Trim Tab

Focus group testing is appealing. I really do get it. In theory, it’s a quick and efficient way to find out what you need to know about what people think or how they use your product or service, but I’m afraid my experience has confirmed what Malcolm Gladwell says about focus groups: “There are consequences to asking people how they feel or what they like. The consequence is, you’ll be led to a false conclusion, because making people think about their preferences makes them stupid.”

The-Problem-with-Focus-Testing-2

Focus group testing isn’t quick or efficient. And it has a dangerous inclination to lead you down false pathways. This photograph of my daughter was the closest I could get to a metaphor for focus groups: they can lead you into the bushes, and when people start thinking too much about what they think they want—especially in the presence of others—they really can become unsure of whether they should be wearing sandals or roller blades.

Cutting to the chase, here is a quick account of the problems we have experienced with focus group testing (there are probably more):

1. Expense

Focus group testing is expensive, relative to other better qualitative discovery methods (see below).

2. Bias

Focus group testing is highly susceptible to bias and suggestion.

You can try and ensure that the questions aren’t leading ones (though it’s hard), but you can never truly eliminate the suggestions and influence of the other participants in a focus group.

3. Focus group testing needs focus

The questions you think you want answers to are often not the questions that really need asking. The real questions are very hard to obtain, and are usually not clearly understood at the outset, so if focus group testing is done early its findings are often superficial.

4. Artificiality

The format of a focus group is not analogous to the actual decision making process, which is more personal and often not shared, and certainly not explained by the decision maker.

5. Moderator & peer influence

We are intensely social creatures, and our ability to detect and respond to the desires of a moderator or a conversation partner are actually quite remarkable. This same ability makes us quite unreliable if someone is  trying to understand likely real-world behaviors based on findings from the particular un-real world social environment created by a focus group.

Here is an absolutely hilarious illustration (thanks to Jimmy Kimmel) of how willing people are to say more or less anything in a public social interaction. It matters not whether they actually have any first-hand knowledge of the subject. It does clearly matter, though, what they think the questioner wants to hear, and neither their true feelings nor their actual likely behavior are illuminated.

6. Stupidity

As Malcolm Gladwell has pointed out, people become stupid when they have to explain why they do things. 

The majority of decisions are made emotionally and intuitively; rationalization and the articulation of reasons through speech is secondary (please read Brain Science and Marketing for more on this topic). When people are asked to rationalize why they behave or believe something, they are actually much more likely to make a different selection than they would have if left to their own devices.

7. Who are the subjects?

People willing to sit through a focus group are a self-selecting group. There are a lot of problems built into this that are difficult to unravel. People willing to do a focus group are a special a category of person and are likely not representative. Also, if your goal is to truly understand something, the “non-believers” are just as important as the brand evangelists (and their friends).

8. Shallowness

What truly matters are the deep emotional and even instinctual reasons that a decision is made. The focus group format is not conducive to getting at these critical insights.

What truly matters are the deep emotional and even instinctual reasons a decision is made. The focus group format is not conducive to getting at these critical insights. Tronvig GroupRelevant to both stupidity and shallowness above, I am reminded of the University of Virginia study “Introspecting About Reasons Can Reduce Post Choice Satisfaction” experiment, conducted by Dr. Timothy D. Wilson and colleagues. This experiment used fine art posters and humorous posters (think kittens) to see if people would make different choices if asked to explain their reasons. The subjects chose the art posters 95% of the time if they did not have to explain why, but this number fell to 64% when they had to give reasons. And those who chose kitten posters were later unhappy with their decision. Asking people to think about their decisions, therefore, creates an artificial mental process that is misleading almost by its very nature. If the poster experiment were a focus group, the researchers would have had to conclude that more than a third (36%) of the audience liked kittens when in fact it was really only 5%. The reality would have been that they only liked kittens DURING the focus group.

Here is how Professor Wilson and company summarize their findings: “The present findings add to the mounting body of evidence that thinking about the reasons for one’s preferences can alter decision making in nonoptimal ways.“ This academic understatement reads to me like a bullet to the heart of focus group testing.

So what to do instead?

Our preferred discovery and research methods include the following:

Persona Modeling

This discovery method, as outlined in Customer Personas (What Sally Can Show You) is a long interview that we use to ascertain the underlying motivations that drive consumer behavior as it relates to a brand or product. In persona interviews we have found that if the interview subject attempts to model someone other than themselves—creating a “persona” which stands in for a certain segment of the target audience—the interview goes deeper and can uncover more fundamental motivations than if the interview subject is simply discussing their own opinions. In effect, they are able to engage in deeper introspection because the fictionalization of the interview has the effect of removing some barriers to revelation. We find that this leads us to a truer and more psychologically profound depiction of the persona’s fears, needs and desires as they relate to a product or service. Our experience with this process has shown that even a slight distance from themselves allows a more candid conversation to take place, and facilitates deeper insights as we dig into what really matters to the consumer.

Behavioral Observation

We watch people interact with the product or service on their own terms in an undirected setting, no guidance, just observation. It is important to understand existing behavior. We are not in the business of advocating that our clients try to change people’s behavior, but rather we help them to understand how their products and services can be made to better interact with existing consumer behaviors.

Long Conversational Interviews

This is not an impersonal interview, but a series of questions that mix the factual with open-ended and personal questions. It requires a skilled and personable interviewer, but it facilitates an intimate and honest conversation. Typically the interview subject is very happy about the interview despite its length, and very often new insights can be extracted from such conversations.

Surveys

We also devise and conduct surveys in order to pull in larger data sets. These are useful for both qualitative and quantitative questions that are difficult to extract from the smaller samples above.

Case Study (for those who might be considering focus testing right now)

Here is one real life example that demonstrates with considerable clarity the relative effectiveness of one of our suggested tools, Persona Modeling, compared to focus group testing. In this case the focus group testing was conducted by a highly respected professional research firm. The Persona Modeling was derived from our own custom two-hour persona interviews. We were asked to integrate the findings from the focus groups into our findings for the same target audiences.

Focus Group Testing Ban, Tronvig Group

The comparison is stark. Above is one of our competitive advantage diagrams, which illustrates the needs of a particular segment of a client’s target audience (represented by a candidate persona) and demonstrates how and if those needs are being met by our client, and whether or not they are uniquely met by the client or by competitors as well. The diagram shows all the specific factors mapped (we’ve left out the factor details in this illustration). Our persona interview factors are shown as numbers, and all the focus group testing factors are shown as letters. We usually do a number of persona model interviews as part of this process, and the relative difference in the findings proved to be about the same in every case. To summarize, the focus groups yielded about 1/3 the information despite costing significantly more.

But that is not all, of critical importance is that the focus group testing also generated false positives, giving the strong impression that certain candidate personas were genuinely viable marketing targets when, in fact, they were shown by our findings not to be. Marketing action based on focus group testing alone would have likely been a very costly mistake.

Finally, the focus group testing was not able to give clear direction regarding which other groups were important for the marketing effort. So focus groups, in our experience, tend not to give clarity where it is most needed, and can spit out false positives. This is very damning. While our discovery makes clear the questions that need to be answered, and allows for carefully focused research to proceed, focus group testing is more likely to burn up money and muddy the water.

‘Nuff said.
a-helpful-starter-guide-to-thinking-like-your-persona-button

___
Photo by James Heaton

Please feel free to share your experience one way or another on focus group testing.

Related posts:
What is marketing strategy?
Find the Pain: Business to Business Marketing
Customer Personas: What Sally can show you.
Brain Science and Marketing: Role of the Limbic & Reptilian Brains

2 Responses to Focus Group Testing Ban (and some better alternatives)

  1. Eduardo says:

    Oh! This is so interesting. I was convinced that Focus Groups were the best way to get real information. Now, you’ve opened my mind. But, I have a doubt: the “Customer Persona” discovery method is applied one-by-one? Or just applied to a person? How can my company can get different views from several people? I ask this because Focus Groups get information from many people, and that´s an advantage for many brand managers.

    Congratulations on the items and services provided by your company that many businesses need today. I´m from Mexico. This is why my English is not very fluid. Sorry!

    • James Heaton says:

      Thanks for the questions Eduardo.

      The persona interview method is actually part of our Discovery Process, and we always interview more than one person, often many. In our full process the persona interviews clarify what the most important questions are that we can then follow-up with research. There are also cases when the insights extracted from the persona interviews themselves are clear and compelling enough to be actionable.

      If X dollars will get you, say, focus groups with 30 people on one hand, or persona interviews (properly done) with 6 people (who represent all the types of people you suspect could be your ideal consumer), then I can assure you the information you can obtain from the persona interviews will be more useful. That said, an extremely well done focus group is probably better than a bunch of really badly done persona interviews.

      The key difference between these two methods is depth. A persona interview has to get past the surface. It has to draw out the emotions that are actually driving behavior. This is where focus groups fall short. A focus group’s structure simply does not allow you to get past the surface.

Leave a Reply

Your email address will not be published. Required fields are marked *


*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>