Focus groups, the beloved method of many British and American marketers, are under attack in the US. An article in New York Magazine drew up a long list of focus groups run in Manhattan.
They included groups for people who grew up in Long Island, victims of severe asthma, travelers who had back-packed through Mongolia, people of Italian descent, Johnnie Walker drinkers and sufferers of profuse, gland-related perspiration.
The catch was that there was one person, Will Leitch, the author of the article, who attended all these groups, but had absolutely no legitimate reason to do so because he had no experience of the topics.
Leitch has learned how to get on the call-sheets of many of Manhattan’s leading research agencies. By lying in the initial qualification interviews about his background and how many groups he had attended, he was able to earn as much as $300 (£214) a session, up to four times a week, as a professional focus group participant.
For years, many marketers have whispered about the single biggest weakness of focus group data – the validity of those being focused upon. Finding participants can be a difficult task, and research agencies can be less than stringent in selection procedures. There is a growing army of semi-professional focus group participants, particularly among retirees, who will happily give their assent to any and all pre-screening questions to pick up some extra income.
Focus group veterans are bad for insight because they usually agree with the moderators’ questions and can embellish or invent experiences to fit the bill. By doing so they will be invited back for more sessions and more rewards. Leitch even describes the ‘cardinal rule’ of focus group faking: don’t offer your opinion, just confirm whatever they want you to confirm.
The article could not have come at a worse time for focus group adherents.
Big players such as Procter & Gamble and Unilever are steadily moving away from this technique toward more valid and insightful qualitative research methods. In particular, ethnography – the study of real consumers with real peers in real locations, engaged in the actual purchase or consumption of products and services – is growing in influence in marketing departments around the world.
Leitch’s story has caused outrage in the US. Several large research organizations, including the Market Research Association, have drafted a press release condemning the actions of the author. Interestingly, they do not censure the agencies that compile lists of participants then use them for different clients with very different target consumers.
Even after his article appeared, Leitch continued to receive calls from agencies that had his details on their lists. He was soon back at work offering heartfelt, on-message, entirely fictional views on a variety of new products. He even overheard members of one of his focus groups discussing his article. ‘The piece was funny,’ recalled one of the participants. ‘But man, I wish he wouldn’t tell everybody’.
Fortunately, in the UK nothing like this could happen. Market research agencies do not keep directories of regular, semi-professional focus group attendees who they call on again and again. They are also extremely rigorous in ensuring that participants are who they say they are. Right?
Courtesy of Marketing Magazine
The Blake Project Can Help: Accelerate Brand Growth Through Powerful Emotional Connections
Branding Strategy Insider is a service of The Blake Project: A strategic brand consultancy specializing in Brand Research, Brand Strategy, Brand Licensing and Brand Education
7 comments
Bruce Humbert
March 3, 2009 at 8:24 am
For years I have been focus group wary – one key to getting value from this – or any research is to get a firm grasp of the context – be behind the glass – watching body language and listening to all that is being said – you cannot get the same insight from watching or listening to tapes.
But if you think Ethnographics is “better” – more “insightful” inherently I have a bridge that I would like to sell you.
A first principle that is worth remembering in all research – quant or qual is that OBSERVATION effects OUTCOME – the Heisenburg uncertainty principle if you will. There are no perfect measures – NONE.
That is why good research requires smart and experienced people to manage and interpret findings.
Jeffrey Henning
March 3, 2009 at 9:29 am
Great post! This was the first time I had heard the history of the focus group. Regarding its future, you left out online communities, which are also cannibalizing some focus-group activity.
For facilities, who obviously must do more, the easy way to prevent false names is to check IDs and to give out cheques instead of gift cards or cash.
Nicolle Zapien
March 3, 2009 at 10:55 am
This article points out one of the potential risks to quality sampling that have been debated for decades by professional researchers and their consumers. Essentially, focus groups (like any other research) can be of particularly poor quality if recruited poorly. Researchers and sample providers have been asked to assure (to the savvy research purchaser) that the sample that is provided is valid, is sufficient and is not “overused” or “filled with professional respondents”. Any reputable sample provider has ways of assuring that this doesn’t happen most of which includes maintaining a huge database that tracks all response from all studies and kicks out anyone who says that he or she is 18 in one study and then 35 for another, or anything else that is contradictory. Further, most reputable firms only invite the same respondent once every 6 months and controls via the phone numbers and IP address for anyone who seems to be sending screening responses from multiple phone numbers or email accounts.
This is why I only do business with folks like Survey Sampling, Socratic Technologies or Greenfield Online, or reputable focus group recruiting firms in the cities that I work (via the Greenbook). It’s more costly but the entire validity of the research is on the line. Normally you can tell the quality of the sample by the price tag. One can purchase a database of 1 million consumers to survey for under $100 these days and use a bulk emailer to avoid being reported to the spam police. Obviously nobody is controlling the quality of that list if I can purchase it and send a study to them and you can too – on the very same day!. But quality consumers for a focus group can cost several hundreds of dollars to recruit – each one. The fees include the database management, the rescreening (calling the consumer back pre-focus group to do a validity check), and constant replenishing of the database that is sample balanced to match the population to protect the pool of potential respondents.
Further, this is one of the reasons why quantitative methods and ethnographic or other phenomenological methods are being used instead of or in addition to focus groups these days to make solid business decisions (and why I tend to use focus groups only in the exploratory phase or when clients request them). We can control for who is in our sample better and get either more depth (ethnographic) or more breadth of data (quantitative) thus controlling for outliers via sheer quantity. This isn’t always appropriate though. For example, if we don’t yet fully know who the target market is or there is any disagreement on who that target is, one can’t accurately recruit “customers” for ethnographic research and it isn’t responsible to make such assumptions and recruit a quantitative pool of who we guess to be the target. A responsible quality recruited focus group makes a lot of sense then, for an exploration of who the target might be. And follow up research quantitatively or using other more robust qualitative methods may be warranted.
The professional marketing research community thoughts on this can be found in: CASRO, MRA, IMRO, Survey Sampling, Quirks and QRA. This article is pretty alarmist and not terribly helpful in directing people on how to use quality sampling services or to know what quality is, in my opinon. Further, this is why one shouldn’t DIY your research and should use a professional. That professional can help you to find the right methodology the correct sampling strategy and sources and assure that you aren’t polluting the sampling pool with your study. Never using focus groups and only using ethnographics won’t get all the qualitative research done well either.
Nicolle Zapien
Brandon R Allen
March 3, 2009 at 4:12 pm
Do focus groups even work? It seems as if there have been a few studies lately about the subject of how customers respond verbally to questions about products and how they truly feel and act. This subject was covered in the book “Buyology”. It seems to me that focus groups aren’t the most effective way to get accurate information about what your customers truly care about.
Gabriel Rossi- Branding
March 3, 2009 at 8:04 pm
Hawthorne effect — people act differently when they know they are being observed.
Marty Neumeier has made a great point in his book “The Brand Gap”…
Great article.
Gabriel Rossi- Brazil
Hoo See Kong
March 12, 2009 at 12:40 am
An interesting discussion! This should not be the fall of focus group, but it does trigger the need to have an improved recruitment process. In Thailand, some MR agencies check the respondent’s ID (based on Identity card) during recruitment. This is doing fine for those MR agencies within their own record on respondents. However the “professional” respondents start to appear in the other agencies, and slowly moving to those agencies not adopting the ID screening. This issue was brought to the attention of Thailand Market Research Society (TMRS) to look into a QC standard to have a list of respondents from ALL qual MR agencies in Thailand for validation.
Sheena Ewing-Brown
March 31, 2009 at 11:38 am
I was interested to read this article, especially the line about how this would never happen in the UK.
Having lived in the UK for several years, my husband and I managed to build our holiday fund quite nicely by attending various focus groups. In fact one such focus group we were told to pretend that we did not know each other.
Unfortunately, like in most industries, people learn to work the system to their own advantage.
Thanks for the articles, I love reading them and they often make me homesick for London.
Comments are closed.