When Research Doesn't Work


"Don't let your marketing dollars go up in smoke!"

Steve McKee has a vested interest in making sure companies are doing their research, and doing it right - he's the president of marketing firm McKee Wallwork Cleveland, author of When Growth Stalls: How It Happens, Why You’re Stuck and What To Do About It, and blogger on his company's website. So when he saw this article in the Wall Street Journal highlighting the risks of focus groups (see the second blurb, entitled Neuroscience: Your Brain on Advertising), he was intrigued.

The article briefly summarizes a study performed by three researchers, one from University of Michigan, one from University of Oregon,and one from UCLA. In the study, 31 smokers who were interested in trying to quit were shown three anti-smoking advertisements. The first commercial, Ad A, employed humor to empathize with the smokers' situation. Ad B was equally empathetic, but took a more serious approach; Ad C was "light-hearted". Participants were asked to rate the ads' effectiveness, based on whether they were likely to call the 1-800 hotline number presented at the end of each commercial. When ranking the ads based on their perceptions, participants ranked them, from favorite to least favorite, B, A, C. These rankings matched those of public advocacy marketing experts who were consulted in the development of the three commercials.

This is where the similarities to your average focus group ends. While the smokers were viewing the three ads, the researchers had them hooked up to a machine measuring brain activity, paying special attention to activity in the medial prefrontal cortex, where behavior change occurs. Based on neural activity, the rankings should have been C, B, A -- quite different from what the test subjects told their observers. Lest you conclude that the research must have been performed incorrectly, the ads' were then used in real-life markets, where their effectiveness matched up with those from the neural activity rankings.

So what's going on here? Were the smokers lying to the researchers? Not necessarily. According to McKee, it's possible that the test subjects were lying to themselves, thus unintentionally tainting the results of the market research. This is why it's important to do more than just the standard research -- you have to decide what method will work best for your needs, analyze the results, and analyze the results properly and accurately. Relying too heavily on focus groups can also lead to problems, especially if you are drawing the wrong conclusions from your market research.

For the most up-to-date information on how to get reliable results from your tests or surveys, be sure to follow the Cvent blog! And if you're looking for ways to analyze the results that you're getting and make sure that they're actionable, Cvent can help with that too. 

blog comments powered by Disqus
Crimes in Design Webinar
Subscribe to our Monthly Newsletter