Tips for Collecting Feedback

Online news sites are abuzz this week with a new study from Carnegie Mellon University which shows that analyzing Twitter data can yield the same results as conducting a public opinion poll. As a survey methodology, it probably also costs less, takes less time and annoys fewer people. But it hardly sounds a death knell for more traditional research methods.

The study claims that by analyzing text in billions of tweets, it’s possible to create an algorithm that accurately gauges things like consumer confidence and presidential approval. So while it may be a great alternative for media outlets to create news from macro opinion analysis, it’s a long journey (and leap of faith) to believe that the same algorithms could be applied to a specific question that a company or organization needs to gauge the answer to.

For one thing, Twitter is more like a thermometer rather than a barometer. It’s great for measuring the temperature or “heat” of a topic, but it can’t really “put a pin” on an opinion score, nor create a benchmark for a change in opinion over time. When Twitter can predict a change in opinion or measure a change in opinion (like a barometer), its utility as a research tool will increase tenfold.

Secondly, because it’s about heat, when the initial rush of conversation dies down, so dies the Twitter talk, although opinions still remain. The Carnegie researchers had to adjust their algorithms to take the temperature of opinion over several days to adjust for the frequency of tweets.

A third caveat is that Twitter users represent a relatively skewed sample of the population. Those who tweet on a regular basis form only a fraction of the overall number of Twitter accounts. (Most Twitter accounts are dormant and many only follow.) Sample bias is definitely an issue to explore if a demographic breakdown of active Twitter users could be conducted.

The researchers admit that the methodology is a long way from being commercially available. So for the moment, marketers and business people will need to be content with online, print, telephone or mall intercept studies and focus on the basics of sampling methodology, questionnaire design, response rate incentives and life-cycle analysis of results.

Sorry Twitter, your time has not yet come in the research world!