Online Survey Methods

Low survey response rates can throw off your entire research project and result in unreliable data. Designing surveys, determining a sample size that meets your research criteria, fielding a questionnaire and analyzing survey data can all take a substantial amount of time and resources. It makes sense then, that you want to do it right, the first time. This means, you need a high response rate.

When you're gathering feedback from customers, employees or another target population, keep these five survey tips in mind to increase response rates.

• Survey length. Different types of surveys have different best practice tips for survey length. But a general rule of thumb to keep in mind is survey questionnaires shouldn't take more than 10 minutes or be more than 30 questions.

• Respondent Fatigue. Keep in mind that survey respondents can get tired of completing your surveys. To combat respondent fatigue, keep your survey questions relevant with varied question types. Seeing the same survey rating scale over and over and being asked to agree or disagree with similar statements gets old fast. Respondent fatigue also comes from asking the same people to complete your questionnaires over, and over, and over, and over... again.

• Question flow. Question flow and respondent fatigue go together. You need to keep your questionnaire flowing in a manner that makes sense. Utilize different types of survey question logic (sub-questions, pipe logic, skip logic, branch logic, link logic, and other types of advanced logic) to improve the respondent experience. Avoid asking questions you already know the answer to, like email address or customer ID. Link what you already know in the backend, instead of asking for it over and over again. If you're worried their information might have changed, select a survey company that allows for pre-population of contact fields so the respondent only needs to verify not re-enter their information.

• Data collection techniques. If you use an outdated data collection method, you may not see a high response rate. I suggest using an online professional survey tool. We've posted many times before about the adoption and reach online survey methods have.

• Data Security. Some market research studies ask questions that are personal in nature. Assure your respondents of the security and anonymity of their answers. Tyson recently shared his thoughts on how to protect survey respondents anonymity and confidentiality.

As is the case with any research method, there are advantages and disadvantages to using an online web survey to collect data about customers, employees or the public-at-large.  For example, online surveys offer you the ability to gather vast amounts of data from many respondents at the same time, get your data back in an electronic form, see real-time results and automate analysis/reporting tasks... and you can do all of this affordably.  You can even use a relatively small research survey sample to accurately estimate the opinions of your larger population (for survey research in general).

But in some cases, it is best to use a “mixed-methods” approach to your research project.  This means you combine the online survey method with another kind of investigation, such as interviews or focus groups, in order to produce more well-rounded data and conclusions.  Here are some examples of when a mixed-methods approach is likely better than an online survey form by itself:

In a recent post, I described the differences between nonprobability and probability sampling methods in online survey designs.  Probability methods are preferred if at all possible, because they allow you to make generalizations from your electronic survey results to a larger population or target audience.  One kind of probability-based sampling technique is called systematic random sampling. 

To employ a systematic random sampling design for your online web survey, you first select a case at random from your exhaustive population list, and then select further cases at identical intervals, determined by how many people you want to sample in total.  If you wanted to sample ten people from a population list of 150, you would then choose every fifteenth person after selecting someone in the first 15 cases (to ensure you will select 10 people in total).

This provides an easy way to obtain a random sample of your population list or sampling frame, because as long as your data is ordered randomly, you can begin simply by selecting any record or case and go from there.  This is an important caveat though: your records must be randomly ordered for a systematic sample to be effective. 

Take this example of survey sampling, let’s say you have a sampling frame (list) of people that is currently ordered alphabetically by last name, and you are interested in subgroup analyses by ethnicity.  It would be wise in this case to rearrange the records into a truly random order (i.e., not alphabetically), because last names from certain backgrounds may be more likely to begin with a particular letter.  While systematic sampling provides an easy way to generate a random sample for online surveys, you do need to be sure there is no hidden order within your population list or sampling frame.

On Tuesday, the public opinion survey results were released by the Wenatchee research firm hired by East Wenatchee and Douglas County. The survey project's goal was to gauge community opinion concerning the development of the Apple Capital Recreation Loop Trail in East Wenatchee, Washington.

According to the survey findings that were released 55% of survey respondents opposed any development along the trail and 15% favored some development, including waterfront shops and restaurants. Unfortunately, the accuracy of the community attitude survey findings are being questioned. They think those who support some development may have been under represented in the survey sample.

The quantitative market research method used for data collection included both paper surveys and online surveys hosed on the More Than a Trail website. The website survey collected survey responses from respondents willing to identify themselves, it was not an anonymous survey. All parties involved in the survey project, according to The Wenatchee World article, agree even though the survey results are not significant there were a huge number of survey respondents. How many is a huge number of survey respondents? 1,249.

Here's how those involved view the survey: like a public hearing. You can only do so much to encourage participation, and 1,249 responses exceeds local government standards. With a public hearing, typically the decision makers only hear from the people who have objections the others don't show up, so they don't get heard. This survey allowed everyone a chance to "show up." The public opinion survey was well promoted with data collection techniques that included publicizing the quantitative research survey on local radio station, newspapers, association chapters, employee newsletters and signs were posted along the trial.

I really like that East Wenatchee is turning to online survey methods as well as paper questionnaires to get an understanding of public and community opinion for issues that will affect, well the community. Their analogy of an online survey being like a public hearing is a pretty good one. But I see where there could be concern about the overall results of their survey questionnaire. It does seem unlikely that 55% of any survey sample or population would be completely against anything - not have various degrees of how much development is too much.

My first reaction would be to look to the fact the website survey was not anonymous. While I understand it's hard to guard against "ballot stuffing" with a website survey online, the fact that it was not anonymous could have deterred people from responding. Why? Well, if they felt their opinion differed from the majority they may have declined to take the survey in case responses were traced back to them. On the same note, survey respondents may have felt social pressures to respond in a certain way. It's a tough call. When you're deciding on your quantitative methods to collect feedback, you need to weight the pros and cons of available survey methodologies and go with what you believe will result in the best, unbiased results.

How many answer choices are too many? Whether you're using a multiple answer or checkbox survey question (Select all that apply) or a single answer or radio button survey question (Please select the statement that best describes your feelings towards XYZ), the number of answer options available for the survey respondent to select matters. But there are a couple things you should consider:


Nothing is more frustrating when a question requires lots of scrolling. Not only do you tend to forget what the question is you're answering, but it takes a lot of time. Depending on the question type (multiple vs single answer) it can be difficult to answer. If you ask, Which of these products are your favorite? and supply the survey respondent with 50 choices, it will be hard to just pick one - they wont even remember them all when trying to decide. At some point respondents will stop reading and pick one so they can move on to the next survey question.

If you are going to ask survey questions which require many answers, make sure they're organized in a meaningful way. For example, if with the help of website polls, I'm using surveys to generate leads for a sales team, I may want to ask a question about which products the website visitor is interested in. It makes sense, in this case, to have a longer list sorted into groups or perhaps alphabetically if you have that many product offerings.

Many demographic questions make use of many survey response options. One that always comes to mind first is What state do you live in? There is a standard for this type of question because it gets asked so frequently. This makes it easy for the respondent to quickly find the correct option and move on.

I can't tell you how many options are too many for your particular survey question. But my suggestion to you is to try to look at it from the respondent's perspective: Would you be annoyed by this list? Could you honestly answer that question? If you were not using an online survey method and instead were creating a paper survey, would the list be manageable in a paper survey?

I stumbled upon an interesting article today discussing the findings of a recent Zagat Survey. Last month, Zagat announced the findings of its Fast Food Survey. Starbucks Corp. was the winner of the best fast-food coffee category. Nate Silver, baseball-numbers analyst turned political-numbers analyst during the 2008 Presidential election, noticed the Zagat results and questioned the online survey method for the data collection. Assuming Zagat used its usual methodology, it wouldn't be a fair survey. The more popular eateries typically get more votes, whether or not they are the best. In this case, Starbucks and Dunkin' Donuts have 7,000 locations compared to Peet's, the 3rd place winner, 200 locations. Many of the people voting in the poll may never have heard, let alone have tried, Peet's coffee.

Carl Bialik, author of the Seeking the Best Coffee in a Popularity Contest post at WSJ Blogs, investigated the online surveyor's method. A little over 6,100 survey respondents were asked, Which fast food chain has the best coffee? and given a list of approximately 30 chains that sell coffee - the most votes won. Starbucks won 38% of the votes, followed by Dunkin' Donuts with 28% and then Peet's with 9%. I agree with Carl, it's quite impressive Peet's managed to snag 9% of the vote while only having locations in six states.

There's an assumption being made here that people only drink what they consider the "best" coffee, but what if they don't know what the best coffee is? I might even wager a guess that many people who don't drink coffee would vote for a company they think has the best reputation. Reputation isn't solely based on marketing dollars alone, but it certainly helps. Starbucks, Dunkin' Donuts, McDonald's (rated 4th best coffee) have a lot more marketing power behind them to promote their products than a smaller company like Peet's. Is this really a fair way to judge the best coffee?

I'm not trying to imply Starbucks has bad coffee, they wouldn't be as successful as they are if the coffee was terrible. I'm just trying to point out how important it is to question survey methodologies when deciding how much faith you want to put in survey reports. You may find the scales were unevenly tipped in one direction and the data is biased.

Want to know who else won the most votes in Zagat's Fast Food Survey? Check out the findings on their website.

Although the Twitter talk has slowed down some, there's still a lot of buzz around how organizations should use Twitter as a resource for customer service, customer feedback, lead generation, etc.

How are you using Twitter?

I think Twitter is a great way to collect customer feedback. I'm still unsure about using Twitter for customer service, because if you do it wrong - it could really hurt your organization's reputation. Make sure you evaluate whether or not it even makes sense for your product. But the most important thing organizations need to understand when it comes to social media is that the conversation is going on - whether you're a part of it or not. It doesn't matter if you have customer survey software and conduct customer questionnaires online or off. The internet has made sharing information with peers so easy, of course it's going to happen and you need to monitor it. Ignoring potential feedback because it's not within your chosen method (ex. surveys online or feedback forms within your restaurant or store) would be silly. With that in mind, don't ever try to control the conversation, customers will not appreciate it and will likely kick you out of the conversation.

So how do you manage feedback and not let it spiral out of control? (See this post about Motrin Moms or #AmazonFail on Twitter for examples.) That's a great question, and I'm not sure anyone has a one size fits all solution. Social media doesn't have a one size fits all solution, each organization needs to figure out their own strategy. For how you deal with feedback, it's the same. You need to set your own rules for what requires action and what does not. My recommendation is if your share of the conversation is small it may be beneficial to take part in as much of the conversation as possible. This means when someone says something both good and bad about your organization. But do not over react to bad feedback. If you get bad feedback, maybe there's a process you need to look at and fix or it's just that someone doesn't like you. If you get bad feedback that you think is unfair, try to follow up in a non-defensive way to understand the problem so you can fix it. Chances are if you solve the problem, you'll receive praise for it, not more hate. Social Media users tend to share the good feedback as well as the bad, which flips the belief that significantly more bad news is shared than good. There's still an imbalance, but it's getting leveler.

My point: Twitter, and other social media platforms, are a great source for customer feedback for customer service feedback to product feedback to any other type of feedback. These resources should be included in your tool box.