Survey Research

With the emergence of widespread internet usage and powerful online survey software, the web has become the survey method of choice for many project managers and market research professionals.  Utilizing an online web survey design allows for levels of efficiency unheard of in traditional mail questionnaire projects, but it also poses new challenges for business survey researchers.  One of the questions that is asked often is, Is my customer feedback sample and/or survey data skewed because I used an online survey?

The answer could be either no, yes, or maybe, depending upon your specific research issues and target demographics.  The demographic that use the internet (and newer technologies in general) the most are younger people.  Those who are new to online technologies, or those who are not tech-savvy, might be intimidated or confused by a web survey form or an email survey invitation from an unknown sender, which can decrease your rates of nonresponse.  Also, you can run into the issue of coverage error for those who do not have internet access, which is more common among lower-income and less-educated individuals.

But overall, the reliability and validity of online survey results can be on par with or better than other modes, especially if you know your target audience well, including their rates of internet usage.  According to a September 2009 survey from the Pew Internet & American Life Project, 77% of all adults - and 93% of those ages 18-29 - use the internet “at least occasionally.”  Less than half of those aged 65 and older use the internet or send and receive email, however.

So while there are drawbacks to online questionnaire designs of which you should be mindful (as there are with any survey modes, such as interviewer bias, etc.), the efficiency and reduced costs are more than enough rationale for most to employ online data collection techniques, either exclusively or as part of a multi-mode survey design.  Knowing and understanding your target audience is the best way to be sure that your chosen mode will produce quality data.

When conducting market research, either through survey forms or another mode, the method of sending a survey pre-notification letter can be a tempting one.

For some time now survey methodologists have been conducting studies of pre-notification letters to determine if they have any affect, either positively or negatively, on survey response rates. An overall analysis of these various independent studies revealed an increase in response rate of approximately 8% when compared against studies where no pre-notification letter was sent.

No one can say for sure exactly why pre-notice survey letters seem to assist in increasing the number of survey responses, but perhaps they help to establish the legitimacy of a survey, contributing to a respondent's feeling of trust and the credibility of the organization conducting the research. Another possibility is that a pre-notice letter builds expectation of arriving mail (be it electronic, paper, or in-person). A third possibility is that a potential respondent is less likely to disregard the survey when it arrives if they are aware it is arriving shortly.

Although survey pre-notification letters are an excellent (but sometimes expensive) way to increase response rates, they are seldom used in marketing research surveys. Each researcher needs to weigh the additional cost of sending out a pre-notice to potential respondents against the probability of a lower response rate.

It's worthwhile to take your population's survey sample size into consideration when coming to this decision. When your sample sizes are small, each individual response affects your survey results even more. Increasing those responses through utilizing a pre-notice letter may counterbalance the increased cost of sending them out to your participants.

As is the case with any research method, there are advantages and disadvantages to using an online web survey to collect data about customers, employees or the public-at-large.  For example, online surveys offer you the ability to gather vast amounts of data from many respondents at the same time, get your data back in an electronic form, see real-time results and automate analysis/reporting tasks... and you can do all of this affordably.  You can even use a relatively small research survey sample to accurately estimate the opinions of your larger population (for survey research in general).

But in some cases, it is best to use a “mixed-methods” approach to your research project.  This means you combine the online survey method with another kind of investigation, such as interviews or focus groups, in order to produce more well-rounded data and conclusions.  Here are some examples of when a mixed-methods approach is likely better than an online survey form by itself:

Surveys are an invaluable tool for researching the community attitudes, employee concerns, product needs, customer loyalty and priorities held by different groups or target audiences.  Designing a questionnaire and collecting survey responses from a sample allows us to draw a profile of the group as a whole, and perhaps perform some correlation analysis to understand the source of those feelings.  The online survey findings can then support fact-based organizational decisions or improvement projects to help continually improve the organization over time.

Survey research can be applied to many venues.  Here are just a few practical applications listed below:

An Internal Employee Survey could identify reasons for low employee retention and provide ideas for reducing those costs, such as a better designed benefit program, improved training opportunities, or problems in the way the organization functions.

A Training Survey can identify how a training program has improved the capabilities of some group and how the training program itself can be improved.  

A Product Satisfaction Survey can identify initial customer experiences with a product, providing data to address unforeseen problems and help the next product release.  

A Market Research Survey can identify customers needs when creating these new service and product offerings.  Surveys can be part of Design for Six Sigma activities.  

An Association Survey, which is similar to market research and customer surveys, can show the member benefits most of interest.

However, a survey program is only valuable if it is properly designed and executed.  While performing a survey project seems deceptively simple – it's just a bunch of questions, and survey software tools make electronic surveys quick and cheap – a small mistake in the survey questionnaire design or survey administration can skew or bias the data, leading to erroneous conclusions.  No organization should ever make critical business decisions based on unreliable or invalid data.

Bad data is worse than no data!

Designing a survey instrument can be one of the most fun aspects of conducting survey research. In this two-part tutorial, we'll first look at some of the "Don'ts" of writing survey questions, and then we will look at some of the "Do's".

Once the brainstorming of ideas phase has been accomplished and you have a good idea what the scope of your research is, it's time to get down to putting question ideas on paper (or online!).

However creating good survey questions can turn into quite an art form, as we all struggle to word our items in such a way that our participants will understand what information we are asking for. Recognizing when you've accidentally designed a survey question that is worded in a non-objective way is the sign of a good researcher. Let's see if you can spot the poorly worded question from this group of sample survey questions:


Now that we've looked at some question items that shouldn't be put into surveys, next time we can define some ways we can create objective question items that will provide us with the results we need!

Some people still believe in using paper surveys instead of designing online questionnaires, sending personalized email survey reminders and utilizing electronic survey methods. Sometimes I have to wonder, Why? Some of the reasons people claim they want to use paper questionnaires instead of web survey forms is because they think it's cheaper. That's not usually the case.

Consider this: Online survey response rates are typically hire than response rates on a direct mail survey. Online survey research designed well will yeild response rates from 20-40%. Let's be conservative and say the paper survey you plan to send out will get a 20% response rate as well. I looked up the cost of a ream of standard multipurpose paper (500 sheets) on Office Depot's website: $7.09. A black ink cartridge for a laser printer is $77.99 and will print on a maximum of 2,200 pages. Most printed surveys will be kept to three to five pages, longer than that your survey form looks over whelming from the very beginning and is going to lower your response rate. This is how that cost quickly adds up:


Keep in mind these costs don't include the staff time of stuffing envelopes to send out the direct mail survey, or the staff time of manually entering in all of the responses into some sort of data collection software, be it Excel or a survey software tool. You can figure it will take an intern a week to send out 5,000 survey responses and another week for them to do the data entry for the 1,000 responses. So that's an additional cost of $600 to $800 for one survey campaign. What if you want to do a quarterly customer satisfaction survey? You're looking at an annual cost of over $25,000.

For me, that would be enough reason alone to look into an online survey software application. With a web survey tool, you have the ability to not only send one invitation, but reminders survey emails as well. Sending reminders, as we've mentioned before, can drastically increase your survey response rate. My suggestion: don't fall for the myth that paper surveys are cheaper than online surveys. It's definitely the other way around.

There advantages and disadvantages to conducting web or electronic surveys as opposed to traditional survey modes such as personal interviewing, telephone and mail.  One of the advantages of an online survey design is a possible reduction of what is known as social desirability bias. 

Social desirability bias occurs when survey respondents offer responses that portray them in a positive or more favorable manner to others. 

When a face-to-face interviewer asks personal or sensitive questions respondents feel have a “good answer” and a “bad answer” (such as criminal behavior), they may underreport bad behavior and overreport good behavior, for example.  While this bias may be reduced in online surveys due to the absence of an interviewer, there are topics that may produce invalid or unreliable data, regardless of survey mode.  This is something to consider before finalizing your online questionnaire design or web survey forms.

Survey questions within the following content areas are especially subject to social desirability in a survey form (i.e., respondents believe particular responses are “better” than others). 

A key marker of a quality survey, whether that survey is an online form or another mode, is replicability of your results over trials. Replicability of your survey results lends credibility to your organization's research work.

One indication that your survey form may have results based on biased samples is nonreplicable results for the same instrument, fielded with the same survey design, from one time to the next.  If the same sample frame produces significantly different results for identical questions, that should perk up our noses to the scent of possibly unreliable results. We should start to look at the criteria used to select our survey sample, and take a fresh look at the questionnaire to see if it is still pertinent to our needs, or the needs of our clients.

This holds true for all modes of survey research, including online survey internet research. Receiving divergent findings in survey results being fielded again and again could lead to differing (and possibly harmful) business decisions.  No company should be left with uncertainty about answers and actions when decisions need to be made based upon the research. This is just one reason why replicability of your research findings is a key ingredient to sound data. When in doubt- replicate!

When designing surveys, researchers often want to find out how people act in addition to how they feel.  Questions about actions and behaviors are especially useful in market research, since you can gain a sense of what customers and consumers are actually doing (as opposed to what they say they’ll do).  However, this area of survey research can be difficult – to the point of being unreliable – so I urge you to follow the guidelines below when asking behavioral questions in your online questionnaire design:

Provide a specific time reference within your survey questions.  This allows for an “anchor” to be set, and can give you a clearer idea of what your respondents do during an average time frame (be it a day or a week).

Make the time frame fairly recent.  It is very difficult for respondents to remember what they did three years ago.  At the very most, stick with questions that reference 12 months ago or less (preferably much less, like the past week or even 24 hours).

Ask only about the respondent’s own behavior.  Unless you’re asking about the actions of small children, it’s best to keep survey questions directed at the individual level rather than for family members or friends.  Asking people about others’ behaviors can provide distorted, unreliable data, as most people overestimate or underestimate certain actions of other people.

Personal and sensitive information should be taken with a grain of salt.  Multiple market research studies have shown the more sensitive a question is, the less likely it is that respondents will answer... and the data provided can be inaccurate, so it is important to consider that people might dance around the absolute truth on these kinds of questions.

USAA Survey says Shoppers Plan to Use Cash this Holiday Season USAA, the insurance and financial company who we've reported tops customer satisfaction surveys time and time again says shoppers are planning to use more cash and less credit this holiday season. Their insurance survey report shows 85% of shoppers plan to use cash and 62% plan to use more cash this year than last year.

They also found that 56% of consumers have been saving cash throughout the year to spend during the holidays. This jives with other survey research that Americans are, in general, saving more and spending less. Perhaps more important to USAA's financial services business, though, is their data which shows that 74% of those who plan to use a credit card for their holiday purchases plan to pay off the balance immediately. 

On a broader note, a retail survey commissioned by the National Retail Foundation reports overall holiday sales will be down about 1% and that two-thirds of families have adjusted their holiday plans due to the economy.

Online retailers continue to be the bright spot, however, with sales expected to increase about 8%, according to Forrester Research. 

It is tempting to include many similar question types with similar response options in your online survey design.  Matrix questions, for example, provide an efficient questionnaire design method to help you gather lots of data in a neat, brief survey form.  It is wise, however, to resist the urge to use too many uniform survey questions and response lists, namely because of two sources of bias that stem from doing so: habituation and acquiescence.

Habituation occurs when respondents begin providing the same answers to survey questions with the same response options.  They start to get in a habit and select the identical response choice for every question.

Acquiescence is related to habituation, and occurs when respondents passively agree with an interviewer or survey questions.  Agree-disagree scales are the most often-used response options in opinion surveys; it is important that you take steps to avoid the chance that respondents will passively agree with your statements in order to quickly complete the questionnaire or provide what they think may be the “right” answers.

To avoid these response biases, you can use online survey software that allows question randomization, break up your matrix questions with other types of questions and scales, and phrase some questions in a manner that makes respondents switch their thinking.  An example of the latter would be to ask a series of positive questions in your survey questionnaire, and then throw in a couple questions worded differently so as not to allow habituation or acquiescence.  Use care up-front in your online questionnaire design to be sure that you'll reduce error and bias in your results.

In the past several years, many market researchers have shifted to using online surveys more frequently. Traditionally, these online surveys were emailed rather than posted online to avoid data skewing.

When creating the online survey design, there are several things to consider. Make sure the electronic survey is simple and easy to use. Too often, online survey forms are cumbersome or technologically dated. Keep it simple, and you won’t run into any problems.

You must also plan carefully to make sure no questions are missed. You can design online surveys so they will not allow survey respondent to move on if mandatory questions are not completed. In addition to requiring survey questions, I would advise setting up the internet survey so respondents cannot complete the questionnaire more than once. Most online survey research software applications have features to create limits on how many times someone can fill out an online survey form. This prevents skewed data – some respondents will try to complete online survey forms multiple times if there is a giveaway or drawing offered as an incentive.

Make sure you read over your electronic survey a few times as well. Proofreading is especially important when designing online surveys. You must have members of your team proofread not only for grammar and clarity, but also functionality. Make sure all the links work, and the survey can be completed with all your caveats and mandatory questions. 

Online surveys have made it possible to gather information much quicker, and when used properly provide an excellent resource for market researchers.

All too often, we begin the survey creation process, but draw a blank.  How should we design a survey?  Should a survey design have the company look and feel?  Or should we design it to be a fresh look?

Having an online surveys tool that has survey templates built into the system is extremely important.  It eliminates the nagging question, "What is a survey design? How do I create a professional looking web 2.0 survey?"  This gives the survey builder that boost to get the creative juices flowing. In the Cvent Web Survey solution, there are over 50 graphical templates, ready to help you in designing surveys. Any color, any pattern, any style. They are all preloaded in the Cvent Web Surveys application. Definitely check these out the next time you are trying to figure out survey research design ideas for your next questionnaire.

Want to learn more about our survey design software? Sign up for a product demonstration.

Within the survey research process, a "Thank You" note is something that is sent a few days to a week after a questionnaire is sent out to your clients. This mailing expresses appreciation for responding and indicates that, if the client survey has not yet been completed, it is hoped it will be soon.

The postcard or email survey "thank you" note is not written in order to overcome respondent doubts or fears about taking the survey, rather, it is meant to jog memories and help re-arrange priorities, bringing your potentially forgotten survey back to the top of potential respondent's "To-Do" pile.

If timely and appropriately worded, the postcard "Thank you" note can arrive just in time to make an appeal that engenders a sense of importance about survey participation, without crossing the line into sounding impatient.

When possible, attempt to have your "Thank you" note be as different as possible from your original survey invitation request for participation letter. The goal is to create a stand-out piece of paper or electronic mail that contrasts with others, creating new stimuli for your potential respondent.

Utilizing these survey tips in conjunction with adding a "Thank You" note to your survey project outline are just some of the ways you can ensure your products are recognized as standing out from the crowd, especially in this all-too-hectic world.

The vast majority of survey research projects are studies at a single point in time of a specified population, such as employees, customers or the general public.  Fewer web survey designs track opinions over time.  This post outlines the different types of surveys carried out by researchers.

Point-in-time surveys are called cross-sectional studies.  They study a single population or sample size during a single specified time-frame, and give us a “snapshot” of opinion data.  Cross-sectional surveys comprise the largest number of projects that are undertaken. 

Longitudinal surveys, on the other hand, are those which study trends over time, and usually consist of cohorts or panel respondents.  These can be further classified into three distinct types of longitudinal designs (trend, cohort and panel).

Completing a study is just the first step in the market research process. A good researcher knows there is plenty of work left to be done analyzing survey data and taking action on survey results.

Stay in touch with the client and schedule presentations of the research. If you have a client contact assigned to the project, go over the preliminary survey data to see what key findings they feel are the most important to share. Create presentations around these key findings. You can also create separate market survey reports for different groups within the same company.

For example, if you have just completed a large product survey that includes feedback on customer satisfaction, the client may want a presentation simply on customer feedback for their customer service team.

A good researcher also has an eye for finding problems. If you notice an area for improvement in the course of your market research analysis, propose a solution to that problem for the client. Take this market research survey example, customers complain about being on hold too long when calling customer service. You could propose several ideas from hiring more staff to setting time goals for staff to talk to customers.

Even though the client may not agree with your solution ideas, ignoring problems you identify through the research is a bad idea. The client is paying you to compile and analyze research data, and they will likely appreciate all your efforts even if the information does not fit in with their current business plan.

I always recommend probability-based survey sampling techniques wherever possible.  Sometimes, however, companies and organizations want to get an initial feel for how consumers and customers will react to a new product or concept.  In addition, early in the process, you may not have the ability to comprehensively identify a target market or sampling frame, and there is no way to produce a representative sample of your population. 

In these instances, it may be useful to employ a snowball sampling technique as a pilot project, or to gain a rough, early grasp on what customers are feeling.  A snowball sample is one in which you use an initial group of respondents as recruiters for additional market research respondents.  In the survey, you ask your original respondents to list several people  they know that might be interested in completing a survey as well.   This is a case where an incentive might prove particularly useful, since you are asking your market research survey sample to provide contact information of their acquaintances.  Snowball sampling is also especially useful if you do not have a predefined list of people to survey, or if you are trying to identify key information-holders or opinion leaders.

Again, there is a significant caveat of snowball and other nonprobability-based business research methods for sampling techniques: they do not produce representative samples, and therefore cannot be used to generalize findings to the overall population.  But if you are just starting out, and do not mind that your market survey sample cannot produce generalizable findings, then a snowball sampling technique is a convenient survey data collection method to obtain larger amounts of preliminary data.

When performing a market research study for clients in certain marketplaces, it is important to remember the competition. If, for example, you were doing a customer market research survey for a fast-food restaurant, Burger One, you would want to gauge survey respondents’ feelings about the competition as well as your client.

You might ask restaurant customer satisfaction questions like:


If respondents frequent Burger One and four other fast-food restaurants, that gives you insight into the survey data provided. If respondents only frequent Burger One, that is helpful to know as well. Take this example of how survey results can be applied to the restaurant's over all marketing strategy:


On the other side of this equation is what Burger One's loyal customer base thinks. If customers who only eat fast-food at Burger One express no interest in the new sandwich, it may not be the best possible sandwich to launch. However, if respondents who frequent competitors more often than Burger One express high levels of interest in the new sandwich, it could spell an opportunity for Burger One to gain new business.

It’s best to analyze market research from all angles when launching a new product – there are multiple factors at play.

Just because you find correlation in your data when analyzing survey results does not mean there is causation. I find this is a common mistake in lots of survey reports when someone is new to survey research or conducting data analysis. Take this example:


Some people see this connection as a causation. Customers are satisfied because you contact them frequently to make sure everything is going well. The problem is, it's not a causation. Causation are extremely hard to prove because you cannot control every factor. For example, you may split your territory by industry and your solution suits some industries better than others. Or clients who are really satisfied simply use the product more often so the customer success managers reach out to them more frequently, because they are more likely to have questions. While those who use the product less have less to be satisfied about and may feel they are wasting those budget dollars.

To be able to prove causation, you need to be able to rule out all other possible explanations for the connection. As you can imagine, that's almost impossible to do since we do not control outside factors influencing the survey respondent or even the greater survey sample. Instead, when situations like these occur, we're seeing a correlation between two things. In my customer survey question examples, there's a correlation between how satisfied customers and how often they are contacted.

This principle does not only apply to customer survey research, it also applies to analyzing employee feedback forms, product surveys, market research and any other type of data collection and analysis.

Did you know there are 6 steps in the market research process?  While this process speaks directly to marketing research professionals, the process applies to HR, customer or education surveys as well:

1. Identify and define the problem.  Before you start any web survey project, you should identify the key issues you hope to be able to solve.  This step should also include clearly defined objectives.
    
2. Develop the approach. In this step, you need to establish a budget, understand influencing factors such as the environment or economy, decide on sampling and survey methods, and formulating hypotheses.
    
3. Research design. Designing a survey or questionnaire is considered the most important step in any survey process.  Question design takes a lot of thought and time.  We like to say, "If you put garbage in, you'll get garbage out."  This means that if the questions are bad, the data will be bad as well.  During the survey research design, keep in mind sampling methods and data analysis factors you intend to use.
    
4. Collect the data. Don't forget to test your survey before to ensure you're fielding the correct data.  Thankfully, with the help of an online survey tool, this step is relatively painless.
    
5. Analyze the Data. The types of analysis you planned to perform on the collected survey data should have been decided in earlier steps, but after collecting the data you have to actually perform the survey analysis.  Analysis can be performed using survey analysis tools like office programs, such as Excel, or more advanced programs such as SPSS - the complexity of the questions will determine this.
    
6. Report, Present, Take Action.  The final step in the market research process is to present your survey research findings and draw conclusions.  While Step 3 is the most important because it defines the outcome of your survey, if you fail to complete this last step and act on the findings in some way, the previous steps don't matter. 

As I mentioned in the beginning, this same process can be applied to any type of project: product evaluations, customer satisfaction questionnaires, public relation surveys, etc.  If you give each step the attention it deserves, each of your online surveys should be a success.