Market Research Design Tips

I’ve previously emphasized how important it is to consider survey reporting not only after you’ve collected data, but at the beginning and throughout a survey project.  Another central tenet of good survey reporting is tailoring your report or presentation to your audience.

The formatting of your market research reports and presentations can vary dramatically depending upon who is going to receive them.  Even the content will differ based on the audience(s) who will read the report or view the presentation.  Here are two big questions to ask yourself about your audience, so you can tailor your reporting tasks accordingly:

1) Is this a lay or technical group of people? 
The more professional your audience is, the more technical you want to be when describing the results.  For example, if you are sending survey results back to respondents, you probably don’t need to go into statistical and methodological detail.  If you are presenting to market research colleagues, on the other hand, you would go into fairly deep detail regarding topics such as statistical significance, margin of error/confidence level, as well as include prior research citations and an abstract of your project.

2) Am I presenting this to senior officials and/or executive decision-makers?
Usually, the higher-up the ladder your audience is, the less time you spend on detail.  So if your answer to the above question is "yes," I like to use a five-minute rule: if someone had to make major business decisions based on your survey results, what could you present in five minutes that would help them make good strategic decisions?  What would be the “take-home” message (i.e., two or three data-driven recommendations based on your results)?

Although surveys are usually single point-in-time snapshots, it’s good to draw basic conclusions unless it would be inappropriate to do so... after all, you’ve likely spent considerable time and effort for the survey project to generate useful insights!

With the emergence of widespread internet usage and powerful online survey software, the web has become the survey method of choice for many project managers and market research professionals.  Utilizing an online web survey design allows for levels of efficiency unheard of in traditional mail questionnaire projects, but it also poses new challenges for business survey researchers.  One of the questions that is asked often is, Is my customer feedback sample and/or survey data skewed because I used an online survey?

The answer could be either no, yes, or maybe, depending upon your specific research issues and target demographics.  The demographic that use the internet (and newer technologies in general) the most are younger people.  Those who are new to online technologies, or those who are not tech-savvy, might be intimidated or confused by a web survey form or an email survey invitation from an unknown sender, which can decrease your rates of nonresponse.  Also, you can run into the issue of coverage error for those who do not have internet access, which is more common among lower-income and less-educated individuals.

But overall, the reliability and validity of online survey results can be on par with or better than other modes, especially if you know your target audience well, including their rates of internet usage.  According to a September 2009 survey from the Pew Internet & American Life Project, 77% of all adults - and 93% of those ages 18-29 - use the internet “at least occasionally.”  Less than half of those aged 65 and older use the internet or send and receive email, however.

So while there are drawbacks to online questionnaire designs of which you should be mindful (as there are with any survey modes, such as interviewer bias, etc.), the efficiency and reduced costs are more than enough rationale for most to employ online data collection techniques, either exclusively or as part of a multi-mode survey design.  Knowing and understanding your target audience is the best way to be sure that your chosen mode will produce quality data.

Quota sampling is frequently used in survey designs, and especially in market research projects.  This technique is a form of "convenience sampling," where respondents are chosen not at random, but because they are available or easier to reach.  A probability-based sampling design is not employed, due to decisions made by the researcher based upon various reasons: the population frame cannot be known, contact information for respondents is unavailable, or even because the time, effort and costs are simply too high for the budget.

Quota sampling is a way that you can gather completed questionnaires, producing adequate amounts of data, from people with different demographic attributes.  Often, market researchers want to ensure they get roughly equal amounts of data from males and females, may be interested only in a specific age range (i.e., their target market/demographic), or would like to know if preferences differ by other characteristics such as ethnicity and income level.

So where does the “quota” come into play?  Well, just as in stratified sampling, the population is divided into mutually exclusive subgroubs, often based on demographic characteristics.  The researcher sets a quota for each subgroup (100 females and 100 males, for example), collects data until the quotas are met, then stops data collection and begins data analysis.  The reason that quota sampling is not a probability-based sampling technique, thereby limiting your ability to generalize, is because respondents are not selected at random.  Quota sampling does go a step further than simply selecting whomever is available without regard to any criteria, and that's why it is used so often.

As is the case with any research method, there are advantages and disadvantages to using an online web survey to collect data about customers, employees or the public-at-large.  For example, online surveys offer you the ability to gather vast amounts of data from many respondents at the same time, get your data back in an electronic form, see real-time results and automate analysis/reporting tasks... and you can do all of this affordably.  You can even use a relatively small research survey sample to accurately estimate the opinions of your larger population (for survey research in general).

But in some cases, it is best to use a “mixed-methods” approach to your research project.  This means you combine the online survey method with another kind of investigation, such as interviews or focus groups, in order to produce more well-rounded data and conclusions.  Here are some examples of when a mixed-methods approach is likely better than an online survey form by itself:

One of the best descriptors attributed to good survey questions is the word specific.  A main goal in designing valid, reliable survey questions is doing everything in your power to make them clear, standardized and unambiguous.  A great way to follow through on that is to make sure your questions are as specific as possible.  The degree of specificity affects how people interpret and respond to your survey questions.  Several examples are listed below to help you analyze your survey questions regarding their level of specificity.

Be clear with demographic questions.  Your objective here should be to use words and categories that your respondents can clearly understand and identify with.  Will you use Census designations for Ethnicity?   Exactly what do you mean by “marital status?”

Define vague concepts, words and phrases.  Your respondents will likely widely differ with regard to their backgrounds, experiences and perceptions.  Words like “justice” and “equality” can mean very different things to people, so it is best to define exactly what you are asking about.

Objective or subjective?  Use verbs that trigger respondents.  Attitude survey questions are usually subjective (i.e., how do you feel about the war?), while behavioral questions are usually objective (i.e., what did you eat for supper last night?).

Always try to attach a time frame to behavioral and recall questions.

For market and product research, identify the actual brand name, and ask how they feel about specific items, not just groups or genres.  For example, don’t use the vague word “furniture” if you’re really interested in how they feel about a table or a chair.

Survey designing software can't look at your questions and tell you if your online survey form will give you the data you want. However, it does make it easier when you're designing a web survey because it takes out the added complications with writing your own code. So make sure next time you're working on an instant survey form, you think about the specificity of your survey questions.

There is sufficient evidence from prior studies suggesting it is a good idea to keep respondents informed of their progress during internet surveys.  Respondents may suffer from fatigue or irritation, and may abandon web survey forms – even if they have only a few questions left to complete the questionnaire.  This may lead researchers and project managers to adjust their online questionnaire design by reordering questions to include “important” or sensitive items earlier, possibly causing more survey respondents to abandon mid-stream.

There are various survey design techniques that can be employed to keep respondents informed of their progress, especially within electronic surveys.  One way is to design web surveys so the entire survey web form can be viewed on a single page; but while this allows respondents to scroll down and see the total length of a questionnaire, this setup is less than ideal. 

Another method is to include words or symbols in headers and transitions that notify progress (such as section numbers, the words "next" and "finally," etc.)  By far however, the most popular and effective method of keeping respondents in the loop is to include a progress bar. 

A progress bar is a graphic or set of words that let respondents know how far along they are in the survey process.  Progress bars are especially useful for shorter, instant surveys, since answering only a few questions will show that they are through a significant portion of the questionnaire.

Cvent Web Surveys software makes it easy to include a progress bar as part of an effective online survey design.  You can even select among three options (percent complete, page x of y, or a bar graphic that fills as respondents move through the survey).

In a recent post, I described the differences between nonprobability and probability sampling methods in online survey designs.  Probability methods are preferred if at all possible, because they allow you to make generalizations from your electronic survey results to a larger population or target audience.  One kind of probability-based sampling technique is called systematic random sampling. 

To employ a systematic random sampling design for your online web survey, you first select a case at random from your exhaustive population list, and then select further cases at identical intervals, determined by how many people you want to sample in total.  If you wanted to sample ten people from a population list of 150, you would then choose every fifteenth person after selecting someone in the first 15 cases (to ensure you will select 10 people in total).

This provides an easy way to obtain a random sample of your population list or sampling frame, because as long as your data is ordered randomly, you can begin simply by selecting any record or case and go from there.  This is an important caveat though: your records must be randomly ordered for a systematic sample to be effective. 

Take this example of survey sampling, let’s say you have a sampling frame (list) of people that is currently ordered alphabetically by last name, and you are interested in subgroup analyses by ethnicity.  It would be wise in this case to rearrange the records into a truly random order (i.e., not alphabetically), because last names from certain backgrounds may be more likely to begin with a particular letter.  While systematic sampling provides an easy way to generate a random sample for online surveys, you do need to be sure there is no hidden order within your population list or sampling frame.

There advantages and disadvantages to conducting web or electronic surveys as opposed to traditional survey modes such as personal interviewing, telephone and mail.  One of the advantages of an online survey design is a possible reduction of what is known as social desirability bias. 

Social desirability bias occurs when survey respondents offer responses that portray them in a positive or more favorable manner to others. 

When a face-to-face interviewer asks personal or sensitive questions respondents feel have a “good answer” and a “bad answer” (such as criminal behavior), they may underreport bad behavior and overreport good behavior, for example.  While this bias may be reduced in online surveys due to the absence of an interviewer, there are topics that may produce invalid or unreliable data, regardless of survey mode.  This is something to consider before finalizing your online questionnaire design or web survey forms.

Survey questions within the following content areas are especially subject to social desirability in a survey form (i.e., respondents believe particular responses are “better” than others). 

It is crucial that you consider reporting a main element of your web survey design at the outset of your research project.  What you can say about your results hinges heavily on the types of analyses your questions and the capabilities of your response scales.  Today, I will outline the difference between the two major branches of statistical analysis available for most survey data: descriptive and inferential.

Descriptive statistics are the basic measures used to describe survey data.  They consist of summary descriptions of single variables (also called “univariate” analysis) and the associated survey sample.  Examples of descriptive statistics for survey data include frequency and percentage response distributions, measures of central tendency (which include the mean, median and mode), and dispersion measures such as the range and standard deviation, which describe how close the values or responses are to central tendencies.

Inferential statistics offer more powerful analyses to be performed on your online web survey data.  As the names suggests, this branch of statistics is concerned with making larger inferences about social phenomena.  This can include associations between variables, how well your sample represents a larger population, and cause-and-effect relationships.  Some examples of inferential statistics commonly used in survey data analysis are t-tests that compare group averages, analyses of variance, correlation and regression, and advanced techniques such as factor analysis, cluster analysis and multidimensional modeling procedures.

By designing online questionnaires and survey web forms with a good idea of what you want to do with your data after it's collected, you can create cohesive, powerful reports and presentations. Need more tips for how to analyze survey data, read some of these data analysis posts.

When designing surveys, researchers often want to find out how people act in addition to how they feel.  Questions about actions and behaviors are especially useful in market research, since you can gain a sense of what customers and consumers are actually doing (as opposed to what they say they’ll do).  However, this area of survey research can be difficult – to the point of being unreliable – so I urge you to follow the guidelines below when asking behavioral questions in your online questionnaire design:

Provide a specific time reference within your survey questions.  This allows for an “anchor” to be set, and can give you a clearer idea of what your respondents do during an average time frame (be it a day or a week).

Make the time frame fairly recent.  It is very difficult for respondents to remember what they did three years ago.  At the very most, stick with questions that reference 12 months ago or less (preferably much less, like the past week or even 24 hours).

Ask only about the respondent’s own behavior.  Unless you’re asking about the actions of small children, it’s best to keep survey questions directed at the individual level rather than for family members or friends.  Asking people about others’ behaviors can provide distorted, unreliable data, as most people overestimate or underestimate certain actions of other people.

Personal and sensitive information should be taken with a grain of salt.  Multiple market research studies have shown the more sensitive a question is, the less likely it is that respondents will answer... and the data provided can be inaccurate, so it is important to consider that people might dance around the absolute truth on these kinds of questions.

It is tempting to include many similar question types with similar response options in your online survey design.  Matrix questions, for example, provide an efficient questionnaire design method to help you gather lots of data in a neat, brief survey form.  It is wise, however, to resist the urge to use too many uniform survey questions and response lists, namely because of two sources of bias that stem from doing so: habituation and acquiescence.

Habituation occurs when respondents begin providing the same answers to survey questions with the same response options.  They start to get in a habit and select the identical response choice for every question.

Acquiescence is related to habituation, and occurs when respondents passively agree with an interviewer or survey questions.  Agree-disagree scales are the most often-used response options in opinion surveys; it is important that you take steps to avoid the chance that respondents will passively agree with your statements in order to quickly complete the questionnaire or provide what they think may be the “right” answers.

To avoid these response biases, you can use online survey software that allows question randomization, break up your matrix questions with other types of questions and scales, and phrase some questions in a manner that makes respondents switch their thinking.  An example of the latter would be to ask a series of positive questions in your survey questionnaire, and then throw in a couple questions worded differently so as not to allow habituation or acquiescence.  Use care up-front in your online questionnaire design to be sure that you'll reduce error and bias in your results.

The vast majority of survey research projects are studies at a single point in time of a specified population, such as employees, customers or the general public.  Fewer web survey designs track opinions over time.  This post outlines the different types of surveys carried out by researchers.

Point-in-time surveys are called cross-sectional studies.  They study a single population or sample size during a single specified time-frame, and give us a “snapshot” of opinion data.  Cross-sectional surveys comprise the largest number of projects that are undertaken. 

Longitudinal surveys, on the other hand, are those which study trends over time, and usually consist of cohorts or panel respondents.  These can be further classified into three distinct types of longitudinal designs (trend, cohort and panel).

I always recommend probability-based survey sampling techniques wherever possible.  Sometimes, however, companies and organizations want to get an initial feel for how consumers and customers will react to a new product or concept.  In addition, early in the process, you may not have the ability to comprehensively identify a target market or sampling frame, and there is no way to produce a representative sample of your population. 

In these instances, it may be useful to employ a snowball sampling technique as a pilot project, or to gain a rough, early grasp on what customers are feeling.  A snowball sample is one in which you use an initial group of respondents as recruiters for additional market research respondents.  In the survey, you ask your original respondents to list several people  they know that might be interested in completing a survey as well.   This is a case where an incentive might prove particularly useful, since you are asking your market research survey sample to provide contact information of their acquaintances.  Snowball sampling is also especially useful if you do not have a predefined list of people to survey, or if you are trying to identify key information-holders or opinion leaders.

Again, there is a significant caveat of snowball and other nonprobability-based business research methods for sampling techniques: they do not produce representative samples, and therefore cannot be used to generalize findings to the overall population.  But if you are just starting out, and do not mind that your market survey sample cannot produce generalizable findings, then a snowball sampling technique is a convenient survey data collection method to obtain larger amounts of preliminary data.

At minimum, two specific characteristics define a good list of response options for survey questions.  First, the categories (response options) must be mutually exclusive, which means they do not overlap with one another.  Second, survey response options must be collectively exhaustive, meaning they provide all possible options that could comprise a response list.  Let’s take a look at examples of common mistakes for each of these characteristics:


You can see while this response list is exhaustive, it does not provide mutually exclusive categories.  For example, if a survey respondent eats out three times per week, he or she could select either (b) or (c) as an accurate response.


In this survey question example, the response categories do not overlap, but they are not collectively exhaustive.  If a survey respondent make less than $10,000 annually, he or she does not have an option that can accurately capture his or her response.  This could be corrected for option (a) by applying the same response type as shown in (e), such as “$29,999 or less."

If you’ve ever looked at results from a public opinion survey or political poll, you’ve no doubt seen the margin or error noted alongside the findings.  Usually the note will read something like margin of error = plus or minus x%, CL 95%.

So what the heck does that mean?

Well, the first part basically tells you how close the results from your selected sample are compared to what you'd find if you surveyed the entire population.  The expression of “plus or minus x%” tells you that the percentages of given responses might be a bit higher or lower “in reality” (i.e., if you surveyed absolutely everyone).

Generalizability to the larger population is also described by an associated measure called a confidence level (CL). This term describes how confident you can be that your results are not due to chance alone.  A confidence level is normally set at either 90%, 95% or 99% (95% has become standard).

Let’s use an example to understand how these two concepts work:

A random sample of Americans were asked whether they preferred cake or ice cream for dessert.  The results showed that 60% preferred ice cream over cake, and 40% preferred cake over ice cream.  This question had a margin of error of +/- 3% at a 95% confidence level. What this means is that you can be 95% confident that the percentage of all Americans who prefer ice cream would fall between 57% and 63% (60 plus or minus 3).  Another way to put it would be as follows: if you conducted 100 surveys of the entire population, at least 95 times you would find that the percentage who preferred cake ranged from 37% to 43%.

There are literally dozens of ways respondents can misread and even misconstrue survey questions.  Some potential errors may be easily identified, while other errors can go unnoticed all the way up until data collection begins.  The possibility of a large number of respondents skipping the same questions, or customers providing invalid feedback because of faulty survey questionnaire design and implementation, are just two of the many reasons you should test-run a survey before sending out the real thing.  Conducting a pretest is the single best way to identify phrases subject to misinterpretations (and question design flaws in general).

Pretests (also referred to as pilot tests) are used to test the validity and reliability of individual survey questions, the entire questionnaire, and/or response scales.  While carrying out a pretest adds a step to your task list, the benefits of sending out a valid instrument far outweigh the costs of a pretest (which for online surveys usually means only extra time, not dollars).

To conduct a pretest, you first need to select a smaller survey sample that is still fairly representative of your target population.  You then have this pretest sample complete their surveys, while providing feedback about your questions, any technical concerns and other elements you feel could introduce bias or error into the process and subsequent results.  These respondents will no longer be eligible for actual survey participation since they will have been exposed to the questions ahead of time, but the process is well worth it.  The final step in the pretest is to analyze your results and decide how you should revise the survey to make it better and easier to complete.

One big caveat: make sure you use people unfamiliar with your research.  If you use colleagues or acquaintances who are knowledgeable about any relevant topics or technical issues, they may overlook correctable errors such as leading and loaded questions, faulty transitions and overly technical terminology.

By taking the extra step of conducting a pretest, you can gain valuable information from a small subset of people.  This will allow you to make the necessary and appropriate revisions, and in turn produce valid, reliable survey questionnaires for your formal survey research project.

If you follow the bloggers on the Cvent Web Survey blog, you know we emphasize survey reporting concerns throughout the entire questionnaire project life-cycle.  This post outlines five general sections that comprise a good report or presentation for a survey research project.  By following this outline, you can be sure that you have a thorough summary of your research objectives, processes and results all in one place at the conclusion of your survey project.

Executive Summary
This is the "drive-by" version of the entire survey project.  It contains a brief summary of your main points, and predominately consists of broad research findings.  You can also think of this as the audience takeaway section: imagine if people had just five minutes to hear your results, and then had to make decisions based upon what they heard.  Include essential, bottom-line insights from your survey project here (and not much else).

Introduction/Background
In this section, you introduce your study, including its purpose, concepts, rationale and variable descriptions (where appropriate).  This is also the place to provide all the background information, as well as any prior research findings/citations upon which your current study is based.  If you have specific hypotheses, they can be placed here after you’ve established rationale.

Methodology
This is the part of the report where you get dirty with the details of how you went about designing and conducting your survey.  Descriptions of survey sampling procedures, survey question and response option/scale design, data collection activities and timelines all belong here.

Results
In the survey results section of your report or presentation, you provide all the survey findings.  This includes detailed tables, charts and other selected graphics that clearly explain what the data say, and can be quite extensive depending on how many survey questions (and research questions) you have.

Conclusion/Discussion/Recommendations
These sections can of course be listed separately, but I’ll group them together here for our general descriptive purposes.  This is where you take the step from findings to decisions.  You’ve detailed your results; now you draw conclusions, discuss anything that might have affected the research or your path forward, and make recommendations about where to go next based on the data.

Before you select a sample or design a single question, the first step in a survey research project is to establish your goals and objectives.  With proper planning, time and attention given to this step, all subsequent steps will flow more smoothly. 

Ask yourself--and answer, in writing--broad questions such as the ones below, and then you can drill down into more specific research questions and move on to procedural matters.

What information am I seeking to gather?  Why do I need it?  What is the “end game” regarding what I’m trying to learn?

Who (exactly) can I get this information from?  How can I contact them?  How many people do I need to contact, and do I have an exhaustive list?  Am I interested in any particular subgroups?

Has anyone else done a similar project in the past?  If so, what did they find out and how could what they did help with my current project?

What kind of data and results do I want my questions to produce?  Am I interested in public opinion, actual behaviors, satisfaction levels, employee feedback/workplace surveys, etc.?  How will I go about collecting and analyzing the data?

How will my results be reported?   What will the report format look like, and who will receive the results?  What kind of summary data do I need, and how can I design questions with response options that will provide me with actionable data?

Establishing clear, attainable objectives and goals is of the utmost importance when undertaking a research study.  Your results will likely only be as clear as your objectives, for better or worse!

I’ve written in the past that it’s best to use probability-based sampling techniques for your employee, market research or consumer survey samples whenever possible.  One of these techniques is called stratified sampling.  It is used when you expect that your population is heterogeneous (different) and that the survey results could vary greatly based upon certain subgroup characteristics.  If you are particularly interested in subgroup analysis within your survey sample, using a stratified sample can increase the accuracy of your results and reduce error estimates.  Here’s how it works:

Let’s say you are interested the television watching habits of American citizens, and you know (from anecdotal evidence or previous theory) that television watching varies widely by gender.  Since you know that males and females have disparate television watching habits, you want to select a sample whose results will accurately reflect the habits and responses for males and females independently.

What you do is first divide the population into mutually exclusive subgroups (or “strata”), and then take a random sample from each of the subgroups.  Using our gender example, you would divide the population into two groups (male and female), then take a random sample from your male and female subgroups, respectively.  You will be able to be more confident in what you say about your results than if you used a simple random sample from the overall population.

Private businesses tend to guard findings from their research efforts very closely.  In some instances, where significant investment dollars and time were spent on sampling frames and reporting, this makes sense.  It's nice to have proprietary research at your disposal to impress clients and customers, and internal employee survey results, for example, may be reserved for senior leaders.

But in most cases, I suggest sharing your survey results with as many people as possible.  The benefits outweigh the perceived drawbacks.  After all, most survey research comes from of a specific point-in-time sample, and whatever narrow edge you may get from keeping the results private will be short-lived.

At minimum, you should share the results of your survey with respondents themselves.  People like to know what they're a part of (and why).  In fact, offering to provide respondents with survey results has been shown to increase response rates! 

It's a great idea to set up partnerships with community leaders before you even send out your community attitude survey.  Publicize your organization and your efforts toward being proactive.  Let your community know that you value them and are interested in feedback and suggestions about the process.  Create anticipation and eagerness to both complete questionnaires and receive the results after the survey is completed.

You can also organize a press release highlighting selected findings from your survey.  This can be posted on your website and in other locations (external websites, community hotspots, etc.).   Call local news organizations if you think the results would appeal to their audiences.  Tell all who will listen how you've made original contributions to the knowledge of your industry, and how you will improve business practices based on your results.