Sample Survey Question

Designing a survey instrument can be one of the most fun aspects of conducting survey research. In this two-part tutorial, we'll first look at some of the "Don'ts" of writing survey questions, and then we will look at some of the "Do's".

Once the brainstorming of ideas phase has been accomplished and you have a good idea what the scope of your research is, it's time to get down to putting question ideas on paper (or online!).

However creating good survey questions can turn into quite an art form, as we all struggle to word our items in such a way that our participants will understand what information we are asking for. Recognizing when you've accidentally designed a survey question that is worded in a non-objective way is the sign of a good researcher. Let's see if you can spot the poorly worded question from this group of sample survey questions:


Now that we've looked at some question items that shouldn't be put into surveys, next time we can define some ways we can create objective question items that will provide us with the results we need!

In a recent post, I described the differences between nonprobability and probability sampling methods in online survey designs.  Probability methods are preferred if at all possible, because they allow you to make generalizations from your electronic survey results to a larger population or target audience.  One kind of probability-based sampling technique is called systematic random sampling. 

To employ a systematic random sampling design for your online web survey, you first select a case at random from your exhaustive population list, and then select further cases at identical intervals, determined by how many people you want to sample in total.  If you wanted to sample ten people from a population list of 150, you would then choose every fifteenth person after selecting someone in the first 15 cases (to ensure you will select 10 people in total).

This provides an easy way to obtain a random sample of your population list or sampling frame, because as long as your data is ordered randomly, you can begin simply by selecting any record or case and go from there.  This is an important caveat though: your records must be randomly ordered for a systematic sample to be effective. 

Take this example of survey sampling, let’s say you have a sampling frame (list) of people that is currently ordered alphabetically by last name, and you are interested in subgroup analyses by ethnicity.  It would be wise in this case to rearrange the records into a truly random order (i.e., not alphabetically), because last names from certain backgrounds may be more likely to begin with a particular letter.  While systematic sampling provides an easy way to generate a random sample for online surveys, you do need to be sure there is no hidden order within your population list or sampling frame.

There advantages and disadvantages to conducting web or electronic surveys as opposed to traditional survey modes such as personal interviewing, telephone and mail.  One of the advantages of an online survey design is a possible reduction of what is known as social desirability bias. 

Social desirability bias occurs when survey respondents offer responses that portray them in a positive or more favorable manner to others. 

When a face-to-face interviewer asks personal or sensitive questions respondents feel have a “good answer” and a “bad answer” (such as criminal behavior), they may underreport bad behavior and overreport good behavior, for example.  While this bias may be reduced in online surveys due to the absence of an interviewer, there are topics that may produce invalid or unreliable data, regardless of survey mode.  This is something to consider before finalizing your online questionnaire design or web survey forms.

Survey questions within the following content areas are especially subject to social desirability in a survey form (i.e., respondents believe particular responses are “better” than others). 

A key marker of a quality survey, whether that survey is an online form or another mode, is replicability of your results over trials. Replicability of your survey results lends credibility to your organization's research work.

One indication that your survey form may have results based on biased samples is nonreplicable results for the same instrument, fielded with the same survey design, from one time to the next.  If the same sample frame produces significantly different results for identical questions, that should perk up our noses to the scent of possibly unreliable results. We should start to look at the criteria used to select our survey sample, and take a fresh look at the questionnaire to see if it is still pertinent to our needs, or the needs of our clients.

This holds true for all modes of survey research, including online survey internet research. Receiving divergent findings in survey results being fielded again and again could lead to differing (and possibly harmful) business decisions.  No company should be left with uncertainty about answers and actions when decisions need to be made based upon the research. This is just one reason why replicability of your research findings is a key ingredient to sound data. When in doubt- replicate!

It is crucial that you consider reporting a main element of your web survey design at the outset of your research project.  What you can say about your results hinges heavily on the types of analyses your questions and the capabilities of your response scales.  Today, I will outline the difference between the two major branches of statistical analysis available for most survey data: descriptive and inferential.

Descriptive statistics are the basic measures used to describe survey data.  They consist of summary descriptions of single variables (also called “univariate” analysis) and the associated survey sample.  Examples of descriptive statistics for survey data include frequency and percentage response distributions, measures of central tendency (which include the mean, median and mode), and dispersion measures such as the range and standard deviation, which describe how close the values or responses are to central tendencies.

Inferential statistics offer more powerful analyses to be performed on your online web survey data.  As the names suggests, this branch of statistics is concerned with making larger inferences about social phenomena.  This can include associations between variables, how well your sample represents a larger population, and cause-and-effect relationships.  Some examples of inferential statistics commonly used in survey data analysis are t-tests that compare group averages, analyses of variance, correlation and regression, and advanced techniques such as factor analysis, cluster analysis and multidimensional modeling procedures.

By designing online questionnaires and survey web forms with a good idea of what you want to do with your data after it's collected, you can create cohesive, powerful reports and presentations. Need more tips for how to analyze survey data, read some of these data analysis posts.

What is a survey? The Merriam-Webster Dictionary defines the word "survey" in this way: "To query (someone) in order to collect data for the analysis of some aspect of a group or area."  These "someones" can also be called your survey sample. Unlike a census, which surveys an entire population, a sample survey gathers information from only a fraction of the individuals within an entire group. Samples are collected in such a way that results from the survey can then be generalized back to the entire population.

Surveys can be conducted in a variety of ways, which can also be termed your survey design. They can be conducted over the phone, through responses written on paper, through in-person interviews, or through online web surveys. Even though the mode in which a survey is conducted can vary, sample members are all asked the same questions in a similar or standard way.

Some survey forms focus on opinions and attitudes (e.g. asking your clients how they feel about a new product line), while others focus on factual questions (household income levels, or age of members). Most online survey design contains a combination of both types of questions. A survey design can also vary in length and time it takes the sample to complete. Some may take only a few minutes of a person's time, while others may take hours or even days to complete. 

Surveys should be carried out only for the purposes of gathering data and information on a certain subject. A quality survey can be determined by looking at the way in which it was conducted, and its purpose.

Get started designing your own survey using one of over 50 graphical templates available, or read more blogs on designing surveys.

There are many options available to market researchers when designing a web survey form. I covered this topic a bit last week when I discussed creating online surveys, and included some dos and don’ts.

Conducting a study for the Web offers several opportunities for market researchers that paper questionnaires do not. For example, you can create an instant survey and check on responses frequently for updates. You may even allow the respondents to view the current results when they complete the survey. This works best for one or two-question quick polls rather than longer studies. But knowing they will see the results immediately is frequently a tipping point for respondents who are on the fence about completing your feedback survey form.

This can be helpful in analyzing survey data as well. For example, suppose results for your online survey form vary wildly throughout one week. One day, 10% of respondents use your client’s brand of glass cleaner. The next day, it’s 65%, but then the following day it drops back down to 20%. This could mean something (possibly that you need to check your recipient list) and is good information to track for your client.

The instant results also allow you to add more questions and gain clarification on points you may not have considered prior to launching the survey form. There are dozens of survey web software tools available that can create instant survey results – it’s a method that is worth trying out, but as always, consider your client’s needs before committing to a particular survey method.

Web 2.0 is a term that is frequently tossed around, but there are still many who may not be entirely clear on the definition. The simplest way to boil it down is that Web 1.0 offered little or no opportunities for feedback, while Web 2.0 includes countless ways to “talk back” on the Web. It can include blogs, forums, social media sites, and interactive informational sites like Wikipedia. The users can control the content to an extent on these types of sites, a difference from the static web sites of Web 1.0.

What does this mean for a market researcher? I’ve already discussed methods to use social media for gathering data and survey respondents, but you can use other Web 2.0 sites in a similar manner. It’s a huge topic that I am sure will be revisited again and again.

Web 2.0 sites are good for research purposes. People who would likely make the ideal survey respondents are using these sites to talk back about products. For example, if you were doing a baking survey, the blog Bakerella.com has hundreds of posts and thousands of comments about various recipes and their merits. It is a good start for conducting internet research about the habits of the true baking enthusiast, and could help you in your online questionnaire design.

Despite the opportunities for feedback Web 2.0 offers, it is important to remember that it does not and should not replace more traditional marketing research methods, like electronic surveys, paper questionnaires and other methods. Although respondents are very forthcoming on blogs like Bakerella.com, for a quality study respondents must be properly vetted to ensure they are the correct group.

It is tempting to include many similar question types with similar response options in your online survey design.  Matrix questions, for example, provide an efficient questionnaire design method to help you gather lots of data in a neat, brief survey form.  It is wise, however, to resist the urge to use too many uniform survey questions and response lists, namely because of two sources of bias that stem from doing so: habituation and acquiescence.

Habituation occurs when respondents begin providing the same answers to survey questions with the same response options.  They start to get in a habit and select the identical response choice for every question.

Acquiescence is related to habituation, and occurs when respondents passively agree with an interviewer or survey questions.  Agree-disagree scales are the most often-used response options in opinion surveys; it is important that you take steps to avoid the chance that respondents will passively agree with your statements in order to quickly complete the questionnaire or provide what they think may be the “right” answers.

To avoid these response biases, you can use online survey software that allows question randomization, break up your matrix questions with other types of questions and scales, and phrase some questions in a manner that makes respondents switch their thinking.  An example of the latter would be to ask a series of positive questions in your survey questionnaire, and then throw in a couple questions worded differently so as not to allow habituation or acquiescence.  Use care up-front in your online questionnaire design to be sure that you'll reduce error and bias in your results.

Creating an evaluative tool to measure an employee's performance can be a daunting task for even the most experienced managers.

Here are some survey design tips to help you create effective performance evaluation materials that will be meaningful for both supervisors and supervisees:

1) Use titles that are less challenging for employees (e.g. calling the instrument an "evaluation" as opposed to a "test")

2) Have a place at the beginning of the job performance evaluation form to clearly delineate the employee being evaluated, such as their name, title, department, and other pertinent job related individual information, like hiring date and date of last review. Other non-job related demographics (such as employee age or eye color) should be left out.

3) Make sure the content the employee is being evaluated on always refers directly back to their position. This can include technical job-related skills, and "softer" characteristics such as courtesy to both clients and co-workers, or punctuality.

4) Employee evaluations lend themselves nicely to the use of Likert scales, but a good evaluation uses verbal measurements as opposed to numeric. For example one end of the survey rating scale would be "Needs Improvement" and the opposite end of the scale "Excellent Performance."

5) Leave plenty of room for written employee performance evaluation comments after each content area. Both the manager AND the employee should write down their thoughts about the content area discussed. This makes both parties feel like they are contributing equally to a conversation, rather than one person telling the other how to act.

6) Include space for concrete development plans and steps to be accomplished, including dates and time lines for the progress to take place. It's also a good idea to include mid-term progress review dates so manager and employee can check in with each other. This ensures both parties are still on target for a successful future review.

7) Allow both the employee and manager to sign the list of employee evaluation questions and responses after reading all parts thoroughly and together. Leave time for discussion of the evaluation. 
 
8) If your company has an HR department, have an appropriate HR supervisor review your staff evaluation form to double-check that all the right notes are hit.

The vast majority of survey research projects are studies at a single point in time of a specified population, such as employees, customers or the general public.  Fewer web survey designs track opinions over time.  This post outlines the different types of surveys carried out by researchers.

Point-in-time surveys are called cross-sectional studies.  They study a single population or sample size during a single specified time-frame, and give us a “snapshot” of opinion data.  Cross-sectional surveys comprise the largest number of projects that are undertaken. 

Longitudinal surveys, on the other hand, are those which study trends over time, and usually consist of cohorts or panel respondents.  These can be further classified into three distinct types of longitudinal designs (trend, cohort and panel).

Completing a study is just the first step in the market research process. A good researcher knows there is plenty of work left to be done analyzing survey data and taking action on survey results.

Stay in touch with the client and schedule presentations of the research. If you have a client contact assigned to the project, go over the preliminary survey data to see what key findings they feel are the most important to share. Create presentations around these key findings. You can also create separate market survey reports for different groups within the same company.

For example, if you have just completed a large product survey that includes feedback on customer satisfaction, the client may want a presentation simply on customer feedback for their customer service team.

A good researcher also has an eye for finding problems. If you notice an area for improvement in the course of your market research analysis, propose a solution to that problem for the client. Take this market research survey example, customers complain about being on hold too long when calling customer service. You could propose several ideas from hiring more staff to setting time goals for staff to talk to customers.

Even though the client may not agree with your solution ideas, ignoring problems you identify through the research is a bad idea. The client is paying you to compile and analyze research data, and they will likely appreciate all your efforts even if the information does not fit in with their current business plan.

I always recommend probability-based survey sampling techniques wherever possible.  Sometimes, however, companies and organizations want to get an initial feel for how consumers and customers will react to a new product or concept.  In addition, early in the process, you may not have the ability to comprehensively identify a target market or sampling frame, and there is no way to produce a representative sample of your population. 

In these instances, it may be useful to employ a snowball sampling technique as a pilot project, or to gain a rough, early grasp on what customers are feeling.  A snowball sample is one in which you use an initial group of respondents as recruiters for additional market research respondents.  In the survey, you ask your original respondents to list several people  they know that might be interested in completing a survey as well.   This is a case where an incentive might prove particularly useful, since you are asking your market research survey sample to provide contact information of their acquaintances.  Snowball sampling is also especially useful if you do not have a predefined list of people to survey, or if you are trying to identify key information-holders or opinion leaders.

Again, there is a significant caveat of snowball and other nonprobability-based business research methods for sampling techniques: they do not produce representative samples, and therefore cannot be used to generalize findings to the overall population.  But if you are just starting out, and do not mind that your market survey sample cannot produce generalizable findings, then a snowball sampling technique is a convenient survey data collection method to obtain larger amounts of preliminary data.

One of the great things about Cvent Web Surveys software application is that it is constantly getting “better.” I, for one, am not aware of another survey software company that can state with confidence that 80% of all product enhancements come directly from the requests of current clients. Cvent, however, has certain processes in place so that every time a client expresses interest in seeing a new feature added to the online survey application, this request is quickly relayed to our technical team.

For those of you who were clients before our most recent product release in August 2009, you definitely noticed at least one big change in your account the first time you logged in after the release. As soon as you logged into your Cvent Web Surveys account, you saw that your surveys were no longer organized into folders on the Survey Selection page. Instead, they are now displayed in “views.”

Now, you may wonder why I chose to write my blog post this week on the transition from folders to survey views. It may seem like a pretty dry topic. I wanted to write on this particular survey subject because I get so many calls from clients asking how to create a new survey view that pull the appropriate surveys into view. If you have a lot of surveys created in your account, then this is a pretty important thing to know how to do, so that you don’t have to sort through all of your company’s surveys just to find the few that you are personally working on!

When you are ready to create a new survey view and filter the appropriate surveys into this view, you need to remember 3 simple steps:

1) Create a survey custom field. You can create survey custom fields under the Administration tab, on the same page that you create contact custom fields. Survey custom fields are primarily used to classify the surveys in your account and pull them into the appropriate views on the Survey Selection page. So, for example, if your marketing department and human resources department are running surveys, you may want to create 2 separate survey views, one for each department. The first step to do this would be to create a survey custom field for department.

2) Create a new view on the Survey Selection page. You can create a new survey view by choosing “add new view” from the Display drop-down menu. When you add the new view, you will need to name it and also specify certain options (i.e. whether you would like the view to be private or public). Finally, at the bottom of the page, you should apply an advanced filter based on the survey custom field you just created for department. For example, if you are adding the survey view for “Marketing Surveys,” you should choose “department” as the field, “equals” as the operator, and “marketing” as the value.


3) Now that you have created the survey custom field and added the new view, all you need to do is pull the appropriate surveys into the view you just created! When you added the new view for “Marketing Surveys,” you should have gotten a message, “no surveys match your criteria.” This is because you have not yet applied the survey custom field at the survey-level! To do this, simply go into an individual marketing survey, and click on  Settings on the top navigation bar. On the General Information page, you should click on the Custom Survey Fields tab. Here you can apply the “marketing” label to the individual survey, so that it will show up in the “Marketing Surveys” view.


Hopefully this post will be helpful to those of you who are struggling with the transition from folders to survey views. Believe me, survey views are completely customizable and will help you organize online surveys in your Cvent Web Surveys software account.

At minimum, two specific characteristics define a good list of response options for survey questions.  First, the categories (response options) must be mutually exclusive, which means they do not overlap with one another.  Second, survey response options must be collectively exhaustive, meaning they provide all possible options that could comprise a response list.  Let’s take a look at examples of common mistakes for each of these characteristics:


You can see while this response list is exhaustive, it does not provide mutually exclusive categories.  For example, if a survey respondent eats out three times per week, he or she could select either (b) or (c) as an accurate response.


In this survey question example, the response categories do not overlap, but they are not collectively exhaustive.  If a survey respondent make less than $10,000 annually, he or she does not have an option that can accurately capture his or her response.  This could be corrected for option (a) by applying the same response type as shown in (e), such as “$29,999 or less."

Perhaps you have an extra special research project that really needs a good response turnout, or you’re simply looking for ways to boost your response rates. Well, maybe its time to try sweetening the pot a little?

Studies have shown that surveys where potential participants were offered monetary incentives of as little as $1 received higher response rates then ones that didn't’t offer any incentives at all. In fact, offering any incentive to your potential survey respondents as a way of saying “Thank You” even before they take your survey is a great way to get started on the road to higher survey returns. Good examples of survey incentives that work are gift cards, checks, gift certificates, and of course good ol' fashioned cash.

The amount isn't as important as the belief by the participant that the offered incentive will actually be provided. Credibility is key. So if you choose to offer incentives to your potential respondents after they have participated in the survey (say for example the option to be entered into a random drawing for a chance to win an iPod) remember to follow through on giving out the reward after you collect feedback from all the survey respondents!

This tip is particularly useful when trying to avoid survey bias. By offering incentives you're more likely to attract people who would not normally participate in your surveys. Learn more about minimizing survey bias from some of our other posts.

I found an interesting poll today about colors preferred by men and women, and it provoked some questions about what are the best colors to use when you create polls or design survey questionnaires. Here's the breakdown from the poll shared in a Lyris Whitepaper:


Why does it matter? Because colors are also a form of non-verbal communication. So whether you're creating an online questionnaire to collect feedback or using an email survey tool to craft email marketing messages for survey invitations, you should care how colors affect those reading your email or completing your customer survey forms.

Colors can cause physical reactions. For example, too much red has been show to increase blood pressure. As you design survey templates, keep in mind how color meanings can affect survey respondents.

Cool colors: Cool colors typically have a calming effect. Keep in mind that cool colors can appear smaller than warm colors and visually recede on the page.

	Blue - As you might have guessed, blue is calming. Almost everyone likes some shade of blue, whether it's a strong and steadfast blue or a light, friendly blue. In fact, in 2008 Pantone selected Blue Iris as the color of the year. As a result of the calming effect blue has, it can make time seem to pass more quickly and help you sleep. However, too much blue can cause the calming effect to go to the extreme and cause you to have the blues. Beyond just being calming, blue can convey richness and sometimes superiority (deep royal blue) or it can convey trust and truthfulness (combining light and dark blue). See how using blues could improve your response rate if it helps people trust you?
	Green - Like blue, green has some calming effects and can make time seem like it's moving quicker, but it also signifies growth, renewal, health and the environment. Like with blue, green has it's own extreme as well, green can mean jealousy or envy and inexperience. With a hint of warmth and coolness, green can create balance, harmony and stability.
	Purple - Over the ages, purple has come to be synonymous with royalty. Since purple comes from red (warm) and blue (cool) it has intriguing qualities of both. Typically deep and bright purples suggest riches, while lighter purples are more romantic and delicate. Keep in mind though, while purple can be noble and spiritual, too much purple can cause moodiness - the same as with too much blue.
	Turquoise - As a blend of blue and green, turquoise can have a soft, feminine qualities or a more sophisticated feel with the darker teals.

Look for parts two and three later this week for warm and neutral color meanings.

When performing a market research study for clients in certain marketplaces, it is important to remember the competition. If, for example, you were doing a customer market research survey for a fast-food restaurant, Burger One, you would want to gauge survey respondents’ feelings about the competition as well as your client.

You might ask restaurant customer satisfaction questions like:


If respondents frequent Burger One and four other fast-food restaurants, that gives you insight into the survey data provided. If respondents only frequent Burger One, that is helpful to know as well. Take this example of how survey results can be applied to the restaurant's over all marketing strategy:


On the other side of this equation is what Burger One's loyal customer base thinks. If customers who only eat fast-food at Burger One express no interest in the new sandwich, it may not be the best possible sandwich to launch. However, if respondents who frequent competitors more often than Burger One express high levels of interest in the new sandwich, it could spell an opportunity for Burger One to gain new business.

It’s best to analyze market research from all angles when launching a new product – there are multiple factors at play.

If you’ve ever looked at results from a public opinion survey or political poll, you’ve no doubt seen the margin or error noted alongside the findings.  Usually the note will read something like margin of error = plus or minus x%, CL 95%.

So what the heck does that mean?

Well, the first part basically tells you how close the results from your selected sample are compared to what you'd find if you surveyed the entire population.  The expression of “plus or minus x%” tells you that the percentages of given responses might be a bit higher or lower “in reality” (i.e., if you surveyed absolutely everyone).

Generalizability to the larger population is also described by an associated measure called a confidence level (CL). This term describes how confident you can be that your results are not due to chance alone.  A confidence level is normally set at either 90%, 95% or 99% (95% has become standard).

Let’s use an example to understand how these two concepts work:

A random sample of Americans were asked whether they preferred cake or ice cream for dessert.  The results showed that 60% preferred ice cream over cake, and 40% preferred cake over ice cream.  This question had a margin of error of +/- 3% at a 95% confidence level. What this means is that you can be 95% confident that the percentage of all Americans who prefer ice cream would fall between 57% and 63% (60 plus or minus 3).  Another way to put it would be as follows: if you conducted 100 surveys of the entire population, at least 95 times you would find that the percentage who preferred cake ranged from 37% to 43%.

Did you know there are 6 steps in the market research process?  While this process speaks directly to marketing research professionals, the process applies to HR, customer or education surveys as well:

1. Identify and define the problem.  Before you start any web survey project, you should identify the key issues you hope to be able to solve.  This step should also include clearly defined objectives.
    
2. Develop the approach. In this step, you need to establish a budget, understand influencing factors such as the environment or economy, decide on sampling and survey methods, and formulating hypotheses.
    
3. Research design. Designing a survey or questionnaire is considered the most important step in any survey process.  Question design takes a lot of thought and time.  We like to say, "If you put garbage in, you'll get garbage out."  This means that if the questions are bad, the data will be bad as well.  During the survey research design, keep in mind sampling methods and data analysis factors you intend to use.
    
4. Collect the data. Don't forget to test your survey before to ensure you're fielding the correct data.  Thankfully, with the help of an online survey tool, this step is relatively painless.
    
5. Analyze the Data. The types of analysis you planned to perform on the collected survey data should have been decided in earlier steps, but after collecting the data you have to actually perform the survey analysis.  Analysis can be performed using survey analysis tools like office programs, such as Excel, or more advanced programs such as SPSS - the complexity of the questions will determine this.
    
6. Report, Present, Take Action.  The final step in the market research process is to present your survey research findings and draw conclusions.  While Step 3 is the most important because it defines the outcome of your survey, if you fail to complete this last step and act on the findings in some way, the previous steps don't matter. 

As I mentioned in the beginning, this same process can be applied to any type of project: product evaluations, customer satisfaction questionnaires, public relation surveys, etc.  If you give each step the attention it deserves, each of your online surveys should be a success.