Survey & Questionnaire Design

Randomizing your online survey form or your answer options within a survey questionnaire, means that the listed items are not asked in the same order for each survey respondent.

So, participant A may see their question in this manner:


While participant B may see the question in this order:


With the advent of electronic surveys, this feature can now be done in an easy and cost-effective manner!

For many survey questions, rotating the order in which answer choices are viewed by your participants will minimize potential within-item bias. Say for example, customer service survey respondents tend to remember (and choose) the last option they read from a list, or conversely, the first option they read from a list, rotating the order of the options means that each option is read last as often as all the other options available. Ensuring that any potential bias is spread out across the options.

By randomizing your answer options for certain questions, you can be more confident that your results aren't over-estimating the number or frequency of participants who prefer dogs, to say, ferrets.

Read more about this topic to learn important tips on when you shouldn't use randomization of your answer options.

When conducting market research, either through survey forms or another mode, the method of sending a survey pre-notification letter can be a tempting one.

For some time now survey methodologists have been conducting studies of pre-notification letters to determine if they have any affect, either positively or negatively, on survey response rates. An overall analysis of these various independent studies revealed an increase in response rate of approximately 8% when compared against studies where no pre-notification letter was sent.

No one can say for sure exactly why pre-notice survey letters seem to assist in increasing the number of survey responses, but perhaps they help to establish the legitimacy of a survey, contributing to a respondent's feeling of trust and the credibility of the organization conducting the research. Another possibility is that a pre-notice letter builds expectation of arriving mail (be it electronic, paper, or in-person). A third possibility is that a potential respondent is less likely to disregard the survey when it arrives if they are aware it is arriving shortly.

Although survey pre-notification letters are an excellent (but sometimes expensive) way to increase response rates, they are seldom used in marketing research surveys. Each researcher needs to weigh the additional cost of sending out a pre-notice to potential respondents against the probability of a lower response rate.

It's worthwhile to take your population's survey sample size into consideration when coming to this decision. When your sample sizes are small, each individual response affects your survey results even more. Increasing those responses through utilizing a pre-notice letter may counterbalance the increased cost of sending them out to your participants.

Last week, we reviewed poorly written survey question items, and what makes them inefficient tools for collecting the data you desire. For our final installment of "Writing Survey Questions the Work," we'll discuss some final points to keep in mind in order to write great questions.

In general when you design a questionnaire, try to write from the respondent's perspective rather than your perspective. Hark back to last weeks post where we discussed writing objectively and from a non-assuming point of view. Writing from neutral viewpoint and keeping far away from internal workplace jargon and slogans that wont be understood by your clients is key.

Having a sample of your survey population and other neutral testers review the wording of your survey is a wonderful way to attain clarity on question items. After spending what may amount to dozens of hours on a survey form project, having an outsider do things like proofread and spell check with a fresh pair of eyes can be well worth the effort!

And as we've talked about in other posts, pre-testing your survey with a segment of your audience is crucial to detecting question item mis-wordings as soon as they occur.

If we have poorly worded question items in our survey, not only will the answers we receive be inaccurate, our data analysis will be inaccurate as well. This could result in poor decisions being made based on data that has either overstated or understated our clients needs for a product or service. 

Staying in tune with the needs of our clients and our target populations starts with skillfully worded survey question items. If your question isn't worded right, your analysis wont be right.

In yesterday's post we looked at how your survey question items, if worded improperly, may be biased, resulting in unusable or incorrect results. Now we know how survey questions shouldn't look, let's delve deeper into why those previous examples were poorly worded, and how with just a few small adjustments you can easily start to hone question items within your survey forms.

When you design internet surveys, how do you write objective questions that won't bias your results one way or the other?

Here are a few main points:


These first few points are a great starting place when you're designing a questionnaire. The final post in this series will look at a few final survey best practice tips for writing great survey questions and the benefits you'll gain from crafting quality items. In the meantime, questions previously stored in our question library already come properly worded and ready for use!

Designing a survey instrument can be one of the most fun aspects of conducting survey research. In this two-part tutorial, we'll first look at some of the "Don'ts" of writing survey questions, and then we will look at some of the "Do's".

Once the brainstorming of ideas phase has been accomplished and you have a good idea what the scope of your research is, it's time to get down to putting question ideas on paper (or online!).

However creating good survey questions can turn into quite an art form, as we all struggle to word our items in such a way that our participants will understand what information we are asking for. Recognizing when you've accidentally designed a survey question that is worded in a non-objective way is the sign of a good researcher. Let's see if you can spot the poorly worded question from this group of sample survey questions:


Now that we've looked at some question items that shouldn't be put into surveys, next time we can define some ways we can create objective question items that will provide us with the results we need!

A key marker of a quality survey, whether that survey is an online form or another mode, is replicability of your results over trials. Replicability of your survey results lends credibility to your organization's research work.

One indication that your survey form may have results based on biased samples is nonreplicable results for the same instrument, fielded with the same survey design, from one time to the next.  If the same sample frame produces significantly different results for identical questions, that should perk up our noses to the scent of possibly unreliable results. We should start to look at the criteria used to select our survey sample, and take a fresh look at the questionnaire to see if it is still pertinent to our needs, or the needs of our clients.

This holds true for all modes of survey research, including online survey internet research. Receiving divergent findings in survey results being fielded again and again could lead to differing (and possibly harmful) business decisions.  No company should be left with uncertainty about answers and actions when decisions need to be made based upon the research. This is just one reason why replicability of your research findings is a key ingredient to sound data. When in doubt- replicate!

What is a survey? The Merriam-Webster Dictionary defines the word "survey" in this way: "To query (someone) in order to collect data for the analysis of some aspect of a group or area."  These "someones" can also be called your survey sample. Unlike a census, which surveys an entire population, a sample survey gathers information from only a fraction of the individuals within an entire group. Samples are collected in such a way that results from the survey can then be generalized back to the entire population.

Surveys can be conducted in a variety of ways, which can also be termed your survey design. They can be conducted over the phone, through responses written on paper, through in-person interviews, or through online web surveys. Even though the mode in which a survey is conducted can vary, sample members are all asked the same questions in a similar or standard way.

Some survey forms focus on opinions and attitudes (e.g. asking your clients how they feel about a new product line), while others focus on factual questions (household income levels, or age of members). Most online survey design contains a combination of both types of questions. A survey design can also vary in length and time it takes the sample to complete. Some may take only a few minutes of a person's time, while others may take hours or even days to complete. 

Surveys should be carried out only for the purposes of gathering data and information on a certain subject. A quality survey can be determined by looking at the way in which it was conducted, and its purpose.

Get started designing your own survey using one of over 50 graphical templates available, or read more blogs on designing surveys.

By now most of us know about the many benefits of online/electronic surveys. Some of the major benefits that come to mind include: Access to a possible world-wide population, negligible distribution costs, data is automatically analyzed or exported to other databases with ease, pre-programming the question order, easily trackable response rates (or choose anonymous responses), automatic reminders, thank you's, and auto-close dates. Additionally there are great perks such as spell checkers, and built-in math calculations.  

Some less well-known benefits that make online instant surveys even more desirable include: adding "help buttons" or links that can provide your respondents with additional explanations, designing surveys that are easily pilot tested, progress bars for respondents, required questions, validation (only "acceptable" answers are allowed), branching, piping and skipping patterns. 

We can already see the many benefits to online web surveys without writing out an exhaustive list. However, here are some considerations to keep in mind before committing yourself to an online survey design: Do you have a survey form software tool that can keep up with advancing technology? That is, does your instant survey software have the ability to include all the features mentioned above and more, such as importing pre-written questions to save time? Does your survey company's software have a "back-up survey" policy in case of technical glitches? Are your surveys hosted on secure URLs to provide respondents with confidence in your electronic survey? Can you customize your online survey form with your own brand? 

Whether you'd prefer paper questionnaires or online survey forms to for quality data collection, Cvent can assist with both. Check out all the useful info that's covered in Cvent's online data collection blogs, or sign up for a free webinar to learn more survey best practices.

Within the survey research process, a "Thank You" note is something that is sent a few days to a week after a questionnaire is sent out to your clients. This mailing expresses appreciation for responding and indicates that, if the client survey has not yet been completed, it is hoped it will be soon.

The postcard or email survey "thank you" note is not written in order to overcome respondent doubts or fears about taking the survey, rather, it is meant to jog memories and help re-arrange priorities, bringing your potentially forgotten survey back to the top of potential respondent's "To-Do" pile.

If timely and appropriately worded, the postcard "Thank you" note can arrive just in time to make an appeal that engenders a sense of importance about survey participation, without crossing the line into sounding impatient.

When possible, attempt to have your "Thank you" note be as different as possible from your original survey invitation request for participation letter. The goal is to create a stand-out piece of paper or electronic mail that contrasts with others, creating new stimuli for your potential respondent.

Utilizing these survey tips in conjunction with adding a "Thank You" note to your survey project outline are just some of the ways you can ensure your products are recognized as standing out from the crowd, especially in this all-too-hectic world.

Creating an evaluative tool to measure an employee's performance can be a daunting task for even the most experienced managers.

Here are some survey design tips to help you create effective performance evaluation materials that will be meaningful for both supervisors and supervisees:

1) Use titles that are less challenging for employees (e.g. calling the instrument an "evaluation" as opposed to a "test")

2) Have a place at the beginning of the job performance evaluation form to clearly delineate the employee being evaluated, such as their name, title, department, and other pertinent job related individual information, like hiring date and date of last review. Other non-job related demographics (such as employee age or eye color) should be left out.

3) Make sure the content the employee is being evaluated on always refers directly back to their position. This can include technical job-related skills, and "softer" characteristics such as courtesy to both clients and co-workers, or punctuality.

4) Employee evaluations lend themselves nicely to the use of Likert scales, but a good evaluation uses verbal measurements as opposed to numeric. For example one end of the survey rating scale would be "Needs Improvement" and the opposite end of the scale "Excellent Performance."

5) Leave plenty of room for written employee performance evaluation comments after each content area. Both the manager AND the employee should write down their thoughts about the content area discussed. This makes both parties feel like they are contributing equally to a conversation, rather than one person telling the other how to act.

6) Include space for concrete development plans and steps to be accomplished, including dates and time lines for the progress to take place. It's also a good idea to include mid-term progress review dates so manager and employee can check in with each other. This ensures both parties are still on target for a successful future review.

7) Allow both the employee and manager to sign the list of employee evaluation questions and responses after reading all parts thoroughly and together. Leave time for discussion of the evaluation. 
 
8) If your company has an HR department, have an appropriate HR supervisor review your staff evaluation form to double-check that all the right notes are hit.

The Grand Island Independent recently published the results of a survey conducted at St. Francis Medical Center in Grand Island, Nebraska.

Some of their key findings? Mentors not only help retain staff, but also increases the quality and quantity of job performance in the workplace.

Approximately one-third of all St. Francis' employees have a mentor, an unusually large number for any organization. Mentors were most likely to be an employee's direct supervisor, and were most effective for new-to-the-job employees.

What do these results mean?  In these tough economic times, utilizing the benefits of mentor-mentee relationships is a great way to improve employee retention programs through decreasing staff turnover, increasing job security and grooming more staff for leadership positions, all huge cost-savers for an organization.

Perhaps most important for both employees and their companies, having mentors in the workplace can help everyone learn to do their job better.  In other words, an increase in workplace performance.

Find out what's working well for your organization, and what may not be working by following in the footsteps of St. Francis Medical Center. Learning more about your business research needs.

Perhaps you have an extra special research project that really needs a good response turnout, or you’re simply looking for ways to boost your response rates. Well, maybe its time to try sweetening the pot a little?

Studies have shown that surveys where potential participants were offered monetary incentives of as little as $1 received higher response rates then ones that didn't’t offer any incentives at all. In fact, offering any incentive to your potential survey respondents as a way of saying “Thank You” even before they take your survey is a great way to get started on the road to higher survey returns. Good examples of survey incentives that work are gift cards, checks, gift certificates, and of course good ol' fashioned cash.

The amount isn't as important as the belief by the participant that the offered incentive will actually be provided. Credibility is key. So if you choose to offer incentives to your potential respondents after they have participated in the survey (say for example the option to be entered into a random drawing for a chance to win an iPod) remember to follow through on giving out the reward after you collect feedback from all the survey respondents!

This tip is particularly useful when trying to avoid survey bias. By offering incentives you're more likely to attract people who would not normally participate in your surveys. Learn more about minimizing survey bias from some of our other posts.

Sometimes we get so excited about creating a survey we want to jump right in and ask the really interesting questions. Those typically consist of finding out about who our clients are, where they're located, etc. (factual questions). In other words, our demographic questions.

But if we don’t take the time to think about placement of these really important factual items within our surveys, we may be inadvertently affecting the results we see. By placing your demographic survey questions (name, gender, race/ethnicity, age, etc.) close to the end of the survey feedback form as opposed to right at the beginning, you are increasing the chances that respondents will fill out these items, leading to richer data collection and more useful results. Respondents are more likely to feel invested in your survey, and will understand why you are asking those personal questions if those sensitive items are placed after survey questions that address the purpose of your survey.

Check out these blog posts to get started with your demographic survey questions.