Survey Creation

Designing a survey instrument can be one of the most fun aspects of conducting survey research. In this two-part tutorial, we'll first look at some of the "Don'ts" of writing survey questions, and then we will look at some of the "Do's".

Once the brainstorming of ideas phase has been accomplished and you have a good idea what the scope of your research is, it's time to get down to putting question ideas on paper (or online!).

However creating good survey questions can turn into quite an art form, as we all struggle to word our items in such a way that our participants will understand what information we are asking for. Recognizing when you've accidentally designed a survey question that is worded in a non-objective way is the sign of a good researcher. Let's see if you can spot the poorly worded question from this group of sample survey questions:


Now that we've looked at some question items that shouldn't be put into surveys, next time we can define some ways we can create objective question items that will provide us with the results we need!

In a recent post, I described the differences between nonprobability and probability sampling methods in online survey designs.  Probability methods are preferred if at all possible, because they allow you to make generalizations from your electronic survey results to a larger population or target audience.  One kind of probability-based sampling technique is called systematic random sampling. 

To employ a systematic random sampling design for your online web survey, you first select a case at random from your exhaustive population list, and then select further cases at identical intervals, determined by how many people you want to sample in total.  If you wanted to sample ten people from a population list of 150, you would then choose every fifteenth person after selecting someone in the first 15 cases (to ensure you will select 10 people in total).

This provides an easy way to obtain a random sample of your population list or sampling frame, because as long as your data is ordered randomly, you can begin simply by selecting any record or case and go from there.  This is an important caveat though: your records must be randomly ordered for a systematic sample to be effective. 

Take this example of survey sampling, let’s say you have a sampling frame (list) of people that is currently ordered alphabetically by last name, and you are interested in subgroup analyses by ethnicity.  It would be wise in this case to rearrange the records into a truly random order (i.e., not alphabetically), because last names from certain backgrounds may be more likely to begin with a particular letter.  While systematic sampling provides an easy way to generate a random sample for online surveys, you do need to be sure there is no hidden order within your population list or sampling frame.

Some people still believe in using paper surveys instead of designing online questionnaires, sending personalized email survey reminders and utilizing electronic survey methods. Sometimes I have to wonder, Why? Some of the reasons people claim they want to use paper questionnaires instead of web survey forms is because they think it's cheaper. That's not usually the case.

Consider this: Online survey response rates are typically hire than response rates on a direct mail survey. Online survey research designed well will yeild response rates from 20-40%. Let's be conservative and say the paper survey you plan to send out will get a 20% response rate as well. I looked up the cost of a ream of standard multipurpose paper (500 sheets) on Office Depot's website: $7.09. A black ink cartridge for a laser printer is $77.99 and will print on a maximum of 2,200 pages. Most printed surveys will be kept to three to five pages, longer than that your survey form looks over whelming from the very beginning and is going to lower your response rate. This is how that cost quickly adds up:


Keep in mind these costs don't include the staff time of stuffing envelopes to send out the direct mail survey, or the staff time of manually entering in all of the responses into some sort of data collection software, be it Excel or a survey software tool. You can figure it will take an intern a week to send out 5,000 survey responses and another week for them to do the data entry for the 1,000 responses. So that's an additional cost of $600 to $800 for one survey campaign. What if you want to do a quarterly customer satisfaction survey? You're looking at an annual cost of over $25,000.

For me, that would be enough reason alone to look into an online survey software application. With a web survey tool, you have the ability to not only send one invitation, but reminders survey emails as well. Sending reminders, as we've mentioned before, can drastically increase your survey response rate. My suggestion: don't fall for the myth that paper surveys are cheaper than online surveys. It's definitely the other way around.

There advantages and disadvantages to conducting web or electronic surveys as opposed to traditional survey modes such as personal interviewing, telephone and mail.  One of the advantages of an online survey design is a possible reduction of what is known as social desirability bias. 

Social desirability bias occurs when survey respondents offer responses that portray them in a positive or more favorable manner to others. 

When a face-to-face interviewer asks personal or sensitive questions respondents feel have a “good answer” and a “bad answer” (such as criminal behavior), they may underreport bad behavior and overreport good behavior, for example.  While this bias may be reduced in online surveys due to the absence of an interviewer, there are topics that may produce invalid or unreliable data, regardless of survey mode.  This is something to consider before finalizing your online questionnaire design or web survey forms.

Survey questions within the following content areas are especially subject to social desirability in a survey form (i.e., respondents believe particular responses are “better” than others). 

A key marker of a quality survey, whether that survey is an online form or another mode, is replicability of your results over trials. Replicability of your survey results lends credibility to your organization's research work.

One indication that your survey form may have results based on biased samples is nonreplicable results for the same instrument, fielded with the same survey design, from one time to the next.  If the same sample frame produces significantly different results for identical questions, that should perk up our noses to the scent of possibly unreliable results. We should start to look at the criteria used to select our survey sample, and take a fresh look at the questionnaire to see if it is still pertinent to our needs, or the needs of our clients.

This holds true for all modes of survey research, including online survey internet research. Receiving divergent findings in survey results being fielded again and again could lead to differing (and possibly harmful) business decisions.  No company should be left with uncertainty about answers and actions when decisions need to be made based upon the research. This is just one reason why replicability of your research findings is a key ingredient to sound data. When in doubt- replicate!

What is a survey design's ideal length? In the past we've told you the typical survey best practice says surveys, whether they're paper or web survey forms, shouldn't be longer than thirty questions. While thirty certainly isn't a huge number, if you tell me you have a thirty question survey you want me to complete, forget it. I'm never going to complete your feedback form. However, if you tell me it's only a 10 question survey, I'm more likely to be on board. That is, unless I start to give you the customer feedback you're looking for and feel like your online questionnaire is longer than 10 questions.

You probably didn't lie to me, at least not if you're read about the dangers of lying about survey length in your email survey messages. It's probably a case of side view mirror syndrome: objects are closer than they appear - or applied to web survey design, questionnaires appear longer than they are.

Not surprising, if your survey form is interesting, the survey respondent is going to perceive it's shorter than it really is. However, the reverse doesn't seem to be true. If your web survey feedback form is boring and on an uninteresting topic, respondents will perceive it's the same length (and too long) whether there are fifteen questions or thirty questions.

It's not just about the actual length of your form when designing a questionnaire online, it's also about the perceived length. Even though your survey is only ten questions long, it may feel a lot longer if your collecting feedback about an uninteresting topic. If your survey questionnaire feels long, you instantly increase your chance of survey respondents abandoning your feedback form.

It is crucial that you consider reporting a main element of your web survey design at the outset of your research project.  What you can say about your results hinges heavily on the types of analyses your questions and the capabilities of your response scales.  Today, I will outline the difference between the two major branches of statistical analysis available for most survey data: descriptive and inferential.

Descriptive statistics are the basic measures used to describe survey data.  They consist of summary descriptions of single variables (also called “univariate” analysis) and the associated survey sample.  Examples of descriptive statistics for survey data include frequency and percentage response distributions, measures of central tendency (which include the mean, median and mode), and dispersion measures such as the range and standard deviation, which describe how close the values or responses are to central tendencies.

Inferential statistics offer more powerful analyses to be performed on your online web survey data.  As the names suggests, this branch of statistics is concerned with making larger inferences about social phenomena.  This can include associations between variables, how well your sample represents a larger population, and cause-and-effect relationships.  Some examples of inferential statistics commonly used in survey data analysis are t-tests that compare group averages, analyses of variance, correlation and regression, and advanced techniques such as factor analysis, cluster analysis and multidimensional modeling procedures.

By designing online questionnaires and survey web forms with a good idea of what you want to do with your data after it's collected, you can create cohesive, powerful reports and presentations. Need more tips for how to analyze survey data, read some of these data analysis posts.

What is a survey? The Merriam-Webster Dictionary defines the word "survey" in this way: "To query (someone) in order to collect data for the analysis of some aspect of a group or area."  These "someones" can also be called your survey sample. Unlike a census, which surveys an entire population, a sample survey gathers information from only a fraction of the individuals within an entire group. Samples are collected in such a way that results from the survey can then be generalized back to the entire population.

Surveys can be conducted in a variety of ways, which can also be termed your survey design. They can be conducted over the phone, through responses written on paper, through in-person interviews, or through online web surveys. Even though the mode in which a survey is conducted can vary, sample members are all asked the same questions in a similar or standard way.

Some survey forms focus on opinions and attitudes (e.g. asking your clients how they feel about a new product line), while others focus on factual questions (household income levels, or age of members). Most online survey design contains a combination of both types of questions. A survey design can also vary in length and time it takes the sample to complete. Some may take only a few minutes of a person's time, while others may take hours or even days to complete. 

Surveys should be carried out only for the purposes of gathering data and information on a certain subject. A quality survey can be determined by looking at the way in which it was conducted, and its purpose.

Get started designing your own survey using one of over 50 graphical templates available, or read more blogs on designing surveys.

There are many options available to market researchers when designing a web survey form. I covered this topic a bit last week when I discussed creating online surveys, and included some dos and don’ts.

Conducting a study for the Web offers several opportunities for market researchers that paper questionnaires do not. For example, you can create an instant survey and check on responses frequently for updates. You may even allow the respondents to view the current results when they complete the survey. This works best for one or two-question quick polls rather than longer studies. But knowing they will see the results immediately is frequently a tipping point for respondents who are on the fence about completing your feedback survey form.

This can be helpful in analyzing survey data as well. For example, suppose results for your online survey form vary wildly throughout one week. One day, 10% of respondents use your client’s brand of glass cleaner. The next day, it’s 65%, but then the following day it drops back down to 20%. This could mean something (possibly that you need to check your recipient list) and is good information to track for your client.

The instant results also allow you to add more questions and gain clarification on points you may not have considered prior to launching the survey form. There are dozens of survey web software tools available that can create instant survey results – it’s a method that is worth trying out, but as always, consider your client’s needs before committing to a particular survey method.

Designing survey invitations so that they render correctly in the person inbox is critical. However, with every email client, from Hotmail to Gmail to Lotus Notes to Outlook, have their own rules for rendering HTML email. In the past, I've given you updates on changes Gmail has made that's effected email rendering. Today, I wanted to share some tips to ensure you're creating emails that will look good in Outlook 2007. With Outlook 2003, the general rule of thumb is if it looks good in Internet Explorer, it will look good in Outlook. Outlook 2007 doesn't play by that rule.

Depending on your target market and who you expect your survey respondents to be, you may care more about these email survey design tips. If you are a business-to-business organization, most of your email marketing list is probably going to have business emails on it as opposed to freemail domains. If you're a business-to-consumer organization, you may care more about the rules Hotmail, Yahoo or Gmail use to render your emails.

Next time you're sitting down to work on your email survey design, keep these Outlook 2007 tips in mind:

Keep styles in line: This is a general email marketing best practice because a lot of email clients don't support embedded CSS. Outlook 2007 does support embedded CSS, but not all properties work the way they should when their rendered. To avoid running into these problems, just keep your styles in line.

Avoid animated files: Animated .gif files are not going to work in Outlook 2007. Instead of animating, the email will just render with the first image in the animation. As long as you're happy with this first image of the animated .gif file, then it's not going to detract from your email message. Just know that when you go to test your email survey invitation in Outlook 2007, nothing's going to happen.

Specify accurate table widths: This is another email marketing best practice for designing online survey invitations. You should use tables to design HTMl emails. If you have multiple columns and your column widths don't add up to the width you specified for the overall size of the table, you're probably going to run into issues when someone tries to read your email in Outlook 2007. To avoid this, just make sure you're doing your math correctly, or use percentages instead of pixels (just make sure you're adding to 100% not 101%).

Use Alt tag: In Outlook 2007, images are blocked by default. While some users may chose to turn this off and have images automatically show up - you shouldn't count on this. To avoid survey respondents from having no idea what your email says, be sure to include alternative text for all images.

No Forms: If you're trying to embed a survey form in an email message, it's not going to work. Outlook 2007 disables embedded forms. Instead of embedding the form, include a link to your electronic survey hosted either on your website or your survey software company's website.

For the most part, these email survey design tips shouldn't be that hard to follow. In fact, if you're using Cvent Web Surveys software, our email survey tool will keep some of these tips in mind for you - and you don't need to do anything. What do I mean? Our easy-to-use HTML editor automatically codes the email for you, so when you insert a picture, it's going to ask you for alternative text. When you are changing background colors or applying different rules to headings, it will make sure to code the in line style versus embedding CSS. If you're not using an email survey software tool that keeps email marketing design best practices in mind, you're going to need to do the research yourself to find out what different clients allow - or don't allow - in emails.

So, it is clear that Cvent Web Surveys software is always getting better. Remember that 80 percent of the changes made in your accounts with our quarterly product releases are enhancements that were requested by current clients. However, it is not only the online survey application that is consistently changing for the better. Here at Cvent, we are also constantly rethinking and revamping our training classes to serve our clients the best that we possibly can.

I wanted to write a post today just to let you know we recently rewrote the script used for the Survey Creation and Management training class. Another member of the Web Surveys Client Services team and I spent about a week meeting with the Senior Vice President of Client Services and Relationship Management to completely revamp this training class in order to better suit your needs. Although the material covered in the training class is by and large the same as it was before, it is our firm belief that the Survey Creation and Management class is now easier to understand and follow than it had been previously.

Just to reiterate, if you recently signed a contract with Cvent, it is highly recommended that you sign up for this class. Survey Creation and Management is offered every Wednesday from 2:30 PM EST to 4:30 PM EST. We also offer separate classes for our clients in the UK and Australia. The class for UK clients is every Wednesday from 8:30 AM to 10:30 AM GMT. The Australia class is offered Thursdays from 3:00 PM to 5:00 PM Australian Eastern Time. Finally, you can also listen to our recorded training classes if your schedule does not permit you to attend the live training.

In summary, if you are a new survey client or a new user of the Cvent system, please attend the Survey Creation and Management training class at your earliest convenience. Since this class goes over how to set up your first electronic survey in Cvent, how to import your contacts in bulk into the system, and how to send your survey invitations with the Cvent email survey tool, it is extremely beneficial for you to take as soon as possible.

Web 2.0 is a term that is frequently tossed around, but there are still many who may not be entirely clear on the definition. The simplest way to boil it down is that Web 1.0 offered little or no opportunities for feedback, while Web 2.0 includes countless ways to “talk back” on the Web. It can include blogs, forums, social media sites, and interactive informational sites like Wikipedia. The users can control the content to an extent on these types of sites, a difference from the static web sites of Web 1.0.

What does this mean for a market researcher? I’ve already discussed methods to use social media for gathering data and survey respondents, but you can use other Web 2.0 sites in a similar manner. It’s a huge topic that I am sure will be revisited again and again.

Web 2.0 sites are good for research purposes. People who would likely make the ideal survey respondents are using these sites to talk back about products. For example, if you were doing a baking survey, the blog Bakerella.com has hundreds of posts and thousands of comments about various recipes and their merits. It is a good start for conducting internet research about the habits of the true baking enthusiast, and could help you in your online questionnaire design.

Despite the opportunities for feedback Web 2.0 offers, it is important to remember that it does not and should not replace more traditional marketing research methods, like electronic surveys, paper questionnaires and other methods. Although respondents are very forthcoming on blogs like Bakerella.com, for a quality study respondents must be properly vetted to ensure they are the correct group.

When designing surveys, researchers often want to find out how people act in addition to how they feel.  Questions about actions and behaviors are especially useful in market research, since you can gain a sense of what customers and consumers are actually doing (as opposed to what they say they’ll do).  However, this area of survey research can be difficult – to the point of being unreliable – so I urge you to follow the guidelines below when asking behavioral questions in your online questionnaire design:

Provide a specific time reference within your survey questions.  This allows for an “anchor” to be set, and can give you a clearer idea of what your respondents do during an average time frame (be it a day or a week).

Make the time frame fairly recent.  It is very difficult for respondents to remember what they did three years ago.  At the very most, stick with questions that reference 12 months ago or less (preferably much less, like the past week or even 24 hours).

Ask only about the respondent’s own behavior.  Unless you’re asking about the actions of small children, it’s best to keep survey questions directed at the individual level rather than for family members or friends.  Asking people about others’ behaviors can provide distorted, unreliable data, as most people overestimate or underestimate certain actions of other people.

Personal and sensitive information should be taken with a grain of salt.  Multiple market research studies have shown the more sensitive a question is, the less likely it is that respondents will answer... and the data provided can be inaccurate, so it is important to consider that people might dance around the absolute truth on these kinds of questions.

By now most of us know about the many benefits of online/electronic surveys. Some of the major benefits that come to mind include: Access to a possible world-wide population, negligible distribution costs, data is automatically analyzed or exported to other databases with ease, pre-programming the question order, easily trackable response rates (or choose anonymous responses), automatic reminders, thank you's, and auto-close dates. Additionally there are great perks such as spell checkers, and built-in math calculations.  

Some less well-known benefits that make online instant surveys even more desirable include: adding "help buttons" or links that can provide your respondents with additional explanations, designing surveys that are easily pilot tested, progress bars for respondents, required questions, validation (only "acceptable" answers are allowed), branching, piping and skipping patterns. 

We can already see the many benefits to online web surveys without writing out an exhaustive list. However, here are some considerations to keep in mind before committing yourself to an online survey design: Do you have a survey form software tool that can keep up with advancing technology? That is, does your instant survey software have the ability to include all the features mentioned above and more, such as importing pre-written questions to save time? Does your survey company's software have a "back-up survey" policy in case of technical glitches? Are your surveys hosted on secure URLs to provide respondents with confidence in your electronic survey? Can you customize your online survey form with your own brand? 

Whether you'd prefer paper questionnaires or online survey forms to for quality data collection, Cvent can assist with both. Check out all the useful info that's covered in Cvent's online data collection blogs, or sign up for a free webinar to learn more survey best practices.

It is tempting to include many similar question types with similar response options in your online survey design.  Matrix questions, for example, provide an efficient questionnaire design method to help you gather lots of data in a neat, brief survey form.  It is wise, however, to resist the urge to use too many uniform survey questions and response lists, namely because of two sources of bias that stem from doing so: habituation and acquiescence.

Habituation occurs when respondents begin providing the same answers to survey questions with the same response options.  They start to get in a habit and select the identical response choice for every question.

Acquiescence is related to habituation, and occurs when respondents passively agree with an interviewer or survey questions.  Agree-disagree scales are the most often-used response options in opinion surveys; it is important that you take steps to avoid the chance that respondents will passively agree with your statements in order to quickly complete the questionnaire or provide what they think may be the “right” answers.

To avoid these response biases, you can use online survey software that allows question randomization, break up your matrix questions with other types of questions and scales, and phrase some questions in a manner that makes respondents switch their thinking.  An example of the latter would be to ask a series of positive questions in your survey questionnaire, and then throw in a couple questions worded differently so as not to allow habituation or acquiescence.  Use care up-front in your online questionnaire design to be sure that you'll reduce error and bias in your results.

I think I have always thought of myself as a "planner." While some people "fly by the seat of their pants," I am constantly thinking ahead to the consequences of my actions. The way I see it, this can be a good thing and a bad thing. Being a "planner" is bad because I am not as spontaneous as some of my friends. The "life of the party" is usually not someone who thinks ahead. However, it is a good thing at the same time, especially when operating in the corporate world, and specifically when designing online surveys.

I wrote a post a couple weeks ago about the importance of thinking ahead to reporting before designing a questionnaire online. I want to address a related issue: survey question logic. We on the Cvent Web Surveys Client Services team always strongly encourage our clients to use survey logic to their advantage. Lately, however, I have been getting lots of calls from clients who are having trouble with the logic they have applied. The reason for this is simple: too many clients do not plan or map out the logic they'll apply to their electronic survey. This lack of planning can cause numerous problems, most notably conflicting survey logic.

In the past two weeks, I have spoken with two clients who applied both branch logic and advanced logic to their online surveys. Both of these clients called because they were concerned there was a technical issue which was prohibiting the logic from functioning as it should. "But, I set up my branch logic correctly, and I know that my advanced logic should be working fine too, so what is wrong?" In both cases, the problem was the clients had applied branch logic which conflicted with the advanced logic, and the survey system therefore did not know where to send survey respondents when they answered a question a particular way. "If only these clients had planned ahead and mapped out the survey logic before launching their surveys," I thought, "then this problem could have been avoided."

Moral of the story: Please take advantage of the wonderful survey logic functionality available with Cvent Web Surveys software. However, at the same time, make sure to plan ahead. Even if you are someone who generally "flies by the seat of your pants," when designing online survey questionnaires, you should be a "planner" for once.

In the past several years, many market researchers have shifted to using online surveys more frequently. Traditionally, these online surveys were emailed rather than posted online to avoid data skewing.

When creating the online survey design, there are several things to consider. Make sure the electronic survey is simple and easy to use. Too often, online survey forms are cumbersome or technologically dated. Keep it simple, and you won’t run into any problems.

You must also plan carefully to make sure no questions are missed. You can design online surveys so they will not allow survey respondent to move on if mandatory questions are not completed. In addition to requiring survey questions, I would advise setting up the internet survey so respondents cannot complete the questionnaire more than once. Most online survey research software applications have features to create limits on how many times someone can fill out an online survey form. This prevents skewed data – some respondents will try to complete online survey forms multiple times if there is a giveaway or drawing offered as an incentive.

Make sure you read over your electronic survey a few times as well. Proofreading is especially important when designing online surveys. You must have members of your team proofread not only for grammar and clarity, but also functionality. Make sure all the links work, and the survey can be completed with all your caveats and mandatory questions. 

Online surveys have made it possible to gather information much quicker, and when used properly provide an excellent resource for market researchers.

All too often, we begin the survey creation process, but draw a blank.  How should we design a survey?  Should a survey design have the company look and feel?  Or should we design it to be a fresh look?

Having an online surveys tool that has survey templates built into the system is extremely important.  It eliminates the nagging question, "What is a survey design? How do I create a professional looking web 2.0 survey?"  This gives the survey builder that boost to get the creative juices flowing. In the Cvent Web Survey solution, there are over 50 graphical templates, ready to help you in designing surveys. Any color, any pattern, any style. They are all preloaded in the Cvent Web Surveys application. Definitely check these out the next time you are trying to figure out survey research design ideas for your next questionnaire.

Want to learn more about our survey design software? Sign up for a product demonstration.

Rarely are we asked the question, What is a survey? Typically questions follow the path of, Why do I need a survey program? What am I going to get out of conducting online web surveys to collect customer feedback? Or How do I get started measuring employee satisfaction with online questionnaire templates?

However, sometimes it's good to define market research and what it means, and answer the rarely asked question, What is a survey?

Market Research Definition (mahr-kit-ree-surch)
The gathering and studying of data relating to consumer preferences, purchasing power, etc., especially prior to introducing a product on the market.

Survey Definition (ser-vey)
Collect quantitative information about items in a population. Surveys of human populations and institutions are common in political polling and government, health, social science and marketing research.

Chances are though, you're still wondering about those other questions. These posts may help you answer those ever burning questions about why you should create and design surveys.

Yes, you read right; we’re not talking about cloning people. But wouldn’t it be awesome if you could clone your electronic surveys? The advantages are endless. If you have an internet survey that has to be sent out to people who attended different conferences on different dates, the headers and welcome pages would be different – as would the responses coming in, but the questions and emails would be almost exactly the same. Or if there’s an old online survey, closed and archived years ago, but you’ve realized you need to send it out again – clone it!

The best part about cloning surveys is that you can still make changes to them – so if you were designing surveys with similar content for five different groups of people, you could create a basic survey with all the questions, headers, templates, security settings, etc. and make minor changes to each of them so that each group gets a customized online web survey, but you only spend a fifth of the time.

When you clone a survey form, all information except the people added to its targeted list, gets carried over. So the next time you spend hours adding complicated branch and pipe survey question logic to an online questionnaire, worrying all the time that you will have to do it again and again, you can breathe a little easy. You always have the option to clone a survey irrespective of whether it’s in Test mode, Active, Closed or even Archived. Once you’ve done the work, you will never have to do it again.

So go ahead and clone your next survey – send it out to different people, generate different reports for each of them – you can even have one as anonymous and the other collecting contact information from respondents. As for cloning people, that debate will rage on.