Questionnaire

When conducting market research, either through survey forms or another mode, the method of sending a survey pre-notification letter can be a tempting one.

For some time now survey methodologists have been conducting studies of pre-notification letters to determine if they have any affect, either positively or negatively, on survey response rates. An overall analysis of these various independent studies revealed an increase in response rate of approximately 8% when compared against studies where no pre-notification letter was sent.

No one can say for sure exactly why pre-notice survey letters seem to assist in increasing the number of survey responses, but perhaps they help to establish the legitimacy of a survey, contributing to a respondent's feeling of trust and the credibility of the organization conducting the research. Another possibility is that a pre-notice letter builds expectation of arriving mail (be it electronic, paper, or in-person). A third possibility is that a potential respondent is less likely to disregard the survey when it arrives if they are aware it is arriving shortly.

Although survey pre-notification letters are an excellent (but sometimes expensive) way to increase response rates, they are seldom used in marketing research surveys. Each researcher needs to weigh the additional cost of sending out a pre-notice to potential respondents against the probability of a lower response rate.

It's worthwhile to take your population's survey sample size into consideration when coming to this decision. When your sample sizes are small, each individual response affects your survey results even more. Increasing those responses through utilizing a pre-notice letter may counterbalance the increased cost of sending them out to your participants.

In the past, I've shared online survey best practice tips you should keep in mind when designing a questionnaire to ensure quality survey data. We like to always share survey best practices, and today is no exception. Here are six more notable online survey tips.

Keep your email survey invitations short: Email marketing best practices always tell us to keep emails short and simple, with one clear call to action. In the case of survey emails, your call to action is the link to access the online web survey form. Be sure to include these important points to help recipients make an informed decision about whether or not to complete the survey:
Send personalized email survey invitations: Emails with a personal salutation result in increased response rates. Send your education survey as "Dear Mr. Wright" or “Hi John,” rather than "Dear Valued Alumni." In addition, you can insert other contact fields like organization names, work cities, amount they last gave, etc. throughout the body of the email, in order to make it highly personalized for each recipient.

Include the survey link in the email: Be sure to use the appropriate link in your outgoing emails. The default link in the emails {[S-RSVP LINK]} takes your respondents to the first page of the survey and skips the Welcome page. However, you might want to change this link to {[S-HOMEPAGE LINK]} if you have added important information on the Welcome page.

Incorporate an opt out link in all emails: Cvent requires that each email contain an opt out clause for all external surveys. It is very important that all of customers maintain compliance with the CAN-SPAM Act.

Launch survey invitations automatically: After you have tested the email invitation and your survey, you are ready to send it out in bulk to all of your invitees. We recommend that you switch the How to Send option on the Edit Settings page to the Send By Date.

Remember: Timing is everything: In order to make sure your email invitations are opened and read, avoid certain days and times. Never send out an email over the weekend or on Monday morning. This is when people’s mailboxes are full, especially with dreaded spam. We recommend sending emails in the middle of the week and the middle of the day.

While designing a web surveys is fairly simple, make sure you are aware of common online survey pitfalls so you can avoid them:

• Plan the correct date and time to send survey invitation emails and consider your target audience. If your survey sample is office workers, it would be best to send it out during the work week when they will be at a computer most of each day. Do a little research on what the best day is – while it is widely known that Fridays, Saturdays and Sundays are not optimal, there is always new information on this topic. Don’t forget to consider the time zone of recipients too – if you are sending something that will land at 7 p.m. their time, it will likely be buried or forgotten by morning.

• Create your questions carefully. In an online web survey, the respondents are obviously on their own. You can’t clarify things for them. Make sure the questions are clear and easy to understand.

• Keep the open-ended questions to a minimum or you will be poring through the answers for hours. Respondents tend to be more forthcoming with electronic surveys, and this is a positive thing. But just because you can get all those open-ended responses, you may not really want them. Always be thinking ahead to how you will analyze survey data and present findings.

• Entice the respondents to take the survey. Many times online surveys fail to offer enough incentive. Even though online surveys are more convenient, the respondents are still volunteering their time and should be rewarded.

As is the case with any research method, there are advantages and disadvantages to using an online web survey to collect data about customers, employees or the public-at-large.  For example, online surveys offer you the ability to gather vast amounts of data from many respondents at the same time, get your data back in an electronic form, see real-time results and automate analysis/reporting tasks... and you can do all of this affordably.  You can even use a relatively small research survey sample to accurately estimate the opinions of your larger population (for survey research in general).

But in some cases, it is best to use a “mixed-methods” approach to your research project.  This means you combine the online survey method with another kind of investigation, such as interviews or focus groups, in order to produce more well-rounded data and conclusions.  Here are some examples of when a mixed-methods approach is likely better than an online survey form by itself:

A survey conducted by research firm BIA/Kelsey found that 9% of small and midsize business use Twitter for marketing purposes. An earlier survey by online marketing resource, MarketingProfs, showed that 84% of small businesses expect they will increase their use of Twitter and other social media outlets in the future. That same report also takes a look at how large businesses - including Dell and Disney - are using Twitter to reach devoted fans and potential customers. Companies large and small have begun using Twitter. I find this list of companies, although old, to be quite impressive!

At Cvent Web Surveys, we often recommend the use of Twitter, Facebook, and other social media outlets to reach a wider audience for your web survey. We've often found it to be most useful in monitoring and acted fast to ensure customer satisfaction. Bad news travels fast these days, and you have to be ready to react. 

If you are distributing your electronic survey in multiple channels (social media, your public-facing website, and through Cvent's email survey software platform) it is important to track where people are coming from. By analyzing where your respondents are coming from, you can more efficiently distribute your surveys in the future or weight responses. This is especially important if you are paying to advertise your survey - perhaps on traditional banner display advertising, website pop-ups, or search ads such as Google AdWords.

Cvent Web Surveys software provides two ways of tracking and analyzing this data. First, we have our "Reference ID". Reference IDs are a short string of text added to the end of your survey URL – "twitter", "yelp", or "google" – for instance, to allow you to track what other website "referred" the survey respondent to complete your online survey. These strings of text will then allow you to use the Cvent's built-in survey reporting to see, analyze, and slice and dice the information to give a clear view of where the survey respondents are coming from. 

Second, Cvent was proud to announce in our last quarterly product enhancement, the addition of Google Analytics integration. Google Analytics is Google's free tool for tracking visitors to your websites. The integration with Cvent Web Surveys software is extremely simple to set up and is a fantastic addition for our clients who use and love Google Analytics.

Now while I heartily recommend using social media as a tool to get your survey in front of more people, there is one thing that is crucial to remember: many businesses block social media websites. Robert Half International, an IT recruiting firm, reported 54% of companies completely block access to sites such as Twitter, Facebook, and even business-oriented LinkedIn. This means, if you're trying to attract those of us who work, you may be facing a higher hurdle than you imagined.

Last week, we reviewed poorly written survey question items, and what makes them inefficient tools for collecting the data you desire. For our final installment of "Writing Survey Questions the Work," we'll discuss some final points to keep in mind in order to write great questions.

In general when you design a questionnaire, try to write from the respondent's perspective rather than your perspective. Hark back to last weeks post where we discussed writing objectively and from a non-assuming point of view. Writing from neutral viewpoint and keeping far away from internal workplace jargon and slogans that wont be understood by your clients is key.

Having a sample of your survey population and other neutral testers review the wording of your survey is a wonderful way to attain clarity on question items. After spending what may amount to dozens of hours on a survey form project, having an outsider do things like proofread and spell check with a fresh pair of eyes can be well worth the effort!

And as we've talked about in other posts, pre-testing your survey with a segment of your audience is crucial to detecting question item mis-wordings as soon as they occur.

If we have poorly worded question items in our survey, not only will the answers we receive be inaccurate, our data analysis will be inaccurate as well. This could result in poor decisions being made based on data that has either overstated or understated our clients needs for a product or service. 

Staying in tune with the needs of our clients and our target populations starts with skillfully worded survey question items. If your question isn't worded right, your analysis wont be right.

One of the best descriptors attributed to good survey questions is the word specific.  A main goal in designing valid, reliable survey questions is doing everything in your power to make them clear, standardized and unambiguous.  A great way to follow through on that is to make sure your questions are as specific as possible.  The degree of specificity affects how people interpret and respond to your survey questions.  Several examples are listed below to help you analyze your survey questions regarding their level of specificity.

Be clear with demographic questions.  Your objective here should be to use words and categories that your respondents can clearly understand and identify with.  Will you use Census designations for Ethnicity?   Exactly what do you mean by “marital status?”

Define vague concepts, words and phrases.  Your respondents will likely widely differ with regard to their backgrounds, experiences and perceptions.  Words like “justice” and “equality” can mean very different things to people, so it is best to define exactly what you are asking about.

Objective or subjective?  Use verbs that trigger respondents.  Attitude survey questions are usually subjective (i.e., how do you feel about the war?), while behavioral questions are usually objective (i.e., what did you eat for supper last night?).

Always try to attach a time frame to behavioral and recall questions.

For market and product research, identify the actual brand name, and ask how they feel about specific items, not just groups or genres.  For example, don’t use the vague word “furniture” if you’re really interested in how they feel about a table or a chair.

Survey designing software can't look at your questions and tell you if your online survey form will give you the data you want. However, it does make it easier when you're designing a web survey because it takes out the added complications with writing your own code. So make sure next time you're working on an instant survey form, you think about the specificity of your survey questions.

When you plan your online market research study, you still need to remember the standard steps for any market research project, plus some online specific tasks:


The time frame for an online study is, as discussed previously, often shorter because of the immediacy of the results. Keep this in mind but be sure to give yourself enough time to analyze survey responses.

Every year it seems like the holidays get earlier and earlier. This year I noticed stores had holiday decorations out before Halloween costumes! While I can attest to retail stores being ready for the holiday shopping season, it makes me wonder, are you?

Online web surveys are just one more way you have the ability to engage with your consumer email lists. Customer surveys should be no brainers. But how can an online survey form apply to holiday marketing strategies? Well that depends, are you setting survey objectives that will impact your marketing strategies?

Here's what I mean: I receive countless retail marketing emails everyday. Most of which I opted in to, however, no one ever asks me about my shopping habits or preferences. These companies could easily put together a retail survey using an online survey application and find out more about who I am as a customer, what I prefer, how frequently I shop in their stores or how frequently I visit their website. From the responses they get back, they can then customize their marketing messages based on how I shop with them. If I only shop online, it may make sense to group these customers together and do a special online promotion. All the customers who only shop in the stores would receive a different promotional offer because they shop differently than I do.

This concept goes back to one I've shared previously: profiling customers to better understand your organization. The holidays are closing in quickly, but it's not too late to build an online survey and collect customer feedback instantly.

How have you designed online survey questionnaires in the past to impact your marketing strategies?

Too often companies make the common mistake of surveying customers after it’s too late. Many will use customer satisfaction surveys to gauge how they did last month or last quarter. In truly proactive, customer-focused organizations customer feedback is used on a daily basis to increase revenue and improve customer satisfaction and loyalty when the timing is critical for the customer.

Most customer feedback collected is of immediate importance to them and requires immediate attention. Waiting until a survey closes to attend to customer needs is a recipe for disaster. This could include missing out on hot leads for new business, up-sell and cross-sell opportunities, or worse not reaching out to identify the dissatisfaction from an irate customer.


Your customer is seeing how much you really care about their business. Why should they waste their time taking a web survey form to help you improve, if there is clearly no actions being taken to apply the feedback you’ve received? Why is a customer going to work with a firm that they feel is wasting their time? Now the customer is not only feeling as if you aren’t concerned about their needs and opinions, you’ve pushed them to look at your competitors. The goal is to take this opportunity to show the customer how much you truly care about their needs.

Having a system in place to streamline and automate your produce with real time alerts will allow you to serve you customers in a more timely fashion, keeping them happy and better yet loyal.

One thing that I think people often overlook when picking out online survey software is the team behind the product. Here at Cvent Web Surveys, we take pride in our customer care team. With today’s blog post, I wanted to share some great news – Cvent’s Client Services team is the winner of the 2009 International Service Excellence Award for Contact Center by the Customer Service Institute of America (CSIA)!

The CSIA is the body delegated by the International Council of Customer Service Organizations (ICCSO) to manage the International Service Excellence Awards. These awards are the world’s peak customer service awards with leading customer service organizations and individuals around the globe being nominated in a variety in categories to recognize their commitment customer service excellence.

We are proud and honored to have received this award. Having been a member of the Cvent Web Surveys Client Services team, I can attest to the level of service we provide.

Cvent is not just a survey software company – we are consultative, and pledge every effort to your survey success and satisfaction. We want our clients to get the data they need, in the way they need to see it. We work with our clients – hand in hand – to ensure no question goes unturned!

To read more about Cvent’s Support and Training, visit our website. Cvent’s Client Services team is available by phone and email 24 x 5 – and we offer weekend support hours as well! In fact, 90% of calls into our client services line are answered in less than 90 seconds, and we have the highest ratio of service personnel to clients in industry.

When choosing an online survey application, customer care cannot be overlooked. In this economy, utilize expert advice to increase responses and data quality and demand more from your survey partner!

Surveys are an invaluable tool for researching the community attitudes, employee concerns, product needs, customer loyalty and priorities held by different groups or target audiences.  Designing a questionnaire and collecting survey responses from a sample allows us to draw a profile of the group as a whole, and perhaps perform some correlation analysis to understand the source of those feelings.  The online survey findings can then support fact-based organizational decisions or improvement projects to help continually improve the organization over time.

Survey research can be applied to many venues.  Here are just a few practical applications listed below:

An Internal Employee Survey could identify reasons for low employee retention and provide ideas for reducing those costs, such as a better designed benefit program, improved training opportunities, or problems in the way the organization functions.

A Training Survey can identify how a training program has improved the capabilities of some group and how the training program itself can be improved.  

A Product Satisfaction Survey can identify initial customer experiences with a product, providing data to address unforeseen problems and help the next product release.  

A Market Research Survey can identify customers needs when creating these new service and product offerings.  Surveys can be part of Design for Six Sigma activities.  

An Association Survey, which is similar to market research and customer surveys, can show the member benefits most of interest.

However, a survey program is only valuable if it is properly designed and executed.  While performing a survey project seems deceptively simple – it's just a bunch of questions, and survey software tools make electronic surveys quick and cheap – a small mistake in the survey questionnaire design or survey administration can skew or bias the data, leading to erroneous conclusions.  No organization should ever make critical business decisions based on unreliable or invalid data.

Bad data is worse than no data!

There is sufficient evidence from prior studies suggesting it is a good idea to keep respondents informed of their progress during internet surveys.  Respondents may suffer from fatigue or irritation, and may abandon web survey forms – even if they have only a few questions left to complete the questionnaire.  This may lead researchers and project managers to adjust their online questionnaire design by reordering questions to include “important” or sensitive items earlier, possibly causing more survey respondents to abandon mid-stream.

There are various survey design techniques that can be employed to keep respondents informed of their progress, especially within electronic surveys.  One way is to design web surveys so the entire survey web form can be viewed on a single page; but while this allows respondents to scroll down and see the total length of a questionnaire, this setup is less than ideal. 

Another method is to include words or symbols in headers and transitions that notify progress (such as section numbers, the words "next" and "finally," etc.)  By far however, the most popular and effective method of keeping respondents in the loop is to include a progress bar. 

A progress bar is a graphic or set of words that let respondents know how far along they are in the survey process.  Progress bars are especially useful for shorter, instant surveys, since answering only a few questions will show that they are through a significant portion of the questionnaire.

Cvent Web Surveys software makes it easy to include a progress bar as part of an effective online survey design.  You can even select among three options (percent complete, page x of y, or a bar graphic that fills as respondents move through the survey).

I realized it only after the emails had been sent.  It was a test survey that wasn’t intended for entry level employees and senior management.  It was an employee evaluation that had initially been slated to go out only to our newest batch of hires.  Only last minute did we discover we could use logic to create one interactive survey online for senior management to evaluate the progress of new hire training, and new hire survey questions for them to evaluate the training process.  The wording, the slang, everything was wrong!  What did I do…

Hopefully, this scenario has appeared solely in a nightmare (and yes, survey nightmares are very real and very terrifying).  But a similar, less daunting situation is common among many: You have an online web survey that will go out to two groups of respondents, the wording in the two would ideally be completely different, and you do not have the time to go into the two templates you’ve created and select the different audiences then manually send the emails.  Not to mention, this method would require using Cvent’s email survey tool's default templates and the Custom Messages.  What if I told you it was possible to create entirely separate email marketing campaigns, set the emails up to go out automatically AND keep the both default templates and custom messages for each group of respondents? 

The multiple email marketing campaigns feature allows you to do just this!  You can select the number of separate email blasts you intend on sending, you can add entirely different contact groups to each campaign, and you can set up your emails to go out automatically. 

1. This feature is perfect if you have a multilingual survey.  You can translate your message into as many languages as you like to ensure maximum comprehension. 

2. Multiple email campaigns is also great for newsletters that are tied to a generic instant survey.  Instead of copying your survey and creating a new email blast every month, you can add an email campaign for the October '09 Newsletter, November '09 Newsletter, December '09 Newsletter and so forth.  You can then get an aggregate view of the data collected across all months.

3. The nightmare scenario outlined above.  It is likely that emails sent to your senior management are not worded the same as emails sent to entry-level new hires.  The formality of the verbiage aside, your new hires might be a bit disconcerted to know that the bosses of the bosses are watching them and evaluating their decisions. 

We all know creating surveys and designing questionnaires can be a bit overwhelming.  Why put yourself through that again just so different people can receive the same questionnaire?

I’ve talked extensively in the past about using Facebook for market research. But there are other Web 2.0 sites that you can use effectively as well.

For example, Twitter is growing more and more. You can search on Twitter for keywords or competing products to see how often they are mentioned, and what people are saying. This is a quick and simple method of gathering background info.

There are many sites where people mark things they like or dislike, like digg and del.icio.us. Everything on digg is submitted by users and then ranked by users. After a story has enough “diggs” it is listed on the front page. You can search for your client and see what types of stories are making their way to the site and get an idea of what kind of buzz is surrounding them. Del.icio.us allows users to tag and bookmark their favorite pages, and then allows all users to access rankings. You can find out how many people have bookmarked your client and competitors.

Of course, both these groups are by their nature narrower than most studies require. Results for each page are primarily involving only regular users of the site. You should broaden your search to find other respondents so your data is more accurate.

Web 2.0 sites are great places to find information and do background research. They also tend to be good places to find electronic survey respondents, as users are web savvy and opinionated. But as I mentioned, be sure not to make your respondent range too narrow.

In yesterday's post we looked at how your survey question items, if worded improperly, may be biased, resulting in unusable or incorrect results. Now we know how survey questions shouldn't look, let's delve deeper into why those previous examples were poorly worded, and how with just a few small adjustments you can easily start to hone question items within your survey forms.

When you design internet surveys, how do you write objective questions that won't bias your results one way or the other?

Here are a few main points:


These first few points are a great starting place when you're designing a questionnaire. The final post in this series will look at a few final survey best practice tips for writing great survey questions and the benefits you'll gain from crafting quality items. In the meantime, questions previously stored in our question library already come properly worded and ready for use!

Marketing departments love client testimonials and customer quotes. They're great to share with the sales team when prospects need referrals, they're helpful to put in powerpoint presentations, and can be a critical piece of any marketing website. Getting those sound bites from clients can be a challenge, however. As you can imagine, the challenge only gets bigger if your organization sells consumer products. Admittedly, the growth of social media has made finding people who are saying good things about your product, services or organization has gotten a little bit easier. But, it could be easier still.

Imagine a world where you had a plethora of client sound bites and quotes to share with the outside world. A marketers dream. No more tracking down sales reps to talk to their clients and mangle their words. Everything is already done for you. Well, if you're conducting client satisfaction surveys or consumer surveys, you should be gathering those sound bites already. You can even consider using website polls and website usability surveys to get those sound bites. Depending on the online questionnaire design, you may still need to contact the customer to get permission to use their quote, but the hard part is still done. You have the quote.

Question remains, how do you pull these customer quotes out of your survey web form? You make sure to add an additional comment box. Customers who love you will typically share those sentiments in the additional comments area, particularly if there are not other open ended questions in the survey for them to share why they like an organization.

Still wondering why your marketing department needs sound bites? Opinions for others is one of the driving forces behind purchasing decisions. If you want to learn more about the importance of customer reviews, check out this past post. It's also worth mentioning, this same idea can work for employee surveys and your HR department for recruiting purposes.

When it comes to marketing, understanding what information your audience wants to see is very important. In the Cvent survey management system, there is an option for Click Tracking. This part of the email marketing tool is great.

Here is how it works: When you create an eNewletter in Cvent, you can embed active links. These links can then be tracked through the Cvent Web Surveys reporting engine.  Unbelievable. Imagine being able to see what articles are important to your readers by running a simple report.  This gives you the insight needed to target specific groups more effectively.

Vocus, a leading provider of on-demand public relations management software, released the results of a recent public relations and marketing survey they conducted. The survey sample was made up of 1,800 marketing and PR professionals. Among Vocus's key findings was the response of 80% respondents that social media would be a key focus for them in 2010. Vocus shared that they believe social media has been a catalyst for the change in the PR industry, where PR is becoming increasingly more important.

How is the PR industry dealing with the fact they're being asked to do more with less? Investing in technology. Sounds like a familiar story, right? Marketers turn to email survey tools to send out email survey invitations instead of having to manually send all email marketing messages through Outlook. Accounting departments use software to help them manage finances instead of the old way of paper, pencils and ledgers. Technology is often used to help professionals manage their work loads, and PR industry is doing the same thing.

Here are some of the other key findings Vocus shared from it's marketing survey:


You can learn more about the survey results by viewing the recording of webinar Vocus hosted yesterday, PR Planning Considerations for 2010.

Just because you find correlation in your data when analyzing survey results does not mean there is causation. I find this is a common mistake in lots of survey reports when someone is new to survey research or conducting data analysis. Take this example:


Some people see this connection as a causation. Customers are satisfied because you contact them frequently to make sure everything is going well. The problem is, it's not a causation. Causation are extremely hard to prove because you cannot control every factor. For example, you may split your territory by industry and your solution suits some industries better than others. Or clients who are really satisfied simply use the product more often so the customer success managers reach out to them more frequently, because they are more likely to have questions. While those who use the product less have less to be satisfied about and may feel they are wasting those budget dollars.

To be able to prove causation, you need to be able to rule out all other possible explanations for the connection. As you can imagine, that's almost impossible to do since we do not control outside factors influencing the survey respondent or even the greater survey sample. Instead, when situations like these occur, we're seeing a correlation between two things. In my customer survey question examples, there's a correlation between how satisfied customers and how often they are contacted.

This principle does not only apply to customer survey research, it also applies to analyzing employee feedback forms, product surveys, market research and any other type of data collection and analysis.