Complex Question Types: The Constant Sum

Sometimes in survey research we need to ask questions that may be a little tougher for the respondent to answer. Typically these can be questions that require recalling something that happened in the past. Other times we need our clients to compute a little bit of math to arrive at an answer they can report. This is where it can get a little tricky for both the survey researcher and survey respondent!

For example we may want to know how much of our weekly work schedule is spent with certain types of clients, and we'd like to know it as a percent.

The question may look something like this:

Job Performance Appraisal Samples: In the last week, what proportion of your time was spent providing services to?
Job Performance Appraisal Sample Question

Now, for the respondent this question actually involves 2 seperate cognitive tasks, which results in lesser quality data for you (and more frustration for them).

Creating this survey question in web format may reduce some of the frustration, as survey platforms can automatically calculate the total percent for respondents and notify them when the total they reported amounts to greater than 100%. While this is a great practical advancement that makes it slightly easier to get the answers we need and provides cleaner data, regardless, adding a math problem (calculating percentages) to a question(s) will reduce the construct validity of your question.

Whether we use paper, phone or a web program that helps the survey respondent total to 100%, surveyors should pause before putting too much faith in responses received from complex question types.
blog comments powered by Disqus
Crimes in Design Webinar
Subscribe to our Monthly Newsletter