Should we step to the right or step to the left? There are times when questionnaire design seems a bit like a dance step. Every researcher during his or her career has asked themselves if they should start a series of scaled questions with the high number (or most positive response) closest to the question or should it be on the far end away from the question? The graphic below is a good example. This came from a routine post-event bank customer satisfaction evaluation.
Like many other post-event surveys I have seen the author has chosen to start the scale with the most satisfied response, in this case 10 – Extremely Satisfied, closest to the question and then counting down to the most negative response. My thinking here is not about the choice of a ten-point scale, but in the direction of the coding and the potential impacts it can have on the survey data analysis.
The author made a wise choice to label the endpoints so that respondents had no doubt that moving from left to right was akin to assigning a lower or less satisfied score. However, it has been posited that by starting off the scale with the most positive response the door is opened for order-effect bias. This form of bias would lead to satisfaction scores that are inflated. This may serve the purposes of some, but is not good for the business as a whole.
Best practice is to start the scale with the most negative response, or the lowest numerical score in order to minimize score inflation. This is an effective approach, especially in countries that read from left to right where we are taught that lower scores should be on the left and higher scores on the right. You could make a counter argument that starting off with high scores, since it is atypical, would lead to greater focus and therefore more accurate scoring.
How do we solve this dilemma? Simple – test two versions of the scale. Preferably this should be done at the same time and via random assignment. In a sense this becomes an A/B test. If the scores for the test group which received the high score on the left are significantly higher than the group receiving high scores on the right then inflation is present and should be accounted for.
