All Survey designers, at some point in their career, will face the challenge of having to maximize information content for their clients while balancing the cognitive load we are asking of our respondents. This post examines two options for eliciting a respondent’s preference – Rank-order versus point allocation exercises.

Rank order questions are in common usage across marketing and other forms of consumer-based research. The primary advantages offered by rank-order are multi-fold – they are easy to program and are available in most if not all survey platforms. This question format creates only a moderate amount of cognitive stress, unless the researcher goes beyond 5 – 7 items. Rank order questions are objective in nature and remove potential subjective bias.

This question format is not without some issues. By design, rank order questions produce ordinal data. This data type can present analytical challenges. For example, items can have the same median ranking. It is also possible that no item receives a top ranking, based on the median rank. You can assign the “most preferred flag” to the item with the greatest response percent at either the top or bottom position, depending on how your question is phrased to work around this.

The point allocation scale is a viable alternative to the rank order question type. In this scenario, respondents are asked to allocate 100 points across the slate of items for consideration. This question type is also known as a Constant Sum. The data produced by a constant sum question is interval-level, which means a full-slate of statistics can be applied against it. What makes it interval is the fact that an item can receive no points or all of the points. It is suitable for data procedures that require numeric data, e.g. regression or analysis of variance. Like the ranking problem, this question type is also objective in nature. It carries little risk of subjectivity bias. The image below shows what a respondent might see for a point allocation or a constant sum scale.

To convert this to a rank-order scale then substitute the following language:

Both methods fall into the domain of commonly accepted practice for survey research. Best practice for both variants is to limit the list to 5 – 7 items and to employ randomization (this minimizes first-order bias). Ultimately, the question format chosen should be a function of your client’s needs and your interest in analyzing the data.

The author: Greg Timpany directs the research efforts for Global Knowledge in Cary, North Carolina, and is partner in the firm Anova Market Research. You can follow him on Twitter @DataDudeGreg.

Greg Timpany is also a Research Rockstar Instructor.