NPS – The one number you’re (probably) doing wrong

The Net Promoter Score – aka “NPS”. We’ve all heard about it and most of us have probably used it in a project before to measure positive word-of-mouth about a brand or website. NPS seems pretty straightforward to measure right? You just ask respondents how likely they are to recommend something on a scale. Easy! Right?

NPS is a popular marketing metric, but what is a good score? We were interested in finding benchmarking data to feed back to our users but we noticed that not everyone uses the same scale when asking the NPS question.

What we uncovered was both unexpected and fascinating.

A quick NPS history lesson

The Net Promoter Score began its journey into our research work from an article in the Harvard Business Review by Fred Reichheld in 2003. In that article Reichheld details the development of ‘the’ question and scale (0 to 10 with 0 = “not at all likely”, 5 = “neutral” and 10 = “extremely likely”) to measure the likelihood of a consumer to recommend the brand. Reichheld outlines the reasons he and his team chose the 0 to 10 scale. The scale should be:

• appropriate to group users into groups that deserve different focus
• intuitive to users
• statistically valid

In the same article, Reichheld points out that by limiting ‘promoters’ to just those who give a score of 9 or 10, companies avoid the danger of inflating the number of users that are ‘really’ loyal. To explain this, assume that you have just been to a website and you are 51% satisfied. Technically, you would be classified as ‘satisfied’ – since your satisfaction rating is over halfway of the scale (out of 100%). But are you ‘really’ satisfied? Probably not. You’re most likely really satisfied if you were 70% satisfied; definitely satisfied if you were 90-100% satisfied. Hence the recommendation of using a 0-10 scale when asking the NPS question.

NPS in practice – Creating false positives

At Loop¹¹, we examined the use of the NPS question across multiple projects that asked the question to respondents. Keeping only projects that had a sample above 30, contain no missing data, and at least a 5-point scale, we found a total of 149 recent projects asking the NPS question using one of four scale types:

• The correct 0-10 scale
• 1-10 scale
• 1-7 scale
• 1-5 scale

We found that only 64% of projects that use the NPS, use it correctly. But does it really matter? What difference does it make to the Net Promoter Score if someone uses a 1-5 scale instead of a 0-10 scale?

To check this, we pooled together the response options for the NPS question from the 149 projects run in Loop¹¹ and we used a handy little formula to convert all incorrect scales into the correct 0-10 scale and then calculated the NPS result. This enabled us to compare the NPS of different scale sizes and see if there was any bias in using a shorter scale.

Scale	Projects	Respondents	NPS	Promoters	Passives	Detractors
0-10	95	17,539	-20.8	25%	30%	46%
1-10	47	4,113	-9.8	30%	31%	40%
1-7	1	94	37.2	51%	35%	14%
1-5	6	1,658	27.2	49%	30%	21%

These results show that as the scale becomes smaller (the range column), the NPS goes up, even when the scale goes from 0-10 to 1-10, the NPS result is likely to be positively biased. The cause of the bias comes from the higher percentage of users being classified as promoters and less as detractors.

Try Loop¹¹ for free!
You’ll have an NPS test running on your website in minutes.

Why is it that a smaller scale leads to a higher NPS result?

The probable cause is more likely to be psychological rather than statistical.

The NPS question asks people how likely they are to recommend a brand or website. Likelihood indicates a probability, and we tend to think of probability as a percentage (i.e. there’s a 50% chance a coin will land on heads), and a percentage is a value between 0% (not at all gonna happen) and 100% (absolutely will happen).

Tourangeau et al (2000) say that a person’s response to such a question conjures up beliefs, feelings, impressions, general values and prior judgements that the person combines together into an overall judgement. We don’t believe that respondents actually compute the probability, but they arrive at this judgement and then compare it to the scale range (using the endpoints as anchors) to find the category most similar to their answer.

If you use a scale smaller than 0-10, the respondent is forced to recalibrate their probability to fit the shorter scale (i.e. 70% likely [a 7 on a 0-10] is equivalent to a 3.5 on a 1-5 scale), but unless you’re Sheldon from The Big Bang Theory, most people cannot make this calculation accurately and end up giving a higher score.

Why would someone use a 5 or 7-item scale?

Many websites will say that a scale should have only 5 or 7 points and a 10 or 11-point scale is too long. Their rationale stems from the belief that a person can only hold seven or so items in working memory, which came from a psychology paper by Miller in 1956. The problem here is that Miller’s argument refers to remembering things, not about comparing your overall judgement to a scale you can see. Asking people to recall what they know about 10 different brands is challenging, but asking them to compare how likely they would recommend a brand or website with an 11-point scale is not challenging; the numbers are there for them to see. In fact, we could almost recommend the 0-10 scale for every scale question you have!

So let’s clear the air a bit; the correct way to ask the NPS question is to ask:

How likely is it that you would recommend XYZ to a friend or colleague?

And use a 0-10 scale as per below:

Not at all likely					Neutral			Extremely likely
0	1	2	3	4	5	6	7	8	9	10

We don’t believe that people who are using incorrect scales are knowingly subverting the NPS result, but their results would create a situation known as “everything-looks-rosy-but-really-it’s-not”; leading to managerial decisions that overlook potential issues.

Another great way to test your website is to find out why visitors are arriving and what they are hoping to find. Google Analytics won’t tell you this. The best way to achieve this is via a True Intent study, which can include an NPS question! Right now we are offering readers of this article the chance to get a True Intent study set-up on their website for free.

Fill out the below form and we’ll send you an email on how to get started.

[wufoo username=”loop11″ formhash=”x1kia99x1lcu8ee” autoresize=”true” height=”380″ header=”show” ssl=”true”]

REFERENCES

• Miller, GA (1956). The Magical Number Seven, Plus or Minus Two. Psychological Review. 63 (2): 81-97
• Reichheld, F (2003). The One Number You Need To Grow. Harvard Business Review. 81 (12): 46-55
• Tourangeau, R, Rips, LJ, and Rasinski, K. (2000). The Psychology of Survey Response. Cambridge University Press