Net Promoter Score (NPS) is a common consumer insight method used to track customer loyalty. Advocates of NPS maintain that if someone is highly likely to recommend a product or service to a friend, they are also highly likely to be a loyal customer and will generate revenue for a company through referrals.
It’s a single question that goes something like this:
“On a scale of 0 to 10, how likely are you to recommend this product or service to a friend or colleague?”
How Net Promoter Score Works
Net Promoter Score tracks one thing and one thing only: how likely a customer will be to recommend a product or service to a friend. That’s it. According to Jared Spool, a leading voice in user experience, the science behind it is a little odd as well.
Rather than calculating the mean from the total of responses, as most surveys of this nature would, NPS answers are broken down into three groups:
- Promoters: Those who score 9-10. They are considered loyal customers who will increase growth.
- Passives: This group falls in the 7-8 range, and they are satisfied, but may not remain completely loyal.
- Detractors: This group scores between 0 and 6. They are considered unhappy customers who can damage a brand.
The score is then calculated by subtracting the percentage of detractors from promoters and ranges from -100 to 100. A “good” score is anything above 0.
This means different answers can actually produce the same result.
Jared Spool uses the simple example of 10 users all giving scores of 0, which is an NPS of -40. That’s the same thing as 10 users who give the answer of 6. Still a -40. Confusing? Yep. Accurately capturing what users actually think? Probably not.
And yet, companies and business leaders endorse Net Promoter Score because it’s a simple number that can be easily tracked and measured. Often, important business decisions are based solely on NPS scores—sometimes to their detriment.
The Problems with Net Promoter Score
The truth is, answers from a single survey won’t tell companies much more than whether the customer feels that they would recommend the product or service to a friend at that exact moment in time. It doesn’t accurately predict future behavior, nor does it account for changes to customer attitudes in the future.
Issues with Net Promoter Score are twofold:
- NPS is riddled with cognitive biases.
- NPS alone fails to get a well-rounded view of the user.
Let’s break this down.
What People Say Doesn’t Always Match What People Do
According to Jakob Nielsen, self-reported data is often three steps removed from the truth. People will do these three things:
- Tell you what you want to hear or what they think is socially acceptable (bandwagon bias)
- Tell you what they remember what they did: Memory is unreliable. (leveling and sharpening biases)
- Rationalize what they did: “If the link were blue, I would have seen it.”
When reporting answers to an NPS solicitation, users are simply approximating that they’ll recommend a product or service. They haven’t actually recommended it, nor do we know if they will, which brings us to the next point.
People Are Terrible at Predicting Their Own Future Behavior
Net Promoter Score largely relies on people being very good a predicting what they’ll do in the future. People are terrible at this. According to leading behavioral scientist Susan Weinschenk, preference and reality are two separate things. People may confidently say what they want or what they’ll do, but when the situation actually occurs, their reaction is much different.
NPS Only Measures One Thing at a Single Point in Time
When a company sends a customer a Net Promoter Score survey, it only captures a single point in time. So, if a customer receives the NPS survey right after they’ve made an online purchase and has no issues with the purchase process, they may give a high NPS score.
But what if the same customer receives poorly made merchandise and is now displeased with the purchase? There’s no NPS survey now to capture that experience and the customer can’t go back and change their answers. Experiences and preferences change over time. NPS only captures one moment.
NPS Tells the What But Not the Why
Net Promoter Score, like web analytics and surveys, is a quantitative research method. Quantitative research methods are good, and necessary, but fall short at building empathy and truly understanding user behavior.
Net Promoter Score will tell you what someone thinks, but it won’t give you a full picture of why they feel that way, or what happened to get them to that point.
Qualitative research methods help tell the full story in combination with quantitative data.
So, if Net Promoter Score doesn’t work, what does?
User Research Rounds Out Net Promoter Score
To get a clear picture of what users think and why they do what they do, user experience designers and product teams should conduct user research often and regularly.
Because no research method is perfect, a multi-pronged research plan should be used.
To determine the right research mix, focus on the questions and goals, and then determine the right research methodology.
The most effective research mix combines qualitative and quantitative methods. One method isn’t empirically better than the other, but one is conditionally better than the other depending on the data and information you need. Using them in tandem will give the most reliable data.
Nielsen Norman Group outlines the various research methods, as well as when and why to use them; in this article, we will examine a few of the key methods used to develop a well-rounded view of users.
While data analysts often have the core quantitative research responsibilities at a company, many user experience researchers tend to rely heavily on qualitative studies, with only a moderate use of quantitative analysis.
Qualitative research involves observation of user behavior and focuses on what people do versus what people say. This is valuable because past behavior is generally a reliable predictor of future behavior. Qualitative research can involve several different methods including interviews, field studies, and moderated usability tests.
Key user research methods to help better understand the user include:
User Interviews
User interviews are a relatively quick and easy type of research to conduct. They can be done remotely or in-person, depending on the budget and time constraints of a project. The key to user interviews is to ask the right questions in the right way.
Things to avoid when conducting a user interview:
- Closed-ended questions where users only answer “yes” or “no.” Ask open-ended questions instead. Think of the interview as a conversation, rather than an interrogation.
- Talking too much. It’s the interviewer’s job to listen and get as much information as possible from the participant.
- Filling the silence. Sometimes an interview participant will be silent. Researchers should avoid trying to fill that space, even if it feels awkward, and allow the participant to gather their thoughts.
Contextual Inquiry
Contextual inquiry takes place in the user’s environment and is a combination of questions and observation with little interference on the part of the researcher. By observing the user in their natural environment, a researcher can capture the most accurate data.
For example, if a research team wants to understand a business professional’s experience in an office, they should visit them at work as opposed to their home. Conversely, if a user researcher wants to understand the nuances of how an elderly person may use an app to monitor their medication intake, they should visit that person in their home.
Experience Sampling
While Net Promoter Score only captures a single moment in time, experience sampling gathers data over time at certain points in the experience. For example, in the case of an online shopper, a user researcher may send them a survey at purchase, once they receive the item and after a few months of using the item.
By gathering continual feedback over the course of an entire experience, UX designers can get a better understanding of the end-to-end experience versus what the customer thinks after a single interaction.
Usability Testing
Usability testing helps designers pinpoint key problem areas of an interface or product. It can be conducted in person or remotely, and be moderated or unmoderated. Budget and resources often determine where and how a usability study takes place, but all usability studies involve the user walking through a prototype or product.
Depending on the product being tested, sometimes a usability study is task-based, meaning it’s divided into several small tasks; other times, it involves the user going through a single flow.
For example, enrollment in healthcare benefits is often a fluid process, even if it seems to have many different steps or screens—breaking it into separate tasks may not work as well as simply allowing the user to go through the steps on their own in a single flow.
Usability tests gather a combination of qualitative observational data and quantitative data. Quantitative measures for usability tests are things like completion rate, error rate, and task difficulty rating.
A/B Testing
A/B testing is one of the quantitative user research methods researchers use. It involves running experiments over a certain period of time on a live website or product using a control (A) and a variant (B).
A/B tests are helpful for testing things like headlines, button colors, form fields, and other single points of interaction in a product.
Before starting an A/B test, a user researcher should form a hypothesis of what they are trying to prove or disprove.
For example, a company may want to test changing their “Add to Cart” button from green to pink. The hypothesis here could be: “Sales will increase if we change our “Add to Cart” button from green to pink.” The A/B test is then run to prove or disprove the hypothesis.
As with Net Promoter Score, A/B tests shouldn’t be relied on to make decisions in a vacuum. Once a team has the results of an experiment, they should look at it in conjunction with their other data in order to make the most informed decision.
Wrapping It Up
Although Net Promoter Score may appear to meet the requirements of a useful business metric, is not enough, and companies that use it exclusively do so at their peril. User experience designers and researchersshould consider a variety of research techniques to support its use.
By implementing ongoing user research to gather behavioral insights, user experience teams will gain a more accurate picture of the customer’s experience, and in so doing, create compelling products that solve real problems.
This article is written by Christine Esoldo and is originally posted at Toptal