This is a series of blogs that explore the use of NPS and address some of the criticisms that have been leveled against it over the years. NPS can be an effective tool in an enterprise storage vendor's arsenal, but how effectively a vendor designs and implements their "NPS program" will determine their success with it.
Over the last several years, there have been several startup enterprise storage vendors that have built a program around the Net Promoter Score (NPS) customer satisfaction metric that has generated a significantly differentiated customer experience. Early vendors to use this metric very successfully were Nimble Storage (now HPE), Nutanix, and Pure Storage. I looked at the NPS metric back in 2016 with a document entitled Why Enterprise Storage Managers Need to Understand the Net Promoter Score (IDC, April 2016), stating that while the number (score) was interesting, what was really interesting about the use of this metric were the processes, programs and culture that those companies committed to it created. Those features are what made the difference for these vendors – the high scores achieved by these vendors (in the 80s and 90s on a scale of -100 to +100) were just a reflection of that. As a yardstick, it's interesting to note that Apple iPhone customers typically generate an NPS in the 60s for that company (the most recent number as of January 2018 was 63).
All enterprise storage vendors use some method to evaluate the customer experience they deliver to their installed base. Some use proprietary, home-grown methods while many of them use NPS but don't publish the number because it is quite low compared to the numbers vendors like Nimble, Nutanix and Pure have made public over the years. In December 2017, Jared Spool, the founding principal of User Interface Engineering, a research, training and consulting firm specializing in web site and product usability, published an article entitled "Net Promoter Score Considered Harmful". In that article, he raises a number of concerns about NPS, coming to the conclusion that the metric has been debunked, and that its continued use will do more harm than good for businesses. Fred Reichheld, a Bain consultant that is considered the father of NPS, hasn't yet responded to this (as far as I know). I do think that Jared raises some valid concerns, but I still believe that use of a simplified, standardized metric like NPS to measure customer experience can have significant value when used correctly. I want to comment on some of Jared's points about NPS.
The way NPS is calculated is "wacky", which presumably make it invalid. Jared shows some NPS calculation corner cases which produce results which don't make any sense. In the business world, quantitative models are used all the time, and while they can be valid if used appropriately, there's a reason for the saying "There's lies, damned lies, and statistics." With some sleight of hand, numbers can pretty much be worked to show a pre-determined outcome, and in many cases are. I'm not going to go into how NPS is calculated (you can read the IDC document referenced above for that), but suffice it to say that it purports to measure an existing customer's willingness to recommend a purchased product to a friend. This willingness (according to Reichheld) directly correlates with a customer's happiness with the product, and the natural assumption is that a product's ability to make current customers happy means that it's very likely to make future customers happy. That, in turn, is expected to correlate with increased future sales.
Let's stop right there, because we need to examine our assumptions. The first assumption here is that a higher number means more happy customers, which presumably should translate to higher future sales. NPS is simple by design (more on that in a moment), and it can be used to help craft a truly differentiated customer experience, but it can also be used incorrectly. Vendors like Nimble, Nutanix and Pure have used it very effectively to not only create lots of happy customers, but have in fact driven change in the enterprise storage industry around what level of service is delivered, how it is delivered, and what customers should expect from their storage vendors. In their cases, high NPS scores have been correlated with revenue growth, but what has really driven their success is all the innovation that went into how they interact with their customers. Much of that (although not all) can be captured if one looks at evolving enterprise storage vendor programs around post-purchase support guarantees and services. There are ten areas in particular that have been affected:
Support case (ticket) handling workflows
Money-back satisfaction guarantees
Data reduction guarantees
Cloud-based predictive analytics
All-inclusive software bundling (of enterprise-class data services)
Hardware investment protection (trade-in credits)
Fixed maintenance costs
Flash endurance guarantees
Non-disruptive technology refresh
I will be publishing more research on these areas later in 2018, but established corporate programs (and in some cases platforms like with cloud-based predictive analytics) provide a predictably good experience with storage systems that did not necessarily exist before. Most enterprise storage vendors offer at least some of these today, whereas five years ago most of them offered none of them (with the exception of a few startups whose actions have now changed the industry). These programs represented a huge change from how enterprise storage vendors did business in the past, and customer satisfaction with them has been extremely high.
But back to NPS. While Jared's examples clearly indicate that NPS does not operate as expected all of the time, I think it is important to think of it in terms of a comparison. What is it replacing? A small local business owner may be able to parse recent purchases and customer interactions and intuit the right strategies to grow their business. As a business scales, however, this quickly becomes unworkable. Professional managers, working with the old adage that you can't manage what you can't measure, began to try and measure customer satisfaction. By the end of the twentieth century, the state of the art for larger businesses were customer questionnaires.
Before you can really understand how to use NPS, it's important to understand how customer satisfaction had traditionally been evaluated. A customer would get a lengthy questionnaire to fill out, and even if they had the time to fill it out there was no closure for them (nobody ever got back with them and they saw no change in the product or service as a result of their inputs). Few customers filled them out, and many that did became frustrated and just answered quickly and off the cuff to get it over with. The result was that few customers gave this type of feedback, and those that did often provided inaccurate data. Providing incentives to complete the questionnaire were often misguided. We've all probably had the car salesman who gave us the questionnaire and told us that if we gave him all high marks on it he would knock an additional $500 off the price of the car. All systems (including NPS) can be gamed in this manner, so part of a successful NPS program has to include ways to deal with (and avoid) gaming. These traditional approaches were complex, expensive, had low return rates, and suffered from inaccurate data in many cases.
To me, a key objective with NPS was to come up with a tool that could collect more comprehensive and accurate data. For this reason, making it simple was paramount. Having a single question to answer makes it simple for the customer. They are more likely to respond and to answer the question thoughtfully and be accurate. Does NPS more accurately predict customer satisfaction than the former method? I haven’t seen any studies that specifically address that, but my guess is that the answer is yes. Is it perfect? No, but if it's better we're moving in the right direction. Could it be improved? Without a doubt. Businesses can experiment with that. When the question is asked (whatever it is), and how often (after every service touch point, once a year, etc.) are also important considerations. While I think Jared brings up some valid points around how NPS is calculated that anyone planning to use it effectively has to address, the simplicity of the approach can't be abandoned (one or two questions vs the lengthy questionnaire). Response rates and accuracy are both important.
Another key advantage to NPS is that it is a standardized metric. The use of many proprietary approaches that are custom-developed by each company does not produce any kind of result that can be used to compare vendors. This means that the number has less predictive value in a competitive world. If in the future we are able to move to the use of more standardized approaches like NPS, and vendors can certify that they have used the generally accepted approach to calculate it, prospective customers can compare them on an "apples to apples" basis. This also opens up the opportunity for third parties to help vendors implement the standardized approach and even certify that vendors are in compliance. This has already happened for NPS, with SatMetrix Systems Inc. being probably the best-known industry example of such a company. SatMetrix was in fact a co-creator of the NPS metric, working with Reichheld over time to help fine tune it.
One final thought on this. Just because a customer bought a product or service from a business doesn't mean that business should want to keep that customer. Certain types of customers are profitable, and certain other types aren't. All businesses should be pre-qualifying to sell to those customers they think fit their definition of a profitable customer and have a way to identify unprofitable customers early on (and get rid of them). Businesses should not concern themselves when they lose customers that are unprofitable – they should focus on retaining (and finding more of) those that are. An NPS number is likely to be much more accurate if its inputs come from the types of customers a business wants to keep, rather than customer types that are not profitable. A business has to know who those customers are for their product or service.
NOTE: I'm splitting this blog into two parts so please look for the next installment on the IDC site: https://idc-community.com/infrastructureanddataamanagement.