A Rating System Based on Buyer Expectations
Any time I make a purchase I have expectations about what I’ll get. These expecations are built by marketing, by my past experience, by what I read on the product page, and by reviews. It doesn’t matter if the product is a bar of soap or a taxi ride. When I finally use the product there are only three possibilities: It can meet my expectations, exceed them, or all too often fail them.
Clayton Christensen taught a framework for looking at products as things we hire to do a particular job. For example, you might hire a raincoat to stay dry, a cup of coffee to focus, or a drive-thru milkshake to feed yourself on the way to work. By this logic, product reviews are employee performance evaluations. If the raincoat has no sleeves then it’s not very good at it’s job, unless part of it’s job is to keep your arms free.
We rarely get to interview the products we buy before we hire them. Customer reviews are the key piece of information we use to inform our purchasing decisions. We don’t trust marketers the way we trust other people who bought the same thing before us. Enterprise companies will ask for references before buying your software; I read Amazon reviews before buying hand soap.
For consumer products, star-based reviews are ubiquitous: Amazon listings, Ebay sellers, Uber drivers, Airbnb rentals, Google Play apps, etc. etc. It’s a way to summarize multiple written reviews into a sigle, comparable metric. In principle this lets you quickly decide between a long list of options. In practice it’s a much weaker signal than intended.
The reason is that no one knows what stars mean. What is a four-star product? What is two-star product? How different are they? Nobody knows! Not even companies who host these reviews.
In the customer experience industry, post-call or post-chat surveys use Likert scales to collect and accumulate feedback. When you’re asked to review your experience with an agent, the prompt will say “on a scale of one to 5, where one means X and five means Y…” because without well-defined extremes, it’s very hard to interpolate.
Even then CX engineers know users are unlikely to take post-call surveys unless they’re really pissed or overjoyed. I read lots of Amazon reviews that begin with versions of “I don’t normally leave reviews, but this time I had to…” A proper replacement for star-based reviews would increase the likelihood that you bother to leave one.
On Amazon/Uber/whatever, 1 star and 5 stars are not defined, nor is how many should count as “good enough.” There are clear drawbacks for both buyers and sellers. In competitive markets, 5 stars has become the baseline for good-enough service: any less may cause problems for your future on the platform. Knowing this changes reviewer behavior: I’m reluctant to review something that’s just okay because I don’t know what impact my review has. I’ll rate something when it’s awesome, when I have a real problem with it, or if I don’t think the answer will get someone fired. This skews ratings towards the extremes.
It should be embarrasing that we haven’t figured out a better rating system, even for services where products are heterogenous and not conducive to Likert scales. It’s not hard to think of alternatives. The demo below builds on the idea of expectation classes to reduce the friction in reviewing products by removing ambiguity.
Instead of stars, this rating system uses a range slider. The slider defaults to the center point, which we’ll say means the product met your expectations. If it did, you can click Submit without thinking. If it exceeded or didn’t live up to your expectations, you can drag the slider to the left (it failed) or the right (it exceeded) to the degree that it did and click Submit. That’s it.
Keep in mind this is just a demo – it hasn’t been tested for accessability, and replcing star-based ratings will mean retraining everyone away from whatever heuristics they use today. But no matter what system we decide on, the present state needs to change.
Now imagine you’re asked to review the last thing you bought:
The product has not been reviewed yet. Submit your feedback to say how your experience compared to your expectations of the product.