How trustworthy are customer reviews?

David Streitfeld of the New York Times has been looking hard at the issue of fake customer reviews.  A year ago, he called out freelancers offering to write positive reviews for a few bucks.  In January, he wrote up a service called VIP Deals that offered rebates in return for positive reviews.  And a couple days ago he published a piece on Todd Rutherford, the founder of gettingbookreviews.com, who sold 4,531 book reviews in 2010 and 2011 at $20-99 each, before backlash from Google and Amazon forced him to shut down the service.

No doubt, there is fakery out there.  The question is: how widespread is it?  4,531 seems like a lot of reviews, but it’s a small part of the billions of customer reviews available on the web.  The most recent NY Times piece says Bing Liu, a computer scientist at the University of Illinois, Chicago specializing in automated text analysis, “estimates that about one-third of all consumer reviews on the Internet are fake.”

Really?

One of the reasons there is so much suspicion of reviews is that so many of them are positive.  Liu has estimated that 60% of the reviews on Amazon are 5 stars and another 20% are 4 stars, but, he says, “almost no one wants to write five-star reviews, so many of them have to be created.”

Here’s an alternative explanation: there’s a lot more customer satisfaction out there than you might guess.  That’s a conclusion you might come to from reading customer answers to shopper questions gathered through TurnTo.  With the TurnTo system, there is almost no possibility for fakery.  When a shopper asks a question, the system chooses a group of people who actually bought the item (based on the transaction records of the store) and emails the question to them.  While the system allows in-line answers on the product detail page, >90% of the answers come in reply to this question email.  So unless there’s a big population of people buying products they don’t need for the purpose of providing artificially positive answers to shopper questions they may never even receive, these answers are legit.

And one of the most striking aspects of the answers provided by these real product owners is how effusively positive they often are.  For example, here are some customer answers to a shopper question about the height of the drip spout on an espresso machine at SeattleCoffeeGear.com.  The question doesn’t ask for any sort of overall evaluation of the machine – it’s just looking for a measurement.  Yet many of the respondents (including me) spontaneously volunteered our enthusiasm for the product.  (You can find this page here.)

Now I don’t want to be Pollyanna about the problem of fake reviews.  I suspect they are much more common on destination review sites like Yelp and Trip Advisor where anyone can submit than on ecommerce sites where the ability to verify purchase is an easy and effective way to police.  It’s also harder to believe the uniformly high ratings sometimes found on products which are judged on subjective personal taste, like books and food; personal tastes differ too much.  As Streifeld points out, even The Great Gatsby (which, first published in 1925, is presumably not attracting many fake reviews) has plenty of neutral and negative reviews (>300 reviews are 1, 2, or 3 stars out of 1,400 total at Amazon).

But while the battle goes on between the fakers and those trying to root them out, it’s possible that in many cases when the reviews are positive, customers might just be happy.