Blog

We don’t usually link to our competitors’ marketing collateral; they don’t usually link to ours.  But when we saw PowerReviews’ latest case study of their “Social Answers” product running on Skechers.com, we thought it would be interesting to check out just how “Social” it really is.  We figured: if this is the site they are featuring in their newest case study, it has got to be their newest product and a best-practice implementation.

So we did a simple test.  We asked a normal shopper question on each of 8 products on Skechers.com through the PowerReviews tool and we asked the identical question about the identical product on Shoes.com, which also sells Skechers shoes and uses TurnTo for Social Q&A.  Like usual, we kept track of when answers were posted, and if the answers were emailed before they were posted we counted the earlier email time.  Half the questions were asked on both sites at around 9:00am eastern, and half were asked at around 1:00pm.

The bottom line: TurnTo generated >3X more answers total, >90% were truly “Social” (from actual past customers), most arrived in 1/3 the time, with 1/4 the staff workload vs PowerReviews.

• Just one hour after the questions were submitted, the 8 questions asked through the TurnTo system had generated 7 social answers.  There were no answers to the questions asked through the PowerReviews system.
• At 8 hours, the TurnTo system had delivered 16 social answers and 2 staff answers.  Still none through PowerReviews.
• At 24 hours, TurnTo was up to 20 social and 2 staff answers.  PowerReviews delivered 8 staff answers – one to each question – but no social answers.  In fact, even after 2 weeks, PowerReviews never produced a social answer.
• At 24 hours, 7 of the 8 questions asked through TurnTo had received at least 2 answers total, including at least one social answer.  One of the TurnTo questions was unanswered.  The 8 questions asked through PowerReviews received one staff answer each.

Skechers says they’ve seen a 30% increase in sales when Q&A is installed on their product pages.  We believe it, and we congratulate them!  Social Q&A is a powerful way to increase conversion and to create the sort of user-generated content that search engines increasingly favor.

But we also wonder: imagine how much better their results would have been if their Social Q&A system delivered 3X more answers with most answers coming from actual past customers in under 8 hours (many in under 1 hour), the way TurnTo’s does?

Appendix: here are the specific questions asked on each shoe type.

We don’t usually link to our competitors’ blogs (they don’t usually link to ours…).  But PowerReviews put up a nice summary of the benefits of ecommerce Q&A that’s well worth a read.  We just want to add one important thing: imagine how much greater these benefits would be if your Q&A system actually generated a large number of fast social answers, instead of – how to put this – bupkis?

[For a downloadable version of this study, click here.]

To date, Q&A on ecommerce sites has been primarily a tag-along application to customer reviews (provided by vendors that specialize in customer reviews). This approach results in a Q&A model that’s more like customer reviews than a true social experience between shoppers and customers, missing the benefits that a truly social approach to ecommerce Q&A provides.

The key to Social Q&A is that shopper questions should reliably and quickly get answered by real customers, and participants should have the ability to go back-and-forth beyond the initial question, if they choose to. If shopper questions receive customer answers only rarely or after an extended period, the shopper is disappointed and the store has missed the chance to provide a fast reminder to the shopper about the purchase she was considering. Further, getting past customers to share their experience with real shoppers is a great way for stores to keep their relationships with the customer base fresh. The rise of social networks has conditioned people to expect a high level of interactivity from social applications – so if a Q&A tool isn’t providing that, it’s not really Social.

On many online stores’ Q&A systems, we’ve observed that most answers come from store staff.  That can be an OK supplement to social answers (especially if the staff are really experts), but the store may be better off directing those questions to a live chat or phone line so the staff can interact with the shopper in real time.  And if a shopper wants to know something subjective – like how the product held up after 3 months, or how it felt, or just if it’s really as fabulous as they hope it is(!) – they may only want an answer from someone like them who really bought the item.  A Q&A system that relies heavily on staff answers also isn’t really Social.

That’s why TurnTo created an approach to Q&A for ecommerce that reliably provides a true Social experience – multiple, fast answers from real purchasers with continuing back-and-forth dialog. To measure the difference between the TurnTo approach and that provided by the leading customer reviews vendors, Bazaarvoice and PowerReviews, we conducted a simple test. We asked 16 shopper questions on a range of sites with Q&A powered by TurnTo and these other vendors, and we tracked how long it took for the answers to arrive.  Here are the aggregated results:

Methodology:  In our test design, we tried to keep the playing field level. We asked general questions that could easily be answered by anyone with experience with the product. We tried to ask the identical question about identical products wherever possible. Where not possible we tried to pick featured items on the Bazaarvoice and PowerReviews sites likely to have high traffic and have been purchased many times (no new arrivals items were used). We tried to pick sites where the Bazaarvoice and PowerReviews Q&A tools were implemented in a highly visible way on the page. That meant that the PowerReviews and Bazaarvoice sites were not always the largest in each vertical (in particular, in the photo gear category), but more often than not, the Bazaarvoice and PowerReviews sites had far more traffic than the TurnTo sites, and they did so in aggregate. We checked the item page where each question was asked at exactly the specified intervals and counted posted answers. We also provided our email address with each question asked and counted answers received by email.  (The Bazaarvoice and PowerReviews stores often emailed answers well before those answers appeared on the sites, in some cases even before the questions appeared on the sites.)  None of the sites were alerted in any way about this test. All questions were submitted on Wednesday, August 10, 2011 between 9am and 11am eastern time.  Here were the test sites that we used:

On each site, we asked 4 questions.  So in total, we asked 16 questions per vendor.  Here are the details of the answers received, by individual site.  (All numbers are for social answers – answers from customers – except those in parentheses, which are answers from store staff.)

Staff answers:  We also tracked answers from store staff.  These are shown in parentheses in the table above.  At the end of the two week test period, the questions on PowerReviews sites received a total of 10 staff answers vs 7 social answers.  The questions on Bazaarvoice sites received a total of 5 staff answers vs 9 social answers.  No staff answers were received on the TurnTo sites – note that 15 out of 16 questions on TurnTo sites received at least 1 social answer within 24 hours.

We encourage you to try this test for yourself.

The raw data:  Here are the urls for all the item pages for all questions in the test.  The asker is “Andrew P”, “Andrew RP” or “Anonymous” – also look for a submit date of August 10th where that is shown.  Note that on the Bazaarvoice and PowerReviews sites, we counted answers received by email, even though some of those answers – in some cases, even the questions – were not posted on the site by the end of the test period.

SunnySports (TurnTo)
http://www.sunnysports.com/Prod/HSRAP.html
http://www.sunnysports.com/Prod/ERKTS.html
http://www.sunnysports.com/Prod/NFCVJWN.html
http://www.sunnysports.com/Prod/TMBCHTM.html

GoJane (TurnTo)
http://www.gojane.com/54695-stuff-pleated-twist-front-clutch.html
http://www.gojane.com/55317-stuff-floral-print-wayfarer-sunglasses.html
http://www.gojane.com/52138-shoes-patent-platform-pump.html
http://www.gojane.com/45392-shoes-mid-rise-combat-boot.html

KitchenwareDirect (TurnTo)
http://www.kitchenwaredirect.com.au/The-5-in-1-Multi-Cooker-5L
http://www.kitchenwaredirect.com.au/Global-Ikasu-7pc-Knife-Block-Set
http://www.kitchenwaredirect.com.au/Red-Pizza-Maker
http://www.kitchenwaredirect.com.au/Scanpan-Classic-Covered-Chef-Pan-32cm

Adorama (TurnTo)
http://www.adorama.com/INKD7000.html
http://www.adorama.com/CA702002ISU.html
http://www.adorama.com/GBSLBK.html
http://www.adorama.com/IDS8GBUC215.html

Sierra Trading Post (PowerReviews)
http://www.sierratradingpost.com/high-sierra-us-snowboarding-team-backpack~p~3901j/
http://www.sierratradingpost.com/marmot-titan-backpacking-tent-2-person-3-season~p~88182/
http://www.sierratradingpost.com/marmot-precip-jacket-waterproof-for-women~p~2283c/
http://www.sierratradingpost.com/asolo-fsn-95-gore-tex-hiking-boots-waterproof-for-men~p~72552/

Johnston & Murphy (PowerReviews)
http://www.johnstonmurphy.com/product.aspx?pid=80786
http://www.johnstonmurphy.com/product.aspx?pid=72913
http://www.johnstonmurphy.com/product.aspx?pid=80374
http://www.johnstonmurphy.com/product.aspx?pid=75845

Hayneedle (PowerReviews)
http://www.hayneedle.com/sale/fagorelectricallinonemulticooker.cfm
http://www.hayneedle.com/sale/calphalonkatanacutlery14pieceknifeset.cfm
http://www.hayneedle.com/sale/presto03430pizzazzpizzamakeroven.cfm
http://www.hayneedle.com/sale/rachaelrayhardanodized10piececookwareset.cfm

Abes of Maine (PowerReviews)
http://www.abesofmaine.com/item.do?item=NKD7000
http://www.abesofmaine.com/item.do?item=CN7020028ISII
http://www.abesofmaine.com/item.do?item=NKCFDC1
http://www.abesofmaine.com/item.do?item=SDCF8GBXA61

Bass Pro Shop (Bazaarvoice)
http://www.basspro.com/Badlands-Diablo-Day-Pack-Backpacks/product/102015/-1507491
http://www.basspro.com/Eureka!-Solitaire-Tent/product/58274/90919
http://www.basspro.com/The-North-Face-Venture-Jacket-for-Ladies/product/10224821/134865
http://www.basspro.com/RedHead%C2%AE-Everest-Hikers-for-Men/product/19193/115209

Bluefly (Bazaarvoice)
http://www.bluefly.com/Rebecca-Minkoff-fire-engine-leather-Beloved-Mini-crossbody-bag/cat20428/315738502/detail.fly
http://www.bluefly.com/Betsey-Johnson-black-contrast-stem-wayfarer-sunglasses/cat60024/314050001/detail.fly
http://www.bluefly.com/Pour-la-Victoire-aqua-patent-leather-Irina-platform-pumps/cat20458/314292102/detail.fly
http://www.bluefly.com/Ash-black-leather-Remix-studded-tall-boots/cat20448/312714301/detail.fly

Walmart (Bazaarvoice)
http://www.walmart.com/ip/Hamilton-Beach-6-Quart-Stay-or-Go-Slow-Cooker-33162/4012120
http://www.walmart.com/ip/Ginsu-14-Piece-Stainless-Cutlery-Set/7942211
http://www.walmart.com/ip/Presto-Pizza-Oven/3218
http://www.walmart.com/ip/Stansport-13-Cast-Iron-Fry-Pan/16349862

Cameras Direct (Bazaarvoice)
http://www.camerasdirect.com.au/webapp/wcs/stores/servlet/Product2_10001_10002_19502
http://www.camerasdirect.com.au/webapp/wcs/stores/servlet/Product2_10001_10002_10620
http://www.camerasdirect.com.au/webapp/wcs/stores/servlet/Product2_10001_10002_11838
http://www.camerasdirect.com.au/webapp/wcs/stores/servlet/Product2_10001_10002_11588

Bazaarvoice is a registered trademark of Bazaarvoice, Inc.PowerReviews is a registered trademark of PowerReviews, Inc.