What to do if your „social“ Q&A doesn’t actually get social answers?

September 27, 2011 by George Eberstadt

[For a downloadable version of this study, click here.]

To date, Q&A on ecommerce sites has been primarily a tag-along application to customer reviews (provided by vendors that specialize in customer reviews). This approach results in a Q&A model that’s more like customer reviews than a true social experience between shoppers and customers, missing the benefits that a truly social approach to ecommerce Q&A provides.

The key to Social Q&A is that shopper questions should reliably and quickly get answered by real customers, and participants should have the ability to go back-and-forth beyond the initial question, if they choose to. If shopper questions receive customer answers only rarely or after an extended period, the shopper is disappointed and the store has missed the chance to provide a fast reminder to the shopper about the purchase she was considering. Further, getting past customers to share their experience with real shoppers is a great way for stores to keep their relationships with the customer base fresh. The rise of social networks has conditioned people to expect a high level of interactivity from social applications – so if a Q&A tool isn’t providing that, it’s not really Social.

On many online stores‘ Q&A systems, we’ve observed that most answers come from store staff. That can be an OK supplement to social answers (especially if the staff are really experts), but the store may be better off directing those questions to a live chat or phone line so the staff can interact with the shopper in real time. And if a shopper wants to know something subjective – like how the product held up after 3 months, or how it felt, or just if it’s really as fabulous as they hope it is(!) – they may only want an answer from someone like them who really bought the item. A Q&A system that relies heavily on staff answers also isn’t really Social.

That’s why TurnTo created an approach to Q&A for ecommerce that reliably provides a true Social experience – multiple, fast answers from real purchasers with continuing back-and-forth dialog. To measure the difference between the TurnTo approach and that provided by the leading customer reviews vendors, Bazaarvoice and PowerReviews, we conducted a simple test. We asked 16 shopper questions on a range of sites with Q&A powered by TurnTo and these other vendors, and we tracked how long it took for the answers to arrive. Here are the aggregated results:

Methodology: In our test design, we tried to keep the playing field level. We asked general questions that could easily be answered by anyone with experience with the product. We tried to ask the identical question about identical products wherever possible. Where not possible we tried to pick featured items on the Bazaarvoice and PowerReviews sites likely to have high traffic and have been purchased many times (no new arrivals items were used). We tried to pick sites where the Bazaarvoice and PowerReviews Q&A tools were implemented in a highly visible way on the page. That meant that the PowerReviews and Bazaarvoice sites were not always the largest in each vertical (in particular, in the photo gear category), but more often than not, the Bazaarvoice and PowerReviews sites had far more traffic than the TurnTo sites, and they did so in aggregate. We checked the item page where each question was asked at exactly the specified intervals and counted posted answers. We also provided our email address with each question asked and counted answers received by email. (The Bazaarvoice and PowerReviews stores often emailed answers well before those answers appeared on the sites, in some cases even before the questions appeared on the sites.) None of the sites were alerted in any way about this test. All questions were submitted on Wednesday, August 10, 2011 between 9am and 11am eastern time. Here were the test sites that we used:

On each site, we asked 4 questions. So in total, we asked 16 questions per vendor. Here are the details of the answers received, by individual site. (All numbers are for social answers – answers from customers – except those in parentheses, which are answers from store staff.)

Staff answers: We also tracked answers from store staff. These are shown in parentheses in the table above. At the end of the two week test period, the questions on PowerReviews sites received a total of 10 staff answers vs 7 social answers. The questions on Bazaarvoice sites received a total of 5 staff answers vs 9 social answers. No staff answers were received on the TurnTo sites – note that 15 out of 16 questions on TurnTo sites received at least 1 social answer within 24 hours.

We encourage you to try this test for yourself.

The raw data: Here are the urls for all the item pages for all questions in the test. The asker is „Andrew P“, „Andrew RP“ or „Anonymous“ – also look for a submit date of August 10th where that is shown. Note that on the Bazaarvoice and PowerReviews sites, we counted answers received by email, even though some of those answers – in some cases, even the questions – were not posted on the site by the end of the test period.

SunnySports (TurnTo)
http://www.sunnysports.com/Prod/HSRAP.html
http://www.sunnysports.com/Prod/ERKTS.html
http://www.sunnysports.com/Prod/NFCVJWN.html
http://www.sunnysports.com/Prod/TMBCHTM.html

GoJane (TurnTo)
http://www.gojane.com/54695-stuff-pleated-twist-front-clutch.html
http://www.gojane.com/55317-stuff-floral-print-wayfarer-sunglasses.html
http://www.gojane.com/52138-shoes-patent-platform-pump.html
http://www.gojane.com/45392-shoes-mid-rise-combat-boot.html

KitchenwareDirect (TurnTo)
http://www.kitchenwaredirect.com.au/The-5-in-1-Multi-Cooker-5L
http://www.kitchenwaredirect.com.au/Global-Ikasu-7pc-Knife-Block-Set
http://www.kitchenwaredirect.com.au/Red-Pizza-Maker
http://www.kitchenwaredirect.com.au/Scanpan-Classic-Covered-Chef-Pan-32cm

Adorama (TurnTo)
http://www.adorama.com/INKD7000.html
http://www.adorama.com/CA702002ISU.html
http://www.adorama.com/GBSLBK.html
http://www.adorama.com/IDS8GBUC215.html

Sierra Trading Post (PowerReviews)
http://www.sierratradingpost.com/high-sierra-us-snowboarding-team-backpack~p~3901j/
http://www.sierratradingpost.com/marmot-titan-backpacking-tent-2-person-3-season~p~88182/
http://www.sierratradingpost.com/marmot-precip-jacket-waterproof-for-women~p~2283c/
http://www.sierratradingpost.com/asolo-fsn-95-gore-tex-hiking-boots-waterproof-for-men~p~72552/

Johnston & Murphy (PowerReviews)
http://www.johnstonmurphy.com/product.aspx?pid=80786
http://www.johnstonmurphy.com/product.aspx?pid=72913
http://www.johnstonmurphy.com/product.aspx?pid=80374
http://www.johnstonmurphy.com/product.aspx?pid=75845

Hayneedle (PowerReviews)
http://www.hayneedle.com/sale/fagorelectricallinonemulticooker.cfm
http://www.hayneedle.com/sale/calphalonkatanacutlery14pieceknifeset.cfm
http://www.hayneedle.com/sale/presto03430pizzazzpizzamakeroven.cfm
http://www.hayneedle.com/sale/rachaelrayhardanodized10piececookwareset.cfm

Abes of Maine (PowerReviews)
http://www.abesofmaine.com/item.do?item=NKD7000
http://www.abesofmaine.com/item.do?item=CN7020028ISII
http://www.abesofmaine.com/item.do?item=NKCFDC1
http://www.abesofmaine.com/item.do?item=SDCF8GBXA61

Bass Pro Shop (Bazaarvoice)
http://www.basspro.com/Badlands-Diablo-Day-Pack-Backpacks/product/102015/-1507491
http://www.basspro.com/Eureka!-Solitaire-Tent/product/58274/90919
http://www.basspro.com/The-North-Face-Venture-Jacket-for-Ladies/product/10224821/134865
http://www.basspro.com/RedHead%C2%AE-Everest-Hikers-for-Men/product/19193/115209

Bluefly (Bazaarvoice)
http://www.bluefly.com/Rebecca-Minkoff-fire-engine-leather-Beloved-Mini-crossbody-bag/cat20428/315738502/detail.fly
http://www.bluefly.com/Betsey-Johnson-black-contrast-stem-wayfarer-sunglasses/cat60024/314050001/detail.fly
http://www.bluefly.com/Pour-la-Victoire-aqua-patent-leather-Irina-platform-pumps/cat20458/314292102/detail.fly
http://www.bluefly.com/Ash-black-leather-Remix-studded-tall-boots/cat20448/312714301/detail.fly

Walmart (Bazaarvoice)
http://www.walmart.com/ip/Hamilton-Beach-6-Quart-Stay-or-Go-Slow-Cooker-33162/4012120
http://www.walmart.com/ip/Ginsu-14-Piece-Stainless-Cutlery-Set/7942211
http://www.walmart.com/ip/Presto-Pizza-Oven/3218
http://www.walmart.com/ip/Stansport-13-Cast-Iron-Fry-Pan/16349862

Cameras Direct (Bazaarvoice)
http://www.camerasdirect.com.au/webapp/wcs/stores/servlet/Product2_10001_10002_19502
http://www.camerasdirect.com.au/webapp/wcs/stores/servlet/Product2_10001_10002_10620
http://www.camerasdirect.com.au/webapp/wcs/stores/servlet/Product2_10001_10002_11838
http://www.camerasdirect.com.au/webapp/wcs/stores/servlet/Product2_10001_10002_11588