User Reviews: Useful, but know what you're using

Type in the words user reviews on Google, and you'll get more than 160 million results. As many as 80 percent of shoppers nationwide now base at least part of their buying decision on these digitized experiences, including the ones on model pages, available to ConsumerReports.org subscribers. You'll also find user reviews on retailer sites like Amazon, Home Depot, Lowe's, Sears, and thousands of others.

User reviews can provide the feeling of reassurance you'd get from a friend or neighbor. Some cover models Consumer Reports hasn't tested. And many offer unique insights about specific models, including breakage and other durability complaints that may not show up in our tests or brand-repair surveys. Even months of rigorous testing may not uncover flaws that could take a year or more of everyday use to surface, which is why we monitor user reviews closely. What's more, an unreliable model may be too new or too limited in sales to affect our survey results for an otherwise-reliable brand.

That's why we were concerned when roughly half of the more than 100 user reviews for the Hoover WindTunnel Bagless S3765-040 and the Kenmore Progressive 27514 canister vacuums cited durability complaints. Both are CR Best Buys. Both also come from brands that have been reasonably reliable in our surveys. We noted those issues in our most recent report on upright and canister vacuums and promised to follow up. Our research confirms that user reviews, while often helpful, may not tell the whole story.

Our experts at the Consumer Reports National Research Center began by recontacting more than 1,200 Hoover and Kenmore canister owners who responded to our online Annual Questionnaire on brand reliability. The number who owned the Hoover WindTunnel and Kenmore Progressive was similar to the user-review total for those models. But our scientific sampling provided a more-representative look at their experiences—good, bad, and indifferent. It also yielded different results:

According to our respondents, both models broke less than the average in our surveys for those canister brands, even when we included broken belts—an inexpensive do-it-yourself repair we don't score in our repair data. Most who owned the Hoover and nearly all who owned the Kenmore said they would definitely or probably buy that model again.

Our research into user reviews for vacuums highlights several factors that can skew reviews for other products or services, wherever you find them:

• People are likelier to review a product they strongly like or dislike than something they find simply okay. Our analysis of more than 1,800 user reviews for uprights and nearly 500 for canisters found that most were either very positive or very negative, with few in between. Good surveys, including our brand-repair surveys, are designed to eliminate those biases. They also eliminate biases related to a product's age and other factors, which user reviews don't do.

• Even the most thorough reviewer may lack the context that comes from extensive side-by-side tests. For example, few could buy, test, and comprehensively compare the roughly 30 cars or trucks, 60 vacuums, and 80 refrigerators and TVs in a typical Consumer Reports Ratings. A model someone loves or hates could be far less lovable or loathsome if the same person compared it with several dozen other models.

• Some reviewers may be especially hard on a product or use it incorrectly. And some may inadvertently praise or pan the wrong model, since vacuums and other appliances can include 20 or more letters and numbers a reviewer may—or may not—get right.

• Users and manufacturers may also try to skew the results by intentionally posting biased reviews themselves or through others. Some reviewers could actually be shills.

All of those issues explain why we don't base our Ratings and model recommendations on user reviews. That doesn't mean they aren't a useful adjunct to our unbiased, scientific tests and brand-repair surveys. User reviews can also tell you about unique features that meet special needs. Just don't assume they're all unbiased or scientific.

The bottom line: Consumer Reports buys and tests more than 3,000 products and analyzes more than 1 million survey responses each year to create our Ratings and brand-repair histories. We're taking a close look at all of our model-specific user reviews in an ongoing effort to learn from them and make them as helpful as possible. Meanwhile, use our Ratings and repair data first for something you'd buy for yourself or a loved one. Then use our user reviews to help inform that decision.

We invite you to share your experiences in our user reviews and forums.