Don't be used by user reviews--make them work for you

The Internet is a digital democratizing force, giving voice to all constituents and connecting a vast wealth of information to every computer. Users can now access direct feedback from owners of nearly every modern product, from gadgets to cars, providing personal perspectives in greater quantity than traditional media. However, not all voices should be considered equal.

At ConsumerReports.org, we have a thriving online community of knowledgeable, insightful members eager to help one another in the pursuit of quality, satisfying products. This feedback is a valued complement to hundreds of tests conducted each year by our teams of engineers and scientists.

I was reminded last week by a press release that even the best user reviews without the context of professional analysis are compromised when newcomer cargurus.com announced its "Midyear most popular cars." The release reads: "Based on the quantity and quality of ratings received by each car on CarGurus, 2007 models were ranked by consumer popularity."

The Jeep Wrangler claimed the honors in the release of being the "most-popular" vehicle, although the site's home page lists it as the third most-popular SUV.

In blogging on this announcement, CNET referred to the Wrangler being "The No. 1 rated car," though the PR clearly indicates it is a popularity contest--not a ranking based on professional judgment, or even the total average user rating. The popularity of this redesigned off-roader caught my attention, but it is the influence of the user reviews as ratings that sent me on this tear.

Sales are strong, even downright robust for the four-door Wrangler variant. Clearly the Wrangler is four-wheeling off dealer lots, but could this Jeep possibly be that well liked among its owners?

Scanning the user reviews at ConsumerReports.org, I see much the same results as CarGurus. Owners, quite frankly, love the Wrangler. Sure, there some common criticisms regarding fuel economy and soft-top removal, but most owners cite "fun" and give it four or five stars.

And yet, Consumer Reports gives it an overall score of 17... out of 100. And yes, 17 is the lowest score in the class. (In fairness, the new Wrangler scores more than 300 percent better than the previous generation, which earned just a 5 overall.)

So, who is right? Both the owners and pro car testers.

Our engineers evaluate cars using scientific, repeatable methods that enable detailed comparisons across product lines. Simply stated, we drive every significant new model, and put it through an exhaustive battery of tests. We do so without fear or favor. The product is what it is, and the results speak for themselves.

Bear in mind that owners are a pre-selected bunch. These are people who love a vehicle enough to plunk down years of hard-earned cash to put it in their driveway and welcome it into the family. As such, they do not represent the more than 16 million new car buyers who chose not to buy the Wrangler this year. And to that end, a Wrangler owner's definition of "fun" is going to be a lot different than how a Chevrolet Corvette or Toyota Prius owner defines it. Ultimately, car owners have a vested interest in liking their chosen vehicle--to admit otherwise, online or off, is to confess to a poor fiscal decision that may be hitting their checkbook for a long time to come.

Both the professionals and the car owners each bring valid, valuable insight to a vehicle--in this case, the Wrangler. The owners' satisfaction speaks well of their first impression being supported by miles of happy driving and their enthused reports can foster excitement. Just remember, user reviews are but one view of the vehicle, and most owners do not have the opportunity, or experience, to evaluate a model in the context of the entire marketplace. 

The Internet is rich with research tools, such as user reviews, forums, and professional ratings. Each has value. To make an informed decision, it would be best to use them all... though be sure to weight the expert opinions accordingly.

--Jeff Bartlett