By the numbers at Consumers Union

	STATISTICS  CR’s experts make sure nothing is left to chance in our testing.

Eaters of chicken and readers of major news media may be forgiven for feeling confused. Consumer Reports told you that 83 percent of the 525 chickens we tested for our January 2007 report on chicken safety carried bacteria that could make you very sick. The U.S. Department of Agriculture, whose job it is to keep our cacciatore clean, labeled our study “junk science,” without even learning our methodology. “There’s virtually nothing or any conclusion that anyone could draw from 500 samples,” said a USDA spokesman.

So wrong. The USDA failed to consider how deeply our team of statisticians and scientists is involved in designing and analyzing our studies. We didn’t pluck 525 chickens out of thin air. We factored in statistical confidence levels and our previous chicken analyses and told shoppers in 23 states exactly how many chickens of which brands to buy in which stores on what day of the week.

That’s par for the course at Consumer Reports. Here are a few of the other ways our statisticians ensure that our work is objective and scientifically sound:

Paint by numbers. Outside our headquarters and on the roof are racks with row upon row of wooden boards, each colored with up to nine paints or stains. There they stay for years to test weathering. But the character of the wood can vary slightly from board to board and throw off the results. So before a brush is dipped, our statisticians design a random pattern of the paints, using each sample more than once. Then they help sort the differences in paint and stain performance from the quirks of the wooden boards themselves.

The big picture. If you’ve ever watched a “Leave It to Beaver” marathon, you know that June Cleaver looks a little less polished by hour 11. It’s not her fault; it’s your eyes. The scores for each of the 180 or so TVs we rate every year are based mostly on picture quality, so it’s important that each TV have an equal chance to shine. The statisticians assign the order and number of evaluations the TVs will have. Trained judges view a number of different images on each TV several times--and the TV brand is masked--so that we can be sure that the only factor that affects scores is how good the picture looks.

The rinse cycle. One winter we were testing washing machines, as always, by washing a measured load of “dirty” clothes in a measured amount of the same detergent. We ran each model in a random order four times. Then our numbers guys analyzed the results and noticed a distinct drop in the performance of all the machines, even those that had done well in earlier rounds. Turns out that our local water-treatment plant had been cleaned out over the holidays, in the middle of our testing. Our team hypothesized that the churned-up water was dirtier than usual, and so were the clothes that it soaked. We adjusted the Ratings, and changed our building’s water-filtering system.

When it comes to testing at Consumer Reports, nothing is left to chance.