Why we take intermittent test failures so seriously

As a testing organization with 73 years of experience, Consumer Reports has often come across products that fail our tests but do so only intermittently, not every time. That was the case most recently with the Orbit Infant System, a car-seat-and-stroller combination whose carrier detached from its base in two of six simulated 30-mph frontal crash tests. Readers of such a report may wonder how intermittent failures arise, and what they mean for consumers trying to make a decision about whether or not a product is safe. This blog addresses those questions and cites three recent examples to explain why we generally take such failures very seriously.
 
There are numerous reasons why products may fail only some of the time. Sometimes, safety standards call for a product to be tested in a variety of ways; the product may fail when used one way but pass when used another way. Test-to-test repeatability and lab-to-lab reproducibility can also contribute to variations in test results. Manufacturing variations may play a role in intermittent failures, as can changes in a product's materials or design. So even under optimal conditions, it is not uncommon to see some variation in test results—up to and including situations in which some samples of a given model pass a test, while others fail.
 
But our point of view is that, when it comes to safety, no test failure is acceptable—and that's especially so where child safety is concerned. Products should be designed with enough margin of safety that they pass each and every time.
 
One recent example: When we first tested the Baby Jogger City Mini stroller, we were troubled to find sporadic problems with the harness restraints. The voluntary industry standard specifies that the harness system must hold a test dummy in place even when the stroller is turned upside down. But in our tests, the results were inconsistent. In further testing, when we pulled directly outward on the buckle it held fast. But if we pulled sideways on the buckle, it could release with as little as two pounds of force.
 
When we shared those results with the manufacturer, it turned out that the company had already changed the design to put stronger buckles on newer strollers. Nevertheless, the Baby Jogger Company, working with the Consumer Product Safety Commission, voluntarily recalled some 41,000 older strollers that had the earlier version of the buckles.

Meanwhile, Consumer Reports testers, who continually update our test procedures, have revised our stroller protocols to include the sideways pull test, even though it is not part of the industry standard test protocol. Reason: We want to anticipate problems that might arise under ordinary, real-world conditions, and not just in an industry or government test.

Another recent instance of intermittent failures: One of our testers was exercising on the Stamina model 1725 elliptical trainer when the left pedal of the machine came loose and fell to the floor. We later found that on both the Stamina 1725, and on the similar Stamina 1772, the bolts holding the pedals in place would sometimes work their way loose as the user pedaled the crank. The problem didn't happen every time—but it happened often enough that we rated both machines a "Don't Buy – Safety Risk." And in July, the CPSC recalled both Stamina trainers.

Perhaps the most dramatic recent example of intermittent failure involved an outdoor gas grill called the Broil King Signet 90. As part of our standard testing, we always put several very fatty ribeye steaks on each grill at high heat in order to see if the dripping fat catches fire. Most top-rated grills flare up momentarily and then stop. But the Signet 90 flared up quite a bit, made a popping sound, and then black smoke came out from under the lid. The testers turned off the propane tank and pulled it out of the cabinet under the grill. But as they did, molten metal dripped down onto the area beneath the grill, just missing one tester's hand. The grill's firebox had melted.

We tested two more Signet 90 grills. In one of them, the firebox also melted. In the other it didn't, but there was cracking and deformation. Yet we cautioned readers about the grill last year, and its maker, Onward Manufacturing of Canada, voluntarily recalled the model and sent buyers a kit intended to fix the problem. (As it happens, we also later criticized the kit).

The point of these three examples is that even when failures are intermittent, we pay serious attention to them if they affect your safety. We believe it is our responsibility to tell consumers that they face a potential risk. Our tests are not designed to explain *why* a product performed as it did; that's a matter for the manufacturer and/or regulators to determine. Nor are we in a position to compel a manufacturer to make changes in design or even to recall a product, as, say, a government agency might. Moreover, our reporting on intermittent failures may mean government agencies and manufacturers don't always agree with us. But it means we are fulfilling our responsibility to our only constituents: Consumers. The fact that a product fails only intermittently is not necessarily a good sign, in our view. We think buyers have a right to expect that the products they buy will be safe to use under real-world conditions each and every time.