May 2007
How our car seat tests went wrong
The missteps we made and some lessons we learned
WITHDRAWN This report from our February 2007 issue was pulled after the tests proved to be flawed.
A series of misjudgments and a key misunderstanding between Consumer Reports and an outside laboratory led to the publication of erroneous crash-test data in our recent report on infant car seats, an
expert investigation and interviews with those involved has revealed.
The report, in the February 2007 issue of Consumer Reports, was made public on Jan. 4 but was withdrawn--along with its test results--just 14 days later when evidence first surfaced that it was flawed.
The report attracted wide public attention because it said 10 of the 12 seats tested provided poor protection. Some seats twisted on their bases or flew apart. We urged recall of two models that got our lowest rating of Not Acceptable.
The withdrawal, which also generated broad publicity, shook the confidence of the public and safety experts in a 71-year-old institution that had enjoyed a largely unblemished record of product testing. "Mistakes are rare at Consumers Union but this one went right to the heart of what we do," says Jim Guest, president of the nonprofit publisher of Consumer Reports. "We had to figure out exactly what went wrong."
Soon after the withdrawal, we asked two independent consultants to review the tests: Kennerly H. Digges, former director of Vehicle Safety Research at the National Highway Traffic Safety Administration (NHTSA), which regulates vehicles and child seats, and Brian O'Neill, former president of the Insurance Institute for Highway Safety (IIHS), which runs its own crash-test program.
Digges and O'Neill were given access to documents and communications concerning the project, and interviewed technical staff from Consumer Reports, the outside laboratory where the tests were run, and NHTSA.
Their review concludes that Consumer Reports set out to raise the bar for car-seat safety but instead stumbled into methodological errors with misleading results.
The project's rationale was simple. NHTSA requires car seats sold in the U.S. to pass a 30-mph front-impact crash test, the same standard to which all new passenger vehicles are held. But many vehicles are also tested in tougher 35-mph front- and 38-mph side-impact crashes as part of the agency's New Car Assessment Program (NCAP) to measure their crashworthiness. Child seats are not required to pass the more rigorous tests, and we wanted to know how they would behave under NCAP-like conditions.
HOW WE TESTED
The tests were performed by the independent laboratory using a piston-driven "sled" that mirrors the acceleration that occupants suffer during a collision, a widely used technique for car-seat crash simulations (see illustration.). While most CR tests are done in-house, we ran 11 percent of last year's tests at outside labs with special equipment or expertise. Our practice is not to disclose their names; we take public responsibility for the results.
The series of misjudgments, Digges and O'Neill said, stemmed mainly from CR's decision to develop and run the side-impact tests without extensive consultation with other experts. We took that step based on our decades-long experience with front-impact simulations--CR was among the first to test child seats this way, back in 1972--as well as our practice of limiting contact with government and industry to avoid influencing the independence of our judgment.
That decision was a mistake, they said. No federal standard exists for simulating 38-mph side impacts, they noted, and "as such, there were large opportunities for tests to go wrong." CR's practice differs from that of some other test organizations, which discuss protocols with manufacturers and others before, during, and after testing. "This openness does not have to mean that the manufacturers can subvert or weaken programs," Digges and O'Neill said, "but it does provide opportunity for important changes to programs to occur and greatly reduces the chances that there will be major criticisms when results are released."
The key misunderstanding concerned the proper speed for the test. In written and oral instructions, CR engineers asked that the side-impact tests be run at 38 mph to mimic the NCAP protocol. Under NCAP, that number refers to the speed of the striking vehicle--a car-sized moving barrier that smashes into a stationary vehicle being tested. But once the two collide, they move off more or less as a unit. The resulting velocity of the struck vehicle, and hence of the crash dummies inside, is only about half that of the striking vehicle, since the striking vehicle's momentum is shared between the two.
The contractor, on the other hand, assumed the 38-mph figure referred to the post-impact speed of the struck vehicle and set up the test accordingly. "This fundamental misunderstanding goes back to the early communications between CU and the contractor," Digges and O'Neill said. The result? Unknown to CU, all the side-impact tests took place under conditions that could occur only if the striking vehicle were traveling at 70 mph or more--close to twice the speed we thought.
That rendered the results nearly meaningless. Relatively few side-impact crashes occur at such speeds, experts say, and in those that do, the greatest risk of injury is from "intrusion," the tendency of the striking vehicle to crush the other car's passenger compartment, which sled tests generally do not simulate.
Once the misunderstanding arose, it was never discovered, despite ongoing contacts and site visits. Our engineers did not have deep knowledge of side-impact sled simulations and relied largely on the expertise of the lab, which has many years of experience in this field. But the contract did not specifically call for the lab to consult on test development, Digges and O'Neill said, adding that the lab "viewed its role as little more than a sled operator. The contractor was willing to run whatever tests were requested." The resulting article was edited and fact-checked by our staff, and reviewed, in an early version, by the lab, without this crucial issue coming to light.
On a separate topic, the two consultants endorsed CR's decision to withdraw its request for a recall of the Evenflo Discovery, one of the two seats rated Not Acceptable. The lab had installed the seat in a manner that it felt adhered to federal regulations and the manufacturer's instructions, and concluded that the seat failed the government-required 30-mph front-impact test. After publication, however, both NHTSA and the manufacturer disputed this interpretation of the rules. When the seat was retested using NHTSA's installation method, it passed, and so we have withdrawn both the recall request and the Not Acceptable rating.
The other Not Acceptable seat, the Eddie Bauer Comfort, has been discontinued, but if you have one and need help installing it, go to www.djgusa.com.
Finally, CR has withdrawn its 35-mph front-impact results, although neither the consultants nor others identified specific flaws like those of the side-impact tests. "Given the lack of a widely accepted test protocol, we think it's better to hold off for now," Guest explained.
A CRITICAL RECEPTION
Publication of the flawed report brought protests from manufacturers whose seats were rated poorly. "We unequivocally stand behind the safety of the Discovery car seat based on over 200 independent tests," Rob Matteucci, Evenflo's chief executive, told TV interviewers. Britax, whose Companion seat failed the faulty test, noted that the same product had been rated No. 1 by Consumer Reports in May 2005 based on earlier 30-mph front-impact tests.
Several manufacturers asked to review the test data with CR engineers, and four have visited our offices. As a matter of policy, CR does share such information, but only for a manufacturer's own products, not for those of its competitors.
NHTSA engineers, who had been shown the article one day before it appeared, also had questions. They visited our Yonkers, N.Y., headquarters on Jan. 12 and then ran tests of their own over the Martin Luther King Jr. holiday that found the flaw. NHTSA shared the details with CR on Jan. 17, and, after confirming the problem, CR pulled the article the next day. Guest then sent an e-mail or letter to nearly 6 million subscribers withdrawing all our conclusions and apologizing.
Several safety experts we interviewed are sharply critical of Consumer Reports for not spotting the problem before publication. "We know that child restraints are remarkably effective at protecting children in crashes, and to find such gross failures didn't seem to line up with what is happening in the real world," says David Zuby, senior vice president of vehicle research at the IIHS. Indeed, NHTSA has found that properly used child restraints may cut the risk of death by as much as 54 percent for toddlers in car crashes and by 71 percent for infants.
CR editors and engineers say they understand the criticism but were swayed by other evidence that seemed to confirm the tests' validity. Two U.S.-made car seats passed completely, for example, as did two European models that were added to the project after other U.S. seats failed. The latter finding seemed logical because European seats, unlike U.S. models, sometimes undergo side-impact testing.
Other experts criticize the article as too alarmist and question whether improving child-seat crashworthiness is really the best way to make children safer. They note that about half of the 450 U.S. children under age 5 who died while riding in vehicles in 2005 were not restrained properly anyway, so stronger seats would have done them no good.
A more effective road to safety, they say, would be to make car seats easier to use, improve the systems that attach seats to vehicles, and--most important--persuade more adults to keep their children restrained for longer. "The safety community has had tremendous success over the past 10 years convincing parents to use seats, and we worried this article might put doubt in their minds," says Kristy Arbogast, director of the Partners for Child Passenger Safety program at Children's Hospital of Philadelphia. Lorrie Walker, the technical adviser of Safe Kids Buckle Up, which has inspected nearly 1 million car seats, says, "We had families call and ask whether they should even keep using their car seats. It took a lot of extra work to make people feel confident in these products again." All 50 states require infants to be in car seats.
"To shut manufacturers out of the process was shortsighted," says Robert Waller Jr., president of the Juvenile Products Manufacturers Association, a trade association, since consultations could have turned up the flaws. "We are willing to work with CR in addressing this issue. We have the same goal--wanting to develop safer products."
The report, in the February 2007 issue of Consumer Reports, was made public on Jan. 4 but was withdrawn--along with its test results--just 14 days later when evidence first surfaced that it was flawed.
The report attracted wide public attention because it said 10 of the 12 seats tested provided poor protection. Some seats twisted on their bases or flew apart. We urged recall of two models that got our lowest rating of Not Acceptable.
The withdrawal, which also generated broad publicity, shook the confidence of the public and safety experts in a 71-year-old institution that had enjoyed a largely unblemished record of product testing. "Mistakes are rare at Consumers Union but this one went right to the heart of what we do," says Jim Guest, president of the nonprofit publisher of Consumer Reports. "We had to figure out exactly what went wrong."
Soon after the withdrawal, we asked two independent consultants to review the tests: Kennerly H. Digges, former director of Vehicle Safety Research at the National Highway Traffic Safety Administration (NHTSA), which regulates vehicles and child seats, and Brian O'Neill, former president of the Insurance Institute for Highway Safety (IIHS), which runs its own crash-test program.
Digges and O'Neill were given access to documents and communications concerning the project, and interviewed technical staff from Consumer Reports, the outside laboratory where the tests were run, and NHTSA.
Their review concludes that Consumer Reports set out to raise the bar for car-seat safety but instead stumbled into methodological errors with misleading results.
The project's rationale was simple. NHTSA requires car seats sold in the U.S. to pass a 30-mph front-impact crash test, the same standard to which all new passenger vehicles are held. But many vehicles are also tested in tougher 35-mph front- and 38-mph side-impact crashes as part of the agency's New Car Assessment Program (NCAP) to measure their crashworthiness. Child seats are not required to pass the more rigorous tests, and we wanted to know how they would behave under NCAP-like conditions.
HOW WE TESTED
The tests were performed by the independent laboratory using a piston-driven "sled" that mirrors the acceleration that occupants suffer during a collision, a widely used technique for car-seat crash simulations (see illustration.). While most CR tests are done in-house, we ran 11 percent of last year's tests at outside labs with special equipment or expertise. Our practice is not to disclose their names; we take public responsibility for the results.
The series of misjudgments, Digges and O'Neill said, stemmed mainly from CR's decision to develop and run the side-impact tests without extensive consultation with other experts. We took that step based on our decades-long experience with front-impact simulations--CR was among the first to test child seats this way, back in 1972--as well as our practice of limiting contact with government and industry to avoid influencing the independence of our judgment.
That decision was a mistake, they said. No federal standard exists for simulating 38-mph side impacts, they noted, and "as such, there were large opportunities for tests to go wrong." CR's practice differs from that of some other test organizations, which discuss protocols with manufacturers and others before, during, and after testing. "This openness does not have to mean that the manufacturers can subvert or weaken programs," Digges and O'Neill said, "but it does provide opportunity for important changes to programs to occur and greatly reduces the chances that there will be major criticisms when results are released."
The key misunderstanding concerned the proper speed for the test. In written and oral instructions, CR engineers asked that the side-impact tests be run at 38 mph to mimic the NCAP protocol. Under NCAP, that number refers to the speed of the striking vehicle--a car-sized moving barrier that smashes into a stationary vehicle being tested. But once the two collide, they move off more or less as a unit. The resulting velocity of the struck vehicle, and hence of the crash dummies inside, is only about half that of the striking vehicle, since the striking vehicle's momentum is shared between the two.
The contractor, on the other hand, assumed the 38-mph figure referred to the post-impact speed of the struck vehicle and set up the test accordingly. "This fundamental misunderstanding goes back to the early communications between CU and the contractor," Digges and O'Neill said. The result? Unknown to CU, all the side-impact tests took place under conditions that could occur only if the striking vehicle were traveling at 70 mph or more--close to twice the speed we thought.
That rendered the results nearly meaningless. Relatively few side-impact crashes occur at such speeds, experts say, and in those that do, the greatest risk of injury is from "intrusion," the tendency of the striking vehicle to crush the other car's passenger compartment, which sled tests generally do not simulate.
Once the misunderstanding arose, it was never discovered, despite ongoing contacts and site visits. Our engineers did not have deep knowledge of side-impact sled simulations and relied largely on the expertise of the lab, which has many years of experience in this field. But the contract did not specifically call for the lab to consult on test development, Digges and O'Neill said, adding that the lab "viewed its role as little more than a sled operator. The contractor was willing to run whatever tests were requested." The resulting article was edited and fact-checked by our staff, and reviewed, in an early version, by the lab, without this crucial issue coming to light.
On a separate topic, the two consultants endorsed CR's decision to withdraw its request for a recall of the Evenflo Discovery, one of the two seats rated Not Acceptable. The lab had installed the seat in a manner that it felt adhered to federal regulations and the manufacturer's instructions, and concluded that the seat failed the government-required 30-mph front-impact test. After publication, however, both NHTSA and the manufacturer disputed this interpretation of the rules. When the seat was retested using NHTSA's installation method, it passed, and so we have withdrawn both the recall request and the Not Acceptable rating.
The other Not Acceptable seat, the Eddie Bauer Comfort, has been discontinued, but if you have one and need help installing it, go to www.djgusa.com.
Finally, CR has withdrawn its 35-mph front-impact results, although neither the consultants nor others identified specific flaws like those of the side-impact tests. "Given the lack of a widely accepted test protocol, we think it's better to hold off for now," Guest explained.
A CRITICAL RECEPTION
Publication of the flawed report brought protests from manufacturers whose seats were rated poorly. "We unequivocally stand behind the safety of the Discovery car seat based on over 200 independent tests," Rob Matteucci, Evenflo's chief executive, told TV interviewers. Britax, whose Companion seat failed the faulty test, noted that the same product had been rated No. 1 by Consumer Reports in May 2005 based on earlier 30-mph front-impact tests.
Several manufacturers asked to review the test data with CR engineers, and four have visited our offices. As a matter of policy, CR does share such information, but only for a manufacturer's own products, not for those of its competitors.
NHTSA engineers, who had been shown the article one day before it appeared, also had questions. They visited our Yonkers, N.Y., headquarters on Jan. 12 and then ran tests of their own over the Martin Luther King Jr. holiday that found the flaw. NHTSA shared the details with CR on Jan. 17, and, after confirming the problem, CR pulled the article the next day. Guest then sent an e-mail or letter to nearly 6 million subscribers withdrawing all our conclusions and apologizing.
Several safety experts we interviewed are sharply critical of Consumer Reports for not spotting the problem before publication. "We know that child restraints are remarkably effective at protecting children in crashes, and to find such gross failures didn't seem to line up with what is happening in the real world," says David Zuby, senior vice president of vehicle research at the IIHS. Indeed, NHTSA has found that properly used child restraints may cut the risk of death by as much as 54 percent for toddlers in car crashes and by 71 percent for infants.
CR editors and engineers say they understand the criticism but were swayed by other evidence that seemed to confirm the tests' validity. Two U.S.-made car seats passed completely, for example, as did two European models that were added to the project after other U.S. seats failed. The latter finding seemed logical because European seats, unlike U.S. models, sometimes undergo side-impact testing.
Other experts criticize the article as too alarmist and question whether improving child-seat crashworthiness is really the best way to make children safer. They note that about half of the 450 U.S. children under age 5 who died while riding in vehicles in 2005 were not restrained properly anyway, so stronger seats would have done them no good.
A more effective road to safety, they say, would be to make car seats easier to use, improve the systems that attach seats to vehicles, and--most important--persuade more adults to keep their children restrained for longer. "The safety community has had tremendous success over the past 10 years convincing parents to use seats, and we worried this article might put doubt in their minds," says Kristy Arbogast, director of the Partners for Child Passenger Safety program at Children's Hospital of Philadelphia. Lorrie Walker, the technical adviser of Safe Kids Buckle Up, which has inspected nearly 1 million car seats, says, "We had families call and ask whether they should even keep using their car seats. It took a lot of extra work to make people feel confident in these products again." All 50 states require infants to be in car seats.
"To shut manufacturers out of the process was shortsighted," says Robert Waller Jr., president of the Juvenile Products Manufacturers Association, a trade association, since consultations could have turned up the flaws. "We are willing to work with CR in addressing this issue. We have the same goal--wanting to develop safer products."
