Is this a scientific survey?
There are generally two criteria that social scientists use to evaluate the quality of a survey: its validity and its reliability. Validity refers to whether the survey actually measures what it says it does. Reliability refers to whether the information generated by the survey would be repeated if the survey were to be conducted again.
We have strong evidence that our survey is both valid and reliable. The questions in the survey are designed professionally by experts in CR's National Research Center, in consultation with our automotive engineers and statisticians. Members of our survey team have more than 30 years of experience in conducting all sorts of consumer surveys. The survey uses an aided response technique that leads respondents through well-defined specific items and gives each respondent the same perspective in answering the questions. The data we report tracks well with other sources of repair and reliability information available on the market. Although we know that auto manufacturers pay close attention to our reports, they have not formally disputed our survey findings, which often identify problems that the manufacturers see in the warranty experience of their vehicles. We conduct a validation test every year and, in more than 30 years, have not found any evidence of bias. From year to year, our subscribers' reports of their problem experiences are fairly consistent; when there is a difference on a particular model, we can often attribute it to known issues with a particular component of a car.
Is the survey based on a representative sample?
A sample is considered to be representative of a population if the relevant characteristics of the population are reflected in the sample. So, considering the population of interest is critical in evaluating the quality of a sample. Our survey sample is drawn from the population of subscribers to ConsumerReports.org and to Consumer Reports magazine. While all subscribers are invited to participate in the survey, participation is voluntary, and there is always the possibility that those who respond are unique in some particular way. For example, subscribers have sometimes questioned whether those who respond are those who have a complaint to make about their cars.
To address this concern, we conduct a validation test every year. A representative sample of 8,000 to 10,000 subscribers are mailed the same questions about problem experiences with their cars at the same time as all subscribers are asked to complete the main survey. Using a combination of incentives and follow-up mailings, we attain at least a 40 to 50 percent response rate on this validation sample. The validation sample is known to be representative of the subscriber population; by comparing responses from this sample to responses of the main sample, we can assess whether the main sample is representative of the population overall. In more than 30 years we have not found any biases on any of the questions on any of the topics.
One reason for this is that our survey is an omnibus survey asking subscribers not only about their cars, but about a dozen other products, about major services they have used (such as insurance, hotels, and health plans), as well as suggestions for CR. The survey also requests participation in our board of directors election. Many subscribers return surveys reporting that they had no problems at all with their cars in the past year. This is true for all makes and models of cars. So, owners with complaints about their cars are not the only ones who return the survey.
Any survey has some sort of sampling frame that limits the people being surveyed. We choose our subscribers as our sampling frame. On average, CR subscribers tend to be more educated and affluent than the general population. With the growth of Consumer Reports online, a wider demographic range of individuals has been surveyed in recent years. However, our reliability questions do not ask respondents about their attitudes or opinions about the reliability of their cars, where one might expect different groups of individuals to have different perspectives. Instead, we ask for factual information about whether specifically defined problems occurred; these types of questions are less sensitive to the nature of the characteristics of the sample itself.
Further, our results track well with other sources of reliability information available on the market, and auto manufacturers have not formally disputed our survey findings, which often correspond to problems that the manufacturers see in the warranty experiences of the population of car owners at large.
Is the survey biased toward Japanese cars?
In our survey of CR subscribers, Japanese vehicles are popular. Also, many Japanese models have had relatively low rates of problems in our survey. But the fact that we received responses on more than 300 makes and models from nearly all domestic, European, and Korean manufacturers shows that our subscribers do not exclusively favor Japanese vehicles and that they buy a wide range of vehicles of all makes and models.
Unlike some other magazines or surveys, we do not take advertisements from any outside manufacturer, so we have no vested interests in the outcome of our survey. We have no agenda other than communicate accurate results of our survey. We do not consider country of origin in our analyses leading to our reliability ratings.
Some Japanese models in our survey have scored below average in reliability, and some American models for example, the Buick Verano and Chevrolet Traverse have scored above average. Those findings provide evidence against pro-Japanese bias on the part of our subscribers.
European luxury brands,especially Audi, have made recent progress in CR's latest surveys. The V6 version of the Audi Q5 and Audi Allroad led with the best predicted reliability scores in the compact luxury SUV and wagon segments respectively, topping the Japanese models.
Your survey results do not match with my experience. Is your survey wrong?
The CR reliability information reflects the problem rate, or percentage of cars that experienced problems, across at least 100 car owners. Even in the most unreliable models, some individual car owners are lucky and experience few or no problems during the 12 months covered by the survey. For example, in one of the worst models in our recent surveys, about 75 percent of the owners reported problems in at least one trouble area over the previous 12 months; of course, this means that about 25 percent of owners reported no problems. Your neighbor or friend might be one of those lucky owners. Of course, the opposite can happen as well—even in a model that tends to be quite reliable, there is an occasional "lemon."
Since the average number of problems is small for most models, is Consumer Reports overemphasizing differences that may not be important?
Beyond statistical significance, we believe these differences are also meaningful to car buyers. We think that car buyers should expect a new car to be entirely problem-free in its first months or years of service. While the difference between a and a may be small, a pattern of several less-than-perfect trouble spots in a brand new car should be cause for concern and does not bode well for a model's long-term reliability. We have not yet seen a single model in our survey that is entirely problem-free. More than that, the Fiat 500L, which had the worst new car prediction score in the 2014 survey, is about 11 times more likely to have a problem than the best, the Scion xB. Those differences among models are important for car buyers to consider in choosing a car. We present these scores for trouble spots primarily to allow consumers to compare the relative incidence of problems among models. While there are no guarantees, you can improve your odds of buying a reliable car if you choose a model that has had a lower rate of problems in the past.
Some people maintain their cars differently from others. How does this affect the Ratings?
The biggest growing area of concern in late-model cars is the audio, entertainment, communication, and navigation systems. Other problem areas include body integrity (squeaks, rattles, and leaks), and body hardware (locks, seats, and doors, Maintenance does not affect those items. Problems in those areas might more likely reflect the inherent design or quality.
How do you account for mileage differences?
Vehicles with higher mileage will most likely experience more problems than vehicles of the same age with lower mileage. We adjust our analyses to minimize differences among models due to varying mileage. Our data are mileage-standardized by dividing cars of each model into groups of high, average, and low mileage, and employing the statistical technique of direct standardization.
How do you know that manufacturers don't "stuff the ballot box?"
In most other surveys that draw their samples from lists of registered car owners, the researcher can control who is mailed a survey. In the Consumer Reports survey, buying a subscription to either the magazine or to ConsumerReports.org allows you to report on your experience with two cars. Some subscribers have wondered whether a manufacturer could just arrange to have their employees fill out questionnaires saying that their cars are reliable, as a way to influence our Ratings. Hypothetically, this is a potential weakness in our survey.
However, there are a number of ways that we can protect against this potential for fraud. For obvious reasons, we do not want to describe in detail the actions we take in this regard. We are confident that no manufacturer has succeeded.
If you state that first-year models are less reliable than later-year models, how can you still recommend some new Japanese models in their first year?
It is true that some newly introduced or redesigned models have more problems than later model years of that design. This happens even to models from the most reliable manufacturers, such as Toyota/Lexus/Scion and Honda/Acura. But despite the decline in reliability due to the new design, if the new/redesigned model still earned an average predicted reliability, and performed well in our testing and independent crash and rollover tests, we will recommend it. We will not recommend any model that has below average predicted reliability regardless of how well it performed in our testing. Occasionally, we will recommend a newly redesigned model with no reliability data specific to that design if the previous generation has a consistently outstanding reliability track record based on our surveys.