How bad is Mercedes reliability, really?

I'm sorry for all the confusion.

This is real research. The goal is to be able to say, "The average Mercedes X-Class requires about 1.2 repair trips per year, while the average Competitor Y requires 0.9 repair trips per year. So the Mercedes requires about one extra repair trip every three years." People can then more easily decide if the advantages of the Mercedes outweigh this disadvantage.

Right now, there's no way to make such a clear comparison. As a result, it's very possible that people overestimate how many repairs they'll have with a Mercedes. Everyone is of course aware of CR's take on Mercedes reliability. What few people realize is that the differences between CR's dots for a 2006 model year car is about 0.03 problems per car. That's about one extra problem for every 30 cars.

For the results of my research to be valid, it's as important to know how many cars have not had problems as how many cars have. The problem with gauging reliability from forums is that there's no way to tell how many cars are not having problems. Also, people with problems are much more likely to post here.

To provide the most valid possible results, my research process only collects data on repairs that haven't happened yet (with the exception of the first month). So people sign up, and afterwards report repairs when they happen.

If there are no repairs, participants simply provide an approximate odometer reading at the end of every quarter. That's how I know the total number of cars to include in the analysis.

Currently over 11,000 people have joined, with over 14,000 cars. The problem is that only a few hundred of these are Mercedes.

As stated in the OP, everyone who helps out will get full access to the results for free.

Credentials: Ph.D. in sociology from University of Chicago, which includes a lot of training in survey research; while there spent five years working at the National Opinion Research Center, a leader in the field.

Your concerns:

Most people who participate consistently participate, whether or not they've had problems.

Unless people check in at the end of the quarter, their responses earlier in the quarter are excluded from the analysis.

When people have gaps in their responses, I follow-up to fill the gaps. This affects only a couple dozen of the couple thousand people who've been responding, about one percent. Only 20-30% of participants report any problems in their first six months of participation.

Your concerns make sense, but the actual data demonstrate that they aren't actually problems.

Revenue:

Not much right now. Eventually there will be two ways to gain access to the results: participate in the research or pay a membership fee. Right now only the first is available. I feel this is a very fair way to structure the research, since those who want others to do all the work will enable those who do the work to get the results for free. With CR, you pay just as much even if you take the time to fill out their surveys.

Your choice whether or not to participate.

As I'm sure your recognize, every car is equally important, since the goal is to have a representative sample.

There are certainly challenges to survey research. But I've designed this research to overcome these challenges as much as possible. My sample sizes are currently small, yet the results have a great deal of face validity (i.e. they make sense).

Most recent results:

TrueDelta Vehicle Reliability Survey results

Next update will be mid-week.

I doubt it'll make much money. Hoping it'll eventually make enough to get my wife off my back. She's not crazy about the current income/time spent ratio. I believe that if I focus on providing the best possible information, the rest will take care of itself.

Routine maintenance and wear items don't get reported.

From the survey form:

Problems with these never count as problems:
Light bulbs -- if not a headlight and takes less than 5 minutes to change
Tires
Suspension alignment
Clutch lining
Fluids (oil, coolant, etc.)
Filters
Battery
Spark plugs and wires
Wiper blades
Belts, including timing belt
Radiator hoses

I also exclude brake pads and rotors from the main analysis. I didn't used to collect data on them at all, but many people wanted me to. So I've started asking people to report brake jobs, but will be reporting any results for them separately.

A few reasons:

1. It would require far more work from participants, and it's hard enough to get people to put in the time currently required

2. Few people are going to admit to not doing the required maintenance. They'd either claim to have done it or, more likely, just not participate

3. People who take part in the research are interested in reliability and are responsible enough to consistently respond to the surveys. Responsible people interested in reliability likely do at least the required maintenance.

4. Most things that break are not things that can be maintained: power window motors, audio systems, alternators, and so forth. And of the items subject to maintenance, only some will be caused by lack of maintenance. Ultimately, a very small percentage of repairs will be due to insufficient maintenance.

5. If need be, I can follow up in cases of major failures to find out if they might have been caused by insufficient maintenance. I don't have plans to do this, but it's a possibility, and would be a much more efficient use of people's time than having everyone report all their maintenance. #2 will still apply, but it will apply regardless.

MarcusF:

Another marketing rep? There is no marketing rep. There's only me. Welcome to the power of the Internet.

Collecting maintenance information wouldn't be too much work for me. It'd be too much work for participants. In other words, far fewer people would participate. And it's hard enough to get people to participate as is.

There are many more things I'd ask if it would not have an impact on participation to ask more questions.

But it would. So it's necessary to figure which questions are most necessary to ask, both in terms of the likely quality of responses and the usefulness of responses.

You're dealing in hypotheticals, and unlikely ones at that. I've been up close and personal with actual data for six quarters now. I read every response. Every three months I tweak the survey based on what I've learned. The actual data have not indicated a need for maintenance information.

It's possible that CR and JD Power do not collect maintenance information for the same reason.

Are the results of this research perfect? Of course not, very little is. But they will be much better than any reliability information currently available.

Wait for something that's perfect, and you'll never have anything.

Hawk20:

Thanks for the info and links.

J.D. Power does a different sort of survey in the UK. It's measuring overall satisfaction, and the reliability component asks for owners' opinions about the reliability of their cars. Good info to have, but not the same as measuring repair rates.

My own research strives to highlight the size of differences. Often these aren't large. I encourage the same approach with this J.D. Power survey. Most of the brands' scores are within a few points of the average of 79.

If one brand has a satisfaction score of 76, and another has a satisfaction score of 82, is that six-point difference really worth paying attention to? This range includes all brands from #6 Jaguar to #29 Peugeot.

Similarly, every brand from #10 Volvo to #25 Mitsubishi falls within two points of the average.

Looked at another way, how different are VWs, Skodas, and Seats? Since I'm in the U.S., I must admit I don't know the answer to this question, other than that they are based off the same platforms and tend to closely resemble one another. Yet Skoda's CSI score is 84.5, VW's is 80.2, and SEAT's is 77.8. Quite a range in the context of this study.

http://www.whatcar.com/NonCar/105774033.jpg