Scientists have found:
A low-fat, high-fiber diet halves the risk of death from all causes.
Antiarrhythmic drugs save the lives of heart attack patients.
Incoming e-mails and phone calls lower IQ by 10 points.
Wrong, wrong and wrong again.
In fact, David H. Freedman argues in his new book Wrong: Why Experts Keep Failing Us—and How to Know When Not to Trust Them, most scientific and medical studies that you see or hear mentioned in news reports turn out to be incorrect, misleading or irrelevant to most people. And, unfortunately, that's true even if they're published in a major peer-reviewed journal. Other commonly offered expert advice, on everything from investing to home repair, is equally suspect, Freedman says.
The forces pushing bad research to the forefront are plentiful, he argues. They start with a bias for publishing surprising new results, and are compounded by a range of researchers' sins, including mismeasurement, manipulation of data, conflicts of interest, ineptitude and outright fraud.
In an interview with the AARP Bulletin, Freedman discussed how consumers should use expert advice and how to recognize its pitfalls.
Q. Let's look at a specific finding. There was a report July 29 in the British Medical Journal that people taking calcium supplements were about 30 percent more likely to suffer a heart attack. If experts are usually wrong, how should people reading about this report react?
A. People shouldn't reject all expert advice that they read about in the papers. What we want is to learn how to separate the stuff that's more likely to be true from the stuff that's less likely to be true. Now it turns out that, for a variety of reasons, medical conclusions about taking supplements or eating certain types of food tend to be among the least reliable of all medical advice.
Q. How does that apply in this particular case?
A. It should set off alarm bells, but I definitely wouldn't recommend stopping calcium based on this, or basically making any medical decisions based on what you read in the paper without talking to your doctor. A supplement like calcium is going to have many different effects on the body. It's going to affect different people in different ways. It's going to affect the same people in different ways at different stages of their lives. There are so many complexities, the fact that someone tries to boil it down to a simple conclusion already means that the advice probably isn't going to be that useful. On the other hand, if there are several more studies that come out over the next few years that start to support this notion that calcium may cause more problems than it solves, then people might seriously consider stopping or limiting their intake of calcium.
Q. What are the clues that a study is likely to be valid?
A. First, it's a very large study that has followed thousands of people for five or 10 or even 20 years.
Also, it's a randomized study, which means one group got a treatment or did one thing and a different group didn't, and people were randomly assigned to the two groups.
And the most important thing is that the finding agrees with findings of many similar studies that have been done over a period of years.
Q. Let's say I hear about a study on some topic that's very important to me. Where would I look, beyond the mass media, for information about what it actually says and whether it's likely to be valid?
A. Obviously, the best place to go is your doctor, and if you don't have a doctor whom you feel comfortable talking with about this sort of thing, then you need a new doctor. Beyond that, go to medical journals. Even if you don't understand all the technical terms, it's still written in English, and it is understandable. It's often surprising for a lay person to look at a medical journal finding and compare it to a news report. Anybody can see that there's stuff that the mass media gets wrong, or leaves out, or distorts in some way.
Q. Are you better off sticking with one medium over another when seeking expert advice, for example newspapers over the Internet?
A. The simple answer is no. Lousy expert findings are everywhere. I'll just pick on the New York Times, which of course is a wonderful publication with some of the world's best journalists. Still, most of what they publish will not hold up. This is because most of what's published in medical journals turns out to be wrong. But I think we need to be especially cautious on the Internet, which has some of the best expert advice out there, and some of the worst. And, unfortunately, search engines like Google are more likely to take you to the bad stuff.
A. The bad stuff tends to be especially popular. And Google tends to bring you to the popular expert advice. But even if people are looking at the very best sources, the medical journals themselves, they're still going to end up with mostly bad expert findings.
Q. Is there any kind of "batting average" that tells you how accurate a medical journal is?
A. No. You run into a whole new set of problems when you try to keep score. The way the medical field itself measures how experts do is by how often they're cited. But this turns out to be a terrible way of doing it. Some findings that turned out to be completely fraudulent are some of the most highly cited findings in history.
Q. Are other kinds of expert advice more reliable?
A. Obviously, older people are more vulnerable to bad financial advice. And financial advice is, as a rule, the most terrible advice we see out there, except for the somewhat conservative, old-fashioned advice that we've all heard a million times: Have a diversified portfolio and don't try to beat the market. That's the most boring advice in the world, and it has held up decade after decade while almost every other kind of financial advice ultimately falls flat.
Q. In your book, you describe your mixed experiences with online help forums. How do you distinguish between the good and bad advice offered by ordinary people online?
A. I really caution against going online, looking at what some people say in a forum and taking it as the truth. I've seen some terrible examples of bad information about prescription drugs or food or exercise that could actually be quite damaging. It's important to go online and look at many, many different sources to develop a sense of who's probably got it right and which stories seem really fishy.
Q. Have you seen any evidence that older experts are wiser than others?
A. You often get some of the very best and very worst advice from more senior experts. Within science, people later in their careers are often freed from the pressure to publish. They already have tenure. Their careers are pretty much settled. So they're not forced to seek these kinds of exciting findings that turn out to be wrong. On the other hand, having become unmoored from having to worry about the opinions of their fellow experts, some will go off on wild tangents. So I do not recommend judging people solely on the basis of age.
Art Dalglish is an editor and writer based in Maryland.
Next ArticleRead This