I Think You'll Find It's a Bit More Complicated Than That (5 page)

So you’ve probably heard that smoking might prevent Alzheimer’s. It comes up in the papers, sometimes to say it’s a true finding, sometimes to say it’s been refuted. Maybe you think it’s a mixed bag, that the research is contradictory, that ‘experts are divided’. Perhaps you smoke, and joke about how it’ll stop you losing your marbles, at least.

This month, Janine Cataldo and colleagues publish a
systematic review on the subject
, but with a very interesting twist. First, they found all the papers ever published on smoking and Alzheimer’s, using an explicit search strategy which they describe properly in the paper – because they’re scientists, not homeopaths – to make sure that they found all of the evidence, rather than just the studies they already knew about, or the ones which flattered their preconceptions.

They found forty-three in total, and overall, smoking significantly increases your risk of Alzheimer’s. But they went further. Eleven of the studies were written by people with affiliations to the tobacco industry. This wasn’t always declared, so to double check, the researchers searched on the University of California’s
Legacy Tobacco Documents Library
, a vast collection of scanned material that has been gathered over decades of legal action.

If you ever want to spend a chilling afternoon living in the head of an industry whose product has been proven to kill a third of its customers, this is the place for you. ‘
The importance of younger adults
’ is a tobacco industry paper that uses financial modelling to explain the importance of recruiting teenage smokers to replace the dying older ones before it’s too late, and explains that ‘repeated government studies have shown less than one third of smokers start after age 18 [and] only 5 per cent of smokers start after age 24’. ‘
Youth cigarette – new concepts
’ from Marketing Innovations Inc. takes these ideas further, into cola- and apple-flavour cigarettes, because ‘apples connote goodness and freshness’.

How much did it matter if the researchers had worked for the tobacco companies? A lot: the risks of Alzheimer’s associated with smoking reported by these papers were on average about a third lower than in those conducted by other researchers, and they produced many papers showing cigarettes were actively protective. If you exclude the eleven papers by researchers associated with the tobacco industry, and look only at the remaining thirty-two, your chances of getting Alzheimer’s as a smoker are vastly higher: for the gamblers out there, comparing a smoker against a non-smoker, the odds of getting Alzheimer’s are higher by 1.72 to 1.

So does that mean we can comfortably ignore all research that comes from people who disgust us? In the 1930s, identifying toxic threats in the environment became an important feature of the Nazi project to build a master race through ‘racial hygiene’.
Two researchers, Schairer and Schöniger
, were working on biological theories of degenerate behaviour under Professor Karl Astel, a scientist who helped organise the vile ‘euthanasia’ operation that murdered 200,000 mentally and physically disabled people.

In 1943 they published a well-conducted case-control study demonstrating a relationship between smoking and lung cancer, almost a decade before any other researchers elsewhere. Their paper wasn’t mentioned in the classic Doll and Bradford Hill paper of 1950, and if you check in the Science Citation Index, it was referred to only four times in the 1960s, once in the 1970s, and then not again until 1988, despite providing a valuable early warning on a killer that would cause 100 million early deaths in the twentieth century. It’s not obvious what you should do with evidence from untrustworthy sources, but it’s always worth appraising its untrustworthiness with the best tools available.

Foreign Substances
in Your Precious Bodily Fluids

Guardian
, 9 February 2008

You’ll find fluoride in tea, beer and fish, which might sound like a balanced diet to you. This week the Health Secretary Alan Johnson announced a major new push for putting it in all tap water, with some very grand promises, and in the face of serious opposition.

In Stanley Kubrick’s film
Dr Strangelove
, General Jack D. Ripper first developed his theories about environmental poisoning and bodily fluids when he experienced impotence, fatigue, and a pervasive sense of emptiness during the physical act of love. He instantly identified the cause: a communist plot to pollute our precious bodily fluids with fluoride.

Bill Etherington MP
calls it a ‘poison’
. Campaigners say Nazis used it to subdue people in concentration camps. According to the
Guardian
’s own (sadly departed) alternative health columnist, fluoride is ‘in the same league as
lead and arsenic
’.

The reality is that anybody making any confident statement about fluoride – positive or negative – is
speaking way beyond the evidence
. In 1999 the Department of Health commissioned the Centre for Reviews and Dissemination at York University to carry out a systematic review of the evidence on
the benefits of fluoridation
for dental health, and to look for evidence of harm. Little new work has been done since.

They found 3,200 research papers, mostly of very poor quality. The ones which met the minimum quality threshold suggested that there was vaguely, possibly, around a 15 per cent increase in the number of children without dental caries in areas with fluoridated water, but the studies generally couldn’t exclude other explanations for the variance. Of course, the big idea with fluoride in water is that it can reduce social inequalities in dental health, because everyone drinks it: but there isn’t much evidence on that either – the work is of even poorer quality, and the results are inconsistent.

So when the British Dental Association says there is
‘overwhelming evidence’
that adding fluoride to water helps fight against tooth decay, they’re with General Ripper. And when
Alan Johnson says
: ‘Fluoridation is an effective and relatively easy way to help address health inequalities, giving children from poorer backgrounds a dental health boost that can last a lifetime,’ he’s really just pushing an admirably old-fashioned line that complex social problems can be addressed with £50 million worth of atoms. The people behind the York review have had to spend a fair amount of time pointing out that
people are misrepresenting
their work.

But since I’m in the mood for some scaremongering, let’s not forget the potential harms. Fluoridation will give around one in eight people mottled teeth (‘fluorosis’). And there’s something else to worry about, if you like worrying. An observational study from Taiwan found a high incidence of bladder cancer in women from areas where the natural fluoride content in water was high. It might easily have been a chance finding – the study in question measured lots of variables, and if you measure enough things, then some of them are bound to come out positive, just by chance. But it could be real.

The problem here is one of small effect sizes. You don’t need a careful designed study to show that falling out of a plane will probably kill you, but finding a link between fluoride and bladder cancer would be a pig to research, because the effect size is small, the exposure is spread over half a century, and the outcome – bladder cancer – takes a lifetime to reveal itself. Welcome to the finer details behind ‘more research is needed’.

And the fascinating thing about public health is that, with population effects, the numbers can start to get very scary, very quickly: in the UK, for example, just a tiny 10 per cent increase in risk would give you one thousand extra new cases of bladder cancer every year. Fear. Actually, I enjoyed that. Maybe I should move to the
Mail
.

How Myths Are Made

Guardian
, 8 August 2009

Much of what we cover in this column revolves around the idea of a ‘systematic review’, where the literature is surveyed methodically, following a predetermined protocol, to find all the evidence on a given question. As we saw in another column,
¹
for example, the Soil Association would rather have the freedom to selectively reference only research that supports their case, rather than the totality of the evidence.

Two disturbing news stories demonstrate how this rejection of best practice can also cut to the core of academia.

Firstly, the Public Library of Science in the US this week
successfully used a court order
to obtain a full trail of evidence showing how pharmaceutical company Wyeth employed commercial ‘ghost writers’ to produce what were apparently academic review articles, published in academic journals, under the names of academic authors. These articles, published between 1998 and 2005, stressed the benefits of taking hormones to protect against problems like heart disease, dementia and ageing skin, while playing down the risks. Stories like this, sadly, are commonplace; but to understand the full damage that these distorted reviews can do, we need to understand a little about the structure of academic knowledge.

In a formal academic paper, every claim is referenced to another academic paper: either an original research paper, describing a piece of primary research in a laboratory or on patients; or a review paper which summarises an area. This convention gives us an opportunity to study how ideas spread, and myths grow, because in theory you could trace who references what, and how, to see an entire belief system evolve from the original data.
Such an analysis
was published this month in the
British Medical Journal
, and it is quietly seminal.

Steven Greenberg from Harvard Medical School focused on an arbitrary hypothesis: the specifics are irrelevant to us, but his case study was the idea that a protein called β amyloid is produced in the skeletal muscle of patients who have a condition called ‘inclusion body myositis’. Hundreds of papers have been written on this, with thousands of citations between them. Using network theory, Greenberg produced a map of interlocking relationships, to demonstrate who cited what.

By looking at this network of citations he could identify the intersections with the most incoming and outgoing traffic. These are the papers with the greatest ‘authority’ (Google uses the same principle to rank webpages in its search results). All of the ten most influential papers expressed the view that β amyloid is produced in the muscle of patients with IBM. In reality, this is not supported by the totality of the evidence. So how did this situation arise?

Firstly, we can trace how basic laboratory work was referenced. Four lab papers did find β amyloid in IBM patients’ muscle tissue, and these were among the top ten most influential papers. But looking at the whole network, there were also six very similar primary research papers, describing similar lab experiments, which are isolated from the interlocking web of citation traffic, meaning that they received no or few citations. These papers, unsurprisingly, contained data that contradicted the popular hypothesis. Crucially, no other papers refuted or critiqued this contradictory data. Instead, those publications were simply ignored.

Using the interlocking web of citations, you can see how this happened. A small number of review papers funnelled large amounts of traffic through the network, with 63 per cent of all citation paths flowing through one review paper, and 95 per cent of all citation paths flowing through just four review papers by the same research group. These papers acted like a lens, collecting and focusing citations – and scientists’ attention – on the papers supporting the hypothesis, in testament to the power of a well-received review paper.

But Greenberg went beyond just documenting bias in what research was referenced in each review paper. By studying the network, in which review papers are themselves cited by future research papers, he showed how these reviews exerted influence beyond their own individual readerships, and distorted the subsequent discourse, by setting a frame around only some papers.

And by studying the citations in detail, he went further again. Some papers did cite research that contradicted the popular hypothesis, for example, but distorted it. One laboratory paper reported no β amyloid in three of five patients with IBM, and its presence in only a ‘few fibres’ in the remaining two patients; but three subsequent papers cited these data, saying that they ‘confirmed’ the hypothesis. This is an exaggeration at best, but the power of the social network theory approach is to show what happened next: over the following ten years, these three supportive citations were the root of 7,848 supportive citation paths, producing chains of false claim in the network, amplifying the distortion.

Similarly, many papers presented aspects of the β amyloid hypothesis as a theory – but gradually, through incremental mis-statement, in a chain of references, these papers came to be cited as if they proved the hypothesis as a fact, with experimental evidence, which they did not.

This is the story of how myths and misapprehensions arise. Greenberg might have found a mess, but instead he found a web of systematic and self-reinforcing distortion, resulting in the creation of a myth, ultimately retarding our understanding of a disease, and so harming patients. That’s why systematic reviews are important, that’s why incremental mis-statement matters, and that’s why ghost writing should be stopped.

Publish or Be Damned

Guardian
, 4 August 2005

I have a very long memory. So often with ‘science by press release’, newspapers will cover a story even though the scientific paper doesn’t exist, assuming it’s around the corner. In February 2004 the
Daily Mail
was saying that cod liver oil is ‘nature’s superdrug’. The
Independent
wrote: ‘They’re not yet saying it can enable you to stop a bullet or leap tall buildings, but it’s not far short of that.’ These glowing stories were based on a press release from Cardiff University, describing a study looking at the effect of cod liver oil on some enzymes – no idea which – that have something to do with cartilage – no idea what. I had no way of knowing whether the study was significant, valid or reliable. Nobody did, because it wasn’t published. No methods, results, conclusions to appraise. Nothing.