In his new book, NPR reporter Richard Harris picks through a problem that may corrode the future fortunes of health research if it isn’t fixed: the reproducibility crisis.
As a former biomedical researcher, I would describe “Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions” as a frustrating read — frustrating and timely. Science is a pyramid. Researchers build on each other’s findings, and this founding creed relies on being able to replicate experiments conducted by other labs. But scientists in biomedicine are struggling to reproduce the work of others, hence the reproducibility crisis. And as Harris explains in 240 or so pages, the problem is costing taxpayers:
The issue of reproducibility in biomedical science has been simmering for many years. As far back as the 1960s, scientists raised the alarm about well-known pitfalls—for instance, warning that human cells widely used in laboratory studies were often not at all what they purported to be. In 2005, John Ioannidis published a widely cited paper, titled “Why Most Published Research Findings Are False,” that highlighted the considerable problems caused by flimsy study design and analysis.
At issue is not simply that scientists are wasting their time and our tax dollars; misleading results in laboratory research are actually slowing progress in the search for treatments and cures. This work is at the very heart of the advances in medicine.
Basic research — using animals, cells, and the molecules of life such as DNA — reveals the underlying biology of health and disease. Much of this endeavor is called “preclinical research” with the hope that discoveries will lead to actual human studies (in the clinic). But if these preclinical discoveries are deeply flawed, scientists can spend years (not to mention untold millions of dollars) lost in dead ends.
Taken out of context, Harris’ book could be used to question the annual $30 billion budget for the National Institutes of Health, which supports 27 different institutes and centers along with hundreds of thousands at universities and organizations. Yet the book is less a laundry list of serious shortcomings, and more a blueprint for ways to improve academic research on the whole.
Once a problem reserved for backroom discussions at conferences, the reproducibility crisis has become firmly mainstream. Last Week Tonight with John Oliver touched on this concept last year with his imprecisely named segment “Science Studies” — most of the research mentioned in the piece involved health, not general science.
Explorations in physics, chemistry and much of biology are readily repeatable. But as Harris explains, such sloppy maneuvers are more dangerous for biology, where “there are many variables and a great deal of natural variation. It’s harder to see phenomena and harder to make sure personal biases don’t creep in.”
Each page of Harris’ book expands on this idea, outlining major flaws in biomedical research conducted in academic settings. Popular promises made with stem cells, genomics and personalized medicine unravel under his pen. Part of one cringeworthy chapter dives into repeated examples of misinterpreted statistics that make results sound more valid than they are. Another was completely devoted to drug studies in mice and their inability to yield remedies that work in humans:
Malcolm Macleod, a neurologist at the University of Edinburgh, worries about biomedicine’s dubious reliance on mice. He has spent much of his career trying to find ways to reduce the brain damage caused by strokes. In hundreds of animal experiments, mostly using mice, drugs have shown promise for treating stroke. Billions of dollars have gone into this research, and yet not one single drug acting on brain cells has been shown to be helpful when tested in people (the drug tPA is effective by breaking down clots and helping restore blood flow, but it doesn’t act on nerve cells damaged by a stroke). Neurologists started calling this long string of failures a “nuclear winter” for stroke research.
“My reading of the animal data for stroke is that it’s not possible to say if they’re good or bad models,” Macleod told me. It could be that experimentally induced strokes in mice are so radically different from a human stroke that there are no real lessons to learn. Or it could be that the experiments have been carried out so loosely that they have led the entire field astray.
Or could it be that stroke and other unremedied conditions, especially neurological ones, are extraordinarily complicated? Rigor Mortis does a great job of exposing pitfalls in modern biomedicine, but on occasion, it overstates these problems and their possible solutions, such as in this example with stroke.
Our brains demand more energy than any other organ. So, when a stroke cuts blood flow to the brain, even for a few minutes, havoc ensues. Some brain cells burst open, leaking inflammatory contents everywhere. Other cells persist and may take days to weeks to perish for myriad underlying reasons.
In an ideal world, doctors would be able to monitor or tinker with individual cells in a human brain, but right now, poking around inside a human head isn’t a feasible or ethical option. Biomedical researchers use mouse models to tease apart what’s behind this mess because rodents, as mammals, share many genetic and physiological traits with humans.
Harris mentions emerging alternatives, like induced pluripotent stem cells or clumps of cells called organoids, which are essentially new-age ways to test drugs in petri dishes. But if you think these models can act as a substitute for the complexity of any organ in a living creature, good luck.
Throughout the narrative, Harris points out several attempts to improve reproducibility. Companies responsible for lab supplies are taking extra precautions to validate materials before selling them. Some research facilities, such as the ALS Therapy Development Institute in Cambridge, Massachusetts, are now devoted to double checking experiments performed by others.
Pharmaceutical companies have taken on some of the burden, given academia feeds private-sector searches for new drugs. A false lead for a clinical trial can cost a company millions of dollars. Yet based on Harris’ book, pharmaceutical companies conduct much of their reproducibility testing in-house and the results aren’t always made public. Also, given pharma’s dependence on academia, one might expect hefty investments into universities, but a 2012 Congressional report found private companies provide only six percent of the nation’s basic research funding.
Harris highlights the competition for research funding as part of the problem. Due to factors like inflation, the rising costs of conducting research and the recession, funds for the National Institutes of Health have fallen 25 percent over the past decade. Tight budgets mean less room for scientists to rigorously test their hypotheses.
Yet the main takeaway from Harris’ discussions of mouse models and other examples of wasteful research is that science needs to slow down. The race for breakthroughs has caused researchers to cut corners when it comes to rigorous exploration — and the book features a number of memorable human patients who are paying the price with their lives:
If we curb our enthusiasm a bit, scientists will be less likely to run headlong after dubious ideas like transdifferentiation, and the public will be less likely to embrace the latest dietary fad. Of course this is a discouraging point of view for patients and advocates looking for rapid progress in the search for treatments and cures. But it’s important to distinguish between speed and haste.