Psychological studies can’t always be reproduced, and that’s OK

A boy takes the Rorschach inkblot test. Photo by Lewis J Merrim/Getty Images

A boy takes the Rorschach inkblot test. Photo by Lewis J Merrim/Getty Images

A large collaborative science initiative called the Reproducibility Project at the University of Virginia recently reran 100 psychology experiments and found that only about one in three studies could be replicated. The results were rapidly disseminated in mainstream and social media with most commentators concluding that psychological science shouldn’t be called a science at all.

Yet, reproducibility failure is common across many scientific disciplines, not just psychological science. For example, the pharmaceutical company Bayer recently reported that it failed to replicate about two-thirds of published studies identifying possible drug targets. During the decade he served as head of global cancer research at the pharmaceutical company Amgen, C. Glenn Begley and his team sought to replicate 53 landmark papers on cancer research published in top journals and conducted by reputable labs. They found that 47 of the 53 could not be replicated.

In 2012, researchers from Nanjing University published a paper on genetics that showed a microRNA in rice could regulate genes in the liver of mice that had eaten the rice. The result was of enormous importance in the field of transgenic crops. But other labs could not replicate the result.

In her recent New York Times essay, Northeastern University psychology professor Lisa Feldman Barrett describes replication failures in genetics and physics as well as psychological science that led to marked progress in these fields. Nor is replication failure a modern phenomenon. Scientific American blogger Jared Horvath describes three famous replication failure cases from the history of science involving the work of such giants as Galileo, John Dalton and Robert Millikan. Science proceeds by fits and starts, and replication failures don’t necessarily spell doom for a scientific endeavor. Instead, they point to refinements that must be made in theory and methods.

Science proceeds by fits and starts, and replication failures don’t necessarily spell doom for a scientific endeavor.

If other sciences also “suffer from a replication crisis,” why then, one wonders, is psychological science singled out for censure? I believe the answer lies in the discomfort we feel from the very idea that the mysteries of human nature can be studied scientifically. In my book, “The Other Side of Psychology,” I drew an analogy between this kind of reaction to psychological science and the reaction Isaac Newton elicited when he used a prism to demonstrate that white light was in fact a spectrum of separable colors. For centuries, people believed that light was the purest of all essences, indivisible and whole. Newton’s scientific demonstration that this “purest of all essences” was in fact a compound of many colors frightened and repulsed many people. The idea that things were not as they seemed, that nature could lie to us and that our powers of perception could be so misled was very threatening, indeed. These concerns seem somewhat reactionary and childish to us now. Yet, while people got used to the idea of scientifically studying light, the idea of scientifically studying the mind was — and frequently still is — met with the same type of skepticism and resistance.

The moral to be drawn from replication failures is that the results of single research papers should not be taken as fact. The very nature of science requires that the confidence we place in our theories and our experimental results should be proportional to the size of the body of evidence upon which they rest. In fact, John Ioannidis, professor of medicine at Stanford, has argued for years that most scientific results are less robust than researchers believe. In a recent interview with the Washington Post, Dr. Ioannidis praised this large-scale study on replication and claimed it should have repercussions beyond the field of psychology.