By Lisa Feldman Barrett | The New York Times | September 1, 2015
IS psychology in the midst of a research crisis?
An initiative called the Reproducibility Project at the University of Virginia recently reran 100 psychology experiments and found that over 60 percent of them failed to replicate — that is, their findings did not hold up the second time around. The results, published last week in Science, have generated alarm (and in some cases, confirmed suspicions) that the field of psychology is in poor shape.
But the failure to replicate is not a cause for alarm; in fact, it is a normal part of how science works.
Suppose you have two well-designed, carefully run studies, A and B, that investigate the same phenomenon. They perform what appear to be identical experiments, and yet they reach opposite conclusions. Study A produces the predicted phenomenon, whereas Study B does not. We have a failure to replicate.
Does this mean that the phenomenon in question is necessarily illusory? Absolutely not. If the studies were well designed and executed, it is more likely that the phenomenon from Study A is true only under certain conditions. The scientist’s job now is to figure out what those conditions are, in order to form new and better hypotheses to test.
A number of years ago, for example, scientists conducted an experiment on fruit flies that appeared to identify the gene responsible for curly wings. The results looked solid in the tidy confines of the lab, but out in the messy reality of nature, where temperatures and humidity varied widely, the gene turned out not to reliably have this effect. In a simplistic sense, the experiment “failed to replicate.” But in a grander sense, as the evolutionary biologist Richard Lewontin has noted, “failures” like this helped teach biologists that a single gene produces different characteristics and behaviors, depending on the context.
Similarly, when physicists discovered that subatomic particles didn’t obey Newton’s laws of motion, they didn’t cry out that Newton’s laws had “failed to replicate.” Instead, they realized that Newton’s laws were valid only in certain contexts, rather than being universal, and thus the science of quantum mechanics was born.
In psychology, we find many phenomena that fail to replicate if we change the context. One of the most famous is called “fear learning,” which has been used to explain anxiety disorders like post-traumatic stress. Scientists place a rat into a small box with an electrical grid on the floor. They play a loud tone and then, a moment later, give the rat an electrical shock. The shock causes the rat to freeze and its heart rate and blood pressure to rise. The scientists repeat this process many times, pairing the tone and the shock, with the same results. Eventually, they play the tone without the shock, and the rat responds in the same way, as if expecting the shock.
Originally this “fear learning” was assumed to be a universal law, but then other scientists slightly varied the context and the rats stopped freezing. For example, if you restrain the rat during the tone (which shouldn’t matter if the rat is going to freeze anyway), its heart rate goes down instead of up. And if the cage design permits, the rat will run away rather than freeze.
These failures to replicate did not mean that the original experiments were worthless. Indeed, they led scientists to the crucial understanding that a freezing rat was actually responding to the uncertainty of threat, which happened to be engendered by particular combinations of tone, cage and shock.
Much of science still assumes that phenomena can be explained with universal laws and therefore context should not matter. But this is not how the world works. Even a simple statement like “the sky is blue” is true only at particular times of day, depending on the mix of molecules in the air as they reflect and scatter light, and on the viewer’s experience of color.
Psychologists are usually well attuned to the importance of context. In our experiments, we take great pains to avoid any irregularities or distractions that might affect the results. But when it comes to replication, psychologists and their critics often seem to forget the powerful and subtle effects of context. They ask simply, “Did the experiment work or not?” rather than considering a failure to replicate as a valuable scientific clue.
As with any scientific field, psychology has some published studies that were conducted sloppily, and a few bad eggs who have falsified their data. But contrary to the implication of the Reproducibility Project, there is no replication crisis in psychology. The “crisis” may simply be the result of a misunderstanding of what science is.
Science is not a body of facts that emerge, like an orderly string of light bulbs, to illuminate a linear path to universal truth. Rather, science (to paraphrase Henry Gee, an editor at Nature) is a method to quantify doubt about a hypothesis, and to find the contexts in which a phenomenon is likely. Failure to replicate is not a bug; it is a feature. It is what leads us along the path — the wonderfully twisty path — of scientific discovery.
Read the original op-ed published in The New York Times.
Lisa Feldman Barrett is Professor of Psychology at Northeastern University and a CLBB Faculty member.
