"Replication is our only hope." "No. There is another"
Psychology is big into replication these days. A lot of people think that a major problem with the field is that many important results have not been replicated, and that this is in part because journals don't like to publish replications (not original or sexy enough).I'm all for replication; it's part of good science. But I've never been that into the whole 'replication movement' that's kicking around, and the reason crystallised for me during a 4am baby feed:
Being able to replicate a study is an effect, not a cause of good scientific practice. So the emphasis on replication as a goal has the whole thing backwards. We should actually be focusing on improving the experiments we run in the first place. If we run better experiments, the replicability will take care of itself.
Better experiments means better theory. We need strong, clearly formulated theories in order to generate strong, clearly formulated hypotheses that we can then test vigourously with robust results. We wrote a paper (Golonka & Wilson, 2012) about how Gibson's ecological psychology, while not a complete theory of psychology, stands as an excellent example of a) how to be theory driven in psychology and b) how well it can work out for you empirically. I'd like to quote the introductory section because I love how it came out and it summarises the argument:
When particle physicists recently found that some neutrinos had apparently traveled faster than light (Adams et al. 2011) it never actually occurred to them that this is what had happened. On the basis of the extraordinarily well supported theory of relativity, the physics community went ‘that's weird - I wonder what we did wrong?’, and proceeded to use that theory to generate hypotheses they could then test. It would take a lot of fast neutrinos to disprove relativity, and even though the result turned out to be caused by a faulty cable, the robust response by physicists stands as an example of the benefits of a good theory.Psychology is an empiricist discipline; it has no core theory and so it leans heavily on it's empirical results to prove that it's doing something interesting. This, I think, is why replication has been held up as the savior of psychology; our training makes us think that saving the phenomena will save the science. But that's not really how it works.
Similarly, the core of modern biology is the theory of evolution. When creationists say ‘we can’t see how a bacterial flagellum which rotates like an outboard motor could possibly have evolved, it’s irreducibly complex’ (e.g. Dembski 2002), biologists are entitled to say ‘we have evidence that lots and lots of other things have evolved. Let’s see if we can figure out how the flagellum did it, and in the meantime, we’re going to operate on the assumption that it did evolve until we have strong evidence to the contrary’. The resulting theory driven empirical work then happily led to a coherent evolutionary story for the flagellum (e.g. Musgrave 2004).
Psychology has many individual theories describing isolated phenomena but no core theory of behavior to guide our research, no analog to the theories of relativity or evolution. This is beginning to cost the discipline. Recently Bem (2011) published a series of experiments purporting to demonstrate evidence of precognition. Bem took several standard psychological experiments and reversed the temporal ordering of the elements. Analyses showed a series of statistically significant effects that suggested that events in the near future were affecting earlier performance. For example, he showed participants a list of words then tested their free recall. After this test, he trained the participants on a subset of the words, and showed that there was improved recall of those words, even though the training had come last. Because he followed the rules of experimental design and had statistically significant results, the Journal of Personality & Social Psychology was unable to find a reason to reject the paper. The editors only noted that “the reported findings conflict with our own beliefs about causality and that we find them extremely puzzling” (Judd & Gawronski 2011: 406, emphasis ours). Note that the cited conflict was with their beliefs about causality, and not, for example, the laws of physics and what they have to say about time travel. This should have been an opportunity for Bem to discuss problems with the standard methods and analyses that produced these physically impossible results (the approach taken in a companion paper by Wagenmakers, Wetzels, Borsboom & van der Maas 2011). Instead, his discussion was framed in terms of a loose reading of quantum physics and an appeal to psychologists to keep an open mind. The paper simply described what had happened, without any real attempt to explain how it had happened. A failure to replicate Bem’s key effects has recently been published (Ritchie, Wiseman & French 2012), but this paper was also entirely empirical and descriptive in nature, with no reference to any underlying theory of how the world works.
Psychology needs a core theory in order to mature as a science. Theory serves a dual role in science. It allows the scientist to identify when a result is likely to be anomaly (e.g. faster-than-light neutrinos), and, more critically, it provides a guide to discovery to structure the search for explanations of novel phenomena (e.g. the bacterial flagellum). The Bem experiments demonstrate how, without a theory, psychology is unable to deal rigorously with anomalous results. This paper will discuss how an example psychological theory (James J. Gibson’s ecological approach to visual perception; Gibson 1966; 1979) has been able to guide discovery and explanation of new phenomena, specifically how people learn to produce a novel coordinated rhythmic movement. It has been able to do this because it is a theory of both the objects of perception and the ecological information that supports that perception. The theory can therefore be used to propose specific mechanisms to explain a given behaviour, rather than simply providing some terms to describe that behavior. We will suggest that the successes of this approach in the area of perceptually guided action stands as a clear model of what a truly theory-driven psychology could achieve.
Take social priming; what a mess. Specific results stand, fall and stand again as people run replication attempts, and the reason for the mess is that social priming is a poorly thought out paradigm to begin with. Results will sometimes replicate; statistically even a broken clock will be 'working' 5% of the time. This is just the nature of the statistical game we play. But none of these replications or failures of replication do anything to make social priming a more scientifically rigorous discipline. (An upcoming paper in Perspectives in Psychological Science seems to agree with me here; HT to Rolf Zwaan via this recent related post).
But even in psychology there is good theory driven work. Golonka & Wilson (2012) use coordinated rhythmic movement as a model. When I run coordination experiments, I never worry about being able to replicate the basic pattern of in-phase being easier than anti-phase, and thanks to the theory based model I use I know why I can count on it. I also discussed some work by Geoff Bingham to contrast with to the typical small-effect psychology that is targeted for replication. When Geoff figured out metric shape perception, after years of hypothesis driven work ruling out ideas, he got to a point where he could make it work, or not work. There is no question about it only working sometimes, when the light is right and the statistical stars align. The moral of the story is that we know focusing on theory works.
Look, go forth and replicate. Your success or failure will tell you something about whether the theory your experiments come from is any good because replicability is an effect of good science. But just don't think that replications can save psychology; only theory can do that, and so it's time to start thinking about what that might look like (Wilson & Golonka, 2013 has our thoughts).
References
Golonka, S., & Wilson, A. D. (2012). Gibson’s ecological approach - a model for the benefits of a theory driven psychology. Avant, 3(2), 40-53. Download
Wilson, A. D., & Golonka, S. (2013). Embodied Cognition is Not What you Think It Is. Frontiers in Psychology, 4. Download
