Psychology's Real Replication Problem: Our Methods Sections

Replication has been a big topic in psychology recently as a) we've suddenly realised we need more of it and b) because there have recently been several high profile replication efforts published (e.g the Many Labs effort; see Ed Yong's summary). Last week Simone Schnall, one of the authors who's work failed to replicate in that project, wrote a blog about her experiences getting replicated. She has issues with the replication (specifically that their data had a ceiling effect on the morality measures which would obscure any differences) but one particular comment caught my eye:

Of course replications are much needed and as a field we need to make sure that our findings are reliable. But we need to keep in mind that there are human beings involved, which is what Danny Kahneman’s commentary emphasizes. Authors of the original work should be allowed to participate in the process of having their work replicated. (emphasis mine)

This idea that there is somehow a requirement to involve authors in efforts to replicate their work. This is nonsense; once you have published some work then it is fair game for replication, failure to replicate, criticism, critique and discussion. In other words, we're all allowed to science the hell out of your work any time we like. We don't need either your permission or your involvement: the only thing we (should) need is your Methods section and if you don't like this, then stop publishing your results where we can find them.

Of course, getting the original authors involved can be very productive; you can chat experimental details, make sure you have covered everything, and generally be collegial about the whole thing instead of adversarial. But there is no obligation to do this, and I'm surprised that people think there is one.

The idea seems to spring from the Kahneman commentary Schnall links too. The commentary in question is called 'A New Etiquette for Replication' and in it Kahneman proposes rules for running replications that he would eventually like to see enforced by journals and reviewers. The rules specify that a replicator must contact the original author ahead of data collection with a detailed experimental plan. The original author has some limited period to get back to them with comments about things they should do differently, the replicator has to either go with this comments or explain why they didn't, and then submit all this correspondence along with their manuscript to show they did everything in good faith. 

Kahneman's reasoning goes like this:

In the myth of perfect science, the method section of a research report always includes enough detail to permit a direct replication. Unfortunately, this seemingly reasonable demand is rarely satisfied in psychology, because behavior is easily affected by seemingly irrelevant factors. For example, experimental instructions are commonly paraphrased in the methods section, although their wording and even the font in which they are printed are known to be significant.
It is immediately obvious that a would-be replicator must learn the details of what the author did. It is less obvious, but in my view no less important, that the original author should have detailed advance knowledge of what the replicator plans to do. The hypothesis that guides this proposal is that authors will generally be more sensitive than replicators to the possible effects of small discrepancies of procedure. Rules for replication should therefore ensure a serious effort to involve the author in planning the replicator’s research.

This blows my mind. Apparently, psychology methods sections routinely do not include specifications of things we know to be significant factors in shaping the behaviours we study and the solution is make sure you talk to the original authors because they might be the only ones who know how to produce their effect. Only the authors of papers know all the magical little things they did that might have had an impact, and so therefore people trying to replicate the study must work with the original authors in order to make sure they do these things too. 

Here's a crazy alternative solution: how about we psychologists all agree to write Methods sections that would pass a first year Research Methods course and include all relevant information required for replication?

I am all for being collegial and working together to find solutions to problems, and Kahneman is not wrong to identify the fact that the big replication efforts have come off as a bit adversarial. But to set this up as a rule, as a norm of behaviour, is ridiculous. If you can't stand the replication heat, get out of the empirical kitchen because publishing your work means you think it's ready for prime time, and if other people can't make it work based on your published methods then that's your problem and not theirs. Let's remember who's responsible for what, people.