Robert Kail, the outgoing editor, had this to say about the ideal Psychological Science paper:
...the ideal Psychological Science manuscript is difficult to define, but easily recognized — the topic is fundamental to the field, the design is elegant, and the findings are breathtaking.There are a few problems here; 'breathtaking' results have the tendency to be wrong, for example, and while I'll all for elegant design, sometimes, to make a breath taking claim, you need to run those 4 control conditions. But my main problem is less with these criteria and more with the papers that apparently meet them.
I've talked a lot about 'embodied' cognition research here, work I think doesn't deserve the name and is typically full of flaws anyway. My two main examples have been how thinking about the future or the past makes you sway forward or backwards and how leaning to the left makes the Eiffel Tower seem smaller. Both of these (quite flawed) papers appeared to great fanfare in Psychological Science. In fact, quite a lot of this work appears in Psych Science, to the point where I nearly made that feature the first thing to check on an embodied cognition paper to know it's not the real deal. So I don't have a lot of confidence in the process of selection at Psych Science to begin with.
This week it got even worse. Kille et al (2012) published a study where they had people sitting on wobbly chairs. They then asked people to rate the stability of some celebrity relationships, and to rate how important they felt several qualities were to a relationship, including several related to stability. People in the wobbly chair condition rated celebrity relationships as less stable than people in the non-wobbly chair condition (effect size (partial eta squared) of .15). They also preferred stability related traits in relationships more (effect size of .1). In both cases, the mean difference was about half a point on a 9 point scale. The authors argue that 'embodiment motivates mate selection preferences'.
When I look to Kail's criteria, I wonder how these papers get published:
Embodied cognition is a fundamental topic, but these papers aren't taking it seriously.
The design was elegant, in that they showed a simple pattern of results. But there was no serious discussion of why these results should have happened. There was instead a brief note at the end that said
Indeed, we suspect that one reason cognition may become embodied is to ensure that one's needs - which may arise from physical states - are met through goal pursuit.Following this logic through, they are claiming that we have evolved to try and resolve temporary postural instability by selecting more stable mates. This makes so little sense when you spell it out, but it sounds so exciting the way they said it. I guess that's why they said it their way rather than mine.
Are the findings breathtaking? Well, I did get a little breathless after reading this but not in that way Kail was intending, I don't think (unless he meant from simultaneously laughing and banging my head against the table for 20 straight minutes). But to be serious, these results are meaningless because, like all this research, there is no clear effort to establish the mechanism by which the postural manipulation and the judgment data are connected to one another. Why would a minor and easily corrected postural instability affect the important task of mate selection (or even the judgment task they are using as a proxy for mate selection)? With no clear task analysis of how this extended, embodied cognitive system is formed and why, we cannot interpret these results.
Finally, this is just more 'small effect size' research, which I've previously argued is not as clever as people think it is. There's a perception in psychology that squeezing out a significant result means you ran a clever experiment and managed to defeat the complexity that is human cognition. It's actually a hint that you have asked the wrong question, I think, and I think this because when you manage to ask the right question, you go from tiny effects to unambiguous results. Small effect sizes are not compulsory for psychology; if we get better at asking our questions, we will get clearer answers.
So how did this get published in Psych Science? It's a good question, and one I don't actually know the answer to. If I had to guess, I'd say it's because it's yet another sexy little result in 'embodied' cognition, and sexy sells. But sexy is hurting our discipline - these tiny, astonishing effects aren't being replicated, are probably wrong, and are, to my mind, fairly average science anyway.
What about all the other papers Psych Science publishes?
I pick on the 'embodied' stuff because I know what to look for. There are good papers in Psych Science. Karen Adolph, my favorite developmental psychologist, recently published some great work about the emergence of walking in infants in this journal, and I'm sure some of the other papers not in my field are good too. But my first reaction to seeing the paper wasn't 'hey Karen, great job', it was 'oh Karen, why did you waste that awesome paper on Psych Science?'. Based on interactions on Twitter I'm not the only person who has come to view what is supposed to be our flagship journal with pity and contempt, even when what it publishes is good.
I'm going to guess that isn't what Kail wanted for the journal, and not what the incoming editor Eric Eich wants either. But until the journal stops publishing all this poorly conceived 'sexy' work, I will never wish Psych Science as a home for any of my papers.
References
Adolph, K. E., Cole, W. G., Komati, M., Garciaguirre, J. S., Badaly, D., Lingeman, J. M., Chan, G. L. Y, & Sotsky, R. B. (2012). How do you learn to walk? Thousands of steps and dozens of falls per day. Psychological Science, 23(11), 1387-1394. DownloadKille, D., Forest, A., & Wood, J. (2012). Tall, Dark, and Stable: Embodiment Motivates Mate Selection Preferences Psychological Science DOI: 10.1177/0956797612457392
