I have a new piece up at the Daily Dot about the Reproducibility Project and why psychology isn’t doomed.
The Internet loves sharing psychology studies that affirm lived experiences, and even the tiniest ticks of everyday people. But somewhere in the mix of all those articles and listicles about introverts, extroverts, or habits that “make people successful,” a debate still lingers: Is psychology a “real science?” It’s a question that doesn’t seem to be going away anytime soon. Last week, the Reproducibility Project, an effort by psychology researchers to redo older studies to see if their findings hold up, discovered that only 36 of the 100 studies it tested reproduced the same results.
Of course, many outlets exaggerated these findings, referring to the re-tested studies (or to psychology in general) as “failed” or “proven wrong.” However, as Benedict Carey explains in the New York Times, the project “found no evidence of fraud or that any original study was definitively false. Rather, it concluded that the evidence for most published findings was not nearly as strong as originally claimed.”
But “many psychology studies are not as strong as originally claimed” isn’t as interesting of a headline. So, what’s really going on with psychology research? Should we be worried? Is psychology a “hopeless case?” It’s true that there’s a problem, but the problem isn’t that psychology is nonscientific or that researchers are designing studies poorly (though some of them probably are). The problem is a combination of two things: Statistical methods that aren’t as strong as we thought and a lack of interest in negative findings.
A negative finding happens when a researcher carries out a study and does not find the effect they expected or hoped to find. For instance, suppose you want to find out whether or not drinking coffee every morning affects one’s overall satisfaction with their life. You predict that it does. You take a group of participants and randomly assign half of them to drink coffee every morning for a month, and the other half to abstain from coffee for a month. At the start and at the end of that month, you give them a questionnaire that assesses how satisfied each participant is with their life.
If you find that drinking coffee every day makes no difference when it comes to one’s life satisfaction, you have a negative result. Your hypothesis was not confirmed.
This result isn’t very interesting, as research goes. It’s much less likely to be published than a study with positive results—one that shows that drinking coffee does impact life satisfaction. Most likely, these results will end up gathering figurative dust on the researcher’s computer, and nobody outside of the lab will ever hear about them. Psychologists call this the file-drawer effect.
Read the rest here.
