When scientists tried to reproduce the results of 100 psychology studies a few years ago, they came to an alarming conclusion: Fewer than half of the studies could be replicated, suggesting the field might be rife with flawed knowledge about human behavior. Now, a few of those same scientists—along with some new colleagues—have taken stock of the field again, by trying to reproduce 21 studies recently published in two of science’s top journals, Science and Nature.
Their results show that there’s been some good progress in making social science studies more transparent and reproducible. The international team of social scientists surveyed journals that publish psychology research and found that many more journals now have transparency policies than in 2013. The team also set up a betting market early on in their reproduction effort. Other scientists could put money on which studies would be successfully replicated. That market showed that scientists are pretty good at judging which papers are sound.
The bad news is that this latest reproduction effort still came to a low reproducibility rate. This time, 13 out of 21 studies held up under redo efforts. In addition, among those successful checks, the results tended not to be as dramatic in the replication as they were originally. The “effect size,” as scientists call it, was, on average, 25 percent lower.
The study’s results are a reminder that, until someone shows they can reproduce the results, even studies published in prestigious journals should be “interpreted very cautiously,” Magnus Johannesson, a professor at the Stockholm School of Economics who worked on this recent replication effort, said in a statement.
Yet the results are also a sign of science’s trustworthiness, as another member of the replication team told the Washington Post. “The reason to trust science is because science doesn’t trust itself,” said Brian Nosek, a psychologist with the University of Virginia who has been active in social science reproduction efforts for the past several years. “We are constantly questioning the basis of our claims and the methods we use to test those claims. That’s why science is so credible.”
