Editor’s Note: A version of this story first appeared on PSmag.com on August 24, 2015, with the headline “No, Social Science Is Not Doomed.” This edited version was published in our November/December 2015 print issue.
Science, it seems, has reached peak p-value. In research, the p-value represents the odds that your finding is a fluke. Jerry Adler wrote about the problems with p-values in our May/June 2014 issue, when he observed an intellectual crisis unfolding in the social sciences (“The Reformation”). The scientific process was failing. The main problem: “Between the laboratory and the published study lies a gap that must be bridged by the laborious process of data analysis,” Adler wrote. In that gap, scientists were performing what Adler called a “statistical sleight of hand,” manipulating data sets until results could be called significant. This fiddling—malicious or not—is known as p-hacking. And here is the real problem with p-hacking, according to Adler: “[I]f you can prove anything you want from your data, what, if anything, do you really know?”
Adler’s question has since been underscored by a series of scandals that have chipped away at the public’s understanding of—and trust in—science. But recently, FiveThirtyEight’s Christie Aschwanden reassured us that science is not, in fact, broken. “The state of our science is strong,” she wrote, “but it’s plagued by a universal problem: Science is hard.” Adler and Aschwanden touch on similar areas of the scientific process where misdeeds arise, and offer many of the same solutions: Research results should be replicated, peer review should be more transparent, and statistical methods should be more closely scrutinized. But all of these are just a partial solution. The other side of the coin is something that Adler touched on only briefly. “We are living in an age that glorifies the single study,” Nina Strohminger, then a post-doc in social psychology, told Adler. According to Aschwanden, this means that our expectations of science are fundamentally flawed. Science is not a synonym for absolute truth; it’s the process by which we search for it.
“The important lesson here is that a single analysis is not sufficient to find a definitive answer,” Aschwanden wrote. “Every result is a temporary truth, one that’s subject to change when someone else comes along to build, test and analyze anew.”
Since We Last Spoke examines the latest policy and research updates to past Pacific Standard news coverage.
For more from Pacific Standard, and to support our work, sign up for our free email newsletter and subscribe to our print magazine, where this piece originally appeared. Digital editions are available in the App Store and on Zinio and other platforms.
