The replication crisis extends well beyond the social sciences. Who will save the biomedical scientists?
By Kate Wheeling
Stem cells viewed on a computer screen. (Photo: Spencer Platt/Getty Images)
On Sunday night, scientific research was on the receiving end of a typically biting and brilliant John Oliver roast. “There is a lot of bullshit currently masquerading as science,” the Last Week Tonight host hollered, skewering numerous (and often contradictory) studies on the benefits and dangers of everything from coffee and chocolate to dehydrated driving and animal affection—and the media that propagates such studies as fact before they are able to be reproduced and confirmed. Pacific Standard has covered the replication crisis in the social sciences at length (see Jerry Adler’s “The Reformation: Can Social Scientists Save Themselves?”). But the same problems plague biomedical research as well.
Basic scientific findings are exceedingly difficult to reproduce. The reasons for that are many and varied — shoddy methods, incomprehensible protocols in papers, and, too often, misconduct — but the end result is the same: A 2015 study in PLoS Biology suggested that roughly $28 billion a year is spent in the United States alone on research that can’t be replicated.
In California, residents have a unique stake in biomedical research, in particular the amount of funding that’s wasted on irreproducible research.In 2004, Californians voted to approve Proposition 71, which created the California Institute for Regenerative Medicine, an agency that funds stem cell research through the proceeds from bond sales. The proposition passed in the wake of George W. Bush-era restrictions on human embryonic stem cell research, and the agency was expected to use $3 billion in bond sales — roughly $295 million a year for more than a decade—to fund stem cell research that wouldn’t have received federal funding.
“About half of California’s families have a child or adult who has suffered or will suffer from a serious, often critical or terminal, medical condition that could potentially be treated or cured with stem cell therapies,” Proposition 71 states.
Roughly $28 billion a year is spent in the United States alone on research that can’t be replicated.
Given its birth story, it’s no surprise that the Institute for Regenerative Medicine is under a lot of pressure to produce results that benefit the public. Alas, science is a slow process, and, so far, no cures have materialized; in 2012, an Institute of Medicine panel called the agency’s translation goals for its research “unrealistic.” Now, the agency is trying to change that by working more closely with private industry, which could also, at least indirectly, encourage more replication studies, according to Elizabeth Iorns, a former cancer researcher and the founder of Science Exchange.
It was the pharmaceutical industry that first brought the biomedical replication crisis to Iorns’ attention several years ago. In 2011, she founded Science Exchange, a company that contracts with academic institutions, government facilities, and research organizations to provide outsourced research services to those who need them. Soon after, she noticed that pharmaceutical companies and venture capitalists frequently came to Science Exchange to request services to replicate scientific studies. Iorns learned that the pharmaceutical industry pours money into replicating early research, but, more often than not, it can’t.
Amgen’s cancer researchers were only able to replicate 11 percent of the “landmark” studies they selected to pursue over a 10-year period; a 2011 analysis by researchers with pharmaceutical giant Bayer found that the company’s in-house findings only lined up with the original results a quarter of the time, at most.
But even the failed replication data was valuable to Bayer. The analysis revealed that, when Bayer scientists were able to reproduce a result in a direct replication experiment, it tended to translate well into clinical applications; in other words, reproducibility seemed to be a useful marker of clinical potential. “What they showed is that this is really a pathway to quickly identify the most robust results, that can then be built upon effectively and translated into the clinic,” Iorns says.
That’s good news for CIRM, which recently created the Accelerated Therapies Public Private Partnership award — an initiative that provides funds for a partnership with private industry to streamline the translation of basic science to application. The award will allow private industry to pick and choose the most promising research avenues out of CIRM, and pursue those for potential application. CIRM will put up a $75 million loan over five years, and businesses will be expected to match that investment.
The award could (indirectly at least) spur replication studies, given that the best way to know which findings might translate to real world application may be to determine which findings replicate. Many scientists and industry personnel are interested in using data mining techniques to identify markers of reproducibility, but until more replication studies are conducted, there’s no data set for them to mine. Which means, at present, there’s no short cut to figure out which studies are most likely to replicate without doing the work.
Right now, funding agencies like CIRM are rarely interested in bankrolling replication studies, and most scientific journals are not interested in publishing such results.
“People are funded to do research so that we can actually find cures for diseases,” Iorns says. “If those results are not reproducible, and can’t be built upon by the industry that develops the drugs, then that undermines the need or the purpose of doing any of the research to begin with.”
||
