Is the way universities and government agencies evaluate scientific work killing science? That’s the question in the air as scientists argue over DORA—not the cute explorer with the backpack and monkey friend, but the Declaration On Research Assessment. DORA is a manifesto issued by a group. Its primary target is a widely-used rating system for scientific journals whose name sounds like the title of a post-apocalyptic thriller: the Impact Factor.
The Journal Impact Factor is an annual score given to scientific journals by Thomson Reuters, a private company with a long history of making tools to track and evaluate scientific publications. The Impact Factor was conceived as a way to measure the overall quality of a scholarly journal, but the DORA authors argue that the Impact Factor is being abused to evaluate individual scientists. The senior scientists who populate committees that make decisions about hiring, funding, and promotion are supposed to be experts capable of independently judging an applicant’s scientific articles. But, the argument goes, instead of making an independent judgment, committees are using the Impact Factor as a shortcut, evaluating applicants by rating of the journals they’ve published in, and not by the actual content of their science. DORA is calling on everyone—scientists, institutions, and publishers—to blunt the impact of the Impact Factor.
If scientists focus on finding the answers to fundamental questions, impact will come as a matter of course.
DORA’S SUPPORTERS ARGUE THAT the emphasis on Impact Factor has pushed scientists’ incentives in an unhealthy direction. Bruce Alberts, the recently retired editor-in-chief of the journal Science (whose Impact Factor is a whopping 31.03), wrote that “Any evaluation system in which the mere number of a researcher’s publications increases his or her score creates a strong disincentive to pursue risky and potentially groundbreaking work,” because groundbreaking work is, as a rule, non-obvious, and people doing groundbreaking work don’t always publish very much.
Not everyone is onboard with DORA’s attack on the Impact Factor. Philip Campbell, editor-in-chief of the journal Nature (whose Impact Factor is a stratospheric 38.6) declined to endorse DORA’s sweeping condemnation of the Impact Factor. Kent Anderson, publisher of the Journal of Bone & Joint Surgery (Impact Factor a respectable 3.272), wrote on his blog that the DORA authors wrongly created the impression that there is “an oppressive regime of publishers wielding the impact factor.” He suggested that “the core issue isn’t the existence of the impact factor or its utilization by journals, but its lazy adoption by academics.”
There is evidence that concerns over the Impact Factor are a symptom of deeper structural problems. Paula Stephan, an economist at Georgia State University, argues that academic science is being damaged by a dysfunctional funding system that has allowed universities to over-expand their research facilities. Universities have recklessly over-expanded because the costs and risks of that expansion are borne by others: the government agencies and state governments that pay for the expansions, as well as the low-cost, temporary graduate students and postdoctoral fellows who work for low pay in exchange for unrealistic promises of career opportunities. Stephan writes that the large expansion in U.S. government funding for biomedical research “triggered universities to hire more people and build more buildings…. Now, this biomedical machine needs increasing amounts of money to sustain itself—larger labs need more grants, which leads to lower success rates, with calls for more funding.” The result is more people are competing for grants, jobs, and tenure, leading to overburdened scientists on hiring and funding committees, who need some kind of short cut to help them wade through the piles of applications—like the Impact Factor.
That the problem lies with the behavior of senior scientists on committees is the argument made by Mark Johnston, a DORA supporter and editor-in-chief of the journal Genetics (Impact Factor a solid 4.389). He writes that “[w]e have met the enemy, and it is us.” Johnston says that if senior scientists decide to stop emphasizing the Impact Factor, the culture will change for the better. This will happen when scientists take “responsibility for setting the standards of science—when we’re serving on grant review panels, when we’re sitting on hiring and promotion committees, in conversations with our colleagues in the hallway, and when we’re in our labs considering journals for our next submission.”
Harvard’s chair of Systems Biology, Mark Kirschner, has gone even further, urging scientists to discard the very idea of Impact Factor as a way to measure cutting-edge science. He writes that “one may be able to recognize good science as it happens, but significant science can only be viewed in the rearview mirror.” Today’s misplaced emphasis leads to sloppy science that is manipulated to “inflate its (narrowly defined) impact.” If scientists focus on finding the answers to fundamental questions, impact will come as a matter of course.
Regardless of the merits of the Impact Factor as a tool, it’s clear that the large international expansion of biomedical science over the last 10 years has made the competition for jobs and funding much more fierce; that increased competition has surely changed the incentive structure. It is unlikely that this incentive structure can be changed by the voluntary actions of senior scientists or scientific journals, because money is at stake. As Paula Stephan writes, “[s]cientists may portray themselves as not being motivated by money, but they and the institutions where they work respond in spades to financial opportunities.”
