Causes for the Persistence of Impact Factor Mania

Hypercompetition for funding and jobs.In the United States, success rates for grant applications are at historic lows, and the imbalance between applicants and academic positions for scientists has reached a crisis point (7–9). Review panels are routinely asked to discriminate between grant applications that seem equally meritorious, and search or promotion committees likewise must decide which of many highly qualified candidates are most deserving of hire or professional advancement. In this environment, quantitative bibliometric tools seem to offer an objective, measurable way to ascertain researcher performance (10).

Paucity of objective measures of the importance of scientific work.There are presently more than 25,000 journals and over one million new articles published each year. The reliance of scientists and granting agencies on the impact factor as a measure of scientific quality is rooted in the need for quantitative measures of importance in science (11). In this context, the impact factor of the journal where work is published is often used a surrogate marker of excellence, despite the fact that citation frequency for the journal does not predict the citation frequency for individual papers (12–14). Although we have proposed a scheme to assess the importance of scientific work (11), our approach remains subjective and does not readily lend itself to quantitative analysis. Hence, impact factor mania is driven by the absence of other readily available parameters to judge the importance of scientific articles.

Hyperspecialization of science.As science has succeeded as an enterprise, it has become ever more specialized (15). The increasing specialization of science has made it increasingly difficult for scientists working in different fields to understand each other’s work. Relying on publication in highly selective journals as a surrogate measure of quality provides a convenient, if intellectually lazy, alternative to attempting to read and understand a paper outside one’s specialty. At least one author has attributed impact factor mania to laziness on the part of senior scientists and bureaucrats (5).

Benefits to selected journals.The high-impact journals produce a highly desirable commodity when a paper is accepted for publication. These journals create a scarcity of publication slots by rejecting the overwhelming majority of submissions, resulting in a monopoly that restricts access and corners the market for excellent articles. This in turn creates a sense of exclusivity that encourages more submissions, as scientists equate low acceptance rates with greater merit (16). Thus, high-impact journals are perversely rewarded for rejecting manuscripts. A central error made by many scientists is in assuming that exclusivity is an indicator of exceptional quality, an assumption that is not always justified (17). Nevertheless, the impact factor of a journal is a better predictor of subsequent citation frequency than article quality (18). Nature, Science, and Cell collectively account for 24% of the 2,100 most-cited articles across all scientific fields (19).

Benefits to scientists.Publication in prestigious journals has a disproportionately high payoff that translates into a greater likelihood of academic success. This has been referred to as acceptance to “The Golden Club,” with rewards in the form of jobs, grants, and visibility (20). There is also a tendency for successful publishing in highly selective journals to beget more success, perhaps in part because greater visibility attracts capable and ambitious trainees and also because editors and reviewers may apply more liberal standards in assessing submissions from authors who publish regularly in such venues (21). This helps to perpetuate the cycle. The tendency for the rich to get richer and the poor to get poorer was designated the “Matthew effect” by the sociologist Robert K. Merton (22). The economic implications of the Matthew effect imply that prominent scientists have an interest in the perpetuation of impact factor mania.

National endorsements.Some nations have developed schemes to rate the productivity of their scientists, depending on the impact factor of the journals in which their papers are published. Brazil has established a “Qualis” scale based on the average impact factor of their publications, which is used to grade students and faculty (23). China has been reported to offer monetary rewards to editors who increase the impact factors of their journals (24), and China, Turkey, and South Korea offer cash bonuses to scientists who publish in journals with high impact factors (25).

Prestige by association.The cachet of the most highly selective journals is readily transferred to its contents. Although one may resent the disproportionate influence of the most prestigious journals, one cannot deny that many important and high-quality research articles are published there, and articles are judged by the company they keep. Of course, there are also many articles published in less prestigious venues of equal or greater quality, and these may be unfairly neglected.

We conclude that impact factor mania persists because it is useful to certain scientists, certain journals, and the bureaucracy of science, particularly in certain nations. However, benefits to individuals and special interest groups do not necessarily translate into an overall benefit for the scientific enterprise.

Distortions in the scientific enterprise.The greatest distortion caused by impact factor mania is the conflation of paper quality with publication venue rather than the actual content of the paper. This encourages the branding of science and scientists with journals in which work is published. The likelihood of obtaining funding, academic promotion, selection for awards, and election to honorific societies becomes dependent, at least in part, on publication venue. The distorted value system has become self-sustaining, with the editors of high impact journals commanding far great power and influence than is healthy for the scientific enterprise.

The full impact of a scientific discovery may not be apparent for many years.The journal impact factor is calculated by dividing the number of total citations over the number of articles published in the previous 2 years. However, the impact of truly important and novel findings often takes more than 2 years to be fully realized (26). Thus, the central premise of measuring the importance of scientific journals or papers on the basis of impact factor is fundamentally flawed. An unanticipated consequence of the way the impact factor is calculated is that the most novel and innovative research may be less attractive to some journals because such work, by its very nature, will have its major influence at a time when it does not contribute to the calculation. Journals that reject manuscripts because they cannot immediately appreciate the importance of the work may be turning down the next penicillin, PCR, or tricarboxylic acid cycle (27). Problems with short-sighted peer review are not limited to journals; an analysis at the National Institute of General Medical Sciences (NIGMS) has concluded that current grant review procedures have an error rate of approximately 30% and result in many meritorious applications going unfunded (28).

Emphasis on high impact means that many meritorious studies will not be funded or published.Asking scientists to conduct only high-impact research creates a strong bias that discourages high-risk research and reduces the likelihood of unexpected breakthrough discoveries. We suggest an analogy between scientists and foraging predators, which employ random Brownian movement when prey are abundant and use more complex yet still random Lévy flights when searching for sparser prey (29, 30). Scientists in search of new discoveries cannot hope to investigate the full scope of nature if they must limit their searches to areas that a consensus of other scientists judges to be important. As Vannevar Bush observed nearly 70 years ago, “Basic research is performed without thought of practical ends … Many of the most important discoveries have come as a result of experiments undertaken with very different purposes in mind … It is certain that important and highly useful discoveries will result from some fraction of the undertakings in basic science, but the results of any one particular investigation cannot be predicted with accuracy” (31).

Journal impact factor and individual article citation rate are poorly correlated.The majority of a journal’s impact factor is determined by a minority of its papers that are very highly cited (13). It is well known that the impact factor does not necessarily predict the citation prospects of other papers published in the same journal (12, 14, 18, 32), and the relationship between impact factor and citation rate may be weakening (33). Furthermore, when the impact of science is analyzed by other criteria, there is at best a very weak positive correlation with the impact factor of the journal in which the work was published, whereas journal rank by impact factor correlates inversely with the reliability of research findings (14, 34–36).

Citation rate is an imperfect indicator of scientific quality and importance.Citation rate is highly dependent on factors such as field size and name recognition (37, 38). Studies suggest that journal impact factor may correlate poorly with the true value of research to a field (39, 40). Review articles and descriptions of new methods tend to be disproportionately cited (41, 42). Moreover, an emphasis on citation rate as a measure of impact perversely discourages research in neglected fields that are deserving of greater study.

Delays in the communication of scientific findings.The disproportionate rewards associated with publication in high-impact journals create compelling incentives for investigators to have their work published in such journals. Authors typically submit their work to multiple journals in serial fashion hoping for acceptance by a journal with the highest possible impact factor. This effort can consume considerable time and resources since investigators often respond to reviewers by performing additional experiments in an effort to convince journals to accept their papers. The multiple submissions also consume reviewers’ and editors’ time and delay the public disclosure of scientific knowledge. Publication delay slows the pace of science and can directly affect society when the manuscript contains information important for drug and vaccine development, public health, or medical care. For an investigator, the time spent in identifying a high-impact journal can result in a loss in citations (4). These effects translate into major inefficiencies in the dissemination of scientific information.

Creation of perverse incentives.The pressure for publication in high-impact journals may contribute to reduced reliability of the scientific literature. Articles retracted due to data falsification or fabrication are disproportionately found among high-impact journals (36), and errors also appear to be more frequently encountered (34, 36). The most selective journals demand clean stories and immaculate data, which seldom match the reality of laboratory investigation, where experiments can produce messy results. Hence, some investigators may be tempted to cut corners or manipulate data in an effort to benefit from the disproportionate rewards associated with publishing in prestigious journals. Impact factor mania also creates great pressures on journals to raise their impact factors, which can lead to practices that range from gaming the system to outright editorial misconduct. Some journals have been reported to pressure authors to include more citations to their own journal in order to artificially increase the journal’s impact factor. Recently a scheme was uncovered in which three journals conspired to cite each other’s papers in a mutual effort to increase their impact factors, a conspiracy that was blamed in part by the use of journal impact factor by the Brazilian funding agency to judge the quality of articles (43).

DORA.The American Society for Cellular Biology, in concert with journal editors, scientific institutions and prominent scientists, organized the San Francisco Declaration on Research Assessment (DORA) in December 2012 to combat the use of the journal impact factor to assess the work of individual scientists (51). Some prominent scientists in the biomedical research community have begun to speak out in support of DORA. Nobel Prize winner Harold Varmus is advocating the redesign of curriculum vitae to emphasize contributions instead of specific publication (52). Dr. Varmus has decried a “flawed values system” in science and laments the fact that “researchers feel they will win funding only if they publish in top journals” (52).

A boycott of high-impact journals.Nobelist and eLife Editor-in-Chief Randy Schekman has recently criticized the monopoly of what he calls “luxury journals” in an editorial published in the British newspaper The Guardian. Schekman has vowed not to publish hereafter in Nature, Cell, and Science, stating that the disproportionate rewards associated with publishing in those journals distorts science in a manner akin to the effects of large bonuses on the banking industry (53). Although such efforts are well-intentioned, we are skeptical that boycotting the impact factor or the “luxury journals” will be effective, because the economics of current science dictate that scientists who succeed in publishing in such journals will accrue disproportionate rewards. This will continue to be an irresistible attraction. Even if the journal impact factor were to disappear tomorrow, the prestige associated with certain journals would persist, and authors would continue to try to publish there. Most scientists do not actually know the impact factor of individual journals—only that publication in such journals is highly sought after and respected. For instance, it is not widely known that Science is actually only ranked 20th among all journals in impact factor, lower than journals such as CA—A Cancer Journal for Clinicians and Advanced Physics. Similarly, we fear that boycotts of specific prestigious journals may hurt the trainees of those laboratories by depriving them of high-visibility venues for their work. In lieu of Science, Nature, or Cell, Schekman has recommended that authors submit their best papers to eLife. However, the managing executive editor has described eLife as “a very selective journal,” further noting that “it’s likely that the rejection rate will be quite high” (54). As long as a critical mass of scientists continues to submit their best work to highly selective journals, impact factor mania, or its equivalent, is likely to persist.

Reform review criteria for funding and promotion.We strongly advocate the assessment of research quality by peers in an individual’s field rather than using simplistic measures based on the quantity of papers and prestige of the publication venue. Academics should resist attempts by institutions to use journal impact factor in promotion or tenure decisions. When evaluating the performance of scientists, a focus should be on contributions rather than publication venue. Academic administrators should be educated that impact factor is an inadequate measure of individual achievement (55) and should be informed of the DORA principles. Individual scientists may sign to support the DORA initiative at http://am.ascb.org/dora/ (56).

Consider the use of diverse metrics.If metrics are to be used for judging individual scientists and their projects, review committees and administrators should consider a diverse range of parameters. Although we do not suggest that the quality of scientists’ work can be reduced to a single number, we do note that alternative citation metrics, such as the H-index and eigenfactor, are probably better measures of scientific impact than the impact factor. Evaluations that employ diverse metrics are likely to provide more insight than assessments based on single criteria.

Increase interdisciplinary interactions.The specialization of science has made it increasingly difficult for scientists to understand research outside their own field (15). However, there is no substitute for actually reading and understanding a scientific article. This requires that scientists who are asked to judge the productivity of another scientist working in a different field must acquire a working familiarity with that field. Increasing opportunities for interaction between researchers from different fields in training programs and at seminars and meetings will help to improve the quality of research assessment as well as stimulate transdisciplinary or multidisciplinary research.

Encourage elite journals to become less exclusive.Letters of rejection often state that “We regret that we receive many more meritorious submissions than we can publish.” Why should not the elite journals expand to accommodate all meritorious articles? Artificial restrictions on journal size serve to perpetuate the current wasteful system that requires authors to cascade serial submissions from one journal to another.

Address current imbalances in research funding and the scientific workforce.As long as there is an unreasonable level of competition for grants and jobs, methods to compare scientists with one another are going to be utilized. Scientists must work with policymakers to alleviate the current shortages of research funding and job opportunities that have created the current crisis situation.

A return to essential scientific values.Ultimately, the only cure for impact factor mania must come from scientists themselves. If scientists fail to curb their current impact factor mania, they will pass onto their trainees a distorted value system that rewards the acquisition of publications in exclusive journals rather than the acquisition of knowledge and one that promotes an obsession with individual career success over service to society. Moreover, the current insistence on funding only high-impact projects is skewing the focus of research efforts and increasing the likelihood that important avenues of investigation will be overlooked. Scientists must return to essential scientific values that place an emphasis on research quality and reproducibility, the advancement of knowledge, and service to society over the accumulation of publications in prestigious journals.