How can impact factors be improved?

Eugene Garfield

British Medical Journal 313:411-3, 1996

Counting references to rank the use of scientific journals was reported as early as 1927 by Gross and Gross.¹ In 1955 I suggested that reference counting could measure "impact,"² but the term "impact factor" was not used until the publication of the 1961 Science Citation Index (SCI) in 1963. This led to a byproduct, Journal Citation Reports (JCR) and a burgeoning literature using bibliometric measures. From 1975 to 1989, CR appeared as supplementary volumes in the annual SCI. From 1990-4, they have appeared in microfiche, and in 1995 a CD ROM edition was launched.

Calculation of current impact factors

The most used data in the JCR are impact factors— ratios obtained from dividing citations received in one year by papers published in the two previous years. Thus, the 1995 impact factor counts the citations in 1995 journal issues to "items" published in 1993 and 1994. I say "items" advisedly. There are a dozen major categories of editorial matter. JCR's impact calculations are based on original research and review articles, as well as notes. Letters of the type published in the BMJ and the Lancet are not included in the publication count. The vast majority of research journals do not have such extensive correspondence sections. The effects of these differences in calculating journal impact can be considerable.³,⁴

Absolute citation counts

The ubiquitous and sometimes misplaced use of journal impact factors for evaluation has caused considerable controversy. They are probably the most widely used of a all citation based measures. They were invented to permit

reasonable comparison between large and small preferentially journals. Absolute citation counts preferentially give highest rank to the largest or the oldest journals. For example, in 1994, articles published in the BMJ regardless of age were cited 37 600 times. Of these, 5800 citations— -- about 15%— were to items published in 1992 and 1993.

Table 1 shows absolute citation counts for nine English language clinical journals as well as the Journal of Biological Chemistry, which is included to emphasise the difference between absolute and relative citation of large journals Table 2 lists these same journals ranked by 1994 impact It is important to note that of the thousands of journals published and cited in SCI only 337 achieved a current impact higher than 3.0. The SCI processes over 3300 source journals and cites thousands more, but it does no t include as sources hundreds of low impact applied and clinical journals. This has been a source of frustration to editors from the Third World. They often ask-how they can improve impact so as to warrant inclusion.

Impact of review journals

It should be apparent from reviewing the two tables that the ranking relationship between quality and citation is not absolute. The Journal of Biological Chemistry is one of the most cited journals in the history of science but it is also among the largest. Publishing over 4000 research manuscripts a year (as do other high volume journals like Physical Review) inevitably leads to considerable variation in quality and impact of individual articles. As a consequence, while these journals publish many papers which become "citation classics," their current impact may not be as high as some smaller journals, especially certain review journals. Indeed, one of the highest impact journals is Annual Review of Biochemistry with a current impact of 42.2. But as the data for Annual Review of Medicine show, this is not an absolute rule. In general, a journal is well advised to publish authoritative review articles if it would increase its impact. Since there are over 40 000 review articles published each year, not all will achieve high impact. Again, selection of reviews about active research fronts is important, as is their timing. Controversial topics may increase impact. A non-medical example is "cold fusion" by Fleischman and Pons,⁵ which has been cited over 500 times. A recent medical example could be the "Concorde study," already cited in 150 papers.⁶ But hundreds of halfbaked controversial ideas are essentially ignored.

Nothing will replace the judgment necessary for editors m select putative citation classics and to reject trivial or outlandish papers. Nevertheless, most reputable journals have at one time or another rejected papers which proved to be blockbusters. In retrospect, we should congratulate editors who publish controversial ideas such as that of Barry Marshall concerning Helicobacter pylori and peptic ulcers.⁷ That paper was well on its way to citation classic status when I nominated Marshall for the John Scott Award of Philadelphia— several years before he received the Albert Lasker award this year.

Method papers

It is widely believed that method papers are cited more than the average and thus increase journal impact. How lovely it would be if every method proved to be another Lowrey method,⁸ cited over 8000 times in 1994 and over 250 000 in its lifetime. But the fact is that method journals do not achieve extraordinary impact since the vast majority of their papers, like clinical tests, are not unusual.

An editor could select authors on the basis of past performance. By checking their citation histories, one could undoubted increase the probability of publishing papers with higher potential impact. Some editors do this instinctively, especially when publishing the first few issues of a new specialty journal. in most cases, the most-cited papers for newer journals appear in the first

Over the years, the increase in multiauthored papers has been apparent. This is matched by an increase in multinational and multi-institutional clinical and epidemiological studies. At the institute for Scientific Information (ISI), unpublished studies support the notion that these papers produce greater impact. ISI's Science Watch has regularly reported on the most-cited current papers in medicine. There is fierce competition among editors to publish these "hot papers." These undoubtedly contribute to increased current impact. But what about long term impact?

Cumulative impact factors

No substitute for judgment

In spite of dozens of presentations by myself and others, there continues to be a certain mystique about journal selection for Current Contents, the Science Citation Index, and the Social Sciences Citation Index. My 1990 essay in Current Contents is still sent to those making such enquiries.¹⁰

Of the 4500 journals covered by SCI and SSCI, probably 3000 can be described as biomedically related. Of these, 500 account for 50% of what is published and 75% of what is cited. Of the 3300 covered in Medline, hundreds of low impact journals are not included by ISI for similar reasons -space and economics.

Other factors in journal evaluation

New journals continue to appear each year and must be evaluated as early as possible. But even Nature in its periodic reviews of new journals requires the passage of time before accepting journals to be reviewed. An experienced evaluator takes into account timing, format, subject matter, past performance, and other indicators such as internationally. The first issue of many journals is full of hope, but they soon exhaust the backlog of material needed to insure continued, timely publication. Clearly a society publisher with years of experience does not launch a journal without a long term commitment— -- and its editorial standards will be well known. Inexperienced publishers often do not live up to their rosy expectations. The inclusion of abstracts and complete author, street, and email addresses are but a few artefacts that are factored in the judgment of minimum quality. ISI may also ask how well a particular specialty or country is represented in its coverage. And, after a few years of history, all other criteria being equal, one can look at a journal's impact.

This article has focused on journal impact factors and their role in what Stephen Lock described as "journalology."¹¹ Like individual authors, a variety of indicators can be used to judge journals in a current or historical sense. Impact numbers are probably less important than the rankings which are obtained. Often there are only slight quantitative differences involved. The literature is replete with recommendations for corrective factors that should be considered, but in the final analysis subjective peer judgment is essential.

Caution in use of impacts as surrogates