The use of journal impact factors, especially in Europe, has created much anxiety. Until a few years ago, it was difficult to get anyone to pay attention to citation data. But then the dam broke and a week did not go by when one or more papers used citation analysis and impact factors. Many of these papers, however, attacked any quantitative studies as though they were synonymous with the use of journal impact factors. So from the outset, let me state the obvious. It is dangerous to use any kind of statistical data out of context. The use of journal impact factors as surrogates for actual citation performance is to be avoided, if at all possible. However, let me give you a few examples when it was not possible and was justifiable.

Several years ago, the Soros Foundation wanted to help Russian scientists in desperate need of funds. How could they, within a three-month period, decide which of 20,000 applicants should receive financial aid?

An arbitrary decision was made � if the applicant had published a paper in a journal with an impact of X, he or she would receive an immediate short-term stipend.

As you can imagine, those Russian scientists who regularly published in international high impact journals had an advantage over those who exclusively published in Russian language journals or other low impact journals. Few Russian language journals achieve better than average impacts. Those that are translated cover to cover do somewhat better, but nothing like what one might assume. Many of my friends, e.g. in the small field of marine biology, were initially hurt by this policy. Even many English language journals in that field do not achieve impacts comparable to fields like physics, biochemistry, etc. But as time passed and normal peer review procedures could be implemented, scientists even in small or low-impact fields were better accommodated.

I feel comfortable telling a group of Scandinavian scientists that it is reasonable to expect that higher impact work is generally reported in English and not in vernacular languages. But surely the Soros Foundation could not say that it would not fund Russian scientists who had exclusively published in Russian. Some Soviet physicists and chemists benefited because cover-to-cover translation journals not only promoted their work, but in some cases artificially increased their impact. The translated versions systematically cited the original Russian papers. However, it is widely believed that if your work is available in English, this alone may increase its citation. We have no significant verification of that. One recent study of the Pasteur Institute journals claims that switching from French to English had no discernible effect on impact but that would imply that only French scientists writing in English or French did the citing.¹.

Let me cite another example. Over 20 years ago, the then new director of the Italian National Research Council adopted a policy similar to that of Soros. However, he established a very low threshold of journal impact. He did not even inquire whether the individual�s work had ever been cited. He was not surprised to learn how many Italian clinical researchers had not published a paper at all, no less in low-impact journals. And many who did publish chose vernacular journals mainly supported by the pharmaceutical industry.

It is well known how politics has affected Italian research funding policies in the past, but today the situation has improved. The interest in journal impact in that country may have started with Professor Luigi Rossi-Bernardi but has accelerated. The recent publication of a book in Italian covering this phenomenon is significant.²The Weight of Academic Quality by Spiridione and Calza was discussed at a G7 Conference in Capri about two years ago. The authors clearly understand how to use citation data appropriately.

So the use of journal impact factors as surrogates can be justified in certain situations. The use in these cases is simply another way of determining that a scientist had or had not published in a journal of minimal prestige. The mere acceptance of a paper in such a journal makes a statement. Even if that paper is never cited, the fact that a respected peer-reviewed journal accepted it, means that the scientist met some minimum international standard. Now we know that this generalization is not always true. I regularly see papers that should never have been published. But we hope that the peer review process minimizes the publication of trivial, me-too salami-slice science. I will leave it to Per Seglen and others to beat the drum further on the question of using journal impacts as surrogates for real citation evaluation. (Figure 1)

Figure 1

Let me turn to the kind of quantitative and qualitative evaluation I have pursued for almost four decades. In 1961, we had to make quick decisions about which journals to cover in the Science Citation Index . ^® We had some experience with Current Contents ^® so we were especially interested in current impact. Over the past 35 years, ISI's journal coverage has grown from 600 to over 8,000 journals. Can anyone seriously question that the most-significant journals are included? Of course, there still are uncovered, as sources, small journals, which may be considered important to a particular nation or specialty. But this alone does not prevent them from being cited.

There are always new journals arising, and now, even purely electronic journals. ISI ^® is always running just to keep in place. Medline has a similar problem. But there can be little doubt thatby using ISI's own database, combined with knowledge of experts, the key journals will be identified. But the task of keeping up is not trivial. Before going on to talk about citation analysis per se, as distinct from journal impact, let me mention the subject of Citation Classics . ^® We are here to celebrate the life and work of Dr. Arne Boyum who is the author of several Citation Classics , including his super Citation Classic on "Isolation of mononuclear cells and granulocytes from human blood," in the Scandinavian Journal of Clinical and Laboratory Investigation . I arbitrarily designate any paper cited over 5,000 times � a super classic. This includes about perhaps 100 to 200 papers.

Figure 3

Time does not allow me to list them all, but here is a list of the ten most-cited super classics for 1981-95. At the top of course is our old friend, the Lowry method. It is still cited thousands of times each year but after 47 years, citations are declining. This occurs with most super citation classics after 20 years.
 

Figure 4

Figure 4 provides a good perspective on the subject of Citation Classics. However, let me digress for a moment. In a commentary I recently submitted to the journal Scientometrics on the topic of "Theories of Citation",³I have offered a list of maxims, some of which are related to Citation Classics.
 

Myth about Size of Field and Impact Factor

There is a widely circulated myth that the size of a field largely determines the impact factors of journals. From outward appearances, it is assumed that the impact of biochemistry journals is high because there are so many biochemists. However, the membership of the ASBMB is not anywhere near as large as other professional societies like the AMA and IEEE or even the newer Society for Neuroscience. Nevertheless, we can agree that biochemistry is a field with many researchers.

The Journal of Biological Chemistry published about 10,000 papers in 1994 and 1995. These papers were cited 71,167 times in 1995. So its current impact factor was 7.45. However, it ranked 85 ^th by impact even though it published more articles in 1996, and over the past 20 years, than any other journal in the world. If size were the primary determinant, this journal should have the highest impact. But that is the point. Size alone of the biochemical or any other literature per se does not determine impact. A large field does indeed produce many citations, but there are also more papers that need to share those citations, so more citations are needed to keep up the average.

If the size of a literature does not determine impact, what does? In 1996, JBC articles cited an average of 42 references per source article. It also had a relatively high immediacy of 1.1. For a discussion of this point, see my paper in Current Contents many years ago.⁴So it is the average number of references per article and their recency which largely determines current impact. A more recent discussion is available.⁵

The size of the literature does affect the number of Citation Classics one can expect. With a large population of papers, there is the much greater probability for highly cited papers to occur. Which makes Figure 4 very relevant.

In the Figure 5 you will see 30 papers that have been cited over 10,000 times from 1981-95. So Dr. Boyum�s 1968 paper is among the 30 or so that have been cited over 10,000 times. Let me add that I have deleted several highly-cited books.

As in the case of the Lowry method, Figure 6 shows how citations peaked about 1985 and are gradually declining.
 

Figure 6

We launched the series called Citation Classics in Current Contents about 25 years ago. When I was in Copenhagen last December, I showed this slide indicating Danish authors of Citation Classics. In the next slide, I�ve listed a group of authors from Norway.

"Separation of blood leucocytes, granulocytes and lymphocytes," Tissue Antigen. 4:269-74, 1974, and Isolation of lymphocytes, granulocytes and macrophages," Scand J Immunol 5(Supp. 5):9-15, 1976.

This brief discussion of Citation Classics permits me to segue into a brief mention of another maxim about citation analysis: I never encountered a Nobel Prize winner in science who had not written one or more Citation Classics. This is one of the most reliable indicators we have for Nobel Class science. However, Citation Classics alone do not merit Nobel Prizes. In the case of Kary Mullis, however, that was the only visible evidence of his bibliometric impact.

This leads to a second maxim � Nobelists publish five to six times as many papers as the average author and their work is cited 30 to 40 times the average.⁶

Perhaps these numbers need to be up-dated since we first reported them 34 years ago and clearly Kary Mullis and a few others are exceptions.

Figure 8A

Figure 8B

To generalize further, the following Figures show the most-cited authors in various periods. We�ve identified those who have received Nobel prizes. As early as 1967 we used citation data to test whether we could algorithmically identify scientists "of Nobel Class."⁷Call this, if you will, mapping "the scientific elite" to borrow Harriet Zuckerman�s term. A new edition of her book The Scientific Elite was recently published by Transaction Press.⁸

Of the 50 most-cited scientists in the 1967 SCI , twelve of them have received Nobel prizes.⁷ Sixteen are living members of the NAS. Many of the others have received Lasker, Gairdner, Wolf, and other non-Nobel-Class awards. These data did not take into account co-authorship. So it is limited to first author counts.

Jumping ahead 15 years, we have a list of the 60 most-cited scientists for 1981-97. A few of the Nobelists on this list include David Baltimore, Michael Bishop, and Joseph Goldstein. Thirty of these 60 scientists are members of the NAS � the 50/50 rule! Most of these names will be familiar to you. Experience tells us that such lists, if extended ten fold, will anticipate 90% of the Nobel Prizes and other high-profile awards such as the Lasker, Wolf, Gairdner, etc.

A simple-minded procedure of this kind does a rather remarkable job of selection considering the worldwide population of over 1,000,000 scientists. A list of several thousand most-likely candidates is quite manageable and could be compared with the membership of the national academies. When the list is categorized by discipline and sub-specialty, it is even more interesting. That�s exactly what we did when we identified the 1,000 most-cited scientists from 1965 to 1978.⁹

Figure 9

Figure 10

Figure 10A

At that time, 250 were members of the US National Academy of Sciences and another 125 were members of foreign academies. It would be interesting but time consuming to determine how many of the remaining 625 have subsequently been elected.

Figure 11 shows the 67 molecular biologists in that group. About 6 have received Nobel prizes. ^9-c Thirty seven, over half, have been elected to the US National Academy of Sciences.

Figure 11

Moving up another ten years, the next slide (#12) provides a more recent perspective on the 1990�s. Seventeen are members of the US National Academy of Sciences as indicated by the check mark. Huber and Baltimore are Nobelists.

After we began research on the SCI ^® in 1962, I met Robert K. Merton. He is the eminent sociologist and member of the Swedish Academy. The Mertonian description of normal science describes citations as the currency of science. Scientists make payments, in the form of citations, to their preceptors. Referees are supposed to help keep that Mertonian principle alive and honest. How much this practice has deviated in recent years, we do not know. But this kind of deviant behavior does not support the MacRoberts Hypothesis¹⁰ that citation analysis is fraudulent simply because the references in the bibliographies of current papers do not reflect all their intellectual influences.

They haven�t for most of the post-World War II period and didn�t have to. The old style of German documentation was long ago dropped by most journals. If there is time, I would be glad to read my letter to JASIS on this question.¹⁰

If you want to do an historical analysis of the impact of an individual�s work, you have to do it year by year. You can construct a reasonably accurate portrayal of influence. You cannot expect each current author in a hot research field to cite the work of all earlier influential authors. Another Mertonian principle is involved here � Obliteration by Incorporation. Even citations to Boyum and Lowry will eventually be obliterated or become simple eponyms without explicit citation.

Figure 12

 
I have deliberately emphasized the use of citation analysis as distinct from journal impact factors. Further, illustrations of such uses are taken from my recent AAAS talk (if time is available, otherwise refer them to my home page). I would like to close with a quotation from the National Research Council report which evaluated 5,000 U.S. university departments:

The clearest relationship between ratings of the 'scholarly quality of program faculty' and these productivity measures occurred with respect to 'citation' - with faculty in top-rated programs cited much more often than faculty in lower-rated programs who published."¹¹

References:

1. back to text BrachoRiquelme RL, PescadorSalas N, & ReyesRomero MA. " Bibliometric repercussions of adopting English as the language of publication," Revista de Investigacion Clinica 49(5):369-372 (September-October 1997).

2. back to text Spiridione, G. & Calza, L. IL Peso Della Qualita Accademica Cooperativa Libraria Editrice Universita di Padova, 1995, 123pgs.

3. back to text Garfield, E. "Concerning Citationology -- Citation Theory and Practice," Scientometrics, In Press.

4. back to text Garfield, E. "Is the ratio between number of citations and publications cited a true constant?"Current Contents No. 8: 5-7 (February 9, 1976). Reprinted in Essays of an Information Scientist ,  Volume 2. Philadelphia: ISI Press, pgs. 419-421, 1977.

5.  back to text Garfield, E. "Dispelling a Few Common Myths About Journal Citation Impacts," The Scientist 11(3):11 (February 3, 1997) http://www.the-scientist.com/yr1997/feb/comm_970203.html

6.back to text Sher, I. H., Garfield, E. "New tools for Improving and Evaluating the Effectiveness of Research ," Research Program Effectiveness, (M.C. Yovits, D.M. Gilford, R.H. Wilcox, E. Staveley, H.D. Lemer, Eds.)  New York: Gordon and Breach, p.135-146, 1966 .

7. back to text Garfield, E. "Citation Indexing for Studying Science," Nature 227:669-671 (August 15, 1970)

8. back to text Zuckerman, H. The Scientific Elite: Nobel Laureates in the United States. New Brunswick and London: Transaction Press, 1996.

9. a. back to text Garfield, E. "The 1,000 Contemporary Scientists Most-Cited 1965-1978. Part 1. The Basic List and Introduction," Current ContentsNo. 41:5-14(October 12, 1981). Reprinted in Essays of an Information Scientist, Volume 5 . Philadelphia: ISI Press. Pgs. 269-278.

9. b.  Garfield, E. "The 1,000 Most-Cited Contemporary Authors. Part 2A. Details on Authors in the Physical and Chemical Sciences and Some Comments about Nobels and Academy Memberships," Current Contents No. 9: 5-13 (March 1, 1982). Reprinted in Essays of an Information Scientist, Volume 5.  Philadelphia: ISI Press. Pgs. 428-436.

9. c.back to text Garfield, E. "The 1,000 Most-Cited Contemporary Authors. Part 2B. Details on Authors in Biochemistry, Biophysics, Cell Biology, Enzymology, Genetics, Molecular Biology, and Plant Sciences," Current Contents No. 21:5-13 (May 24, 1982). Reprinted in Essays of an Information Scientist, Volume 5. Philadelphia: ISI Press. Pgs. 533-541.

9. d.  Garfield, E. "The 1,000 Most-Cited Contemporary Authors. Part 2C. Details on Authors in Hematology, Histology, Immunology, Microbiology, Physiology, and Virology," Current Contents No. 22:5-13 (May 31, 1982). Reprinted in Essays of an Information Scientist, Volume 5.  Philadelphia: ISI Press. Pgs. 542-550.

9. e. Garfield, E. "The 1,000 Most-Cited Contemporary Authors. Part 2D. Details on Authors in Cardiology, Endocrinology, Gastroenterology, Nephrology, Neurobiology, Neurology,  Neuropharmacology, Nuclear Medicine, Oncology, Pathology, Pharmacology, Psychiatry, and Surgery,"  Current Contents No. 24:5-13 (June 14, 1982). Reprinted in Essays of an Information Scientist, Volume 5. Philadelphia: ISI Press. Pgs. 562-574.

9. f. Garfield, E. "The 1,000 Most-Cited Contemporary Scientists. Part 3. Details on Their Current Institutional Affiliations," Current Contents No. 27:5-20 (July 5, 1982). Reprinted in Essays of an Information Scientist, Volume 5. Philadelphia: ISI Press. Pgs. 591-606.

10. back text Garfield, E. "Validation of Citation Analysis," Journal of the American Society for Information Science 48(10):962-964 (1997).
 

11. back to text Goldberger ML, Maher BA, and Flattau PE (Editors), Research-Doctorate Programs in the United States: Continuity and Change. Washington, DC: National Academy Press, 1995, 704 pgs.