Letters to the Editor
The Impact Factor and Using It Correctly
Eugene Garfield
Der Unfallchirurg, 48(2) p.413, June 1998.
I would like to comment on the statement by Prof. H-J. Oestern and Prof. J. Probst of the German Trauma Society which appeared in the October 1997 issues of your journal as well as the Anaesthesist. The authors assert that the work of German specialists in these fields is published "primarily" in German language journals. And without any supporting data they assert that the impact factor is not appropriate for judging scientific achievements in trauma surgery and most important that "its use leads to an unjustified disadvantage in comparison with other fields."
Where has it been written that such comparisons should be made? In a series of unrelated assertions, none of which are supported by data, claims of bias are made repeatedly. Would these authors assert that German scientists, even in trauma surgery, do not publish in the international journals? In 1997, scientists from Germany published over 77,000 papers in Science Citation Index® covered journals -- almost 7.8% of the total ISI® database. About 12,000 of those articles were published in German.
A detailed analysis and definition of the field of trauma surgery is necessary in order to determine the percentage of papers from each country and their relative performance. An agreed upon list of cohort source journals is essential.
The authors talk about self-citations, but there is no evidence that this changes impact factor. In fact, for the smaller journals, self-citations may be the major source of citations, especially when they are new.
The authors allege that there is discrimination against other unspecified journals. Are they aware that all references, regardless of the journals in which they were published, are included in the SCI® data?
Unfallchirurg and other Springer journals are included in SCI. Which are the journals that are missing whose citations might improve their impact? In fact, as we add more low-impact journals we find that they cite the high-impact journals heavily and thereby increase their already high impact. But these generalizations need to be supported by specific studies in each field.
The idea that non-English language journals do not have a chance to be cited is untrue. The SCI processes all references regardless of the journal cited, with the exception of those in exotic alphabets such as Chinese and Japanese. German scientists who publish in English-language journals are never forbidden to cite relevant work in German. One might argue that English-speaking readers may not read the original German journals, but today the use of English titles and abstracts means that few important articles are missed. But even that in itself does not guarantee citation. Authors cite one another because they become familiar with their work through many channels including international meetings and educational exchanges. And hopefully good refereeing keeps most references relevant.
It is absurd to make invidious comparisons between specialist journals and multi-disciplinary general journals like Nature and NEJM. To compare journals you should stick to a particular category as is explained very carefully in the Guide to Journal Citation Reports®. Incidentally, the anomaly of the old journal Clinical Research is explained in a footnote on page 7 of the JCR® printed guide. That journal primarily contains meeting abstracts and its title has been changed to Journal of Investigative Medicine.
The source of much anxiety about Journal Impact Factors comes from their misuse in evaluating individuals, e.g. during the Habilitation process. In many countries in Europe, I have found that in order to shortcut the work of looking up actual (real) citation counts for investigators the journal impact factor is used as a surrogate to estimate the count. I have always warned against this use. There is wide variation from article to article within a single journal as has been widely documented by Per O. Seglen of Norway and others.
These questions will be discussed in Oslo in April by myself and Dr. Seglen and then in May at the Conference of Biology Editors in Salt Lake City. All editors are all welcome to attend.
For detailed information on over 8,000 journals, ISI’s Journal Performance Indicators can be a valuable source of data for comparing journals. These ISI databases are available through David Pendlebury at ISI (215-386-0100,
x1411). Long-term impact factors rather than current impact may be more appropriate in certain clinical fields.
Citation data and analysis should always be used in combination with other indicators when evaluating departments or individuals. For nation by nation comparisons, there is very little controversy about the use of citation indicators. Further, they have been used in the USA to evaluate 5,000 departments at the leading universities. Similar research assessment exercises are performed in the UK.
To test the validity of the ISI data you should identify a cohort of experts in trauma surgery and see how their citation records compare. This would augment an article-by-article citation audit of articles published in your journal. This can be done by contacting Pendlebury or it can be done by use of the various public online vendors of citation indexes or the Web of Science.
With best wishes.
Eugene Garfield
PS The following reference is of possible interest since it was written
by a German scientist and demonstrates that even journals not included in
SCI can have their impacts calculated.
"How to evaluate journal impact factors "
Stegmann J
FREE UNIV BERLIN,
UNIV CLIN BENJAMIN FRANKLIN,
D-12200 BERLIN, .
Nature 390: (6660) 550-550 DEC 11 1997