Figure A
Academics rely on peer review, meaning subjective human judgment, for evaluating research performance. However, modern databases which index millions of research papers permit us, through publication and citation analysis to view the research world with some new objective perspectives which can validate peer review opinions. This process can provide better-informed judgments to those responsible for funding research.
|
Figure A |
ISI launched Current Contents(r) in the 1950s and then in the early 1960s, the Science Citation Index(r). SCI(r) included not only full bibliographic information on each journal article, including author names and addresses, article titles, journal names, and pagination, but also the references listed or cited in these papers. The Science Citation Index enables you to find out what someone has published, a traditional index function, but also what other papers have cited them. The idea is this: if you know a paper or book that is related to your own interests, then other papers that cite that publication may be important to you as well. In this schematic diagram, you see how a paper by Harold Urey has been indexed in the SCI. I used this slide recently to remind people that the question of life on Mars was hot news 35 years ago too!
The SCI offered researchers a new way of searching the literature. The method had been utilized in American case law for decades. Let me emphasize that both Current Contents and the Science Citation Index were created for purposes of information retrieval and dissemination -- not research analysis or evaluation. The latter use came to the fore as computer technology advanced and ISI's indexes reached "critical mass." But even a few years after we started the SCI, and Social Sciences Citation Index(r) in the late 1960s, the files were already used for evaluation.
Sociologists of science and, soon after them, research administrators and science-policy analysts, went to the library shelves to use the printed Science Citation Index. This group sensed that citations were useful indicators of the utility, significance, and influence of scientific work: Simply, the more citations a paper received, the more useful it probably had been to the scientific community. Citation scores were seen as akin to polling figures; as with polling data, one can and does encounter artifacts in the data (negative citation, self citations, citation circles or cabals, wherein friends cite friends). Time does not permit me to discuss all the pros and cons of citation analysis. There is a vast literature on it. Today I am going to concentrate on showing you real data that leave little room for debate. You can judge whether the tabulations I'll show you match your perceptions of what is important or significant in the field of sports medicine.
Let me close this introduction by referring you to the recent voluminous report of the National Research Council which ranked over 5,000 academic departments in the United States:
"The clearest relationship between ratings of the 'scholarly quality of program faculty' and these productivity measures occurred with respect to 'citation' - with faculty in top-rated programs cited much more often than faculty in lower-rated programs who published."
Thirty years of detailed studies have demonstrated the validity of citation analysis. Citation analyses correlate well with peer judgments of significance. Thus, citation analysis should generally bring to light which papers, people, institutions, nations, and journals have contributed in greatest measure to the advance of research in a specific field. That's the approach we've taken with sports medicine.
There is a fundamental distinction that must be made in a multi-disciplinary field like sports medicine. On the one hand, there is what we can call the literature of sports medicine. Then there is the literature of interest to sports medicine researchers.
The first part of my presentation utilizes the literature of sports medicine -- narrowly defined by the top 53 sports medicine journals. The second part of my talk concerns the additional scientific journals that sports researchers depend upon from fields like physiology, nutrition, cardiology and orthopedics. Citation analysis is controversial but its relevance is rarely disputed when it draws on large aggregations of data. In this talk, I will provide a global macro-perspective by nation, institution, or journal. Then we'll take a microview of the most-published and most-cited authors. At an even more specific level, we'll look at long-term, high-impact papers and then more recent hot papers.
|
Here are the top 25 journals out of 53 that we used to create the literature of sports medicine database. (This database is available from David Pendlebury of ISI.) They are arranged by the date they were added to the ISI database. That date is in the last column. From 1981-96, 23,354 papers were published in these 53 journals.
These 23,354 papers were cited a total of 68,754 times by the end of
1996, for an average of 2.94 citations per paper for the field. The corresponding
figure for the whole ISI file, including SCI and SSCI is about 9.1. More
about that later.
National Performance: Output, 1981-96Figure:2
|
In the next Figure (3), we've ranked countries by the number of papers published in our sports medicine database from 1981 to 1996. US and Canadian authors dominate this field and together produce over 65% of the papers. This is in contrast to 44% when you consider ISI's total database covering all fields of research.
National Performance: Impact, 1981-96Figure:3
|
To obtain a more qualitative perspective, we can create a more level playing field by using impact -- citations per paper -- and in this slide you see that several other smaller countries from Sweden to Canada perform better on this basis. Keep in mind that as the number of published papers gets smaller, the figures are less robust since a few papers can distort the result.
Figure : 4
SPORTS MEDICINE 1981-96
|
In Figure 4, we have plotted the annual output in sports medicine journals year by year from 1981-96. There is a chronological bias introduced in 1993 or 1994 when ISI added more journals to cover this field for Focus on Sports Medicine.
The middle portion of this slide shows the cumulative number of citations
to papers published in each year. The bottom portion shows the cumulative
impact.
Figure : 5
SPORTS MEDICINE, 5-Year Periods, 1981-96Figure 5: The graphs summarize output, citations, and citation impact for Sports Medicine in five-year periods for 1981-96.Another perspective on these data is provided in Figure 5 where we have plotted data for 5-year periods, thus smoothing out some of the year-to-year variations.
|
Figure:6 |
Figure:7: INSTITUTIONAL PERFORMANCE, 1981-96 (Institutions publishing 14 or more papers ranked by impact)
In Figure 7, we've ranked universities by impact, that is, average cites per paper. This particular slide is preliminary data which does not yet take into account variations in the 6,000 addresses involved, of which 4,000 occur but once! So 2,000 institutions publish 2 or more papers each. 1370 published 3 or more and 1,030 published 4 or more.
Figure:7
INSTITUTIONAL PERFORMANCE, 1981-96
|
|
Moving down another level of aggregation, we get to individual authors.
These data take into account all authors --
not just first authors. I presume that most of these names are familiar
to you.
Figure 9: AUTHOR PERFORMANCE, 1981-96 - TOP 20 IN CITATIONS
In Figure 9, we've listed authors by total citations and there is quite a shift in the rankings.
AUTHOR PERFORMANCE, 1981-96
|
Figure 10: AUTHOR PERFORMANCE, 1981-96 - TOP 20 IN IMPACT (those who have published 16 or more papers, or one each year).
In Figure 10, we have listed 20 authors by impact -- cited per paper -- and this produces yet another ranking.
| |
In Figure 11 we've ranked journals by productivity.
Journal Output, 1981-96
|
Figure 12: JOURNAL CITATIONS, 1981-96 - Top 15 Journals
In Figure 12, by total citations. As in all other fields, a small group of journals accounts for a large percentage.
|
Figure 13: CUMULATIVE JOURNAL IMPACT, 1981-96 - TOP 16 Journals
At the outset, I said we would start by discussing the journals of sports medicine. But what about the other journals used by sports medicine researchers?
|
In 1993, ISI launched a product called Focus on: Sports Science & Medicine. Slide 14A will give you an idea of the range of sports this covers. However, it is the medical outcomes of these activities that are our primary concern today.
Focus On: Sports Science & Medicine is a bibliographic information product with abstracts which is sold to academic sports medicine physicians, exercise physiologists, kinesiologists, orthopedists, physical therapists, athletic trainers, coaches, and university & medical librarians. Current Awareness searching is available via monthly diskette. A retrospective CD-ROM was introduced in 1996 which now covers 1993 to 1996.
Figure 13A: (below) Shows the range of sports covered in Focus On.
FOCUS ON: SPORTS SCIENCE & MEDICINESports Covered
|
Figure 14 : FOCUS ON SPORTS SCIENCE -- FULLY COVERED JOURNALS NOT INCLUDED IN DATABASE USED FOR THIS TALK
In contrast to the database created for the previous discussion, Focus on Sports Science and Medicine(r) fully covers not only the journals of sports medicine but also 20 leading journals outside of sports medicine proper. These are listed in Figure 14 below...
Figure: 14
Focus On:Sports Science & MedicineAdditional journals fully covered in Focus OnActa Orthopaedica Scandinavica American Journal of Physical Medicine & Rehabilitation Annual Review of Physiology Archives of Orthopaedic and Trauma Surgery Archives of Physical Medicine and Rehabilitation Arthroscopy Aviation Space and Environmental Medicine Clinical Biomechanics Clinical Orthopaedica and Related Research European Journal of Applied Physiology and Occupational Physiology Journal of Applied Physiology Journal of Back and Musculoskeletal Rehabilitation Journal of Bone and Joint Surgery - American Volume Journal of Bone and Joint Surgery - British Volume New England Journal of Medicine Physiological Reviews Scandinavian Journal of Rehabilitation Medicine Spine Sports Medicine Standards and Malpractice Reporter |
Figure 14A: FOCUS ON SPORTS SCIENCE - SELECTIVELY COVERED JOURNALS.
In addition, in Figure 14A there are dozens of selectively covered journals.
I have little doubt that you will recognize the relevance of these journals. They were selected by using another ISI database called Journal Citation Reports(r). This annual statistical report is now published on CD-ROM and permits you to examine, for all 8,000 ISI journals, the citing and cited patterns for each journal. Let me illustrate these two types of listings.
Figure: 14A
Focus On: Sports Science & Medicine
Acta Physiologica Scandinavica
|
Figure 15: JCR LISTING FOR MEDICINE AND SCIENCE IN SPORTS AND EXERCISE - CITING JOURNAL LISTING
In Figure15, we have the list of journals most-cited by one journal -- Medicine and Science in Sports and Exercise, the largest contributor to this field. This slide shows it as the citing journal. The physiological orientation of the journal is evident. Of the ten journals most cited by authors publishing in this journal, 5 are leading physiology journals.
Figure : 15
 |
Figure 16: below: JCR REPORT FOR AMERICAN JOURNAL OF SPORTS MEDICINE - CITED JOURNAL LISTING
In this case, we get a cited journal perspective of the journals which cite the American Journal of Sports Medicine. It has a definite emphasis on orthopedics, not surprising considering that it is published by the American Orthopedic Society of Sports Medicine. Five of the ten journals that cite this journal are in orthopedics.
Figure : 16
 |
Figure 17: CURRENT IMPACT - JCR PRINTOUT SHOWING JOURNALS RANKED BY IMPACT ON TOP - AND THE IMPACT FACTOR CALCULATION FOR MEDICINE AND SCIENCE IN SPORTS AND EXERCISE
As a final perspective on journals, I've listed the 12 journals in the JCR sports science category by current impact factor. This is based on 1995 citations to 1994/93 papers. These numbers are relatively low compared to molecular biology and other fields or disciplines. This is due to factors such as immediacy, half-life and average number of references per paper, and the number of review articles included. These rankings are quite different from those in Slide #14 where I showed you 15-year cumulative impact data.
Figure 17
 |
Figure 18A : MEDICINE AND SCIENCE IN SPORTS AND EXERCISE -- YEAR-BY-YEAR CUMULATIVE IMPACT FROM JOURNAL PERFORMANCE INDICATORS FILE
Figure 17A
 |
Figure 18:: JCR SOURCE DATA INCLUDING REVIEW ARTICLES It is also important to note that certain journals publish many review articles. These include Medicine and Science in Sports and Medicine and the Scandinavian Journal of Medicine and Science in Sports. Review articles are generally cited more than research or clinical reports and would tend to inflate impact numbers.
Figure 18
 |
MOST-CITED PAPERS IN SPORTS MEDICINE, 1981-96
|
Let's move now to the lowest level of classification, the individual paper. In this slide, I've listed the five most-cited papers 1981-96. The paper by G. A. V. Borg of the University of Stockholm appears to be the citation superstar of this field. It is a Citation Classic by any standard.
All of these papers were published in the 1980s so I'm sure that some of them are not cited much any more because they are part of the accepted wisdom of the field.
However, the paper by Haupt continues to be cited about ten times per year -- not surprising considering the topic.
All twelve papers were cited more than 130 times.
For a view of more recent highly cited papers in the field, we can restrict our view to papers published from 1994-1996; the most cited of these we call "hot papers," because they are both highly cited and of recent vintage. In fact, all nine of the papers on this slide and the next one were published in 1994, so it took about 2 1/2 years to achieve these results.
Figure: 20
HOT PAPERS IN SPORTS MEDICINE, 1994-96
|
Incidentally, the average sports medicine paper receives .5 cites per year whereas the average cited item in the SCI overall is cited 2.25 times per year. Let me remind you as I said earlier that the average sports medicine paper received 2.94 cites over the 16-year period in contrast to 9.1 for the SCI which verifies that the half-life for this field is much greater.
