Aslib Proceedings

Emerald Article: Journal diffusion factors: a new approach to measuring

research influence
Ian Rowlands

Article information:
To cite this document: Ian Rowlands, (2002),"Journal diffusion factors: a new approach to measuring research influence", Aslib
Proceedings, Vol. 54 Iss: 2 pp. 77 - 84
Permanent link to this document:
http://dx.doi.org/10.1108/00012530210435211
Downloaded on: 13-12-2012
References: This document contains references to 13 other documents
Citations: This document has been cited by 15 other documents
To copy this document: permissions@emeraldinsight.com
This document has been downloaded 943 times since 2005. *

Users who downloaded this Article also downloaded: *

Ian Rowlands, (2002),"Journal diffusion factors: a new approach to measuring research influence", Aslib Proceedings, Vol. 54 Iss:
2 pp. 77 - 84
http://dx.doi.org/10.1108/00012530210435211

Ian Rowlands, (2002),"Journal diffusion factors: a new approach to measuring research influence", Aslib Proceedings, Vol. 54 Iss:
2 pp. 77 - 84
http://dx.doi.org/10.1108/00012530210435211

Ian Rowlands, (2002),"Journal diffusion factors: a new approach to measuring research influence", Aslib Proceedings, Vol. 54 Iss:
2 pp. 77 - 84
http://dx.doi.org/10.1108/00012530210435211

Access to this document was granted through an Emerald subscription provided by BROWN UNIVERSITY

For Authors:
If you would like to write for this, or any other Emerald publication, then please use our Emerald for Authors service.
Information about how to choose which publication to write for and submission guidelines are available for all. Please visit
www.emeraldinsight.com/authors for more information.
About Emerald www.emeraldinsight.com
With over forty years' experience, Emerald Group Publishing is a leading independent publisher of global research with impact in
business, society, public policy and education. In total, Emerald publishes over 275 journals and more than 130 book series, as
well as an extensive range of online products and services. Emerald is both COUNTER 3 and TRANSFER compliant. The organization is
a partner of the Committee on Publication Ethics (COPE) and also works with Portico and the LOCKSS initiative for digital archive
preservation.
*Related content and download information correct at time of download.

Journal diffusion
factors: a new approach
to measuring research
influence
Ian Rowlands
The author
Ian Rowlands is a Senior Lecturer, CIBER: Centre for
Information Behaviour and the Evaluation of Research,
Department of Information Science, City University,
London, UK.
Keywords
Libraries, Research, Measurement, Journal publishing
Abstract
This paper introduces a new bibliometric tool, the journal
diffusion factor. An argument is presented that the
bibliometric indicators commonly used to measure the
quality of research (journal impact factor, immediacy
index and cited half-life) offer little insight into the
transdisciplinary reception (thus the wider influence) of
journals. The journal diffusion factor describes a neglected
dynamic of citation reception and is intended as a
complementary partial indicator for research evaluation
purposes, to be read alongside existing well-established
indicators.
Electronic access
The research register for this journal is available at
http://www.emeraldinsight.com/researchregisters
The current issue and full text archive of this journal is
available at
http://www.emeraldinsight.com/0001-253X.htm
Aslib Proceedings
Volume 54 . Number 2 . 2002 . pp. 77±84
# MCB UP Limited . ISSN 0001-253X
DOI 10.1108/00012530210435211
77
Journal impact
Journal impact factors (JIFs) are widely
accepted as a simple, useful and convenient
shorthand notation for calibrating the relative
standing of serials titles (Garfield, 1994). Their
publication, on an annual basis by ISI1, is
eagerly awaited by bibliometricians, research
administrators, journal publishers and
(increasingly) by academics and researchers.
The results can facilitate or hinder success in
grant competitions, influence journal
acquisitions or disposal policies, and make or
break the careers of researchers, editors and
publishers. Impact factors matter.
Impact factors naturally have their critics.
More thoughtful members of the
bibliometrics community have pointed to
certain technical problems with the way that
impact factors are usually calculated (see for
example Davis, 1998; Moed et al., 1996;
Moed and van Leeuwen, 1995; Lewison,
2001) in a spirit of constructive criticism.
Other commentators, unconvinced that a
concept as subtle as ``research quality'' can
really be inferred from counts of citation
tokens, are openly hostile to what they regard
as the bean-counting tendencies of those who
use impact factors for evaluating research
(Lehrl, 1999).
The perspective of this paper is to blissfully
ignore both the technical and principled
objections to ISI1 journal impact factors and
to try to strike out instead in a new direction.
Like them or loathe them, impact factors are
here to stay. The question posed here is
whether they actually represent what they are
generally taken to represent: a journal's
standing, prestige or (more prosaically) its
research utility, or whether they should be
interpreted more carefully and within
narrower terms of reference.
What do journal impact factors
measure?
JIFs are determined by computing the ratio of
recent citations to recently published articles
The author is indebted to Ellen Bonnevie of the
Royal School of Library and Information Science,
Copenhagen, for her helpful advice and
encouragement.
Received 10 December 2001
Revised draft received 3 January 2002
Accepted 18 January 2002
 Ian Rowlands
and reviews. ``Recent'' is an elastic concept,
but in practice usually means two or five
years. In the example below, a two-year
diachronous journal impact factor for Aslib
Proceedings for 1998 is calculated.
Number of recent citations (the
numerator)
The number of citations received during 1998
by Aslib Proceedings articles and reviews
published in the previous two years, 1996 and
1997:
1996 + 1997 = 21 citations.
Number of recent publications (the
denominator)
The number of articles and reviews published
in Aslib Proceedings during 1996 and 1997:
1996 + 1997 = 75 articles and reviews.
Impact factor calculation
JIF98= 21/75 = 0.3
All things being equal, impact factors should
offer a rational basis for comparing the
relative ``performance'' of two or more titles.
However, all things are rarely equal, and it
will serve us well to remember that impact
factors are heavily influenced by subject field,
document type and journal size (Mabe and
Amin, 2000), by numbers of citations (Egghe,
1988), and by research level, shifting fashions
and publication policy (Lewison, 2001) as
well as by absolute quality, whatever that
means. Tomer (1986) concludes that impact
factors are useful insofar as they aid the
identification of what is probably the
relatively small number of journals that
receive disproportionately high numbers of
citations. He also points out that they are
otherwise uninformative derivations,
producing rank orderings which are not
dissimilar to uncorrected rates of citation
(Tomer, 1986).
Intellectual influence and quality are
difficult ideas to conceptualise, let alone
measure, and it is noticeable how careful
some bibliometricians are with their use of
language. Thus Mabe and Amin (2000) note
simply that impact factors measure the way
that a journal ``receives citations to its articles
over time'', while Bordons and Zulueta
(1999) consider that impact factors are better
regarded as a measure of a journal's
``visibility'' rather than its scientific quality.
78
Volume 54 . Number 2 . 2002 . 77±84
Nevertheless, throughout the scientific and
research policy literature, impact factors are
implicitly and explicitly associated with values
of research quality, utility and influence. Is it
reasonable to impute a single partial indicator
with so much intellectual baggage?
Chambers Dictionary defines ``quality'' as
relating to a ``high grade of excellence'',
``utility'' as ``usefulness, the power to satisfy
wants'', and ``influence'' as the ``power of
producing an effect'' or ``exertions of friends
at court, wire pulling, and the like''. Thinking
about the motivations that structure authors'
citation and publication behaviour, elements
of all three definitions seem to have some
relevance! But the lack of clarity surrounding
the independent variables which are
associated with impact factors in empirical
studies should be a source of some concern,
especially given the controversial nature of
much data-driven research evaluation.
Citations and the ripple effect
The dynamics of how ideas are transferred from
one author to another, and from one discipline
to another, is a central concern within scholarly
communication and it is surely relevant to some
conception of research quality or influence. If
we regard new ideas as being like pebbles
thrown into a pond (where the surface of the
pond represents the general research literature)
we can draw upon two potentially useful
metaphors: the size of the splash as the pebble
hits the surface of the water, and the
characteristics of the resulting ripples.
Mabe and Amin (2000) relate the three
standard bibliometric measures (impact
factor, immediacy index and cited half-life)
published in ISI1's annual Journal Citation
Reports series to a generalised citation curve.
Their work is carefully presented, and I
apologise to both authors by playing fast and
loose with their ideas and relating them to the
pebble-pond metaphor. Mabe and Amin
argue that impact factors measure the relative
area under the citation curve within the
two-year time window (i.e. the mass of the
pebble); that the immediacy index represents
the initial gradient of the citation curve (i.e.
the acceleration of the pebble as it hits the
surface of the water); while cited half-life
indicates the rate of decline of the citation
curve (i.e. the time taken for calm to begin to
return to the surface of the pond).
 Ian Rowlands
If these analogies hold, then it follows that
journal impact factors are best regarded as
only partial indicators of research impact, at
least in the limited technical sense of
describing particular aspects of citation
dynamics. To fully understand the ``splash'',
they need to be read carefully alongside
immediacy and cited half-life. However, and
this is the main point of this paper, they tell us
nothing directly about the subsequent ripples,
the ``breadth'' of the reception of a particular
journal in the marketplace for ideas.
This paper introduces a simple ``ripple''
measure which is intended to complement the
three established ``splash'' indicators
discussed above: a journal diffusion factor.
A note on journal citation distributions
Since the focus of this article is on the ripples
rather than the characteristics of the
immediate splash, some preliminary
observations on journal citation distributions
are presented here, so as to locate what
follows in its proper context.
Much of the initial work for this article
employed online bibliometric techniques,
using the Dialog Classic implementation of
the Science Citation Index1 and the Social
Science Citation Index1. A set of citations to a
given journal title may be retrieved using an
expression such as the one in Figure 1.
This search expression defines the set of
citing papers accruing to a particular journal
title from all SSCI1-indexed publications
appearing over a two-year period (1996 and
1997). Throughout, a limiting expression
restricts the citing documents to articles and
review papers.
The Dialog RANK command can be used
to analyse a set of citing papers by their source
journals, generating a rank-frequency
distribution as in Figure 2. A wider time
window (ten years) is used here to make the
highly asymmetric nature of the citation
distribution more readily apparent.
Figure 1 Retrieving a set of journal citing papers from SSCI1 in Dialog Classic
79
Volume 54 . Number 2 . 2002 . 77±84
Two points leap out immediately from this
distribution. The first is that the journal itself,
Scientometrics, accounts for the largest single
contribution to its own citations (a common
phenomenon), followed by a highly
productive core of closely related journals.
The second is that what follows is a long
diffuse tail including many citations from
titles outside of the immediate discipline.
Yanovsky (1981) suggests that a better
understanding of the factors that shape
journal citation patterns such as these could
be used to inform serials acquisitions policies,
as well as providing insights into disciplinary
structures.
Introducing the journal diffusion factor
The distribution in Figure 2 speaks to the
question of the extent of the ripples that result
from the diffusion of ideas in Scientometrics
into the general research literature. The
simplest and perhaps most intuitive indicator
of journal diffusion is the ratio of citing
journals to source citations. The journal
diffusion factor (JDF) is presented here in
standardised form as the average number of
citing journals per 100 source citations within
a given time window. This new indicator can
easily be determined in an online
environment such as Dialog Classic by
following the general example of Christensen
et al. (1997). A two-year journal diffusion
factor for Scientometrics for 1998 may be
derived as follows in Figure 3.
The set created by the search expression for
the numerator shows the number of citing
papers published in 1998 (as articles or
reviews, search limit not shown), which refer
to Scientometrics articles from 1996 and 1997.
The subsequent RANK analysis on the citing
papers' journal field identifies 18 unique
terms, i.e. distinct journal titles. (It is strongly
recommended that these terms should be
inspected, to avoid potential bias due to
changes of journal name or variants in the way
that journal names are represented by ISI1.)
 Ian Rowlands Volume 54 . Number 2 . 2002 . 77±84

Figure 2 Dialog RANK analysis of citations to Scientometrics

Figure 3 Online method for calculating journal diffusion factors from SSCI1 in Dialog Classic

The number of citations in the denominator is
found by simply inspecting the online
postings using a simple correction
(Christensen et al., 1997). It should be noted
that this method recovers numbers of
80
individual citations rather than citing papers
since there may be more than one unique
citation to Scientometrics in a given article.
The journal diffusion factor is expressed
here as the number of unique citing journal
 Ian Rowlands Volume 54 . Number 2 . 2002 . 77±84

titles per 100 citations, and may range
theoretically between one and 100. As with
journal impact factors, the size of the time
window may be varied (a two-year factor is
calculated in this instance), and both
synchronous and diachronous methods might
be employed (Ingwersen et al., 2000).
The data in Table I comprise 42 library and
information science titles ranked by two-year
journal diffusion factors for 1998. The
diffusion factors range between 38.2 (Social
Science Information) to 5.1 (Journal of
Government Information) with an average value
of 20.0. The third column shows the rankings
that obtain when the journals are ranked by
the comparable journal impact factors as
published by ISI1 in the Journal Citation
Report for 1998. The two lists are very

Table I LIS journals ranked by 1998 two-year journal diffusion factor

Ranked by journal Ranked by journal
Journal title diffusion factor impact factor
Social Science Information 1 28
Aslib Proceedings 2 35
Journal of Education for Library & Information Science 3 39
Libri 4 29
E-Content (formerly Database) 5 36
Information Systems Journal 6 14
Information Society 7 1
Journal of Scholarly Publishing 8 27
Online 9 32
Journal of Information Science 10 23
Scientist 11 41
Journal of Information Technology 12 15
Information Systems Research 13 5
Library Resources & Technical Services 14 31
Bulletin of the Medical Library Association 15 37
Serials Librarian 16 34
International Journal of Information Management 17 17
Library & Information Science Research 18 13
Library Journal 19 40
Library Hi Tech Journal 20 26
International Journal of Geographical Information Science 21 4
Information Processing & Management 22 18
Reference & User Services Quarterly 23 25
Interlending & Document Supply 24 42
Journal of Documentation 25 8
Law Library Journal 26 33
Program 27 10
NFD Information Wissenschaft & Praxis 28 38
Library Trends 29 16
Library Quarterly 30 9
Social Science Computer Review 31 20
Telecommunications Policy 32 22
Information & Management 33 7
Library Collections, Acquisitions & Technical Services 34 30
College & Research Libraries 35 12
MIS Quarterly 36 2
Journal of Academic Librarianship 37 19
Government Information Quarterly 38 24
Journal of the American Medical Informatics Association 39 3
Journal of ASIS 40 6
Scientometrics 41 11
Journal of Government Information 42 21

81
 Ian Rowlands
distinct: only 14 of the 42 journals even
remain in the same half of the distribution
when ranked in this way (note that Aslib
Proceedings leaps up 33 places!).
As a very general principle, journals in the
top third of the list comprise titles which are
more oriented towards the general academic
reader (e.g. Libri), the practitioner (e.g.
E-Content, formerly Database), or which have
a high social sciences content (e.g. Social
Science Information), or all three
characteristics (e.g. Aslib Proceedings). Titles
in the lower third of the list are generally more
highly specialised and scholarly in nature (e.g.
Scientometrics, Journal of Government
Information). Since diffusion and impact
factors are designed to measure different
aspects of the reception process, it is perhaps
hardly surprising that the ranked listings are
so different. It is worth bearing in mind that
``high diffusion'' does not at all imply a large
relative number of citations given that the
diffusion factor standardises the number of
citing titles per 100 citations. Nonetheless,
the introduction of this new indicator does
suggest some interesting questions about
what citations can tell us about research
quality or influence. Consider Figure 4.
Which scenario depicts the most ``influential''
journal, B or C? This question is posed simply
as an illustration of the additional dimension,
which the notion of diffusion brings to
thinking about what journal ``impact'' really
means. Its application to understanding
transdisciplinarity and boundary spanning
issues are strikingly evident, and the indicator
might be of great interest to editors and
journal publishers.
Figure 4 Different modes of influence: journal diffusion and impact
82
Volume 54 . Number 2 . 2002 . 77±84
Interpretation of the data in Table I is difficult
and possibly premature. It may be, for
example, that high diffusion journals play an
important role in science as ideas aggregators,
spinning together ideas from closely-related
specialities and knitting them into the fabric
of neighbouring disciplines. Conversely, low
diffusion journals may play a quite distinct
role by providing a sharp focus for specialists
in, for example, bibliometrics (Scientometrics)
or federal information policy (Journal of
Government Information) to explore the limits
of their field.
Formal characteristics of the journal
diffusion factor
As noted earlier, it is intended that the journal
diffusion factor should be read alongside
existing ISI1 indicators, notably the journal
impact factor, immediacy index and cited
half-life. This section explores the formal
characteristics of the diffusion factor in
relation to these existing and widely used
tools.
Journal titles were selected using random
number tables from four contrasting ISI1
subject categories:
(1) astronomy and astrophysics;
(2) immunology;
(3) information science and library science;
and
(4) orthopaedics.
In each case, the samples represent at least 50
per cent of all the titles from the relevant
subject category. The fields were selected to
 Ian Rowlands
represent a variety of research levels and
styles: from a basic science (astronomy and
astrophysics), through an intermediate
science with strong basic and clinical aspects
(immunology) to an applied discipline
(orthopaedics) and a social science field
(LIS). Two-year journal diffusion factors
were computed using the method above and
the remaining data taken from the 1998
Journal Citation Reports (ISI1) on CD-ROM.
In three of the literatures studied, the
journal diffusion factor exhibited a significant
negative correlation with the traditional
impact factor (see Table II). This seems
counter-intuitive, since one would have
perhaps expected that high quality research
would be rewarded both by rapid citation
uptake (impact) and broad reception
(diffusion).
In fact, when the effect of the absolute
number of citations is controlled for, the
apparent relationships between journal
impact and diffusion, and between journal
immediacy and diffusion disappear entirely.
The sensitivity of both journal impact
(previously observed by Egghe, 1988) and
diffusion factors to citation frequency
suggests that journal rankings of the kind
presented as Table I should be interpreted
with great caution. This also constitutes an
argument for using longer time windows (five
years, say) in constructing diffusion indicators
to give greater stability.
Since journal diffusion and impact factors
appear to be statistically independent, at least
for the literatures studied here, it may be
concluded that they are tracking different
aspects of the dynamics of reception. But how
are we to interpret high or low diffusion
scores? The final column in Table II presents
the correlation coefficients for journal
diffusion factors and export ratios. Export
ratios represent the proportion of citing
papers that are published outside the field, as
defined by ISI1 subject category: thus, a
citation from an article published in a
Table II Journal diffusion factor correlates (Pearson coefficients)
ISI1 subject category
Astronomy and astrophysics
Immunology
Information science and library science
Orthopaedics
Notes: * Correlation is significant at the 0.01 level; ** Correlation is significant at the 0.05 level
Volume 54 . Number 2 . 2002 . 77±84
biochemistry journal to one in an
immunology journal is regarded here as an
``export''. These export ratios were computed
using a simple online technique.
Journal diffusion and exports show a
moderately strong and significant correlation.
When corrected for absolute numbers of
citations, the partial correlation observed
becomes even stronger. It seems then that
diffusion factors may be read as an indication
of the degree of transdisciplinary reception of
the papers in a journal (this finding needs
further investigation, since ISI1 subject
categories apply at the aggregated level of the
journal, not the unbundled paper).
Conclusion
The journal diffusion factor proposed here
can be read as an indication of the extent of
the ripples that flow from the publication of a
particular journal as it is subsequently cited.
There is prima facie evidence to suggest that
the journal diffusion factor is highly correlated
with ``exports'' of ideas across disciplinary
boundaries and that it may therefore be of
great interest in mapping research influence.
It is sufficiently distinct from the journal
impact factor, both conceptually and in terms
of its statistical behaviour, to warrant its
serious consideration in research evaluation.
The patterns which citations make as they
accumulate through time and across
literatures are indeed complex and it is almost
impossible to imagine how a single summary
indicator could capture the essence of
something as subtle as scholarly influence.
Garfield impact factors are certainly a useful
tool in research evaluation, but they are often
crudely applied and loosely interpreted in
practice. In warning against the perils of
over-reliance on a single indicator in
bibliometrics, my colleague, Grant Lewison,
is fond of pointing that out we buy cars on the
basis not solely of the engine size, number of
Journal Immediacy Cited Export
n impact factor index half-life ratio
34 ±0.21 ±0.22 0.04 0.39*
66 ±0.38** ±0.40** 0.21 0.43**
42 ±0.37* ±0.16 0.35 0.21
36 ±0.42* ±0.43* ±0.29 0.54**
83
 Ian Rowlands
doors, colour, price, or the ready availability
of spare parts but by all these characteristics
and more taken together. The utility and
limitations of the journal diffusion factor
should be seen in that light.
References
Bordons, M. and Zulueta, M.A. (1999), ``EvaluacioÂn de la
actividad cientõÂfica a traveÂs de indicadores
bibliomeÂtricos'', Revista EspanÄola de Cardiologia
Online, Vol. 52 No. 10, available at:
www.revespcardiol.org/cgi-bin/wdbcgi.exe/cardio/
mrevista_cardio.fulltext?pident=190 (accessed
6 December 2001).
Christensen, F.H., Ingwersen, P. and Wormell, I. (1997),
``Online determination of the journal impact factor
and its international properties'', Scientometrics,
Vol. 40 No. 3, pp. 529-40.
Davis, J.D. (1998), ``Problems in using the Social Sciences
Citation Index to rank economics journals'',
American Economist, Vol. 42 No. 2, pp. 59-65.
Egghe, L. (1988), ``Mathematical relations between impact
factors and average number of citations'',
Information Processing & Management, Vol. 24
No.5, pp. 567-76.
Garfield, E. (1994), ``The impact factor'', Current
Contents1, Vol. 25, 20 June, pp. 3-7.
Ingwersen, P., Larson, B. and Wormell, I. (2000),
``Applying diachronous citation analysis to research
84
Volume 54 . Number 2 . 2002 . 77±84
program evaluations'', in Cronin, B. and Atkins, H.B.
(Eds), The Web of Knowledge, ASIS Monograph
Series, Medford, NJ, pp. 373-87.
Lehrl, S. (1999), ``Evaluating scientific performance by
impact factors: the right for equal opportunities'',
Strahlentherapie und Onkologie, Vol. 175 No. 4,
pp. 141-53.
Lewison, G. (2001), ``Impact factors revisited'',
unpublished manuscript.
Mabe, M. and Amin, M. (2000), ``Impact factors: use and
abuse'', Perspectives in Publishing, Vol. 1, available
at: www.elsevier.com/homepage/about/ita/editors/
perspectives1.pdf (accessed 6 December 2001).
Moed, H.F. and van Leeuwen, T.N. (1995), ``Improving the
accuracy of Institute for Scientific Information's
journal impact factors'', Journal of the American
Society for Information Science, Vol. 46 No. 6,
pp. 461-7.
Moed, H.F., Van Leeuwen, T.N. and Reedijk, J. (1996),
``A critical analysis of the journal impact factors of
Angewandte Chemie and the Journal of the
American Chemical Society: inaccuracies in
published impact factors based on overall
citations only'', Scientometrics, Vol. 37 No. 1,
pp. 105-16.
Tomer, C. (1986), ``A statistical assessment of two
measures of citation: the impact factor and the
immediacy index'', Information Processing &
Management, Vol. 22 No. 3, pp. 251-8.
Yanovsky, V.I. (1981), ``Citation analysis significance of
scientific journals'', Scientometrics, Vol. 3(n),
pp. 223-33.