Article

Evaluation

Homing in on excellence:
17(2) 117–131
© The Author(s) 2011
Reprints and permission: sagepub.
Dimensions of appraisal in co.uk/journalsPermissions.nav
DOI: 10.1177/1356389011400891
Center of Excellence program evi.sagepub.com

evaluations

Tomas Hellström
Lund University, Sweden

Abstract
Excellence has become a watchword for goal setting and assessment in science and technology
policy. While the concept has been around for many years in its commonsense meaning, it
is now explicitly used for science governance; however, so far little work has been done to
detail, operationalize and systematize the dimensions of value present in academic evaluators’
use of the concept. This article is based on a close reading of a mid-term evaluation of several
centers of excellence in Sweden, the goal of which is to achieve a detailed understanding
of the evaluative components of the concept. By applying template analysis to evaluation
documents, seven main components, largely referring to the organizational/institutional
aspects of excellence were identified. These are analyzed and used to provide insights
for evaluation scholars and practitioners into the tensions and possibilities present in the
excellence concept.

Keywords
centers of excellence, program evaluation, research evaluation, scientific quality

Introduction
While quality has long been on the agenda for research evaluation, and the concept of excellence
has been present within this discourse as a general evaluative term, it is only recently that it has
gained prominence in policy discourse ‘in its own right’ (Atkinson-Grosjean, 2006; European
Commission, 2008). It is intuitive to think about excellence as a measure of quality in the sense of
equaling high or very high (‘excellent’) academic quality. In such instances, excellence usually

Corresponding author:
Tomas Hellström, Center for Innovation, Research and Competence in the Learning Economy (CIRCLE), Lund University,
Box 117, SE-221 00 Lund, Sweden.
Email: tomas.hellstrom@circle.lu.se

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

118 Evaluation 17(2)

refers to a level of quality in the output of academic activity (see Jackson and Rushton, 1987).
However, when research evaluations proliferate as governance mechanisms for public science,
process evaluations, especially those connected to large program investments, become more prom-
inent. As a result excellence is increasingly operationalized as the set of organizational and process
variables likely to generate high quality outputs (Heinze et al., 2009; Tjissen, 2003). In effect a
separation has been argued between:

•• excellence connected to the actual research process, referring to traditional outputs of scien-
tific inquiry; and
•• excellence in the scientific endeavor at large, including education, third stream activities as
well as management and leadership of a social group (Tjissen, 2003).

It is in this last respect that excellence is emerging as the integrating glue for the governance of
science. The question is, how do we conceptualize excellence in operational terms which are spe-
cific enough to discriminate among programs, which are amenable to concrete improvements in
governance, and which can still be general enough to apply across a number of fields?
Lamont (2009), in a recent study on peer review in grant application processes, points out that
criteria for excellence differ across fields, even within the human sciences, and that several of these
criteria are elusive and visceral: they may for example address the moral qualities of an applicant
or project. In addition, Lamont argues that excellence is produced at many sites, by many actors
and with varying motivations. This variety of actors and interests has led Balderston (1995) to coin
the notion of ‘a politics of excellence’. However, while excellence no doubt has politics, indeed its
formulation and deployment presents a superb empirical expression of the politics of science, there
is also a very strong sense in which excellence is not politics. ‘Excellence is a quintessential poly-
morphic term’ says Lamont (2009: 159), and we may reflect that this is exactly why it is becoming
such a powerful boundary object in today’s science policy discourse. Being an essentially con-
tested concept as well as a boundary object, it creates a space where political and cognitive values
can coexist. Excellence is a term for the political and the scientific community: this because its
evaluative dimensions clearly vary within a common theme which most researchers can relate to,
and it is often tangible enough for external interests to partake and discuss its implications.
This article is centered on a case study of a mid-term evaluation of eight centers of excellence,
spanning the social and natural sciences, and funded by the Swedish Research Council. Being a
mid-term evaluation, the focus was put on process and organizational issues, and how these were
valued as contributing to the sustainability and impact of the centers. The case specifically focuses
on a close reading of the evaluation reports, with an aim to identify how the evaluators choose to
justify and frame their assessments in terms of expressed values or hypothesized causes of excel-
lence. The motivation for this article is twofold: first, the case study aims to elucidate how the
expression of excellence found in evaluation reports concretizes yet complicates the concept. A
second aim is to assess the critical dimensions and contestations involving the concept as presented
in the literature, and show how its use in research evaluation depends on resolving certain tensions
present in academic accounting. This analysis aims to provide a guide for research evaluators uti-
lizing the concept.
The structure of the article is as follows: a number of theoretical precursors to this study will be
outlined, with the aim to capture the dual dimension of excellence as a quality imputed to success-
ful organization (section one) and a set of normative criteria for evaluation (section two). Second,
the methodological approach of the article will be outlined, followed by the case study. Finally, the
article will discuss a number of critical points for research evaluation, derived from the case.

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

Hellström: Homing in on excellence 119

Appraising scientific excellence

Research evaluation is always performed against the background of certain valued processes and
outcomes. Philosophy of science has traditionally divided these into social and epistemic values
(e.g. Lacey, 1999). Social values refer to processes and outcomes which derive from and have
implications for society at large, and to what type of effects of research, forms of inquiry (e.g. its
effect on living beings) and types of institutions are acceptable from the point of view of society’s
values. Epistemic values on the other hand have to do with value commitments which drive choices
of method, forms of inference and theoretical/methodological trade-offs in the research process. It
might also have to do with valuing certain kinds of empirical phenomena in a domain rather than
others, and what type of qualities a good theory should possess (Doppelt, 2008). Kuhn mentioned
accuracy, consistency, scope and simplicity as valued qualities of a theory, but emphasized how
these could be weighted and interpreted differently at different times and in different subjects, as
well as stand in contradiction to each other when practically applied. Such contradictions, for
example between simplicity and accuracy, may generate conflict and ultimately change in research
traditions (Kuhn, 1977).
In addition to social and epistemic values there is a third kind, a hybrid of these two, namely
values regarding how best to implement a research program as an organizational and institutional
activity. This is an issue where both epistemic and social values enter in, specifically in two ways:
first with regard to what kind of organizational arrangements best stimulate the realization of
(appropriate) epistemic values, and second what types of such organizations best realize those
social arrangements which we value ‘for their own sake’, i.e. the way that science can also be a
palatable social activity. The latter aspect may be the most challenging one to accept; however, it
is clear from how organizational research evaluation is conducted that such considerations actually
have a place. This third kind of value is studied under the label of social epistemology (Fuller,
2000; Goldman, 1999; Longino, 2002), and it forms, as we will see, a normative basis for evalua-
tion of excellence. The empirical case study will illustrate concretely how epistemic and social
values, while analytically separate, come to overlap in various ways and with different implica-
tions for research program evaluation. In the following section of this review we look closer at how
excellence has been identified as a social and epistemic quality in research. After that the emphasis
is put on how normative criteria for excellence have been formulated in evaluation exercises and
other types of policy contexts. It is important to note that these theoretical preliminaries are not
aspiring to an exhaustive review of the field, but rather to present and analyze a number of repre-
sentative contributions against which the empirical material can be discussed.

Aspects of excellence
What defining markers of excellence in research and research organizations could form a backdrop
for evaluation? Balderston (1995) discusses various dimensions of excellence from a management
perspective. Of central importance he claims is the extent to which the institution satisfies some
conditions for long-term viability, where the most central components are a sound governance
structure ensuring autonomy and self-direction, and a broadly accepted commitment to academic
values. By emphasizing the role of autonomy and academic values in excellence Balderston
accepts, with some qualification, the idea that peer-judgment is established in ‘the province of
knowledgeable peers in [a] field’ (p. 352), that is excellence is ultimately what your peers value as
excellent. Excellence as the ability to attract academic ‘stars’, high levels of recruitment selectivity
and broad, but essentially collegial, consultation for resource allocation are key organizational cor-
relates to this view.

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

120 Evaluation 17(2)

However, Balderston also points out that while excellence tends towards ‘the selective, the criti-
cal, the fundamental, the cosmopolitan, the long-range’ (p. 359), there are also trade-offs which
may put some of these values into question. One is the risk of overinvesting in narrow, basic proj-
ects with long gestation periods, the uncertainty of which may threaten the survivability of an
institution. Another is the compatibility with the other values of the university, for example those
that relate to more ‘democratic’ and equitable goals such as local engagement and social improve-
ments. Consequently Balderston mentions three key institutional aims for guiding selection of
excellence programs: (a) compatibility of aims between center/program and university, (b) effec-
tiveness and mutual reinforcement of such programs, and (c) acceptability to their most important
constituencies. These values blend together an organizational and a collegial rationale for
excellence.
Balderston’s arguments are mainly relevant on a strategic university level; however, several
authors also point towards fairly concrete organizational and group aspects of excellence. For
example, Hemlin et al. (2004) in a review of a number of studies on research climate, suggest that
certain aspects of leadership are key to excellent research environments; namely, clear coordinated
objectives, excellent visionary leadership, group participation in leadership and well managed staff
selection. Similar to Balderston, they emphasize how academic culture/climate – a primary focus
on research and a genuine research culture – stimulates high quality outputs from research. In addi-
tion, internal and external communications have proven to be strong correlates to high level aca-
demic performance, as have diversity in age and background been of import.
In their classic cross-discipline study on research laboratories in the USA, Pelz and Andrews
(1966) too pointed to the importance of interaction between scientific colleagues for research pro-
ductivity, and to the role played by joint goal setting for research. Intra-organizational communica-
tion was found to be important for research productivity, and while such communication did not
necessarily have to lead to organizational consensus, a key factor for success was shared enthusi-
asm for the same type of problems. In a similar but more contemporary study on creative research
groups in nanotechnology and genetics, Heinze et al. (2009) found that extra-mural collaborations
play an even greater role for research excellence than was assumed in these previous reports.
Successful groups draw on larger collaborative networks, provide a link between disjointed peers
and work under conditions that reflect multidisciplinary contacts. Heinze et al. (2009) also point
out how a scientific actor who operates in the intersection of a diversity of research groups may
generate more original research by having a greater variety of perspectives and knowledge avail-
able (e.g. Burt, 2004). However, such a position may not be the most optimal one for diffusing new
ideas, rather cohesive collaborative groups and high trust networks seem to function best for such
purposes (e.g. Fleming et al., 2007). This naturally poses a dilemma for evaluation: ideally, both
diversity and cohesion should be present in a research group to some extent – however, the exact
trade-offs may vary. As will be seen, the empirical cases described in the present study suggest that
criteria employed by evaluators will sometimes pull in different directions on this dimension.
Excellence is also closely connected with funding outcomes. Laudel (2006) attempts to capture
some of the more important conditions for fund acquisitions, which is a key proxy for excellence.
In a qualitative interview study with German and Australian physicists, Laudel identified a number
of promoting conditions for funding relating to center/program specific aspects; namely, topic of
research (e.g. diverse funding landscape, large epistemic room for maneuver and availability of
collaborators) and quality of scientists (e.g. significance, continuity and amount of prior research,
reputation of applicants and proposal quality). Laudel notes that not all such promoting conditions
are quality related. For example, while mainstream/low risk research and know-how about fund-
raising are clearly important for fund acquisition, they hardly point directly to scientific excellence.

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

Hellström: Homing in on excellence 121

In fact, Hornbostel’s (2001) work on third party funding at German universities suggests that exter-
nal fund acquisition is only an appropriate indicator of excellence when such funding is common
in the field, when there is a qualified peer-review system operating, when there is a mix of resources
in the system, and when an essential infrastructure for research is available. External funding has
this character of cause-effect ambiguity, where it is not always clear if such funding is a good indi-
cator of excellence because (a) it actually promotes scientific quality, or (b) external income is in
fact generated by scientific quality. Both assumptions can of course be questioned, and none by
itself implies that large amounts of external funding is good for long-term excellence. Criticism of
this indicator can be raised in the case of (a) on account that external competitive funding may lead
to mainstreaming and short-termism in research (Whitley, 2007), and in the case of (b) on the basis
of observed ‘resource mediated Mathew effects’, that is where prior funding rather than quality
begets more funding (Gillett, 1991).

Values and the assessment of excellence

What types of values and qualities of research are used for assessing whether excellence is present
or forthcoming? This discussion has focused on elaborating assessment metrics as well as discuss-
ing the values which academics and others apply in trying to identify and stimulate excellence in
research. In one study, which points toward the heterogeneity of the concept, Pounder (2000)
assessed how general organizational performance indicators measuring quality may be applied to
Higher Education Institutions (HEIs). The assessment involved university academics in Hong Kong
validating a self-rating scale with regard to how an institution relates to quality in every aspect of its
operation (it may thereby be generalized to centers of excellence as well). It was found that quality
could not be represented in a unitary form: participants simply could not agree on any determining
signifiers, such as a set of institutional behaviors generating quality. On the other hand, participants
agreed on reliable scales for efficient communication/information management, planning and goal-
setting, productivity and cohesion. Pounder suggests that it may be in order to shift focus to such
indirect organizational prerequisites for quality, such as communication, rather than attempt focus
on the quality construct itself in conducting assessments. The implication is that while excellence
may not be a unitary/generalizable concept, it may be reducible to a number of constituent parts or
factors, the interpretation of which should be allowed to vary across disciplines.
Similar observations have been made with regard to benchmarking excellence across programs.
Tjissen (2003) points to the contextual nature of the concept, and how this may affect attempts to
create successful benchmarking measurements. Research programs have their own qualitative
standards in terms of scientific achievement, and not all dimensions of achievement are compara-
ble across groups. An important challenge in this regard is to identify the goal and purpose of the
organization as a starting point for selecting excellence criteria. Such analysis translates into a
stakeholder perspective where it becomes important to identify specific ‘recipients’ of excellence.
These may vary in composition and importance across programs. For example, while it may be
feasible to compare academic quality in terms of quality of publications and journals where they
appear, as well as on more tangible activities such as international advisory appointments and
prizes, items as for example the cumulative quality of a program or its originality in terms of meth-
ods and approaches are considerably more difficult to assess comparatively (Tjissen, 2003).
Ultimately, the subject dependent relationships between contextual (e.g. originality) and more gen-
eralizable (e.g. publications, citations, etc.) indicators will have to be established on the basis of
case studies and sound reasoning. In the end then, this is an internal question for science and espe-
cially for the disciplines themselves.

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

122 Evaluation 17(2)

However, Mansilla and Gardner (2006) note the difficulty in applying traditional disciplinary
values to interdisciplinary research in emerging problem oriented fields, where the lack of viable
sources for comparison may undercut traditional peer-review processes. An interview study includ-
ing several successful interdisciplinary environments revealed that scientists often related to those
disciplinary sources that allowed them to converge on an epistemic/practical goal of some kind. As
a result, the authors argue, the assessment criteria for excellence in these contexts should be more
pragmatic, and put the emphasis on standards such as viability, workability and impact of the
research, rather than for example unification or simplicity. The social and organizational corollaries
to such values may be the integration of different disciplinary traditions within one project, and the
ability to formulate and pursue new ‘hot’ topics in an unexpected way. This corresponds closely to
Longino’s (1990) ‘feminist theoretical virtues’, which include traditional epistemic virtues (or val-
ues) such as novelty, but replaces universalism with ‘ontological heterogeneity’, and introduces
‘mutuality of interaction’ as an internal value for science. Some of these values, for example ‘appli-
cability to current human needs’ and ‘accessibility’, seem to satisfy goals other than purely scientific
ones. These may be viewed as more pragmatic or political goals, however as policy and science
merge within the framework of excellence, these values are gradually taking on an epistemic guise.
Fischer et al. (2001) provide an example from practice in their study of the Canadian
Government’s development of the Networks of Centers of Excellence. They noted that the excel-
lence concept was driven by stimulation of leading edge long-term research coupled with partner-
ships and collaboration across disciplinary and sectoral boundaries (e.g. with industry). The
evaluation of the centers eventually came to focus on the quality of the research, networking and
network management, training of qualified personnel, knowledge exchange and technology exploi-
tation. Science was given 20 percent of the weighting and the emphasis was put on network man-
agement and third stream activities, which were taken to be ‘as important’ (or a bit more important
as it were) as excellence in science. The driving values of the program were multidisciplinary and
institutional diversity, research management and clear goal setting, and the use of networks to
leverage external competences. However it was noted that while scientific reviewers easily evalu-
ated the science part, they found it difficult to operationalize those other intermediary qualities.
The Canadian Networks of Excellence example, where scientific quality is reduced to a fifth of
the evaluative weighting, raises an important issue namely to what ends science is ultimately
invested in. Schmoch and Schubert (2009) argue that in adopting indicators for excellence it is
important to analyze to what extent such factors and proxies actually represent and stimulate the
ultimate goal of science, which they argue is new knowledge creation. And again the results seem
to suggest that excellence is a subject specific quality. For example in reference to the role of third-
party funding, which is a traditional proxy for assessing the quality of a research program, Schmoch
and Schubert show empirically that there is a non-linear relationship between such funding and
research performance: at a certain point a larger amount of external funding undercuts research
output, and as a result an optimal funding mix needs to be maintained. In addition, these non-linear
relationships are shown to be field dependent. So, for example among the departments studied by
Schmoch and Schubert, biotechnology successfully handled a larger amount of external funding
before dropping in performance (up to 82 percent of the funding total), while micro-economics had
a cut-off point of 47 percent (p. 13). In addition, the authors note how success in attracting external
funds may lead to a neglect of the type of infrastructure development required for sustainable
excellence. When ‘new knowledge creation’ is taken as a necessary condition for excellence, the
issue of field specific institutional mix becomes even more obvious. The question for evaluators
must then be: ‘how can excellence be assessed for this field, and under what conditions will it
prevail over time?’ Schmoch and Shubert propose the useful notion of ‘system excellence’ to

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

Hellström: Homing in on excellence 123

account for how a number of instrumental and mediating factors such as education, communica-
tion, infrastructure and other governance variables sustain excellence in a system, and furthermore
that a functional balance needs to be maintained between such factors.
This review already directs attention to some of the findings of the empirical study presented
below, at least as far as the more general dimensions of research value are concerned. We have
observed how value may be preliminarily divided into epistemic and social categories and how it
is possible to view these as endpoints in a spectrum, with some epistemic categories being more
social and vice versa. Second, the review has identified various implications of the distinction
between outcome and process categories for excellence, potential contradictions found there, as
well as how certain valued process-outcome relations are especially difficult to pin down. A recur-
rent theme among these previous studies is the relevance of a disciplinary spectrum for quality
assessments; that is the field specificity of excellence in research. In what follows the method
employed for this study will be outlined. After that, the results from the empirical cases will be
presented and discussed, at which point these aspects of evaluation of excellence will be further
elaborated and new aspects of the problematic identified.

Research approach
Empirical background
The Swedish Linneus program is an initiative of the Swedish Research Council and aims at ‘a
concerted, long-term, and strategic investment in Sweden’s leading research environments’ (VR,
2006). The aim of the Linnues Grant is further to ‘enhance support for research of the highest
quality . . . to encourage universities and colleges to prioritize research fields and to allocate funding
for them’ (VR, 2006). The mechanism for this program was to establish well-funded centers of
excellence, and support these for a duration of ten years. The program was initiated in 2005 with
the competitive selection of ten basic research environments, focused on frontline and interdisci-
plinary research of internationally high standard. In 2006 another 20 environments received sup-
port, with a combined funding of ca 140 Million Swedish Kronor (approximately 16 Million Euros)
annually for ten years. Two years later these were subject to a mid-term process evaluation. The
disciplinary composition of the evaluation committee was broad: professors of higher education
policy, economics, molecular medicine, sociology, and physics made up the group. The committee
prepared reports based on interviews, primary and secondary data such as publications and yearly
reports, as well as a hearing session. The terms of reference stipulated a focus on issues regarding
organization, cooperation, and leadership in reference to the plans presented in the program appli-
cations. In addition to these three topics, each of the evaluation reports were advised to also cover
opportunities created by the Linneus grant and strategic and international implications of the
center. This study focuses on a sample of eight of the 20 Linneus grant environments awarded in
2006. Each of the Linneuscenters is reported in a five page document, making up a total of 40
pages of evaluation text. The sample represents a cross-section of the disciplinary mix of the 20
awarded applications in terms of social science, physical, engineering and bio-medical science
respectively.

Procedure
The approach to textual analysis used here is derived from template analysis (King, 1998) and
rhetorical criticism (Foss, 2004). The procedure in this case followed a standard approach by first
going through the evaluation reports in detail, identifying evaluative statements and motivations

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

124 Evaluation 17(2)

which denoted high quality/excellence. These statements may be referred to as ‘evaluative mean-
ing units’ following the parlance of Interpretative Phenomenological Analysis (e.g. Giorgi, 1997)
or simply as ‘evaluation units’, indicating that these short designators of value represent together
the evaluators’ total designation of excellence awarded to a center. Evaluation units were created
by assigning codes in the form of short descriptive labels or simply keyword summaries to positive
evaluative statements in the report. Codes or units were then clustered into broader themes based
on commonalities identified by the researcher. Themes were re-interpreted and broken down into
lower level categories according to the same method of identifying similarities and co-extensive
qualities in the evaluation units. This resulted in seven themes depicting general focus points for
evaluative statements referring directly to or implying qualities of excellence. These themes each
had two to five sub categories, which represent more concrete operationalizations of the theme.

Evaluation of the Linneus Centers of Excellence

The following themes and categories appear as they were ‘discovered’ or constructed out of the
material, and their order has no specific meaning. Each theme is provided with a sub-section
heading, and each category is underlined in the text.

Organizational capabilities
One of the most noticeable focus points of the evaluation was the research programs’ structural capacity,
in term of how they enabled research activities to be carried out interactively and sustainably, and how
well they were set up to be responsive to outside opportunities. A central aspect of this can be referred
to as enabling structural qualities. The committee valued program features such as a flat organizational
structure which provides social ‘legibility’ to its members, but also important here was a certain scale of
operations. Growth was considered positive including the building up of a critical research infrastruc-
ture. Formally organized exchanges among members of the programs were a sign of dynamism. In the
leadership dimension, an informal, organic leadership style was considered important, as well as a flex-
ible work style among staff. At the same time attention to psycho-social aspects of work was encour-
aged. Typical quotes included: ‘This is a flat structure . . . in our view well suited to overseeing an
environment’ and ‘leadership [is] ‘organic’ in the sense that it does not require formal organization’.
Another organizational capability visible in evaluator statements was that of an organizational
structure enabling rapid responses. This included a leadership structure which could accommodate
rapid growth; that is, a capacity to scale up from the small to the large organizational format. One
interesting mechanisms pointed out by the group in this regard was the use by one program of
‘action groups’ for opportunity driven and time-limited missions, e.g. shorter research interven-
tions. This model would presumably devolve action capacity and foster a culture of collaboration.
To quote the panel: ‘[these] temporary sub-units are . . . a powerful instrument indeed for rapidly
instilling a culture of collaboration’.
Finally under this heading we find a third category: dynamic internal cooperation. One key
value here is the long-term durability of the cooperation. Historical observed continuity may be a
guide to the future; if cooperation existed in the past it is likely to continue. It was also considered
important that research spanned several environments, and that it brought together research groups
and created synergies and critical mass among groups. This was referred in one instance as a ‘cul-
ture of collaboration’. In two cases the panel illustrated this by pointing to how practical project
work had been organized based on an interdisciplinary model, and how leadership had encouraged
movement between research groups.

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

Hellström: Homing in on excellence 125

Strategic capacity
This evaluative theme considers values expressed by the evaluators concerning the capacity to
form and execute strategy, and instill direction for the program. The first category, situational
awareness, refers to the ability of leadership and measures taken to understand the strategic context
of the program. Notions such as ‘clear vision’ among leaders and dialogue between leaders and in
groups were considered valuable, for example for identifying bottlenecks to progress in research.
Closely connected to this was the ability to radically reposition when need or opportunity emerged.
One instrument in this regard quoted by the panel was the use of an advisory board to ‘evaluate the
quality of the environment’s output as the basis for considering possible options open to the envi-
ronment . . . of a strategic nature’.
A second category here concerns the executive capacity of the program. This includes leader-
ship’s ability to capitalize on unpredicted results, to allocate resources accordingly and to make
reasonable trade-offs between long- and short-term focus. One issue identified was that of the chal-
lenge to translate scientific brilliance into institutional excellence. This, the committee reflected
requires leadership skills and project management. Executive capacity also involves the governing
structure of the program. For example the evaluators note positively how one program possessed a
strong scientific advisory board with executive power to change the leadership should the condi-
tions warrant such a move. In the panels assessment a long term commitment to the program is an
important quality of its leaders, but so too is the formal capacity to shift out leadership when
needed. Executive capacity in addition included the ability to collaborate within the leadership
group and the board. However, the importance of formal structures for executive decisions was
emphasized.

Resource strategy
This theme is concerned with how resources are acquired and distributed in the organization.
External acquisition refers to how successful the organization has been in attracting additional
funding, as well as qualitative aspects of such funding, for example if the program has been the
main applicant for a grant. A diversity of funders was recurrently emphasized as positive, espe-
cially so when multiple competitive bids had been secured over time. A long-term and aggressive
strategy for funding was favorably mentioned. External funding provided by sources outside of the
national or even the EU context was pointed out as central accomplishments. One key issue was
how funds were put in use, and the qualitative differences in implications of funding. Several times
were mentioned the use of funding to stimulate cooperation, and the targeting of funds aimed at
such activities. The challenge in accessing the right types of funds for sustaining interdisciplinary,
large-scale research was mentioned by the committee in one case, and the ability to attract such
funding indicated prominence. The following assessment is telling: ‘The strategy that the leader-
ship has drawn [for funding] for the long term is thus aggressive and proactive . . . the coming years
will see increasing attention paid to ‘grantsmanship’.
Finally, internal allocation relates to structures and procedures for evaluating and distributing
resources within the program. A few times positive mention is made of internal competitive or
quasi-competitive’ project funding, i.e. resource allocation made on the basis of proposals.
Evaluation dimensions mentioned in one case included: scientific quality, long-term sustainability,
involvement with other researchers in the center. Yet another interesting observation by the com-
mittee related to when internally funded projects were required to be run by members from at least
two faculties, this in order to ensure multi-disciplinary involvement.

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

126 Evaluation 17(2)

External cooperation and networking

This theme is central to all the assessments in the evaluation and it comprises a broad spectrum of
activities. The evaluators put an emphasis on high quality partners to the programs: that is high
level of selectivity in partnering was appreciated (for example with other centers of excellence),
and particularly international (EU and overseas) connections. The standing of the universities
involved was an issue which was recurrently mentioned to in the assessments, and diversity of
network partners in terms of disciplines and types of institutions was also positively noted. A typi-
cal assessment was that ‘international cooperation strikes us as being high level and naturally
selective’.
However, cooperation was also expected to generate sustainable partnerships. It was positively
remarked on when international connections showed stability over time; this was taken to reflect
the reputation of the program and the actors involved. Long-term ties with other successful envi-
ronments were seen as indications of quality. The fact that many of these network ties were ‘inher-
ited’ from seniors active in the environment was taken as a sign of long-term viability. Longer term
agreements for example concerning faculty and student exchange were viewed favorably. The
category operational partnerships indicates a focus on the activity level of external network con-
nections, that is whether centers were active in setting up and running partnerships, and if the
cooperation was ‘live’ in the sense of actual joint research being conducted. It also discussed how
networks were used to access resources, such as new knowledge and talent. One aspect of this
category relates to what if anything makes an environment attractive to potential partners, for
example if they have access to useful resources, databases or the like. Two ways of making partner-
ships more operative were to create dual appointments with scholars from other environments
(adjunct professorships) and enabling doctoral student exchanges.
Cross-sectoral cooperation was pointed out as one important network activity for the centers of
excellence. The assessments mainly took this to refer to industry projects (in Sweden as well as in
Europe and overseas) and university-industry linkages in general. This category was only noted for
two programs however, and did not figure as much as one could have expected.

Human capital base

This theme relates to talent and competence present in the research group, and to mechanisms
ensuring a viable human capital base. The committee recurrently commented on the presence and
need for academic leadership competence, in order to ensure the sustainability of the programs.
Some of the centers were commended on having secured such leadership competence via an advi-
sory board of internationally renowned scholars, who could be drawn upon for governing the
center. Closely related was the issue of securing a smooth transition in academic leadership when
seniors retired. The importance of continuity in leadership, also the more informal type represented
by senior colleagues, was addressed several times, and centers having integrated this concern into
their strategy were commended, e.g. ‘the center’s development plan takes this [retiring seniors]
into account . . . that the ‘changing of the guard’ is anticipated shows there is also continuity in the
midst of change!’.
As an addendum to this was the question of a conscious and worked through recruitment policy.
A proactive recruitment strategy for the center is vital, as pointed to several times by the evaluators.
Recruitment policies have to be proactive, have a high level of ambition, utilize the center’s
strengths and compensate its weaknesses. One way that several centers achieved this was to secure
access to talent. This is broader than simple recruitment and relates to such issues as the types of

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

Hellström: Homing in on excellence 127

scholarly degrees that exist among staff, the utilization of network partners to access talent, the
conscious attempt to recruit outstanding young PI’s, and utilizing the education system to prime
students for future recruitment. It also related to the international spread of applicants to new
positions - that is how widely a center was able to throw its net in recruiting new talent. For example
in reference to the use of network partners, one center was applauded on ‘[its] clear use . . . of these
long-term relationships in recruiting its younger and highly promising talent’.

Epistemic capability
This theme covers five categories of appraisal. Knowledge access refers to a center’s ability to gain
access to outside knowledge, when it is needed, for example by developing it in-house or in other
ways connecting competence to the centre. Related to this is a supply side dimension, where the
center maintains capabilities, for example in data processing, that make it attractive to others. Such
exchange of capabilities may be a key requirement for leveraging the center’s knowledge resources,
as pointed out by the committee: ‘Cooperation is . . . the only way that permitted the rapid exploita-
tion of the sheer volume of information . . . ’. As second type of epistemic capability is the intel-
lectual coherence of the research program. Intellectual coherence may be indicated by previously
successful projects and the way that the group has built up its strength in established areas of excel-
lence and then added new areas incrementally. Good dialogue between leaders and the group with
regard to pertinent problems, bottlenecks in research, etc. was also considered key. One example
of this brought out by the evaluators was the establishment of identifiable ‘subjects of consolida-
tion’ and ‘forward thrust’ subjects, i.e. where a group selects what issues to gather around and what
future problems to pursue. This active social formulation of research topics requires the center to
develop learning mechanisms. Such mechanisms can consist of internal ‘training’ in the contribut-
ing research traditions; for example one center received acclaim for having developed ‘journal
clubs’, or reading groups, where researchers could develop understanding across subject lines.
Another mechanism for learning was the use of an active scientific advisory board to identify
future research options and to evaluate output in two-year cycles.
Multi-disciplinary involvement was a recurrent issue throughout the evaluation. Cross-disciplinary
exchanges were applauded, and movement between groups encouraged where it occurred. If a center
spanned several environments, it was taken up for positive mention especially if such interplay could
be concretized through one organizational activity or the other, e.g. ‘Each platform is managed by a
senior and a junior researcher, each from a different disciplinary background, a judicious decision
since it takes fully into account the strategic importance of sustaining interdisciplinarity.’ Finally
the issue of intellectual synergies relates to how existing groups were conjoined to achieve critical
mass, especially where such criticality was clearly a result of the program. In addition, if such cross-
fertilization of ideas is still happening even after long-term cooperation, this was viewed as positive.
Concrete yet academic outcomes of cooperation were positively noted, e.g. ‘. . . the rapid cross-
fertilization of ideas between the groups participating [led to] new areas of expertise emerging’.

Research-teaching nexus
The final theme relates to statements on how the centers connected their research activities to
undergraduate and graduate education. The evaluators took positive notice of educational activities
involving cross-faculty cooperation leading to, for example, new teaching tracks. This was sug-
gested to increase environmental visibility and to create bi-directional influences between teaching
and research. Graduate training/talent development in particular relating research training carried

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

128 Evaluation 17(2)

out by faculty received encouraging mention. Forms of researcher-student synergies were appreci-
ated, and viewed positively as an instantiation of the unity of science and education. For example
the primacy of a ‘Humboldtian nexus’ was mentioned in one case, and in other instances the con-
nection between research and undergraduate education was suggested to be important for the long-
term perspective. To quote from one of the assessments: ‘finally we are pleased to note an equally
significant component in the environment’s long term development, namely, the attention paid to
developing a preparatory programme driving down to undergraduate level, aimed at students hav-
ing the will to embark on research training in [this] demanding field’.

Homing in on excellence: A contingency approach for evaluators?

Previously it was suggested that the excellence concept might be a useful boundary object, due to
its flexibility and adaptability, for both academics and policy makers interested in science gover-
nance. The test of a boundary object is that it offers some ground for constructive discussion via a
shared framework, which does not offer a clear advantage to any one party (it might offer a hidden
or latent advantage to one group, which may be revealed through critical discourse analysis. Such
analysis is not the main purpose of this article, however). The strength of the concept seems to be
that both academics and policymakers want science to do well. Excellence in scientific research is
rewarded by the academic community as part of its basic reward structure (cf. Merton, 1973: ch.
4), and policymakers can see in excellent science a means to other things such as industrial pros-
perity, national prestige and increasing educational levels. In the latter case scientific excellence is
benefited by policymakers having a policy theory for science which can be summarized, regardless
of intermediary variables, as good science yields good society, i.e. some version or the other of the
still so popular ‘endless frontier’ view on science, inaugurated by Vannevar Bush (1945). However,
there are complicating factors here which, if context is disregarded, may generate conflict. First, on
the side of science, the concept of excellence that is becoming dominant, and which is partly exem-
plified by the above case, seems to favor a specific type of governance and organizing of science
– one which may not function well across all subjects. A second and larger issue is that while the
academic community tends to reward excellence in research, the current policy use of the concept
shifts focus to the achievement of a set of hypothesized conditions for greatness rather than actual
academic performance. The viability and secondary effects of moving from output to process eval-
uation in science is far from clear and needs to be analyzed. The results of this study suggest a few
preliminary answers to these two problems.
The first issue is related to how effective a boundary object the process/organizational view of
excellence exemplified in this study is for science itself. Many of the qualities brought forward in
the appraisals discussed above, for example the ability to respond rapidly on the organizational
level, situational awareness among leaders, intellectual coherence, stable and formalized networks
and coherent recruitment policies, are more easily realized in some environments rather than oth-
ers. To use Whitley’s (2007) classification, these qualities would probably be most applicable in
fields with a low variability of research goals and approaches, low diversity of audiences, high
centrality of prestige fields and a cohesive elite structure. In other words, by emphasizing such
qualities the excellence concept would reward mature natural science programs with the potential
for economies of scale (big science). As suggested by Schmoch and Schubert (2009), however, the
diversity of external funding sources applauded by the reviewers in this study, would probably not
be a positive indicator for at least some of the environments. Programs closer to economics where
productivity trails off at about 47 percent of external funding should be evaluated differently in this
regard than programs closer to physics and biology which successfully handles a higher proportion

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

Hellström: Homing in on excellence 129

of external funding (see above). On the other hand, other qualities pointed out by the reviewers, for
example informal leadership, flexible work style, and transdisciplinarity may be more successfully
served by disciplines with heterogeneous elite structures, high people to problem ratios and high
variability of research goals and approaches, i.e. those subjects on the opposite side of the ‘disci-
plinary spectrum’ from the potentially large scale natural science programs. Yet again, certain
qualities would be easier to realize in the typical engineering subjects, or what Becher (1994) refers
to as entrepreneurial, pragmatic or purposive disciplines. One may speculate that such fields would
likely stand a good chance to realize qualities such as cross-sectoral cooperation, multi-disciplinary
involvement and strategic capacity, while faring less well on intellectual coherence. The key point
here is that not simply because excellence is equally desirable across all these fields does it mean
that once one starts to operationalize the concept on a more detailed level, will every field have the
same chance to comply across the spectrum, nor that they should attempt to do so. We may refer to
this as an ‘epistemic contingency’ requirement on excellence to which evaluators must pay heed.
The second issue relates to what type of policy theory underlies the excellence concept as
espoused here. The question in point is the viability of viewing excellence as a mix of process and
output variables, where the process variables can be awarded higher import. Pounder (2000)
wanted to raise this shift towards processes to a norm by suggesting that quality should be assessed
on organizational dimensions rather than on academic outputs. Some authors like Pounder and also
Fischer et al. (2001) seem to suggest that the trend towards organizational/process focus and away
from academic outputs/outcomes, responds to what policy finds manageable or transparent, that is
organizational variables are more generally accessible to policy actors than cognitive ones. In a
sense such an interpretation would assume that management, networking etc. are generalizable or
neutral competences, which can be applied and assessed regardless of the scientific content of an
activity. In light of previously reviewed research such generalizability of process variables is most
likely more illusion than reality (Hornbostel, 2001; Lamont, 2009; Mansilla and Gardner, 2006;
Pounder, 2000; Tjissen, 2003). If policy actors would start promoting certain research because it
has epistemic affinity with a particular organizational model, because available assessment tools
yield ‘better results’ with those organizations, that would certainly be a dead end for research
evaluation. And while it is clear that many academically successful environments tend to display
specific cultural or organizational features, it is far from obvious that these features generate aca-
demic success by themselves.
Based on observations from the Linneus appraisals, however, one may want to argue that some
aspects of excellence are closer to academic values (values realized through a research activity)
than to general organizational or management processes, and again some features are clearly more
ends than means oriented. Looking at the themes in the order presented above, under organiza-
tional capabilities, ‘dynamic internal cooperation’ seems to come closest to the epistemic dimen-
sions and in terms of strategic capacity ‘situational awareness’ is closer to the core of scientific
activity than is ‘executive capacity’. With respect to resource strategy, the ‘internal allocation’
system is closest to the research/organizational nexus; the human capital base shows similar fea-
tures, with ‘academic leadership’ and ‘access to talent’ on the research side of the scale and ‘leader-
ship transition’ and ‘recruitment policy’ on the organizational side. Epistemic capability is generally
closer to academic than to organizational content, and so are the issues concerning the research-
teaching nexus. The point of this simple exercise is to show how the indicators for excellence –
while clearly organizationally framed – can be used to put emphasis on epistemic content or on
intermediary organizational factors. The choice will be contingent on, among other things, the
purpose of the program, its institutional and its disciplinary context. Furthermore, it serves as an
illustration of the importance of being conscious of what side of the epistemic-organizational scale

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

130 Evaluation 17(2)

one is operating as an evaluator, and that one side (i.e. the organizational) may be more sensitive
to disciplinary and other types of differences predicated on field and interests. That being said, it is
only by disaggregating the excellence concept in the way illustrated by this case, into operational
and detailed items of evaluation, that such a contingency approach can be realized.

References
Atkinson-Grosjean J (2006) Public Science, Private Interests: Culture and Commerce in Canada’s Networks
of Centres of Excellence. Toronto: University of Toronto Press.
Balderston FE (1995) Managing Today’s University: Strategies for Viability, Change and Excellence. San
Francisco, CA: Jossey-Bass.
Becher, T (1994) The significance of disciplinary differences. Studies in Higher Education 19(3): 151–61.
Burt RS (2004) Structural holes and good ideas. American Journal of Sociology 110(2): 349–99.
Bush V (1945) Science the Endless Frontier: A Report to the President by Vannevar Bush, Director of the
Office of Scientific Research and Development, July 1945. Washington, DC: United States Government
Printing Office.
Doppelt G (2008) Values in Science. In: Psillos S and Curd M (eds) The Routledge Companion to Philosophy
of Science. London: Routledge, 302–13.
European Commission (2008) A Strategic European Framework for International Science and Technology
Cooperation. Brussels, 24.09.2008, COM(2008) 588 final.
Fischer D, Atkinson-Grosjean J and House D (2001) Changes in academy/industry/state relations in Canada:
the creation and development of the Networks of Centers of Excellence. Minerva 39: 299–325.
Fleming L, Mingo S and Chen D (2007) Collaborative brokerage, generative creativity and creative success.
Administrative Science Quarterly 52: 443–75.
Foss SK (2004) Rhetorical Criticism. Long Grove, IL: Waveland Press.
Fuller S (2000) The Governance of Science: Ideology and the Future of the Open Society. Buckingham: Open
University Press.
Gillett R (1991) Pitfalls in assessing research performance by grant income. Scientometrics 22: 253–63.
Giorgi A (1997) The theory, practice, and evaluation of the phenomenological method as a qualitative research
procedure. Journal of Phenomenological Psychology 28: 235–60.
Goldman AI (1999) Knowledge in a Social World. Oxford: Oxford University Press.
Heinze T, Shapira P, Rogers JD and Senker JM (2009) Organizational and institutional influences on creativ-
ity in scientific research. Research Policy 38: 610–23.
Hemlin S, Allwood CM and Martin BM (eds) (2004) Creative Knowledge Environments – The Influences on
Creativity in Research and Innovation. Cheltenham: Edward Elgar.
Hornbostel S (2001) Third party funding of German universities: an indicator of research activity?
Scientometrics 50: 523–37.
Jackson DN and Rushton JP (1987) Scientific Excellence: Origins and Assessment. Newbury Park, CA:
SAGE.
King N (1998) Template analysis. In: Symon G and Cassell C (eds) Qualitative Methods and Analysis in
Organizational Research. London: SAGE.
Kuhn T (1977) The Essential Tension: Selected Studies in Scientific Tradition and Change. Chicago: Chicago
University Press.
Lacey H (1999) Is Science Value Free? Values and Scientific Understanding. London: Routledge.
Lamont M (2009) How Professors Think. Inside the Curious World of Academic Judgement. Cambridge, MA:
Harvard University Press.
Laudel G (2006) The ‘quality myth’: promoting and hindering conditions for acquiring research funds. Higher
Education 52: 375–403.
Longino H (1990) Science as Social Knowledge. Princeton: Princeton University Press.
Longino H (2002) The Fate of Knowledge. New Jersey: Princeton University Press.

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015

Hellström: Homing in on excellence 131

Mansilla VB and Gardner H (2006) Assessing expert interdisciplinary work at the frontier: an empirical
exploration. Research Evaluation 15(1): 17–29.
Merton RK (1973) The Sociology of Science: Theoretical and Empirical Investigations. Chicago: University
of Chicago Press.
Pelz DC and Andrews FM (1966) Scientists in Organizations: Productive Climates for Research and
Development. New York: John Wiley and Sons.
Pounder JS (2000) Evaluating the relevance of quality to institutional performance assessment in higher edu-
cation. Evaluation 6(1): 66–78.
Schmoch U and Schubert T (2009) Sustainability of incentives for excellent research – the German case.
Scientometrics 81(1): 195–218.
Tjissen RJW (2003) Scoreboard of research excellence. Research Evaluation 12(2): 91–103.
VR (2006) Press releases 2006-07-07. URL (consulted 24 July 2009): http://www.vr.se
Whitley R (2007) Changing governance of the public sciences. In: Whitley Rand Gläser J (eds) The Changing
Governance of the Sciences. Dordrecht: Springer, 3–27.

Tomas Hellström is a professor of innovation, entrepreneurship and knowledge creation at Lund University,
Sweden. He has a PhD in Theory of Science from Gothenburg University, and focuses on the management of
research organizations, innovation processes and evaluation methodology. Please address correspondence to:
Center for Innovation, Research and Competence in the Learning Economy (CIRCLE), Lund University, Box
117, SE-221 00 Lund, Sweden. [email: tomas.hellstrom@circle.lu.se]

Downloaded from evi.sagepub.com at FUDAN UNIV LIB on May 8, 2015