2015 Technopolis Group Guide To Impact Assessment of Research Infrastructures

2015
Evaluating and Monitoring the SocioEconomic Impact of Investment in

Research Infrastructures
www.technopolis-group.com
Evaluating and Monitoring the SocioEconomic Impact of Investment in Research

Infrastructures
technopolis |group|, 2015
Elina Griniece, Alasdair Reid and Jelena Angelis
www.technopolis-group.com
Table of Contents
1. Introduction
1.1 Why evaluate the socio-economic impact of research infrastructures?
1.2 Previous evaluations of the impact of research infrastructure investment
1.3 The aim of this guidance document
2. Logical framework for socio-economic impacts of investment in research

infrastructure
4
2.1 Design and construction phase
2.2 Operational phase
3. Impact assessment methods and tools
13
3.1 Impacts on economy
13
3.2 Impacts on human resource capacity
14
3.3 Impacts on innovation
14
3.4 Impacts on scientific activity
15
3.5 Impacts on society
15
3.6 IA implementation process, its challenges and limitations
17
Bibliography
18
Table of Figures
Figure 1: Typology of research infrastructures ................................................................ 1
Figure 2: Stakeholders and beneficiaries of investment in research infrastructure ....... 2
Figure 3: Logical framework for socio-economic impact assessment of investment in
research infrastructure ..................................................................................................... 5
Figure 4: Indicators for economic impacts of RI construction phase ............................. 6
Figure 5: Indicators of innovation impact of RI construction phase .............................. 7
Figure 6: Indicators for economic impact of RI operational phase ................................. 8
Figure 7: Indicators for impact of RI operation on human resource capacity ................ 9
Figure 8: Indicators for impacts on innovation of RI operational phase ...................... 10
Figure 9: Indicators for impact on scientific activity ..................................................... 12
Figure 10: Indicators for impacts on society .................................................................. 13
Figure 11: Implementation of an impact assessment of research infrastructure .......... 16
1. Introduction
1.1 Why evaluate the socio-economic impact of research infrastructures?
Research infrastructures (RI) refer to facilities, resources (including human) and
related services needed by the research community to conduct research in any
scientific or technological field. Research infrastructures include:
Major equipment or group(s) of instruments used for research purposes;

Permanently attached instruments, managed by the facility operator for the
benefit of researchers, industrial partners and society in general;
Knowledge-based resources such as collections, archives, structured
information or systems related to data management, used in scientific
research;
Enabling information and communication technology-based (ICT) or einfrastructures such as grid, computing, and software communications;
Any other entity of a unique nature that is used for scientific research.1
Due to the large number of research communities and complex research needs, there
are very different types of research infrastructures with specific characteristics. Figure
1 provides an overarching typology of RIs.
Figure 1: Typology of research infrastructures
Type of research
infrastructure
Description
Examples
Single-site facility
Unified body of
equipment at one physical
location
High-performance laser system,

clean room, coastal observatory,
centre of competence
Distributed facility
Network of distributed
instrumentation or
collections, archives and
scientific libraries
ELI: European Light

Infrastructure; Council of
European Social Science Data
Archives
Mobile facility
Mobile vehicles specially

designed for scientific
research
Research vessels, satellite and

aircraft observation facilities
Virtual facility
ICT-based system for

scientific research,
including high-capacity
communication networks
and computing facilities
European Grid Computing

Infrastructure; Digital Research
Infrastructure for the Arts and
Humanities (DARIAH)
Setting-up, or renovating, RI usually requires a considerable level of financial

investment and a long-term operation strategy. For example, in the environmental
sciences, many research infrastructures bring their scientific return only after decades
of sequential data recording. The investment strategy requires careful planning of the
operation phase and possible future reinvestments. This entails the purchase of
technologically advanced equipment, clustering of specific skills to enable its
deployment and devising appropriate governance structures. RIs can also be closely
linked to other research and innovation establishments, such as vocational schools,
universities, private and public research centres, research hospitals, business
incubators and science and technology parks.
1 Definition used by the European Commission
Evaluating the socio-economic impact of research infrastructures

Technopolis Group, 2015
While RIs are designed for research needs, the impacts of these facilities reach beyond
fuelling scientific excellence. The advanced technical opportunities and the
concentration of skilled human capital and know-how can foster innovation, create
new or expand the existing markets, attract inward investment, increase economic
activity and potentially have an impact on the social and cultural life in a particular
region. In this regard RIs can be viewed as focal points for continuous interaction
between scientific, technological and socio-economic development.2
The construction and operational phase of RIs are largely funded from public budgets
either through national funding from one or several countries, or a mix of national and
EU funding. For this reason it is crucial to understand the return on investment in RI
to support informed decision-making. Yet it is difficult to quantify and understand this
return in conventional commercial terms. Investment in RI brings a broad range of
benefits that spreads across wider society rather than serving merely the direct
stakeholders (owners and users of RI). Official statistics do not sufficiently describe
the variety of benefits associated with the development and, more importantly,
exploitation of RI. It calls for more elaborate and fine-tuned approaches to account for
the impacts that the RI investment brings on science, economy and society.
Figure 2: Stakeholders and beneficiaries of investment in research infrastructure
1.2 Previous evaluations of the impact of research infrastructure investment

Currently there is no unified framework for the impact assessment (IA) of investment
in research infrastructures. Various conceptual frameworks exist in parallel
comprising a range of observable direct and indirect effects and longer-term impacts.3
The existing studies focus on three types of impacts: 1) the direct and indirect
economic benefits of spending large amounts of public money in a single location; 2)
the industrial knowledge spillovers realised by contractors that design, build and
equip research facilities and spin-offs that provide specialist technical services back to
the facility (or other facilities); and 3) the local economic effects and high-technology
Rizzuto, C. (2012) Benefits of Research Infrastructures beyond Science, presentation at ERF Workshop
The Socio-Economic Relevance of Research Infrastructures, 31 May-1 June 2012, Hamburg
3 See, for example, project Research Infrastructures: Foresight and Impact (RIFI), project on Evaluation of
Research Infrastructures in Open innovation and research systems (EVARIO), Research Infrastructure
Group of UK Science & Technology Facilities Council, Czech Metodika: Evaluation Methodology for
Research Infrastructures
2

clusters that grow up around some of the larger facilities.4 A heterogenous set of
methods is applied to capture these effects of RI. Most of them address standard
economic impacts (direct effects) and to some extent economic multipliers.5
Comprehensive and methodologically demanding studies are still rare. Core aspects of
RI benefits, such as their impact on human and social capital formation and
innovation, are not extensively explored. Up to date, the existing literature provides
insufficient evidence to support claims that investment in RI (even large-scale) attract
and retain talent and promote innovation.6 Therefore, important socio-economic
contributions remain poorly understood. In this respect, significant work is
undertaken by FP7 project Research Infrastructures: Foresight and Impact (RIFI). A
Foresight Enriched Research Infrastructure Impact Assessment Methodology
(FenRIAM)7 was developed during the project, which significantly contributed to the
understanding of the impact of RI on learning and capacity of RI operators, suppliers
and users. A key lesson is that there is a need to treat the impact pathways in greater
detail and substantiate them with empirical evidence.
Most previous concept papers and evaluations have been devoted to studying largescale research infrastructures, or so-called big science facilities that are of European
and international significance. Empirical work on investment in mid-size
instrumentation of national and macro-regional significance barely exists. This is
mainly the result of the fact that RI investments have not yet been subject to regular
evaluations. There is a scope for further methodological developments and their
applications in evaluations to evolve from narratives of socio-economic effect of RIs to
more empirical models and measurement techniques. This knowledge would help to
move from simple ex-post detection of intended and unintended returns of RIs to a
better understanding and planning of future investment.
1.3 The aim of this guidance document

The aim of this guidance document is to disentangle and characterise the concrete
benefits of RI investment for different stakeholders and to build a schematic impact
assessment framework that can be used in evaluations to trace the core impact
pathways. Section two introduces a logical framework for the impact assessment of RI.
Recognising that each RI is embedded in specific socio-economic conditions, there will
never be a one-size fits all approach to mapping all socio-economic impacts. This
work should be regarded as a step towards defining a typology of possible effects and
accounting for the conditions and patterns that enable their creation and diffusion.
The third section elaborates on the methods and tools that are suitable for the
measurement of respective impacts. The identified impact pathways involve complex
and multidimensional phenomena, which in most cases cannot be captured with
standard quantitative indicators, but require a thorough triangulation of quantitative
and qualitative evidence. The section elaborates on the appropriate indicators for each
type of impact and provides additional guidance on the respective data requirements,
their potential sources and advice on collection routines. Reflections on the limitation
and challenges of the application of this IA framework are outlined in conclusion.
4 Simmonds, P. et.al (2013) Big Science and Innovation. Report for the UK Department for Business,
5
Innovation & Skills

Multipliers - Further economic activity (e.g. jobs, expenditure or income) associated with the inputs to or
outputs from the project. According the European Commission (DG REGIO) an income multiplier is a
secondary effect resulting from increased income and consumption generated by the public intervention
(investment). Multiplier effects are cumulative and take into account the fact that part of the income
generated is spent again and generates other income, and so on in several successive cycles. In each cycle,
the multiplier effect diminishes due to purchases outside the territory. The effect decreases much faster
when the territory is small and when its economy is open.
6 Horlings, E. et.al (2012) The societal footprint of big science. Report of the Rathenau Instituut, the Hague,
7
the Netherlands
http://www.fenriam.eu/

2. Logical framework for socio-economic impacts of investment in

research infrastructure
Figure 3 provides a schematic overview of the various direct and indirect benefits that
arise during two main periods of a research facilitys lifecycle:
1) design and construction phase; and
2) operational phase of research facilities.
The specificities of each of the pathways and their contributions to other impacts that
are singled-out in this two-dimensional graph are described in detail in the following
sub-sections of this chapter. The types of impacts are positioned in the graph to
visually reflect better their relation to each other, not an order of their importance.
Societal impacts can be of very diverse nature and lead to broad, overarching
outcomes. In order to provide a concise graphical representation of IA framework
these impacts are not schematically reflected in this graph.
This IA framework is predominantly designed for single-sited and geographically
confined national or macro-regional distributed research infrastructures. With regards
to highly distributed and virtual RI case-specific a range of additional factors have to
be considered. These types of RI usually are decentralised administratively and
financially and include much larger geographical areas of impact. They can have
mainly or only virtual access without the requirement to cover physical support to
individual users. Advanced ICT-based technical solutions and well-designed access
rules may ensure much larger outreach with dispersed impact on user and stakeholder
groups. These characteristics of highly distributed and virtual RI require more casespecific approach to impact assessment.
2.1 Design and construction phase

2.1.1 Economic impacts
During the design and construction phase a wide range of technical and scientific
knowledge is applied to set-up the required facilities. The physical construction and
refurbishment of buildings and their accompanying networks and utilities engages
many companies ranging from construction material suppliers to technical and legal
service providers. The procurement of core construction services typically involves
local companies8, which creates additional revenue and increases temporary
employment opportunities in the region. Despite the fact that the impact of many of
these expenditures is usually confined and short-lived for smaller-scale RI, there can
be a significant further multiplier effect on the local economy, for example, due
improved reputation credentials for firms that supply high-tech facilities and an
increase in the local tax base. In-depth analysis of the local industrial base and
mapping of relevant global supply chains is required to scope the total economic effect
of the RI investment.
Locally-owned businesses, franchise or corporate branch operating within a local area

Figure 3: Logical framework for socio-economic impact assessment of investment in research infrastructure
Evaluating the socio-economic impact of research infrastructure

Figure 4: Indicators for economic impacts of RI construction phase

No
Indicators
Explanatory remarks and examples
1.
Number of commercial suppliers

for RI design and construction
phase
Suppliers can be mapped per sector, field

of activity, size, technology group (lowtech/high-tech), geographic scope
(national/international/multinational)
2.
Scale of commercial suppliers

turnover increase due to RI
For applied method to estimate utilities in

monetary terms see the study Economic
utility resulting from CERN contracts9
3.
Scale of commercial suppliers

employment increase due to RI
Measured in FTE and grouped per

category of employees (e.g. technical staff,
R&D staff)
2.1.2 Impacts on innovation

Research facilities under construction may require specific design and building
solutions to serve the purposes of a particular research field (e.g. laboratories, clean
rooms, etc.). Such requirements foster collaboration among the involved suppliers,
facility mangers and scientists in developing innovative design solutions and building
functionalities. Local companies can benefit from accumulated know-how and
complementary skills that expand their competitive advantage in other markets (e.g.
enable winning international procurements of similar research facilities, secure service
contracts as external consultants to other developers). In cases where foreign
companies demonstrate a superior skills base and are selected to address particular
design and construction tasks, the benefits of the investment flow out of the national
economy. Moreover, in the RI construction phase, the universities and responsible
government agencies may acquire additional and more sophisticated project
management skills that can have knock-on effects on the potential for public sector
innovation.
The purchase of the necessary scientific instruments and supporting equipment is
generally open to international bidders, as the supplier base for such specific
commodities and accompanying services is often very specialised and, in certain cases,
the specific equipment may not exist off-the-shelf on the market. Hence, the
procurement of equipment can have a considerable impact on extending the
technological frontier and fostering innovation10. As a result, the impact from this
procurement will be distributed across specific global supply chains for scientific
instrumentation.
Bianchi-Streit, M. et al (1984) Economic utility resulting from CERN contracts (second study). Report of
the European Organisation for Nuclear Research (CERN)
http://indico.cern.ch/event/66952/contribution/0/2/material/paper/0.pdf
10
For the theoretical background of this nexus see work of von Hippel, E. (1976) The Dominant Role of
Users in the Scientific Instrument Innovation Process, Research Policy 5 (3)

Figure 5: Indicators of innovation impact of RI construction phase

No
Indicators
1.
Number of joint
development activities
with suppliers
The novel technical requirements for the

construction (design or building techniques) of
scientific facilities or the equipment require
close interaction between the recipient
organisation and one or more suppliers. This
can be realised through a process of precommercial public procurement11, design
contest, a forward commitment procurement or
a competitive dialogue. In these cases, the
recipient organisation defines specific technical
requirements or criteria to be met but not the
precise solution (in the form of a specific piece
of equipment, etc.).
2.
Number of contracts
concluded for high-tech or
specialist services that
require development, or
calibration of
designs/equipment to
meet specific requirements
The focus should be on the extent to which

specific design or technical requirements for
scientific buildings or equipment induce an
impact on learning and skill development
amongst supplier firms, notably local (national)
firms.
2.2 Operational phase

Two main categories of impacts of the operational phase of RI exist: 1) impacts from
the routine operation, maintenance and upgrading of RI; and 2) impacts from the use
of research facilities.
2.2.1 Economic impacts

Benefits from RI operation include the assessment of the performance characteristics
of the research facilities. This includes a review of RI performance addressing aspects
of quality, access, reliability of the infrastructure, account of all the services provided
and assessment of the optimal usage of equipment. Among the criteria to review is the
flexibility of staff to support external users, feasibility of contractual arrangements for
granting access, quality of accompanying services (e.g. tech transfer, incubation
services). Performance aspects have a direct impact on the efficiency and effectiveness
of RI operations, user satisfaction, and consequently on the generated revenues to RI
and its users.
Maintenance and operation of RI involves longer-term effects on employment in
higher education establishments, science parks and companies, such as additional jobs
for scientists, technicians, administrative and support personnel working on the RI.
The operation of facilities also includes expenditure on goods and services and routine
upgrades with additional procurement design and associated equipment. These
activities have a further multiplier effect on the local economy and relevant global
supply chains.
11
http://www.innovatiefaanbesteden.be/files/409

Figure 6: Indicators for economic impact of RI operational phase

No
Indicators
1.
Number of scientists,
students, state-owned or
private enterprises that
benefitted from RI services
Service can be categorised into broad types,

such as: free or for a fee access to equipment
(including technical support from qualified
technicians), testing or metrology (carried out
by the RI staff in laboratories or using mobile
equipment), contract research, consultancy,
hosting doctoral students, etc.
2.
Total amount of funding

generated from services,
grants and joint projects
Per service type, source of research grant and

type of joint project
3.
Number of new directly

and indirectly created jobs
Per category (scientific/ technical/

administrative) and wage level; the aim is to
scope how RI impacts on an increased
employability of human resources.
4.
Total amount of
expenditure on personnel,
operations, maintenance
5.
Total RI capacity
utilisation
RI capacity utilisation
external business users
Financial sustainability of
RI
For centralised large-scale facilities the rule of

thumb is to calculate around 10% of investment
costs for annual operation expenditure. Yet for
most other types of RI this percentage is
significantly higher.12 For distributed RI there
are higher costs due to coordination,
management and consultation tasks requiring
qualified personnel. Operating costs for RI in
the humanities and social sciences are usually
the most significant expenditure.13
RI access hours used as % of the total available
access time
RI access hours used by business entities as % of
the total available access time
Measured as % of the total costs funded from
the provided services, received grants and
realised joint projects
6.
7.
2.2.2 Impacts on human resource capacity

The build up of human resource capital is a major benefit of RI. Modern research
facilities can serve as magnets to retain and attract talent. This includes local
researchers, technicians, students and global talent through recruitment of permanent
staff and the promotion of the flow of visiting researchers and foreign students.
RI has a crucial role to play in training and skills development. Research facilities
concentrate skilled staff that hold tacit knowledge how to operate them. Only with the
help of skilled technicians and experienced researchers that students and other
interested stakeholders can learn to set up experiments and interpret their results.
This tacit dimension of knowledge transfer makes RIs entry points into networks of
knowledge, expertise and practice. To determine the impact of RI on skills
development it is important to explore how the exchange between the younger staff
Wissenschaftsrat (2013) Report on the Science-Driven Evaluation of Large Research Infrastructure
Projects for the National Roadmap (Pilot Phase)
13 Wissenschaftsrat (2011) Recommendations on Research Infrastructures in Humanities and Social
Sciences
12

(MAs, PhDs, post-docs) and experienced researchers and technicians is structured and
promoted. Equally important is to appraise the ways tacit knowledge is transferred to
private sector users.
The diffusion of knowledge gained by the RI management, operators and users may
have a significant societal value. For example, the research commercialisation skills
acquired by the management and scientists at a RI is an important benefit with farreaching implications on the local innovation system. Closer formal and informal
social networks increase interpersonal trust and knowledge sharing. This consequently
augments the quantity and diversity of knowledge that is available to RI users. Social
networks create new forms of interaction among the actors of the innovation system,
stimulate learning environments and increase the awareness of users (potential and
existing) about the scope of knowledge that is available on RI. In short, social capital
enables the full exploitation of accumulated human capital.
The mobility of people and emergence of new cooperation networks is a major
mechanism through which knowledge circulates and diffuses the benefits to wider
economy. Graduates leave RI with the knowledge of the most recent scientific results,
skills using advanced instrumentation and ability to apply them to complex problem
solving. Internships and recruitment of graduates mean that students act as a form of
social glue and they often create long-term links between scientists and businesses.14
However, if the local academic and industrial environment does not provide sufficient
absorption capacity for these advanced skills, the accumulated human capital may
remain underutilised or emigrate.
Figure 7: Indicators for impact of RI operation on human resource capacity
No
Indicators
1.
Number of new jobs for

research and technical staff
attracted from abroad as %
of the total number of staff
employed on RI
Per category (permanent/visiting) and per year;

the aim is to understand the influx of new
knowledge that is directly linked with RI
2.
Number of Master thesis

defended, where
knowledge and skills
gained on RI were
exploited
3.
Number of graduates
trained on RI
4.
Number of foreign
students as % of all
students trained on RI
Data on the post-diploma
employment path of those
graduates trained on RI
Per year of completion and per scientific field;

Master
theses
are
considered
as
a
training/education impact on skills profile of
population; the number of PhD thesis
completed is counted in the impacts on
scientific activity
Per stage (MA, PhD) and per year; distinction
should be made on skills gained by PhD
students that improve their general academic
and employability profile and skills directly
applied in PhD thesis work; the latter should
account for impacts on scientific activity
Per stage (MA, PhD) and per year
5.
Descriptive data should include at least

indication on the share of graduates finding
employment in academia/public sector/private
sector; further statistics on types of graduate
employment (part time/full time) and sector of
activity are desirable
14 Bozeman, B. (2000) Technology transfer and public policy: a review of research and theory. Research
Policy, 29(45)

2.2.3 Impacts on innovation

Innovation can result from RI through industry collaborations with academic research
groups and contract research that result in knowledge transfer. Research facilities
generate knowledge that businesses cannot acquire through their networks. More
exposure to RI knowledge networks through formal and informal social interaction
increases the possibility that private sector discerns new opportunities for exploiting
intellectual and technical capital available on RI.
Spin-offs are one way in which knowledge generated using RI can diffuse into the
market in the form of innovative product, process or service. Other avenues for
innovation impacts arise from the commercialisation of research through licences or
joint ventures with existing companies.
There are important barriers for private sector use of RI due to different objectives and
cultural factors. The industrial community is often less interested in using RI directly
(e.g. placing industrial researchers in RIs to carry out R&D). There is more interest in
acquiring technical training and co-developing new IPR.15 In some sectors (like food or
agriculture) the industrial landscape can be dominated by a large number of SMEs
that, individually, lack resources to engage in research collaboration.
The table below provides a number of indicators that can be used to track the impact
of RIs on knowledge transfer and innovation. One example of a framework for
measuring the quantity of knowledge transfer from the research base is the Scottish
Funding Council's (SFC) Knowledge Transfer Metrics Return16.
Figure 8: Indicators for impacts on innovation of RI operational phase
No
Indicators
1.
Number of collaborative
research projects and
volume of funding
Collaborative research carried out at that the RI

might involve both regional and international
researchers using the equipment for shorter-orlonger periods. The funding generated from
service fees charged to research team will enable
reinvestment in equipment. Qualitative research
on the research teams working in or with the RI
can be used to map (e.g. using social network
analysis) the place of the RI in global research
networks.
2.
Number of R&D projects

commissioned by
companies and volume of
their funding
Contract research is one step closer to

generating a socio-economic impact since firms
purchasing research services will only do so in
order to improve production processes
(enhancing productivity) or develop prototypes
of new products or services. Tracking trends in
volume of funding provides an indicator of the
longer-term financial sustainability of the RIs.
Similarly, tracking the renewal of the client
base (contracts with new clients) rather than
just number of companies contracting research
is useful.
15 ERF Workshop The Socio-Economic Relevance of Research Infrastructures, 31 May 1 June 2012,
Hamburg, Germany
16 The metric is used by the Scottish Government to measure as a proxy measure of the quantity, but not the
quality,
of
knowledge
exchange
activities
undertaken
by
http://www.gov.scot/About/Performance/scotPerforms/TechNotes/knowledge

Scottish
universities.
10
No
Indicators
3.
Number of technologies,
prototypes, industrial
designs developed and
transferred
4.
Number of start-ups and

spin-offs created with
support from RI services
growth in turnover/value
added and employment
5.
Number of feasibility or
market studies for
industrial investment and
application of technologies
6.
Procurement contracts
signed for development
and upgrade of research
equipment

Depending on the RIs intellectual policy policy,
the results of collaborative and contract
research may be potentially exploited through
proof of concept and prototyping either by the
RI itself or in co-operation with academic or
industrial partners. Ideally, the monitoring of
outcomes should go beyond number counting
and aim to track whether prototypes have been
scaled up or further developed commercially.
The number of start-ups/spin-offs is an
indicator commonly used to assess economic
impact. However, there is a need to avoid
unrealistic expectations when setting targets for
numbers of spin-offs as this involves IP transfer
to the new companies potentially closing future
avenues of academic research. Moreover, many
spin-offs fail to grow substantially and there is
an need to track growth in turnover and
employment on a longitudinal basis (over 5-10
years) to fully understand impact.
Commercialisation of research results will
involve, traditionally, market research or
industrial scale feasibility study prior to
investment. The number of (e.g. proof of
concept/prototyping) projects that move to the
stage of industrial investment is a strong
indication of the economic relevance of the
research. Tracking over time (5-10 years) the
actual investment in the application of new
technologies by businesses provides a stronger
indication of economic impact.
Tracking the number and type of procurement
contract for the development of innovative
instruments and products (rather than simply
off-the-shelf purchases from equipment
suppliers provides an indication of the impact of
the RI on the scientific equipment market this
is particularly pertinent if suppliers are
regionally based.
2.2.4 Impacts on scientific activity

The impact of RI on scientific activities can result in accumulation of new knowledge
and methodologies to push the boundaries of fundamental science. There is an
inherent tension between the development of non-proprietary research, which is
published in relevant scientific journals and can be accessed by other interested
parties, and propriety research where knowledge and technical data, which can be
basis for specific inventions, are not disclosed openly.
Another linkage to scrutinise is the impact of the use of RI on scientific productivity,
international recognition and reputation. The availability of advanced equipment has
an effect on shaping how scientific communities organise themselves. It can
significantly increase the productivity of research teams, as they do not need to seek
and arrange experimentation and testing opportunities on limited-availability facilities
or abroad (e.g. access to Grid computing facilities substitutes the need to queue for
supercomputer service time).

11
Open access to RI also induces more regular inward flow of researchers promoting
closer involvement of local teams in international research networks (brain exchange).
Such interaction can generate important learning effects for experienced and earlystage researchers alike. Increase in international recognition of leading scientists and
the capacity of research teams can attract further international competitive funding to
the research system. In addition, researcher experience gained during the set-up and
operational phase of RI can have further policy impact through contributions to RI
roadmaps.
Figure 9: Indicators for impact on scientific activity
No
Indicators
1.
Number of articles
published in the ISI level
international scientific
journals as a direct result
of research using RI
2.
Number of
methodologies/designs
developed
International patents
granted and published
patent applications (all
types)
3.
4.
Number of PhD
dissertations completed
5.
Number of scientific events

organised on research
topics directly relating to
RI services

The aim should be to track the dynamics of
publishing behaviour due to RI; peer-reviewed
articles applies as an indicator of scientific
activity to most scientific fields; in humanities
and some sub-fields of social sciences books,
book chapters and monographs are more
relevant indicator
Applies to those scientific fields where this
output is relevant to measure scientific
productivity
Indicator is relevant for most scientific fields in
natural sciences, engineering, life sciences and
medicine; in humanities and social sciences
other types of IPR (e.g. copyrights) should be
considered
Per year of completion and per scientific field;
includes those PhD dissertations which have
been predominantly or partly-based on the use
of RI services or equipment
Data on the frequency, types of events,
geographic origin and institutional affiliation of
participants can provide some proxy on the RI
impact on mutual learning, knowledge exchange
and researcher involvement in major scientific
networks; some conclusions can be drawn on
the critical mass and synergy effects with other
RI facilities in the region
2.2.5 Broader impacts on society

Regarding wider social impacts, it is important to outline that research infrastructures
can play an important role in scientific communication and scientific education. RI
can be used to inspire school students to learn STEM subjects. For example, largescale RI may organise open days for the general public or for schools. The policydecision on investment in RI can also be widely reflected in the press leading to
increased public awareness of science.
Research outcomes from using RI services can lead to concrete innovative products,
and services that are taken up and diffused in society (for example, new medical
instruments, diagnostics, treatments). While such impacts on society are broad,
indirect and very difficult to attribute and quantify, it is nevertheless important to
scope the potential contribution of RI to solving various societal problems.
There can also be concrete benefits that result from improving local infrastructure,
urban planning and community services as the RI investment may revitalise certain
areas with important indirect societal benefits. These developments can further shape
local cultural activities and citizen lifestyles.

12
Figure 10: Indicators for impacts on society

No
Indicators
1.
Number of organised RI open

days for wider public and any
available data on participant
satisfaction with the events
Per year and per target group (e.g.

schools/general public); aim to provide an
indication on the impact of public
awareness of science
2.
Number of press articles on the

investment in research
infrastructure
Number of new or improved
products, services, solutions as a
result of research using RI that
are diffused in society
Account of improved local
infrastructure, community
services, increase in local
cultural/recreational activities
due to RI
Per year and per type of publication (print

media/online articles/specialist
publications)
This indicator involves especially
significant time-lag; the aim to scope the
possible contribution of RI to specific
societal outcomes
Aim to scope the possible contribution of
RI to the outcomes
3.
4.
3. Impact assessment methods and tools

3.1 Impacts on economy
To measure the direct economic effect of the RI design and construction phase
requires an analysis of all suppliers that have won the procurement tenders to scope
the contract values per sector of activity and service provider affiliations
(local/foreign/multinational). Relevant supply chains should be mapped to determine
the scope of the output, value added and additional employment that has been created
locally and the benefits that have been diffused internationally. Desk analysis of the
proposal and contractual documentation and survey among suppliers can be used to
collect this information.
The operational phase requires a comprehensive look at the activities undertaken by
the managers responsible for establishing and running the research facility. This
involves a detailed account of facilities expenditure on salaries, purchase of goods and
services from financial records and direct output generated. Desk research and
interviews with RI managers/coordinators can be used to collate and analyse financial
and activity data.
The multiplier effect on the economy is addressed through an input-output analysis.
This ripple effect includes the indirect economic effects that occur from all secondary
industries supporting the procured activities and induced effect representing the
changes in consumption of respective households. The input-output model is a
quantitative econometric technique that represents the interdependencies between
different sectors of a national/ regional economy. An input-output analysis estimates
the value of the purchases that flow between the identified supply chains in a given
period of time. Building IO model it is possible to derive economic multipliers.
Methodological approaches have been developed on applying cost-benefit analysis to
RI projects17, yet these are suited more for ex-ante project evaluation as it takes a long
time period for the effects of RI investment costs to materialise. Nonetheless, the
applicability of CBA type approaches can be explored. For instance, the effects on RI
17 See e.g. Clarke, S. et.al. (2013) Project Preparation and CBA of RDI Infrastructure Projects. JASPERS
Staff Working Paper

13
users can be scoped through application of contingent valuation techniques.

Contingent valuation is a method that uses surveys to draw out the willingness of
actual and potential users to pay for certain services. The method involves assigning of
monetary values to non-market goods and services based on preferences.
Data requirements
Overview of contractual information with RI suppliers (design, construction,
maintenance, operations, renewal)
Data on the affiliation of suppliers (local/foreign/multinational)
Data on employment, wages
Facilities annual expenditure broken down by function in time series
Strategic plans and annual activity reports
3.2 Impacts on human resource capacity

To account for the impacts on human resource capacity it is necessary to capture the
structural changes in organisational behaviour that arise as a result of the research
infrastructure. Monitoring statistics on the number of staff and students attracted, in
particular from abroad, can shed a light on the dynamics of knowledge exchange that
RI is facilitating. It is necessary to understand what are the formal organisational
aspects, how training on RI is carried out and what are the practices for informal
know-how transfer (interaction with technicians, networks). Surveys among students,
vising and permanent researchers, technicians and RI managers can be employed for
necessary data gathering. Particularly important is to explore the spaces of student
and researcher interaction with industry. Longitudinal analysis of graduate career
paths is a valuable approach.
In addition to the available monitoring statistics and information from surveys, data
collection strategies for tracing impacts on human resource capacity should involve
extensive qualitative research techniques. These can include semi-structured
interviews with facility owners and a representative population of researchers and
students. A case study approach is an appropriate way to capture the dominant
patterns of knowledge diffusion and the degree of social capital formation that the
establishment of RI has promoted.
Data requirements
Relevant monitoring data, including e.g. statistics on the numbers of attracted staff and
students, (locally and from abroad) and career paths of graduates that have used RI for
training or MA/PhD thesis
Database of concluded service contracts
Documentation of human resource development strategies, user guides and other
documented evidence
Annual activity reports
3.3 Impacts on innovation

Impacts on innovations arise via multiple impact pathways. The most feasible method
is to apply quantitative analysis to scope economic effects of innovation (e.g. value of
IP portfolio) and use qualitative investigation to understand in detail the knowledge
spillovers. Monitoring data on knowledge transfer activities can be supplemented with
supplier surveys (for tracing procurement-led innovation and skills formation) and indepth interviews (with research teams, spin-offs, relevant industry representatives).
Narrative descriptions of individual case studies that look in detail into specific aspects
of innovation impact (e.g. impact via prototyping, industrial demonstrators) would be
supportive for the overall assessment. If any routine user surveys are executed by RI
management this would be very useful source of material to better understand user
experiences, motivations and wider effects of RI use.

14
Data requirements
Relevant official statistics and structured monitoring data (i.e. regular progress reports
and data for measure project level key performance indicators)
Database of users (academic, industrial) over the lifetime of the facility
Database of supplier contracts (held either by the funding agency or by beneficiary
higher education or public research institutes)
3.4 Impacts on scientific activity

Impacts on scientific activity can be analysed through a bibliometric analysis of the
research teams working on the RI. Especially valuable would be to see the evolution of
co-publications over a period of time. A time-series/network analysis of the
participation in international research project consortia (FP7, H2020) may shed some
light on the visibility of research groups on the international research landscape.
All quantitative techniques should be supplemented with in-depth interviews with key
researchers to deepen, test and validate the claims. There are many limitations, as the
knowledge spillovers that arise as a result of the intellectual advances may be largely
invisible to facility owners and stakeholders.
Data requirements
Database of researchers working on the RI
Relevant official statistics and structured monitoring data
Annual activity reports
3.5 Impacts on society

The broader impacts that research infrastructure investment have on society is the
least scrutinised area due to the fact that such impacts are difficult to trace and
quantify. While direct causal chains are almost impossible to establish, social science
methods can be employed to estimate the potential contribution of RI. Surveys among
schools, public sector, relevant stakeholder groups can be used for this purpose.
Further interviews with relevant target groups for RI communication and PR activities
and the involved researchers can be undertaken. Impact on STEM education and
public understanding of science can be best explained through analytical narrative in
the form of case studies.
Data requirements
Documented materials on RI public relations activities (open days, public lectures,
seminars)
Available data from satisfaction and feedback surveys of participants to RI
communication events
Track of media publications/online content regarding RI

15
Figure 11: Implementation of an impact assessment of research infrastructure

16
3.6 IA implementation process, its challenges and limitations

Figure 11 provides an overview the IA implementation steps and use of analytical
techniques. During the inception phase desk research is performed on all available
documented evidence to determine the information gaps that should be covered
through additional data collection methods. In the data collection phase surveys are
launched beforehand. In-depth interviews and the application of quantitative
techniques can be run in parallel. Case studies are usually chosen after the first
preliminary findings to ensure that the most relevant aspects and impact areas are
scrutinised in detail. The synthesis phase of findings ideally should involve a peer
review and an international expert team to ensure appropriate benchmarking of RI
performance. A full impact assessment takes usually around six to nine month period.
Data shortages are likely to be the most important challenge due to the limited data
caption routines during the lifetime of research facility. Database development with
appropriate quantitative data is a foundation for carrying out an in-depth impact
assessment. This calls for anticipatory collection of relevant data at an early stage of RI
project implementation. The principle should be that the amount if work for
researchers is kept to a minimum. RI management should be encouraged to regularly
assess key sources of information and collect relevant data on an on-going basis. These
efforts should include as a minimum data which is readily available or which can be
collected automatically, e.g. CVs, completed doctorates, concluded contracts, etc.
Bearing in mind the challenge of data availablity and quality, the expectations have to
be realistic in terms of the quantifiable findings that can result from the evaluation.
Further challenges and limiting factors to IA of RI include:
The difficulty of capturing multidimensional and complex research outcomes and
impacts with simple metrics that are used as core indicators. For instance, the
accuracy of input-output models to determine multipliers is questioned. Findings
from the application of quantitative methods should be well-balanced with
qualitative insights and domain expert validation.
Issue of displacement or the extent to which economic benefits occur at the

expense of other economic activities is hard to address in the analysis.
Delineation between short-term, medium and long-term effects will be very

difficult due to restraints in data availability in time series.
It is hard to separate between RI induced benefits and those fostered by other

determinants. The impacts of RI should be accompanied by analysis of parallel
initiatives in higher education and support measures for enterprises. National
legal frameworks (e.g. rules for skilled migration, IPR) are all important
determinants.
The establishment of systematic evaluation practice for investment in research

infrastructures is key to expand reference material and foster learning for more
comprehensive analyses of RI impacts.

17
Bibliography
Autio, E. (2014) Innovation from Big Science: Enhancing Big Science Impact Agenda.
Report to UK Department for Business, Innovation & Skills. Available at:
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/28
8481/bis-14-618-innovation-from-big-science-enhancing-big-science-impactagenda.pdf
Autio, E., Streit-Bianchi, M., and Hameri, A.-P. (2003) Technology Transfer and
Technological Learning Through CERNs Procurement Activity, CERN Scientific
Information Service. Geneva
Bianchi-Streit, M. et al (1984) Economic utility resulting from CERN contracts (second
study). Report of the European Organisation for Nuclear Research (CERN). Available
at: http://indico.cern.ch/event/66952/contribution/0/2/material/paper/0.pdf
Bozeman, B. (2000) Technology transfer and public policy: a review of research and
theory. Research Policy, 29(45)
Clarke, S. et.al. (2013) Project Preparation and CBA of RDI Infrastructure Projects.
JASPERS
Staff
Working
Paper.
Available
at:
http://www.jaspersnetwork.org/display/for/Project+Preparation+and+CBA+of+RDI
+Infrastructure+Projects
European Strategy Forum on Research Infrastructures (2011) Evaluation Report 2011.
Available
at:
http://ec.europa.eu/research/infrastructures/pdf/esfri_evaluation_report_2011.pdf
Fotakis, C. (2010) Analyses of FP7 supported Research Infrastructures initatives in the
context
of
ERA.
Available
at:
http://ec.europa.eu/research/evaluations/pdf/archive/fp7-evidencebase/experts_analysis/c.%20fotakis_-_research_infrastructure.pdf
Hallonsten, O., Benner, M. and Holmberg. G (2004) Impacts of Large-Scale Research
Facilities- A Socio-Economic Analysis. A study done at the Research Policy Institute,
Lund
University.
Available
at:
http://rifi.gateway.bg/upload/docs/public_doc_REPORT_impact_of_large_scale_R
I.pdf
von Hippel, E. (1976) The Dominant Role of Users in the Scientific Instrument
Innovation Process, Research Policy 5 (3)
Hickling Arthurs Low Corporation (2013) Return on Investment in Large Scale
Research Infrastructure. Report for National Research Council Canada. Available at:
http://www.triumf.ca/sites/default/files/HAL-ReturnOnInvestmentStudy-May2013.pdf
Horlings, E. et.al (2012) The societal footprint of big science. Report of the Rathenau
Instituut. Available at: http://www.rathenau.nl/en/publications/publication/thesocietal-footprint-of-big-science.html
National Research Council Canada (2013) Return on Investment in Large Research
Infrastructure.
Available
at:
http://www.triumf.ca/sites/default/files/HALReturnOnInvestmentStudy-May-2013.pdf
OECD (2014) International Distributed Research Infrastructures: Issues and Options.
Available at: http://www.oecd.org/sti/sci-tech/international-distributed-researchinfrastructures.pdf
OECD (2014) Report on the Impacts of Large Research Infrastructure on Economic
Innovation and on Society: Case Studies at CERN. Available at:
http://www.oecd.org/sti/sci-tech/CERN-case-studies.pdf

18
Rizutto, C. and Wood, J. (eds.) (2013) RAMIRI Handbook. Deliverable of FP7 project
Realising and Managing International Research Infrastructures. Available at:
http://www.ramiri-blog.eu/index.php?n=Main.HomePage
Rizzuto, C. (2012) Benefits of Research Infrastructures beyond Science, presentation
at ERF Workshop The Socio-Economic Relevance of Research Infrastructures, 31
May-1 June 2012, Hambourg
Sallee, C.M., Watkins, S.D and Rosaen, A.L. (2011) The Economic Impact of Fermi
National Accelerator Laboratory. Anderson Economic Group report to the University
of
Chicago.
Available
at:
https://ovprnl.uchicago.edu/sites/research.uchicago.edu/files/Fermilab_Economic_I
mpact_Full_Study.pdf
Simmonds, P. et.al (2013) Big Science and Innovation. Technopolis Group report for
the UK Department for Business, Innovation & Skills. Available at:
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/24
9715/bis-13-861-big-science-and-innovation.pdf
Wissenschaftsrat (2011) Recommendations on Research Infrastructures in
Humanities
and
Social
Sciences.
Available
at:
http://www.wissenschaftsrat.de/download/archiv/10465-11_engl.pdf
Wissenschaftsrat (2013) Report on the Science-Driven Evaluation of Large Research
Infrastructure Projects for the National Roadmap (Pilot Phase). Available at:
http://www.wissenschaftsrat.de/download/archiv/2841-13_engl.pdf
Zuijdam, F. et. al (2011) The role and added value of large-scale research facilities.
Technopolis Group report. Available at: http://www.technopolis-group.com/wpcontent/uploads/2011/02/1379_Report_Large-scale_Research_Facilities_EN1.pdf
Please cite this publication as:

Griniece E., A Reid and J. Angelis (2015): Guide to Evaluating and Monitoring SocioEconomic Impact of Investment in Research Infrastructures. Technopolis Group.
Tallinn. Estonia
Front cover photo: Centre for New Pharmaceutical and Health Technologies of the
Santaka Valley, Kaunas, Lithuania. Source: http://www.mitnija.lt/

19

2015 Technopolis Group Guide To Impact Assessment of Research Infrastructures

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

2015 Technopolis Group Guide To Impact Assessment of Research Infrastructures

Caricato da

Copyright:

Formati disponibili

2015

Evaluating and Monitoring the SocioEconomic Impact of Investment in

Evaluating and Monitoring the SocioEconomic Impact of Investment in Research

technopolis |group|, 2015

Elina Griniece, Alasdair Reid and Jelena Angelis

1.1 Why evaluate the socio-economic impact of research infrastructures?

1.2 Previous evaluations of the impact of research infrastructure investment

1.3 The aim of this guidance document

2. Logical framework for socio-economic impacts of investment in research

2.2 Operational phase

3. Impact assessment methods and tools

3.1 Impacts on economy

3.2 Impacts on human resource capacity

3.3 Impacts on innovation

3.4 Impacts on scientific activity

3.5 Impacts on society

3.6 IA implementation process, its challenges and limitations

Major equipment or group(s) of instruments used for research purposes;

High-performance laser system,

ELI: European Light

Mobile vehicles specially

Research vessels, satellite and

ICT-based system for

European Grid Computing

Setting-up, or renovating, RI usually requires a considerable level of financial

1 Definition used by the European Commission

Evaluating the socio-economic impact of research infrastructures

1.2 Previous evaluations of the impact of research infrastructure investment

Evaluating the socio-economic impact of research infrastructures

1.3 The aim of this guidance document

Innovation & Skills

Evaluating the socio-economic impact of research infrastructures

2. Logical framework for socio-economic impacts of investment in

2.1 Design and construction phase

Locally-owned businesses, franchise or corporate branch operating within a local area

Evaluating the socio-economic impact of research infrastructures

Evaluating the socio-economic impact of research infrastructure

Figure 4: Indicators for economic impacts of RI construction phase

Explanatory remarks and examples

Number of commercial suppliers

Suppliers can be mapped per sector, field

Scale of commercial suppliers

For applied method to estimate utilities in

Scale of commercial suppliers

Measured in FTE and grouped per

2.1.2 Impacts on innovation

Evaluating the socio-economic impact of research infrastructure

Figure 5: Indicators of innovation impact of RI construction phase

Explanatory remarks and examples

The novel technical requirements for the

The focus should be on the extent to which

2.2 Operational phase

2.2.1 Economic impacts

Evaluating the socio-economic impact of research infrastructure

Figure 6: Indicators for economic impact of RI operational phase

Explanatory remarks and examples

Service can be categorised into broad types,

Total amount of funding

Per service type, source of research grant and

Number of new directly

Per category (scientific/ technical/

For centralised large-scale facilities the rule of

2.2.2 Impacts on human resource capacity