Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Technology Management 30
New Surveys on R&D Metrics
In an effort to answer these questions, the current group,
Measuring the Effectiveness of R&D, administered two
surveys. The rst, survey A, was largely a repeat of the
original 1994 survey; the intent of this instrument was to
provide data to facilitate comparison in metrics usage
between the original survey and this one and to measure
changes in practice with regard to R&D metrics. Survey
A was administered at IRIs 2009 Annual Meeting; at-
tendees were typically high-level R&D managers, vice
presidents of R&D, and chief technical ofcers (CTOs).
The survey listed the 33 metrics included in the original
1994 survey and asked respondents to rank them by or-
der of importance in their organizations. We did not ask
respondents to identify their organizations, but we did
ask if they represented for-prot or not-for-prot corpo-
rations.
The second survey, Survey B, was a broader instrument
that asked participants to rate the expanded list of 50
metrics on their importance to the respondents organi-
zations, using a scale of 1 (no importance) to 5 (top im-
portance). Again, organizations were not identied, but
we did ask respondents to identify the industry in which
they worked. The group used this data to distinguish re-
lationships between preferred metrics and industry,
company type, and innovation game. Additionally, the
survey solicited open-ended comments on the value and
usage of metrics and methods used to collect data.
Respondents were also asked to describe metrics that
they felt were needed and list metrics that their corpora-
tion was using but that were not on the list. Survey B
was administered to attendees at IRIs 2008 Member
Summit; again, this group comprised high-level R&D
managers, vice presidents of R&D, and CTOs.
Results
Survey A. Survey A was completed by 56 respondents
from both corporate and not-for-prot organizations;
the 1994 survey did not ask about for-prot versus
One of the key
challenges of
implementing R&D
metrics is matching
metrics to the
various levels and
functions of the R&D
organization.
Figure 1 . The Technology Value Pyramid (TVP)
SeptemberOctober 2011 31
not-for-prot status, but given that IRIs membership did
not include not-for-prot organizations at that time, the
original sample most likely included only for-prot cor-
porations.
Data from Survey A were compared to data from the
1994 administration of the same survey ( Table 1 ). The
2009 data are sorted by for-prot or not-for-prot to al-
low a better comparison of the data from 1994 to 2009.
Interestingly, the top three metrics from 1994nancial
return to the business, strategic alignment with the busi-
ness, and projected value of the R&D pipelinemain-
tained their importance for for-prot corporations in the
2009 survey, but only the strategic alignment metric was
judged of high value by the not-for-prot group.
A number of metrics ranked higher in 2009 rankings
than they did in 1994, including achievement of R&D
pipeline objectives, quality of R&D personnel, level of
business approval of projects, comparative manufac-
turing costs, and effectiveness of transfer to manufac-
turing; none of these made the top 12 list in 1994, and
all did in 2009. Other metrics lost ground, including
portfolio distribution of R&D projects, market share,
current spending level for technology, and customer
satisfaction surveys; all were in the top 10 in 1994, and
none were in 2009. Metrics ranked highly by for-prot
respondents to the 2009 survey revealed a focus on -
nancial returns, strategic business alignment, and qual-
ity, while not-for-prot respondents were focused on
strategic alignment, accomplishment of milestones,
quality of people, portfolio distribution, and clarity of
project goals. The common thread in both groups was
strategic alignment.
Survey B. Survey B asked about companies practices
with regard to R&D metrics; the survey also asked
participants to rate the metrics currently included in the
TVP structure. There were 52 respondents.
Questions about company practices with regard to col-
lecting and using metrics were revealing. While a mi-
nority of respondents indicated that their companies
did not collect any metrics, most companies apparently
feel compelled to measure the effectiveness of their
R&D efforts. The most-reported uses for R&D metrics
included:
Providing data for project justication and decision-
making processes;
Enabling portfolio analysis, balancing, and tracking;
Calculating ROI for R&D;
Enhancing efciency in product development;
Driving performance and dening goals;
Ensuring that R&D is aligned to the business strat-
egy; and
Benchmarking against intracompany units as well
as outside companies.
About half of participants reported that their companies
collect data for metrics on a quarterly basis; a quarter
collect data either semiannually or annually. Most data
must be collected by hand; only a small percentage of
participants reported that their companies had automated
data collection and metric generation processes. ROI
calculations, according to respondents, require some
sort of project time tracking, in addition to other data
collection.
Participants also rated the importance to their compa-
nies of each entry in the expanded list of TVP metrics
(including those added after the original 1994 study),
sorted by TVP level ( Table 2 ). As in survey A, nancial
return is a dominant theme, especially in the top two
levels of the TVP. The top metrics for the value
Table 1 . Summary of Survey A results: Top ten metrics in 1994 and 2009
1994
2009
For-Prot Not-For Prot
Financial return to the business Financial return to the business Strategic alignment with the business
Strategic alignment with the business Strategic alignment with the business Accomplishment of project milestones
Projected value of R&D pipeline Projected value of R&D pipeline Quality of R&D personnel
Sales or gross prots from new products Gross prot margin Portfolio distribution of R&D projects
Accomplishment of project milestones Product quality and reliability Clarity of project goals
Portfolio distribution of R&D projects Sales or gross prots from new products Product quality and reliability
Market share Accomplishment of project milestones Rating of project benets by customers
Customer satisfaction surveys Achievement of R&D pipeline objectives External peer evaluation of R&D
Development cycle time Quality of R&D personnel Customer rating of technical capabilities
Gross prot margin
Product quality and reliability (tie)
Level of business approval of projects
Comparative manufacturing costs (tie)
Number of technical reports
Research
Technology Management 32
creation level are all nancial, as are most of the top
metrics at the strategy level. Other important strategy
metrics include strategic alignment and probability of
success. Foundation metrics are associated with quan-
titative measures, such as number of patents, as well as
more abstract metrics associated with people develop-
ment and creativity.
In an attempt to distinguish how innovation strategy in-
uenced the choice of metrics, we analyzed results in
the context of innovation strategy, using Roger Millers
concept of innovation games ( Miller and Floricel
2004 ). Miller and his colleagues identied eight distinct
approaches to innovation strategy, each most likely to
be used by rms in particular industrial contexts; for
instance, pharmaceutical rms are likely to engage in
science-based journeys while biotechnology rms en-
gage in technology races and cellphone manufacturers
engage in battles of architectures ( Miller and Floricel
2004 ; Tirpak et al. 2006 ; Miller and Olleros 2008 ).
Clearly, the structure of the innovation game should in-
uence the shape and objective of the R&D effort and,
hence, the metrics an organization uses to measure the
effectiveness of its R&D effort.
Because of our small sample size, the original list of
eight innovation games yielded data sets too small to
provide meaningful information. As a result, for the pur-
poses of analysis, the eight innovation games were con-
solidated into four: new and improved (standalone)
industries, such as chemicals or petroleum; pushing the
envelope (integrated systems) industries, such as com-
puters, machines, aircraft, or computer chips; consumer
products industries, such as food, soap, or cosmetics;
and services to innovator rms, such as nancial, techni-
cal, and governmental services. Representatives taking
the survey from the services sector were typically from
government labs, universities, and research institutes;
these were largely not-for-prot organizations.
Once overall data were analyzed for responses from
each of the four dened innovation games, the results
revealed some interesting variations ( Table 3 ). Financial
return, for instance, is much less important in services
than it is in the other groups, perhaps a reection of the
preponderance of not-for-prot organizations in this cat-
egory. With reduced emphasis on nancials, the founda-
tion metrics focusing on intellectual property, R&D
processes, and people development rank much higher.
Product quality is much more important in consumer
products than in the other groups, and the metrics for
integrated systems industries is different from the others
in that pathways for exploiting technology, environmen-
tal management, process efciency, and management
systems are signicantly more important than they are in
other sectors.
Respondents to Survey B also identied a number of
metrics that they felt should be added to the 50 currently
included in the TVP; these fell into four broad groups:
Open innovation metrics, including external vs. in-
ternal R&D, percent of projects outsourced, and exter-
nal innovation;
Metrics to measure the effectiveness of R&D pro-
cesses beyond nancial, including, for instance, mea-
sures of efciency in project management;
Value creation metrics for technical service and sup-
port for existing products; and
Table 2 . Survey B results: Top ve metrics by TVP level
Metric Ranking
Outcome
Financial Return 1
Gross Prot 2
Market Share 3
Projected Value of Pipeline 4
IP Management 5
Strategy
Financial Return 1
Projected Value of Pipeline 2
Gross Prot 3
R&D Investment/Sales 4
Strategic Alignment 5
Foundations
IP Management 1
Number and Quality of Patents 2
People Development 3
Creativity 4
Cost versus Budget 5
Most companies
apparently feel
compelled to
measure the
effectiveness of their
R&D efforts.
SeptemberOctober 2011 33
Not-for-prot metrics to measure the development
of new capabilities, awards, professional activities,
and bibliometrics.
Of particular interest were calls for additional metrics in
R&D effectiveness and value creation. While the litera-
ture does describe some metrics in the area of R&D ef-
fectiveness beyond nancial measures (e.g., McGrath
1994), most companies found these existing measures
inadequate. Similarly, many participants reported a need
for metrics to measure the effectiveness of technical ser-
vice and support groups, which are often associated with
R&D budgets, in order to show their value to the com-
pany. The closest metrics currently available in this area
are related to cost savings, but these do not adequately
address the issue.
Respondents also identied a number of issues with ex-
isting metrics, including the difculty of collecting data
and calculating and quantifying metrics, the lack of uni-
form standards for different business units or global
sites, and inconsistency in the application of business-
specic metrics across the company. Respondents were
sometimes skeptical of the value of metrics, attitudes re-
ected in comments that the easiest-to-measure metrics
generally have the lowest value, that metrics are trailing
measures that tend to focus on the short term, and that
metrics tend to focus on inputs rather than outcomes.
Respondents also worried that a reliance on metrics can
lead to gaming the numbers to make a group look good.
Respondents generally commented on the danger that
metrics can inuence behavior, for good or bad, and that
poor choices of metrics can lead to bad business deci-
sions. Finally, they commented on the difculty of creat-
ing metrics that are meaningful to shareholders. And
they lamented that, even with perfect metrics, doing all
the right things does not guarantee success.
Discussion
The comparative analysis enabled by Survey A reveals
some interesting trends, but also suggests areas where
R&D focus has remained the same. Financial return,
strategic alignment, and projected pipeline value were
the leading metrics in 1994, and they remained so in
2009, in spite of dramatic changes in the context and
understanding of R&D in the intervening decade and a
half. These three metrics remain important because they
reect core sources of value. The quantitative nancial
measures (return and value of pipeline) are indicators of
the absolute value of R&D in terms the world under-
stands as generally accepted accounting. Strategic align-
ment measures the likelihood that the R&D effort will
have a signicant impact on the core business. Taken
together, these three measures also offer a wide perspec-
tive of measurement. Financial return is a retrospective
measure, while projected pipeline value and strategic
alignment are outcome measures. Not-for-prot organi-
zations, which do not have the direct nancial criteria
for success of most corporations, tend to place more im-
portance on strategy and foundation than on value cre-
ation metrics.
Table 3 . Survey B results: Top metrics by innovation game
Innovation Game
TVP Level
Value Creation Strategy Foundation
New & Improved
(Standalone)
Financial Return
Projected Value of R&D
Pipeline
Gross Prot
Gross Margin
Financial Return
Projected Value of R&D Pipeline
Gross Prot
Gross Margin
None identied at statistically
signicant level.
Pushing the Envelope
(Integrated Systems)
Financial Return
Projected Value of R&D
Pipeline
Gross Prot
Gross Margin
Financial Return
Projected Value of R&D Pipeline
Gross Prot
Gross Margin
R&D Investment as % of Sales
People Development
Intellectual Property Management
Number and Quality of Patents
Consumer Products Financial Return
Product Quality & Reliability
Gross Margin
Gross Prot
Market Share
Financial Return
Gross Margin
Gross Prot
R&D Investment as % of Sales
Probability of Success
People Development
Intellectual Property Management
Number and Quality of Patents
Services Intellectual Property
Management
Financial Return
Financial Return Idea Generation and Creativity
R&D Process
People Development
Quality of Personnel
Number and Quality of Patents
Research
Technology Management 34
Shifts in ranking may reect a number of changes in cor-
porate culture and the context of innovation: the effects
of the recession that began in 2008, which has driven a
more inward and short-term focus and increased worries
about nances, cash ow, and excess spending; the shift
from total quality management (TQM) with its emphasis
on quality and the voice of the customer; the increased
attention to corporate governance; and the rise of global
competition. Recessionary anxieties are reected in the
increased importance of nancial measures, such as the
value of the R&D pipeline, manufacturing costs, and
transfers to manufacturing, and in the decreased signi-
cance accorded to such metrics as portfolio distribution,
market share, spending on technology, and customer sat-
isfaction. The increased importance of business approv-
als and transfer to manufacturing may be a reection of
a closer focus on corporate governance. The shift away
from TQM as a management paradigm may have driven
down the importance of customer satisfaction as an
R&D metric, allowing other measures to take its place.
A third explanation for the emphasis on costs is global
competition. In 1994, China was just beginning to re-
place some U.S. manufacturing. In todays worldwide
marketplace, low price has become one of the most im-
portant criteria for product sales. This is reected in the
increased importance of manufacturing costs and trans-
fers to manufacturing as R&D metrics.
There were also some differences noted in the metrics
preferred depending on a companys innovation game
and between corporations and not-for-prot organiza-
tions. For instance, consumer product companies tend to
rely more heavily on quality metrics than other compa-
nies do, possibly because these industries lack the well-
dened specications and standards that generally guide
companies working in the standalone or integrated sys-
tems areas. Companies in these areas are more likely to
emphasize number of defects, efciency, and system
management and R&D metrics. Consumer products
companies, on the other hand, must deal in a consumer
market where a contaminated or poor quality product
could generate major lawsuits, negative publicity, and
bad will. Future events could inuence following sur-
veys; for instance, we might expect a higher emphasis
on quality metrics in the petrochemical industry in the
wake of the 2010 BP oil spill in the Gulf of Mexico.
There are also discernible differences in preferred met-
rics between for-prot and not-for-prot organizations.
Not-for-prot companies are typically less concerned
about nancials and product sales; instead, they are fo-
cused on meeting goals and managing people and port-
folios. The common thread linking corporations and
not-for-prots is strategic alignment; both are concerned
to make sure theyre working on the right projects to
support the core business.
As the R&D model is changing from exclusively inter-
nal development to an open innovation perspective,
metrics depicting these changes, and measuring their
outcomes, are required. Proposals for such metrics as
percent of new intellectual property from licensing in or
percent of new business created by licensed technology
have been proposed. Fine-tuning these and integrating
them into the TVP are important next steps; one part of
that process will be determining in which levels of the
hierarchy they belong. Since R&D will continue to
evolve, identifying and integrating new metrics is a con-
tinuous process. The TVP should continue to develop as
the repository of these metrics, along with their deni-
tions, recommended uses, and references.
Conclusion
Our respondents, along with the data they provided, of-
fer some concrete advice for selecting R&D metrics.
First, it is important to recognize that some metrics are
more important to specic industries and innovation
strategies than others. Choosing a metric, or bench-
marking results, is best done within similar business
types. In a similar vein, before choosing a metric, it is
important to understand what the company (and its
stakeholders) will do with the results. To be effective,
metrics must align with the business and technical
strategies as well as with the immediate goals of mea-
surement. The number of metrics that are collected
should be limited to the minimum necessary to acquire
a complete picture; at the same time, a few robust and
As the R&D model
is changing to an
open innovation
perspective, metrics
depicting these
changes, and
measuring their
outcomes, are
required.
SeptemberOctober 2011 35
clearly measurable metrics should be selected from
each portion of the TVP.
Beyond the surveys, the ROR subcommittee has success-
fully integrated the TVP process into the IRI website. An
updated list of the metrics with contemporary language and
examples, and their denition and usage has been captured
to enhance future knowledge sharing. As new metrics are
developed, these will need to be incorporated into the list.
Ultimately, a goal would be for a user of the TVP to com-
plete a survey and be able to compare the results with those
of others playing the same innovation game.
Even as measuring R&D performance becomes more im-
portant, metrics remain difcult and sometimes elusive.
IRI should continue with working groups that continue to
monitor, survey and make available to others these evolu-
tions. The TVP is an ideal system to support this effort.
References
Andrew, J.P., Hanns, K., Michael, D.C., Sirkin, H.L., Taylor,
A. 2008. Measuring Innovation Survey 2008: Squandered
opportunities. Boston Consulting Group. http://www.bcg.com/
documents/le15302.pdf (accessed November 11, 2010) .
Cosner , R. R. 2010 . The Industrial Research Institutes R&D trends
forecast for 2010 . Research-Technology Management 53 ( 1 ):
14 22 .
Donnelly, G., and Fink, R. 2000. A P&L for R&D. CFO Magazine
February: 4450 .
European Industrial Research Management Association (EIRMA).
2004. Assessing R&D Effectiveness: WG62 Report. EIRMA
Working Group Reports. Paris. www.eirma.org/pubs/reports/
wg62.pdf (accessed April 11, 2011) .
Germeraad , P. 2003 . Measuring R&D in 2003 . Research-Technology
Management 46 ( 6 ): 47 56 .
Hauser , J. R. 1996 . Metrics to value R&D: An annotated bibliography.
Sloan Working Papers . Cambridge, MA : Sloan School of
Management . Massachusetts Institute of Technology .
McGrath , M. E. , and Romeri , M. N. 1994 . The R&D Effectiveness
Index: A metric for product development performance . Journal of
Product Innovation Management 11 ( 3 ): 213 220 .
Miller , R. , and Floricel , S. 2004 . Value creation and games of
innovation . Research-Technology Management 47 ( 6 ): 25 37 .
Miller , R. , and Olleros , X. 2008 . To manage innovation, learn
the architecture . Research-Technology Management 51 ( 3 ):
17 27 .
Roussel , P. A. , Saad , K. N. , and Erickson , T. J. 1991 . Third Generation
R&D: Managing the Link to Corporate Strategy . Boston, MA :
Harvard Business Press .
Tipping , J. , Zeffren , E. , and Fusfeld , A. 1995 . Assessing the value
of your technology . Research-Technology Management 38 ( 5 ):
22 39 .
Tirpak , T. , Miller , R. , Schwartz , L. , and Kashdan , D. 2006 . R&D
structure in a changing world . Research-Technology Management
49 ( 5 ): 19 26 .
Reprints
THE ART OF TECHNOLOGY MANAGEMENT
Forty-nine selections the Managers at Work department of RESEARCH-TECHNOLOGY MANAGEMENT
are now available in paperback. To order, see inside back cover.
Leadership and R&D Productivity
To Motivate, Set Goals
Motivating Research Scientists to Reach for
the Eagles
Wining the Respect of Subordinates
How Am I Doing?
Become A Better Coach
Exercising Leadership in the Corporation
What Does the CEO Want?
Are You Credible With Your CEO?
Encouraging Little C and Big C Creativity
Fostering Breakthrough Innovations
What to do About NIH
To Innovate Faster, Try the Skunk Works
Overcoming Roadblocks to
Commercializing Industrial R&D Projects
Overcoming Roadblocks to the Marketplace
Technology Transfer: A GM Managers
Strategy
Metanoic Society Helps Shell Commercialize
Product Ideas in Half the Time
Making Customer Visits Pay
Teams Speed Commercialization of R&D
Projects
Building TeamsWhat Works (Sometimes)
Before you Try Team Building
Creating High-Performance Teams
Rules of Thumb for Project Management
Four Steps to Better Staff Meetings
Red Flags at Dawn, or Predicting R&D
Project Termination at Start-Up
Hiring People Who Do Good Research
Revisiting the Dual Ladder at General
Mills
Misusing the Dual Ladder, or the The Case
of Joe Mertz
Building Careers at Procter & Gamble
...AND MORE
Research
Technology Management 36