UNIT 63 - BENCHMARKING

UNIT 63 - BENCHMARKING

 A. INTRODUCTION

 Two types of benchmarking

 Benchmark script

 B. QUALITATIVE BENCHMARKS
 C. QUANTITATIVE BENCHMARKS

 Performance evaluation (PE)

 Subtasks for GIS PE
 Requirements for a quantitative benchmark
 GIS PE is more difficult

 D. EXAMPLE MODEL OF RESOURCE UTILIZATION

 Subtasks
 Products and data input
 Frequency required
 Execution of tasks
 Prediction
 Forecast
 Summary of phases of analysis

 E. APPLICATION OF MODEL

 Three phases of benchmark

 Qualitative benchmark
 Quantitative benchmark
 Model

 F. LIMITATIONS
 G. AGT BENCHMARK EXAMPLE

 Project Background

 REFERENCES
 EXAM AND DISCUSSION QUESTIONS
 NOTES
This unit is contains far more information than can possibly be covered in a single lecture. The
middle sections, D and E, contain a detailed technical review of a benchmark model. Depending
on the abilities and interests of your students you may wish to omit these sections and move on
to the description of the AGT benchmark in section G, or focus the lecture on the technical
aspects and omit the descriptive example.

UNIT 63 - BENCHMARKING

A. INTRODUCTION

 benchmarking is a key element in minimizing the risk in system selection

o often the customer does not have precise plans and needs - these will be
determined to some extent by what the GIS industry currently has to offer
 no vendor's product yet meets the requirements of an ideal GIS
o customer needs reassurance - real, live demonstration - that the system can deliver
the vendor's claims under real conditions
 the GIS industry is still young, there are two few success stories out there
 a benchmark allows the vendor's proposed system to be evaluated in a controlled
environment
o customer supplies data sets and a series of tests to be carried out by the vendor
and observed by the customer
o an evaluation team is assembled and visits each vendor, performing the same
series of tests on each system
o tests examine specific capabilities, as well as general responsiveness and user-
friendliness
 reinforces the written response from the vendor by actual demonstration of capabilities
o demonstration is conducted in an environment over which the customer has some
control - not completely at the vendor's mercy as e.g. at trade show
demonstrations
 equipment is provided by the vendor, data and processes must be defined by the customer
 a benchmark can be a major cost to a vendor - up to $50,000 for an elaborate benchmark
o in some cases part of these costs may be met by the customer through a direct
cash payment

Two types of benchmarking

 qualitative benchmark asks:

o are functions actually present?
o do they live up to expectations?
o are they easy to use?
 quantitative benchmark asks:
o does the proposed configuration have the necessary capacity to handle the
planned workload?

Benchmark script
handout - Benchmark script example (2 pages)
  benchmark uses a script which
o details tests for all of the functions required
o permits both:
 subjective evaluation by an observer (qualitative)
 objective evaluation of performance (quantitative)
 must allow all of the required functionality to be examined
 failure of one test must not prevent remainder of test from being carried out
 must be modular
o customer must be able to separate the results of each test
 conditions must be realistic
o real data sets, realistic data volumes

B. QUALITATIVE BENCHMARKS

 in the qualitative part of the benchmark it is necessary to evaluate the way the program
handles operations
 functions cannot be evaluated simply as present or absent

overhead - Qualitative assessment

 functions are not all equally necessary - they may be:

o necessary before any products can be generated, e.g. digitizing
o necessary to some products but not others, e.g. buffer zone generation
o necessary only to low-priority products, i.e. nice to have

C. QUANTITATIVE BENCHMARKS

 in quantitative tests, procedures on problems of known size are executed

o analysis of results then establishes equations which can be used to predict
performance on planned workload
o e.g. if it takes the vendor 1 hour to digitize 60 polygons during the benchmark,
how many digitizers will be needed to digitize the planned 1.5 million polygons to
be put into the system in year 1?
o this is known in computer science as performance evaluation

Performance evaluation (PE)

 developed in the early days of computing because of need to allocate scarce computing
resources carefully
 a subfield of computer science
 requires that tasks be broken down into subtasks for which performance is predictable
 early PE concentrated on the machine instruction as the subtask
o specific mixes of machine instructions were defined for benchmarking general-
purpose mainframes
 e.g. the "Gibson mix" - a standard mixture of instructions for a general
computing environment, e.g. a university mainframe
  multi-tasking systems are much more difficult to predict because of interaction between
jobs
o time taken to do my job depends on how many other users are on the system
o it may be easier to predict a level of subtask higher than the individual machine
instruction
o modern operating systems must be "tuned" to perform optimally for different
environments
 e.g. use of memory caching, drivers for input and output systems

Subtasks for GIS PE

 specifying data structures would bias the benchmark toward certain vendors
o e.g. cannot specify whether raster or vector is to be used, must leave the choice to
the vendor
 similarly, cannot specify programming language, algorithms or data structures
 a GIS benchmark must use a higher level of subtask
o an appropriate level of subtask for a GIS benchmark is:
 understandable without technical knowledge
 makes no technical specifications
o e.g. "overlay" is acceptable as long as the vendor is free to choose a raster or
vector approach
o e.g. "data input" is acceptable, specifying digitizing or scanning is not
 therefore, a GIS PE can be based on an FRS and its product descriptions, which may have
been generated by resource managers with no technical knowledge of GIS

Requirements for a quantitative benchmark

 need a mathematical model which will predict resource utilization (CPU time, staff time,
plotter time, storage volume) from quantities which can be forecast with reasonable
accuracy
o numbers of objects - lines, polygons - are relatively easy to forecast
o technical quantities - numbers of bytes, curviness of lines - are less easy to
forecast
 the mathematical form of the model will be chosen based on expectations about how the
system operates
o e.g. staff time in digitizing a map is expected to depend strongly on the number of
objects to be digitized, only weakly on the size of the map (unless large maps
always have more objects)
o requires a proper balance between quantitative statistical analysis and knowledge
about how the procedures operate

GIS PE is more difficult

 GIS PE is more difficult than other types of PE because:

o uncertainties over the approach to be adopted by the vendor (data structure,
algorithms)
 o high level at which tasks must be specified
o difficulty of forecasting workload
o no chance of high accuracy in predictions
 however even limited accuracy is sufficient to justify investment in
benchmark

D. EXAMPLE MODEL OF RESOURCE UTILIZATION

 this section describes a mathematical model developed for a quantitative benchmark

overhead - Model of resource utilization

handout - A model of resource utilization

Subtasks

 begin with a library of subtasks L

o this is the set of all GIS functions defined conceptually
o e.g. overlay, buffer zone generation, measure area of polygons, digitize

Products and data input

 FRS identified a series of products

o identified as R1, R2,...,Ri,...
 each product requires a sequence of subtasks to be executed
 data input also requires the execution of a series of subtasks for each dataset, e.g. digitize,
polygonize, label

Frequency required

 each product is required a known number of times per year

o Yij is the number of times product i is required in year j
 knowledge extends only to the end of the planning horizon, perhaps year 5

Execution of tasks

 execution of a subtask uses resources

o e.g. CPU, staff or plotter time
 these can be quantitatively measured
o e.g. CPU time measured in seconds
o e.g. staff time in minutes
o note: indications are (Goodchild and Rizzo, 1987) that staff time (human) is more
predictable than CPU

time (machine) because of complications of computer accounting systems,

multitasking etc.
  Mak is the measure of resource k used by subtask a
o k is one of the resources used
o a is one of the subtasks in the library L

Prediction

 in order to predict the amount of resources needed to create a product, need to find a
mathematical relationship between the amount of resource that will be needed and
measurable indicators of task size
o e.g. number of polygons, queries, raster cells, lines
 Pakn is predictor n for measure k, subtask a
 Mak = f(Pak1,Pak2,...,Pakn,...)
o e.g. the amount of staff time (Mk) used in digitizing (a) is a function of the
number of polygons to be digitized (Pak1) and the number of points to be
digitized (Pak2)
 the general form of the prediction function f will be chosen based on expert insight into
the nature of the process or statistical procedures such as regression analysis
o e.g. use the results of the benchmark to provide "points on the curve" with which
to determine the precise form of f

Forecast

 given a prediction function, we can then forecast resource use during production with
useful, though not perfect, accuracy

Wkit is the use of resource k by the tth subtask required for a single generation of product
i

Wki = sum of Wkit for all t is the amount of the resource k used by all subtasks in
making product i once

Vkj = sum of (Wki Yij) for all i is the amount of resource k used to make the required
numbers of all products in year j

Summary of phases of analysis

overhead - Summary of phases of analysis

1. Define the products and subtasks required to make them

2. Evaluate each subtask from the results of the qualitative benchmark

3. Analyze the system's ability to make the products from the qualitative evaluations in
(2) above

4. Obtain performance measures for known workloads from the results of the quantitative
benchmark
 5. Build suitable models of performance from the data in (4 ) above

6. Determine future workloads

7. Predict future resource utilization from future workloads and performance models, and
compare to resources available, e.g. how does CPU utilization compare to time available?

E. APPLICATION OF MODEL

 this section describes the application of this model of resource use in a benchmark
conducted for a government forest management agency with responsibilities for
managing many millions of acres/hectares of forest land
 FRS was produced using the "fully internalized" methodology described in Unit 61
 FRS identified 33 products
o 50 different GIS functions required to make them out of a total library of 75
 GIS acquisition anticipated to exceed $2 million

Three phases of benchmark

1. data input - includes digitizing plus some conversion of existing digital files

2. specific tests of functions, observed by benchmark team

3. generation of 4 selected products from FRS

 these three phases provided at least one test of every required function
 for functions which are heavy users of resources, many tests were conducted under
different workloads
o e.g. 12 different tests of digitizing ranging from less than 10 to over 700 polygons

Qualitative benchmark

 each function was scored subjectively on a 10-point scale ranging from 0 = "very fast,
elegant, user-friendly, best in the industry" to 9 = "impossible to implement without
major system modification"
o score provides a subjective measure of the degree to which the function inhibits
generation of a product
o maximum score obtained in the set of all subtasks of a product is a measure of the
difficulty of making the product

Quantitative benchmark

 since this was an extensive study, consider for example the quantitative analysis for a
single function - digitizing
 digitizing is a heavy user of staff time in many systems
 delays in digitizing will prevent system reaching operational status
 o digitizing of complete database must be phased carefully over 5 year planning
horizon to allow limited production as early as possible
 as stated above, benchmark included 12 different digitizing tasks
 resource measure of digitizing is staff time in minutes
 predictors are number of polygons and number of line arcs
o line arcs are topological arcs (edges, 1-cells) not connected into polygons, e.g.
streams, roads
o other predictors might be more successful - e.g. number of polygons does not
distinguish between straight and wiggly lines though the latter are more time-
consuming to digitize - however predictors must be readily accessible and easy to
forecast
 sample of results of quantitative benchmark: polygons line arcs staff time (mins) 766 0
930 129 0 136 0 95 120
 benchmark digitizing was done by vendor's staff - well- trained in use of software, so
speeds are likely optimistic

Model
overhead - Models of time resources required

 expect time to be proportional to both predictors, but constants may be different

m = k1p1 + k2p2 m is measure of resource used p is a predictor - p1 is polygons, p2 is

line arcs k1, k2 are constants to be determined

Results

 the equation which fits the data best (least squares) is:

m = 1.21 p1 + 0.97 p2

o i.e. it took 1.21 minutes to digitize the average polygon, 0.97 minutes to digitize
the average line arc
 to predict CPU use in seconds for the digitizing operation:

m = 2.36 p1 + 2.63 p2

o i.e. it took 2.36 CPU seconds to process the average polygon

 uncertainties in the prediction were calculated to be 34% for staff time, 44% for CPU
time
o suggests that humans are more predictable than machines
 adding together staff time required to digitize the forecasted workload led to the
following totals: Year Time required (minutes) 1 185,962 2 302,859 3 472,035 4 567,823
5 571,880 6 760,395
 the average working year has about 120,000 productive minutes in the daytime shift
o by year 6 the system will require more than 6 digitizing stations, or 3 stations
working 2 shifts each, or 2 stations working 3 shifts each
 o this was significantly higher than the vendor's own estimate of the number of
digitizing stations

required, despite the bias in using the vendor's own staff in the digitizing
benchmark

F. LIMITATIONS

 difficult to predict computer performance even under ideal circumstances

o GIS workload forecasting is more difficult because of the need to specify
workload at a high level of generalization
o the predictors available, e.g. polygon counts, are crude
 the model is best for comparing system performance against the vendor's own claims, as
implied by the configuration developed in response to the RFP
o it is less appropriate for comparing one system to another
 it assumes that the production configuration will be the one used in the benchmark
o staff will have equal levels of training
o hardware and software will be identical
o it is difficult to generalize from one configuration to another - e.g. claims that one
CPU is "twice as powerful" as another do not work out in practice
 however, any prediction, even with high levels of uncertainty, is better than none
o after a quantitative benchmark the analyst probably has better knowledge of
system performance than the vendor

G. AGT BENCHMARK EXAMPLE

Project Background

 in 1983, Alberta Government Telephones (AGT) had been operating a mechanized

drawing system for 5 years
o however, lack of "intelligence" in automated mapping system was increasingly
hard to justify given growing capabilities of GIS
o management was showing interest in updating the record-keeping system
 an FRS and RFP for an AM/FM system were developed by a consultant in cooperation
with staff
o three companies were identified as potential suppliers and a benchmark test was
designed
 tests included placement and modification of "plant" (facilities), mapping, report
generation, engineering calculations, work order generation
 tests were designed to be progressively more difficult
o all vendors were not expected to complete all tests
 data and functional requirements analysis were sent in advance to all vendors for
examination
o actual benchmark script and evaluation criteria were not sent in advance
o vendors were asked to load supplied data in advance of benchmark
o methods chosen to load and structure data were part of the evaluation
 o visits were made to each vendor 5 weeks before the actual benchmark to clarify
any issues
 providing the data before the script is typical of benchmarks for systems that are
primarily query oriented
o prevents planning for the queries that are presented in the script
o on the other hand, benchmarks for systems that are product oriented will normally
provide the script in advance
 in the AGT case, actual benchmarks were conducted by a team of 3, spending one full
working week at each vendor
o during the benchmark the vendor's staff were responsible for interacting with the
system, typing commands, etc.
o the benchmark team acted as observers and timekeepers, and issued verbal
instructions as appropriate
 must recognize that the vendor's staff are more familiar with the system than the typical
employee will be during production
o thus the benchmark is biased in favor of the vendor in its evaluation of user
interaction - the vendor's staff are presumed to be better than average digitizer
operators etc.
 during the benchmark, the intent of each phase of testing was explained to the vendor
o positive and negative evaluations were communicated immediately to the vendor
o the project team met each evening to compare notes
 a wrapup session at the end of the benchmark identified major difficulties to the vendor,
who was invited to respond
 when the three benchmarks were completed the results were assessed and evaluated and
became part of the final decision-making stages

REFERENCES

Goodchild, M.F., 1987. "Application of a GIS benchmarking and workload estimation model,"
Papers and Proceedings of Applied Geography Conferences 10:1-6.

Goodchild, M.F. and B.R. Rizzo, 1987. "Performance evaluation and workload estimation for
geographic information systems," International Journal of Geographical Information Systems
1:67-76. Also appears in D.F. Marble, Editor, Proceedings of the Second International
Symposium on Spatial Data Handling, Seattle, 497-509 (1986).

Marble, D.F. and L. Sen, 1986. "The development of standardized benchmarks for spatial
database systems," in D.F. Marble, Editor, Proceedings of the Second International Symposium
on Spatial Data Handling, Seattle, 488-496.

EXAM AND DISCUSSION QUESTIONS

1. Discuss the Marble and Sen paper listed in the references, and the differences between its
approach and that presented in this unit.
2. How would you try to predict CPU utilization in the polygon overlay operation? What
predictors would be suitable? How well would you expect them to perform based on your
knowledge of algorithms for polygon overlay?

3. Since a computer is a mechanical device, it should be perfectly predictable. Why, then, is it so

difficult to forecast the resources used by a GIS task?

4. Compare the approach to GIS applications benchmarking described in this unit with a standard
description of computer performance evaluation, for example D. Ferrari, 1978, Computer
Systems Performance Evaluation. Prentice Hall, Englewood Cliffs, NJ.

5. In some parts of the computing industry, the need for benchmarks has been avoided through
the development of standardized tests. For example such tests are used to compare the speed and
throughput rates of numerically intensive supercomputers, and of general-purpose mainframes.
Are such tests possible or appropriate in the GIS industry?

6. GIS product definition exercise - 2 following pages.

Back to Geography 370 Home Page

Back to Geography 470 Home Page

Back to GIS & Cartography Course Information Home Page

Please send comments regarding content to: Brian Klinkenberg

Please send comments regarding web-site problems to: The Techmaster
Last Updated: August 30, 1997.
 Benchmarking
From Wikipedia, the free encyclopedia

This article is about the business term. For the geo locating activity, see Benchmarking (geo locating). For
other uses, see Benchmark (disambiguation).

Benchmarking is comparing one's business processes and performance metrics to industry bests and best
practices from other companies. Dimensions typically measured are quality, time and cost. In the process of
best practice benchmarking, management identifies the best firms in their industry, or in another industry where
similar processes exist, and compares the results and processes of those studied (the "targets") to one's own
results and processes. In this way, they learn how well the targets perform and, more importantly, the business
processes that explain why these firms are successful.
Benchmarking is used to measure performance using a specific indicator (cost per unit of measure, productivity
per unit of measure, cycle time of x per unit of measure or defects per unit of measure) resulting in a metric of
performance that is then compared to others.[1]
Also referred to as "best practice benchmarking" or "process benchmarking", this process is used in
management which particularly shows VEMR strategic management, in which organizations evaluate various
aspects of their processes in relation to best practice companies' processes, usually within a peer group
defined for the purposes of comparison. This then allows organizations to develop plans on how to make
improvements or adapt specific best practices, usually with the aim of increasing some aspect of performance.
Benchmarking may be a one-off event, but is often treated as a continuous process in which organizations
continually seek to improve their practices.

Contents
[hide]

 1History
 2Procedure
 3Costs
 4Technical/product benchmarking
 5Types
 6Tools
 7Metric benchmarking
 8See also
 9References

History[edit]
The term bench mark, or benchmark, originates from the chiseled horizontal marks that surveyors made in
stone structures, into which an angle-iron could be placed to form a "bench" for a leveling rod, thus ensuring
that a leveling rod could be accurately repositioned in the same place in the future. These marks were usually
indicated with a chiseledarrow below the horizontal line. Benchmarking is most used to measure performance
using a specific indicator (cost per unit of measure, productivity per unit of measure, cycle time of x per unit of
measure or defects per unit of measure) resulting in a metric of performance that is then compared to others. In
1994, one of the first technical journal named"Benchmarking: An International Journal" was published.
In 2008, a comprehensive survey[2] on benchmarking was commissioned by The Global Benchmarking
Network, a network of benchmarking centres representing 22 countries.
 1. Mission and Vision Statements and Customer (Client) Surveys are the most used (by 77% of
organizations) of 20 improvement tools, followed by SWOT analysis (strengths, weaknesses,
opportunities, and threats) (72%), and Informal Benchmarking (68%). Performance
Benchmarking was used by 49% and Best Practice Benchmarking by 39%.
2. The tools that are likely to increase in popularity the most over the next three years are Performance
Benchmarking, Informal Benchmarking, SWOT, and Best Practice Benchmarking. Over 60% of
organizations that are not currently using these tools indicated they are likely to use them in the next
three years.
3. Mission and Vision Statements and Customer (Client) Surveys are the most used (by 77% of
organizations) of 20 improvement tools, followed by SWOT analysis (strengths, weaknesses,
opportunities, and threats) (72%), and Informal Benchmarking (68%). Performance
Benchmarking was used by 49% and Best Practice Benchmarking by 39%.
4. The tools that are likely to increase in popularity the most over the next three years are Performance
Benchmarking, Informal Benchmarking, SWOT, and Best Practice Benchmarking. Over 60% of
organizations that are not currently using these tools indicated they are likely to use them in the next
three years.

process
 There is no single benchmarking process that has been universally adopted. The wide appeal and
acceptance of benchmarking has led to the emergence of benchmarking methodologies. One seminal
book is Boxwell's Benchmarking for Competitive Advantage (1994).[3] The first book on benchmarking,
written and published by Kaiser Associates,[4] is a practical guide and offers a seven-step approach.
Robert Camp (who wrote one of the earliest books on benchmarking in 1989)[5] developed a 12-stage
approach to benchmarking.
The 12 stage methodology consists of:

1. Select subject
2. Define the process
3. Identify potential partners
4. Identify data sources
5. Collect data and select partners
6. Determine the gap
7. Establish process differences
8. Target future performance
9. Communicate
10. Adjust goal
11. Implement
12. Review and recalibrate
The following is an example of a typical benchmarking methodology:

 Identify problem areas: Because benchmarking can be applied to any business process or function, a
range of research techniques may be required. They include informal conversations with customers,
employees, or suppliers; exploratory research techniques such as focus groups; or in-depth marketing
research, quantitative research, surveys,questionnaires, re-engineering analysis, process mapping, quality
control variance reports, financial ratio analysis, or simply reviewing cycle times or other performance
indicators. Before embarking on comparison with other organizations it is essential to know the
organization's function and processes; base lining performance provides a point against which
improvement effort can be measured.
 Identify other industries that have similar processes: For instance, if one were interested in improving
hand-offs in addiction treatment one would identify other fields that also have hand-off challenges. These
could include air traffic control, cell phone switching between towers, transfer of patients from surgery to
recovery rooms.
 Identify organizations that are leaders in these areas: Look for the very best in any industry and in any
country. Consult customers, suppliers, financial analysts, trade associations, and magazines to determine
which companies are worthy of study.
 Survey companies for measures and practices: Companies target specific business processes using
detailed surveys of measures and practices used to identify business process alternatives and leading
companies. Surveys are typically masked to protect confidential data by neutral associations and
consultants.
 Visit the "best practice" companies to identify leading edge practices: Companies typically agree to
mutually exchange information beneficial to all parties in a benchmarking group and share the results
within the group.
 Implement new and improved business practices: Take the leading edge practices and develop
implementation plans which include identification of specific opportunities, funding the project and selling
the ideas to the organization for the purpose of gaining demonstrated value from the process.

Costs[edit]
The three main types of costs in benchmarking are:

 Visit Costs - This includes hotel rooms, travel costs, meals, a token gift, and lost labor time.
 Time Costs - Members of the benchmarking team will be investing time in researching problems, finding
exceptional companies to study, visits, and implementation. This will take them away from their regular
tasks for part of each day so additional staff might be required.
 Benchmarking Database Costs - Organizations that institutionalize benchmarking into their daily
procedures find it is useful to create and maintain a database of best practices and the companies
associated with each best practice now.
The cost of benchmarking can substantially be reduced through utilizing the many internet resources that have
sprung up over the last few years. These aim to capture benchmarks and best practices from organizations,
business sectors and countries to make the benchmarking process much quicker and cheaper.[6]

Technical/product benchmarking[edit]
The technique initially used to compare existing corporate strategies with a view to achieving the best possible
performance in new situations (see above), has recently been extended to the comparison of technical
products. This process is usually referred to as "technical benchmarking" or "product benchmarking". Its use is
well-developed within the automotive industry ("automotive benchmarking"), where it is vital to design products
that match precise user expectations, at minimal cost, by applying the best technologies available worldwide.
Data is obtained by fully disassembling existing cars and their systems. Such analyses were initially carried out
in-house by car makers and their suppliers. However, as these analyses are expensive, they are increasingly
being outsourced to companies who specialize in this area. Outsourcing has enabled a drastic decrease in
costs for each company (by cost sharing) and the development of efficient tools (standards, software).-

Types[edit]
Benchmarking can be internal (comparing performance between different groups or teams within an
organization) or external (comparing performance with companies in a specific industry or across industries).
Within these broader categories, there are three specific types of benchmarking: 1) Process benchmarking, 2)
Performance benchmarking and 3) Strategic benchmarking. These can be further detailed as follows:

 Process benchmarking - the initiating firm focuses its observation and investigation of business processes
with a goal of identifying and observing the best practices from one or more benchmark firms. Activity
analysis will be required where the objective is to benchmark cost and efficiency; increasingly applied to
back-office processes where outsourcing may be a consideration. Benchmarking is appropriate in nearly
every case where process redesign or improvement is to be undertaking so long as the cost of the study
does not exceed the expected benefit.
 Financial benchmarking - performing a financial analysis and comparing the results in an effort to assess
your overall competitiveness and productivity.
 Benchmarking from an investor perspective- extending the benchmarking universe to also compare to
peer companies that can be considered alternative investment opportunities from the perspective of an
investor.
 Benchmarking in the public sector - functions as a tool for improvement and innovation in public
administration, where state organizations invest efforts and resources to achieve quality, efficiency and
effectiveness of the services they provide.[7]
 Performance benchmarking - allows the initiator firm to assess their competitive position by comparing
products and services with those of target firms.
 Product benchmarking - the process of designing new products or upgrades to current ones. This process
can sometimes involve reverse engineering which is taking apart competitors products to find strengths
and weaknesses.
 Strategic benchmarking - involves observing how others compete. This type is usually not industry
specific, meaning it is best to look at other industries.
 Functional benchmarking - a company will focus its benchmarking on a single function to improve the
operation of that particular function. Complex functions such as Human Resources, Finance and
Accounting and Information and Communication Technology are unlikely to be directly comparable in cost
and efficiency terms and may need to be disaggregated into processes to make valid comparison.
 Best-in-class benchmarking - involves studying the leading competitor or the company that best carries out
a specific function.
 Operational benchmarking embraces everything from staffing and productivity to office flow and analysis of
procedures performed.[8]
 Energy benchmarking - process of collecting, analysing and relating energy performance data of
comparable activities with the purpose of evaluating and comparing performance between or within
entities.[9] Entities can include processes, buildings or companies. Benchmarking may be internal between
entities within a single organization, or - subject to confidentiality restrictions - external between competing
entities.

Tools[edit]
Further information: List of benchmarking methods and software tools

Benchmarking software can be used to organize large and complex amounts of information. Software
packages can extend the concept of benchmarking and competitive analysis by allowing individuals to handle
such large and complex amounts or strategies. Such tools support different types of benchmarking (see above)
and can reduce the above costs significantly.

Metric benchmarking[edit]
Another approach to making comparisons involves using more aggregative cost or production information to
identify strong and weak performing units. The two most common forms of quantitative analysis used in metric
benchmarking are data envelope analysis (DEA) and regression analysis. DEA estimates the cost level an
efficient firm should be able to achieve in a particular market. In infrastructure regulation, DEA can be used to
reward companies/operators whose costs are near the efficient frontier with additional profits. Regression
analysis estimates what the average firm should be able to achieve. With regression analysis, firms that
performed better than average can be rewarded while firms that performed worse than average can be
penalized. Such benchmarking studies are used to create yardstick comparisons, allowing outsiders to
evaluate the performance of operators in an industry. Advanced statistical techniques, including stochastic
frontier analysis, have been used to identify high and weak performers in industries, including applications to
schools, hospitals, water utilities, and electric utilities.[10]
One of the biggest challenges for metric benchmarking is the variety of metric definitions used among
companies or divisions. Definitions may change over time within the same organization due to changes in
leadership and priorities. The most useful comparisons can be made when metrics definitions are common
between compared units and do not change so improvements can be verified.