Tender Evaluation Process Notes

May 2009
 Tender Evaluation Process Notes

Contents

1. INTRODUCTION, PURPOSE AND SCOPE........................................................... 3

2. TENDER RESPONSE EVALUATION AND SELECTION.......................................... 4

2.1 OVERVIEW ...................................................................................................... 4
2.2 EVALUATION PROCESS ........................................................................................ 4
2.3 EVALUATION TEAM .......................................................................................... 13
2.4 EVALUATION CRITERIA ...................................................................................... 16
2.5 EVALUATION AND SCORING PROCEDURE ................................................................. 17
2.6 COMMUNICATION WITH RESPONDENTS .................................................................. 18

Page 2 of 18
 Tender Evaluation Process Notes

1. Introduction, Purpose and Scope

This document contains some notes on a possible approach to tender evaluation.

Page 3 of 18
 Tender Evaluation Process Notes

2. Tender Response Evaluation and Selection

2.1 Overview

The objectives of the evaluation process are twofold:

• To select the most appropriate proposal to meet requirements

• To create an evaluation audit trail and set of supporting documentation that
will support the evaluation process and final decision.

The output from the process will be a Vendor and Solution Evaluation and
Selection (VSES) report.

The document describes the evaluation criteria that were applied to responses to
the tender.

Any evaluation process is liable to subjectivity. The evaluation process described

below can be subjective in:

• The selection of the evaluation factors and the relative weights assigned
• The scores assigned to each vendor

The purposes of a formal evaluation process are:

• To eliminate subjectivity as much as possible from the evaluation process

• To provide transparency in the evaluation process
• To ensure that all proposals are evaluated in the same way

2.2 Evaluation Process

The generalised solution evaluation process can be customised to meet the exact
needs.

The VSES report will have the following general structure:

Part A – Overview
1. Executive Summary
Part B – Short List Selection
2. List of Vendors Contacted
3. Product Decision Model
4. RFP Findings
5. Summary of Short List Recommendation
Part C – Finalist Selection
6. Reference Checks and Site Visits
7. Functional Demonstration Findings

Page 4 of 18
 Tender Evaluation Process Notes

8. Technical Review Findings

9. Solution Demonstration Planning Lab Findings
10. Summary of Finalist Recommendation
Part D – Appendices Containing Supporting Material (optional)

Schematically, the high-level view of the evaluation process that is proposed to

be used during the evaluation and selection of a reporting solution is:

Page 5 of 18
 Tender Evaluation Process Notes

In summary the evaluation process consists of the following elements:

Page 6 of 18
 Tender Evaluation Process Notes

• Evaluation Team - the evaluation team consists of individuals with

appropriate skills to be able to evaluate responses received from vendors

• Evaluation Process - the evaluation process is the set of steps to be

followed by the evaluation team to conduct an evaluation of the responses

• Evaluation Criteria - this is a set of criteria by which the responses will be

assessed

• Evaluation Score - the score is an objective assessment of the merits of a

response based on the application of the evaluation criteria

• Evaluation Report - this is a document describing the evaluation process

and its results

In more detail, the steps in the process are:

Step Description
Start VSES The amount of effort involved in starting the Vendor and Solution
Process Evaluation and Selection (VSES) step depends on:

How recently you made any VSES scope and direction decisions
•
Whether the team has already been selected
•
How familiar each team member is with the functional area
•
within which the application is to be deployed, and the systems
currently in use in BoI to support Monthly Financial Performance
Reporting
• How experienced the team is with vendor and solution
evaluation and selection.
Confirm Scope Review previous decisions made about the scope of the project.
and Approach Discuss any scope changes and adjust the work plan to reflect
approved changes.

Consider the urgency of need, and determine how it impacts the

selection strategy.

Determine the timing and general strategy for contract negotiation.

Determine what parties to involve in defining procurement

procedures, handling legal questions, and negotiating the contract
so that the right people can be involved at appropriate points.
Establish and Assemble and organise a solution evaluation and selection team.
Train the Observe the following guidelines:
Evaluation
Team • Include technical personnel who understand the systems
management and support processes that are being implemented
• Include business personnel who understand the operational
objectives, business processes and needs of Executive

Page 7 of 18
 Tender Evaluation Process Notes

Step Description
stakeholders and sponsors of the project.
• Include subject matter experts in Best Business practice in
Business Performance reporting as needed.
• Include experts who understand the market context within
which the vendors operate so as to ensure an understanding of:
o The capability and capacity of service providers on the
local market where resources will be required
o The extent to which the vendors solutions will need to
be customised to meet the requirements identified in
the RFP
o The market for services based activities
(consulting/training/application development etc)
• Provide for a workshop facilitator with a solid background in the
subject area if you plan to use facilitated workshops as part of
the evaluation and selection process.
• Keep the core group small but representative of the functional
areas the product most affects.
• Include as support members anyone involved in defining
procurement procedures, handling legal questions, and
negotiating the contract.
• Include management personnel responsible for approving the
selection

Once you have established the evaluation and selection team,

designate team members to be the primary and secondary vendor
contacts. The secondary vendor contact works with the vendor
when the primary contact is unavailable, such as during vacation or
illness. These individuals represent the interests of the client and
the systems integrator, answer questions about the solution
evaluation and selection process, strategise with the vendor,
arrange vendor contact with other members of the team, and are
empowered to make informal commitments.

Next, designate a team librarian to administer the vendor

scheduling, mail all vendor requests, record all vendor contact,
obtain and distribute vendor materials, and maintain a filing system
for vendor materials.

Walk through the evaluation and selection activities with team

members to familiarise them with the general process. If the team
is relatively inexperienced with solution evaluation and selection,
consider using a team-building exercise to preview the process.
Scale down and simulate evaluation and selection activities. Save
detailed discussions for just-in-time training sessions.

Discuss the objectives and principles of the evaluation and selection

process with the team to ensure that everyone has a consistent
understanding.

Page 8 of 18
 Tender Evaluation Process Notes

Step Description
Compile Compiling criteria when selecting a solution involves:
Function
al • Developing or gathering criteria appropriate to the evaluation
Criteria method
• Avoiding excess precision
• Keeping the focus on business priorities
• Designing a score sheet that facilitates getting and analysing
vendor responses
• Incorporating the criteria into the score sheet.

These criteria will be taken from the tender.

In this activity, define the functional criteria - the requirements as

seen from the business user’s point of view. These requirements
represent the business processes the solution needs to
accommodate. For more information on developing application
selection criteria and designing a score sheet, see the Product
Evaluation technique.
Design functional criteria and group them as follows:

• Business processes
• Data
• External system interfaces
• Business performance measurement.

Assemble the functional criteria into one document, the Functional

Criteria List.
Compile Define technical criteria to represent the solution design and
Technical technical infrastructure aspects of the solution. The technical
Criteria requirements concern the solutions operation and infrastructure.

To enhance readability and minimise oversights, develop categories

of technical criteria in the areas of:

• Systems management and support process

• Operational interface
• Technical infrastructure
• Volume and technical performance.

Assemble the technical criteria into one document, the Technical

Criteria List.

These criteria will be taken from the tender.

Compile The buyer of a reporting solution is buying more than a service. In

General most cases he or she is also developing a long-term relationship
Criteria with the vendor, expecting some, if not all, of the following:

Page 9 of 18
 Tender Evaluation Process Notes

Step Description
• Initial transition, configuration and implementation services
• Custom services
• Consulting services
• Future changes
• Customer support services, such as a hot line and a user group.

General criteria include “vendor” and miscellaneous criteria. These

criteria are factors that influence the solution purchase decision but
are not functional or technical in nature. General criteria require
that you answer them through an iterative process, which requires
different tactics:

Concentrate on criteria that would eliminate vendors from the short

list. Consult with legal and purchasing personnel to extract general
criteria that can quickly narrow the solution selection.
Add criteria to ensure the team understands the vendor’s decision-
making and delivery processes, schedules, and organisation before
making a final solution selection and developing an implementation
plan.

With functional and technical criteria, you are usually looking for a
specific response. With general criteria questions, you are looking
for potential risks and information needed for deployment. Collect
responses to these criteria by asking the vendor directly, by asking
references who use the software, and by doing your own research.

Develop categories to enhance readability and minimise oversights.

Assemble the categories of general criteria into one document, the
General Criteria List.

These criteria will be taken from the tender as well as other more
specific requirements that extend the material in the tender.
Determine To narrow solutions options to a reasonable number, design an
Scoring overall package selection framework to communicate and
Approach summarise findings. Use weighting to reflect the relative
importance of criteria; use scoring to indicate how well the vendor’s
solution meets the criteria.

Weighting and scoring are often represented numerically, giving the

illusion of objectivity. Keep in mind that these numbers are really
just a quick means to communicate subjective judgments. When
making the package decision, check that the bottom-line numbers
are consistent with your overall judgment.
Conduct RFP The objective of this activity is to obtain and evaluate
Response comprehensive vendor proposals and presentations that explain
Evaluation how their products meet the specific criteria, and at what estimated
cost. The RFP has information instructing vendors to respond
formally, in writing, and possibly in a presentation. Vendors are

Page 10 of 18
 Tender Evaluation Process Notes

Step Description
compared uniformly and strictly as to how well they meet the
criteria, comply with the instructions, and present their solution.

The responses to the RFP will be evaluated with respect to the

criteria developed above.
Develop Script During the solution meetings with vendors, the team needs to
Cases compare solutions fairly and evenly, even though different solutions
are being proposed and discussed. This is hard for the team to do if
each vendor is allowed to determine the meeting’s sequence,
contents, and emphasis. By prescribing that all vendors follow a
planned sequence of requirements, or scripts, you:

• Ensure consistency of content coverage across vendors

• Ensure the vendor is presenting those features most important
to you, not just those the vendor puts forward as strong points
• Can more easily spot features and functions that do not quite
match expectations
• Can more easily distinguish how the features and functions
differ among solutions.
Select Vendor Conduct a workshop or a work session to obtain team consensus on
Short List whether or not to proceed with solution evaluation and to reduce
the list of vendors under consideration to three or fewer. In
general, use a workshop for large solutions or solutions with a
major impact such as the service being requested.
Check Vendor Check vendor references as another way to confirm that the
References vendors’ solution and services could meet the requirements.
Develop a questionnaire to record reference responses. Include
questions covering relevant functional, technical, and general
criteria. Involve members of the team in reference checking. This
distributes the work, counteracts individual bias, and provides
experience useful in subsequent VSES activities.
Conduct Have the short list vendors conduct functional and operation walk-
Functional and throughs using the script cases provided, so you can:
Operational
Review • Confirm that the solution, can process operate to your
Meetings satisfaction
• Determine, based on your own observations, which solution
provides the best answer to needs
Perform Conduct a technical review of the solution with the vendor’s
Technical technical staff. The vendor’s technical staff members are usually
Review involved in the provision of the solution and its underlying technical
components and are not as oriented as the sales staff toward
selling the solution. They usually paint a more realistic picture of
the solution and the vendor’s ability to support it. They can also
confirm technical infrastructure details about operating the solution.

The technical review typically has several components:

Page 11 of 18
 Tender Evaluation Process Notes

Step Description
• Demonstration of technical facilities and elements
• Review of remaining questions on technical and general criteria
responses
• Support and operational model comparison
• Performance investigation
• Review of solution architecture
• Conceptual technical infrastructure design
• Review of services delivery and quality
• Software, support and Operations costing
• Review the software release and patching strategy the vendor
employs

The importance and, hence, the amount of effort spent preparing

for and conducting the various components of the technical review
vary based on several factors.

During the technical review, each team member updates the

relevant criteria scores on a copy of the current Technical and
General Criteria Lists. The team members take notes justifying the
scores and identify other risk areas, strengths, and weaknesses. At
the end of the technical review, the team meets to reach consensus
on the technical review findings.
Visit Decide whether to visit vendor reference sites. Plan the approach,
Reference visit the sites, then consolidate the findings.
Sites
Perform This activity is a critical juncture for confirming the vendor’s
Financial financial estimates. Help make sure the vendor’s cost estimates are
Analysis as correct as possible by confirming the preliminary cost estimates.
Select Finalist The team is now prepared to select a finalist. Review the decision
to confirm that the reporting approach should still be pursued.
Determine contract negotiation timing and approach. Conduct a
workshop or a work session to select the solution finalist. In
general, use a workshop for a large solution such as that being
requested.
Complete Complete the evaluation process.
Evaluation
Complete the project records.

Finalise the evaluation report.

Notify the losing vendors. Prepare for any vendor loss meetings.

Conduct any vendor loss meetings

Assess satisfaction with the evaluation project.

Summarise the project results and the lessons learned.

Page 12 of 18
 Tender Evaluation Process Notes

Step Description
Review and recognise the team performance.

Close out the project records

This is a high-level view of the evaluation process that will be used.

The following additional collateral will be created and maintained during the
evaluation and selection process

• Evaluation Matrix. Create an Evaluation Matrix to record evaluations of

vendor responses to an RFS or RFP.
• Solution Decision Model. Create a Solution Decision Model reflecting how
you measure and evaluate the different evaluation components, such as RFP
responses, demonstrations, and site visits.
• Vendor Demonstration Support Materials. Create a Script Case Plan,
Process Scenario Worksheet, Script Case Summary Score sheets, and EBP
Summary Score sheets to plan vendor demonstrations, record observations,
and summarise team evaluations of the demonstrations.
• Solution Comparison Matrix. Update the Solution Comparison Matrix to
keep track of significant vendor and solution evaluation factors.
• Vendor Reference Questionnaire. Create a Vendor Reference
Questionnaire to record questions and answers for checking a vendor’s
references.
• Negotiated Vendor Agreements. Support the creation of Negotiated
Vendor Agreements. Create the agreements to bind the vendor and company
in a clearly defined and enforceable way regarding a solution the vendor will
supply.
• Strengths and Weaknesses List. Create Strengths and Weaknesses Lists
to record individual and team assessments of vendor and solution strengths
and weaknesses.

2.3 Evaluation Team

The evaluation team may be structured as follows:

Page 13 of 18
 Tender Evaluation Process Notes

The team will be lead by a team manager.

The team may be divided into a number of groups:

• Functional - this group will be responsible for evaluating the functional

aspects of the responses

• Technical - this group will be responsible for evaluating the technical aspects
of the responses

• General - this group will be responsible for evaluating general factors such as
the responding companies and the finances

No members of the evaluation team will be connected with any of the

respondents.

The team manager may be assisted by a number of specialists in particular

aspects of the development and operation of a reporting system.

The responsibilities of the Evaluation Team Manager will be:

• Managing the evaluation process

Page 14 of 18
 Tender Evaluation Process Notes

• Consulting with specialists as required

• Documenting the evaluation process

• Assign weights to the evaluation factors in conjunction with the Group Leaders

• Conducting meetings of the evaluation group leaders and of the whole team

• Preparing the final report on the evaluation

• Reviewing and understanding responses from respondents

• Acting as the single point of contact between respondents and other team
members

• Allocating and scheduling work for the evaluation groups

• Reviewing the outputs from the evaluation groups and seeking clarification

The responsibilities of the evaluation group leaders will be:

• Conducting meetings of the evaluation group

• Preparing evaluation task lists and other material to assist evaluation

• Reviewing and understanding responses from respondents

• Co-ordinating any questions to and responses from respondents from group

members

• Documenting the work of the group

• Reporting to the team manager as required

• Allocating and scheduling work between the members of the group

• Reviewing and discussing the work of the group as it takes place

• Summarising the results of the evaluation group

The responsibilities of the members of the individual group members will be:

• Reviewing and understanding responses from respondents

• Understanding the evaluation process

• Documenting their work

• Performing their assigned work in a timely and efficient manner

Page 15 of 18
 Tender Evaluation Process Notes

• Seeking clarification where required

• Responding to queries and requests for updates from the group leader and
team manager

2.4 Evaluation Criteria

The evaluation criteria will be taken from the RFP. They will be grouped. Each
criterion will be assigned a weight to reflect its relative importance in the required
solution.

The following table contains an illustrative list of evaluation criteria:

Category Factor Weight % Total

Category
Weight %
Response Response document 3.00% 3.00%
Document
Vendor Respondent stability 4.00%
Respondent references 4.00%
Respondent knowledge and 4.00%
experience
Respondent personnel 3.00%
Respondent certifications 1.00%
Project delivery ability 4.00% 20.00%
Methodology Methodology 2.00%
Risk Management 2.00%
Standards 1.00% 5.00%
System Design System design 3.00%
System performance and 2.00%
throughput
System usability 3.00%
Growth and expansion 2.00% 10.00%
Implementation Timescale 4.00%
Team 3.00% 7.00%
Software Tools Development language(s) 2.00%
Database management system 2.00%
Other software components 2.00% 6.00%
Security Connection to End User Company 2.00%
user directory
Flexibilility 2.00% 4.00%
Hardware Server infrastructure 1.50%
Client infrastructure 1.50%
Network infrastructure 1.00% 4.00%
System Operation User reporting 4.00%

Page 16 of 18
 Tender Evaluation Process Notes

User analysis 4.00%

System administration and 3.00%
management
Office interface 2.00%
Mail interface 2.00% 15.00%
Financial System development 3.00%
Software licenses 3.00%
Support and maintenance 2.00%
Hardware costs 1.00%
Upgrade costs 1.00%
Overall costs 10.00% 20.00%
Future Growth in data volumes 2.00%
Growth in users 2.00%
Additional reports 2.00% 6.00%
TOTAL 100.00% 100.00%

2.5 Evaluation and Scoring Procedure

The evaluation process involves the following steps:

• Weighting or classifying criteria based on importance to the overall project

• Scoring vendors ability to match criteria

• Multiplying score by the weight to derive final notional score

Importa Total Responde .. Responde Respond .. Respond

nce Possibl nt 1 nt M ent 1 ent M
Weightin e Score Score Score Score
g Score (Unweigh (Unweigh (Weighte (Weighte
(1-5) ted) ted) d) d)
Category 1
Factor 1
...
Factor N
...
Category P
Factor 1
...
Factor N
TOTAL

Deriving a scoring for the cost of the response could involve some formal
comparison of all the costs.

Page 17 of 18
 Tender Evaluation Process Notes

Respondents could be either assigned a simple score based on their rating in a

table of scores. However, this does not take into account large variations in
costs.

The cheapest response could be assigned the maximum number of points.

The differences between each response and the lowest response could be
calculated. The amount of the total difference each respondent is responsible for
can be used to calculate the number of points to be assigned to the vendor.

In the following example, the cheapest response is 75 and the largest in 100. The
total of all the differences from the cheapest is 110. So a respondent who
proposed a cost of 100 is responsible for nearly 24% of the total variation. The
respondent's score is the maximum score reduced by this percentage.

Cost Difference From Percentage of Points Assigned

Lowest Total Variation
75 0 0.00% 100
80 5 4.55% 95
85 10 9.09% 91
85 10 9.09% 91
90 15 13.64% 86
95 20 18.18% 82
100 25 22.73% 77
100 25 22.73% 77
110

2.6 Communication With Respondents

During the evaluation process, it may be necessary to communicate with

respondents.

All communication will be in writing, by letter, FAX or e-mail. Responses from

respondents must be in writing. These responses will be made available to
members of the Evaluation Team. The communications will take the form of a list
of issues requiring clarification. The communications will not disclose any
information on the perceived strength or weakness of the respondent’s proposal.

All respondent communications will be via a single member of the Evaluation

Team.

Page 18 of 18