Scenario-Based Ecommerce Technology Evaluation Process

May 17, 2010
Scenario-Based eCommerce
Technology Evaluation Process
by Brian K. Walker
for eBusiness & Channel Strategy Professionals
Making Leaders Successful Every Day

For eBusiness & Channel Strategy Professionals
May 17, 2010

Scenario-Based eCommerce Technology
Evaluation Process
Using The STEP Methodology To Move Beyond The RFP In Evaluating
eCommerce Technology
by Brian K. Walker
with Patti Freeman Evans and Brendan McGowan
Exec ut i v e S u mma ry
The eCommerce technology selection process is broken. Both buyers and technology providers are
feeling the pain. The courtship ritual of requests for proposals (RFPs), vendor demos, and reference
checks routinely fail to show how technologies and solutions will meet fundamental business needs.
Solutions become homogenized by this process, and buyers and sellers alike are hard pressed to find
ways to differentiate. How will the solution address the business requirements, enable the desired
customer experience, or meet the technology and integration objectives differently from other options?
To help clients answer these important questions, Forrester has devised and vetted a better way to make
these high-cost complex decisions. This approach — a scenario-based technology evaluation process
(STEP) — will lead to a better fit between buyer and seller and one that ultimately leads to multichannel
eCommerce success.
tabl e o f Co nte nts NOT E S & R E S OUR CE S

2 The Problems With The eCommerce Selection Forrester interviewed both vendor and user
Process Today companies for this research.
It’s Not Just The Buyers That Are Underserved
By Traditional RFPs Related Research Documents
“2010 US Online Retail Technology Investment
4 Time To Reinvent The Process — Scenario-
Based Technology Evaluation Outlook”
January 14, 2010
Placing STEP In Context
“Keys To Successful eCommerce Platform Projects”
The RFP Is Dead; Long Live The RFI
August 3, 2009
STEP: Running A Scenario-Based Technology
Evaluation
What IT MEANS
14 How Scenario-Based Technology Evaluation
Can Fit Into A Larger Project
How Forrester Can Help
14 Forrester’s Resources Can Help You Succeed
© 2010, Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited. Information is based on best available
resources. Opinions reflect judgment at the time and are subject to change. Forrester®, Technographics®, Forrester Wave, RoleView, TechRadar,
and Total Economic Impact are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. To
purchase reprints of this document, please email clientsupport@forrester.com. For additional information, go to www.forrester.com.
2 Scenario-Based eCommerce Technology Evaluation Process
The Problems with the eCommerce Selection Process Today

Today, the typical eCommerce technology selection process follows a fairly prescribed and well-
worn path (see Figure 1).1 That process places a large emphasis on the RFP as the primary selection
tool. Add to that the invitation for the vendors to come in with their demo jockeys and run through
a highly scripted demo of the product, focusing on what they — the vendors — care about (see
Figure 2). But this traditional process fails at a very high rate because:
· Business stakeholders don’t know what to ask for. Large-scale eCommerce technology
selection is not something any eCommerce business does every day. Businesses often lack clear
goals and objectives for their projects, context for the tools they need to succeed, and clarity on
what to expect from the technology in a given category. They also often lack an understanding
of how a new technology must integrate into their existing environment in order to deliver
value. Whether it’s their first time through the process or their 10th, businesses need to focus
on their business problems and opportunities and see how technology and services will enable
them over time.
· The RFP becomes the kitchen sink. Since it can be so difficult to define what is needed up front,
the natural inclination is to ask for everything and sort it out later. The RFP process is often the
dumping ground for every possible requirement, collected from every possible corner of the
business. Little attention is paid to value of the capability, such as return on investment (ROI),
or how it adds to the business’ ability to drive the online experience. This scope creep makes the
rest of the process more difficult with vendors that lead the pack and manage to check off the
most requirements, irrespective of the value they can bring by doing so. If you are a vendor, the
purpose of an RFP response is to stay in the game. It is very tempting to say “yes” to something,
even if you know it will be difficult. It is very hard to say “no,” especially to an ill-defined
requirement that can be easily interpreted in multiple ways.
· The RFP exposes the “what” while hiding the “why” or “how well.” Unfortunately, RFPs
provide a very normalizing effect on describing how well vendors can meet a business’
requirements. Templates for RFPs are often shared among businesses, so your RFP sounds like,
looks like, and includes the same requirements as many other RFPs.2 Vendors respond to RFPs
by having teams who busily copy and paste to get the responses out quickly. Little time is spent
getting to know you, asking clarifying questions, or trying to understand the “why” behind
what you are asking for. In the end, little may jump out to really differentiate vendors in an RFP-
driven process, and you may miss what makes a particular vendor a great fit for your business
needs.
· The RFP is devoid of context. Typically, the RFP will contain a description of the current
state and then launch into a long list of requirements with little explanation of why these
requirements are important or how you can evaluate the capabilities of the vendor in your
May 17, 2010 © 2010, Forrester Research, Inc. Reproduction Prohibited

Scenario-Based eCommerce Technology Evaluation Process 3
context. As one senior vice president (SVP) of eCommerce of a large multichannel retailer who
is frustrated by multiple project failures put it, “Can you identify one eCommerce platform
project that has been a success? The track record is terrible. Are we just making bad picks, or are
they really this bad?”
It’s Not Just The Buyers That Are Underserved By Traditional RFPs
As a vendor, it can be frustrating for a competitor that is a year or more behind you in a certain
capability to come across as if it can support something just as well as you can. Many vendors report
that the typical technology selection processes level the playing field too much, making it hard
to determine the difference between what and how the vendors meet requirements. The typical
selection processes do not offer vendors the opportunity to differentiate and really explain what they
can do. There is not an opportunity for vendors to illustrate their knowledge of the business problem
and how they are well positioned (or not) to meet the business and technology requirements with a
client over the long haul. Often, vendors do not — or don’t have a chance to — understand the full
context of the prospective clients’ needs and don’t ask enough questions, leaving the clients with
scripted demos and presentations that fail to put the vendors’ products into their context and don’t
illustrate potential value.
Figure 1 Typical eCommerce Technology Selection Process
Justification Request
Vendor demos Negotiation Project plan
of project for proposal
56754 Source: Forrester Research, Inc.
Figure 2 Demos And References Matter Most To Selection Quality
“From your perspective, which two selection process steps were most important in the outcome
of the deal?”
Reference calls/visits 75%
Scripted demonstrations 64%
Limited-use product trial or proof-of-concept test 31%
A formal request for proposal (RFP) 22%
Other 9%
Base: 36 software vendors with application selection project experience
(multiple responses accepted)
Source: May 2007 Application Selection Projects Online Survey
© 2010, Forrester Research, Inc. Reproduction Prohibited May 17, 2010

Time to reinvent the process — scenario-based Technology evaluation

There is a better way. In researching how to improve the process, we have developed a framework
that should drive better eCommerce technology selections.3 At the core of this improved process is
an emphasis on business needs and a requirement that the vendors under evaluation demonstrate
how they address those needs. The approach requires businesses to define scenarios that represent
key business needs and requires the vendors to demonstrate how their products serve those
scenarios. In doing so, clients also recognize that long, burdensome lists of requirements become
less critical to their selection process than the domain knowledge represented by the vendors and
how they meet these needs with varying quality. Thus, key benefits of the process include:
· Specific knowledge of how well your goals and challenges are understood and supported.
Rather than specifying your requirements — which can be a very difficult challenge itself — and
trying to judge from an RFP response, you present your problems and goals at a higher level.
You then see specific demonstrations of how well the vendor understands these issues and meets
these goals.
· Key differences in the products that emerge more clearly. With the scenario-based evaluation
process, you will see how intuitive the tools are to you and your team as well as have a greater
opportunity to take advantage of specific vendors’ capabilities because of how the scenario is
supported. A scenario-based evaluation puts these tools into your context and forces vendors
to show you specifically how they meet your needs, removing the possibility that they will only
highlight strengths or bypass problem areas in a vendor-driven demo.
· Clear explanation of how the capability is working with other clients. By asking the vendors
to step through the scenarios, you will more clearly understand what a core feature of the
product is or what must be supported through customization. This is much harder to detect in a
written RFP response or vendor-driven demo.
· Insight into what is supported by base applications versus via customization. By not being
given too much time to prepare, the vendors will be demonstrating the scenarios with their
base application. Within STEP, vendors should be encouraged to explain how customization
can enable a scenario even when not demonstrable. Customization or configuration is not
necessarily a bad thing, but it should be well understood up front.
· Valuable insight into the culture, people, and vision of the vendor. When selecting
eCommerce technology, you are not picking a commodity product or service. You are instead
picking a long-term partner, one you will likely be working with for five to 10 years. You will
rely on partners for updates, upgrades, fixes, and creative solutions. How well they understand
and relate to your business, and serve related customers, may mean the difference between a
successful relationship and a failed investment. The scenario-based evaluation process is the
best way to get insight into how the vendors think, their domain knowledge and understanding
of your business, as well as the quality, communication style, and fit of their people.

Placing STEP In Context

STEP is a critical component in a larger technology selection process and must be placed inside that
larger context. Generally, a well-run vendor selection process will have three key dimensions:
· Evaluating how the vendor supports business requirements. This will include how the
application or service will meet customer experience, business management, and marketing
requirements. Often, these are the requirements that will make up the bulk of a written
RFP. This also extends to how well the vendor can meet future business needs through the
application’s flexibility as demonstrated by given functionality, professional services provided by
the vendor, and the vendor’s product road maps. This is the selection dimension that scenario-
based technology evaluation primarily addresses.
· Evaluating how well the vendor will meet technology requirements. For on-premises
solutions intended to be run and supported internally, evaluation of architecture, scalability, and
development approach is critical. The technology team will typically take the lead in evaluating
the application architecture and integration capabilities. Its members will also look at how well
the application can be supported by the current IT staff ’s skill sets and knowledge. This is where
training and staff-augmentation needs (and costs) may be identified. However, increasingly,
eCommerce technology is being delivered in hosted/managed or software-as-a-service (SaaS)
delivery models, in which case this area of evaluation may be less critical.4 Instead, this may be
viewed in a long-term manner for potentially bringing the application in-house down the road,
or it may be skipped all together.
· Evaluating the vendor’s stability, client satisfaction, and ability to meet legal requirements.
Often, these steps may be managed or dictated through a vendor management and procurement
policy or department. Included here is an evaluation of the vendor’s financial situation, stability,
and potential for change of control, such as being acquired. Legal counsel will take the lead
in evaluating standard agreements and service-level agreements (SLAs) and provide advice
on potential problem areas. Typically, this evaluation area will inform how partner-friendly
the vendor is but may not ultimately determine the best fit since the business and technology
requirements will carry greater weight.
The RFP Is Dead; Long Live The RFI

While the RFP-driven process is very problematic — for the reasons we already outlined — the
RFP deliverable requested from vendors can be very useful. However, repositioning it as a request
for information (RFI) — and not a proposal — more properly aligns the document with being a
contributor to the overall decision process. We recommend that clients use RFIs as a means to:
· Collect background information on the prospective vendors. This information will be

important to assessing the financial stability of the vendor. While it is always important, this
part of the process is becoming more and more critical as companies evaluate working with

vendors that deliver solutions via SaaS or hosted/managed models because they will rely on
the stability of vendors and their solutions for the day-to-day online business and customer
experience.
· Get the vendors on record for negotiation and accountability. The RFI can be a way to hold
the vendors accountable for capabilities they say that they deliver and a way to gain information
useful in negotiations.
STEP: Running A Scenario-Based Technology Evaluation

A good scenario will require vendors to get into the mindset of a merchant and a marketer and
showcase how their tools enable the business to succeed. By using a few core best practices and
a basic process flow, running a STEP should be both easy and liberating for the project selection
team. Forrester suggests that you focus on three to four well-developed scenarios, with back-office
business process, customer experience, and operations capabilities woven into them. Future-facing
requirements and opportunities can be included as bonus elements to ensure that you are inviting
the vendor to demonstrate both leading capabilities and core ones.
There are essentially three key phases of STEP (see Figure 3):
Figure 3 Key Steps In A Scenario-Based Technology Evaluation
1) Prepare
• Assemble selection team, and establish a team lead.

• Clarify high-level requirements.
• Develop scenarios.
• Determine selection criteria and weighting.
• Identify shortlist of vendors.
2) Evaluate
• Conduct scenario-based labs.

• Score vendors based on their demonstrated capability through scenarios.
• Conduct technology evaluations.
• Conduct reference, background, and analyst checks.
3) Decide
• Determine top two vendors based on scoring.

• Obtain management buy-in.
• Begin contract negotiations.
• Validate via proof of concept.

1. Prepare. Lay the groundwork for STEP.
Assemble and organize the vendor selection team. By building a well-rounded team
comprised of multiple disciplines, you will gain much-needed buy-in to the overall selection,
identify issues up front including potential feature gaps, identify additional opportunities, and
educate stakeholders on your business and how it works (see Figure 4).
Ensure a clear and approved project charter, business case, and budget. Often, large
technology and customer experience projects begin without clear objectives, goals, or business
cases. Developing a project charter and gaining buy-in from selection team members and
executive leadership can streamline the process of final selection and decision-making later in
the process.
Define high-level business, customer experience, and technical requirements. A business

should never select technology for technology’s sake. Technology must always serve a business
objective with the goal of delivering customer value in a sustainable and profitable manner. In
order to make a viable technology and vendor selection, you must have clear — albeit high-level
— business requirements, a vision for your desired customer experience, and clear technical and
integration requirements and method definitions.5 This analysis will help clarify the appropriate
shortlist of vendors to evaluate and help you define your scenarios.
Figure 4 Forming A STEP Selection Team
Role Responsibilities
Project manager Running the selection process, ensuring consistency, being available to respond to
vendor questions
eCommerce business Ensuring that business requirements are met and that the tools will suit the
leader business
eCommerce merchant Ensuring that merchandising requirements are met and that tools are usable
eCommerce marketer Ensuring that marketing requirements are met and that interfaces are workable
eCommerce creative Ensuring that creative and usability requirements are met and that the tools for
director the creative resources are usable and effective
Technology leader Ensuring that the integration, hosting, technical compatibility, stability, and
redundancy requirements are met
Operations leader Ensuring that order management and customer service requirements are met
Handling contract negotiation and compliance with corporate policies such as

Legal/governance privacy, Children’s Online Privacy Protection Act (COPPA), Sarbanes-Oxley (SOX),
and payment card industry (PCI) standards
Finance Ensuring that business case meets standards and that financial reporting and
SOX compliance requirements are met

Know how you want to own and operate the technology. In today’s eCommerce technology
market, there are solutions delivered in a wide variety of models — often from the same
company. But comparing licensed/on-premises software, hosted/managed applications, as well
as SaaS and full-service offerings all within the same process is very problematic. Often, these
represent different ownership, control, accounting, and management options. By deciding how
you want to pay — license versus revenue share versus utilization — and how you want to own
and operate — doing everything in-house versus outsourcing only application support versus
outsourcing everything — you will help narrow down the list of appropriate vendors up front in
your STEP approach.
Determine your shortlist of three to four vendors to take through STEP. Based on your
high-level requirements and how you want to own, operate, and pay for the solution, you can
then determine an appropriate shortlist of vendors to work with.6 Use additional aides such as
industry research, interactions with industry analysts, or knowledgeable consultants to support
this process and gain third-party perspectives if necessary.7
Field an RFI to obtain key information. The RFI will be useful in evaluating procurement
selection criteria such as financial viability, partners, services offered, and references. In
addition, asking for a review of base capabilities may be important not only to the evaluation
process but also for holding the vendor accountable through the contracting process and
beyond.
Prepare to conduct technical evaluations if appropriate. Depending on the way in which the
technology or service will be run — and by whom — a technical evaluation may or may not
be necessary. But even if there is a lightweight Web services integration — such as JavaScript
tags — some level of technical evaluation should be performed. If you are going to own licenses
and launch this application in your environment, then it is critical. If you rely on a third-party
service provider, have the provider perform this evaluation and provide its input, perhaps with
an IT or developer resource from your company participating as well. This evaluation should
look at the architecture, scalability, coding standards, services standards, and integration
patterns.
2. Evaluate. Set up scenarios for use in evaluations.
Understand how your business works today. Know the smooth and rough points of your
operations from a business-process standpoint. Evaluate some of the business-critical processes
necessary to manage the site and customer experience once your new site is live and into the
future.
Set up real-world cases. Use examples that have really happened if you have then, which will
help make things more real. For example, one client used a scenario that included a product

feature on Oprah in which it had only two days’ notice. Another client used an eCommerce site
launch in Japan.
Design scenarios around multiple business processes. For example, what processes are in
place to add product, content, and new pages to your site. Look at these processes in the context
of how they are tied to overall initiatives such as launching new brand/products or launching in
new countries.
List workflow steps and how you want the system to respond. Illustrate the roles involved
in executing these business processes and important workflows. Require that the scenario is
demonstrated using those multiple roles and how each individual will interact with the tools —
such as email, the tools user interface, or even short message service (SMS) text messages.
Don’t stop with the base execution of the tasks. Build analysis, testing, and optimization tasks
into the scenarios and ask to see real examples of how this will work. In some cases, this will
require a third-party application to be used in concert with the vendors’ products. And if that’s
the reality, then so be it — it’s better to see it and understand it up front.
Determine scenario scoring criteria and weighting. Develop a straightforward scorecard

to use in scoring the vendors on critical dimensions such as usability of the tools, ability
to demonstrate the scenario effectively, and understanding of the business reasons behind
the scenario. All selection team participants will use the scorecard to evaluate the vendors
individually during the scenario sessions and as a platform for discussion as you center in on
the vendor(s) you feel best meet your business needs and opportunities.
Communicate these scenarios to the vendors in written form. Work to develop scenarios (see
Figure 5). Provide vendors with data and brand assets available to allow them to fully respond to
your needs (see Figure 6). Give each vendor the same limited amount of time to work with the
scenarios — we suggest two weeks — before presenting to you.
Lay out clear ground rules for vendor presentations. Vendors will present their solutions
in live lab settings, which Forrester recommends to be 45 minutes in length. All vendors
will adhere to a consistent set of ground rules and explain their work by showing the tools
while executing the scenario (see Figure 7).8 Advise them to act the part that’s illustrated in
the scenario — such as a merchant, marketer, or customer service agent — and use the tools
accordingly. Instruct the vendors that if they do not provide the ability to execute certain
steps or if they require or rely on a third party, then they should explain clearly why they don’t
provide the ability or demonstrate the third-party tool if possible.
Bring vendors in for lab demonstrations of scenario execution. Again, the labs are designed
for the vendors to demonstrate how you would use their tool in specific scenarios. Do not

allow the vendor to log into the tool as an administrator and skip steps, unless you feel that you
understand the workflow tools enough and are looking to speed up the process. And insist on a
live demonstration of the tools and in the current version available for license and use today. It
is great to hear about planned enhancements, but you do not want to see works in progress, beta
products, or, worst of all, demo-ware (also lovingly referred to as “vaporware”).
Figure 5 Sample STEP Scenarios For eCommerce Platform Selection
Scenario No. 1: Add new products.

Scenario: 15 new products from a new designer need to be added to the Web site immediately.
Task:
- A merchandiser needs to add 15 apparel products.
• Each product comes in multiple sizes and colors.
• Products are imported in a raw state with minimal information from a legacy merchandising system.
- A new category must be created for the products.
• The products must be cross-populated among other categories.
- At least one product requires complete setup.
• Product content includes images, copy, descriptions, etc.
• Products require scheduling for go-live.
• Products must be excluded from “free shipping” promotions.
Scenario No. 2: Implement microsite.
Scenario: Your firm just received word that a very popular talk show host will be featuring a review of some of
the company’s products next week. You need to get a microsite up and running with a subset of the
company’s products as soon as possible.
Task:
- Set up a microsite with celebrity branding.
• Create landing page, category page, and product pages.
- Using existing product catalog, assign products to the microsite.
- Set up site for SEO and SEM, targeting via incoming links.
Bonus:
Version microsite in one or more languages and one or more currencies with US-based fulfillment
Scenario No. 3: Set up promotions with A/B and targeting.

Scenario: Set up two promotions.
Task:
- First promotion: “Buy two, get third free”
• Exclusion by category
• Exclusion by manufacturer
• Exclusion by specific product
- Second promotion: “Free shipping” at cart sizes of $50, $100, and with no threshold
• Use A/B test to show eﬀectiveness of each variation.
• Show reporting interface (including if through partner).
Bonus:
Show how these promotions may be targeted at specific groups: new customers, customers who have not
bought in one year, and customers who come through a specific link.

Figure 6 Example Of Scenario-Based Technology Evaluation Scorecard Framework
Score Weighted Detailed

Criteria Basis for assessment Weight (1 [low] to 5 [high]) score notes
Ability to meet See separate = [weight %]
scenario No. 1 scenario scorecard % X [score]
Ability to meet See separate
scenario No. 2 scenario scorecard %
Ability to meet See separate
scenario No. 3 scenario scorecard %
Usability of tools Intuitive ease of use, %

role and process
orientation,
interoperability
Functional fit to How well system meets

business requirements key functional require- %
ments, based on RFI
Reporting capabilities Ad hoc tools and
standard reports, %
suitability for
business users
Technical criteria
Configurability of
Software flexibility rules, process
(configuration without
workflows, %
customization) data formats and UI
without customization
Match to existing Use of existing tools
technology and technology skills
standards and tools %
within the
organization or vendors
Integration with Ability to integrate
key systems into existing
technology skills within %
the organization or
vendors
Other criteria
Cost Overall cost — licenses,
implementation,
ongoing (highest %
cost gets lowest score)
Vendor viability Financial condition,
customer base, market %
presence, long-term
vision
Customer references Customer satisfaction
scores based on %
three reference calls
Total score

Figure 7 Tips For Running Scenario-Based Technology Evaluation Labs
Purpose
Demonstrate the ways in which your solution addresses our eCommerce business needs.
Ground rules
• Each vendor will be asked to work through the same scenarios.
• We ask that each scenario be demonstrated in 45 minutes, with 15 minutes at the end for Q&A.
• Your demonstration must feature the current, presently-available version of the product.
• You can indicate how the next revolution of the product will improve/evolve the capability, but the lab-based
evaluation is intended to be a review of the current product.
• It must be a live demonstration of the tools:
• If your tools are accessed through a Web browser, we want a live demonstration through a Web browser.
• If your tools are desktop tools, please have an instance running in real time that you can use.
Evaluating
• Your product’s ability to address the given scenarios
• Tools user interface (UI), including workﬂow tools
• Overall tool set
• Our need to understand the breadth, depth, and clarity of product fit when it comes to the common
eCommerce tasks of the merchandiser, marketer, content contributor, and administrator — even beyond
those employed in the given scenarios
3. Decide. Conduct STEP.
Score vendors based on their capabilities demonstrated through scenarios. Immediately

following each vendor session, allocate 30 minutes to have the team individually score,
aggregate, and discuss what the team experiences during the lab. This step allows the team to
compare notes and observations, clarify and identify follow-up questions for the vendor, and
outline issues to discuss with references. Ask the team to describe why they gave specific scores.
If the team disagrees or views the vendor very differently, do not try to necessarily come to
agreement, but make sure the different views are documented. This is also a great opportunity
to capture the cultural alignment of a vendor to your business. Decide by finalizing your
evaluation and making a selection.
Conduct reference, background, and analyst checks. Among the most important ways you
will learn about a vendor is through reference checks. Ask the vendor for four or five references,
even if you are planning to interview fewer, and then work your network to find one or two
unofficial references you can talk to about their experience to get a balanced view of the vendor.
You may also ask your chief financial officer (CFO) or accounting department to conduct
financial background checks, and it is not uncommon for a vendor to submit financials for
review. This is also a good time to get the perspective of and validation from third-party experts
and industry analysts.

Select your vendors for price and contract negotiation. While a clear leader may emerge from
the STEP approach, common sense tells us that it can be important to have a backup plan and
balancing alternative in the works. Forrester recommends that you select the top two vendors
from the STEP approach for detailed statement-of-work definitions, price negotiations, and
contract negotiations. By investing time in a rigorous process to define the work, terms, and
price, you will avoid surprises once work begins and have a clear path to resolving issues if they
occur. Having two vendors in the process can ensure that you both get a good deal and have a
reasonable and balanced view of what terms to drive — such as SLAs with meaningful penalties
for noncompliance.
If time allows, and depending on technology, validate through a proof of concept (POC).
Literally test-drive the application. Use your scenarios with your data and your resources (or
those you have contracted with) to launch a POC/beta to discover challenges and opportunities
before the ink is put to paper. This step may be seen as a luxury, but with the advent of cloud-
based services — from Amazon.com, Microsoft, and IBM, among others — POCs are becoming
less expensive and time-consuming than in the past. Many eCommerce technology applications
are experimenting with running their applications in these environments, and the POC is a
great use of these instances. If you are using SaaS-based technology solutions, it should be even
easier to run POC and beta tests of the applications. Some vendors may balk at allowing their
technology to be used before a contract is final, in which case negotiate a short-term opt-out —
such as three months — if the technology or service does not meet you expectations.
Finalize procurement process and contract negotiation. This phase of the selection process is
often governed by companywide vendor selection processes and resources that are responsible
for adhering to vendor selection standards. If members of the procurement and legal teams have
been involved throughout the STEP approach and negotiations with the vendor are proceeding
well, this should be a relative formality. Typical sticking points will be contract opt-outs and
SLAs. Address these up front by collecting these options in the RFI, and be clear about your
expectations throughout the process.
Prepare the transition to your project development process. Congratulations, you have found
the vendor that will supply the technology or service you need to drive your business forward
and deliver the customer experience to help you compete and differentiate. But that is of course
just the beginning. Now the real work begins. Plan a transition that includes the business
sponsor and project manager from the selection process (and possibly others as well) to help
the project team — such as designers and developers — hit the ground running, understand the
value and opportunity, and streamline communication with the vendor. In the hopefully rare
instances where expectations are not being met and escalation is necessary, having the selection
team resources involved in the project directly will be very important.

W h at i t M e a n s
How SCenario-Based technology Evaluation Can Fit Into A Larger Project

Selecting technology to meet business or customer experience needs is typically one relatively
small, though critical, step in a larger project. For example, a site-redesign-and-relaunch
project will have many other critical phases that will lead to the success or failure of the project
independent of the technology or service provider you choose. STEP, however, can be integrated
in some useful and synergistic manners, such as:
· Identifying consumer scenarios for the design and testing of the project. The scenario
development for the technology selection may in fact help illuminate customer scenarios —
such as the discovery of, research of, comparison of, as well as multichannel shopping and
conversion of products. A scenario-based design process can feed scenarios straight into the
technology and service selection by clearly identifying how the site and tools should support
and respond to the customer experience.9
· Exposing how the business will manage and drive the business and site. By articulating
the business use cases through scenarios, it can become much clearer what may be
required to manage a new site and drive the customer experience. This may be a process
of rationalizing an innovative design against the pragmatic realities of running it every day.
Often, these tools are overlooked in a project, and STEP can help illustrate these early in a
project and in evaluating what technologies will lead to scalable, efficient operations of the
business and service roles required to excel at eCommerce.
· Guiding how the site and customer experience can be optimized over time. Effective
tools are critical to allowing the business and customer experience design teams to test and
optimize the site and multichannel commerce experiences. By identifying and evaluating
how these tools work through scenarios — not just that A/B testing is supported — you will
know how easy, or even fun, it will be for your teams to test, innovate, and deliver an optimal
experience to your customers and drive great business results. The STEP approach will make
sure that you are not stuck with a compromised and scaled-back design or one in which
some bad assumptions and design decisions were made. You need to tune your site over
time, and the tools you will evaluate through STEP are going to be critical to your success.
H o w F o r r este r C a n He l p
Forrester’s resources can help you succeed

Forrester advises clients through eCommerce technology selection in multiple ways:
· Inquiry process. Forrester clients may purchase inquiry access, which can be very useful
in connecting with relevant analysts to discuss your situation, project plan, and high-level
requirements as well as getting insights into the vendors appropriate for you. Many clients
use this in their shortlisting process, and this can be very useful in the STEP approach.

· Advisory support for eCommerce technology strategy assessment. Forrester can help
you optimize your opportunities to leverage existing and new investments in eCommerce
platforms and technology based on your business goals and the changing needs of your
customers. Forrester can provide a perspective grounded in research and experience in
supporting road map development, business cases, and eCommerce and multichannel
solution strategy with the goal of ensuring our clients’ success.
· Advisory support for eCommerce platform vendor selection projects. Using STEP,
Forrester can help you with your vendor selection process, serving as a well-informed guide
regarding the process, typical eCommerce requirements, and vendors in the eCommerce
technology marketplace. Forrester analysts are regularly briefed by eCommerce platform
providers. Analysts also publish research on major vendors in areas such as business-to-
business (B2B) eCommerce platforms, business-to-consumer (B2C) eCommerce platforms,
full-service providers, global eCommerce solutions, and solutions focused on digital product
delivery. Forrester analysts can provide objective, rigorous, and independent advice and
perspective on which vendors are best suited to support your eCommerce requirements
and guide you in executing a STEP evaluation. Forrester Consulting also supports RFI/RFP
development and complete vendor selection processes.
Endnotes
1
With this report on eCommerce technology selection, we are primarily focused on large application and
solution types such as platforms, search applications, content management, Web analytics, and marketing
suites. The methodology articulated in this report can be adapted to selection of any technology or service,
but dependent on the investment required and risks associated, a streamlined process may be called for.
2
It is also important to realize that vendors often have access to competitors’ RFP responses from other
projects they bid on. The goal becomes to respond in the RFP to stay in the game and make it to the final
cut.
3
The scenario-based evaluation approach is based on extensive research with our eCommerce clients, the
vendor community. We have put this to the test in our own technology evaluation processes to hone the
process and provide a framework for you to use in your own technology selections.
4
Online retailing looks to emerge from the economic downturn healthier than ever as a channel and as
a business. For a majority of online retailers, those results will be manifested in 2010 through increased
spending on eCommerce technology. Forrester’s Q4 US Online Retail Executive Online Survey shows that
eCommerce platform solutions, content management solutions, and marketing solutions look to benefit
the most from this increased spending. Also eCommerce executives are shifting how they look to purchase
and support these investments, with a marked shift away from internally supported technology and toward
externally supported solutions such as fully hosted/managed eCommerce solutions. See the January 14,
2010, “2010 US Online Retail Technology Investment Outlook” report.

5
A solid technology foundation is one of the keys to success for eCommerce businesses looking to optimize
their customer experiences, market effectively and creatively, and scale their business to make more
meaningful contributions to their companies. Yet many eCommerce business and technology professionals
will tell you that they struggled to get their eCommerce technology foundations to where they needed
them to be. Forrester worked with eCommerce business and technology leaders to identify important
lessons learned and key success factors to help businesses avoid trouble and meet the increasingly critical
eCommerce platform needs to run and optimize their businesses and customer experience. See the August
3, 2009, “Keys To Successful eCommerce Platform Projects” report.
6
To shortlist your vendor list effectively, rely on Forrester inquiry (emails or 30-minute calls with analysts)
and advisory consulting (analyst interactions based on our research, with a short timeline from client
interest to delivery). Source: Forrester Research (http://www.forrester.com/My/Contacts).
Additionally, Forrester’s reports are designed to assist you in the decision-making process. Source: Forrester
Research (http://www.forrester.com/rb/search/results.jsp?autoN=1&oNtt=ebusiness+vendor+selection&oN
tx=mode+MatchAllPartial&N=0+133001+51006&oNtk=MainSearch).
7
Forrester has many research reports that may prove very useful in helping you identify your appropriate
shortlist of vendors. See the January 14, 2010, “Interim Update To Forrester Wave™ Evaluation Of B2C
eCommerce Platforms” report, see the December 7, 2009, “Market Overview: B2B eCommerce Platforms”
report, and see the July 14, 2009, “Open Source eCommerce Solutions: Truths And Myths” report.
Additionally, Forrester clients who have access to inquiry can access Forrester analysts who cover specific
technologies to review their high-level requirements, goals, and how they want to own and operate a
solution in order to get the much-needed input for a shortlist of appropriate vendors. This is particularly
useful when a Forrester client’s requirements and needs are relatively straightforward.
Lastly, many technology and industry verticals have a group of seasoned and knowledgeable consultants
to aid clients. Forrester also provides these services as a neutral third party through advisory offerings.
This is particularly useful for clients with more complex requirements and needs. Contact Forrester for
more information on how Forrester analysts can help in this way. Source: Forrester Research (http://www.
forrester.com/My/Contacts).
8
A tip for vendors in a STEP approach: Have one resource acting the part in the scenario and using the
tools (or if multiple roles are defined, have more than one show the tools), and then have another resource
providing the commentary to explain the work in the context of the tools, the philosophy behind the
application, and what is out-of-the-box, configured, and custom work.
9
Firms know that customer experience is important — but they deal with it haphazardly. As a result,
customers suffer through needlessly painful interactions. That’s why firms need a more disciplined approach
to customer experience. Forrester recommends that companies adopt Scenario Design, a concept built on a
simple assumption: No experience is inherently good or bad, it can only be judged by looking at how well it
helps customers achieve their goals. This approach requires companies to continually ask — and answer —
three questions: Who are your users, what are their goals, and how can you help them achieve those goals?
See the July 19, 2004, “Scenario Design: A Disciplined Approach To Customer Experience” report.

Making Leaders Successful Every Day
Headquarters Research and Sales Offices

Forrester Research, Inc. Forrester has research centers and sales offices in more than 27 cities
400 Technology Square internationally, including Amsterdam; Cambridge, Mass.; Dallas; Dubai;
Cambridge, MA 02139 USA Foster City, Calif.; Frankfurt; London; Madrid; Sydney; Tel Aviv; and Toronto.
Tel: +1 617.613.6000
Fax: +1 617.613.5000 For a complete list of worldwide locations
visit www.forrester.com/about.
Email: forrester@forrester.com
Nasdaq symbol: FORR
www.forrester.com
For information on hard-copy or electronic reprints, please contact Client Support

at +1 866.367.7378, +1 617.613.5730, or clientsupport@forrester.com.
We offer quantity discounts and special pricing for academic and nonprofit institutions.
Forrester Research, Inc. (Nasdaq: FORR)

is an independent research company
that provides pragmatic and forward-
thinking advice to global leaders in
business and technology. Forrester
works with professionals in 20 key roles
at major companies providing
proprietary research, customer insight,
consulting, events, and peer-to-peer
executive programs. For more than 26
years, Forrester has been making IT,
marketing, and technology industry
leaders successful every day. For more
information, visit www.forrester.com.
56754

Scenario-Based Ecommerce Technology Evaluation Process

Caricato da

Informazioni sul documento

Descrizione originale:

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Scenario-Based Ecommerce Technology Evaluation Process

Caricato da

Copyright:

Formati disponibili

May 17, 2010

Making Leaders Successful Every Day

May 17, 2010

tabl e o f Co nte nts NOT E S & R E S OUR CE S

The Problems with the eCommerce Selection Process Today

May 17, 2010 © 2010, Forrester Research, Inc. Reproduction Prohibited

Figure 1 Typical eCommerce Technology Selection Process

56754 Source: Forrester Research, Inc.

Figure 2 Demos And References Matter Most To Selection Quality

© 2010, Forrester Research, Inc. Reproduction Prohibited May 17, 2010

Time to reinvent the process — scenario-based Technology evaluation

May 17, 2010 © 2010, Forrester Research, Inc. Reproduction Prohibited

Placing STEP In Context

The RFP Is Dead; Long Live The RFI

· Collect background information on the prospective vendors. This information will be

© 2010, Forrester Research, Inc. Reproduction Prohibited May 17, 2010

STEP: Running A Scenario-Based Technology Evaluation

Figure 3 Key Steps In A Scenario-Based Technology Evaluation

• Assemble selection team, and establish a team lead.

• Conduct scenario-based labs.

• Determine top two vendors based on scoring.

56754 Source: Forrester Research, Inc.

May 17, 2010 © 2010, Forrester Research, Inc. Reproduction Prohibited

1. Prepare. Lay the groundwork for STEP.

Define high-level business, customer experience, and technical requirements. A business

Figure 4 Forming A STEP Selection Team

Handling contract negotiation and compliance with corporate policies such as

56754 Source: Forrester Research, Inc.

© 2010, Forrester Research, Inc. Reproduction Prohibited May 17, 2010

2. Evaluate. Set up scenarios for use in evaluations.

May 17, 2010 © 2010, Forrester Research, Inc. Reproduction Prohibited

Determine scenario scoring criteria and weighting. Develop a straightforward scorecard

© 2010, Forrester Research, Inc. Reproduction Prohibited May 17, 2010

Figure 5 Sample STEP Scenarios For eCommerce Platform Selection

Scenario No. 1: Add new products.

Scenario No. 3: Set up promotions with A/B and targeting.

56754 Source: Forrester Research, Inc.

May 17, 2010 © 2010, Forrester Research, Inc. Reproduction Prohibited

Figure 6 Example Of Scenario-Based Technology Evaluation Scorecard Framework

Score Weighted Detailed

Usability of tools Intuitive ease of use, %

Functional fit to How well system meets

© 2010, Forrester Research, Inc. Reproduction Prohibited May 17, 2010

Figure 7 Tips For Running Scenario-Based Technology Evaluation Labs

56754 Source: Forrester Research, Inc.

3. Decide. Conduct STEP.

Score vendors based on their capabilities demonstrated through scenarios. Immediately

May 17, 2010 © 2010, Forrester Research, Inc. Reproduction Prohibited

© 2010, Forrester Research, Inc. Reproduction Prohibited May 17, 2010

How SCenario-Based technology Evaluation Can Fit Into A Larger Project

Forrester’s resources can help you succeed

May 17, 2010 © 2010, Forrester Research, Inc. Reproduction Prohibited

© 2010, Forrester Research, Inc. Reproduction Prohibited May 17, 2010

May 17, 2010 © 2010, Forrester Research, Inc. Reproduction Prohibited

Headquarters Research and Sales Offices

For information on hard-copy or electronic reprints, please contact Client Support

Forrester Research, Inc. (Nasdaq: FORR)

Potrebbero piacerti anche