Internet Research

A framework and methodology for evaluating e-commerce Web sites

Rian van der MerweJames Bekker

Article information:
To cite this document:
Rian van der MerweJames Bekker, (2003),"A framework and methodology for evaluating e-commerce Web sites", Internet
Research, Vol. 13 Iss 5 pp. 330 - 341
Permanent link to this document:
http://dx.doi.org/10.1108/10662240310501612

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

Downloaded on: 16 June 2015, At: 01:01 (PT)

References: this document contains references to 21 other documents.
To copy this document: permissions@emeraldinsight.com
The fulltext of this document has been downloaded 4601 times since 2006*

Users who downloaded this article also downloaded:

Chung-Shing Lee, (2001),"An analytical framework for evaluating e-commerce business models and strategies", Internet
Research, Vol. 11 Iss 4 pp. 349-359 http://dx.doi.org/10.1108/10662240110402803
Mei Cao, Qingyu Zhang, John Seydel, (2005),"B2C e-commerce web site quality: an empirical examination", Industrial
Management & Data Systems, Vol. 105 Iss 5 pp. 645-661 http://dx.doi.org/10.1108/02635570510600000
Eric Allen, Jerry Fjermestad, (2001),"E-commerce marketing strategies: an integrated framework and case analysis", Logistics
Information Management, Vol. 14 Iss 1/2 pp. 14-23 http://dx.doi.org/10.1108/09576050110360106

Access to this document was granted through an Emerald subscription provided by emerald-srm:499410 []

For Authors
If you would like to write for this, or any other Emerald publication, then please use our Emerald for Authors service
information about how to choose which publication to write for and submission guidelines are available for all. Please visit
www.emeraldinsight.com/authors for more information.

About Emerald www.emeraldinsight.com

Emerald is a global publisher linking research and practice to the benefit of society. The company manages a portfolio of
more than 290 journals and over 2,350 books and book series volumes, as well as providing an extensive range of online
products and additional customer resources and services.
Emerald is both COUNTER 4 and TRANSFER compliant. The organization is a partner of the Committee on Publication Ethics
(COPE) and also works with Portico and the LOCKSS initiative for digital archive preservation.
*Related content and download information correct at time of download.

Introduction

A framework and
methodology for
evaluating e-commerce
Web sites
Rian van der Merwe and
James Bekker

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

The authors
Rian van der Merwe is Research Assistant and Student,
School of Marketing, Curtin University of Technology, Perth,
Australia.
James Bekker is a Lecturer in Information Systems and
Computer Simulation, Department of Industrial Engineering,
University of Stellenbosch, South Africa.
Keywords
Electronic commerce, Internet, Function evaluation
Abstract
The corporate Web site is an indispensable part of any ecommerce venture. If users find it difficult to do electronic
transactions, they will visit another online source, or go to a
physical source instead. Evaluating the performance of ecommerce Web sites has received some attention in the past
few years, but there is still a pressing need for a more
comprehensive approach. This paper attempts to address
this need by providing an e-commerce Web site evaluation
framework and method built on solid business principles,
and using multidimensional scaling to analyse evaluation
data and present the results in a graphic manner for easy
interpretation.
Electronic access
The Emerald Research Register for this journal is available at
http://www.emeraldinsight.com/researchregister
The current issue and full text archive of this journal is
available at
http://www.emeraldinsight.com/1066-2243.htm
Internet Research: Electronic Networking Applications and Policy
Volume 13 . Number 5 . 2003 . pp. 330-341
# MCB UP Limited . ISSN 1066-2243
DOI 10.1108/10662240310501612

The Internet and e-commerce have become

extremely important avenues for firms in many
industries to interact with their customers and
other stakeholders. E-commerce, with the Web
as primary infrastructure, offers several tangible
and intangible benefits (Kim et al., 2003). For
any firm with an online presence, the Web site
is the platform used to communicate with
customers and to facilitate business
transactions. Web sites that do not provide
positive experiences may cause customers to
decide that it is easier to go to a physical store
rather than purchase online. Forrester Research
estimates that ``poor Web design will result in a
loss of 50 per cent of potential sales due to users
being unable to find what they want, and a loss
of 40 per cent of potential repeat visits due to
initial negative experience'' (Cunliffe, 2000).
The question is, however, how should
companies go about conceptualising and
continually improving their Web sites.
Managers need information on Web site
performance in order to improve effectiveness
and usability. Currently, no generally accepted
method exists to evaluate the performance of
Web sites systematically and comprehensively.
There is a need for a complete and accurate
method to evaluate the performance of Web
sites, not only individually but also in
comparison with the Web sites of other
companies in the same industry. This article
attempts to address this need by focusing on
two objectives:
(1) To develop a framework and criteria for the
comprehensive evaluation of e-commerce
Web sites.
(2) To use this framework and sound statistical
reasoning to develop a method that can be
used to evaluate Web sites quantitatively.
This paper starts with a brief review of the
literature on Web site evaluation, followed by a
discussion of a framework and method for
evaluation. The paper concludes with the
identification of further managerial applications
and research opportunities, and also points out
some of the limitations of the methodology.

330

A framework and methodology for evaluating e-commerce Web sites

Internet Research: Electronic Networking Applications and Policy

Volume 13 . Number 5 . 2003 . 330-341

Rian van der Merwe and James Bekker

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

Web site evaluation: a review of the

literature
Boyd Collins developed the first formal
approach to the evaluation of Web sites in late
1995. He founded the Infofilter project, a
model intended for librarians who needed to
evaluate the quality of information on the
Internet. The model was based on six criteria,
developed by combining evaluation criteria for
printed media, and considering what was
relevant for Web sites. These criteria are
content, authority, organisation, searchability,
graphic design and innovative use. Although
innovative at the time, it is clear that the
Internet and its uses have changed to such an
extent that these criteria are no longer sufficient
to evaluate Web sites. The Infofilter project
ceased operation in July 1997 because its
participants realised that it was becoming
obsolete. It seems, however, that many authors
have not realised that these criteria have to a
large extent lost their relevance. Using common
Internet search engines to find literature on
Web site evaluation reveals hundreds of Web
sites claiming to provide Web site evaluation
criteria (see Alexander and Tate, 1996). When
studied closely it becomes clear that most of
these sites use the Collins criteria as a platform
to which they arbitrarily seem to add their own.
There is little evidence of any research on which
these are based, which makes them unreliable
to a large extent. The biggest problem with
them is that they are based on traditional
marketing strategies for printed media.
The past few years have also witnessed an
increase in the number of research articles on
Web site evaluation published in academic
journals. The advice given in these publications
ranges from extremely rigid (``Use no more than
three images per page'') (D'Angelo and Little,
1998) to very vague (``Design the Web site for
content, not appearance'') (D'Angelo and
Little, 1998). Generally, however, these
publications are much more reliable than the
multitude of Web sites claiming to provide
criteria for Web site evaluation. They even
move a step beyond mere front-end evaluation.
Berthon et al. (1996) developed a model for the
evaluation of Web site effectiveness based on
hierarchy of effects models, which are well
known in buying behaviour. Other approaches

include the static-quality/dynamic-quality

approach by Day (1997); user-defined criteria
developed by Misic and Johnson (1999); the
Web site quality evaluation method (QEM) by
Olsina et al. (1999) and automatic classification
and evaluation by Bauer and Scharl (2000).
Literature on Web site evaluation that focuses
specifically on e-commerce sites is scarce. One
approach is the Web assessment model (WA
model), which was developed in 1997 at the
Competence Centre for Electronic Markets
(CCEM) at the University of St Gallen in
Switzerland. Schubert and Selz (1999) describe
the development of this model in detail. Liu et
al. (2000) made effective progress in the field by
developing criteria for the development of
e-commerce Web sites using the inputs of
Fortune 1,000 companies' Web masters.
While the approaches to Web site evaluation
discussed above have made some contributions,
they only focus on a specific aspect of a Web
site, and none provides a method for the
comprehensive evaluation of e-commerce Web
sites. Ideally, these approaches need to be
combined and extended to develop a
framework that evaluates every aspect of an
e-commerce Web site, as proposed by Kim et al.
(2003).

An e-commerce Web site evaluation

framework
For reasons that become clear in the next few
paragraphs, the proposed framework for the
evaluation of an e-commerce Web site
addresses three different levels of detail. The
first level consists of five criteria categories,
which serve as a structure for evaluation,
covering all the relevant aspects of e-commerce
Web sites. The second level involves four
criteria groups within each of the criteria
categories, and provides more detail and
definition to the categories. On the third and
most detailed level, five individual criteria are
developed within each of the criteria groups,
and are used to perform the evaluation. The
levels are as follows.
Level 1
The purpose of the framework and its criteria
categories is to find a way to comprehensively

331

A framework and methodology for evaluating e-commerce Web sites

Internet Research: Electronic Networking Applications and Policy

Volume 13 . Number 5 . 2003 . 330-341

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

Rian van der Merwe and James Bekker

evaluate e-commerce Web sites. To fulfil this

purpose, the customer buying cycle is used as a
framework for e-commerce Web site evaluation
(see Robinson et al., 1967; Schoel and
Guiltinan, 1992, for two approaches to the
customer buying cycle). This was done for two
reasons:
(1) Comprehensive evaluation of all aspects of
e-commerce Web sites. By following every
action a customer will perform on an
e-commerce Web site when a purchase is
made, the buying cycle will ensure that all
the relevant parts of the Web site are
evaluated.
(2) Total customer focus. The customer buying
cycle puts the focus on the processes that
customers follow when they pay for
products/services. They are the users of the
Web sites, so any evaluation framework
must focus specifically on their experience
of the sites.
As mentioned above, there are several views in
the marketing literature on customer buying
processes (see Berthon et al., 1996; Churchill et
al., 1993). A basic thread runs through these
approaches, and this paper regards the
customer buying cycle as consisting of four
distinct phases. These phases are shown below
with an appropriate example to explain each of
them:
(1) Need recognition. A potential customer
realises a specific need, for example, he/she
would like to read a book on a particular
subject, say aviation in the Second World
War.
(2) Gather information. The customer goes to a
bookstore and finds all the resources on
aviation during the Second World War.
(3) Evaluate information. The customer reads
the summaries of the books on the subject,
and decides which of the resources he/she is
going to buy.
(4) Make purchase. The customer goes to the
checkout counter and buys the book he/she
has selected.

Figure 1 shows how these links were made and

an explanation follows below.
During the need-recognition phase, potential
customers will visit a Web site that they believe
will be able to fulfil their specific needs. The
interface is very important at this stage, because
this determines the user's first impression of the
site. The interface refers to the visual aspects of
the site how attractive it is visually and what
backgrounds, fonts and colours are used, etc.
Although Berthon et al. (1996) state that, on
the Internet, the buying cycle actually starts
before the need-recognition phase, when the
customer tries to find the site, the approach
followed in this paper focuses specifically on the
Web site itself.
During the information-gathering phase,
customers explore the site and search for the
products/services in which they are interested.
During this phase, navigation is the most
important aspect of the site. This refers to the
process of moving around between the different
pages of the Web site. Aspects such as the
straightforward, logical structure of the site, and
menus that are easy to understand are
important in this phase.
During the third phase of the buying cycle
(evaluate information), customers make choices
about which products/services they want to
purchase. The content of the site is, therefore,
very important. Content refers to the actual
information on the site, for example, the
amount and quality of information on the site.
In the final phase of the buying cycle (make
purchase), customers purchase the products/
services they have chosen. The reliability of the
site is extremely important at this stage and is
defined as consisting of two aspects:
(1) The degree to which a customer is able to
use the order process on the site easily and
effectively.
(2) The degree to which the company is able to
fulfil its promises and obligations to
customers every time a purchase is made.

To use this buying cycle as a framework for the

evaluation of e-commerce Web sites, it is
necessary to find links between each of the
buying phases and a specific aspect of a Web
site that is most important during that phase.

Finally, several technical aspects of a Web site

are essential to its proper functioning. As
shows, these aspects are important during all
the phases of the buying cycle, and are,
therefore, the backbone of the site.

332

A framework and methodology for evaluating e-commerce Web sites

Internet Research: Electronic Networking Applications and Policy

Volume 13 . Number 5 . 2003 . 330-341

Rian van der Merwe and James Bekker

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

Figure 1 Development of e-commerce Web site evaluation framework

Level 2
With the first level of the evaluation framework
completed, the second level to be developed is
the criteria groups within each category. These
groups were derived from the literature on Web
site evaluation using popular criteria listed by
several authors (e.g. Berthon et al. 1996; Abels
et al. 1999; Cunliffe, 2000; Bauer and Scharl,
2000; Kim et al., 2003). These criteria were
then classified according to the researchers'
judgement under the most appropriate category
defined on level 1. Four groups per criteria
category were identified, as shown in Figure 2.
Each of these groups is explained in Table I.

Level 3
The third level of the evaluation framework
consists of specific criteria within each criteria
group. Although it cannot be completely
eliminated, the purpose of the criteria on this
level is to minimize subjectivity during the
evaluation process. These criteria are proposed
by the researchers and are debatable. Five
criteria were developed within each of the 20
criteria groups described above, resulting in a
total of 100 criteria. The criteria are shown in
Table II-VI. Note that language is not part of
the criteria, as it is assumed that e-commerce is

Figure 2 E-commerce Web site evaluation framework and criteria groups

333

A framework and methodology for evaluating e-commerce Web sites

Internet Research: Electronic Networking Applications and Policy

Volume 13 . Number 5 . 2003 . 330-341

Rian van der Merwe and James Bekker

Table I Description of e-commerce Web site evaluation criteria groups

Phase

Criteria group

This criteria group evaluates/measures

Interface

Graphic design principles

The effective use of colour, text, backgrounds, and other general graphic
design principles
The effectiveness of the graphics and multimedia used on the site
Whether or not the text is concise and relevant, and the style good
The degree to which the interface is designed to handle exceptions, for
example, text-only versions of pages

Graphics and multimedia

Style and text
Flexibility and compatibility
Navigation

Logical structure
Ease of use
Search engine

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

Navigational necessities

The organisation and menu system of the site

The ease of navigation to find the pages that the user is looking for
The search engine's ability to find the correct pages easily and provide
clear descriptions of the search results
Other important aspects of navigation like the absence of broken links
and ``under-construction'' pages

Content

Product/service-related
information
Company and contact
information
Information quality
Interactivity

Whether or not the products/services are described precisely and

thoroughly
Whether or not it is easy to find information on the company, its
employees and its principals
The currency and relevance of the content on the site
How much input the user has on the content displayed on the site

Reliability

Stored customer profile

The registering process and how the company uses the stored customer
profile
The effectiveness and ease of use of the online order process
The company's actions from order placement until the order is delivered
How the company communicates and helps its online customers

Order process
After-order to order receipt
Customer service
Technical

Speed
Security
Software and database
System design

Different aspects of the loading speed of the site

Security systems and the ways used by the company to protect
customers' privacy on the site
Flexibility in terms of different software used. Also looks at the data
software and data communication systems used on the site
The correct functioning of the site and how well it integrates with
internal and external systems

done in English, even within non-English

countries.

Evaluation concept
Evaluation data for a given Web site are
gathered by going through the list of evaluation
criteria and awarding a value to each item. This
value will be an indication of how well the Web
site adheres to the specific criterion. All the
individual values for the criteria in a specific
group are then summed to a unique score for
that group. The group scores within each
category are then summed, which generates a
scale of criteria category scores indicating how

well a Web site performs in each of the

categories.
As soon as more than one site is evaluated
using this procedure, the data can be presented
in tabular form. The result is a matrix of values
achieved, with the column variable being
criteria categories and the row vector being
different Web sites within the same industry. A
given matrix therefore contains the results for
different Web sites in one industry. The values
in the cells of the matrix represent the score of
each site for a specific criteria category, as
calculated from the evaluation criteria. An
example is shown in Table VII.
One of the problems associated with this type
of table is the lack of ease and efficiency of
interpretation of the contained information.

334

A framework and methodology for evaluating e-commerce Web sites

Internet Research: Electronic Networking Applications and Policy

Volume 13 . Number 5 . 2003 . 330-341

Rian van der Merwe and James Bekker

Table II E-commerce Web site evaluation criteria interface

Table IV E-commerce Web site evaluation criteria content

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

1 Interface

3 Content

Group 1:
Graphic design
principles

Home page concise and clear

Effective use of white space
Effective and consistent use of colour
Effective and consistent use of backgrounds
Effective graphics/typeface/colour combinations

Group 1:
Product/servicerelated content

Extensive product/service information available

Price information and cost benefits communicated
Adequate breadth of product range
Adequate amount of advertising of own products
Adequate amount of advertising by other companies

Group 2:
Graphics and
multimedia

Site visually attractive

Graphics and multimedia make a contribution to the
understanding and navigation of site
Icons easy to understand
Not excessively used
Size of media: no negative impact on loading times

Full company information available

Group 2:
Terms and conditions easily accessed
Company and
contact information E-mail addresses of employees available
Telephone and fax numbers available
Postal and physical addresses available

Group 3:
Style and text

Style of pages consistent

Typefaces consistent and easy to read
Correct spelling and grammar
Text concise and relevant
Purpose of site made clear on home page

Group 4:
Flexibility and
compatibility

Pages sized to fit in browser window

Printable versions of certain pages available
Text-only version available
Foreign language support available
Accommodation made for disabled users

Group 3:
Content current and updated
Information quality Content relevant to purpose of site
Content concise and non-repetitive
High perceived quality of product/service
Quality of advertisements high
Group 4:
Interactivity

User able to customise content to fit needs

Large amount of personalisation possible
Easy access to online community
High perceived value of online community
High degree of interactivity in community

Table III E-commerce Web site evaluation criteria navigation

2 Navigation

Table V E-commerce Web site evaluation criteria reliability

4 Reliability

Group 1:
Logical structure

Intelligible, straightforward organising scheme

Content logically structured in different sections and
levels
Menus understandable and straightforward
Site map/table of contents available
Consistent navigation throughout site

Group 2:
Ease of use

Easy
Easy
Easy
Easy
Easy

Group 3:
Search engine and
help function

Easy to use search engine

Search engine accurate
Good description of search engine findings
No search engine errors
Help function easy to use

Online order tracking available

Group 3:
After-order to order Effective payment settlement
Confirmation of order dispatch sent to customer
receipt
On-time delivery
Full order delivery

Group 4:
Navigational
necessities

No broken links
No ``under-construction'' pages
Links clearly discernible, well labelled and defined
Clear label of current position on site
Effective use of frames, non-frames version available

Group 4:
Customer service

to
to
to
to
to

find site
explore specific idea or subject
return to main page
find specific information
access complete product/service range

Group 1:
Stored customer
profile

Easy to register on site

High perceived benefits from registering
Easy to log on to site
Adjustable customer profile stored
Guided ordering using customer profile available

Group 2:
Order process

Transparent, interactive and easy order process

Easy selection of generic services
Alternative methods of ordering/payment available
Good stock availability
Acknowledgement of order sent to customer

335

Feedback forms available

Good after-sales support
Quick reply to e-mail enquiries
General courtesy of company good
Good after-sales relationship building

A framework and methodology for evaluating e-commerce Web sites

Internet Research: Electronic Networking Applications and Policy

Volume 13 . Number 5 . 2003 . 330-341

Rian van der Merwe and James Bekker

Table VI E-commerce Web site evaluation criteria technical

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

4 Technical
Group 1:
Speed

Fast home page loading speed

Fast sub-page loading speed
Good perceived use of cacheing
Good consideration of non-broadband users
Good perceived speed of database

Group 2:
Security

Security systems accredited

Secure payment systems used
Privacy of users protected
Security protocols well communicated
Security certificates adequate

Group 3:
Software and
database

Good cross-browser capability

Users advised on ideal browser and resolution
Database software adequate for size of database
Good data transfer between systems
No perceived duplication of data in database

Group 4:
System design

Precise operation and computation

Good resolution compatibility
Good integration with systems of users
Good integration of different systems on site
Global accessibility to products/service available

The next section introduces correspondence

analysis as the technique used to analyse the
evaluation data and represent it graphically,
which enables easy and efficient interpretation.

Correspondence analysis
In order to quantify, interpret and assess the
evaluation data, it is imperative to find a
suitable technique to analyse data that are
presented in this form. The purpose of
multidimensional scaling techniques is to
condense large amounts of data to as few
dimensions as possible, while still describing it
adequately. Formally, multidimensional scaling
can be defined as ``[a set of mathematical
techniques] concerned primarily with the

spatial representation of relationships among

behavioural data'' (Green et al., 1989). To
paraphrase this definition, multidimensional
scaling techniques seek to find an
understandable visual representation of the
similarities, dissimilarities or ordered rankings
between different data points. All that is needed
to perform multidimensional scaling is a matrix
of numbers that expresses all the similarities,
dissimilarities or rankings within a group of
objects like the matrix discussed at the end of
the previous section.
Correspondence analysis is part of
multidimensional scaling and will be used here.
It originated with a French analyst called
Jean-Paul Benzecri in the early 1960s (see
Greenacre, 1984), and can be defined as ``a
technique for displaying the rows and columns
of a data matrix (primarily a two-way
contingency table) as points in dual lowdimensional vector spaces'' (Greenacre, 1984).
The strength of this technique is that it provides
a way to examine relations not only between
row variables and column variables individually
but also between row and column variables
together. This implies that, if used in the
context under discussion, it would be possible
to compare the similarities that different Web
sites have with each other, or the similarities
that different criteria groups have with each
other. Furthermore, it would also be possible to
compare individual Web sites with the different
criteria groups. The mathematical concepts of
correspondence analysis are discussed
extensively in Greenacre (1984) and Weller and
Romney (1990).
Correspondence analysis enables one to
examine data and visually show the complex
relationships between different Web sites,
between different criteria, and between sites
and criteria, on a low-dimensional scale (usually
two dimensions).

Table VII Example of a two-way contingency matrix for Web site evaluation
Criteria category Criteria category Criteria category Criteria category Criteria category
1
2
3
4
5
Web
Web
Web
Web

site
site
site
site

A
B
C
D

67
87
78
37

78
74
64
57

63
48
49
85

336

78
93
56
73

23
67
83
65

A framework and methodology for evaluating e-commerce Web sites

Internet Research: Electronic Networking Applications and Policy

Volume 13 . Number 5 . 2003 . 330-341

Rian van der Merwe and James Bekker

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

E-commerce Web site evaluation method

A Web site evaluation method is now proposed.
The term method, as used in this context, is
defined as all the procedures followed to
evaluate Web sites from the gathering of
evaluation data to the interpretation of the
output results. Since evaluation must be done
on the same basis, the method requires that the
e-commerce Web sites be evaluated by
industry. Each industry must be clearly defined,
while Web sites must be classified and assigned
to a specific industry. To ensure objectivity, it is
suggested that at least three evaluators do the
evaluation. The three-step evaluation method
follows, and it is supported by an example (Van
der Merwe, 2001).
Step 1: Gather e-commerce Web site
evaluation data and present the data in a
two-way contingency matrix
Each of the evaluators individually assesses the
Web site in question against the 100 evaluation
criteria. The scoring method requires that the
evaluator scores the Web site according to the
level to which he agrees with the statement
made by each criterion. For each criterion a
score between 2 (strongly disagree) and +2
(strongly agree) is awarded. Only four scoring
options are provided (2, 1, 1 or 2) so the
evaluator is forced to decide whether he agrees
or not a neutral answer (0) is not possible. A
fifth option, not applicable, is provided for cases
in which the criterion is not at all relevant to the
sites in the industry under evaluation. In this
case, a score of 0 is awarded to all the sites in
that industry, which ensures that it has no
positive or negative effect on the outcome of the
evaluation.
After the evaluations are completed the group
should go through the criteria together, discuss
their opinions, and award an agreed final score
to each criterion. Any subjectivity that exists
within the criterion is thus reduced to a large
extent. This process can be facilitated through
online group collaboration (see Bamber et al.,
1995). The sum of the final scores of the five
criteria in each criteria group gives the 20
criteria group totals. The four group totals
within each of the five criteria categories in the
evaluation framework are then summed and

these five totals provide a scale of evaluation of

data for the site.
This process is repeated for all the Web sites
that are evaluated in a specific industry. When
all the sites have been evaluated, a contingency
matrix is created which is used for
correspondence analysis. As an example, this
method is applied to six Web sites in the online
books industry. After an experimental
evaluation, the matrix shown in Table VIII was
obtained.
Step 2: Perform correspondence analysis
on the matrix and represent the results
graphically
In this step, computer software is used to
perform correspondence analysis on the
contingency matrix and to plot the results. An
example of the plot that resulted from
correspondence analysis of the data in Table
VIII is shown in Figure 3.
Step 3: Interpret the results in the
graphical representation and make
recommendations
The interpretation process should be done in
three distinct phases:
(1) Use the correspondence analysis graph to
draw basic conclusions about the
performance of each of the Web sites and
the profile groups that can be identified.
This is done by assessing the proximities of
points to each other. Three aspects of the
graph are considered: the criteria category
profiles on their own, the Web site
categories on their own, and the criteria
category profiles and Web site profiles
together.
(2) Use these preliminary findings and
combine them with the original data in the
contingency matrix to refine the
conclusions.
(3) For each Web site, use its individual criteria
group scores to make recommendations on
possible ways to improve the site, and how
to do so.
The following conclusions can be drawn from
the correspondence analysis results shown in
Table VIII and Figure 3.

337

A framework and methodology for evaluating e-commerce Web sites

Internet Research: Electronic Networking Applications and Policy

Volume 13 . Number 5 . 2003 . 330-341

Rian van der Merwe and James Bekker

Table VIII Contingency matrix for the books industry

Site name

Site address

Amazon.com Books
Barnes & Noble
Exclusive Books
Kalahari.net
PC Books, Inc.
VS Online

www.amazon.com/books
www.bn.com
www.exclusivebooks.com
www.kalahari.net
www.pcbooks.co.za
www.vsonline.co.za

Interface Navigation
17
15
14
15
19
6

35
29
35
11
4
14

Content
30
25
18
13
1
9

Reliability Technical
31
27
27
24
16
6

28
19
23
12
22
11

Total
141
115
117
45
16
28

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

Figure 3 Correspondence analysis graph for the book industry

Analysis of criteria category profiles

Based on the lack of proximity between criteria

The other three sites have quite different

profiles from this group and from each other.

category profiles, it is evident that the criteria

Analysis of criteria category and Web site profiles

together
First, it is evident that Amazon.com Books,
Barnes and Noble and Exclusive Books all lie
more or less in the centre of the graph, with the
five criteria forming a rough circle around them.
This implies that these three sites generally
received the highest scores for the criteria.
From their close proximity to the Navigation
criteria category, it can be deduced that these
three sites not only received higher scores for
the Navigation category than for the other
categories, but also that their scores for

categories form no distinctly similar groups in

this case. This implies that their profiles differ.
In other words, two or more specific criteria do
not feature considerably more on some sites
than on others.
Analysis of Web site profiles
Based on proximities, it is evident that the sites
of Amazon.com Books, Barnes and Noble and
Exclusive Books form a group of similar
profiles. It appears therefore that these sites
received similar scores for some of the criteria.

338

A framework and methodology for evaluating e-commerce Web sites

Internet Research: Electronic Networking Applications and Policy

Volume 13 . Number 5 . 2003 . 330-341

Rian van der Merwe and James Bekker

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

Navigation are higher than the scores that the

other three sites received for this category.
Kalahari.net was positioned almost halfway
between the content and reliability criteria
categories. Although its total score was not very
high, it seems that, compared to the other
categories, this is where Kalahari.net received
the highest scores.
PC Books and VS Online lie on the periphery
of the graph and not in close proximity to any of
the categories. It is, therefore, clear that they
have not received particularly high scores for
any of the criteria.
Summary
Based on the above discussion, the following
conclusions can be made:
.
Clearly, Amazon.com, Barnes and Noble
and Exclusive Books are the industry
leaders in terms of Web site effectiveness.
Particularly in terms of navigation, the
other sites can learn a lot from these three.
.
Kalahari.net needs major improvements,
but its overall performance is not too bad.
They should start by improving the
interface, technical and navigation aspects
of the site.
.
VS Online and PC Books are in a bad
condition and drastically need to improve
their sites in all areas.
.
When the correspondence analysis graph
and the contingency matrix are analysed
together, it is clear that all sites need to
improve their interfaces. Kalahari.net and
PC Books in particular have to make drastic
improvements in this area.
The graphical analysis can then be extended by
looking at the individual site results for each site
and deciding on specific criteria groups and
individual criteria that have to be addressed.
Managers can use the information from the
evaluation results to identify :
.
areas where their Web sites need
improvement;
.
other Web sites that perform well in the
aspects that need improvement (for
benchmarking purposes); and
.
the areas in which their Web sites perform
well.

This completes the proposed method for

comprehensive evaluation of e-commerce Web
sites. The method is summarised in Figure 4.

Conclusions, recommendations,
managerial and research implications
At present, firms engaged in e-commerce are
limited in their ability to gauge their online
performance based on objective criteria. Their
ability to compare their performance with other
companies in the same industry is also
restricted. The technique presented in this
paper is structured and repeatable, and gives
managers the opportunity to evaluate the as-is
situation of their Web sites (assuming that they
wish to do so) with respect to those of other
companies. They can use the approach to
identify where they perform well, and where
there is room for improvement. Managers can
determine which sites within their industry
perform better than they do in particular areas,
and why. This objective information can be
used to make strategic decisions on how to
improve the efficiency and performance of their
Web sites.
Practising and academic marketing
researchers can also apply the technique.
Practitioners can identify who the leaders are in
a specific industry, as well as why they do so
well. Academics can conduct cross-industry
research, which may reveal aspects of Web sites
within one industry that may have been
neglected in others. They will be able to
quantify the performance of Web sites and use
this as a dependent or independent variable in
research directed at determining the
consequences of Web site performance.
There are, however, a few reservations about
the individual criteria:
.
Although the objectivity of criteria was a
major driving force behind the
development process, the criteria still entail
a certain measure of human subjectivity, as
mentioned above. Although the evaluation
framework and criteria groups seem to be
reasonably inclusive, it may be necessary to
modify or even replace some of the criteria
to ensure greater objectivity in the process.
The criteria of level 3 requires further
evaluation, and should perhaps be

339

A framework and methodology for evaluating e-commerce Web sites

Internet Research: Electronic Networking Applications and Policy

Volume 13 . Number 5 . 2003 . 330-341

Rian van der Merwe and James Bekker

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

Figure 4 Summary of the e-commerce Web site evaluation method

identified through consensus of experts in

the field.
Some of the criteria cannot be evaluated
without intricate knowledge of the ordering
process of the companies being evaluated.
These criteria are essential to the
evaluation; therefore, e-commerce
companies that are evaluated must be
prepared to explain their process.

With regard to correspondence analysis, the

following must be noted:
.
Interpretation of correspondence analysis
results tends to become less accurate as the
number of sites to be evaluated decreases.
It is, therefore, recommended that the
graph is used to make initial conclusions,
and that the contingency matrix is then
used to validate and elaborate on these
conclusions.
.
One drawback of the technique is that the
results deal with relative positions.
Suppose, for example, that the point of a
specific Web site lies very close to a certain
criteria category on the plot. This does
mean that the site scored well in that
category, compared to its other scores, but
it does not necessarily mean that it received
the highest score for that category of all the
sites that were evaluated. Other sites with
high overall scores and a more even
distribution of scores may well have scored
higher in that category. Also, no
conclusions can be drawn from the
correspondence analysis graph regarding
the absolute scores of the Web sites. The
correspondence analysis graph has no way
of communicating this piece of information
to the user. The recommendation is that

the graphs be studied with the contingency

matrix close at hand, so that an accurate
conclusion can also be drawn about the
absolute performance of the Web sites in
the industry under consideration.
It is recommended that further study be
conducted in the wider field of statistical
mapping. In this paper, the focus was on
correspondence analysis as a specific technique
within this field. Other multidimensional
techniques such as metric scaling or cluster
analysis may be better suited to perform the
analysis of evaluation data. Furthermore, the
technique of intercoder reliability (see
Kassarjian, 1977) could be useful to assess the
reliability of the evaluators themselves. Further
research should evaluate the performance of the
proposed method against that of other existing
evaluation methods (see, for example, Kim et
al., 2003).
It is also important to note that it takes a
considerable amount of time to complete an
evaluation. The researchers found that it took
an average of two to three working days per
industry to evaluate the Web sites thoroughly.
The evaluators have to spend time getting to
know the site (if they have not used it
considerably before), and a dedicated effort is
needed to evaluate each of the 100 evaluation
criteria. It is, however, not a trivial undertaking
to evaluate an e-commerce Web site, and it is
therefore essential that enough time be spent to
perform an accurate and useful evaluation.
Although many factors have to be considered
for an Internet company to be successful, the
Web site is one of the essential aspects that have
to be effectively in place for e-commerce
business to succeed. The evaluation method

340

A framework and methodology for evaluating e-commerce Web sites

Internet Research: Electronic Networking Applications and Policy

Volume 13 . Number 5 . 2003 . 330-341

Rian van der Merwe and James Bekker

proposed in this paper aims to assist managers

to achieve this, should they wish to improve
their e-commerce enterprises.

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

References
Abels, E.G., White, M.D. and Hahn, K. (1999), ``A user-based
design process for Web sites'', OCLC Systems &
Services, Vol. 15 No. 1, pp. 35-44.
Alexander, J. and Tate, M. (1996), ``Evaluation criteria'',
available at: http://hoku.lcc.hawaii.edu/kaz/100/
evalCriteria.htm
Bamber, M., Hill, M.C. and Watson, R.T. (1995), ``Group
support systems can improve your meetings'', The CPA
Journal, Vol. 65 No. 3, pp. 32-7.
Bauer, C. and Scharl, A. (2000), ``Quantitative evaluation of
Web site content and structure'', Internet Research:
Electronic Networking Applications and Policy, Vol. 10
No. 1, pp. 31-43.
Berthon, P.R., Pitt, L.F. and Watson, R.T. (1996), ``The World
Wide Web as an advertising medium: toward
understanding of conversion efficiency'', Journal of
Advertising Research, Vol. 36 No. 1, pp. 43-53.
Churchill, G.A. Jr, Ford, N.M. and Walker, O.C. Jr (1993),
Sales Force Management, 4th ed., Richard D. Irwin,
Homewood, IL.
Cunliffe, D. (2000), ``Developing usable Web sites a review
and model'', Internet Research: Electronic Networking
Applications and Policy, Vol. 10 No. 4, pp. 295-307.
D'Angelo, J. and Little, S.K. (1998), ``Successful Web pages:
what are they and do they exist?'', Information
Technology and Libraries, Vol. 17 No. 2, pp. 71-81.
Day, A. (1997), ``A model for monitoring Web site
effectiveness'', Internet Research: Electronic
Networking Applications and Policy, Vol. 7 No. 2,
pp. 109-15.
Green, P.E., Carmone, F.J. and Smith, S.M. (1989),
Multidimensional Scaling: Concepts and Applications,
Allyn and Bacon, Needham Heights, MA.
Greenacre, M.J. (1984), Theory and Applications of
Correspondence Analysis, Academic Press, London.

Kassarjian, H.H. (1977), ``Content analysis and consumer

research'', Journal of Consumer Research, Vol. 4, June,
pp. 8-18.
Kim, S.-E., Shaw, T. and Schneider, H. (2003), ``Web site
design benchmarking within industry groups'', Internet
Research: Electronic Networking Applications and
Policy, Vol. 13 No. 1, pp. 17-26.
Liu, C., Arnett, K.P. and Litecky, C. (2000), ``Design quality of
Web sites for electronic commerce: Fortune 1000
Webmasters' evaluations'', Electronic Markets, Vol. 10
No. 2, pp. 120-9.
Misic, M.M. and Johnson, K.L. (1999), ``Benchmarking: a tool
for Web site evaluation and improvement'', Internet
Research: Electronic Network Applications and Policy,
Vol. 9 No. 5, pp. 383-92.
Olsina, L., Godoy, D., Lafuente, G.J. and Rossi, G. (1999),
``Specifying quality characteristics and attributes for
Web sites'', paper presented at the 1st ICSE Workshop
on Web Engineering (WebE-99), Los Angeles, CA,
available at: http://citeseer.nj.nec.com/265461.html
Robinson, P.J., Faris, C.W. and Wind, Y. (1967), Industrial
Buying and Creative Marketing, Allyn and Bacon,
Boston, MA.
Schoel, W.F. and Guiltinan, J.P. (1992), Marketing, 5th ed.,
Allyn and Bacon, Needham Heights, MA.
Schubert, P. and Selz, D. (1999), ``Web assessment
measuring the effectiveness of electronic commerce
sites going beyond traditional marketing paradigms'',
Proceedings of the 32nd Hawaii International
Conference on System Sciences, Maui, Hawaii.
Internet and the Digital Economy Track, 01/99, IEEE
Computer Society Press, CA, available at:
www.businessmedia.org/netacademy/publications.nsf/
all_pk/1142
Van der Merwe, R. (2001), ``E-commerce Web site
evaluation'', MSc Master's thesis, Department of
Industrial Engineering, University of Stellenbosch,
South Africa.
Weller, S.C. and Romney, A.K. (1990), Metric Scaling:
Correspondence Analysis, Sage University Paper Series
on Quantitative Applications in the Social Sciences,
Series nr. 07-075, Sage Publications, Newbury Park,
CA.

341

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

This article has been cited by:

1. Gajendra Sharma, Wang Lijuan. 2015. The effects of online service quality of e-commerce Websites on user satisfaction. The
Electronic Library 33:3, 468-485. [Abstract] [Full Text] [PDF]
2. Ping Sun, David A. Crdenas, Rich Harrill. 2015. Chinese customers evaluation of travel agency website quality: A decision tree
analysis. Journal of Hospitality Marketing & Management 150416144358002. [CrossRef]
3. Devashish Das Gupta, Utkarsh. 2014. Assessing the website effectiveness of top ten tourist attracting nations. Information
Technology & Tourism 14, 151-175. [CrossRef]
4. Gaylen N. Chandler, J. Christian Broberg, Thomas H. Allison. 2014. Customer Value Propositions in Declining Industries:
Differences between Industry Representative and High-Growth Firms. Strategic Entrepreneurship Journal n/a-n/a. [CrossRef]
5. Corrado lo Storto. 2013. Evaluating ecommerce websites cognitive efficiency: An integrative framework based on data envelopment
analysis. Applied Ergonomics 44, 1004-1014. [CrossRef]
6. Layla Hasan, Anne Morris, Steve Probets. 2013. Ecommerce websites for developing countries a usability evaluation framework.
Online Information Review 37:2, 231-251. [Abstract] [Full Text] [PDF]
7. Abida Ellahi, Rahat H. Bokhari. 2013. Key quality factors affecting users' perception of social networking websites. Journal of
Retailing and Consumer Services 20, 120-129. [CrossRef]
8. Layla Hasan, Anne Morris, Steve Probets. 2012. A comparison of usability evaluation methods for evaluating e-commerce
websites. Behaviour & Information Technology 31, 707-737. [CrossRef]
9. Vasiliki Vrana, Kostas Zafiropoulos. 2011. Associations Between USPs and Design Characteristics of Mediterranean Countries'
Websites. Journal of Hospitality Marketing & Management 20, 766-790. [CrossRef]
10. Jos Carlos Pinho, Maria de Lurdes Martins, Isabel Macedo. 2011. The effect of online service quality factors on internet usage.
International Journal of Quality & Reliability Management 28:7, 706-722. [Abstract] [Full Text] [PDF]
11. Chad Lin, YuAn Huang, Rosemary Stockdale. 2011. Developing a B2B web site effectiveness model for SMEs. Internet Research
21:3, 304-325. [Abstract] [Full Text] [PDF]
12. Wen-Hsien Tsai, Wen-Chin Chou, Chien-Wen Lai. 2010. An effective evaluation model and improvement analysis for national
park websites: A case study of Taiwan. Tourism Management 31, 936-952. [CrossRef]
13. Wen-Chih Chiou, Chin-Chao Lin, Chyuan Perng. 2010. A strategic framework for website evaluation based on a review of the
literature from 19952006. Information & Management 47, 282-290. [CrossRef]
14. Hyung Jun Ahn. 2010. Evaluating customer aid functions of online stores with agent-based models of customer behavior and
evolution strategy. Information Sciences 180, 1555-1570. [CrossRef]
15. Glin Bykzkan, Jbid Arsenyan, Grdal Ertek. 2010. Evaluation of E-Learning Web Sites Using Fuzzy Axiomatic Design
Based Approach. International Journal of Computational Intelligence Systems 3, 28-42. [CrossRef]
16. R. Dale Wilson. 2010. Using clickstream data to enhance businesstobusiness web site performance. Journal of Business &
Industrial Marketing 25:3, 177-187. [Abstract] [Full Text] [PDF]
17. Gulcin Yucel, Ahmet F. Ozok. 2010. Quantifying ergonomic quality of governmental websites. Electronic Government, an
International Journal 7, 233. [CrossRef]
18. Jean-Claude Usunier, Nicolas Roulin, Bjrn Sven Ivens. 2009. Cultural, National, and Industry-Level Differences in B2B Web
Site Design and Content. International Journal of Electronic Commerce 14, 41-88. [CrossRef]
19. Robert A. Opoku. 2009. Mapping Destination Personality in Cyberspace: An Evaluation of Country Web Sites Using
Correspondence Analysis. Journal of Internet Commerce 8, 70-87. [CrossRef]
20. Francisco Javier Miranda, Ramn Sanguino, Toms M. Baegil. 2009. Quantitative assessment of European municipal web sites.
Internet Research 19:4, 425-441. [Abstract] [Full Text] [PDF]
21. Fotini Patsioura, Maro Vlachopoulou, Vicky Manthou. 2009. A new advertising effectiveness model for corporate advertising web
sites. Benchmarking: An International Journal 16:3, 372-386. [Abstract] [Full Text] [PDF]
22. Indranil Bose, Alvin Chung Man Leung. 2008. Assessing anti-phishing preparedness: A study of online banks in Hong Kong.
Decision Support Systems 45, 897-912. [CrossRef]
23. Eleni Panopoulou, Efthimios Tambouris, Konstantinos Tarabanis. 2008. A framework for evaluating web sites of public authorities.
Aslib Proceedings 60:5, 517-546. [Abstract] [Full Text] [PDF]
24. Stewart Adam, Michael D Featherstone. 2007. A comparison of web use in marketing by local government in the United States
and Australia. Journal of Database Marketing & Customer Strategy Management 14, 297-310. [CrossRef]

Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)

25. GLIN BYKZKAN, DA RUAN. 2007. EVALUATING GOVERNMENT WEBSITES BASED ON A FUZZY
MULTIPLE CRITERIA DECISION-MAKING APPROACH. International Journal of Uncertainty, Fuzziness and KnowledgeBased Systems 15, 321-343. [CrossRef]
26. Young Namkung, Seo-Young Shin, Il-Sun Yang. 2007. A Grounded Theory Approach to Understanding the Website Experiences
of Restaurant Customers. Journal of Foodservice Business Research 10, 77-99. [CrossRef]
27. Glin Bykzkan, Da Ruan, Orhan Feyziolu. 2007. Evaluating e-learning web site quality in a fuzzy environment. International
Journal of Intelligent Systems 22:10.1002/int.v22:5, 567-586. [CrossRef]
28. Ray Welling, Lesley White. 2006. Web site performance measurement: promise and reality. Managing Service Quality: An
International Journal 16:6, 654-670. [Abstract] [Full Text] [PDF]
29. R. Ufuk Bilsel, Glin Bykzkan, Da Ruan. 2006. A fuzzy preference-ranking model for a quality evaluation of hospital web
sites. International Journal of Intelligent Systems 21:10.1002/int.v21:11, 1181-1197. [CrossRef]
30. Ray Welling, Lesley White. 2006. Measuring the Value of Corporate Web Sites. Journal of Internet Commerce 5, 127-145.
[CrossRef]
31. Talha Harcar, John E. Spillan. 2006. Exploring Latin American family decision-making using correspondence analysis. Journal
of World Business 41, 221-232. [CrossRef]
32. COSTAS ZAFIROPOULOS, VASILIKI VRANA, DIMITRIOS PASCHALOUDIS. 2006. Patterns of Information Services
on Hotel Websites: The Case of Greece. Anatolia 17, 55-74. [CrossRef]
33. Nikos Manouselis, Constantina Costopoulou. 2006. Quality in metadata: a schema for ecommerce. Online Information Review
30:3, 217-237. [Abstract] [Full Text] [PDF]
34. Charles Oppenheim, Louise Ward. 2006. Evaluation of web sites for B2C ecommerce. Aslib Proceedings 58:3, 237-260. [Abstract]
[Full Text] [PDF]
35. Wayne Huang, Taowen Le, X. Li, S. Gandha. 2006. Categorizing web features and functions to evaluate commercial web sites.
Industrial Management & Data Systems 106:4, 523-539. [Abstract] [Full Text] [PDF]
36. Costas Zafiropoulos, Vasiliki Vrana, Dimitrios Paschaloudis. 2006. The internet practices of hotel companies: an analysis from
Greece. International Journal of Contemporary Hospitality Management 18:2, 156-163. [Abstract] [Full Text] [PDF]
37. Irene Pollach. 2005. Corporate selfpresentation on the WWW. Corporate Communications: An International Journal 10:4,
285-301. [Abstract] [Full Text] [PDF]
38. Jrgen KaiUwe Brock, Yu (Josephine) Zhou. 2005. Organizational use of the internet. Internet Research 15:1, 67-87. [Abstract]
[Full Text] [PDF]
39. Muneesh Kumar, Mamta SareenEvaluating Web Sites of Municipal Corporations 143-157. [CrossRef]