Available online at www.sciencedirect.

com

Journal of Interactive Marketing 23 (2009) 321 331

www.elsevier.com/locate/intmar

An Experimental Study of the Relationship between Online Engagement and

Advertising Effectiveness
Bobby J. Calder a , Edward C. Malthouse b, & Ute Schaedel c
a

Marketing Department, Kellogg School of Management, Northwestern University, USA

Department of Integrated Marketing Communications, Medill School of Journalism, Northwestern University, USA
c
Department of Media Management, Hamburg Media School, University of Hamburg, Germany

Abstract
We discuss consumer engagement with a website, provide a systematic approach to examining the types of engagement produced by specific
experiences, and show that engagement with the media context increases advertising effectiveness. Based on experiments using measurement
scales involving eight different online experiences, we advance two types of engagement with online media Personal and Social-Interactive
Engagement. Our results show that both types are positively associated with advertising effectiveness. Moreover, Social-Interactive Engagement,
which is more uniquely characteristic of the web as a medium, is shown to affect advertising after controlling for Personal Engagement. Our
results offer online companies and advertisers new metrics and advertising strategies.
2009 Direct Marketing Educational Foundation, Inc. Published by Elsevier Inc. All rights reserved.
Keywords: Online advertising; Engagement; Consumer behavior; Context effects; Online media; Internet marketing

Introduction
Media provide a context for advertising that may affect
consumer responses to advertising. Many studies have
investigated possible media context effects. The most general
conclusion is that when consumers are highly engaged with a
media vehicle they can be more responsive to advertising (e.g.,
Aaker and Brown 1972; Bronner and Neijens 2006; Coulter,
1998; Cunningham, Hall, and Young 2006; DePelsmacker,
Geuens, and Anckaert 2002; Feltham and Arnold 1994;
Gallagher, Foster, and Parsons 2001; Nicovich, 2005; Wang,
2006). While this conclusion is not surprising, media buyers do
not consider consumer engagement with a media vehicle in
their decisions, except in secondary, ad-hoc ways. For example,
the price of print advertising is determined by circulation, the
location of the ad within the publication and characteristics of

Corresponding author.
E-mail addresses: calder@northwestern.edu (B.J. Calder),
ecm@northwestern.edu (E.C. Malthouse),
u.schaedel@hamburgmediaschool.com (U. Schaedel).

the ad such as the number of colors; and algorithms used to

place banner and sidebar ads do not consider consumer
engagement with the hosting site.
There are many explanations for why consumer engagement with the surrounding media context is not considered
when making advertising decisions. One reason, as we will
demonstrate in the next section, is that many practitioners and
academics do not agree on what engagement is. Making
matters worse, related terms such as involvement and
experience are also used in the academic and trade literatures
without any consensus over whether or how they are different
from engagement.
At the same time, advertisers are searching for ways to
overcome the problems of ad clutter and avoidance (Cho and
Cheon 2004). Leveraging the media context is a potential
solution since advertisers have (at least some) control over
where their ads appear and we know that context can affect
reactions to ads. Moreover, online media is gaining prominence
and spending on online advertising is growing at a rapid pace
(Shankar and Hollinger 2007). It is important to better
understand how engagement is related to the effectiveness of
advertising in the context of online media.

1094-9968/$ - see front matter 2009 Direct Marketing Educational Foundation, Inc. Published by Elsevier Inc. All rights reserved.
doi:10.1016/j.intmar.2009.07.002

322

B.J. Calder et al. / Journal of Interactive Marketing 23 (2009) 321331

The contribution of this paper is twofold. First, we define

consumer engagement with a website and its relationship to
online experiences. As summarized below, other work has
explored distinct online experiences and related concepts. This
article conceptualizes engagement as a second-order construct
that is manifested in various first-order experience constructs.
We theorize that our engagement construct is causally related to
consumer responses to online advertising. Second, we develop
measures of engagement and test our theory by evaluating
whether these measures are associated with consumer evaluations of a banner advertisement. We close with a discussion on
how understanding engagement can help the online firms
manage their sites and advertisers improve the effectiveness of
their ads.
Engagement, experiences, and advertising effectiveness
What is engagement?
Most people know what engagement with media feels like.
Those who are engaged with, for example, a television
program or website have a certain connection with it and
probably view or visit it often. But it is difficult to define the
concept of engagement beyond loose descriptions such as
feeling a connection and using it often.
We begin with what engagement is not. Our conceptualization of engagement is different from others who have
characterized it in ways that we regard as consequences of
engagement. Marc (1966), for example, defines engagement as
how disappointed someone would be if a magazine were no
longer published. Syndicated market research often asks
whether a publication is one of my favorites, whether a
respondent would recommend it to a friend or is attentive.
Many equate engagement exclusively with behavioral usage.
That is, they define engaged people as those who visit the site
often, spend substantial time on the site, or have many page
views. The Advertising Research Foundation (ARF) gives the
definition media engagement is turning on a prospect to a
brand idea enhanced by the surrounding context (ARF, 2006).
Clearly engagement has many different meanings.
We argue that all of the meanings discussed above are
consequences of engagement rather than engagement itself. It is
engagement with a website that causes someone to want to visit
it, download its pages, be attentive to it, recommend it to a
friend, or be disappointed if it were no longer available.
Likewise, researchers have known for years (see citations in
Introduction) that the media context can turn on a prospect to
some advertised brand, but again, this is a consequence of
engagement. Engagement is antecedent to outcomes such as
usage, affect, and responses to advertising.
To think about what engagement really means, let us return
to the basic notion of a sense of being connected with
something. We feel this intuition is essentially correct, but
needs elaboration to be useful. The fundamental insight is that
engagement comes from experiencing a website in a certain
way. To understand engagement we need to understand the
different experiences that consumers have in connecting with

the site (see Fig. 1). Consumer engagement with a website is a

collection of experiences with the site.
We define an experience as a consumer's beliefs about how a
site fits into his/her life. For example, content can be engaging
because users have a utilitarian experience with it. That is, they
believe that the site provides information to help them make
important decisions and accomplish something in their lives.
Other content can be engaging because it provides users with an
intrinsically enjoyable experience, enabling them to unwind and
escape from the pressures of daily life.
To be engaging, different sites need not deliver the same
experiences. Some sites could be engaging because they provide
high levels of a utilitarian experience while other sites could be
engaging because they are intrinsically enjoyable. Experiences
are not necessarily mutually exclusive and some content could
engender high levels of multiple experiences. It is necessary to
realize that there is more than one path to engagement and that
the different paths are realized by offering different experiences.
Consider, for example, the travel section of www.nytimes.com.
Some articles could engage readers by creating a utilitarian
experience, where the reader believes the articles give useful
advice about what to do and where to stay at certain destinations.
Other articles could be engaging because they offer intrinsic
enjoyment. A narrative story about some travel adventure could
relax readers and transport them to a different place and not
provide utilitarian how-to detail. Similarly, different consumers could have different experiences with the same content.
In the language of measurement models, experiences are
first-order constructs while engagement is a second-order
construct. We shall use the term experience whenever we
refer to a specific set of consumer beliefs about a vehicle such as
utilitarian or intrinsic enjoyment, and the term engagement
whenever we refer to the overall experiences of a vehicle.
Online experiences
It follows from the above discussion that we need to
determine the first-order experiences before we can measure
this second-order construct of engagement. There are many
independent streams of research examining consumers'

Fig. 1. Engagement and its consequences.

B.J. Calder et al. / Journal of Interactive Marketing 23 (2009) 321331

experiences online and with media in general. While there is

substantial overlap between the experiences posited by the
different streams, unfortunately they are not entirely consistent.
Certain experiences exist in some frameworks but not others.
Among the experiences that consistently exist in multiple
frameworks, there are often subtle differences in the way in
which they are conceptualized. In some cases, multiple
experiences under one framework are subsumed by a single
experience of another.
Uses and gratifications (U&G) theory (sometimes called an
approach rather than a theory) provides a functionalist
explanation of why people use media and has been an active
area of research within communications since the 1940s (e.g.,
see Ruggiero 2000 for a recent survey). The U&G literature is
vast; McQuail (1983, pp. 823) gives a concise summarization
that is often cited:
Information finding out about relevant events and
conditions in immediate surroundings, society and the world;
seeking advice on practical matters or opinion and decision
choices; satisfying curiosity and general interest; learning,
self-education; gaining a sense of security through
knowledge.
Personal identity finding reinforcement for personal
values; finding models of behavior [sic]; identifying with
valued others (in the media); gaining insight into one's self.
Integration and social interaction gaining insight into the
circumstances of others; social empathy; identifying with
others and gaining a sense of belonging; finding a basis for
conversation and social interaction; having a substitute for
real-life companionship; helping to carry out social roles;
enabling one to connect with family, friends and society.
Entertainment escaping, or being diverted, from
problems; relaxing; getting intrinsic cultural or aesthetic
enjoyment; filling time; emotional release; sexual arousal.
The utilitarian experience discussed above is an example of
information in the U&G framework and the intrinsic enjoyment
experience is an example of entertainment.
U&G approaches have been used in interactive marketing.
For example, Nambisan and Baron (2007) applied a variation
of the U&G constructs to explain virtual customer environments with four experiences: cognitive, social integrative,
personal integrative, and hedonic. Bronner and Neijens (2006)
measure eight experiences that are consistent with the U&G
approach: practical use, social, identification, pastime, transformation, stimulation, information, and negative emotion.
Childers et al. (2001) discuss utilitarian and hedonic (a type of
entertainment in the U&G approach) experiences as
explanations of online shopping behavior. The same approach
is also followed by Fiore et al. (2005) and Cotte et al. (2006).
Flow is another construct that has received substantial attention
(e.g., see Hoffman and Novak 2009) and is consistent with the
U&G approach of understanding the consumer experience with
media.
Media engagement is particularly interesting in the case
of websites. It is commonly thought that online media are

323

experienced differently than more traditional media such as

television and print. This difference is often described as
leaning forward versus leaning backward. The online
experience is thought to be more active, participatory and
interactive. The internet is also thought to be more social in
nature because it can be used for sharing and communicating
and it therefore breeds social engagement (Mathwick 2002;
Rappaport 2007). Ruggiero (2000, p. 15) highlights the need
to include interactivity in U&G framework. Previous
studies have tended to focus on this experience at a high
level or for specific applications. For example Thorbjrnsen
et al. (2002) examined the overall amount of experience
people have with the web but deal only with the level of
experience and not the nature of that experience. Nambisan
and Baron (2007) discuss an interaction experience in
virtual customer environments. Tremayne (2005) addresses
the meaning of interactivity and concludes that it can be
viewed either as a process of message exchange or as a
perceptual variable. Others have studied interactivity in the
form of word of mouth (e.g., Brown et al. 2007; Dwyer 2007;
Sen and Lerman 2007). Prahalad and Ramaswamy (2004) and
Sawhney et al. (2005) discuss the co-creation experience.
It is unnecessary for purposes of this article to sort out
differences in the ways that various frameworks have
conceptualized experiences because, for the purpose of
measuring engagement, all we need is a set of experiences
that can serve as indicators of the engagement construct domain.
No set of indicators would be exhaustive of this domain but this
is not required from a measurement point of view.1 Our
approach is to develop scales for a representative set of
experiences that parallel those noted in the literature. We shall
then factor analyze the experience measures and test whether
they could plausibly be manifestations of a second-order
engagement construct or constructs. The above discussion
indicates that websites may deliver different types of experiences than traditional media, as characterized by the four
McQuail (1983) U&G aspects.
Engagement and advertising effectiveness
The conceptual framework (Fig. 1) posits that engagement
and experiences are antecedent to reactions to ads. We seek to
test this relationship as an indicator of the predictive validity
of our measures. There has been relatively little previous
research on the impact of the online media context on
advertising. Existing studies have approached this at either a
1
The question arises of whether to treat experiences and engagement as
formative or reflective. We follow Jarvis, Mackenzie, and Podsakoff's (2003)
criteria for making the decision. We treat both as reflective (a Type I secondorder factor specification in the language of Jarvis, Mackenzie, and Podsakoff).
In the case of experiences, the items are manifestations of some experience, are
interchangeable, and should covary. The items we have used represent a sample
from the respective construct domains, e.g., there are many ways that a person
can have a utilitarian experience and different items could represent the
construct domain equally well. Thus, experiences are reflective according to the
Jarvis et al. framework. We also think of engagement as a reflective construct
because we view experiences as manifestations of engagement (reflective)
rather than as defining characteristics (formative).

324

B.J. Calder et al. / Journal of Interactive Marketing 23 (2009) 321331

Selecting experience scales

could be used as indicators of engagement. Ideally these scales

should produce an acceptable fit in a measurement model and
have good psychometric properties such as acceptable reliability and convergent and discriminant validity. We are unaware of
any previous studies of online experiences that measure such a
broad range of experiences with these high standards.
The present study uses the CalderMalthouse (CM) set
of media experiences (Calder and Malthouse 2004, 2005;
Malthouse, Calder, and Tamhane 2007). We briefly summarize
their methodology and argue that these experiences span the
engagement domain. CM conducted over 400 hour-long, indepth interviews with consumers about the role that specific
websites, newspapers, magazines, and TV news programs play
in their lives. They analyzed the transcripts for common themes
and created hundreds of Likert-scale items. The items were
included on surveys of website visitors, newspaper and
magazine readers, and TV news viewers. Exploratory factor
analysis identified 22 online experiences, 44 newspaper
experiences, 39 magazine experiences, and 12 TV news
experiences. The values of coefficient alpha suggested that
most of the scales were reliable (some had weak reliability
because of too few items). None of the CM studies estimate
confirmatory factor analysis models. Some experiences are
common across media, while others are specific to a particular
medium (e.g., media websites). CM also showed that their
experiences are associated with usage (site usage, readership,
and viewership) and, in the case of magazines, reactions to
advertising.
In this research we had to select eight experiences from the
22 CM online ones, due to constraints on survey length and
respondent fatigue. Requiring our experience measures to have
an acceptable fit in a measurement model also limits the number
of experiences that we can include.2 In reviewing the original 22
experiences, we decided that some did not fit in the construct
domain because they describe the site itself rather than how the
site fits into the consumer's life. For example, one of the
experiences was about credibility of the site and another was
about the site being easy to use. Several experiences were also
dropped because they were specifically about the advertising on
the site.
The eight experiences and their items are displayed in Table 1.
They were selected with a stratified sampling procedure from the
remaining experiences so that there would be at least one from
each of the four McQuail U&G categories (the strata) that
characterize more traditional media, and others, such community and participation and socializing, that are particularly
relevant to online media. We tried to avoid picking too many
experiences from any single McQuail U&G category. For
example, two of the remaining experiences fit under McQuail's
information category: makes me smarter, which is about
keeping people up-to-date on issues that concern them, and
utilitarian, which is more about advice and how-to
information. Using the flip of a coin we decided to include
utilitarian. Likewise, the original CM experiences intrinsic

As indicated, to accomplish the objectives of this study, we

needed measurement scales for a set of online experiences that

2
Hatcher (1994, p. 260) recommends using a maximum of 2030 indicator
variables in a measurement model and we will use 37.

very high level or very specifically. To wit Bronner and

Neijens (2006) compare the experiences of different types of
media with the experiences of advertising content. They find,
for instance, that the experience of usefulness with a site is
related to the ads on that site being experienced as useful.
And Wang (2006) finds in the context of an online game that
an online ad inviting users to play a game was more effective
than an ad that did not, suggesting that the game-ad might
have benefited from the game context. Previous work has also
focused specifically on the use of interactivity in online ads
(e.g., Ariely 2000; Chatterjee, Hoffman, and Novak 2003;
Pavlou and Steward 2000). Hupfer and Grey (2005) test the
effect of the offer (e.g., whether there is a free sample) and
user mode (e.g., goal-directed) on attitudes towards the brand
and ad.
There are several theoretical explanations for why engagement should affect reactions to advertising including affect
transfer (e.g., Broniarczyk and Alba 1991, p. 215) and
categorization theory (Cohen and Basu 1987). Dahln (2005)
does a literature review of media context effects and
summarizes three possible theoretical rationales for why context
should affect reactions to ads. The first is the mood
congruencyaccessibility hypothesis: The ad context makes
a certain mood or affect more accessible and relieves the
processing of stimuli with similar moods or affects (p. 90). The
second is the congruity principle: the medium and the
advertised brand converge and become more similar in
consumers' minds (p. 90). The third is that the context serves
as a cognitive prime that activates a semantic network of
related material that guides attention and determines the
interpretation of the ad (p. 90). It should be noted that these
explanations are not alternative explanations but rather all of
them are plausible mechanisms for how media context can
affect advertising. They lead us to formally hypothesize:
Hypothesis 1. Engagement with the surrounding online media
vehicle context increases advertising effectiveness.
Methods and results
Our methodology consisted of several steps, each of which
will be discussed in this section. The first step was to select
scales to measure experiences that span the construct domain
and provide indicators of engagement. Next we executed a
survey that employed the scaling measures of experiences and a
quasi-experimental design to evaluate advertising effectiveness.
The survey data allowed us to evaluate the psychometric
properties of our experience scales and engagement by
estimating a confirmatory factor analysis measurement model
for the experience scales and then a second-order factor model
for engagement. The final step was to test the research
hypothesis that engagement increases ad effectiveness.

B.J. Calder et al. / Journal of Interactive Marketing 23 (2009) 321331

Table 1
Question wording and parameter estimates from confirmatory factor analysis
measurement model.
Experience

Item

It inspires me in my own life.

This site makes me think of things in new
ways.
This site stimulates my thinking about lots
of different topics.
This site makes me a more interesting person.
Some stories on this site touch me deep down.
Social Facilitation
I bring up things I have seen on this site in
( = .88)
conversations with many other people.
This site often gives me something to talk
about.
I use things from this site in discussions or
arguments with people I know.
Temporal ( = .90)
It's part of my routine.
This is one of the sites I always go to
anytime I am surfing the web.
I use it as a big part of getting my news for
the day.
It helps me to get my day started in the
morning.
Self-Esteem and Civic Using this site makes me feel like a better
Mindedness ( = .91) citizen.
Using this site makes a difference in my life.
This site reflects my values.
It makes me more a part of my community.
I am a better person for using this site.
Intrinsic Enjoyment
It's a treat for me.
( = .87)
Going to this site improves my mood, makes
me happier.
I like to kick back and wind down with it.
I like to go to this site when I am eating or
taking a break.
While I am on this site, I don't think about
other sites I might go to.
Utilitarian ( = .88)
This site helps me make good purchase
decisions.
You learn how to improve yourself from this
site.
This site provides information that helps me
make important decisions.
This site helps me better manage my money.
I give advice and tips to people I know based
on things I've read on this site.
Participation and
I do quite a bit of socializing on this site.
Socializing ( = .88) I contribute to the conversation on this site.
I often feel guilty about the amount of time I
spend on this site socializing.
I should probably cut back on the amount of
time I spend on this site socializing.
Community ( = .88)
I'm as interested in input from other users as
I am in the regular content on this site.
A big reason I like this site is what I get from
other users.
This site does a good job of getting its
visitors to contribute or provide feedback.
I'd like to meet other people who regularly
visit this site.
I've gotten interested in things I otherwise
wouldn't have because of others on this site.
Overall, the visitors to this site are pretty
knowledgeable about the topics it covers so
you can learn from them.
Stimulation and
Inspiration( = .88)

Stand.
loading
.85
.84
.78
.79
.71
.85
.85
.81

325

enjoyment, entertains and absorbs me and a way to fill my

time all fit under McQuail's entertainment category and we
selected the first (at random). The social facilitation
experience was selected as a representative of McQuail's
integration and social interaction category. The self-esteem
and civic mindedness experience represents the personal
identity category.
We claim that these eight experiences are representative of
the engagement construct domain. Of course, other sets could
also represent the domain, but our approach is entirely
consistent with our objective of developing indicators of
engagement. For example, we would not expect our engagement measure to change in a substantive way if we had used
makes me smarter rather than utilitarian from the
information category.

.85
.83

Survey methodology

.84

The second step was to sample users of media websites.

Eleven online media websites were used in the present
confirmatory study.3 These sites represent a convenience
sample, but include broad range of different types of media
sites including those with national reputations (e.g., Reuters.
com and Washingtonpost.com), special interest sites (e.g.,
about.com) and local sites (e.g., king5.com). The target
population, identified with a screening question, was people
who used the site at least once a month. Subjects were recruited
from the visitors on the particular sites, who were redirected to
an online survey. The sample sizes for the 11 sites ranged from
n = 203 to n = 2006, with a median sample size of n = 1141 and a
total sample size of n = 11,541. Respondents were asked about
their usage and experiences with the particular site.

.80
.86
.85
.76
.75
.88
.83
.85
.82
.65
.71
.81

Measurement models for experiences and engagement

.83

We develop measures of online engagement using a two-step

process. First, we estimate a confirmatory factor analysis
measurement model to study the psychometric properties of our
experience measures. Second, we develop second-order engagement factors by applying exploratory factor analysis to the
eight experiences and then fitting a second-order confirmatory
factor analysis model.
The first step in developing the online engagement measures
is to estimate a measurement model for the experiences,
allowing each possible pair of experiences to be correlated. Fit
statistics are provided in Table 2. Question wording, factor
loadings, and the values of coefficient alpha are provided in
Table 1. There were 37 items used to measure the 8 experiences.
All eight scales are highly reliable, with coefficient alpha
ranging from .87 to .91. In the measurement model, each of
the 37 items had a parameter fortheloading and error variance
8
(37 + 37 = 74), and there were
= 28 parameters for the
2

.76
.81
.74
.86
.77
.82
.78
.84
.85
.59
.80
.73
.66

3
The sites are about.com, Washingtonpost.com, PalmBeachPost.com,
Reuters.com, DallasNews.com, Projo.com, King5.com, AZFamily.com,
WFAA.com, KHOU.com, and PE.com.

326

B.J. Calder et al. / Journal of Interactive Marketing 23 (2009) 321331

Table 2
Summary of confirmatory factor analysis model.

Parameters
GFI
CFI
NNFI
RMSEA

Measurement model

Second-order CFA model

102
.9155
.9482
.9426
.0472

87
.9029
.9392
.9343
.0505

Note. n = 5942 with 37 items.

covariances between every pair of experiences, giving a total of

102 parameters. GFI, CFI, and NNFI all exceed .90, indicating
an acceptable fit.
Convergent validity was assessed with the t-values of the
factor loadings, computed as the ratio of the loading to the
standard error of the item. Convergent validity is supported
when t-values reach an absolute value greater than 2. The
minimum t-value was 48.2, providing evidence in support of
the convergent validity of the indicators. We assess discriminant validity with the chi-square difference test. For each of
the 28 pairs of experiences we estimated a separate
measurement model identical to the one shown in Table 2,
except that the covariance between the pair is fixed at 1. The
chi-square statistics between the models were computed, and
range from 4132 to 12,073. The differences have chi-square
distributions with 1 df, and are very highly significant,
supporting discriminant validity.
Pearson correlations between the experiences are provided in
Table 3. Note that the correlations follow a pattern that suggests
the possibility of second-order factors. The first six experiences
are moderately correlated with each other, with values between
.42 and .72. Participation and Socializing (7) is substantially

Table 3
Correlation matrix (treatment group only).
Pearson correlation
1
Experience
1 Stimulation and
Inspiration
2 Social Facilitation
3 Temporal
4 Self-Esteem and Civic
Mindedness
5 Intrinsic Enjoyment
6 Utilitarian
7 Participation and
Socializing
8 Community
Engagement
9 Personal Engagement
10 Interactive
Engagement
Advertising
11 Click Intention
12 Attitude Towards Ad

10

.56
.51 .55
.65 .57 .47
.65 .52 .62 .63
.62 .52 .42 .72 .58
.24 .19 .19 .29 .33 .35
.51 .41 .32 .53 .53 .59 .56
.79 .75 .78 .82 .81 .71 .32 .51
.52 .43 .43 .69 .61 .67 .77 .77 .74

.24 .19 .15 .25 .23 .27 .12 .23 .27 .26
.30 .23 .19 .31 .29 .31 .14 .27 .34 .32

Note. All correlations are significantly different from 0 at the .0001 level.

less correlated with the first six, but moderately correlated with
the Community experience (8). Community is somewhat less
correlated with the first six experiences. This correlation
structure suggests that there is a higher-order factor structure
generating the data.
Therefore the second step in developing the measurement
model is to identify the second-order engagement factors. To do
this we did both an exploratory and a confirmatory factor
analysis. We performed an exploratory factor analysis with a
varimax rotation on the first-order experiences and found two
eigenvalues greater than 1. The rotated factor loadings are
provided in Table 4 and show two interpretable factors,
hereafter called Personal Engagement and Social-Interactive
Engagement. The first six experiences from the correlation
matrix have the largest loadings on Personal Engagement,
although Community also has a cross-loading greater than .3.
Participation and Socializing as well as Community have the
largest loadings on Social-Interactive Engagement, but several
other experiences have sizable cross-loadings. The Utilitarian
experience likely cross-loads on Social-Interactive Engagement
because much of the advice and tips could be coming from the
community of users rather than from content created by
employees of the site itself. Self-esteem likely cross-loads
because contributing to an online conversation could contribute
to one's self-esteem.
We then estimated a second-order confirmatory factor
model, which is a more parsimonious model for the 37 37
covariance matrix than the measurement model for experiences.
The objective was to test whether it is plausible that the Personal
and Social-Interactive Engagement latent variables generate the
observed correlation structure between the experiences and
items. Personal and Social-Interactive Engagement will be used
in the subsequent analyses of advertising effectiveness. Instead
of having 28 covariances between the experiences, we assume
that correlations between the experiences are due to two secondorder factors. This model can represent the correlations between
the experiences with only 12 factor loadings shown in Table 4
above, and one additional term for the covariance between the
second-order factors. Fit statistics are also shown in Table 2
above, with CFI, GFI, and NNFI all greater than .9 suggesting a
good fit. Fig. 2 shows the parameter estimates of the second-

Table 4
Exploratory factor analysis loadings of first-order experiences.
Experience

Factor 1
Personal
Engagement

Factor 2
SocialInteractive

Social Facilitation
Temporal
Stimulation and Inspiration
Self-Esteem and Civic
Mindedness
Intrinsic Enjoyment
Utilitarian
Participation and Socializing
Community

.768
.753
.744
.710

.375

Note. Loadings less than .3 were omitted.

.701
.612
.361

.366
.472
.881
.755

B.J. Calder et al. / Journal of Interactive Marketing 23 (2009) 321331

order factor structure. The loadings for the 37 items were very
similar to those from the measurement model above and have
been omitted. Note that the second-order factor model finds a
significant correlation between the two engagement latent
variables. In the analyses that follow, we estimate the two
engagement factors using a weighted average of the experiences, with the factor loadings as weights.
Personal Engagement is manifested in experiences that are
similar to those that people have with newspapers and
magazines. For example, experience items such as This site
makes me think of things in new ways or This site often gives
me something to talk about could also apply to a newspaper or
magazine. Social-Interactive Engagement, however, is more
specific to websites. Items such as I do quite a bit of socializing
on this site and I contribute to the conversation on this site
would not characterize a newspaper or magazine, and we did
not hear such statements in our qualitative interviews for these
media. While Social-Interactive Engagement is more closely
associated with the web, aspects of it can be found for other
media. For example, A big reason I like this site is what I get
from other users could also apply to the letters-to-the-editor
page of a daily newspaper. The Utilitarian experience is a
manifestation of both forms of engagement. Service oriented
websites (e.g., bhg.com Better Homes and Gardens) will
have a prominent utilitarian component as will user-contributed
advice sites (e.g., Yahoo!Answers or chowhound.com).
In sum, the measurement model and values of coefficient
alpha have shown that the eight experiences have been
measured reliably and support the convergent and discriminant
validity of the scales. The second-order analysis shows
two engagement factors, Personal Engagement and SocialInteractive Engagement. Personal Engagement is manifested in
experiences that have counterparts in magazines and news-

327

papers while Social-Interactive Engagement is more specific to

websites. As reflected the loadings in Fig. 2, with Personal
Engagement, users seek stimulation and inspiration from the
site, they want to use the site to facilitate their interactions with
other people, they feel the site affirms their self-worth, they get
a sense of intrinsic enjoyment in using the site itself, they feel it
is useful for achieving goals, and they value input from other
users. With Social-Interactive Engagement, users experience
some of the same things in terms of intrinsic enjoyment,
utilitarian worth, and valuing the input from the larger
community of users but in a way that links to a sense of
participating with others and socializing on the site. Thus
Social-Interactive Engagement is motivated both intrinsically
and extrinsically, but in this case it is the social relevance of
these, rather than their personal or individual quality, that is
associated with the larger engagement experience. And it is the
valuing of input from the community and sense of participating
with others and socializing that gives Social-Interactive
Engagement its dominant character.
The relationship between engagement and advertising
effectiveness
We now test the hypothesis (H1) that engagement predicts
ad effectiveness. Users of the 11 media websites were
intercepted during their visit to the site and asked to complete
a survey. Participants answered questions about their use of,
and experiences with, this website. They were then shown an
ad for orbitz.com (an online travel agency) and asked to rate
it using standard copy-testing measures and their intention to
click on the ad. A travel agency was used because travel is
potentially relevant to most internet users and this category
often advertises with banner ads. We shall relate engagement

Fig. 2. Second-order factor structure.

328

B.J. Calder et al. / Journal of Interactive Marketing 23 (2009) 321331

and experiences with the media to these ad ratings as a test of

predictive validity.
Note that participants were intercepted while actually
visiting the site, though they did not actually see the ad on the
site. This manipulation of media context is not the same as
encountering the ad while actually on the site but actually
provides a strong test of the hypothesis. If the site experiences
affect reactions to the ad in this test, the effect would be
expected to be, if anything, smaller than in the case of actually
seeing the ad on the site.
One threat to validity is that the mere measurement of the
experiences of a given site might itself affect reactions to the ad.
Whereas this would imply that all experiences would affect the
ad equally, it is at least possible that some of the experiences
could be differentially sensitive to measurement (measurement scale interaction). In this way, merely thinking about
how a site gives advice and tips could have produced a higher
rating of the ad. Another threat is that any effect on advertising
is not due to experiences with a particular site context, but to
experiences with sites in general (which are correlated with the
particular site participants are told the ad is on). Alternatively,
the different experiences individuals had with their site and the
responses to the ads in general could be construed as an
individual difference not dependent per se on using any
particular sites. To assess these threats we used a context-free
control group design. The most important thing about the
control group is that the ad was identified only as a banner and
not linked to any particular site.
Of the 11,536 intercepted on the 11 sites, 1502 were
randomly assigned to the context-free control group, which
was asked about their experiences with reading news sites in
general and told only that the ad was a banner. If any effects
of the experiences on the ad are due to simply rating the
experiences and/or thinking about sites in general while
taking a survey, then the control group should respond in a
similar way to those asked about a specific site. The treatment
group being different from the control group indicates that the
results do not reflect mere measurement or experiences with
sites in general but rather measure the effects of experiences
with specific sites.
We have two measures of reactions to an ad. First, we
developed a multi-item scale to measure attitude towards the ad.
Respondents were asked How well does each of the following
words describe the ad in the [site name]? The study included
the items interesting, lively, helpful, believable, attractive,
imaginative, and soothing (7-point scale from Does not
describe the ad at all to Describes the ad very well). These
items were selected to be typical of those that are commonly
used to test reactions to advertising stimuli (see Bearden and
Netemeyer 1999, Chapter 5) and to fit the ad tested here. The
value of coefficient alpha was .93, indicating a reliable scale. As
a second, complementary measure of reactions to the ad,
respondents were asked: How likely are you to click on this
ad?
Correlations between the experience, the engagement
factors, and the advertising variables are also provided in
Table 3 above. All correlations are positive and highly

Table 5
Estimates from separate models including context-free control group.
Experience

Attitude
toward ad

Intention to
click

Stimulation and Inspiration

Social Facilitation
Temporal
Self-Esteem and Civic Mindedness
Intrinsic Enjoyment
Utilitarian
Participation and Socializing
Community
Personal Engagement
Social-Interactive Engagement

.63
.42
.32
.61
.56
.62
.60
.29
.81
.90

.16
.11
.09
.27
.06
.15
.12
.18
.20
.23

.67
.44
.32
.62
.60
.68
.67
.35
.85
.99

.19
.12
.08
.27
.04
.16
.15
.21
.20
.27

Note. p b .05 marked in italic. p b .01 marked in bold.

significant, indicating that higher experience and engagement

levels are associated with more ad effectiveness, supporting
H1.
Correlations, however, do not account for the different
sites, control for confounding factors such as use of online
travel sites, or rule out measurement effects. We now conduct
a more stringent test of the relationship between the ad ratings
and experiences/engagement by comparing the slopes of the
treatment group (those who were told that the ad appeared on
a specific site) and the context-free control group (those told
the ad was not linked to a site) using an ANCOVA model.
The model includes a different, fixed-effect4 intercept for
each site (j for site j = 1, , 11), a dummy x1 = 1 indicating
the respondent was in the control group, a measure for the
use of online travel agents in general x2, the engagement
rating x3 (as a continuous variable on a 5-point scale), and an
interaction term between experience rating and the control
group dummy (x1x3):
y = aj + b1 x1 + b2 x2 + b3 x3 + gx1 x3 :
The parameter 3 is the slope for engagement in the
treatment group, indicates how much larger or smaller the
engagement slope is in the control group compared with the
treatment group, and 3 + gives the slope for the control
group. We can test whether the slopes in the treatment and
control groups are different with H0: = 0.
The model is estimated separately for each of the
8 experience and 2 engagement measures, with the results
summarized in Table 5 below. Parameter estimates for the
intercept terms 1, , 11, 1, and the slope for product usage
2 are omitted in Table 6 for clarity.5 All of the treatment-group
experience slopes 3 are positive and highly significant,
4
Fixed effects are used rather than random since we have a convenience
sample of sites.
5
In all models, the product usage variables have very highly significant
positive effects. Likewise, across models the extra sums of squares are large
and highly significant for the site intercepts, allowing us to reject the null
hypothesis that all 11 sites have the same intercept. The control group dummy
shifting the intercept (2) is occasionally significant, but the signs change across
models suggesting that the significant results could be type I errors.

B.J. Calder et al. / Journal of Interactive Marketing 23 (2009) 321331

Table 6
Estimates for the model with both types of engagement as predictors.
Dependent
variable

Personal
engagement

Social-Interactive
Engagement

Online travel
agency use

Attitude
towards ad
Intention to
click

.536 (.033)

.443 (.039)

.106 (.006)

.479 (.043)

.590 (.050)

.244 (.008)

Note. p b .05 marked in italic. p b .01 marked in bold.

consistent with the conclusions from the correlation matrix

above supporting H1.6
Testing whether these results are due to measurement effects,
the -values of both dependent variables and for both Personal
and Social-Interactive Engagement are highly significant,
indicating that engagement has a stronger effect on ad ratings
when the respondent associated an ad with a particular site, and
supporting H1 from above. The -values for most, but not all,
of the individual experiences are also significant. As we
indicated above, the manipulation of media context is relatively
weak, and the effect sizes might well be larger if respondents
were actually experiencing the particular site when they were
exposed to the ad.
Having established that both types of engagement are
associated with advertising effectiveness, we now examine
whether Social-Interactive Engagement affects reactions to ads
after controlling for Personal Engagement by including both in
the model, as well as use of online travel agents in general (x2)
and separate, fixed-effect intercepts for individual sites.7 We
shall use only the treatment group in this analysis. The results
are summarized in Table 6. The coefficients for both types of
engagement are highly significant and roughly of comparable
size, indicating that both forms of engagement are important in
predicting advertising effectiveness.
Managerial implications
This research has many applications to both managers at
online companies that host ads and those making advertising

6
The experience slopes for the control group (b3 + g) are also significantly
different from 0, which could be due to any of the threats to internal validity
mentioned above or to the method of recruiting subjects used in this study
(members of the control group were also intercepted from the sites under study
and some may not have completely understood that they were to answer
questions about sites in general rather than the one from which they were
recruited).
7
It could also be tempting to include all 8 experiences in a single regression
model, but such a model is theoretically questionable because there is no
correct model (i.e., set of experiences included as predictor variables, which
are sampled from the construct domain) and all inference on slope coefficients
will be suspect. If experiences are manifestations of high-order engagement
constructs the experience measures will be correlated (creating multicollinearity), and must share in explaining the dependent variables. Thus the experience
effect sizes will depend on the size of the sample from the construct domain.
For example, with a sample of 4 experiences, each experience will have the
opportunity to explain more of the dependent variable than with a sample of 8.
Multicollinearity suggests that the magnitude of the slopes will also be highly
sensitive to the particular sample of experiences drawn.

329

decisions. First, managing a website involves engineering a set

of experiences for the visitors, and then measuring the extent to
which the visitors have the intended experiences. The scales
presented in this paper enable a website to track both
experiences and higher-level engagement. Such measurements
could provide an early warning that the intended experiences
are not being created. Likewise, advertisers and online
companies that produce websites are searching for medianeural metrics for the purpose of common-currency comparisons, e.g., a website with a print vehicle (Winer 2009).
Engagement and experience metrics could serve this purpose.
Second, some managers of advertising vehicles are using
engagement as a way to differentiate themselves from
competitors and retain advertisers. Their basic argument is as
follows: highly engaged readers are more likely to be exposed
to ads; ads carried by vehicles with more engaged readers will
therefore be more effective; and a vehicle with highly engaged
readers should command a premium price for advertising space,
or at least have an advantage in retaining advertisers. Our
research supports this reasoning and practice.
Conclusion and future research
It is commonly believed that the web is different from
other media in terms of leaning forward instead of backward,
being more interactive, more social, and so forth. In this
research, we identified and measured eight different types of
consumer experiences with online news websites and showed
that the measures are reliable with high discriminant,
convergent and predictive validity. Based on a factor analysis
of the eight experiences, we identified two different kinds of
engagement. One factor, Personal Engagement, is manifested
in experiences that are very similar to those that people have
with newspapers and magazines. For example, people have
social experiences with both print and online content by
bringing up an article they read; just as reading a newspaper
at the breakfast table can be habitual, so can reading a
website. The second factor, Social-Interactive Engagement, is
weighted more to experiences that are more unique to the
web, such as participating in discussions and socializing with
others through a site. These experiences give SocialInteractive Engagement its dominant social character. This
finding gives empirical support and specificity to the idea that
the Internet is a different kind of medium.
This work set the stage for examining the effect of online
media engagement on advertising. We related experiences and
engagement to the ratings of a banner ad using a quasiexperimental design. The results show that both Personal and
Social-Interactive Engagement affect reactions to the banner
ad. Therefore, in addition to the Personal Engagement context
effects that have been demonstrated previously for traditional
media, the interactive component of a user's experience with
a website is also shown to affect advertising. The results of a
regression model including both types of engagement indicate
that Social-Interactive Engagement affects reactions to ads
after controlling for Personal Engagement. We conclude that
online media do involve a distinct form of engagement and

330

B.J. Calder et al. / Journal of Interactive Marketing 23 (2009) 321331

that this engagement has its own impact on advertising

effectiveness.
Our conclusions are subject to the limitations of our
methodology. Three points should be kept in mind. First, no
matter how representative the ad used in this study might be,
further research is called for to examine different product
categories and types of advertising execution. It is possible to
formulate many hypotheses in this regard. For example, ads that
are more interactive may have even stronger relationships (i.e.,
greater slopes) with Social-Interactive Engagement. Second, it
would also seem desirable to conduct future research with actual
insertion of ads on websites rather than only intercepting users
on the sites. This might have some value in being a more
realistic methodology with potentially better external validity.
We note, however, that at best achieving external validity
through matching a research setting with some real context is
always fraught with difficulty (Calder, Phillips, and Tybout
1983; Sternthal, Tybout, and Calder 1987). It is never possible
to duplicate the exact context, or even to know what key
variable might be missing. In our view additional work with ads
varying along theoretically motivated dimensions would be
valuable. Third, we have tested the relationship between
engagement ad effectiveness for 11 websites. It is desirable to
test this relationship with more sites.
Taking into consideration the limitations of this study, we
believe that the effects of online media experiences on
advertising are potentially pervasive and need further investigation. While previous research has suggested the importance
of online experiences and the possibility of context effects on
advertising, the present study provides a systematic approach to
examining the types of engagement produced by specific
experiences with online sites and shows that it is engagement
that produces the context effect on online advertising. Further,
the distinctive Social-Interactive Engagement associated with
the web not only increases advertising effectiveness but does so
independently of the type of engagement usually associated
with more traditional media. This implies that interactive
marketers may find online media to have added potential as a
marketing tool.
Finally, the principle of engagement and its effects on
communication effectiveness could be extended to other media
such as mobile media and the social media. The use of mobile
media is growing rapidly and more consumers are engaged with
their mobile devices than before (Shankar and Balasubramanian
2009). Furthermore, social media provide a glue for further
consumer engagement. Understanding the effects of engagement on communication effectiveness in such media is
important for both researchers and practitioners.
Acknowledgments
The authors would like to express their sincere appreciation
to Venkatesh Shankar and two anonymous reviewers for their
detailed comments and careful reviews. They would also like to
thank the On-Line Publishers Association and Northwestern
University's Media Management Center for organizing the
study.

References
Aaker, David and P.K. Brown (1972), Evaluating Vehicle Source Effects,
Journal of Advertising Research, 12(4), 11.
ARF, (2006). Engagement. Retrieved 23 July, 2007, www.thearf.org/research/
engagement.html.
Ariely, Dan (2000), Controlling the Information Flow: Effects on Consumers'
Decision Making and Preferences, Journal of Consumer Research, 27(2),
233.
Bearden, William and Richard Netemeyer (1999), Handbook of Marketing
Scales: Multi-Item Measures for Marketing and Consumer Behaviour
Research. Thousand Oaks, CA: Sage Publications.
Broniarczyk, Susan and Joseph Alba (1991), The Importance of the Brand in
Brand Extension, Journal of Marketing Research, XXXI, 21428.
Bronner, Fred and Peter Neijens (2006), Audience Experiences of Media
Context and Embedded Advertising: A Comparison of Eight Media,
International Journal of Market Research, 48(1), 81.
Brown, Jo, Amanda Broderick, and Nick Lee (2007), Word of Mouth
Communication Within Online Communities: Conceptualizing the Online
Social Network, Journal of Interactive Marketing, 21(3), 220.
Calder, Bobby and Edward C. Malthouse (2005), Experiential Engagement with
Online Content Web Sites and the Impact of Cross-Media Usage,
Proceedings of 12th Worldwide Readership Research Symposium, Prague,
October.
and Edward C. Malthouse (2004), Qualitative Media Measures:
Newspaper Experiences, The International Journal on Media Management,
6(1/2), 123.
, Lynn Phillips, and Alice Tybout (1983), Beyond External Validity,
Journal of Consumer Research, 10(1), 112.
Chatterjee, P., Donna L. Hoffman, and Thomas P. Novak (2003), Modeling the
Clickstream: Implications for Web-Based Advertising Efforts, Marketing
Science, 22(4), 520.
Childers, Terry, Christopher Carr, Joann Peck, and Stephen Carson (2001),
Hedonic and Utilitarian Motivations for Online Retailing Shopping
Behavior, Journal of Retailing, 77, 51135.
Cho, Chang-Hoan and Hongsik John Cheon (2004), Why do People Avoid
Advertising on the Internet?, Journal of Advertising, 33(4), 8997.
Cohen, Joel and Kunal Basu (1987), Alternative Models of Categorization:
Toward a Contingent Processing Framework, Journal of Consumer
Research, 13, 45572.
Cotte, June, Tilottama Chowdhury, S. Ratneshwar, and Lisa Ricci (2006),
Pleasure or Utility? Time Planning Style and Web Usage Behaviors,
Journal of Interactive Marketing, 20(1), 4557.
Coulter, Keith (1998), The Effects of Affective Responses to Media Context on
Advertising Evaluations, Journal of Advertising, 27(4), 41.
Cunningham, Todd, Amy Shea Hall, and Charles Young (2006), The
Advertising Magnifier Effect: An Mtv Study, Journal of Advertising
Research, 4, 46.
Dahln, Micael (2005), The Medium as a Contextual Cue: Effects of Creative
Media Choice, Journal of Advertising, 34(3), 89.
DePelsmacker, Patrick, Maggie Geuens, and P. Anckaert (2002), Media
Context and Advertising Effectiveness: The Role of Context Appreciation
and Context/Ad Similarity, Journal of Advertising, 31(2), 49.
Dwyer, Paul (2007), Measuring the Value of Electronic Word of Mouth and its
Impact in Consumer Communities, Journal of Interactive Marketing, 21
(2), 6379.
Feltham, Tammie and Stephen Arnold (1994), Program Involvement and Ad/
Program Consistency as Moderators of Program Context Effects, Journal
of Consumer Psychology, 3(1), 51.
Fiore, Ann Marie, Jihyun Kim, and Hyun-Hwa Lee (2005), Effect of Image
Interactivity Technology on Consumer Responses Toward the Online
Retailer, Journal of Interactive Marketing, 19(3), 3853.
Gallagher, Katheine, K. Dale Foster, and Jeffrey Parsons (2001), The Medium
Is Not the Message: Advertising Effectiveness and Content Evaluation in
Print and on the Web, Journal of Advertising Research, 41(4), 57.
Hatcher, Larry (1994), A Step-by-Step Approach to Using the SAS System
for Factor Analysis and Structural Equation Modeling. Cary, NC: SAS
Institute.

B.J. Calder et al. / Journal of Interactive Marketing 23 (2009) 321331

Hoffman, Donna and Thomas Novak (2009), Flow Online: Lessons Learned
and Future Prospects, Journal of Interactive Marketing, 23(1).
Hupfer, Maureen and Alex Grey (2005), Getting Something for Nothing: The
Impact of a Sample Offer and User Mode on Banner Ad Response, Journal
of Interactive Advertising, 6(1), http://jiad.org/article74.
Jarvis, Cheryl Burke, Scott Mackenzie, and Philip Podsakoff (2003), A Critical
Review of Construct Indicators and Measurement Model Misspecification in
Marketing and Consumer Research, Journal of Consumer Research, 30,
199218.
Malthouse, Edward C., Bobby J. Calder, and A.C. Tamhane (2007), The
Effects of Media Context Experience on Advertising Effectiveness,
Journal of Advertising, 36(6), 7.
Marc, Marcel (1966), Using Readership Quality in Magazine Selection,
Journal of Advertising Research, 6(4), 913.
Mathwick, Charla (2002), Understanding the Online Consumer: A Typology of
Online Relational Norms and Behavior, Journal of Interactive Marketing,
16(1), 40.
McQuail, Denis (1983), Mass Communication Theory, an Introduction.
London: Sage Publications.
Nambisan, Ratish and Robert Baron (2007), Interactions in Virtual Customer
Environments: Implications for Product Support and Customer Relationship
Management, Journal of Interactive Marketing, 21(2), 4262.
Nicovich, S.G. (2005), The Effect of Involvement on Ad Judgment in a Video
Game Environment: The Mediating Role of Presence, Journal of
Interactive Advertising, 6(1).
Pavlou, Paul and David Steward (2000), Measuring the Effects and
Effectiveness of Interactive Advertising: A Research Agenda, Journal of
Interactive Advertising, 1(1), 5.
Prahalad, C.K. and Venkat Ramaswamy (2004), Co-Creation Experiences: The
Next Practice in Value Creation, Journal of Interactive Marketing, 18(3),
514.

331

Rappaport, Stephen (2007), Lessons from Online Practice: New Advertising

Models, Journal of Advertising Research, 47(2), 135.
Ruggiero, Thomas (2000), Uses and Gratifications Theory in the 21st
Century, Mass Communication and Society, 3(1), 337.
Sawhney, Mohanbir, Gianmario Verona, and Emanuela Prandelli (2005),
Collaborating to Create: the Internet as a Platform for Customer Engagement
in Product Innovation, Journal of Interactive Marketing, 19(4), 417.
Sen, Shahana and Dawn Lerman (2007), Why are you telling me this? An
Examination into Negative Consumer Reviews on the Web, Journal of
Interactive Marketing, 21(4), 7694.
Shankar, Venkatesh and Sridhar Balasubramanian (2009), Mobile Marketing:
A Synthesis and Prognosis, Journal of Interactive Marketing, 23(2),
11829.
and Marie Hollinger (2007), Online and Mobile Advertising: Current
Scenario, Emerging Trends, and Future Directions, MSI Report No. 07206, Marketing Science Institute.
Sternthal, Brian, Alice Tybout, and Bobby Calder (1987), Confirmatory Versus
Comparative Approaches to Theory Testing, Journal of Consumer
Research, 14(114).
Thorbjrnsen, Helge, Magne Supphellen, Herbjrn Nysveen, and Per Egil
Pederson (2002), Building Brand Relationships Online: A Comparison of
Two Interactive Applications, Journal of Interactive Marketing, 16(3), 17.
Tremayne, Mark (2005), Lessons Learned from Experiments with Interactivity
on the Web, Journal of Interactive Advertising, 5(2), http://www.jiad.org/
article62.
Wang, Alex (2006), Advertising Engagement: A Driver of Message
Involvement on Message Effects, Journal of Advertising Research, 46(4),
355.
Winer, Russell (2009), New Communications Approaches in Marketing:
Issues and Research Directions, Journal of Interactive Marketing, 23(2),
10817.