Psychologically Motivated Techniques for Emotional Sound in

Computer Games
Inger Ekman, inger.ekman@tml.hut.fi
Department of Media Technology,
Helsinki University of Technology
P.O.Box 5400
FIN-02015 HUT

Abstract. One main function of sound in games is to create and enhance emotional impact. The expressive model for game sound
has its tradition in sound design for linear audiovisual media: animation and cinema. Current theories on emotional responses to
fiction are mainly concerned with linear medial, and only partly applicable to interactive systems like games. The interactivity
inherent to games introduces new requirements for sound design, and suggests a break in perception compared with linear media.
This work reviews work on emotional responses to fiction and applies them to the area of game sound. The synthesis is
interdisciplinary, combining information and insights from a number of fields, including psychology of emotion, film sound theory,
experimental research on music perception and philosophy. The paper identifies two competing frameworks for explaining fictional
emotions, with specific requirements, and signature techniques for sound design. The role of sound is examined in both cases. The
result is a psychologically motivated theory of sound perception capable of explaining the emotional impact of sound in film, as well
as identifying the similarities and difference in emotional sound design for these two media.

question we are concerned with here is how to tackle all these

1 Introduction issues while maintaining the emotional functions of sound.
The importance of sound for establishing mood in
Surprisingly little academic investigations into sound have
audiovisual media is well recognized. It should come as no
addressed this issue. Also, while there are many books on
surprise that viewing a movie without sound can strip it of its
game sound design, the majority of literature deals either
emotional impact, making the events pictured on screen
with the technological tools for creating the sounds or
seemingly distant and of little relevance. The potency of
managing the overall process of sound design e.g. [2][25].
sound is also rather well exploited at least in contemporary
The best foundations for game sound design decisions can be
gaming. The model for sound design in these games is often
found in the fragmental accounts written by designers
that of film sound; game sounds should be “Cinematic” [3] or
[22][3][13], sharing glimpses of their intuition and
“Bigger than life” [22].
experience in the craft through audio post mortems and
project descriptions.
However, it is a well known fact that games differ from film.
Also sound design in games is different from film sound on
Underlying many successful design decisions is an implicit
several points. An obvious disparity resides in the technical
understanding of what kind of sounds and actions create
realization of game sound compared to film sound. For one,
emotional impact. The aim of this study is to bring together
the expected technical setups for game and film consumption
various sources of information, to contribute to an explicated
are of a vastly different nature. Technical aspects are,
version of this knowledge. Such knowledge may ultimately
however, outside the scope of this paper. Instead, the focus is
aid the design process, and it should be of benefit to
on the special representational requirements that games, as
researchers working to understand the phenomenon of game
interactive systems, put on the sound design as compared to
sound. To this purpose the paper will jointly address two
those within static linear media. Interactivity creates a
close questions, namely why and how sound works to create
fundamental differentiation in what constitutes as the sound
emotion. Asking why looks for the reason behind having
design for film, and games, respectively. This is to say that
emotional reactions to sound emanating from fictional worlds
the final mix for a film differs by nature from the end result
in the first place. Answering this question requires looking at
of game sound design, usually a set of programmatically
the relationship between player and game, and the role of
expressed rules for combining/manipulating sound samples
sound in shaping this relation. The second question, how,
as a player interacts with the system. While technology
deals with the mechanisms underlying emotional reactions to
dictates the alternatives on both sides, the distinction between
sound, seeking cognitive and biological explanations to the
the two is present by nature of film vs. game, and exists
emotional experiences of a certain sound used in a certain
independently of technological issues. This aspect is the
context. Finally, a preliminary synthesis of these findings
focus of this investigation.
sketches out some practical sound design implications for
emotional impact in film and games, respectively.
These are the challenges game sound has to conform to: 1)
Whereas film sound is written to a fixed set of actions, seen
This work serves several purposes. First my hope is that an
through predefined points of view, game sounds must support understanding of the underlying processes of sound will
alternative paths and viewpoints through the story. 2) Film
provide useful concepts for practical work, by bringing to
sound is written to sequences of actions with known light the alternative possibilities and approaches for
durations. Game sounds must span action sequences of
impacting the player experience. This view is in no way
flexible temporal duration. 3) Games ask players to become attempting to challenge the value of intuition to design, nor to
active and play. Hence, sounds must support action, respond
explain away or trivialize the talent that goes into quality
to player control and often survive high repetitiveness. The sound design. On the contrary, explanations of why and how
masterful sound designs influence emotion are as much a
celebration of the skills that went into designing them in the
first place as tools for shaping new experiences in the future.
I am also positive that a bit of clearly enunciated arguments
for why sound holds so much power can be of use, especially
in defending sound when it has to compete for attention (or
budget) with visual effects. Finally, this paper links into the
academic discussion on sound design, and provides a
synthesis of relevant literature from several vastly different
fields. The topic of sound design is still advanced in many
separate camps, partly because it is interdisciplinary in nature
and that requires tapping into so many fields all at once. The
goal in this paper is not to provide an exhaustive review.
Rather it offers a selection of findings from different sources,
suggesting both where future connections may be found and
directions how some of them may be explored further.
The structure of the paper is as follows: Chapter 2 discusses
the nature of emotion in general, and looks at emotion theory
applied to film and games. Chapter 3 considers the processes
wherein film sounds influence emotional response and
consider several sources of emotion in sound. Chapter 4
offers a summary on emotional effect of sound and compares
emotional sound in film with sound in the interactive context
of games.
2 Emotions and Emotional Responses in
Film and Games
In psychology literature, whole chapters have been written
about defining emotions. Perhaps it is due to the subjective
nature of emotional experiences, as well as culturally
negotiated nature of what emotions are suitable and for
whom, that it is hard to pin down precisely what constitutes
an emotion. Intuitively, however, the struggle to define
emotion seems puzzling. Most of us would seem to know, by
intuition, what sort of experience the word emotion refers to.
For the purpose of this text, let us though consider briefly
what it implies. Oatley and Jenkins [20] 96] mention the
following three defining features:
- Emotion is usually caused by a person (consciously or
unconsciously) evaluating an event of some
significance in relation to a goal or situation.
- The core of an emotion is a readiness to act, and
emotion influences actions by preparing for certain
types of actions and causing a sense of urgency.
- An emotion is usually experienced as a distinctive type
of mental state, sometimes accompanied or followed by
bodily changes, expressions, actions.
Emotions are thus evaluations of a specific (visceral and
urgent) type, which signal events of critical importance and
relevance to the perceiver. They provide the perceiver with
an evaluation of the event, producing (typically) either a
positive or negative sensation. This sensation is referred to as
valence. They also produce a sense of urgency. Generally,
the activation part of emotions is called arousal.
2.1 Fictional Emotions?
From a biological viewpoint the function of emotions seems
clear. They provide the foundations for successful behaviour
in an environment with promises and perils. Emotional
relations to film and games, however, is an example of
emotional relation to fictional events. This has been a very
controversial subject, and arguments go back and forth about
whether all these emotional reactions, which have their cause
in completely fictive events, are even real emotions at all.
And if they are real, what is their cause? To shed some light
on these questions, and as an inventory of the tools available
for looking at the effects of sound, let us turn for a moment to
psychology. What psychological support exists for emotional
reaction to fiction?
The simplest objection to calling emotional responses to
fiction real is related to the reality of the events underlying
these responses. Most emotion theories require that evaluated
events should be of real significance to the evaluating
subject. This is precisely the concern underlying Tan’s
argument [27]. Investigating the emotional responses in film
viewing, he bases the validity of his model in Frijda's [9]
theory of emotions and discusses in length the compatibility
of his theory with Frijda's basic tenets. In short, he argues
that emotional responses can arise to fictive events as long as
they are perceived as apparent reality. This is achieved
mainly by the diegetic effect, a cognitive and perceptual
illusion in the viewer' head. The illusion is maintained with
several cinematic techniques, usually by dissolving and
eradicating the medium. The witness position, Tan argues, is
one of the very means in which the cinema excuses the
viewer's passivity and explains, in keeping with the diegetic
effect, the viewer's ability to see it all.
Another way to make room for fictional emotions is to allow
emotional responses not only to real events, but to the
fictions of our imagination. Damasio does just so: he calls the
latter type as-if emotions, and describes in detail the
processes in which the mind simulates affective responses
[7]. According to Damasio, fundamentally affective
processes are part of all rational decision-making, and
function as a means of making decisions – we reason by
simulations on how the outcome might feel. From this point
of view, fiction would be just another way of running
simulations.
2.2 Emotional Responses to Film
There are two dominant approaches to understanding how
film elicits emotional responses. One approach is based on
Freudian psychoanalysis, invoking as a key concept the
Lacanian mirror stage, which “typifies an essential libidinal
relationship with the body-image” [14]. This stage is called
forth by identifying in film, a process that links the self with
the film experience through subconscious mental responses
(the basis of which is rooted in early childhood and
formations of the Ego). An alternative approach explains the
emotional reaction to film through cognitive appraisal of
portrayed fictional events. Instead of identifying
subconsciously, the viewer responds emotionally because of
their cognitive investment in the fictional events. The
viewers' emotional responses are related to motivational
processes [27][9][16].
According to the appraisal theory, the key to understanding
emotions is to understand the role fictive events play to the
viewer. Tan identifies two types of emotion at play in the
viewing situation: fiction emotions (F emotions, emphatic
emotions) and artefact emotions (A emotions, non-empathic
emotions). Empathetic (F) emotions are emotional reactions
to the story; they require empathy with the main character.
Non-empathetic artefact emotions are linked to sensory
pleasures – the viewer enjoys the good looks of the
protagonist or the beautiful scenery. [27]
Non-empathetic emotions require no appreciation of events.
The mechanism by which emphatic emotions are created is
more complicated. Tan suggest the film viewer connects to
the fiction is through a witness position. This position of
agreed-upon inaction (even in the most anxious of moments,
we are content to just watch the film) dictates the relations
between viewer and film events. Our commitment to the
protagonist's cause gives rise to emphatic relations. However,
the inactivity dictated by the viewing position also creates
tensions, such as worry or frustration when we know more
than the protagonist, but cannot change the situation. On the
other hand, inactivity may allow us to more deeply
empathize, since no action is asked of us.
2.3 Emotions in Games
The interaction in games usually has the player controlling a
main character (or other objects in the game). Whereas
cognitive appraisal of game events lends itself to an
emotional impact, a key question in games is how the
narrative reality is maintained. Especially, games remove the
passive viewing position, previously suggested a main
contributor of empathic effect in film. Also, the typical
structure of games – featuring repetitive action, often only
loosely bound together by a story – is challenging to
empathetic emotion.
On the surface, it would seem that the active nature of
players in games break the passivity that Tan proposes so
significant in relation to justification of emotions. However,
it turns out the activity of games lends itself rather well to the
purposes of explaining emotion and the cognitive appraisal
model has also been invoked in relation to computer games
and emotions [21][15]. Activity, while challenging the
passive ride of empathetic emotion, provides an alternative
frame in which to evaluate actions.
Perron [21] explicitly works from Tan’s theory to add
gameplay emotions (he calls them G emotions), that arise
from the cognitive appraisal of game situations. Perron’s G
emotions arise just because, in games, the player is invested
in acting. Emotional evaluation is fuelled by care for the
progress of the game. Lankoski [15] offers a detailed
breakdown of emotions within the gaming context,
demonstrating how different goal evaluations give rise to
basic emotions and their combinations.
In games with a protagonist, gameplay emotions may equal
care for the protagonist, but this care is essentially different
from empathetic emotion: From the perspective of gameplay,
the protagonist is a means, a tool, for playing the game (and
achieving the personal goal of completing the task).
Whereas the protagonist can also provide a vessel for
empathetic emotions, to some extent, the two frameworks for
emotional evaluation - empathetic and gameplay - are
competing. As Perron [21] notes, even in games, both the
main story line as well as individual plot elements and
narrative turning points are indeed furthered from time to
time through filmic means, using stills, predetermined
animation sequences/ dialogue or cut scenes. During these
moments, the player is stripped of control and, effectively,
reduced into a witness position. Empathetic emotion comes
with loss of activity. A similar point is made by Lankoski
[15] when he suggests the empathetic capability of the player
is inversely related to the cognitive challenge of action - in
the heat of a battle, there is little time to ponder the
protagonist's feelings.
3 Roles of Sound in Emotional Appraisal
The appraisal theory would seem to give little explicit
explanation of how sound contributes to emotion. The
question is to what extent sound influences the process of
appraisal. It seems clear that for both film and games, sound
is part of the process enforcing the inherent emotionality of
events as appraised within each framework. However, the
practical and functional approach to sound design is different.
In film, sound makes actions seem real and consequential for
the viewer, both factors mentioned as prerequisite to
empathetic or F emotions [27]. Within this category of
emotions, two specific cases can be identified for sound.
One goal of film (and game) production is to make things on
screen seem real. A silent two-dimensional representation
has limited apparent reality on its own, but adding sound
helps perceiving the pictures on screen as physical bodies.
Especially important in this aspect is the use of Foley, or
synchronized sound effects.
As another case of supporting realism, an integral role for
sound at least in classical (Hollywood) film production has
been to hide the medium. An example of such is the common
practise of continuing sounds over cuts, making them less
apparently noticeable. Many other conventions of film sound
also contribute to the invisible medium effect, either directly
or indirectly. Thus, a role of sound is to create a sense of
immediacy.
In the context of games, the significant emotional investment
goes into advancing goal-related progress. Most current
games rely on both auditory and visual content to represent
the game world. Crucially, in the context of interaction,
sound comes to take on a new task: facilitator and
confirmatory of action. Historically, technical considerations
have long dictated tools for sound expression in interactive
context, which has forced (and allowed) game sound to
deviate from some film sound conventions. The role for
sound in games is at least partly dictated by a functional
approach and a sound's impact defined by its capability of
supporting and facilitating gameplay.
Finally, in discussing empathic emotions we mentioned that
some appraisals require no cognitive investment in the story,
but are linked purely to sensory pleasures. Interestingly, there
is no explicit mention of negative affect in the context of
artefact emotions. The same artefact emotions are present in
games as well, where part of the game can be enjoyed (or
not) apart from aspects of gameplay-related progress.
Soundtrack CD sales gives strong evidence that artefact
emotions functions with regards to both film and game
music. Beautiful pieces of music obviously give themselves
to being appreciated as such, disregarding of whether the
viewer is attending to the story. Further, it is motivated to
broaden the spectrum of artefact emotions to include also
negative effects. With thought to how visual material is used
for shock effects (e.g. displaying blood and entrails), it is
easy to imagine a similar process in which unpleasant
sounds, regardless of story, could produce negative affect.
Both in film and games, sound provokes sensory pleasure
and displeasure.
3.1 The Realism Fallacy
The above categories identify the main roles of sound in
creating and steering the emotional experience. However, as
we shall soon see, there are unresolved contradictions hiding Inferences” [31]. This view is further fortified by subsequent
within these categories. Namely, the effects within the two results from experimental psychology, leading researchers to
latter categories of sound (gameplay and non-empathetic) suggest that there exists such a thing as unconscious emotion
seem to contradict the traditional sound design goal of [30]. For example, Öhman [32] has demonstrated fear
narrative realism. reactions in people who were presented with spider and snake
pictures subconsciously; that is, people became frightened of
Appraisal theory of emotion holds that empathetic emotional pictures they never even realized they had seen.
processing and value judgement is guided by conscious
attention to a story, and this cognitive investment is In light of unconscious value judgements, it is easy to read
heightened in the perceived realism of portrayal. On the other Tan's non-empathic emotions are but one example of these:
hand, it allows emotional experiences that arise from the sounds of the film are invoking emotion by nature of their
appreciation of the artefact: the pictures and sounds of the perceptual properties, unrelated to story at hand. This also
film/game as such. Similar appreciation is present in games, a opens up a way to understand and predict what would result
medium where technological artistry is elaborately in a positive (or negative, for that matter) value judgement.
showcased, often even used in promotion. One especially interesting factor is the importance of
familiarity and perceptual fluency for eliciting positive value
Tan’s view is that artefact emotions detach the viewer from judgements [23]. By this account, beauty is defined by the
the story, drawing attention away from the narrative towards ease in which a stimulus can be processed.
the film as artefact, thus making the actions within the
narrative less consequential for the viewer. This position 3.3 Misattribution and Making Sense of
raises a complicated question, namely how to interpret such Emotions
sounds that appear to transgress the borders of realism,
Affective responses include paraphernalia of bodily
despite experientially supporting narrative.
responses (pounding heart, sweaty palms) and may also bring
with them a certain action tendency (e.g. fight-or-flight). In
For example, while discussing the effect of sentiment, Tan
fact, it has been suggested that one possible function in which
and Frijda [28] 62] mention sound, especially orchestral
we do consciously attend to our affective processes is by
music, as one possible source for the awe-inspiring. Awe
appreciation of the abrupt changes in the felt background
requires a sense of overwhelming power, and this role is
state, or what Russell [24] has called core affect. This change
partly to be played by sound. The effect, according to Tan
would lead us seeking for a cause of our altered state, leading
and Frijda, is the emotional function of total submission, a
us via cognitive process to attribute the jolt to the most
feeling underlying e.g. crying. Sound is thus a tool in
plausible event in our environment.
portraying power, and heightening sentiment. The question
is, by which channel this non-empathetic heightening of
Whether or not we are willing to accept unconscious affect as
sentiment is capable of influencing the (empathetic)
the source of emotions, we can agree that when consciously
evaluation of narrative events.
attended to, emotions tend to have an object. Emotions are
evaluations of something. To be able to function properly, we
The problem of border transgression is most apparent in two
must be able to determine an object for our emotions,
sound conventions that would seem to deviate from the
something to be afraid of, or pleased by. This distinguishes
purpose of reality. In what seems like a blatant contradiction,
emotion from moods, which are long-term affective states
they invoke a sense of realism in highly unrealistic sounds.
without an object. However, and here comes the catch, the
One is the use of musical scoring. The other is the use of
events we cognitively allocate, as objects do not necessarily
sound effects that mismatch what is seen on screen. Similar
have to be the true cause of our initial affective response. In
breaches are omnipresent in games, as well, where sound
fact, when it comes to reasoning why we feel like we do, we
elements effortlessly transgress borders, allowing objects
are prone to make mis-attributions and erroneously
within the story world (diegesis) to refer to non-diegetic
appreciate our affects quite differently from their real causes
space, and vice versa [8][12].
even in everyday life.
The above concerns are about where to draw the line of
A classical example of misattribution is a study by Schachter
realism and about how emotional effects communicate across
and Singer [26]. They injected subjects with doses of
different categories of judgement. Upon closer scrutiny, both
adrenaline, a hormone associated with an excited body state.
questions generalize to the way non-empathetic affect
Depending on the situation that followed the injection,
influences other sources of emotion (empathetic or
subjects judged their aroused state as either anger or elation.
gameplay- related appraisal). To proceed further, we need an
It can be argued that in most cases, our appreciation of our
explanation on how non-empathetic emotions arise, and a
emotional state is at least partly determined by context.
way of predicting when and how artefact emotions lend
emotional meaning to other evaluative processes.
3.4 Misattribution and Emotion in Film and
Games
3.2 Unconscious Affective Processes
Misattribution is a process wherein the contextual appraisal
The theoretical frameworks dealt with above have considered
of perceived emotional ‘raw material’ lending emotional
affect by means of perceived experience. Nevertheless, it
meaning to an outside cause, irrelevant to that particular
seems many of the associations and effects of sound in both
emotional stir. Now consider a similar process at work during
film and games are working on an unconscious level.
film viewing or when playing a game, with music, sounds,
pictures, actions, all mingling to create emotional impacts. Is
Several findings state to the fact that at least some
it not probable, that at some point, the true causes of our
evaluations of stimuli are made precognitively. Zajonc was
feelings might remain oblivious to us? Is it not possible, that
among the first to point this out in his essay, famously
we, stirred by our passions, unwittingly, in deciphering the
entitled “Feeling and Thinking: Preferences need no
cause of our emotions take them to be caused by whatever
the film serves to us on a silver(screen) plate? Could it be
that we just happen to attend to a game event, and assume our
emotions are caused by that event, when in fact they are not?
This is, essentially, what Annabel Cohen proposes. Cohen
[5][6] has dealt extensively with the difficult question why,
and how, something as obviously constructed as the film
score, does not completely destroy the sense of realism in a
film. On the contrary, as many composers will confirm, a
carefully chosen (or composed) piece of music will actually
heighten the sense of reality in a film. Music also seems to
lend a great deal of emotion to events, in ways other than
proposed by the cognitive appraisal theory. Cohen's answer
lies in a congruence-associationist model of film viewing
[18], whereby music focuses attention on those objects in the
film that are congruent with the sound. At the same time, the
conscious attention is directed away from non-associated
sounds, and attending suppressed for stimuli irrelevant for
ongoing cognitive processes.
The emotional impact comes from the fact that sounds, even
when unattended to, will nevertheless affect perception of
objects in the film. Cohen [6] highlights the importance of
temporal unity as a binding factor and predictor of which
parts of the sound will draw attention. She calls our attention
to animation and the technique called mickey-mousing,
whereby sound effects are replaced by short musical motifs.
Their temporal matching allows these music snippets to
replace the original sounds of the events, at the same imbuing
both the events, and the objects part of the action, with
specific characteristics.
The account on musical meaning in film provides an equally
useful tool for approaching the question about other film
sounds as well. Applied to object sound, the theory suddenly
appears much less mysterious: Consider how we find out
properties of objects in real life. What we do is handle the
object – tap it, stroke it, bang it against something. By
perceiving synchronic sounds, we find out the normal sound
of a chair, a balloon or a mandolin. Now, turn that process
around and we have precisely what sounds do in a film (and,
to a great extent in games as well): now the temporal unity of
event and sound defines the object through what sound it
makes. Longer chains of events, if temporally matched, cause
similar perceptions. When approached from this angle, it is
not so odd that sounds in fiction may deviate somewhat from
their real life counterparts without seeming false or
unrealistic. What may seem surprising is that a whole sum of
temporally congruent sounds may become involved in the
same process, from simple Foley through more elaborate
layers of sound effects all the way to music.
3.5 Where do The Emotions Come From?
So far we have established that unconscious emotional
processes may ‘contaminate’ temporally congruent events
through misattribution and shown how this may influence
perception of events in film. The big question remains, where
do the emotions come from?
The most frequently researched category of emotional sound
is music. Huron [10] describes musical emotions in terms of
fulfilling expectations: the interplay between anticipated and
sounded music progression creates patterns of dynamic
tension and relaxation. Musical expectations arise from
several sources, most of them cultural, but according to some
studies at least part of the functions of musical meaning
appear to be universal [19].
The existence of universal function of musical emotion
suggests there may be other sources of emotion humans draw
from in their interpretations of music. The obvious case is
that music invokes memories and connotations awaken by
that music. However, those would not be universal, even less
so than culturally learned expectations. Van Leuwen [17]
turns to the human body, suggesting that the most primitive,
and also a common link between sound and emotion for all
humans, is the perception of our own bodies. Especially the
vocal system sets a reference point through simultaneous
experience of how it feels and what it requires to produce a
certain sound.
Another suggestion is that evaluation of some sounds has
biological motivation. This appears to be the fact with the
startle response (the phenomenon in which we jump to
someone shouting ‘boo’ at us), which aside from providing
for pranks also makes us more alert for dangers and
automatically directs our attention toward potentially harmful
events. However, there may well be other ways in which our
perception of sound is evolutionary determined. For example
Huron [11] suggests that the perceived cuteness of sounds
may be an evolutionary adaptation that promotes parenting.
Value judgements (which are the raw stuff of emotion) seem
to be going on even on the lowest level of perception. A
classical example within music (and other perceptual)
research is the mere exposure effect, wherein a stimulus is
judged as likeable merely as the function of familiarity.
Investigations into a phenomenon called perceptual fluency
suggest that emotional processes are influenced by the very
ease of processing [23]. These findings would imply that
such things as differences in perceptual clarity (think signal-
to-noise ratio) of audio influence the emotional impact of
sound, such as perceived beauty or likeability.
The critical requirement for musical expectations to arise is
that it is attended to as music. Further, there appears to be
boundaries in our listening schemes that separate different
styles of listening. Notably, musical listening, in which
sounds are perceived as sounds, is not the only form of
attending to events. An illustrative deviation from this frame
are listening styles provoked by compositional techniques
invoking other listening styles, like musique concrète, where
the use of real world sounds provokes listening, not at sounds
and patterns, but for causes – Chion aptly refers to this as
causal listening [4]. This is a special case of music, perhaps
seldom used in film, but appearing more and more in games.
In these cases, the framework of listening is perhaps more
determined by evolutionary and low-level perceptual
processes of meaning-making than musical listening modes.
3.6 Realism Revisited
We should now attempt a new understanding of realism.
Within fiction, realism is not an absolute, but a nominator for
a certain level of fit, an apparent realism or credibility. On
the narrative level, realism allows taking the story seriously
enough to allow emotions of empathetic quality. Good fit is
determined by whether the sound is credible (or illustrative)
of a certain sound source [1], pp 190]. When the Foley artist
(the person responsible for creating sounds to on-screen
events) smashes pumpkins in his studio, he does so in order
to produce such sounds with good fit with on screen events.
Many times, the sounds produced have little to do with the
actual event seen on the film screen – indeed, often non-
realistic sounds are purposefully used to make the action
sound better. It is, for example, recognized that walking on
cornstarch sounds much 'more real' on film than the actual
sounds of walking on snow.
A possible explanation underlying the perceived realism of
some Foley sounds is the notion of prototypicality. A
prototype is an object that inhabits central perceptual
characteristics of a given category. Prototypes do not
necessarily exist in reality, they are mental constructs of our
perceptual system. The prototypical chair is the average of all
chair perceptions of your brain, and by definition, it will be
the 'chairest' chair of them all. Experimental psychology has
established that people perceive prototypes as more easily
recognized [27], also more beautiful, and trustworthy [23]
than other category members.
Similarly, narrative reality determines how sound behaves
within the diegesis, and how the source sounds should sound
when listened to from different Points of Audition1, such as
listening behind a wall or under water. At the core, then, also
immediacy is but one way of creating a sense of realism. In
film, it serves to reinforce the witness position, being present
but out of control. Tan [27], pp 25] considers this in his
analysis of the camera point of view, mentioning how even in
first person view the camera is often a bit off, making space
for someone to ‘look over the shoulder’. In the case of sound,
the heightened, focussed sound including over-clear dialogue
can be considered realistic if we view it as a portrayal not of
the scene, but of the experience of listening to the scene.
Consider this: while our environment usually contains a
multitude of sounds, we only attend to a select few at a time.
We are also exemplary at picking out and following these
sounds. A person with normal hearing has no difficulty in
following a single conversation in a room filled with people,
the phenomenon so aptly named the ‘cocktail-party’ effect.
Thus, what would seem presented as the films sounds is not
the scene itself from a given point in space, but the scene as
heard if listened to by attentive ears.
The narrative realism of a sound is thus not in faithful
reproduction of sound sources, nor of their environments.
The apparent realism of a sound in the context of narrative is
defined by how representative a sound is of a certain event.
Sounds that are highly representative have good narrative
fit. High narrative fit supports empathetic emotion.
Importantly, however, the evaluation of sounds spans several
layers. Below the level of narrative meaning are (partly)
unconscious processes whereby sounds are judged
emotionally. For example, a sound can have good fit
narratively, but poor legibility, because the signal-to-noise
ratio is so high. Importantly, breaches in this level are
disruptive to the perception of sound. We have seen that
perceptual fluency is also capable of influencing affective
evaluations [23]. Thus, as with the narrative fit, unconscious
processing of sound influences emotional judgements. These
affects are unrelated to the narrative content of the sound, but
tap into the notion of artefact emotions. Depending on their
nature, they can cause pleasure or displeasure, which can
then be attributed to other temporally congruent events.
1 http://www.gamasutra.com/features/20070322/bridgett_
Similar to Point of View in camera techniques, but
using sound.
Finally, in interactive systems interpretations of sound take
on a new role, conveying functional information [29]. By this
task, sounds are evaluated on a third level, in how well they
serve a functional value, Jørgensen [12], pp 49] refers to this
as a sounds functional fidelity. On this stage, emotional
evaluations are no longer determined only by the sound itself,
but by the utility of a sound for the higher goal of performing
goal-related actions. The value of functional sound depends
on how the functional aspect supports game progress, the
utility of sound. The utility of sound is connected to goal-
related cognitive evaluations. High utility enforces gameplay
emotion.
4 Comparison of Emotional Sound in Film
and Games
The cognitive appraisal framework provides two alternatives
for emotional relation to fictive events: the passive witness
position allows empathetic emotion, while the active player
draws emotional meaning from goal-related evaluation.
These two frameworks ask the viewer/gamer to take on a
different attitudes towards the fiction and appear to be
competing. They also rely on different strategies for sound
design.
For film sound, the effort is usually on heightening narrative
reality. This is achieved through detailed attention to
narrative fit, often striving for high apparent reality. Focus is
on advancing the narrative, heightening and clarifying those
specific actions that are necessary for following the story’s
progress (usually the top priority is on dialogue).
For games, the focus point is different, as games have to
support player action. In games, the task of many sounds is
primarily to provide feedback about actions. Hence, narrative
fit is often sacrificed for utility. To the extent that auditory
cues are used to guide actions, they are treated with utmost
respect for legibility. For example, even in the case of
instructions with diegetic source (a non-player character,
voice mail, etc.) it is common that auditory instructions
remain heard even if the character runs away from their
diegetic source.
An interesting avenue for sound design in games is to shift
focus from music to the emotional impacts of Foley and
sound effects. A possible alternative for emotionality in
games is in environmental sounds, which is already used in
many games, where ambient sounds are beautifully merged
with musically suggestive elements and event sounds into a
sonic landscape in the spirit of musique concrète. However,
for this approach to be systematically explored, there is need
for better understanding of how everyday sounds influence
emotions. In these investigations, theories of unconscious
emotion may prove especially informative.
References
[1] Bordwell, D. and Thompson, K. 1985. Fundamental
Aesthetics of Sound in the Cinema. In Weis, e. and
Belton, J. (eds.) Film Sound, Theory and Practice.
Columbia University Press. 181-199.
[2] Brandon, A. 2005. Audio For Games. Planning, Process
and Production. New Riders.
[3] Bridgett, B. 2007. Audio Postmortem: Scarface: The
World is Yours. Available at
pfv.htm [Accessed 29.5.2008]
[4] Chion, M. 1994. Audio-Vision. Sound on Screen.
(Translation by Claudia Gorbman). Columbia University
Press.
[5] Cohen, A. 1990. Understanding musical soundtracks.
Empirical Studies of the Arts 8, 111-124.
[6] Cohen, A. 2001. Music as the source of Emotion in
Film. In Juslin, P. and Sloboda, J (eds.) Music and
Emotion. Oxford University Press. 249-272.
[7] Damasio, A. 2005. Descartes' Error: Emotion, Reason,
and the Human Brain. Penguin, paperback reprint
(1994).
[8] Ekman, I. 2005. Understanding Sound Effects in
Computer Games In Proc. Digital Arts and Cultures
2005, Kopenhagen, Denmark.
[9] Frijda, N. H. 1986. The emotions. Cambridge University.
[10] Huron, D. 2007. Sweet Anticipation: Music and the
Psychology of Expectation. MIT press. Paperback
reprint (2006).
[11] Huron, D. 2005. The Plural Pleasures of Music. Proc.
2004 Music and Music Science Conference. Kungliga
Musikhögskolan & KTH (Royal Institute of
Technology), 1-13.
[12] Jørgensen, K. 2007. ‘What are Those Grunts and Growls
Over There?’ Computer Game Audio and Player Action.
Ph.D. dissertation, Copenhagen University.
[13] Kutay, S. 2006. Bigger Than Big: The Game Audio
Explosion. A Guide to Great Game Sound. Available at
http://www.gamedev.net/reference/articles/article2317.a
sp [Accessed 29.5.2008]
[14] Lacan,J. 1951. Some reflections on the ego. (fut lu par
Lacan à la British Psycho-Analytical Society le 2 mai
1951) Available at: http://aejcpp.free.fr/lacan/1951-05-
02.htm [Accessed 29.5.2008]
[15] Lankoski, P. 2007. Goals, affects, and empathy in
games. Paper presented at Philosophy of Computer
Games, Reggio Emilia, Italy. Available at:
http://www.mlab.uiah.fi/~plankosk/blog/?p=53.
[Accessed 29.5.2008]
[16] Lazarus, R. 1991. Emotion and adaptation. Oxford
University Press.
[17] Leewen, T. van. 1999. Speech, Music, Sound.
Macmillan.
[18] Marshall, S. and Cohen, A. 1988. Effects of Musical
Soundtracks on Attitudes toward Animated Geometric
Figures. Music Perception 6, 95-112.
[19] Narmour, E. 1990. The analysis and cognition of basic
melodic structures: The implication-realization model.
University of Chicago Press.
[20] Oatley, K. and Jenkins, J. 1996. Understanding Emotion.
Blackwell Publishing.
[21] Perron, B. 2005. A Cognitive Psychological Approach to
Gameplay Emotions. Proc. DiGRA 2005 Conference:
Changig Views – Worlds in Play.
[22] Prince, B. Tricks and Techniques for Sound Effect
Design. Computer Game Developers Conference 1996.
Available at
http://www.gamasutra.com/features/sound_and_music/0
81997/sound_effect.htm [Accessed August 20, 2008.]
[23] Reber, R.; Schwarz, N. and Winielman, P. 2004.
Processing Fluency and Aesthetic Pleasure: Is Beauty in
the Perceiver's Processing Experience? Personality and
Social Psychology Review, 8 (4). 364-382.
[24] Russell, J. 2003. Core Affect and the Psychological
Construction of Emotion. Psychological Review 110 (1),
145-172.
[25] Sanger, G. 2003. The Fat Man on Game Audio: Tasty
Morsels of Sonic Goodness. New Riders.
[26] Schachter, S., & Singer, J. 1962. Cognitive, Social, and
Physiological Determinants of Emotional State.
Psychological Review, 69, 379-399.
[27] Tan, E. 1994. Film-induced affect as a witness emotion.
Poetics 23, 7-32.
[28] Tan, E. and Frijda, N. 1999. Sentiment in Film Viewing.
In Plantinga, C. and Smith, G. (eds.) Passionate Views.
Film, Cognition, and Emotion. Johns Hopkins. 48-64.
[29] Tuuri, K.; Mustonen, M.-S.; Pirhonen, A. 2007. "Same
sound - Different meanings: A novel scheme for modes
of listening." Proc. AudioMostly 2007, Ilmenau,
Germany.
[30] Winkielman, P. and Berridge, K. 2004. Unconscious
Emotion. Current Directions in Psychological Science,
13 (3). 120-123.
[31] Zajonc, R. B. 1980. Feeling and Thinking: Preferences
Need No Inferences. American Psychologist, 35, 151-
175.
[32] Öhman, A. 2005. The role of the amygdala in human
fear: Automatic detection of threat.
Psychoneuroendocrinology 30, 953-958.