Student: Alex van Oosterzee

Student nr.: 3810089

Program: MA New Media and Digital Culture
Course: Digital Music Cultures
Lecturer: Drs. S. Bannier
How Audio in Videogames Influences Gameplay
1




Inhoud
Introduction ................................................................................................................................ 2
Part 1 | The Role of Game Audio ............................................................................................... 4
Part 2 | Assassins Creed 2 analysed ........................................................................................ 11
Conclusion ................................................................................................................................ 15
Literature .................................................................................................................................. 17



2

Introduction
In analysing videogames people often look at what the effect of visuals or a narrative in the
game does to its gameplay, unfortunately although game audio or game sound has greatly
developed over the last years there isnt a lot of research about this topic. According to Zach
Whalen the game music industry spends a large amount of money in the creation of music but
in the academic world game audio is hardly being researched (Whalen 2004).
Videogames are often examined using concepts and theories derived from film studies
literature. The term immersion is often linked to the persuasive character of videogames in
that videogames take up all the players attention and give them the feeling of being inside the
game. This paper will examine what the role of game audio in relation to gameplay is and
what functions game audio can have. In mediastudies the terms immersion and flow are often
linked to the ability of videogames to drag people into playing them and are dependent on
how well the game can be played. In Fundamental Components of the Gameplay Experience:
Analysing Immersion Laura Ermi and Frans Myr discuss immersion to be one of the most
important factors of gameplay and one of the main reasons why players get sucked into
videogames (Ermi & Myr 2005). Whilst their analysis of the phenomenon is broad and
contains the notion of flow theory, the role of audio in games doesnt get very clear. The same
goes for Gordon Callejas revision of the term immersion, although he mentions that it is
unwise to discount the power of sound he does not elaborate on why or how videogame audio
should be treated (Calleja 2007). Other descriptions of immersion show the same flaw in that
they do not take into account the role that audio plays in getting immersed or achieving a state
of flow.
In trying to understand the importance of audio in videogames I will examine what functions
audio in games have. I will focus on both diegetic and non-diegetic sound and will take a look
3

at Sander Huiberts IEZA-model. In doing this I want to formulate an answer to the question:
How does audio in videogames influence gameplay? As mentioned earlier the concepts of
flow and immersion will be closely considered in what the effects of in-game audio are.
After analysing the functions of game audio I will put my findings to the test by analysing the
video game Assassins Creed 2 (Ubisoft 2009). I will use the IEZA model as a basis and by
examining a quest in the game I will link all described concepts to the audio that is heard
while playing.

4

Part 1 | The Role of Game Audio
As a starting point in this paper I will create a theoretical framework that shows what the
different views on game audio are. I will focus on the relation to the in-game environment and
the players environment.
I will start from the notion that game audio can be seen as a physical event. Game audio
according to Nevinsound is as opposed to what is graphically represented on screen not a
simulation. It is the only true 3-dimensional physical event that happens during gameplay
(Nevinsound 2009). Another interesting argument Nevinsounds states is that we perceive
audio as a story through references, it is therefore a physical gate through which players
immerse into the virtual environment (Nevinsound 2009). In other words, when we hear a
sound we can interpret this by the reference we made to that sound in our head. In Audio and
Gameplay: An Analysis of PvP Battlegrounds in World of Warcraft Kristine Jrgensen
argues the same but names it slightly different. She talks about how the context in which we
hear a sound is of great importance. Jrgensen states that we cannot comprehend sound in
isolation but that the situation in which the sound is heard decides the interpretation of the
informative content of the sound signal (Jrgensen 2008). She even calls it an informative
system that gives the player intuitive information. Videogames according to Jrgensen can be
seen as audiovisual environments and many of the sounds in these environments correspond
to sounds in the real world. It can therefore be said that although the audio comes from a
virtual world it can be perceived as real.
When the boundaries between what is real and unreal or virtual seem to disappear we can
speak of immersion. Since there is a lot written about immersion I will first examine what
immersion is before I continue to look at the role audio plays in getting immersed. To not go
too far into the whole discussion about immersion, since that is not the focus of this paper, I
5

will just take a look at a small subset of definitions on the subject. Laura Ermi and Frans
Myr describe immersion by referring to Janet Murray as: Immersion can be defined as the
sensation of being surrounded by a completely other reality [] that takes over all of our
attention, our whole perceptual apparatus (Ermi & Myr 2005, p4). Ermi and Myr have
created what they call the Gameplay Experience Model which consists of three types of
immersion. The types are sensory, challenge based and imaginative immersion. All three
types aim at a different part of the game, in the first type the focus lies on the audiovisual
execution of games. Ermi en Myr argue that videogames have evolved into audiovisually
impressive, three-dimensional and stereophonic worlds that surround their players in a very
comprehensive manner (Ermi & Myr 2005, 7). The second type, challenge-based
immersion, is based on interaction and achieving a satisfying balance of challenges and
abilities. In the last type, imaginative immersion, the players get the chance to empathise with
the character and use their imagination. All three types of immersion as described by Ermi
and Myr focus on a different part of the gaming experience. However if we look at how
audio can contribute in these forms of immersion I argue that the role of audio is very high
since audio in modern day games is almost indispensable. Karen Collins argues that the role
of audio is the most present in the imaginative immersion. She claims that the interruption of
gameplay by a glitch or incoherence between what is represented on screen and heard can
lead to a detraction of getting or being immersed (Collins 2008).
When talking about immersion Sander Huiberts thesis is of great interest. As he states the
connection between audio and immersion is important but there has not been a lot of research
done on this topic (Huiberts 2010, 37). He gives an extensive summary on what is written
about immersion but what all descriptions of the phenomenon seem to have in common is that
they concentrate on the player, as in the player gets immersed in the game. Huiberts concludes
his summary on immersion by stating that there are three basic aspects to the phenomenon.
6

Immersion gives people the feeling of being transported in and surrounded by the game
world, absorption is a key part of defining immersion and immersion lets the player identify
with the character or game world (Huiberts 2010, 39). Absorption according to Huiberts is the
experiencing of a strong connection with the act of playing a game. Mark Grimshaw, Craig
Lindley and Lennart Nacke did research on the importance of sound in first-person shooter
games and concluded that the complete absence of sound while playing a game negatively
influenced the feeling of immersion (Grimshaw et al., 2008). Another interesting finding from
this study is that especially diegetic sound seems to be an enabling factor in experiencing
challenges and a state of flow.
The distinction between diegetic and non-diegetic sound is one that is made very often when
analysing sound in games. It is the difference between sound that originates from within the
game world, sound that the game character can hear, and sound that originates outside the
game world, often described as background music or sounds connected to menus (Van Tol &
Huiberts 2008). Kristine Jrgensen describes the diegetic level of sound as a communicative
situation between the virtual gameworld and the avatar. The non-diegetic level, although she
does not mention the term non-diegetic, is communicative situation between the game system
and the player (Jrgensen 2008). She gives another good explanation of the division between
the two types of sound by stating that the terms are used to separate elements that can be said
to be part of the depicted fictional world from elements that the fictional character cannot see
or hear and which should be considered non-existent in the fictional world (Jrgensen 2011,
78). An interesting argument Jrgensen makes is that there is communication that moves
across borders and stands between diegetic and non-diegetic, she calls this type of game audio
transdiegetic and describes transdiegetic sounds as sounds that seem in some respect to place
themselves somewhere between the two, either by being virtually extradiegetic sounds that
communicate to entities internal to the diegesis (typically the avatar, for example extradiegetic
7

music that alerts about upcoming dangers), or they are diegetic sounds that directly address
entities external to the diegesis (Jrgensen 2007, 3). The extradiegetic level of sound is
described as sound that comes from an external source (Jrgensen 2007) or as the narrators
level, a narrator who is not part of the game world but tells something about it (Prince 2003).
In combining the diegetic and non-diegetic sound the physical event, as Nevinsound
described, can be created. It is the player who can perceive the difference between the two
forms of sound but combined it can be seen as one thing, game audio. Sound in games can not
only be divided into diegetic and non-diegetic sound, since videogames are an interactive
medium the sound that is used in these games can be described as non-linear and adaptive
(Carter and Worth 2009; Collins 2007). The sounds get non-linear or adaptive since they can
be influenced by the players actions. Although there are cut scenes in a lot of videogames
that cant be skipped without resetting or shutting down the game, which makes the sound
linear, most sounds in videogames are dubbed as non-linear or adaptive (Collins 2007).
Since it is clear that audio can contribute in getting and staying immersed it is important to
somehow classify different types of game audio. Richard van Tol and Sander Huiberts created
the IEZA model to make an attempt to classify the types of game audio. The model is created
to make a distinction between the diegetic and non-diegetic sounds in games. Besides the
division between diegetic and non-diegetic sound the model also distinguishes between sound
that gives information about what van Tol and Huiberts have called the setting and the activity
of the game. Sound that gives information on the setting of the game are sounds that give for
example information about the area a player finds himself in. Sound that gives information
about the activity is sound that is linked to a characters in-game activities. See figure 1 for a
representation of the IEZA model. In part 2 of this paper I will make an analysis of the game
Assassins Creed 2 to show what the importance of sound in videogames is and how the
different types of sound contribute to the gameplay. Before I continue to this analysis I will
8

give a short explanation of the four categories (Zone, Effect, Affect and Interface) of which
the model exists.









The diegetic side of the IEZA-model
In the diegetic side of the model van Tol and Huiberts have placed the categories of Zone and
Effect. The first category in the diegetic side of the model is the Effect category. In here
audio is found that is cognitively linked to specific sound sources belonging to the diegetic
part of the game (van Tol & Huiberts 2008). It is audio that comes from sources that are in
the game world, these sources can either be on- or off-screen but are definitely derived from
within the game world. In other words, sounds in this category can be perceived by the
character but do not necessarily have a source that can be seen at the time of hearing the
sound.
Figure 1: The IEZA-model
9

In the Zone category sounds that are linked to the environment in which the game is played
can be found. It are sounds that represent similar environments in the real world. The sound in
this category is often referred to as background or environmental sound. The thing that
separates the Zone category from the Effect category is that Effect category is most often
linked and synchronised to a players in game activities and events.
The non-diegetic side of the IEZA-model
The categories Interface and Affect make up the non-diegetic side of the model. The category
of Interface holds sounds that do not belong to the fictional game world. It are sounds that
give the information about the game that are part of the non-diegetic part of the game. This
means they give information on the game that is not perceivable for the character. In most
cases these sounds can be found in the form of score-counters or pop-up menus. These sounds
are often easily recognizable in that they differ from diegetic sounds because most situations
that are connected to this type of sound do not have a real life equivalent.
To describe the last category of the model I will quote van Tol and Huiberts description since
this also clearly separates the Affect category from the Interface category. Van Tol and
Huiberts state that Affect, consists of sound that is linked to the non-diegetic part of the
game environment and specifically that part that expresses the non-diegetic setting of the
game. Examples include orchestral music in an adventure game and horror sound effects in a
survival horror game. The main difference between Interface and Affect is that the Interface
category provides information of player activity and events triggered by the game in the non-
diegetic part of the game environment, while the Affect category expresses the setting of the
non-diegetic part of the game environment (van Tol & Huiberts 2008).
Another interesting way of analysing videogames is by using Ingmar Ekmans framework for
understanding sound (Ekman 2005). The problem I have with this framework however is that
10

although non-diegetic sounds are part of the framework the focus lies too much on the
diegetic sounds in games which in a way doesnt give enough attention to the non-diegetic
sounds.
11

Part 2 | Assassins Creed 2 analysed
Assassins Creed 2 is an adventure game which lets you play as Ezio Auditore Da Firenze, a
15
th
century assassin who wants to avenge the death of his father and two brothers. For this
analysis I will break the quest Nothing ventured, nothing gained
1
down and examine how all
earlier discussed sorts of audio contribute to the gameplay experience. To summarize, I will
look at diegetic, non-diegetic and transdiegetic sounds and I will apply the IEZA-model since
this gives me the opportunity to cluster different sorts of audio under the right category.
Nothing ventures, nothing gained - analysis
The quest starts after walking up to Leonardo da Vincis house and knocking on the door.
While walking the sound of footsteps in the grass and sand and the singing of birds can be
heard, when knocking on the door a realistic sound of knuckles on wood is played and a cut-
scene starts to play. During these actions before knocking on the door a violin is playing a
tune to create an exciting ambiance. In this part before the actual quest has started I have
already located four different sounds which can be allocated to both the diegetic and non-
diegetic sides of the IEZA model. The sounds of the footsteps and the knocking on the door
can be both placed in the diegetic side of the model under the Effect category since they are
both sounds that come from the game world and can be perceived by both the game character
and the player. The birds singing is also something that Ezio can hear and therefore diegetic,
however it differs from the other sounds in that it is trying to represent the environment in
which Ezio is walking. That is why that sound can be placed in the Zone category. The violin
tune that is played cant be heard by the game character and is therefore automatically placed
in the non-diegetic side of the model. It is a sound that contributes to the setting of the game.

1
To experience what it is to play this quest watch http://www.youtube.com/watch?v=X170FMoQPEQ
12

After the door is opened with a squeaky sound, another diegetic sound in the Effect category,
a cut scene starts in which Ezio meets with Leonardo. Both characters speak with an Italian
accent to contribute to the setting of 15
th
century Italy. The voices echo of the walls to
emphasize the type of room Leonardo lives in. In the background we can still hear an ominous
orchestral sound that tells the player something is about to happen. The cut scene gets
intermitted and with a tingling sound the start of the quest is indicated. After a short freeze the
cut scene continues on a rooftop where we still hear Ezio and Leonardo talk but this time
without an echo. We hear wind as we are high above the city and hear the noise of eagles
circling around the rooftop. When climbing in Leonardos invention, some early flying
machine, we hear the sound of wood creaking under the weight of Ezios body. The cut scene
seemingly slides from a linear part of the game to the non-linear part in which the player
controls Ezios movements again. Just as in the first part leading to the cut scene multiple
sound sources can be traced. The dialog that Ezio and Leonardo have in the cut scene is part
of the diegetic side of the model and can be placed in the Effect category. However the echo
we hear when they are talking in Leonardos chamber supports the idea of the chamber being
big and spacious, hence that part of the dialog can be placed in the Zone category. As
mentioned before the orchestral sound can be placed in the non-diegetic category of Affect.
The tingling sound that signals the intermission in the cut scene is a type of sound we didnt
encounter yet. It is a sound that clearly distinguishes itself from the other sounds that are
present at that moment to indicate something that has to be clear to the player, namely the
start of the quest. It is a sound that falls under the category of Interface belonging to the non-
diegetic side of the model. Although it interrupts the cut scene and with this interruption
might interrupt a players feeling of immersion it is also trigger since the player knows that not
only the quest is started, it can also be the moment for him to take over the control of the
13

game. After the intermission we hear a dialog again and we hear wind and the sound of an
eagle. Just as the sound of the birds earlier, the sound of the eagle is under the Zone category.
The cut scene now has ended and the non-linear part of the game, the part where the player
decides what to do, continues. Ezio now is flying in Leonardos machine and we hear wind
floating by however on a higher volume than before indicating that Ezio is traveling at a
higher speed. By choosing to either make turns or fly straight ahead the sound of the wind
slightly changes as to make it more cohesive with the sound experienced when flying in real
life. Also it gives you feedback on how to navigate in the gameworld. What is interesting
when playing this quest, and others, is that Ezio is apparently talking to himself. However the
information he gives is very relevant for the player (why am I crashing?! and Ill never
make it to the Palazzo!). At that moment all other sounds, birds whistling or people talking
for example, have disappeared. At some point, most often after crashing the machine, the
tingling sound plays again signalling the start of another cut scene. When crashing we hear
the sound of crunching wood. In this cut scene we hear most of the sounds as described in the
first cut scene. The most interesting sound in this part of the quest the most interesting sound
is that of Ezio talking to himself. It is a sound that is diegetic in that it comes from Ezio
however it is completely aimed at the player to inform him about thinking of a solution to the
problem. This can be seen as the transdiegetic sound Jrgensen described. It is a sound that
comes from the game world but is aimed directly at the player, it is the character talking to the
player. When the cut scene ends the quest is completed and the tingling sound, an Interface
sound, plays again verifying that the quest is completed successfully.
As we can see a players action can influence the types of sounds that are played in
videogames. Often these diegetic or non-diegetic sounds are linked and can have an effect on
the subsequent sounds that are played (Collins 2008) but it is clear that sound plays a large
role as a support for the virtual environment and the fictional setting of the game (Jrgensen
14

2008). In the case I just described this can be taken to extremes since the quest wont start at
all when the player decides to not knock on Leonardos door. However it is clear that game
audio is not only linked to what is seen onscreen but also to what we perceive as sounds
referring to real life sounds.

15

Conclusion
As a start of this paper I tried to define what role audio plays in videogames. I established that
sound is more than just vibrations that reach our ear, it is a physical event that we as players
of a game all perceive in a different manner. I started looking at how sound influences
immersion and found that although there is a lot written about immersion, the main focus of
research was on how visuals contribute to getting immersed. From the three forms of
immersion that Ermi and Myr describe sound is of most importance in the imaginative
immersion according to Karen Collins since an interruption in the sound would have the most
influence in that particular form of sound.
In using the IEZA-model I tried to make clear the distinction between diegetic and non-
diegetic sound. In addition I applied Jrgensens concept of transdiegetic sound in my
analysis of Assassins Creed 2. I also integrated some notions of linearity and non-linearity to
show how game audio is played and how users can influence this.
In my analysis it became clear that audio is not merely in videogames for entertainment.
Although it has to be said audios role mainly seems to be to contribute to getting immersed in
a game, it is not the only role. In the cut scenes I described, sound (especially diegetic sound)
is used to explain a narrative that develops through time. The non-diegetic sounds we hear,
the orchestral music, however is used to create a setting in which the game is played, it is used
as an extra sensorial way of making the game exiting.
As my analysis of Assassins Creed 2 shows the mix of diegetic, non-diegetic and even
transdiegetic sounds is what lures players into playing the game. To formulate a swift answer
to my research question: The role of sound in videogames is twofold. On the hand it is there
to give players information about the narrative of the game and feedback on how to navigate
16

through the gameworld. On the other hand it is there to lure players into playing the game, in
other words it is there to create a feeling of immersion.
17

Literature
Calleja, Gordon. 2007. Revising Immersion: A Conceptual Model for the Analysis of Digital
Game Involvement. In Proceedings of DiGRA 2007 conference.
Carter, Dan & Michael Worth. 2009. Adaptive Audio: A Beginner's Guide to Making Sounds
for Videogames. Gamecareerguide.com. Retrieved from
http://www.gamecareerguide.com/features/696/adaptive_audio_a_beginners_guide_.php
Collins, Karen. 2007. An Introduction to the Participatory and Non-Linear Aspects of Video
Game Audio. Eds. Stan Hawkins and John Richardson. In Essays on Sound and Vision.
Helsinki : Helsinki University Press
Collins, Karen. 2008. Game Sound: An Introduction to the History, Theory, and Practice of
Video Game Music and Sound Design. Cambridge: The MIT Press
Ekman, Ingmar. 2005. Meaningful Noise: Understanding Sound Effects in Computer Games.
In Proceedings of the 2005 conference on Digital Arts & Culture.
http://people.uta.fi/~ie60766/work/DAC2005_Ekman.pdf
Ermi, Laura & Frans Myr. 2005. Fundamental Components of the Gameplay Experience:
Analysing Immersion. In Proceedings of DiGRA 2005 Conference: Changing Views Worlds in Play.
Retrieved from http://www.uta.fi/~tlilma/gameplay_experience.pdf
Grimshaw, Mark, Craig A. Lindley & Lennart Nacke. 2008. Sound and Immersion in the
First-Person Shooter: Mixed Measurement of the Player's Sonic Experience. Paper presented
at Audio Mostly-a conference on interaction with sound. Pite, Sweden
Huiberts. 2010. Captivating Sound: The role of audio for immersion in computer games.
PhD thesis, Utrecht School of the Arts
18

Jrgensen, Kristine. 2007. On transdiegetic sounds in computer games. In Northern Lights:
Film and Media Studies Yearbook. Vol 5, 1, 105-117
Jrgensen, Kristine. 2008. Audio and Gameplay: An Analysis of PvP Battlegrounds in World
of Warcraft. In The international journal of computer game research. Vol. 8. Retrieved from
http://gamestudies.org/0802/articles/jorgensen
Jrgensen, Kristine. 2011. Time for New Terminology? Diegetic and Non-Diegetic Sounds
in Computer Games Revisited. In Game Sound Technology and Player Interaction: Concepts
and Developments. Ed. Mark Grimshaw. New York: Hershey
Nevinsound. 2009. Game Audio: Immersion & Interaction. Accessed January 20, 2013.
Retrieved from http://nevinsound.com/2009/03/13/game-audio-immersion-interaction/
Prince, Gerald. 2003. A Dictionary of Narratalogy. Nebraska: University of Nebraska Press
Tol, Richard van & Sander Huiberts. 2008. IEZA: A Framework For Game Audio.
Gamasutra.com. Retrieved from
http://www.gamasutra.com/view/feature/3509/ieza_a_framework_for_game_audio.php
Ubisoft Entertainment. 2009. Assassins Creed II. [PlayStation 3]. Sony Computer
Entertainment. Played 16 January, 2013
Whalen, Zach. 2004. Play Along An Approach to Videogame Music. In the international
journal of computer game research. Vol. 4. Retrieved from
http://www.gamestudies.org/0401/whalen/