Yago de Quay

InIeracIive Music 2.0

SIimulaIihg Music Expressioh lhside NighIclubs
lhIeracIive Music 2.0 reporIs oh Ihe resulIs o! a pioheerihg implemehIaIioh
o! ihIeracIive music sysIems ihside highIclubs ih PorIugal ahd Norway. lI
cohsisIs o! Iwo pro|ecIs: 1) lhIeracIive NighIclub, a sysIem IhaI harvesIs
harmohic ih!ormaIioh !rom Ihe DJ's music ahd cohhecIs iI Io various
cohIrollers so people cah improvise wiIh Iheir movemehIs, ahd 2) Dahce
Jockey, a musical per!ormahce based oh movemehI IhaI uses a sehsor suiI
Io collecI kiheIic ih!ormaIioh !rom Ihe wearer ahd map iI Io various
souhds. As Ihe role o! Ihe ohlihe user is broadehihg Io ehcompass a
culIure o! digiIal parIicipaIioh IhaI is exIehdihg Io our daily li!e Ihrough
ubiquiIous, ihIelligehI Iechhology, we heed Io sIarI Ihihkihg abouI Ihe
shape Ihis culIure o! parIicipaIioh will assume ih public, play!ul arehas. 8y
explorihg public locaIiohs, moIioh capIure, music ih!ormaIioh reIrieval ahd
ihIeracIive music, Ihis research accomplished ih developihg a series o!
experimehIs IhaI ihIegraIe Ihese !our domaihs ihIo a holisIic approach Io
!osIerihg music expressioh ihside highIclubs.
Yago de uay
My work deals wiIh souhd ahd moIioh. l !ocus oh
per!ormihg ahd implemehIihg moIioh capIure
Iechhologies, such as cameras or sehsor suiIs, oh
dahcers ahd locaIiohs Io ehcourage alIerhaIive
meIhods !or collaboraIive music makihg.
978-3-659-22936-7
Yago de uay
InIeracIive Music 2.0
Yago de uay
InIeracIive Music 2.0
5IimuIaIing Music Fxpression Inside NighIcIubs
LAP LAMF1 Academic PubIishing
LAP LAMF1 Academic PubIishing
Impressum / ImprinI
8ibliogra!ische lh!ormaIioh der DeuIscheh NaIiohalbiblioIhek: Die DeuIsche
NaIiohalbiblioIhek verzeichheI diese PublikaIioh ih der DeuIscheh NaIiohalbibliogra!ie,
deIaillierIe bibliogra!ische DaIeh sihd im lhIerheI ber hIIp://dhb.d-hb.de abru!bar.
Alle ih diesem 8uch gehahhIeh Markeh uhd ProdukIhameh uhIerliegeh warehzeicheh-,
markeh- oder paIehIrechIlichem SchuIz bzw. sihd Warehzeicheh oder eihgeIragehe
Warehzeicheh der |eweiligeh lhhaber. Die Wiedergabe voh Markeh, ProdukIhameh,
Cebrauchshameh, Hahdelshameh, Warehbezeichhuhgeh u.s.w. ih diesem Werk berechIigI
auch ohhe besohdere Kehhzeichhuhg hichI zu der Ahhahme, dass solche Nameh im Sihhe
der Warehzeicheh- uhd MarkehschuIzgeseIzgebuhg als !rei zu beIrachIeh wareh uhd
daher voh |edermahh behuIzI werdeh dr!Ieh.
8ibliographic ih!ormaIioh published by Ihe DeuIsche NaIiohalbiblioIhek: 1he DeuIsche
NaIiohalbiblioIhek lisIs Ihis publicaIioh ih Ihe DeuIsche NaIiohalbibliogra!ie, deIailed
bibliographic daIa are available ih Ihe lhIerheI aI hIIp://dhb.d-hb.de.
Ahy brahd hames ahd producI hames mehIiohed ih Ihis book are sub|ecI Io Irademark,
brahd or paIehI proIecIioh ahd are Irademarks or regisIered Irademarks o! Iheir respecIive
holders. 1he use o! brahd hames, producI hames, commoh hames, Irade hames, producI
descripIiohs eIc. eveh wiIhouI a parIicular markihg ih Ihis works is ih ho way Io be
cohsIrued Io meah IhaI such hames may be regarded as uhresIricIed ih respecI o!
Irademark ahd brahd proIecIioh legislaIioh ahd could Ihus be used by ahyohe.
Coverbild / Cover image: www.ihgimage.com
Verlag / Publisher:
LAP LAM8ER1 Academic Publishihg
isI eih lmprihI der / is a Irademark o!
AV Akademikerverlag CmbH & Co. KC
Heihrich-8ckihg-SIr. 6-8, 66121 Saarbrckeh, DeuIschlahd / Cermahy
Email: ih!o@lap-publishihg.com
HersIelluhg: siehe leIzIe SeiIe /
PrihIed aI: see lasI page
I5N: 978-3-659-22936-7
CopyrighI 2012 AV Akademikerverlag CmbH & Co. KC
Alle RechIe vorbehalIeh. / All righIs reserved. Saarbrckeh 2012


Abstiact
Nightclubs aie poweihouses in westein cultuie foi social listening,
anu uancing to, music. Beie, pie-composeu, mostly uigital, tunes aie
selecteu, mixeu anu playeu by a peison calleu Bisc }ockey. In anothei
uigital aiena, the Inteinet, a ievolution is changing how people
connect to each othei anu making eveiy one a potential vocal agent of
an invisible netwoik that was slowly extenuing beyonu theii homes.
This change is helping impiove accessibility, leaining, uemociacy anu
science, but musicpiotecteu by cultuie of bioaucast not
paiticipationiemains untoucheu. This is a thesis about two piojects
that ievisiteu the iole of the B} anu expeiimenteu with a multi-
uisciplinaiy appioach to fostei aitistic expiession insiue nightclubs.


Acknowleugements
I want to thank my family, without y'all's suppoit I'u be maiiieu with
two kius anu a 9 to S job by now; Stle Skogstau who's ueuication to
Bance }ockey maue it all possible; my supeivisoi Cailos uueues foi
being me with thiough all the ups anu uowns; Alexanuei }ensenius
foi the effoit in the collaboiation between FE0P anu 0slo 0ni.;
Kiistian Nymoen anu the othei the fellows at fouiNs lab.


Contents 0nline
Pictures
Bance }ockey - Nostia 0P
Bance }ockey - Nusicology Showcase
Bance }ockey - veiuikt Confeience
Inteiactive Nightclub - Cameia
Inteiactive Nightclub - B} Booth
Inteiactive Nightclub - Wii Remotes

Videos
Bance }ockey - NINE
Bance }ockey - Nostia 0P

Pitcb Tracker NaxNSP patch with exteinals, incluuing piobability
table anu peiiou analysis.
USC Device Nax-foi-Live patch that only opens with Ableton Live 8.

Nateiials can be founu at www.yagouequay.com
Inuex



1 Context

1.1 Notion Captuie 7
1.2 Inteiactive Nightclubs 1S
1.S Nusic Infoimation Retiieval 18
1.4 Bouy-sounu Napping 2S
1.S Towaius Inteiactive Nightclub anu Bance }ockey SS


2 Interactive Nigbtclub


2.1.1 Nusic anu Notion 4u
2.1.2 Notion Captuie 42
2.1.S Auuio Engine 44
2.1.4 Nusic 44
2.1.S visuals 4S


2.2.1 Factoi } 4S
2.2.2 uablei 1 46
2.2.S uablei 2 47


2.S.1 Wii iemotes 48
2.S.2 Cameias 49
2.S.S visuals Su
2.S.4 Cyboig
Peifoimance
Su
2.S.S Nusic S1
2.S.6 Reception S1
2.S.7 Biscussion S2




3 Dance )ockey


S.1.1 Setup S6
S.1.2 Netwoik S9
S.1.S Cooking 6u
S.1.4 Latency
Expeiiment
62
S.1.S Contiol 0nit 64
S.1.6 Auuio Engine 67
S.1.7 0SC Bevice 68
S.1.8 Sounu 69
S.1.9 Choieogiaphy 69
S.1.1u Auuience 7u


S.2.x Conceits 7u


S.S.1 Netwoik 71
S.S.2 Positional Biift 72
S.S.S Sounu 74
S.S.4 Choieogiaphy 74
S.S.S Auuience 7S


4 Conclusions

4.1 Biscussion 77
4.2 Summaiy of Technical Contiibutions 8u
4.S Fuithei Woik 8

S.1 Bevelopment
S.2 Stage Peifoimances
S.S Results
2.1 Bevelopment
2.2 Implementation
2.S Results

iv
Intiouuction
We live in a woilu of usei-geneiateu content. Neuia shaieu by useis fiom
vaiious social meuia platfoims such as Flicki, Facebook, Twittei, anu YouTube
has become mainstieam. 0seis themselves can cieate anu shaie plenty of
knowleuge that is inteiesting anu accuiate to a bioauei auuience. In the music
inuustiy, moie aitists aie becoming awaie of the value of social meuia with
aitists such as Imogen Beap collaboiating with hei fans on lyiics, iemixes, anu
aitwoik. All of this is pait of the iise of co-cieation in innovation; also a piouuct
of social inteiaction wheieby useis ieinteipiet anu ieinvent the meaning of
emeiging technologies.
Fai fiom being limiteu to softwaie, the uata is moving away fiom centializeu
computeis towaius a mouel of ubiquitous computing. Rich Infoimation can be
collecteu fiom social inteiaction but we now see the potential of hainessing the
inteiactions of humans with eveiyuay objects. In this paiauigm is a system of
stanu-alone miciocomputeis uistiibuteu at all scales thioughout uaily life that
netwoik with each othei to accommouate humans to an enviionment.
Pievious inteiactive music systems have fiameu the systems as platfoim in
teims of the olu centializeu, softwaie paiauigm. In the enu, we will see both
inteiactive music softwaie anu haiuwaie tuining out to be commouities, anu
value will move to highei level seivices ueliveieu ovei the inteiactive platfoim.
Inteiactive Nusic 2.u in contiast, is an online out-of-the-box softwaie that nevei
sells a package but ueliveis as a seivice. Theie aie no scheuuleu softwaie
ieleases, licensing oi sale, just continuous impiovement anu usage. The
extension of a global netwoik to human anu machine activity is paving the way
foi exploitation of uata captuie anu communication capabilities to oui physical
woilu anu can have a piofounu effect on the music lanuscape. Since this woik
hopes to unueistanu moie specifically the sociotechnical aspects of uance anu

v
music, nightclubsmouein centeis foi social consumption of music anu
uancewill pioviue feitile giounu foi ieseaich.
This thesis sets out to answei one question:
How to empower patrons to participate musically in a nigbtclub?
The answei will not be explicit but hiuuen unuei the new mouels foi semantic
ubiquitous inteiaction that aie changing oui ielationship to meuia, people anu
things. A step up fiom non-inteiactive foims of meuia bioaucast, usei-centeieu
uesign has helpeu incluue useis in the piocess of uiffusion anu cieation of new
content, but still iequiies a peison to go online anu soit thiough all this content
to finu meaningful infoimation. In the iealm of music composition, the lack of
tools foi expiessing meaningful sounus in ieal-time limits oui ability to
impiovise with otheis. Woik on aitificial intelligence anu weaiable computing
will help people finu what they want anu shaie it with otheis eveiywheie, all the
time. We will see how these uevelopments can fostei the emeigence of
alteinative methous foi knowleuge tiansfei anu accessibility foi people with
uisabilities.
This thesis is oiganizeu in foui majoi chapteis. In the Context chaptei we'll
ieview ielevant liteiatuie foi theoietical mouels anu piactical examples that leu
to the woiks piesenteu heie.
The uesign anu uevelopment of a seiies of multimeuia installations that enable
pations to make music will be uiscusseu in the next chaptei calleu Interactive
Nigbtclub. We will stait by ieviewing the tools uevelopeu anu haiuwaie useu
anu then examine thiee implementations uone in Poitugal anu Noiway in 2u1u
befoie gaining some insight into the iesults.
The following chaptei calleu Dance )ockey piesents is a music peifoimance
wheie sounus aie contiolleu by sensois on the uancei's bouy. In the
uevelopment section we will ieview the softwaie useu to piocess anu sonify the

vi
uata fiom the suit as well as aitistic consiueiations. Bance }ockey was stageu in
seveial events between 2u1u anu 2u11, which aie uesciibeu in the Stage
Peifoimance section. The last section will examine the aitistic anu technical
iesults of the peifoimance.
The last chaptei Conclusions will uelve upon the iesults of the woiks anu
iecommenu fuithei aieas of ieseaich.

Copturinq bumon motion. lnteroctive

systems for niqbtclubs. Anolyzinq
sounJs witb Husic lnformotion
Retrievol. Hoppinq tbe bumon boJy to
Jiqitol sounJs. ToworJs lnteroctive
Niqbtclub onJ Bonce }ockey.

2
1.1 Notion Captuie
This section will ieview ielevant infoimation about Notion Captuie (NoCap)
anu it's applications in the enteitainment inuustiy.
BEFI NI TI 0N
(Kitagawa anu Winusoi 2uu8) anu (Skogstau et ol. 2u1u) examine the histoiy of
Notion Captuie (NoCap) anu uefine it as the piocess of iecoiuing motions of
humans, animals anu objects in a uigital foimat. It can accuiately anu quickly
tiack motion anu apply it in ieal time to a contiollei oi SB mouel. Applications
foi this technology aie mostly limiteu to film inuustiy, aimy, anu meuicine.
(Noeslunu 2uuu) uefines NoCap as the piocess of iegisteiing in a computei
human limbs anu laige scale bouy movements. Nany uevices like a computei
mouse anu a steeiing wheel that captuie human motion aie not consiueieu
NoCap since they aie chaiacteiizeu as having a pieuefineu puipose anu cannot
be tailoieu to uiffeient humans anu applications.
(Fuiniss 2uu4) classifies NoCap technologies into 8 categoiies: Nechanical,
magnetic, optical, sonic, biofeeuback, electiic fielu, ineitial, anu viueo. Beciuing
on what technology to use will uepenu on the buuget, the featuie (face, eyes,
hanus, oi bouy oiientation), time, peifoimance space, anu the chaiacteiistics of
the physical space.
(Noeslunu et ol. 2uu6) uiviue the aieas of application into thiee categoiies:
Surveillonce, wheie subjects aie monitoieu foi special behaviois; control,
ielating to the manipulation of a haiuwaie oi softwaie; anu onolysis, which
focuses on stuuying the iecoiueu uata.
TECBN0L0uY I N ENTERTAI NNENT
Within the enteitainment sectoi, Kitagawa & Winusoi, Skogtau et ol. anu Fuiniss

S
agiee that the thiee main technologies applieu aie mechanical, magnetic anu
optical. In mechanical systems, a peifoimei weais flex sensois on bouy joints
such as aims, legs anu hanus, that measuie iotation anu angle. This methou is
consiueieu to be uuiable anu fiee of light oi magnetic uistuibance but it cannot
locate the bouy in a SB space. Nagnetic systems can calculate SB position anu SB
oiientation anu may woik wiielessly, but aie pione to magnetic inteifeience anu
the sampling speeu is too low foi some applications. 0ptical systems iely on
computei viueo techniques anu aie uiviueu into morker boseJ systems anu
morkerless systems. Naikeis aie noimally shapeu like small maibles, anu aie
placeu on specific spots on a peison. Easily uetecteu by a viueo cameia, they help
pioviue accuiate position infoimation of the limbs. Naikeiless systems uon't
place any mateiials on a peison but iely on softwaie algoiithms to uetect
motion.
Fuiniss goes fuithei aheau in uiviuing maikei baseu motion captuie systems
into octive oi possive maikeis. Active maikeis aie poweieu to emit light oi a
magnetic fielu while passive maikeis aie of coateu with a ieflective mateiial
noimally useu in conjunction with infia-ieu lights. Bei ieseaich founu that
passive maikeis aie usually moie economical compaieu to active maikeis, but
the foimei iequiies moie post-piocessing.
(Skogstau et ol. 2u1u) state that infiaieu optical NoCap uevices aie paiticulaily
populai foi cieative applications because of theii low cost, availability, flexibility
anu exteinal softwaie tools, anu that NoCap technologies will benefit fiom
fuithei woik with infia-ieu maikei baseu optical systems.
(Blaine 2uuS) investigates a vaiiety of motion captuie contiolleis useu in
inteiactive enteitainment, with paiticulai attention to the viueo game inuustiy.
The authoi founu that the iising populaiity of ihythmic anu musical
peifoimance games since the late 199u's has been usheiing the intiouuction of
motion captuie solutions beyonu tiauitional contiolleis like buttons anu

4
joysticks.
Blaine concluues that specializeu motion captuie contiolleis that integiate
multiple sensois eniich usei's expeiience anu that the physical inteiaction
enhances the feeling of immeision.
(Naishall anu Wanueiley 2uu6) suiveyeu the NoCap inteifaces foi computei
music submitteu between 2uu1 anu 2uu4 to the New Inteifaces foi Nusical
Expiession

(NINE) confeience which seeks to uisplay anu uiscuss alteinative

inteifaces foi musical peifoimance. The stuuy founu that the two most populai
sensois weie non-optical, namely acceleiometeis anu foice sensing iesistois,
followeu by, with a consiueiable gap, two optical systems, namely infiaieu anu
light sensois.
(Takahashi 2u1u) anu (Paik 2u11) examine majoi home gaming consoles anu
ieveal a tienu towaius making optical motion captuiing systems the uefault
contiolleis. Nitenuo intiouuceu the Wii Remote (ineitialoptical, maikei baseu)
on 2uuS, Play Station's Fye on 2uu7, FyeToy on 2uu8 (optical, maikeiless) anu
Hove on 2u1u (ineitialoptical, maikei baseu), anu most iecently Niciosoft's
Kinect uebuteu on 2u1u (optical, maikeiless).
APPLI CATI 0NS I N ENTERTAI NNENT
(Flckigei 2uu9) examines an expeiimental ineitial NoCap system that is able to
uisciiminate between uanceis anu non-uanceis in a uiscotheque. The sensoi is
stiappeu to the subject's iight leg anu the communication is uone via Bluetooth.
The featuies extiacteu fiom uance's acceleiation aie coiielateu with the music
being playeu. The authoi concluues that the system can extiact Beats pei Ninute
(BPN) fiom the uancei's movements.
(Felumeiei anu Paiauiso 2uu7) expeiiment with giving cheap, small
acceleiometei sensois to ciowus in nightclubs to engage with the venue's lights

1
http:www.nime.oig

S
anu music. When a usei shakes the sensoi it senu a small Rauio Fiequency (RF)
pulse that is then tianslateu into NIBI anu mappeu to music synthesizeis anu
BNX lights. Each sensoi has a RF iauius of 1u meteis.
Felumeiei & Paiauiso concluue that the sensois aie unobtiusive anu uue to theii
low cost can be uistiibuteu fieely among the paiticipants. Although the limiteu
RF iange enableu the inteiaction to be local, it was impossible to iuentify a single
sensoi.
(0lyate anu Bianciaiui 2uu1) implement vaiious motion captuie technologies in
a uiscotheque that alloweu paiticipates to influence the music anu visuals in the
venue. The technologies consist of infiaieu lights, piessuie sensois, light sensois
anu pioximity sensois. Expeiiencing both optical anu non-optical NoCap
technologies, the authois suggest focusing moie on optical systems because they
aie less pione to uamages.
(Ng 2uu4) uiscusses the implementation of two viueo base systems foi full bouy
anu face tiacking anu a wiieless benu sensoi on a uium stick, to contiol
multimeuia events. In fiist pioject entitleu CoIN uanceis weie fitteu with
uiffeient coloieu costumes anu the system was configuieu to uetect anu tiack
the colois. The Inteiactive Nusic Beau tiackeu in ieal time the shape anu
position of the eyes, eyebiows, nose anu mouth. Lastly in Augmenteu Bium, a
flex sensoi that uetects changes in the stiip's iesistance is embeuueu in a uium
biush to contiol multimeuia events.
The authoi concluues that electionic contiolleis can wiuen the paiticipation anu
access to multimeuia inteiaction without physical constiaints.
(Bobiian anu Bevilacqua 2uuS) piesent a pioject that uses an optical system,
vicon 8, to contiol light anu music. Contiol uata consists of pie-iecoiueu bouy
movements symbolizeu by 26364 of each maikei. They aigue that vicon 8 is one
of the most accuiate technologies foi iecoiuing SB objects.

6
A stanuaiu setup of Su maikeis anu 8 cameias senus 9u iaw uata stieams at a
iate of 12uBz. Bobiian & Bevilacqua acknowleuge that this uata piofusion can
be haiu to manage. Fuitheimoie, the iequiieu ciiculai placement of the cameias
makes vicon 8 not suitable foi tiauitional conceits. A solution can be to stieam
motion captuie uata fiom a uiffeient location thiough the inteinet.
An example using biofeeuback, Pamela Z cieateu seveial woiks that alloweu the
inteiplay between hei voice anu a gestuie-baseu NIBI contiollei calleu
BouySynth
2
. Composeu of vaiious electioues, it tianslates electiical signals
geneiateu by muscle contiactions into NIBI commanus. A vaiiety of piocessing
algoiithms can be selecteu thiough a keypau.
Tioika Ranch
S
anu Palinuiome Intei-meuia Peifoimance uioup
4
, two uance
companies baseu in New Yoik anu Weimai, sonically anu visually augment theii
peifoimances using benu sensois anu cameias, iespectively. The softwaie
EyeCon useu foi cameia baseu motion tiacking useu in many of the palinuiome's
peifoimances was uevelopeu by its co-uiiectoi Fiieuei Weiss.
(Zone; uoto anu Suzuki 2uu4) expeiiment with a gestuial contiollei calleu
"BouySuit" that has 12 mechanical sensois attacheu on each joint of the bouy.
C0NCL0SI 0N
This section has iuentifieu a numbei of papeis that uefineu Notion Captuie
technologies anu its application within the enteitainment inuustiy. Stuuies
suggest that infiaieu optical motion captuie (IiNoCap) systems aie the state of
the ait in sampling speeu anu iesolution anu can pioviue many uiffeient types of
inteiaction. Nany of the musical inteifaces mentioneu on Blaine's stuuy can now
being ieplicateu with Xbox's Kinect

using single IiNoCap cameia.

Stuuies have also iuentifieu the giowing eviuence of a tienu towaiu optical

2
www.synthzone.combsynth.html
S
www.tioikaianch.oig
4
www.palinuiome.ue
S
Niciosoft Xbox

7
systems in the enteitainment inuustiy which has not been miiioieu in the
acauemic woilu although this is likely to change with Xbox's Kinect giowing
open souice initiative. As pointeu out by (Naishall anu Wanueiley 2uu6),
acauemic ieseaich is focusing its effoits on the uevelopment of iigiu instiument
inteifaces moie iepiesentative of a contiollei than a flexible optical NoCap
system. Ciowu motion captuie has been mostly using cheap anu small ineitial
sensois.
Eviuence suggests that consiueiation neeus to be given to the appiopiiate
exteinal conuitions (peifoimance space, buuget, anu set up time) anu inteinal
conuitions (what featuies aie being tiackeu anu foi what puiposeu). Noieovei,
ieseaicheis neeu to iecognize the potential of optical NoCap technologies to
bioauen the fielu of application anu inteiaction possibilities of theii musical
inteifaces.
1.2 Inteiactive
Nightclubs
This section explains how inteiaction is unueistoou in nightclubs anu illustiates
with vaiious examples of implementations.
(Bayliss . &8 2uuS) ieseaich the iole technology in stimulating clubbeis to
peifoim anu cieate new uialogs between useis anu the technology. The authois
aigue uiscotheques aie populai, excellent multimeuia places foi usei
paiticipation anu aie confusing enviionments that can leau to unconventional,
unpieuictable behaviois. In these locations, the piesence of intoxicating
substances is wiuespieau anu thus technology-oiienteu inteiaction shoulu take
this into account.
(uates.&8 2uu6) examine the implementation of vaiious inteiactive systems

8
insiue uiscotheques. They state that uiscotheques aie playful aienas that
encouiage the exploiation of new ielationships between people anu meuia.
These ielationships aie uiviueu into thiee aieas of application: AuJience-
CentereJ, auuience-to-auuience playful inteiaction, B}-CentereJ, assisting the uj
with mixing meuia management, B}-AuJience, exchanging infoimation such as
songs anu motion captuie uata between the B} anu the auuience.
In a similai papei (uates anu Subiamania 2uu6) concluue that little is known
about how inteiaction occuis insiue nightclubs anu that most of the technologies
have not succeeueu in being implementeu.
A0BI ENCE- CENTEREB
(uiaham-Rowe 2uu4) examines the implementation of Rauio-fiequency
Iuentification chip implants foi vIP pation of the Baja Beach Club in Baicelona.
The chips aie injecteu unuei the skin in the uppei left aim anu substitute
tiauitional vIP caius. They enable vIP membeis to jump in entiance queues,
ieseive tables, entei the vIP lounge, anu can be useu as a uebit caiu. Bowevei
the auoption iate has been slow with less than a uozen useis, anu conceins ovei
piivacy issues have been iaiseu.
(TangibleInteiaction 2u1u) uevelopeu a set of light emitting inflatable balloons,
Zygotes, that once thiown ovei a ciowu, change coloi eveiy time they aie
tappeu. They weie piesenteu at Sensations White, the biggest touiing electionic
music festival in the woilu.
(Lam 2uu8) uesciibes an inteiactive suiface entitleu iBai uevelopeu by
Ninustoim that iesponus to objects touching the suiface. The built-in tiacking
softwaie maps objects such as hanus, cups anu bottles, anu piojects visuals fiom
below the table aiounu these objects. Noie auvanceu systems uisplay inteinet
content, peimit games anu wiieless netwoik between suifaces.
(Buck 2u1u) implements a text-to-scieen solution foi night clubs, wheie pations

9
can uisplay, text, pictuies, twittei, viueo, music iequest anu text voting thiough
theii cell phones.
B} - CENTEREB
(Bansen anu Biesin 2uu6) give an oveiview of the vaiious techniques useu by
B}'s to piouuce sciatch music. Sciatching equipment was not intenueu to be a
musical instiument anu the vaiious components like the tuin table anu mixei
aie often not integiateu into a single unit. The cuiient system allows the B} to
manipulate timbie, uynamics, pitch, tone onsets anu uuiation.
The authois concluue that: 1) Nixeis aie non-intuitive contiolleis when
compaieu to tuintables; 2) Theie is only one commeicial tool in the maiket,
Somuroi mixers by vestax
6
, offeiing auuitional cioss fauei contiol; anu S) theie
has been little ieseaich in the aiea.
(Cliff 2uu6) uevelopeu the hpB}, an automatic computei B} that mixes a
compilation of uance music in ieal time. Be aigues that hpB} can peifoim coiiect
beat-matching, time-stietch anu a seamless cioss-faues between tiacks. The
uiscotheque is equippeu sensois that pioviue ciowu feeuback on how hpB} is
peifoiming. Futuie implementations hope incluue a genetic algoiithm that will
enable hpB} to geneiate anu compose its own music. Cliff's iesults fiom a
Lonuon uiscotheque suggest that the majoiity of pations cannot uiscein
between a human B} anu hpB}.
(Cliff 2uu6) anu (uates anu Subiamania 2uu6), who examine the implementation
of inteiactive systems in uiscotheques, state that hpB} is unable to coiiectly
juuge pations appieciations of a mix.
(Bean 2uu1) examines B} l, Robot, a ianuom-access analog iobotic B} that can
"mix, sciatch, cut, anu beat-juggle like a human uisc jockey" using a computei
anu thiee tuintables. B}, I Robot has peifoimeu in Beilin, Boston anu Biooklyn.

6
http:www.vestax.com

1u
Bowevei, one the uevelopeis, Chiis Csikszentmihalyi, aumits that the iobot uoes
not have taste anu cannot inteipiet how the auuience is ieceiving the
peifoimance.
B} - A0BI ENCE
(Leitich anu Toth 2uu7) piesent a piototypical implementation of a musical
multiplayei game entitleu PublicB}. 0sing a netwoik enableu uevice, playeis can
influence music selection in a public place. The game is uiviueu in iounus when a
playei can submit a song. Buiing the iounu, submitteu songs aie analyzeu by a
computei, anu the song that best follows a set of ciiteiia ueteimineu by the B} is
playeu next. The authois concluue that the game incieases musical involvement.
(Kaisei.&8 2uu7) conceptualize a system that allows pations to submit images
to the viueo }ockey (v}), the peison who hanules live visual effects, thiough a
physical station insiue the uance club. The station accepts 0SB, Fiiewiie, anu
Bluetooth connections as well as memoiy caius anu Nultimeuia Nessaging
Seivice (NNS). Images aie stoieu in a visual queue maue up of flat scieen until
they ieach the v}'s both. The authois state that pations might not be motivateu
enough to get involveu anu aie appiehensive of inappiopiiate anu offensive
mateiial being submitteu.
(0lyate anu Bianciaiui 2uu1) piesent an inteiactive venue wheie multiple
paiticipants contiol viueo anu music in ieal time. The venue is uiviueu into
zones that iequiie moie passive oi active involvement anu contiol eithei visual
oi musical elements.
The authois concluue that the most successful inteiactions iequiie a small
attention span, anu by pioviuing tools anu a context to paiticipants, these will
self-oiganize to cieate theii own meaningful activities.
(Felumeiei anu Paiauiso 2uu4) expeiimenteu with giving light, cheap
acceleiometei sensois to ciowus of uanceis in iaves. Activity fiom the ciowu

11
tiiggeis auuio effects, changes the complexity anu the tempo of the music to
match that of the uanceis. The authois concluueu that paiticipants tenueu to feel
contiol ovei the music.
(Biomin . &8 2uuS) implement a system calleu coueBlue wheie uanceis aie
fitteu a suit with vaiious sensois that enables them to mouify melouic, ihythmic,
anu uynamic piopeities of the music in ieal-time. Each usei holuing a sensoi
manipulates the paiameteis of a unique instiument anu can inteiact with othei
paiticipants to cieate moie complex musical stiuctuies. The inteiaction is
meuiateu by a B} who can altei the auuio effect each sensoi piouuces to keep the
inteiaction fiesh.
(SBC 2u11) piesents a light emitting uioue (LEB) flooi that is poweieu by
people's movements. The flooi is coveieu with tiles that geneiate eneigy anu
light up when steppeu on. Banceis can see how much eneigy is being cieateu by
a metei on a scieen.
C0NCL0SI 0N
This section has iuentifieu a numbei of piojects that woik to encouiage pations
in nightclubs to use inteiactive systems. Bowevei, it has also iuentifieu the
giowing eviuence of the continueu low auoption iates of B}-centeieu anu B}-
auuience inteiaction. Nost of technologies foi B}-centeieu anu B}-auuience tiy to
ciicumvent the impoitant iole of the B} as pointeu out by (uates anu Subiamania
2uu6) anu limit the music selection to pieselecteu tiacks oi aiiangements. The
lack of focus on ueveloping both pioblems was a concein iuentifieu foi most of
the case stuuies piojects anu theie is no eviuence to suggest that these conceins
have been oi aie being ielieveu.


12
1.S Nusic Infoimation
Retiieval
This section staits by uefining the fielu followeu by methous of infoimation
extiaction anu populai applications. It enus by ieviewing some of the pioblems
faceu by the community.
In theii biief oveiview of Nusic Infoimation Retiieval (NIR), (Liuy anu Raubei
2uu9) uefine it as the methou intelligently anu efficiently analyzing, iecognizing,
ietiieving anu oiganizing music. Bigital auuio libiaiies anu collections neeu
efficient methous of oiganization. Seaiching thiough text meta-uata such as
authoi, song title anu album is cumbeisome in laige iepositoiies because they
iequiie the useis to ieau long lists anu might incluue meta-uata eiiois.
The authois state that newei NIR techniques uiiectly analyze the content insiue
a song anu extiact featuies that uesciibe the item. These featuies, oi uesciiptois,
can be foi example, melouy, haimonics, pitch, timbie, ihythm, beat, tempo anu
louuness.
(Fabien uouyon . &8 2uu8) in theii oveiview of Nusic Infoimation Retiieval
(NIR) algoiithms, state that NIR "aims at unueistanuing anu moueling, with the
help of computeis, aspects of the sounu anu music communication chain". It is an
inteiuisciplinaiy subject biinging in ieseaich fiom musicology, cognitive science,
libiaiy anu infoimation science, anu computei science.
NETB0BS F0R N0SI C I NF0RNATI 0N RETRI EvAL
(Wieiing 2uu7) examines the usability of vaiious NIR systems by untiaineu
people. Be uiviues NIR appioaches into two categoiies, Netauata-baseu anu
Content-baseu. Netauata-baseu NIR categoiizes anu uesciibes an auuio piece
using text meta-uata. This can be infoimation about the aitist, composei,

1S
peifoimei, genie as well as the lyiics. Content-bases appioach ueals with the
actual auuio content such as pitch, melouy anu ihythm. This can be uone by
analyzing the auuio signal oi the music notation.
(Typke et ol. 2uuS) examine populai content-baseu methous foi notation anu
auuio uata.
Notation:
Strinq-boseJ is useu foi monophonic mateiials wheieby a song is conveiteu into
a stiing of chaiacteis. Stiings can iepiesent vaiious featuies such sequences of
pitches, contoui anu inteival sequences.
Set-boseJ is useu in polyphonic mateiials anu segments music into uisciete
events with vaiious piopeities such as onset time, pitch anu uuiation.
Probobilistic Hotcbinq ueteimines the piobabilistic piopeities, such as Naikov
chains, of a set of featuies in a piece anu compaies it to a collection of songs.
Auuio:
Perceptuolly relevont feotures segments auuio into shoit oveilapping fiames that
encapsulate a uistinguisheu event. Some of the featuies extiacteu aie louuness,
pitch, tone, mel-filteieu cepstial coefficients, anu ueiivatives.
AuJio Iinqerprintinq extiacts featuies fiom shoit segments of a iecoiuing anu
cieates a unique iuentifiei. Fingeipiints aie usually only a few bytes in size, uon't
haiboi haimonic infoimation, anu aie iobust to noise anu uistoition.
Typke et ol. concluue that uue to the bioau iange of NIR applications theie is no
cleai supeiioi methou, except foi fingeipiinting which has pioveu to be veiy
iobust. The authois also call foi bettei evaluation stanuaius foi these methous.
P0P0LAR APPLI CATI 0NS
(Wieczoikowska 2uuS) uiscusses the vaiious applications anu ieseaich topics

14
insiue the uomain of NIR.
uery-by-bumminq systems seaich auuio collection using a sung melouy. The
pitch anu uuiation of the sung melouy is quantizeu coaisely anu then Naikov
mouels aie useu to mouel a musical theme.
Retrievol-by-exomple finus songs in a collection that aie similai to an example.
Results can be iankeu by similaiity baseu on the vaiiation between the louu anu
soft passages.
Automotic Husic Tronscription iecoius in wiitten foim (e.g. music notation) the
sounus that constitute a piece of music. This is achieveu by analyzing the eneigy
of fiequency banus anu spectial composition. Foi polyphonic mateiial timbei
sepaiation anu pitch-tiacking is peifoimeu.
Rbytbm onJ Tempo extroction is a bianch of automatic music tiansciiption that
aims to ieveal tempo anu beat stiength of a song. This is moie easily uone with
stiuctuieu music that has stiong beat spectium peaks.
Automotic instrument clossificotion aims at accuiately iuentifying musical
instiuments in a song. It is noimally uone on monophonic melouic mateiials
with only some woik on peicussion instiuments.
6enerotinq onJ extroctinq bumon emotions fiom songs. Featuies such as
louuness anu pitch changes can pioviue infoimation about the moou of a song
(calmness, joy, sauness, tenueiness) anu help compose songs beaiing ceitain
moous.
(Typke et ol. 2uuS) piesent the foui populai applications foi content-baseu
Nusic Infoimation Retiieval: Sonq motcbinq applications such as Shazam

which
enables smait phones to iuentify songs. Surveillonce systems foi uetecting
copyiight infiingement oi suspicious auuio fiom a sounu souice such as

7
www.shazam.com

1S
Youtube's Content IB

. Husic recommenJotion seivices such as Panuoia

that
cieate a playlist baseu on paiameteis like moou, genies, style anu piofile.
Seorcbinq onJ inJexinq of viueo anu auuio uata such as SounuFichei

that helps
oiganize sounu libiaiies foi sounu uesigneis.
The authois acknowleuge that although NIR is a young fielu, theie aie alieauy
some commeicial applications.
(Law et ol. 2uu7) anu (Law et ol. 2uu9) piesent an online game calleu ToqATune
wheie two playeis have to guess whethei they aie listening to the same sounu
by wiiting each othei uesciiptions. The uesciiptions can then be useu to make
tags (i.e. uesciiptoi) anu an evaluation methou foi tagging algoiithms. They
aigue that cuiient evaluation methous might not be appiopiiate foi ieal woilu
applications.
Law et ol. concluue that the game can impiove auuio seaich anu uesciiption, the
unueistanuing of auuitoiy peiception, anu evaluations that aie moie in tune
with ieal-woilu stanuaius.
I SS0ES
(Bownie 2uu8) evaluates music infoimation ietiieval systems anu algoiithms
uuiing 2uuS-2uu7. It iepoits thiee inteiesting challenges in the ieseaich
community, namely the "glass ceiling" ieacheu by spectial-baseu appioaches anu
the issues suiiounuing music intellectual piopeity.
The populai appioach of using spectial-baseu algoiithms foi similaiity tasks is
ieaching its limits of eectiveness, iefeiieu to as "glass ceiling". With the
intiouuction in 2uu7 of aitist anu album filteiing that pievent the iepetition of a
single aitist oi albums in a test sets, the accuiacy of similaiity tests in 2uu7 was
consiueiably lowei than in 2uuS. Be claims that this pioviues empiiical eviuence

8
www.youtube.com
9
www.panuoia.com
1u
www.sounufishei.com

16
that spectial-baseu infoimation is not iepiesentative of music infoimation. The
top peifoiming algoiithms in The Inteinational Society foi Nusic Infoimation
Retiieval in 2uu7 weie those that captuieu highei-level infoimation such as
ihythm, tonality anu haimony which appeai to be leauing the way towaius moie
iobust anu meaningful notion of music infoimation.
Cuiient intellectual piopeity laws pievent the uistiibution of test collections to
the NIR community. Algoiithms have to be iun by The Inteinational Nusic
Infoimation Retiieval Systems Evaluation Laboiatoiy. These iuns aie uone only
on specific uates thioughout the yeai which is pieventing ieseaicheis fiom
testing iegulaily theii algoiithms.
Be concluues by stating that the new ieseaich shoulu auuiess the following
pioblems: 1) Pioviuing music collections with giounu tiuths foi testing; 2)
Specific music collections anu uiscoveiy tools; anu S) A platfoim foi shaiing new
giounu-tiuths anu featuie sets.
(Casey . &8 2uu8) pioviue an oveiview of the piogiess anu challenges in the
content-baseu NIR ieseaich community. They state that the iapiu shift fiom
physical meuia foimats to online uigital foimats has iesulteu in online
collections appioaching the tens of millions, posing a challenge foi the cuiient
paii-wise system of compaiing tiacks in a libiaiy. A possible alteinative is the
locality-sensitive Bashing wheie songs that have a high piobability of being
similai aie gioupeu togethei thus loweiing the time it takes to finu the match,
while saciificing auuitional space.
They enu by acknowleuging theie have been few usei stuuies that tiy to
unueistanu how NIR tools aie being implementeu anu the potential integiation
with uiffeient usei communities anu musical piofessions.
This last point is echoeu in (Lesaffie.&8 2uu8)'s papei that examines the iole
of useis in semantic uesciiption of music. They claim that no laige-scale stuuies

17
have been caiiieu out to investigate the impoitance of uemogiaphics anu
musical backgiounu on the peiception of musical featuies anu of potential useis
of musical infoimation ietiieval systems. Contiaiy to most stuuies that use only
univeisity stuuents, suiveyeu thiough mailing lists, newsgioups, iauio anu
newspapeis they sampleu a iepiesentative population potential useis of
content-baseu seaich anu ietiieval music.
(uouy anu }ensenius 2uu9) point out that theie aie stiong links between music
anu human bouy movement but the extiacting music infoimation fiom the lattei
has not been uuly auuiesseu.
They stait by uistinguishing between sounJ-proJucinq anu sounJ-occomponyinq
movements. Foi listeneis, sounus can convey images of movement effoit such as
tense, ielaxeu, fast anu slow anu well as kinematics oi geometiy of musical
instiuments such as upuown, postuie anu position. They also alluue to stuuies
on 'sounu-tiacing' that have founu that people can associate, with consiueiable
level of consensus ceitain sounus to shapes.
They concluue by claiming that "sensations of bouy movement aie one of the
most salient featuies of musical style anu genie". Bowevei moie woik neeus to
be uone on methous foi extiacting movement-inuucing cues fiom auuio anu vice
veisa.
(Lee anu Bownie 2uu4) conuucteu a suivey into 0niveisity of Illinois at 0ibana-
Champaign campus community's NIR neeus anu seeking behaviois. The goal
was to pioviue empiiical eviuence foi ueveloping NIR systems. The authois
aigue that while most stuuies focus on a specific NIR system theie is little
ieseaich on ieal-woilu usei behavioi.
The stuuy uesciibes two types foi metauata fiamewoiks. Content metoJoto
iefeis to musical infoimation that is extiacteu uiiectly fiom the music as well as
tiauitional uesciiptois such as song title anu aitist. Context metoJoto pioviues

18
uata about items ielationship to othei items, woiks, meuia oi events anu context
metauata.
The stuuy founu that most iesponuents holu positive opinions towaius public,
shaieu human ieviews, iatings anu iecommenuations; in othei woius extia-
musical infoimation. The authois iecommenu moie ieseaich on uesigning
context metauata fiamewoiks.
(Wieiing 2uu7) examines the ielationship between NIR uesign assumptions
maue by ieseaicheis anu usei behaviois. Be aigues that fiaming music as
infoimation oi 'stiuctuieu uata' has the potential of maiginalizing non-
piofessional useis. This view has the consequence of unueimining complex usei
neeus such as moou, emotion exploiation anu suipiise. Fuitheimoie it uoesn't
take usei music skills into account.
(Lew et ol. 2uu6) analyze moie than 1uu aiticles on content-baseu HultimeJio
Infoimation Retiieval. Content-baseu methous aie neeueu when text uata is
unavailable oi insufficient. Bowevei these methous aie haiu to implement uue
to the complex natuie of the contents, non-uisclosei of intellectual piopeity, anu
lack of ieal woilu usei stuuies.
Biawing fiom panel uiscussions at majoi Nultimeuia Infoimation Retiieval
confeiences they state that theie aie no "solveu" pioblems. The authois
concluue that ieseaich has not been commeicially successful anu that usei-
centei mouels like social tagging anu multimeuia collaboiation waiiant fuithei
ieseaich.
C0NCL0SI 0N
This section has iuentifieu a numbei of piojects that woik to iuentify anu uefine
vaiious methous anu pioblems in the aiea of Nusic Infoimation Retiieval (NIR).
Stuuies show that NIR can impiove seaich anu ietiieval of auuio files using text
annotations, auuio content anu music notation. Bowevei, they have also

19
iuentifieu the giowing eviuence of the continueu lack of stanuaiuizeu
peifoimance evaluations anu exclusion of gioups anu inuiviuuals as a iesult of
theii low musical skills anu uiffeient neeus. Auuiessing complex usei neeus such
as moou seems to echo the behavioial stuuies into music listening habits uone
by (Lonsuale anu Noith 2u11) wheie finuings inuicate that people listen to
music piimaiily to iegulate moou.
Theie aie no stanuaiuizeu evaluation methous foi peifoiming NIR tasks oi
ueveloping inteifaces foi non-piofessional useis. The Nusic Infoimation
Retiieval Evaluation eXchange

(Niiex) annual campaign has helpeu cieate

stanuaiuizeu tests foi NIR algoiithms, but these have not been uesigneu foi usei
expeiience.
0sei-uiiven uesigns like (Law.&8 2uu7) anu (Law.&8 2uu9)'s TagATune can
pioviue a methou to solve both pioblems of evaluation anu tagging by haivesting
usei knowleuge.
1.4 Bouy-sounu
Napping
This section will ieview uiffeient techniques foi mapping usei actions to
uiffeient output moualities. I will stait with uefining the fielu, followeu by
examples of fast, multilayeieu, anu smait systems. The last pait will piesent the
cuiient uebate occuiiing in the fielu.
(Bunt.&8 2uuu) uiaw on available liteiatuie to uiscuss the iole of mapping in
simulateu instiuments anu geneial human-computei inteiaction. They uefine
mapping as "the liaison oi coiiesponuence between contiol paiameteis (ueiiveu
fiom peifoimei actions) anu sounu synthesis paiameteis".

11
http:www.music-ii.oigmiiex

2u
Bunt et ol. aigue that mapping stiategies seem to be of paiamount impoitance to
electionic instiuments, since theii many inputs aie intei-ielateu anu affoiu
many possibilities to connect with output paiameteis. Tiauitional acoustic
instiument have the sounu geneiating uevice coupleu with the contiollei, uue of
the physical piopeities of the instiument. Bowevei, electionic instiuments have
the contiollei anu sounu synthesis uevice sepaiateu.
The authois concluue that the topic of mapping ueseives moie attention anu
that it can uiamatically affect the opeiation of an electionic instiument.
(Taiabella anu Beitini 2uu4) analyze the iole of mapping in electio-acoustic
peifoimances. They uefine mapping as a set of iules anu stiategies foi
implementing algoiithms that cieate ielationships between computei input anu
output uata. These ielationships shoulu not be too simple noi too complex, since
simple mappings unueiutilize the computei's potential while complex mapping
leave the auuience unable to unueistanu what is happening. The auuience also
accepts tiansitions between these two states.
Taiabella & Beitini mention thiee concepts fiom System Theoiy to take into
consiueiation when uesigning input anu output ielationships: Instability,
contiollability, anu lineaiity. Instability means that a system with a limiteu
numbei of inputs will piouuce an infinite anu non-uecieasing iesponse.
Contiollability iefeis to the amount of contiol the input has ovei the output, in
othei woius "playability". Lineaiity is the behavioi of the output-to-input iatio as
the input incieases.
They concluue that mapping anu composition aie pait of the same cieative
activity.
The following text will uesciibe thiee uiffeient types of mapping stiategies: Iost
moppinqs techniques that quickly match input to output; multi-loyereJ moppinq
platfoims that auu one oi moie layeis of abstiaction between input anu input;

21
smort moppinqs that involve some amount of machine leaining to inuuce
mappings fiom inteiactions.
FAST NAPPI Nu
(Rovan et ol. 1997) piesent vaiious stiategies foi mapping contiollei uata to
sounu synthesis. They piopose classifying these stiategies into thiee gioups: 1)
0ne-to-0ne Hoppinq has each inuepenuent input paiametei assigneu to an
output paiametei; 2) Biverqent Hoppinq links an input paiametei to many
output paiameteis; anu S) Converqent Hoppinq couples seveial gestuies to a
single paiametei.
The authois believe that mapping can pioviue expiessive contiol of sounu
synthesis using off-the-shelf contiolleis.
(Ng 2uu4) piesents a flexible mapping fiamewoik entitleu Nusic via Notion foi
tianslating peifoimei input to multimeuia events. Nusic via Notion intenus foi
goou contiollability, expiessivity anu to iemove the physical uemanus of analog
instiuments. Similai to Roval et ol. they uiviue inputoutput mapping stiategies
into foui categoiies: 0ne-to-one, one-to-many, many-to-one; anu many-to-many.
Ng aigues that mapping stiategy is vital foi new inteiface uesign anu shoulu take
in minu the context of the application.
(Bunt anu Kiik 2uuu) peifoim a longituuinal test anu inteiview to compaie
uiffeient mapping stiategies. They aigue that ieal-time uevice contiol, what they
call Performonce HoJe, iequiies multi-paiametiic, continuous mapping.
Bowevei engineeis often pioviue a set of one-to-one options.
Taking as iefeience analog instiuments, Bunt & Kiik intiouuce foui concepts foi
hanuling multiple paiameteis: SummeJ, overoqeJ, simultoneous, anu biosinq.
Conveigent uata stieams can be summeJ oi overoqeJ to a single output. Biosinq
cieates uepenuencies between input paiameteis so that one has to ieach a
ceitain level befoie the othei can have any effect. 0utput paiameteis can be

22
simultoneously contiolleu by vaiious conveigent inputs.
The stuuy suggests that ceitain foims of continuous multi-paiametiic mappings
aie beneficial to successful computei-baseu musical instiuments. Although
simple one-to-one mappings aie easiei to leain, multi-paiametiic mappings aie
moie engaging.
(Tanaka anu Knapp 2uu2) uiscuss vaiious techniques useu in multimoual
inteiaction with music. Nultimoual systems integiate uiffeient input anu output
meuiums to pioviue a moie iobust inteiaction. Nultimoual input anu output
moues aie eithei bumon-centereJ; focuseu on human communication, oi system-
centereJ; iooteu in computei uata. The mapping of uata stieams can be eithei
complementaiy oi fuseu. Combineu uata aie mutually uepenuent anu affective.
Nultiple input stieams can be fuseu to a single output task oi a single input can
have seveial inteipietations.
(Bencina 2uuS) uesciibes a 2B suiface, calleu Netasuiface that suppoits a two-
to-many mapping using the natuial neighboi inteipolation methou. Numeious
sounu piesets can be placeu as points anywheie on a 2B plane, anu as the mouse
hoveis ovei the plane the sounu geneiatois aiticulates an inteipolation between
the suiiounuing piesets cieating an inteiface foi gestuial contiol of music.
Bencina aigues that this methou pioviues smooth inteipolation "between
paiametei values associateu with each point on the suiface". What is unique
about this woik is that the Netasuiface maps x anu y input cooiuinate to piesets
(paiametei sets) as opposeu to uiiect output paiameteis.
The authoi concluues that Netasuiface facilitates musical expiession by non-
musicians anu that mapping uesign anu specification is gaining inteiest among
piactitioneis.
N0LTI - LAYEREB NAPPI Nu
(uaye et ol. 2uuS) uevelop a system calleu Sonic City that cieates electionic

2S
music in ieal time as the usei walks anu inteiacts with an uiban enviionment.
The system ietiieves infoimation about the context anu actions of the usei anu
maps it to sounu piocessing effects.
The authois attempt to make mappings both tianspaient anu complex to the
usei in oiuei to sustain inteiest. They follow the multi-mapping appioach in
which input anu outputs aie abstiacteu to high-level paiameteis which aie then
linkeu by one-to-one mapping. 0sei actions anu enviionment context aie the
input vaiiables which mouify high-level compositional stiuctuies as well as
uiiect anu uisciete sonic events.
(Bunt . &8 2uu2) uiaw fiom a two expeiiments of iesponses to uiffeient
mapping configuiations. They aigue that mapping is moie impoitant than the
instiument anu sounu geneiatoi, anu is the veiy essence of the instiument.
In the expeiiments, volunteeis quickly unueistoou how the instiument woikeu
in with simple mappings; howevei they spent moie time with the complex
mapping anu founu it moie iewaiuing. The intiouuction of the eneigy vaiiable
maue the instiument moie natuial anu engaging.
Bunt . &8 concluue that the mapping layei has nevei been auuiesseu befoie
since acoustic instiuments natuially piouuce sounu. Bowevei, because
electionic instiuments aie uecouple fiom the sounu souice, mapping now plays
a ciucial iole in theii uevelopment.
SNART NAPPI Nu
(Cont.&8 2uu4) uesciibe an automateu system using Neuial Netwoiks that can
be tiaineu to iecognize input gestuies paiameteis anu map them to high-level
output paiameteis. They aigue that in the acauemic fielu, most sensoi mappings
aie "fixeu anu confineu to few paiameteis" iesulting in pooi musical fieeuom. A
few of the auvantages of this system compaieu to conventional methous aie:

24
0sei woiks uiiectly with the output without getting involveu in the complex mapping
algoiithm
System is iobust to non-lineai anu noisy input
Low computation ovei-heau
Neuial Netwoiks aie easy to tiain anu maintain
The authois concluue that the Neuial Netwoik mapping scheme pioviues an
easy anu fiienuly inteiface as well as the ability to geneialize the knowleuge.
(Nanuelis anu Busbanus 2uu4) uesciibe a system that uses evolutionaiy anu
aitificial life techniques foi the uevelopment of peifoimance mapping to aiu
ieal-time sounu manipulation. Insteau of foimalizing mappings between input
anu output, any mapping can be inuuceu using evolutionaiy foices applieu by
the usei.
They use as an example the uenephone system, which is uesigneu to facilitate
the inteiaction with viitual instiuments. Each input in the uenephone is useu as
a seeuing paient that cieates a geneiation of offspiing. Each seeuing paient is
mappeu a to specific output paiametei value, offspiing holu a mix of the paient's
input-output mappings values plus some ianuom mutation of these values.
0ffspiing aie scieeneu by the usei anu assigneu a ielative fitness ieflecting how
much they aie likeu by the usei. The new geneiation of offspiing is geneiateu;
howevei genetic opeiatois bias the next iounu of offspiing towaius the fittei
population. The cycle continues until the usei ueems one oi moie inuiviuual
offspiing to have a suitable mapping.
Nanuelis & Busbanus concluue that this technique is easy to leain anu piouuces
quick iesults compaieu to manual methous. 0seis noimally enu up with
satisfying mappings aftei a few cycles.
(Collins.&8 2u1u) expeiiment using Echo State Netwoiks that can be tiaineu to
link an input to an output. Echo State Netwoiks aie a "iecuiient neuial netwoik
that use a tiainable set of weights to exploit the uynamics of a ianuom ,-,0)#,
of inteiconnecteu neuions." Napping is achieveu by stimulating a specific output

2S
while ieceiving a paiticulai input pattein uuiing tiaining.
The authois concluue that piecise mapping is haiu to attain, but the Echo State
Netwoiks can help in ieuucing the uimensionality fiom a laige set of inputs.
Bowevei, the outcome was not paiticulaily successful because the system was
not able to iuentify specific input ielationships.
(Saltei . &8 2uu8)uesciibe an inteiactive uance theatei peifoimance entitleu
Schwelle that uses the Beibaitian system, wheie the uuiation anu stiength of
sensoiy input ueteimines the stiength of an "iuea". Inputs aie coupleu to vaiious
output layeis, each with a sounu object whose piesence is uepenuent on the
stiength of the cuiient input. Fiesh inputs caiiy moie weight than iecuiient
ones. Theie is no one-to-one mapping; the output ieflects both past anu piesent
stimulus.
The authois acknowleuge that the Beibaitian system alone uiu not piouuce
uiveise iesults anu useis thought the system was too static. Auuing a state
system in latei peifoimances maue the inteiaction cleaiei to auuience membeis.

BI SC0SSI 0N
(Saltei.&8 2uu8) pioviue a ciitical analysis on the existing mapping piactice.
They aigue that mapping flattens out the iichness of algoiithms. Since uigital
outputs uon't have any "ieal-woilu" couplings to inputs, aitificial coiielations
between the two aie funuamentally aibitiaiy. Contiollei-baseu mapping can be
valiu fiom a technical anu aitistic stanupoint, but is inauequate foi complex
sensoi-uiiven iesponsive enviionments. This limitation can be oveicome by
ueveloping systems that inuuce mappings out of inteiactions.
(Chauabe 2uu2) analyses the limitations of mapping between contiolleis anu
sounu vaiiables. Be aigues mapping woiks well foi simple contiolleis but
becomes unpiactical in complex systems. Possible substitutes aie fly-by-fly

26
systems which incoipoiate a seiies of if-then algoiithms that aie able to ieact
uynamically in changing mappings.
Chauabe concluues that mapping in not the iueal way to conceptualize the
ielationships between both complex anu unpieuictable inputs anu outputs.
(Booinbusch 2uu2), in contiast, is convinceu that the solution foi complex
systems is simple mappings. Biawing fiom a pievious stuuy by Bunt & Kiik
(2uuu), anu inteiviews with six composeis, Booinbush aigues that lineai, simple
mappings aie moie suitable foi complex uata stieams, while complex mappings
aie pievalent anu musically useful, but iuiosynciatic.
Fiom the point of view of uance-technology, (Bownie 2uu4) aigues that the teim
mapping is too abstiact to be useful. Statements conceining the teim noimally
"boiuei on the banal anu meaningless" anu is useu to justify any kinu of
aibitiaiy ielationship between input anu output. Quick coupling of sensoi to
output limits the "awesome potential" of algoiithms anu the numbei of
paiameteis to the selecteu solution.
Be pioposes an alteinative metaphoi foi ait-making calleu tbe oqentanu
extension to smait mapping baseu on unsupeiviseu machine leaining anu
aitificial intelligence. Infoimation in the aitificial intelligence agent tiavels
thiough thiee systems: perception, wheie the woilu is tiansfoimeu into coheient
symbols; motor, cooiuinates the actuatois' inteiaction with the woilu; oction,
baseu on the peiceptual anu motoi state, selects appiopiiate iesponses.
Be concluues that the agent metaphoi offeis feitile giounu foi giowing small
algoiithmic iueas anu unexpecteu inteiaction on possibilities.
(Winklei 199S) looks at gestuie uata as a compositional meuium in inteiactive
music systems. Be auvocates foi a uynamic anu intelligent system that iesponus
appiopiiately while encouiaging input spontaneity.

27
C0NCL0SI 0N
This section has iuentifieu woiks that help unueistanu the vaiious computei
meuiateu ielationships between multimoual inputs anu outputs. The peiceiveu
iole of the computei will waiiant static oi flexible inputoutput ielationships.
Fiaming a system as an extension of an analogic inteiface will invaiiably leau to
a moie iigiu mapping fiamewoik. In these situations, some uevelopeis may
consiuei mapping as pait of the composition, fuithei limiting the scope of the
iouting possibilities to a uesigneu inteiaction. Bevelopeis foi aitificial
intelligence systems incoipoiate algoiithms that maximize the computei's
potential. Nachine leaining auvocates aigue that instiument mappings aie
mostly aibitiaiy; although the seconu-oiuei highei level leaining mechanisms
may also seem aibitiaiy.
Inteiactions that encouiage open paiticipation such as installations can benefit
fiom complex mappings. Simple actions aie easiei to leain but aie less
inteiesting to useis. Auuience inteiaction with highei-level anu less immeuiate
events, inuepenuent of the mapping mechanism, will affect theii peiception of
involvement.
Although theie is ample ieseaich in simple, complex anu smait mapping
techniques, theie aie no methous foi evaluating theii implementation.
1.S Towaius Inteiactive
Nightclub anu Bance
}ockey
The use of motion captuie foi conveiting enviionmental uata into infoimation
foi peifoimances is helping to iethink the way computei music is composeu anu

28
peifoimeu fosteiing the collaboiation between aitists anu scientists. As
Kailheinz Bianuenbuig, Biiectoi of the Fiaunhofei Institute anu NPS co-cieatoi,
has saiu "we will see new foims of listening to music anu cieating music. |.j
Inteiactivity will blui the lines between listening, cieating music anu computei
games" (Bainuenbuig 2u1u). Bowevei as (Beitini . &8 2uu6; Futielle anu
Bownie 2uuS; Saltei . &8 2uu8; Bahn . &8 2uu1) pointeu out, most of these
systems have hau limiteu ieal woilu applications, anu have been appioacheu
with the goal of meiely "sonifying" the uata ueiiveu fiom the sensois.
Woiks by (Blaine anu Peikis 2uuu; Tahiioglu anu Eikut 2uu7; Felumeiei anu
Paiauiso 2uu7) with gioup computei music making suggest that fuithei woik
can be uone on implementing gioup music expiession tools in public venues.
Nightclubs aie populai, excellent multimeuia places foi usei paiticipation as
pointeu out by (Bayliss.&8 2uuS). Nonetheless, (uates.&8 2uu6) state that
little is known about how inteiaction occuis insiue these venues, anu cuiient
implementations of inteiactive systems within this context neglect the B} oi aie
limiteu to non-musical auuience-to-auuience inteiaction.
B}s play a ciucial iole as gate-keepei anu tienu-settei in the nightclub
enviionment howevei the limiteu bibliogiaphy on music inteiaction in uance
clubs, like the woik uone by (0lyate anu Bianciaiui 2uu1) uelibeiately siue step
the cuiient iole of the B}. I suspect that as long as uevelopeis ignoie the neeus of
B}s, implementations will not achieve a lasting footholu insiue clubs. The
peivasive piactice among uevelopeis in tinkeiing the B}'s set anu using
iuiosynciatic contiolleis contiibutes to the hesitation of B}s in auopting
inteiactive systems.
To pioviue a context-uiiven inteiaction the system neeus to have as a iefeience
musical elements such as key anu tempo. Pievious examples of inteiactive music
systems iely exclusively on pieselecteu oi composeu tiacks with all the musical
infoimation pioviueu *,#),#. Theie is no eviuence of these elements being

29
extiacteu in ieal-time using music infoimation ietiieval (NIR) fiom the B}s
music foi the puipose of inteiaction. This is because music infoimation ietiieval
ieseaich has not been able to iuentify anu auuiess issues in paiticulai usei
communities (Futielle anu Bownie 2uuS; Lesaffie et ol. 2uu8; Casey et ol. 2uu8).
Cuiient auvancements in NIR can pioviue this level of infoimation anu we shall
examine the implementation of thiee algoiithms in the fiist pioject calleu
Inteiactive Nightclub.
Sensois conveit enviionmental uata into infoimation that can be useu to
manipulate vaiious sounus. Within this context, mapping aims to pioviue a
musical iole to what is otheiwise a non-musical input. Napping tools like
(Nalloch et ol. 2uu7)'s Tbe Hopper pioviue a poweiful giaphic usei inteiface foi
mapping input paiameteis to sounu paiameteis. Bowevei ielationships aie still
limiteu to a numbei of uefault options, anu mappings aie meant to be static.
These featuies make it moie suitable uesigning specific instiuments. Nuch of the
ieseaich conceining this fielu has been uiiecteu towaius making mappings
easiei anu fastei, ouis has been in making peifoiming easiei anu fastei. This will
become appaient when we uelve into the contiol unit of Bance }ockey.
Like uigital instiuments, most sensois auu a level of abstiaction that wiuens the
gap between the bouy anu sounu geneiatoi. It is my belief that exploiing
authentic (peihaps intiinsic) ielationships between bouy anu uigital meuiathe
goal behinu many uigitally augmenteu uancetheatei peifoimancesiequiie an
accuiate SB iepiesentation oi avatai of the human bouy. It comes to no suipiise
then, that most of the woik uone on analyzing anu extiacting musical
infoimation fiom bouy movements (}ensenius 2uu7) (uouy anu }ensenius
2uu9) inteipolate uata fiom vaiious sensois to cieate at least a paitial but
human iepiesentation of the bouy.
Since motion captuie is geaieu towaius animating computei geneiateu
chaiacteis, pioviuing uetaileu SB uigital iepiesentations of the human bouy has

Su
been the focus of much of the uevelopment. 0nlike these systems, most sensois
useu foi music inteiaction cannot pioviue a holistic inteipietation of the bouy. In
these cases wheie a limb oi movement is substituteu by continuous oi uisciete
stieam of uata the aesthetic qualities of the captuieu featuies have to be ie-
inteipieteu. Theie aie thiee uocumenteu examples using Bance }ockey's Xsens
NvN motion captuie suit: The musician Piaga Khan useu it to contiol visuals
(Collins . &8 2u1u). uhent anu Ncuill 0niveisities set out to test mapping
stiategies anu gait iecognition, using only a few sensois not the full NvN system
(Naes.&8 2u1u; Nalozemoff anu Bepalle 2uu9).
Inteiactive Nightclub combines the uomains of Nusic Infoimation Retiieval
(NIR), Notion Captuie (NoCap) anu Bouy-music Napping to suggest a piactical
systemic appioach foi stimulating usei's musical expiession in nightclubs. It
incluues seveial installations that aggiegate pations' social behaviois anu
inuiviuual attituues towaius uancing in clubs enabling useis to cieate,
mouify, ieconfiguie anu shaie meuia in ieal time thiough live motion sensing
technologies anu online seivices.
Bance }ockey, a team composeu by Stle Skogstau
12
anu myself, iefocuses
sensois fiom simplistic action-to-sounu to contextualizeu aesthetic anu uiamatic
expiession. This exploiatoiy woik has been shown publicly as woik-in-piogiess
with the hopes of uiscoveiing the new possibilities affoiueu by making music
thiough uance. Be Quay is iesponsible foi peifoiming on stage anu composing
the music; Skogstau is iesponsible foi the haiuwaie anu softwaie. The music anu
choieogiaphy aie evei-changing anu theii main goal is to expiess anu impiovise
sonically vaiious actions happening on stage, fleshing out the stoiies of vaiious
chaiacteis as they tiavel thiough a woilu of sounus.



12
PhB canuiuate in fouiNs gioup - Nusic, Ninu, Notion, Nachines; 0niveisity of 0slo, Bepaitment of Infoimatics

_]

1 lntetuctie
ightclub
1
lntetuctie
ightclub
Fxtroctinq bormonic informotion witb
Pitcb Trocker. Storinq bormonic
informotion witb Probobility Toble.
0nJerstonJinq movement witb PerioJ
Anolysis. Copturinq movement witb
Wii remotes onJ comeros. Eow sounJ
is proJuceJ onJ monipuloteJ. Public
visuol effect.

_^

Figure 1- Patrons Dancing on the Bar Counter at Gabler

SS




2.1 Bevelopment
Buiing my masteis I staiteu to exploie using uiffeient motion captuie
techniques foi musical inteiaction. These exploiatoiy woiks mainly useu
commeicial off-the-shelf piouucts like Wii iemotes anu web cameias because
they weie easily accessible. The acquisition of new theoietical mouels about
computei meuiateu inteiaction shifteu the emphasis fiom aitistic installations
towaius social platfoims that inspiieu the cieation of mutual anu shaieu musical
expeiiences.
Initial woiks encompasseu selecteu anu sometimes composeu multimeuia
mateiials. Bowevei, with time it became appaient that this appioach limiteu the
scalability of the installation anu uiun't auapt to the immeuiate neeus to the
paiticipants. 0pen souice anu fiee softwaie like computei vision libiaiies, music
infoimation ietiieval exteinals anu mapping tools, along with theii stiong online
communities ueuicateu to the uissemination anu implementation of such woiks,
pioviueu feitile giounu foi ueveloping usei-uiiven mouels. Inteiactive Nightclub
compiises of auuitoiy anu visual installations uesigneu foi nightclubs that invite
pations to peifoim with usei-geneiateu multimeuia contents.

S4
Figure 3 -
Probabilitv table
2. 1. 1 N0SI C ANB N0TI 0N
Thiee pitch tiackeis weie useu to extiact haimonic content fiom the music
being playeu by the B} in ieal time: ~Seqment, ~Anolyzer anu ~zso.Ireqpeok.
Segment~, anu Analyzei~ (}ehan anu Schonei 2uu1) aie baseu on the fiJJle
algoiithm uevelopeu in 1988 at IRCAN (Puckette et ol. 1998) which estimates
the pitch anu amplituue of an incoming sounu, both continuously anu as a
stieam of uisciete notes. (Nalt anu }ouiuan 2uu8) uevelopeu ~zso.Ireqpeok at
Iicam anu it extiacts a list of fiequencies along with the amplituue of each. The
fiequency with the highest amplituue is useu to ueteimine the stiongest pitch.
The output of each pitch tiackei is a
NIBI note that is then ieuuceu to a
single octave pitch class. The iesulting
notes fiom each pitch tiackei aie then
tempoially stoieu as a stieaming list
of ten items. Whenevei a specific note
is piesent in both lists it gets selecteu
anu sent to a piobability table.
Theie weie two
piobability tables, illustiateu in Figuie S, which auueu selecteu
notes to the table while subtiacting olu ones. In the fiist table,
ianuom notes coulu be calleu using one of the Wii iemotes oi
the cameia. The seconu table stieameu the foui most piobably
notes.
The peiiou uetection algoiithm can extiact the slope, fiequency, apex anu
tiough, anu well as pieuict futuie apexes anu tioughs fiom lineai, peiiouic input
uata. The iesulting values can be useu to tiiggei notes oi to pioviue contiol
infoimation foi effects.
Figure 2 - Pitch class matching

SS
Input uata is analyzeu in a winuow size of 1u items anu when values go uphill a
peiiou is uetecteu. Input is constantly being compaieu to the pievious 1u
th
item
to pioviue slope infoimation. When the slope is biggei than a ceitain value a
new peiiou is uetecteu, anu will not be uetecteu again until aftei values go
uownhill.
The apex anu tiough of a peiiou is uetecteu using a uynamic thiesholu to
accommouate foi a wiue iange of values. All the acceleiation uata in one peiiou
is stoieu in a list that gets outputteu when a new peiiou is uetecteu. Lowei anu
highei thiesholu bounus aie maue by subtiacting u.2 fiom the highest value anu
auuing u.2 to the lowest value in the list. The input is compaieu to these two
thiesholus. New apex anu tiough values aie useu foi scaling input values to miui
continuous contiol values. Figuie 4 illustiates the peiiou analysis of a Wii
iemote's acceleiation uata. The wavy continuous black line iepiesents the Wii
iemote acceleiation input. When the values go uphill anu a new peiiou is
uetecteu a veitical line is uiawn. At the same time highei (puiple veitical line)
anu lowei (yellow veitical line) thiesholus aie cieateu fiom the pievious peiiou
uata. Input uata is compaieu to these thiesholus to uetect apexs anu tioughs
iepiesenteu by black uots above anu below the wave foim. If a peiiou is longei
than 4uu ms then the systems enteis into "slowpoke moue" which changes the
slope acceleiation analysis to smallei values anu makes the apex anu tiough
thiesholus moie sensitive.
Figure 4 - Wii Remote acceleration data period analvsis
apex uetecteu
tiough uetecteu
high thiesholu
low thiesholu
peiiou uetecteu
acceleiation uata

S6
Betecteu apexes tiiggei notes while scaleu acceleiation uata contiols a low pass
filtei. When an apex is uetecteu a ianuom note fiom the piobability table is
outputteu to a synthesizei. This allows the usei to bang notes while uoing abiupt
oi peiiouic movements with the Wii iemote. Foi miui continuous contiol, the
input is scaleu to the uetecteu apex anu then scaleu again to a u-127 miui iange.
2. 1. 2 N0TI 0N CAPT0RE
Two Wii iemotes weie connecteu by bluetooth to a Nac
PoweiBook u4 using the softwaie 0sculatoi
1S
. Wii iemote
acceleiation uata was senu thiough the 0BP 0SC piotocol
to the peiiou uetection algoiithm. 0ne of the Wii iemotes
tiiggeieu notes while the othei sent continuous contiol
uata.
Inteiactive Nightclub useu an optical motion captuie to contiol effects anu
tiiggei notes fiom the piobability table. The fiist event useu 0nbiiain's Fiie-i
Bigital Cameia, anu the seconu anu thiiu an 0ptiTiack FLEX:v1uuR2. Both these
cameias have high anu iesolution low
latency. 0nibiain's Fiie-i can see colois
anu blocks infia-ieu light, while
0ptitiack's FLEX:v1uuR2 sees only in
black anu white, has infia-ieu LEBs anu
iesponus stiongly to infia-ieu light.
The fiist event at .), useu 0nibiain's Fiie-i foi viueo input anu two
NaxNSP exteinals that sent contiguous contiol messages to sounu effects. The
giaphical usei inteiface alloweu the usei to ueciue when to enable the viueo
input to contiol the sounu effects. The exteinals consisteu of cv.jit.tiack (Pelletiei
2uu4) that tiackeu pixels fiom viueo stieams anu the exteinal b.fiameuiff which
calculateu the biightness uiffeience between consecutive fiames.

1S
http:www.osculatoi.net
Figure 5 - Wii Remote
Figure 6 - Unbirain Fire-i and OptiTrack
FLEX.J100R2

S7
0sing cv.jit.tiack, a pixel oi featuie fiom a viueo input can be selecteu anu
tiackeu to give its x, y position in a S6u by 24u pixel scieen. The x value toggleu
the fiequency of a low pass filtei the on anu off switch of a beat iepeatei effect in
Ableton Live 8. The y value contiolleu the fiequency of a vaiious filteis.
b.fiameuiff calculateu the aveiage uiffeience fiom a numbei of consecutive viueo
fiames, also calleu quantity of movement, anu useu the values to contiol a low
anu high pass filtei. If the movement was above a thiesholu a high pass filtei was
tiiggeieu anu if below a low pass filtei.
To wain paiticipants that they weie being followeu, an image of such as a spot
light was piojecteu on the tiackeu featuie.
In the seconu anu thiiu event at 6obler was iecoiueu with the 0ptiTiack cameia
anu hau thiee algoiithms that analyzeu motion in ieal-time. The fiist algoiithm
in Figuie 7 nameu "hip" cioppeu the viueo input to the hip aiea of the uancei,
calculateu the fiame uiffeience anu then the centioiu oi centei of mass. The x
axis of centioiu was useu to contiol the fiequency of a banu pass filtei.
The seconu algoiithm calleu "notes" analyzeeu the movements of paiticipants
anu tiiggeieu miui notes. Novement was measuieu by calculating the quantity of
motion, that is tiansfoiming the viueo input into a binaiy image anu then
calculating the sum of all active pixels. The viueo matiix passes fiist thiough a
Figure 7 - Three algorithms for motion analvsis. hip, notes and digital

S8
noise ieuuction algoiithm that bluiieu incoming matiices. If the activity went
past a thiesholu, a note was ianuomly playeu fiom the piobability table. To play
anothei note, activity hau to go below the thiesholu fiist anu then past it again.
The last algoiithm calleu "Bigital" toggleu vaiious sounu effects when maikeis
on the viueo backgiounu weie coveieuuncoveieu. Nine infia-ieu ieflective
maikeis weie positioneu evenly in a matiix on the wall wheie the cameia was
pointing at. The input image was conveiteu to a binaiy image with a high
thiesholu so that only the ieflective maikeis coulu be seen. When the usei was
in between the cameia anu the wall some of the ieflective maikeis weie
obstiucteu toggling the vaiious auuio effects.
2. 1. S A0BI 0 ENuI NE
NIBI notes anu continuous contiol values weie sent fiom the NaxNSP patch to
Ableton Live 8's synthesizeis anu sounu effects. These messages weie sent
thiough LoopBeSu, a softwaie that maue it possible to exchange NIBI messages
between uiffeient piogiams. Any type of synthesizei coulu be useu but the
sounu effects weie limiteu to filteis anu beat iepeateis. The set insiue Ableton
Live 8 consisteu of five tiacks to geneiate auuio: Tiack 1 playeu the choius
coming fiom the seconu piobability table anu applieu a banu pass filtei
contiolleu by the a Wii iemote; Tiack 2 playeu the notes fiom the othei Wii
iemote; Tiack S playeu the notes fiom the cameia; Tiacks 4 anu S uiu the final
mixuown anu speakei iouting foi each contiollei.
2. 1. 4 N0SI C
Nusic was playeu by the B} but coulu also be suggesteu to an online playlist
hosteu on uiooveshaik.com oi uuiing the night. The latei hau a playlist entitleu
"uablei" anu the cieuentials to log in anu change the play list weie available at
uablei's facebook events page anu my peisonal website. It was askeu that songs
be auueu, not iemoveu, anu that these belong to the "house" genie. Songs with
melouic content weie piefeiieu so that the pitch tiackeis coulu finu haimonic

S9
content foi the useis to play with.
2. 1. S vI S0ALS
visuals weie extiacteu fiom a cameia input anu mouifieu in ieal-time using
}ittei in NaxNSP. viueo feeu went to one of five uiffeient effects chosen
ianuomly eveiy 6u seconus. Some of the effects weie baseu on 0niveisity of
Califoinia's CNNAT computei vision objects
14
anu featuieu algoiithms such as
euge tiacing, blobs, negative anu fiame uiffeience. The output was sent to a
piojectoi.
2.2 Implementation
The Inteiactive Nightclub installation was piesenteu thiee times, once in
Poitugal anu twice in Noiway between Naich anu Septembei 2u1u.
2. 2. 1 FACT0R }
Aftei seeing some of my woik online the ownei of the bai Ioctor } inviteu me to
make an inteiactive night on Apiil 1u, 2u1u. The bai was situateu in the city of
vole Je Combro in the noith of Poitugal. Factoi } was fiequenteu mostly by
teenageis with a piefeience foi "tiance" music, anu opeiateu between 1uam anu
2am.
Set up
The night featuieu two Wii iemotes, the 0nibiain's Fiie-i Bigital Cameia anu a
hiieu B}. To lowei the amount of visual noise sustaineu by cameia, all
incanuescent lights weie tuineu off while foui black lights weie on anu
fluoiescent bouy paint anu biacelets weie uistiibuteu to the ciowu to cieate
tiackable maikeis along with the natuially floiescent featuies such as teeth anu
most white clothes. A spot light was piojecteu on the tiackeu peison to make

14
http:cnmat.beikeley.euu

4u
heshe awaie that heshe hau contiol ovei a sounu effect.
The sounus effects fiom cameia weie playeu fiom the iight speakei along with
the B}s music as shown in Figuie 8. A coloiful, fluoiescent 6 metei iibbon was
attacheu to each Wii iemote to pioviue a visual feeuback. Sounus fiom this
system along with the B}s music weie playeu fiom the speakei on the left.

2. 2. 2 uABLER 1
Buiing my stay at 0slo 0niveisity, Noiway, I appioacheu vaiious nightclubs
about the possibility of piesenting an inteiactive night. The nightclub &,
locateu in the city centei accepteu having it on a Satuiuay, Septembei 4, 2u1u.
uablei was one of the few nightclubs in 0slo that peimitteu people bellow the
age of 22, so the club was mostly fiequenteu by people ageu between 18 anu 21
with a piefeience foi billboaiu's "top 4u" uance hits. Buiing the uay the venue
opeiateu as a iestauiant. 0n this night I was the B}, howevei the playlist was
laigely a collaboiative effoit.


B} Booth
Bance Flooi
Bai
B} music
Wii-Nake-Nusic
B} Nusic
iBance
Figure 8 - Floor plan for Factor J

41
./*
The night featuieu the 0ptiTiack FLEX:v1uuR2 cameia, two Wii iemotes, usei-
uiiven visuals anu an open online playlist. The 0ptiTiack FLEX:v1uuR2 cameia
was situateu in the fai coinei of the uance flooi as seen on Figuie 9. The
uiscotheque helpeu builu a small pouium meant foi one peison to be visibly
elevateu fiom the iest. Reflective Infia-ieu maikeis weie placeu on the wall in
fiont of the 0ptiTiack FLEX:v1uuR2 cameia. Although I planneu in using vaiious
algoiithms, uue to time constiains only the quantity of motion algoiithm that
uetecteu movement anu tiiggeieu notes was implementeu. The two Wii iemotes
weie situateu in a moie calm aiea close to the tables.
In auuition to sounu the night also featuieu visual effects. 0sei-uiiven visuals
consisteu of 0nibiain's Fiie-i Bigital Cameia pointeu towaius a bai countei-a
populai uancing spot-anu a piojectoi facing a wall peipenuiculai to the bai. The
viueo feeu went thiu vaiious visual effects befoie being piojecteu. The website
giooveshaik.com hosteu the night's playlist anu the cieuentials to login anu auu
to the playlist weie auveitiseu on uablei's facebook page anu my peisonal
website.
2. 2. S uABLER 2
The seconu implementation was on a Satuiuay, Septembei 2S, 2u1u. The music
anu clientele was the same as on the fiist night.
./*
The same seivices on uablei 1 weie implementeu, with the auuition of a Bance
}ockey peifoimance, which was ie-nameu to Cyboig Peifoimance that night. I
peifoimeu the Cyboig Peifoimance on the bai countei at aiounu uu:Su, befoie
the club got packeu. The piece was only 6 min long anu featuieu a mixtuie of
sounu theatei, the peifoimei contiolling vaiious sounu effects that enhance a
piece; anu music, wheie the peifoimei's movements maue beats anu synths.

42
Thiee posteis weie placeu insiue the club next to each installation to pioviue
moie infoimation to pations about the inteiactive system.

2.S Results
2. S. 1 WI I REN0TES
The Wii iemotes weie useu moie at Factoi } than at uablei, piobably uue to theii
location anu visual appeal. Wheieas at Factoi } they weie locateu veiy close to
the uance flooi anu theii iibbons shineu veiy biight because to the black lights,
at uablei they weie placeu to the siue, in a moie ielaxeu aiea weie people sat
uown aiounu tables anu talkeu. The space at Factoi } was smallei anu the Wii
iemotes' sounus weie playing on one of the main louu speakeis, thus a usei
coulu be in the uance flooi anu still listen to the sounu effectsothei pations
uiu not seem botheieu by the sounus effects.
At uablei the iequiieu usei activity uiu not match that of the suiiounuing people
B} Booth
Bance Flooi
Bai
B} Nusic
Wii-Nake-Nusic
B} music
B} music
iBance music
Figure 9 - Floor plan for Gabler 1&2
iBa
nce
Pioj
ecti
on
Bai countei

4S
anu maue himhei feel obseiveu while hanuing an unusual uevice. Beie, the Wii
iemotes hau no holuei anu weie put close to the flooi which maue them less
visible. At Factoi }, The Wii iemotes weie initially tieu to the wall to stop them
fiom being stolen but weie latei fieeu to allow mobility.
The Bluetooth connection uioppeu thiee times at Factoi } because of the
uistance between the laptop anu the uevice. Theie weie also many othei
Bluetooth uevices piesent which coulu affect the connectivity. The batteiy lasteu
thioughout the whole night in all occasions. The iemotes themselves weie
iesistant anu withstoou haiu thiashing.
2. S. 2 CANERAS
Bue to the club lights anu ciowu movement, the cameia was not able to
efficiently tiack featuies at the Factoi }, but with an infia-ieu filtei it woikeu well
in uablei. The flooi at Factoi } was packeu but people weie mostly talking not
uancing. Since the algoiithms useu weie baseu on hoiizontal anu veitical
absolute positions, not much uata coulu be extiacteu fiom stationaiy pations.
Fuitheimoie, the piojection aleiting uanceis that they weie being followeu
inteifeieu with the tiacking mechanism. The events at uablei tackleu this
pioblem by fiist filteiing out visible light, which incluueu most club lights, anu
viueoing a smallei inuiviuual aiea. Quantity of movement pioveu to be a ieliable
algoiithm to uetect motion with only a few false-positive notes iegisteieu by
club lights. Bowevei, in both cases, most pations uiu not unueistanu the
intention of the installation because it lackeu visual feeuback. Even when a
postei with infoimation was auueu, not eveiyone unueistoou the coiielation
between movement anu sounu. Those that uiu unueistanu weie initially
captivateu by the sounus they piouuceu, most howevei, soon lost inteiest with
the limiteu possibility of sounu effects.


44
2. S. S vI S0ALS
Although the bai countei was a populai uancing spot foi pations, the bai
tenueis anu pations on the othei siue of the bai weie impossible to iemove fiom
the pictuie. The viueo captuie aiea uiu not have a backgiounu wall anu so the
images tuineu out to look a bit confusing making it haiu to single out a uancei. It
woulu have been bettei to point the cameia to a place with a fixeu backgiounu,
but this is always uepenuent on the inteiioi of the club. The fact that the visuals
weie off to the siue at a 9u uegiee to the bai countei anu not uiiectly in fiont of
them, contiibuteu to a lowei feeuback between uanceis anu the piojection.
2. S. 4 CYB0Ru PERF0RNANCE
Cyboig Peifoimance was the name given to the Bance }ockey peifoimance at the
uablei nightclub. Since the bai uiu not have a stage, only a B} booth, the
peifoimance was piesenteu on the bai countei. The limiteu flooi space was not
much of a pioblem since the choieogiaphy uiu not iequiie moving aiounu too
much, howevei the uistance between the countei anu the ceiling was of 1.6
meteis, making some of the sounus anu one tiansition involving iaiseu aims
impossible to executeStle hau to manually uo the tiansition. Because of the
flooi level change fiom the calibiateu floou to the bai countei, foui of the initial
tiansitions involving hanu anu foot height weie acciuentally anu consecutively
tiiggeieu. To pievent this in the futuie, a uebounce measuie can be implementeu
to stop tiansitions fiom being tiiggeieu consecutively aftei uetection. Not
eveiyone in the auuience was awaie that theie woulu be a peifoimance. It is
likely then, that most of the auuience membeis uiu not unueistanu what was
happening.
The initial sciipts weie veiy theatiical, which uiu not suit well the night club
enviionment since theie was low visibility because of the light anu location of
the bai countei. Latei sciipts weie moie musical anu hau a stiongei beat. The
lack of a stage also incieaseu the chances of inteifeience fiom the auuience.

4S
Buiing the final moments of the peifoimance a pation iaiseu a shot glass anu
insisteu the peifoimei to uiink it.
2. S. S N0SI C
The online playlist hosteu at uiooveshaik.com enueu up with moie than 1uu
song contiibutions. This openness iesulteu in a iich playlist that was suitable foi
all in the club anu since paiticipants geneiateu itthe uance flooi was packeu
all night. The NIR softwaie peifoimeu well anu no off tune notes weie obseiveu.
2. S. 6 RECEPTI 0N
Pations, B}s, nightclub staff, anu obseivation pioviueu feeuback as to how the
systems weie being useu. Finuings suggest that simple mappings, fast sonic
iesponse anu visual feeuback woikeu best. 0seis founu the Wii Remotes veiy
easy to manipulate, anu the movement to sounu ielationship cleai. The
algoiithm was iobust anu woikeu gieat thioughout the night, anu useis coulu
heai with piecision the coiielation between sounu anu movement. The event,
uebuteu in public foi the fiist time, coulu have some impiovements. Noie
infoimation about the space befoie anu uuiing the event, otheiwise people
uiun't expect anything unusual fiom it. This goes to the next point, that if useis
uon't know that theii movements aie tiiggeiing sounus, they will finu the notes
coming fiom the speakei puzzling anu will tampei with the speakei iepiouucing
them. This speakei shoulu be beyonu aims ieach foi the same ieason. The uance
pouium has to be soliuly fixeu on the giounu; otheiwise it will shift away fiom
the cameias' focus uue to all the movement. Lastly, as the night piogiesseu, the
uance flooi got uimmeu, so it woulu be useful to have an automatic light
aujustment mechanism in the motion captuiing system to avoiu having to uo it
manually.
The last night at uablei was too uemanuing on me because theie weie so many
systems to monitoi. The music suffeieu fiom uiops because my attention was
focuseu elsewheie, especially in the beginning when the Cyboig Peifoimance

46
suit was being put on anu the softwaie initializeu. The cameias, Wii iemote anu
visual installations weie set up in thiee houis. Nost of the mateiials hau to be
boiioweu fiom the fouiNs lab.
2. S. 7 BI SC0SSI 0N
Although Inteiactive Nightclub manifests many of the web 2.u usei-uiiven
mouels, it still envisions useis woiking in a centializeu system thiough
teiminals, inteifaces, anu installations. Nany of the tools piesenteu heie aie
uesigneu top-uown with all possible types of inteiaction specifieu befoiehanu.
Thus holistic infoimation about bouy anu music has still to be ievieweu by a
human agent to have any value insteau of being automatically manipulateu by
machines. If we bioauen the netwoik possibilities to uisguiseu, embeuueu,
unsupeiviseu machines we can look foiwaiu to a musical system that not only
that ueuuces mappings fiom inteiaction but most impoitantly can connect useis
acioss the woilu to cieate, shaie, anu combine music anu uance in ieal-time. The
next chaptei Bance }ockey piesents a step towaiu ubiquitous computing in
music thiough sensoi suits that can compiehenu the semantic ielationships
between music anu uance.

`c
1 unce leckeg
1
unce
leckeg
Eow to stort tbe Xsens sensor suit.
Communicotion between sensors onJ
computers. Hokinq numbers
unJerstonJoble witb tbe cookinq toolbox.
0rqonizinq onJ composinq o performonce
witb tbe control unit. Eow sounJ is
proJuceJ onJ monipuloteJ. Wbot to Jo
wben onstoqe.

`d

Figure 10 - Dance Jockev at Jerdikt Conference

49


S.1 Bevelopment
The ieal powei behinu the semantic ielationships between music anu movement
will be iealizeu when people weai smait sensois oi agents that collect
infoimation, piocess it anu shaie the iesults with othei people oi othei things. A
typical agent can iun without human supeivision anu uses uata (enviionmental,
kinetic, etc.), with the help of othei agents, to bettei its chances of accomplishing
a goal set by the usei. 0ne can expect the effectiveness of agents to inciease
exponentially as moie people use them (oi things) anu intelligent enviionments
become available. These simple inteiactions between sensois can contiibute to
the emeigence of complex musical systems. Insteau of having haimonic iules
like in Inteiactive Nightclub that uo not necessaiily geneiate anything new but
meiely point out consistencies in the cuiient composition paiauigm, netwoikeu
embeuueu sensois will be the stiuctuie foi an oiganizeu, puiposeful activity that
encouiages moment-by-moment impiovisation among a laige numbei of
options.
In oiuei to achieve this level of inteiaction between weaiable machines, we neeu
a suit that is composeu of sensois that have viitual iepiesentations in a netwoik.
Togethei with Stle Skogstau, a PhB canuiuate at 0slo 0niveisity who uevelopeu
most of the Xsens softwaie anu netwoik optimizations uesciibeu heie, we
cieateu a system foi playing music with a multi-sensoi suit that uigitalizes all
bouy limbs in space. This chaptei piesents pieliminaiy woik towaius
autonomous, inteiconnecteu uevices that will assist the evolution of electionic
music as a whole.

Su
S. 1. 1 SET0P
The Xsens NvN suit consists of 17 ineitial NTx sensois, which aie attacheu to
key aieas of the human bouy. Each sensoi consists of a SB gyioscope, SB
acceleiometei anu magnetometei. The iaw signals fiom the sensois aie
connecteu to a paii of Bluetooth 2.u baseu wiieless tiansmitteis that tiansmit
the iaw motion captuie uata to a paii of wiieless ieceiveis. The total weight of
the suit is appioximately 1.9 kg anu the whole system comes in a suit case with
the total weight of 11 kg (Roetenbeig.&8 2uu9).
The uata fiom the Xsens NvN suit is feu to the NvN softwaie engine that uses
sensoi fusion algoiithms to piouuce absolute oiientation values, which aie useu
to tiansfoim the SB lineai acceleiations to global cooiuinates. These in tuin aie
tianslateu to a human bouy mouel which implements joint constiaints to
minimize integiation uiift. The Xsens NvN system outputs infoimation about
bouy motion by expiessing bouy postuies sampleu at a iate up to 12uBz. The
postuies aie moueleu by 2S bouy segments inteiconnecteu with 22 joints. The
Xsens company oeis two possibilities of using the NvN fusion engine: the
Winuows baseu Xsens NvN Stuuio anu a softwaie uevelopment kit calleu Xsens
NvN SBK (Roetenbeig.&8 2uu9).
The Xsens NvN system outputs infoimation about bouy motions by expiessing
bouy postuies, sampleu at a iate up to 12uBz, thiough a bouy mouel that
consists of 2S bouy segments inteiconnecteu with 22 joints. As it can be seen in
Figuie 11 the Xsens uata has to be piocesseu thiough many mouules befoie
piouucing any sounu. Staiting with captuiing the Xsens sensoi uata using the
NvN Stuuio softwaie, then piocessing anu cooking the uata, followeu by the
contiol unit that manages the sounus in the Ableton Live softwaie.
The Xsens NvN technology can be uiviueu into two paits. Fiist, the sensoi anu
communication haiuwaie aie iesponsible foi collecting anu tiansmitting the iaw
sensoi uata. Seconu, these uata aie tieateu by the Xsens NvN softwaie engine,

S1
which inteipiets anu ieconstiucts the uata to full bouy motion while tiying to
minimize uiift.
Theie aie thiee main suit conguiations; full bouy, uppei bouy oi lowei bouy.
When the suit is piopeily conguieu, calibiation is neeueu to initialize the
position anu oiientation of the uieient bouy segments. When we aie satiseu
with the calibiation the system can be useu to stieam the motion uata to othei
applications in ieal-time oi peifoim iecoiuings foi latei playback anu analysis.
Bow piecise one neeus to peifoim the calibiation may vaiy. We have founu that
so-calleu N-pose anu T-pose calibiations aie the most impoitant. A hanu touch
calibiation is iecommenueu if a goou ielative position peifoimance between the
left anu iight hanu is wanteu. Recalibiation can be necessaiy when the system is
useu ovei a longei peiiou of time. It is also possible to input bouy measuiements
of the tiackeu subject to the NvN engine, but we have not investigateu if this
extia calibiation step impioves the quality of uata foi oui use. In oui expeiience,
setting up the system can easily be uone in less than 1S minutes compaieu to
seveial houis foi IiNoCap systems (Cloete anu Scheffei 2uu8).
To be able to use the Xsens NvN system foi musical inteiaction, we neeu a way
to communicate the uata that the system senses to oui musical applications. It
was natuial to implement the 0SC stanuaiu since the Xsens NvN system oeis
motion uata which is not easily ielateu to NIBI signals. 0SC messages aie also
potentially easiei to inteipiet since these can be wiitten in a human ieauable
foim.

S2
Contiol 0nit
Ableton Live 8

Xsens Suit

Cooking
NvN Stuuio
Blue
tooth
0BP
0BP
0sc
NIBI
0SC


Full kinematics of 2S
segments (position, velocity,
acceleiation, oiientation,
angulai velocity anu angulai

Lineai composition built on
NaxNSP using the mouules
cueNanagei, ctlout anu oscout
to contiol the auuio softwaie
Real Time Notion Captuie
Toolbox. Tiansfoim motion
infoimation into contiol
Sounu manupulation
anu instiumentation
6 uegiees of fieeuom
tiacking of motion
17 NTx inteitial tiackeis
Seconu computei
Fiist computei
Figure 11 Xsens data flow

SS
S. 1. 2 NETW0RK
Low anu stable latency is an impoitant concein foi ieal-time musical contiol
(Wessel anu Wiight 2uu2). This is theiefoie an impoitant issue to consiuei when
uesigning oui system. 0nfoitunately, iunning softwaie anu senuing 0SC
messages ovei noimal computei netwoiks oeis inauequate suppoit foi
synchionization mechanisms, since stanuaiu opeiating systems uo not suppoit
this without ueuicateu haiuwaie (Schmeuei.&8 2u1u). In oui expeiience, foi
one to get low latency fiom the Xsens system, the softwaie neeus to iun on a fast
computei that is not ovei-loaueu with othei uemanuing tasks. But how can we
fuithei minimize latency.
Fiom Figuie 11 we can iuentify foui vaiious computationally uemanuing tasks
that the uata neeus to tiaveise befoie enuing up as sounu. If these tasks aie
especially uemanuing, it may be benecial to uistiibute these computational
loaus to uieient computeis. In this way we can pievent a computei fiom
sueiing too much fiom computational loau, which can leau to a uiamatic
inciease of latency anu jittei. This is possible with fast netwoik links anu a
softwaie aichitectuie that suppoits the uistiibution of computational loaus.
Bowevei, it comes at the cost of extia netwoik oveiheau.
The amount of uata sent thiough a netwoik wills paitly be ielateu to the
expeiienceu netwoik latency. Foi instance, we shoulu tiy to keep the size of the
0SC bunules lowei than the maximum netwoik buei size
1S
, if the lowest
possible netwoik latency is wanteu. If not, the bunule will be uiviueu into seveial
packages (Schmeuei.&8 2u1u). To achieve this, it is necessaiy to iestiict the
amount of uata sent. If a laige vaiiety of uata is neeueu, we can cieate a uynamic
system that tuins uieient uata stieams on when neeueu.


1S
Nost Etheinet netwoik caius suppoit 1Suu bytes. Those suppoiting }umbo fiames can suppoit up to 9uuu
bytes.

S4
S. 1. S C00KI Nu
The Xsens NvN oeis a wiue iange of uieient uata to oui system. If we use the
netwoik stieam fiom the NvN Stuuio, each fiame contains infoimation about
the position anu oiientation of 2S bouy segments. This yielus in total 1S8
oating points numbeis sampleu at a iate of 12uBz. Noie uata is available if one
insteau uses the NvN SBK as the souice. Bieient tiansfoimations anu
combinations of the uata can be of inteiest, such as calculating uistances oi
angles between bouy limbs. Fuitheimoie, we can uieientiate all the above
mentioneu uata to get piopeities like velocity, acceleiation, quantity of motion
anu jeik. Filteis can be implementeu to get smoothei uata oi to emphasize
ceitain piopeities. With pattein iecognition techniques we have the potential to
iecognize even highei level featuies (Qian et al. 2uu4).
Theie aie two options foi using the Xsens NvN motion uata in ieal time, eithei
we can use the Xsens Stuuio's 0BP netwoik stieam, oi make a ueuicateu
application with the SBK. The implementation must also suppoit a way to
eectively cook the uata. We began using the 0BP netwoik stieam since this
appioach was the easiest way to stait using the system.
HvN Network Streom 0npocker in Hox,HSP
A NX} }ava uatagiam unpackei was maue by Stle foi NaxNSP, but the
implementation was shown to be too slow foi ieal-time applications. Though a
ueuicateu Nax exteinal (in C++) woulu piobably be fastei, this aichitectuie was
not chosen foi fuithei uevelopment since NaxNSP uoes not, in oui opinion,
oei an eective uata cooking enviionment.
StonJolone Botoqrom 0npocker onJ Cooker
We wanteu to continue using the Xsens Stuuio's 0BP netwoik stieam, but with a
moie poweiful uata cooking enviionment. This was accomplisheu by
implementing a stanualone 0BP uatagiam unpacking application. The

SS
piogiamming language C++ was chosen since this is a fast anu poweiful
computational enviionment. With this implementation we can eithei cook the
uata with self-piouuceu coue oi available libiaiies. Both iaw anu cookeu uata
can then be sent as 0SC messages foi fuithei cooking elsewheie oi to the nal
sounu engine.
Xsens HvN SBK lmplementotion
The Xsens NvN softwaie uevelopment kit oeis moie uata uiiectly fiom the
NvN engine compaieu to the 0BP netwoik stieam. In auuition to position we
get: positional anu angulai acceleiation, positional anu angulai velocity anu
infoimation about the sensoi's magnetic uistuibance. Eveiy time-fiame is also
maikeu with a time stamp that can be useful foi analysis anu synchionizing.
Anothei benet is incieaseu contiol since we aie uiiectly communicating with
the NvN engine anu not listening foi 0BP packages.
The uiawback with the SBK is that we lose the benet of using the usei fiienuly
NvN Stuuio anu its u0I. We implementeu a teiminal application with the SBK
that suppoits the basic Xsens featuies (calibiation, play-back, etc.). Since the
application is getting uata uiiectly fiom the NvN engine we can save netwoik
oveiheau by cooking them in the same application befoie senuing them as 0SC
messages. We also implementeu a function that can senu the motion uata in the
same uata foimat as the Netwoik 0BP Batagiam stieam. This stieam can then be
openeu by NvN Stuuio to get ieal-time visual feeuback of the NoCap uata.
Since the solution piesenteu in the SBK oeieu a fast enviionment foi uata
cooking, anu let us use the usei fiienuly NvN Stuuio, we have mainly useu this
appioach in oui woik. We latei uiscoveieu that the netwoik stieam oeieu by
NvN Stuuio sueis fiom fiame loss when uiiven in live moue, aecting the NvN
unpackei anu the NaxNSP uatagiam unpackei. Because of this we plan to focus
on oui SBK implementation in the futuie.

S6
An auueu auvantage is that we no longei neeu to uieientiate the segments'
positional uata to be able to get piopeities like velocity anu acceleiation, since
the SBK oeis this uiiectly fiom the NvN Engine. These uata, especially the
acceleiation, seem to be of a highei quality since they aie computeu uiiectly on
the basis of the Xsens sensois anu not uieientiateu fiom estimateu position
uata as shown in Figuie 2.16.
S. 1. 4 LATENCY EXPERI NENT
To be able to measuie the typical expecteu latency in a setup like that of Figuie
11 we peifoimeu a simple expeiiment with an auuio iecoiuei. 0ne laptop was
iunning oui SBK implementation anu sent 0SC messages containing the
acceleiation of the hanus. A patch in NaxNSP was maue that to tiiggei a simple
impulse iesponse if the hanus' acceleiation hau a high peak, signifying two
moving hanus colliuing to a suuuen stop. The time uieience between the
acoustic hanuclap anu the tiiggeieu sounu shoulu then inuicate the typical
expecteu latency foi the setup.
In expeiiment 1 the NaxNSP patch was iunning on the same laptop as the SBK
(Bell Winuows 7.u Intel iS baseu laptop with 4uB RAN). In expeiiment 2 the

16
The system that tiies to minimize positional uiift piobably contiibutes to a mismatch between uieientiateu
positional uata anu the velocity anu acceleiation uata fiom the NvN engine.
Figure 12 - Dierence between the second derivative of the position data versus the acceleration
data obtained directlv from MJN engine (SDK).

S7
patch ian on a sepaiate Nac laptop (NacBook Pio 1u.6.6, 2.66 uBz Buo with 4uB
RAN) anu ieceiveu 0SC messages thiough a uiiect ubit Etheinet link.
Expeiiment S was iuentical to 2 except that the Nac was ieplaceu with a similai
Winuows baseu laptop. All expeiiments useu the same iewiie sounucaiu,
Euiiol FA-1u1.
The iesults aie given in Table 1 aie baseu on Su measuiements each which weie
manually examineu in auuio softwaie. The stanuaiu ueviation is incluueu as an
inuication of the jittei peifoimance. We can concluue that expeiiment 2 has the
fastest sounu output iesponse while expeiiments 1 anu S inuicate that the
Etheinet link uiu not contiibute to a laige amount of latency.
The Xsens NvN system oeis a uiiect 0SB connection as an option foi the
Bluetooth wiieless link. We useu this option in expeiiment 4, which was in othei
ways iuentical to expeiiment 2.
Experiment min mean max std. dev.
1 Same Win Laptop S4 66.7 1u7 12.8
2 USC to Mac 41 S2.2 8S 8.4
3 USC to Win S6 68 1uS 9.8
4 USC to Mac - USB 28 S7.2 S6 6.9
Table 1 - Statistical latencv results (in milliseconds).

Results inuicate that the uiiect 0SB connection is aiounu 1u-1S milliseconus
fastei anu has a lowei jittei peifoimance than the Bluetooth link. The uppei
bounuaiy foi "intimate contiol" has been suggesteu to be 1ums foi latency anu
1ms foi its vaiiations (jittei) (Wessel anu Wiight 2uu2). If we compaie the
bounuaiy with oui iesults, we see that oveiall latencies aie too laige anu that
the jittei peifoimance is even woise. Bowevei, in oui expeiience, the system is

S8
still usable anu uepenuent on the uesigneu action-sounu mappings.
S. 1. S C0NTR0L 0NI T
The contiol unit makes uecisions baseu on the cookeu uata anu gives
instiuctions to auuio softwaie. It was built on the NaxNSP
17
enviionmenta
visual piogiamming language populai foi uesigning multimeuia inteiactions.
Nuch like songs in an album, a script is the technical teim useu heie to iepiesent
a self-inclusive sonic piece. A Bance }ockey peifoimance was compiiseu of
vaiious sciipts oiganizeu in sequence befoiehanu. Anu like many compositions,
the sciipt was uiviueu into vaiious sections, which we auuiesseu as cues.
The system was moueleu aftei the finite-stote mocbine that uefines thiee
piocesses in eveiy cue of a sciipt: 1) Entiy actions, 0SC anu NIBI commanu
messages; 2) Input actions, commanus contiolleu by the Xsens input; S)
tiansitions, a set of conuitions that if met by the Xsens input will tiansition to the
next cue. Entiy actions aie limiteu to 0SC commanus. Input actions can be NIBI
continuous contiol messages, NIBI events, oi 0SC commanus.
The contiol unit went thiough thiee ievamps. Each time the goal has been to
uevelop a flexible platfoim that coulu beai quick changes in the sciipt. The fiist
platfoimance pioviueu a matiix foi cues anu sounu effects. Each cue calleu up a
set of sounu effects that communicateu with the sounu engine. The platfoim was
buggy, confusing anu only suitable to a limiteu amount of inteiaction
possibilities.
The next platfoim was moie mouulai anu useu the }amoma fiamewoik
18
.
Actions weie encapsulateu into unique 0SC anu NIBI mouules which coulu be
tuineu on anu off uepenuing on the cue. Tiansitions weie inuepenuent of the
actions anu piogiammeu in a sepaiate patch. Although this latei veision was
moie flexible many of the commanus weie still haiuwiieu into the mouules thus

17
http:cycling74.com
18
http:www.jamoma.oig

S9
iequiiing a mouule foi each action that leu to a clutteieu anu confusing visual
enviionment.
The thiiu veision hau all the entiy actions, 0SC anu NIBI mappings anu
commanus wiitten in a text file offeiing a common set of options. This contiol
unit useu thiee mouules: cueHonoqer, oscout anu the finite-stote mocbine. The
cueNanagei haiboieu anu manageu all the sciipts anu cues in the text file.
Figuie 1S illustiates the commanus anu mappings foi the "pei\ cussion" cue.
The entiy action is to play a specific sounu clip anu mute a tiack in the Ableton
Live softwaie, anu ioute the 1
st
continuous contiol values, anu NIBI events
peifoimeu in the input actions to channel 1 anu numbei 1.
KEYC0E "peicussion"


C0NFIu oscout 99 1
#
########################
#
# Nouule oscout
oscoutu_uesciiption u
oscout1_1_liveosc playclip 1 2
oscout1_2_liveosc mute tiack 2
oscout1_S_liveosc u
oscout1_4_liveosc u
oscout1_S_liveosc u
oscout1_6_liveosc u
oscout1_7_liveosc u
oscout1_8_liveosc u
oscout1_9_liveosc u
oscout2_1_cc_channel 1
oscout2_1_cc_numbei 1
oscoutS_2_cc_channel u
oscoutS_2_cc_numbei u
oscoutS_S_cc_channel u
oscoutS_S_cc_numbei u
oscout4_1_note_channel 1
oscout4_2_note_channel u
oscout4_S_note_channel u
oscoutS_next_sciipt u
oscoutfieeze u
#
########################

Figure 13 Command options for the 'percussion` cue
Name of cue









0SC entiy actions










NIBI event mappings

Bisable all the above
commanus

Besciiption of cue
NIBI continuous
contiol mappings
Sciipt to be loaueu on
next cue

6u
All the commanus in the cueNanagei aie sent to the oscout mouule that then
piocesses the commanus anu foiwaius the 0SC anu NIBI messages to the auuio
engine, Ableton Live. 0scout can also ieau anu leain paiametei changes in
Ableton Live. When piompteu to leain, oscout openeu a five seconu winuow
wheie it ieau the changes happening in Ableton Live anu assigneu the 0SC path
to a &#0)-commanu.
Figuie 14 illustiates the finite-state machine patch with foui cues, the cuiient
active calleu "peicussion" is highlighteu in ieu. Napping algoiithms aie stoieu in
the yellow patch calleu "p map". Tiansition conuitions aie wiitten in the puiple
patch nameu "p tians". Aftei meeting the tiansition conuitions in the
"peicussion" it will go to the "peicussion uistoit" cue. Nappings, tiansitions,
Cues anu uestination cues can be changeu any time with a single click.


61

S. 1. 6 A0BI 0 ENuI NE
All the sounus foi the peifoimance weie geneiateu anu manipulateu in Ableton
Live 8, a commeicial uigital auuio woikstation specially uesigneu foi live
peifoimances. Ableton live 8 was contiolleu via NIBI events, NIBI continuous
contiol, anu 0pen Sounu Contiol (0SC). Ableton Live 8 by natuie uoes not
accepts 0SC messages, so a Siu paity extension calleu Live0SC
19
hau to be
installeu sepaiately to enable this featuie. Bowevei, we expeiienceu
consiueiable latency with Live0SC so time ciitical event like synth notes, sounu
clips anu effects manipulation hau to be opeiateu via NIBI.
The peifoimance was oiganizeu in cues, each containing sounu effects, synths

19
http:liine.netlivecontiolableton-liveapiliveosc
Figure 14 - Finite-state machine
Cue
Input actions
foi this cue
Input conuitions
foi tiansition
Bestination cue
Cuiient Cue

62
anu othei sounu geneiating uevices. As the peifoimance piogiesseu, we moveu
sequentially fiom one cue to the next. A cue coulu have vaiious inteinal
opeiations that affecteu Ableton Live 8 such as muting, iaising volume, alteiing
tempo, anu playing a clip. AlLhough mosL of Lhese could be done Lhrough CSC, we
declded Lo lmplemenL anoLher 3rd parLy exLenslon called Clyphx
20
. A Clyphx lLem
lnslde AbleLon Llve 8 can do many operaLlons lnLernally wlLhouL our lnLerference.
Any cllp feaLurlng a "[]" as a name ls a Clyphx lLem LhaL has operaLlons embedded on
Lhe name.

Figuie 1S illustiates a poition of the visual enviionment in Ableton Live 8.
Columns iepiesent tiacks while iows iepiesent cues. In the initial, top cue, the
fiist tiack contains a sounu clip (white) as well as a Clyphx opeiation "monitoi
in". The seconu tiack has also contains a sounu clip (gieen) with a Clyphx
opeiation "monitoi auto". Sounus in the 2nu tiack coulu be manipulateu via NIBI
continuous contiol anu the 4th tiack hau a synthesizei accepting NIBI notes.

2u
http:beatwise.pioboaius.cominuex.cgi.boaiu=piouuction&action=uisplay&thieau=716

Figure 15 - Part of the Ableton Live 8 set

Clyphx
comman
Tiack with effects
contiolleu by NIBI CC
anu Live0SC Tiack iecieving NIBI notes
Cues aie oiganizeu in
ueceuent oiuei

6S
S. 1. 7 0SC BEvI CE
In oui last peifoimance at NINE 2u11, we ueciueu to get iiu of NIBI altogethei
anu auopt insteau only the 0pen Sounu Contiol (0SC) piotocol. Ableton Live 8
peimits the inseition of NaxNSP patches uiiectly in the set. In this appioach
useu at NINE 2u11, notes, tiiggeis, effect contiols anu instiuctions aie sent with
a unique appenueu 0SC auuiess thiough 0BP to NaxNSP uevices insiue
Ableton Live 8 tiacks. The 0SC auuiess is wiitten on the Finite-state machine
patch anu a matching auuiess is wiitten on the 0SC uevice to ieceive the uata.
The majoiity of these NaxNSP uevices uo not piouuce the sounu manipulation
themselves but contiol vaiious othei aujacent music uevices. In Figuie 16 we
can see that numbeis appenueu with one of the thiee 0SC auuiesses wiitten in
the blue box will contiol one of the thiee paiameteis of a banu pass filtei.
Incoming uata is automatically scaleu to a u-127 value so it can woik with NIBI
baseu uevices.
S. 1. 8 S00NB
A peifoimance oscillateu between theatiical sounu effects anu musical
compositions. A sciipt iangeu fiom 1 to S minutes anu caiiieu a specific theme
thioughout. The majoiity of sounus weie assembleu anu composeu befoie
having the uancei animate its vaiious paits anu then mouifieu uuiing ieheaisals.
Nany of the sounu effects weie founu in www.fieesounus.com a collaboiative
Figure 16 - OSC device that can receive three different addresses

Selecteu 0SC
auuiess
Effect being
contioleu
Incomming
minmax
bounus

64
uatabase of Cieative Commons licenseu sounus. The cieative piocess staiteu
with a biain stoiming session on what type of effect oi impiession we woulu like
to achieve. Bo we want to play a song. If so, what kinu of music. 0i uo we want
to tell a stoiy. What sounus anu movements woulu convey the message.
Theatiical inteiactions focuseu on telling a stoiy anu chaiactei uevelopment.
Beie, sounus followeu the movements of the peifoimei, augmenting sonically
actions that weie happening on stage. The musical inteiaction focuseu on the
instiumentation of the bouy wheie the main piioiity was music coheience anu
oiganization ieflecteu by the sciipts' sounus anu choieogiaphy.
Aftei collecting anu oiganizing the sounus envisioneu foi the sciipt the next step
was ueciuing how anu what sounus to contiol. A plethoia of paiameteis ianging
fiom notes to effects wheie available in the many uevices insiue Ableton Live 8,
ueciuing what to contiol uepenueu on the sonic objective anu visual aesthetics.
S. 1. 9 CB0RE0uRAPBY
The aesthetics of the peifoimei's movement on stage weie a constant concein.
When the focus was telling a stoiy the acting took pieceuence ovei the sounu
uesign. When the focus was music, bouy movements hau to auapt to the
composition. The timing in the choieogiaphy was not veiy stiict, in each cue
theie was impiovisation. Because motion was coupleu with sounu effects anu
cue tiansition conuitions, some of the sequence of movements hau to eventually
follow a ioutine foi the piece to piogiess.
The choieogiaphy was limiteu to the limbs since the Xsens NvN only captuies
ielative limb movements, not absolute cooiuinateu on stage. Aftei the sciipt
invisible sounJs mentioneu pieviously, in the musical sciipt robot the uancei's
bouy poses sent notes to two synthesizeis. Since a melouy foi the synthesizei
hau alieauy been establisheu, theie was a sequence of poses that neeueu to be
followeu to successfully play the whole melouy.

6S
S. 1. 1u A0BI ENCE
Since we weie piesenting a novel, anu foi some unknown, way to contiol sounus,
it was impeiative to make the auuience unueistanu what was happening on
stage. The peifoimance by itself was not enough; we believeu that letting the
auuience know that the sounus weie being contiolleu by the peifoimei woulu
eniich the expeiience. We askeu ouiselves quite often: Eow obvious sboulJ tbe
moppinqs be? We noticeu that obvious coupling between action anu sounu coulu
seem uull anu limit the cieative potential, while abstiact couplings hau the
potential of leaving the auuience clueless on what was happening on stage.
S.2 Stage Peifoimance
Bance }ockey went thiough thiee majoi ievamps stageu foui times in 0slo,
Noiway anu once in Poitugal between Septembei 2u1u anu Nay 2u11.
S. 2. 1 BEPARTNENT 0F N0SI C0L0uY SB0WCASE
The fouiNs laboiatoiy wheie I was uoing ieseaich at was associateu with the
Nusicology uepaitment. Foi theii Fall opening ceiemony, they askeu Stle anu
me to showcase oui ieseaich along with othei peifoimances. We ueciueu to use
one of my pievious musical compositions calleu "Aiabic uioove" which featuieu
a stiong foui-on-the-flooi beat along with two synthesizeis, a piano anu a Aiabic
singing loop. The song was aiounu S minutes long; the instiumentation
iemaineu untoucheu but the numbei of sections weie tiimmeu uown anu
stietcheu. The softwaie was still a piototype, with veiy confusing piogiamming
since we hau been woiking foi only 2S uays on it. The melouy was playeu by the
choieogiaphy that useu mostly uppei bouy positions.
S. 2. 2 uABLER, vERBI KT ANB N0STRA 0P
The uablei (0slo, Noiway on Septembei 2S), veiuikt (0slo, Noiway on
Novembei 1) anu Nostia 0P peifoimance (Poito, Poitugal on Naich 19 anu 2u)

66
all featuieu auuitional cues to the Nusicology showcase mentioneu above,
making them aiounu 1u min in length. This piece staiteu with moie theatiical
sounus that accompanieu the action happening on stage to help establish the
link between sounu anu movement. The last section maue moie use of hips to
tiiggei sounus.
S. 2. S NEW I NTERFACES F0R N0SI CAL EXPRESSI 0N
The NINE peifoimance on Nay S1
st
, 2u11, featuieu vaiious changes to the music
anu piesentation. Because this piece auueu many new sciipts it enueu up being
aiounu 2u minutes. Since this was a peifoimance at a confeience about
alteinative contiolleis theie was no neeu to euucate the auuience about the link
between movement anu music. The oveiall goal was to play expeiimental sounus
as well as populai electionic music.
In teims of piesentation this peifoimance maue much moie use of the stage
flooi anu hau gaiments that coveieu the sensois.
S.S Results
S. S. 1 NETW0RK
We uiscoveieu that the Xsens NvN Stuuio's (veision 2.6 anu S.u) netwoik stieam
is not able to senu all fiames when iunning at 12uBz in ieal time moue on oui
computei. At this iate it skips 1u to 4u peicent of the fiames. This uoes not neeu
to be a signicant pioblem if one uses "time inuepenuent" analysis, that is
analysis that uoes not look at the histoiy of the uata. But if we peifoim
uieiential calculations on the Xsens uata stieams, theie will be laige jumps in
uieientiateu values uuiing lost fiames, hence noise. This was paitly uealt with
the implementation uesciibeu in Xsens HvN SBK lmplementotion. Whenevei
fiames aie uetecteu as missing, the softwaie will peifoim an inteipolation.
Bowevei, fiame loss is still a majoi pioblem since we aie not getting all the

67
motion captuie uata anu can lose impoitant uetails in the uata stieam. Foi
instance, if a tiiggei algoiithm is listening foi some suuuen action, a couple of
lost fiames can make the event uniecognizable.
Xsens species a maximum iange up to 1Su meteis in an open elu (Xsens
2u11). In oui expeiience the wiieless connection can easily covei an aiea with a
iauius of moie than Su meteis in open aii. Such a laige aiea cannot be piactically
coveieu using IiNoCap systems.
We peifoimeu conceits in five uieient venues. Buiing the two ist conceits we
expeiienceu no pioblems with the wiieless connection. Buiing the thiiu
peifoimance we wanteu to test the wiieless connection by incieasing the
uistance between the Xsens suit anu the ieceiveis to about 2u meteis. The
wiieless link also hau an auueu challenge since the conceit was helu in a
confeience venue wheie we expecteu constant WIFI tiamc. This setup iesulteu
in pioblems with the connection anu auueu latency. The uistance shoulu
theiefoie piobably be minimizeu when peifoiming in venues with consiueiable
wiieless iauio tiamc. 0n the fouith anu fifth peifoimance we took piecautions
to ensuie that the uistance between the Xsens anu the computei was smallei.
Bowevei, WIFI inteifeience, mangentic uistuibance anu uistance still aie
eveiyuay issues.
We believe that the Xsens NvN suit, in spite of its shoit-comings in latency, jittei
anu positional uiift, oeis useful uata quality foi musical settings. Bowevei, the
iepoiteu peifoimance issues shoulu be taken into account when uesigning
action-sounu couplings. We have not been able to ueteimine whethei the Xsens
NvN system pieseives the motion qualities we aie most inteiesteu in compaieu
to othei NoCap systems, noi how theii peifoimance compaies in ieal life
settings.
S. S. 2 P0SI TI 0NAL BRI FT
The sensois in the Xsens NvN suit can only obseive ielative motion anu

68
calculate position thiough integiation which intiouuces uiift. To be able to
obseive this uiift we conuucteu a simple test by letting a subject walk along a
iectangulai path (aiounu 6x7 meteis) foui times. Figuie 17 shows a hoiizontal
positional uiift of about 2 meteis uuiing the 9u-seconu long captuie session. We
can theiefoie concluue that Xsens NvN is not an iueal NoCap system if absolute
hoiizontal position is neeueu.
21
The lack of uiift in the veitical uiiection howevei
as can be seen in the iight plot in Figuie 17 is expecteu since the NvN engine
maps the uata to a human bouy mouel anu assumes a xeu ooi level.
If the motion captuie aiea consists of uieient ooi heights the NvN engine will
match the senseu iaw uata fiom the suit against the ooi height wheie the suit
was calibiateu. This can be aujusteu in the post piocessing, but the ieal-time
uata will suei fiom aitifacts uuiing ooi level changes.
The magnetic uistuibance is ciitical uuiing the calibiation piocess but uoes not,
to oui expeiience, altei the motion tiacking quality uiamatically. Buiing a
conceit we expeiienceu signicant magnetic uistuibance, piobably because of
the laige amount of electiical equipment on stage. But this uiu not inuence the
quality of NoCap uata in such a way that it alteieu oui peifoimance.

21
The piouuct NvN Notionuiiu will impiove this uiift.
Figure 17 - Plots of the captured hori:ontal (left) and vertical (right) position of the head.


69
S. S. S S00NB
0ui initial sciipts weie less musical anu hau a utilitaiian puipose of intiouucing
anu uemonstiate the ielationship between the peifoimei anu the sounus.
Couplings between sounu anu movement weie maue veiy obvious with the
sounus accompanying eveiy action, similai to the Nickey Nousing technique
useu in film scoiing. 0nce that ielationship was establisheu in the eyes of the
auuience, the following sciipts moveu on to slightly moie abstiact couplings.
Bue to the contiol units' stiuctuie, mappings between the Xsens anu Ableton
Live 8 paiameteis coulu be quickly cieateu enabling fast piototyping anu
expeiimentation. Each sciipt useu a wiue iange of tiacks, effects, sounus anu
instiuments. Collective sets encompassing vaiious sciipts foi a peifoimance
weie maue on an au-hoc basis. Since theie was no stanuaiu way in oiganizing
tiacks anu uevices in Ableton Live 8, collecting vaiious sciipts into one set was
an aiuuous task. The iesult was a less flexible peifoimance set wheie sciipts
coulu not easily be swappeu.
S. S. 4 CB0RE0uRAPBY
Two sciipts hau choieogiaphy anu the iest was impioviseu. The choieogiaphy
was limiteu to the limbs since the Xsens NvN only captuies ielative limb
movements, not absolute cooiuinates on stage. In the sciipt ,)).the uancei's
bouy poses sent notes to two synthesizeis. Since a melouy foi the synthesizei
hau alieauy been establisheu, theie was a sequence of poses that neeueu to be
followeu to successfully play the whole melouy. In the sciipt #*)*, I tiaineu
with a Bip Bop instiuctoi a shoit choieogiaphy that was latei sonifieu into a
uubstep song. vaiious limbs weie mappeu to uiffeient instiuments anu effects
that iequiieu caieful timing anu contiol. The iesulting mapping was too haiu to
execute successfully anu on time.
The peifoimance at NINE 2u11 uebuteu gaiments that coveieu the Xsens suit. It
consisteu of a baggy sweatpants, a thin hooueu sweatshiit anu tennis shoes.

7u
Togethei they concealeu all the sensois anu cables making the suit moie
comfoitable to weai. Biuing the sensois also helpeu me, the uancei, fiame the
peifoimance in teims of uance anu music not sensoi anu music.
S. S. S A0BI ENCE
Bance }ockey was stageu foui times in 0slo, Noiway anu once in Poito, Poitugal
between Septembei 2u1u anu Nay 2u11, attiacting moie than one thousanu
attenuees. Nany of the auuience membeis piofesseu enjoying the conceit, its
novel appioach to peifoiming electionic music anu the quality of the music anu
uance. They also weie able to unueistanu most the mappings between the
movements anu the sounus.

c]
4 0enclusiens

4
0enclusiens
Conclusions from literoture. Biscussinq
tbe results from lnteroctive Niqbtclub
onJ Bonce }ockey. Summory of
JevelopeJ softwore. Iuture Work.

72
4.1 Biscussion
In this thesis I ievieweu two inteiactive music piojects uevelopeu uuiing my
masteis, anu stuuieu how the conveigence of motion captuie, music infoimation
ietiieval anu mapping has helpeu implementing them into nightclubs. The fiist
pioject calleu Inteiactive Nightclub piesents a set of motion captuie uevices that
can be manipulateu to auu layeis of synth instiuments to the music playing in a
nightclub. The seconu pioject entitleu Bance }ockey, in stiaight collaboiation
with Stle Skogstau, is an expeiimental music peifoimance that ielies on a
netwoik of weaiable sensois to ueiive a uetaileu full bouy uigital avatai of the
usei.
Notion Captuie (NoCap) was initially uesigneu foi the aimy, meuicine anu
animation anu only iecently has it founu its way into music. The appiopiiation of
this technology by musicians is still in its infancy anu most of the publications
focus moie on oveicoming technological baiiieis than biining new aesthetics
into music. Noieovei, anu piobably uue to the oiigins of NoCap, most systems
aie intenueu to be useu by a single peison; optical systems cannot single out
people in a ciowu anu ineitial systems foi ciowus cannot stieam complex usei
uata.
B}s have iesisteu auopting B}-centeieu anu B}-auuience inteiaction in theii
piactice. Consiueiing that the B} is the centeipiece of the enteitainment in
nightclubs, most inteiactive systems ciicumvent oi inteifeie with the iole of the
B}. Theie have been no compiehensive, empiiical stuuies to measuie pation
appieciation of inteiactive systems. The potential of public aienas to facilitate
the auoption of music making tools, whilst acknowleugeu, pioviues few cases of
actual piojects. Noving these tools fiom the lab to the geneial public still seems
to be the most challenging pait.
Nost of the Nusic Infoimation Retiieval stuuies uiscusseu iecommenu moie

7S
ieseaich on usei behavioi. Theie is an inteiest in involvement with enu useis,
although it appeais to be moie of a giowing awaieness of usei neeus iathei than
active involvement anu collaboiation. This awaieness-involvement gap may be
explaineu by the fact that NIR has piioiitizeu ieseaich on seaich anu ietiieval of
music collections foi evei-giowing online uatabases.
Examples of Bouy-music Nappings apply a set of similai techniques involving
one oi moie layeis between input output. Napping comes fiom a top-uown
appioach that has focuseu on uevelopment of new technology at the cost of
lowei paiticipation anu limiteu possibilities of inteiaction. Smait mappings aie
exploiing uiffeient techniques unuei the umbiella of machine leaining.
Inteiaction mouels baseu aiounu machine leaining anu aitificial intelligence
offei a bottom-up methou of inteiaction with potentially highei levels of
paiticipation anu possibilities of inteiaction. Bowevei, much of the softwaie
technology that can enable this level of intelligence is still in its eaily stages.
An oveiall look at the fielus of NoCap, Inteiactive Nightclub, NIR anu Bouy-
music Napping uiscusseu in the liteiatuie ieview will make eviuent a stiong
tenuency to focus on technology not self-expiession. Noieovei, these fielus aie
tieateu as sepaiate silos with only a few anecuotal juxtapositions. Looking at
closing this gap I uevelopeu a seiies of expeiiments that integiate these foui
uomains into a holistic appioach to fosteiing music expiession insiue nightclubs.
Interactive Nigbtclub's goal was to leveiage pations' anu B}s' musical habits
(uance anu tiacks) anu algoiithmic uata management to ieach out to the entiie
nightclub. The seivices offeieu weie not in the foim of an installationthough
they weie ueliveieu by a collection of motion captuie uevices. Noi uoes its
inteiactive system even piouuce the content that it enables pations to inteiact.
Inteiactive Nightclub exists in the space between haiuwaie anu softwaie anu
content seivei, as an inteimeuiaiy between the pation anu his oi hei inteiactive
music expeiience.

74
The most challenging pait consisteu in ueveloping tools that coulu be flexible to
each nightclub's enviionment anu stimulate usei paiticipation. The music playeu
by the B} was unpieuictable so I iesoiteu to NIR to analyze it in ieal-time. Nuch
like the "hoiseless caiiiage" fiameu the automobile as an extension of the
familiai, off-the-shelve contiolleis like Wii iemotes anu cameias piomoteu
viitual instiuments to ieplace physical instiuments. In the enu, Inteiactive
Nightclub was able to efficiently haivest haimonic content fiom the B} anu seive
it to a vaiious pations so they coulu collaboiate anu impiovise musically without
the uangei of evei sounuing . Bowevei, it became appaient that inteiaction
shoulu ieach beyonu teiminals to weaiable agents in oiuei to secuie "the long
tail", that is, the collective, peivasive powei of the small inteiactions that make
up the bulk of the system's content. This is wheie Bance }ockey steppeu in.
Dance )ockey collects kinetic infoimation fiom the weaiei anu uiffuses it
thiough a netwoik so othei people anu things can make music with it. This
inteiactive music system is not sepaiate, but an extension of the pievious one, in
which movement is given meaningful uesciiptois that bettei enabling people
anu computeis to coopeiate musically.
Although a commeicial NoCap suit was useu, vaiious netwoik stieaming
optimizations hau to be uone befoie we coulu focus on aitistic expiession. In
geneial, electionic music is not suitable foi impiovisation, anu tiauitionally
uance follows music; only now with the auvent of usei innovation uo we see
ieinteipietation of emeiging technologies that aie piovoking these establisheu
piactices anu bluiiing the uivision between both aits. Stle Skogstau, a PhB
canuiuate at 0slo 0niveisity, uevelopeu a stanualone softwaie uevelopment kit
that was fastei anu moie ieliable than the softwaie accompanying the suit.
Initially we hau uifficulty in aligning music anu uance aesthetics with Bance
}ockey. The uevelopment of a flexible inteiaction platfoim helpeu biiuge the gap
of execution between movement anu sounu leauing to moie piouuctive
ieheaisals anu peifoimances. The powei behinu Bance }ockey was in its ability

7S
to offei specific useiuevice-iuentification anu netwoik capabilities, seiving as
the basis foi a complex musical system that emeiges fiom the continuous
inteiaction with othei people anu things.
I see a futuie wheie people co-cieate music in ieal-time iegaiuless of physical
anu geogiaphical constiains. As a musician I have leaineu that, thioughout the
histoiy of mankinu, music anu technology co-evolve shaping, anu being shapeu
by, human expiession anu cieativity. The giowing iole of the usei in innovation
will fostei intelligent agents that meuiate people's neeus, anu social machines
that impiove the uiffusion of peisonal iueas to a wiuei auuience. Amplifieu by
synchionous actions of othei people, this spontaneous iipple effect will piouuce
unimaginable cultuial tienus. The vaiiety anu intiicacy of these iecombination
piocesses contiibute piofounuly to the cuiient uiveisity of peifoimance
stiuctuies anu meanings within the aits. 0ltimately, leaining, cieating, anu
shaiing music anu uance, will become inuistinguishable.
This thesis has only sciatcheu the suiface of the tiue potential behinu the
musical ielationships among people anu smait machines. We have openeu new
aieas foi fuithei ieseaich that call foi multi-uisciplinaiy appioach. Foi music to
be shaieu effectively we neeu to elaboiate on methous foi machine leaining,
uevelop motion captuie uevices foi people with uisabilities, cieate ontologies
that uefine ielations between musical concepts, wiuen the ieach of wiieless
netwoik, anu finally assume an open uata way of life.






76
4.2 Bevelopeu Softwaie
In the implementation of Inteiactive Nightclub the B} has the fieeuom to put on
any music, which is analyzeu foi haimonic content in ieal-time by the Pitcb
Tracker. Baimonic uata is stoieu tempoiaiily on, anu fetcheu piobabilistically
fiom the Probability Table. To make music, pations have at theii uisposal
cameias anu Wii iemotes that passeu thiough the Period Analysis, a ieal-time,
uynamic peak, tiough anu peiiou finuei foi one-uimensional motion captuie
uata. All the above tools aie built in the NaxNSP piogiamming enviionment.
In the Bance }ockey peifoimance, uata fiom the Xsens NvN suit is cookeu anu
tiansfoimeu into 0BP packages using Stle Skogstau's C++ Xsens MVN software
development kit which pioviues lowei latency anu fiame loss compaieu to the
suit's accompanying softwaie. The 0BP packages aie ieceiveu by the Control
Unit, a uynamic, mouulai platfoim that haibois multiple bouy-sounu mappings
anu instiuctions to the auuio engine. The USC Device insiue Ableton Live 8
pioviues a fast way to paii 0SC messages fiom any contiollei to input
paiameteis of viitual instiuments anu effect.
4.S Futuie Woik
An inteiesting expeiiment at this point woulu measuie uanceis' satisfaction of
inteiactive musicuance systems in compaiison with the non-inteiactive,
tiauitional, musicuance expeiience. Results will help guiue futuie ieseaich anu
uevelop sustainable inteiactive music mouels foi nightclubs as well as othei
enteitainment technologies. Possible ieseaich questions aie: Since any client in
a uance club can inteiact with the music, how will quality be ensuieu. Bow to
match uanceis' musical expectation with musical output. Which bouy
movements aie going to influence the music.

77
ibliegtuphg
Bahn, Cuitis, Tomie Bahn, anu Ban Tiueman. 2uu1. Physicality anu Feeuback: A Focus on the
Bouy in the Peifoimance of Electionic Nusic. Papei ieau at Inteinational Computei
Nusic Confeience, at La Babana, Cuba.
Bainuenbuig, Kailheinz. 2u1u. The Big Question. WireJ.
Bayliss, A., S. Lock, anu }.u. Sheiiuan. 2uuS. Augmenting expectation in playful aiena
peifoimances with ubiquitous intimate technologies. Papei ieau at PixelRaiueis 2, 6-8
Apiil 2uu4, at Sheffielu.
Bencina, Ross. 2uuS. The metasuiface: applying natuial neighboui inteipolation to two-to-
many mapping. In ProceeJinqs of tbe 200S conference on New interfoces for musicol
expression. vancouvei, Canaua: National 0niveisity of Singapoie.
Beitini, uiaziano, Nassimo Nagiini, anu Leonello Taiabella. 2uu6. An Inteiactive Nusical
Exhibit Baseu on Infiaieu Sensois. In Computer Husic HoJelinq onJ Retrievol, euiteu by
R. Kionlanu-Naitinet, T. voiniei anu S. Ystau: Spiingei Beilin Beiuelbeig.
Blaine, Tina. 2uuS. The conveigence of alteinate contiolleis anu musical inteifaces in
inteiactive enteitainment. In ProceeJinqs of tbe 200S conference on New interfoces for
musicol expression. vancouvei, Canaua: National 0niveisity of Singapoie.
Blaine, Tina, anu Tim Peikis. 2uuu. The }am-0-Bium inteiactive music system: a stuuy in
inteiaction uesign. In ProceeJinqs of tbe SrJ conference on Besiqninq interoctive
systems: processes, proctices, metboJs, onJ tecbniques. New Yoik City, New Yoik, 0niteu
States: ACN.
Casey, Nichael, Remco veltkamp, Nasataka uoto, Naic Leman, Chiistophe Rhoues, anu
Nalcolm Slaney. 2uu8. Content-Baseu Nusic Infoimation Retiieval: Cuiient Biiections
anu Futuie Challenges.
Chauabe, }oel. 2uu2. The limitations of mapping as a stiuctuial uesciiptive in electionic
instiuments. In ProceeJinqs of tbe 2002 conference on New interfoces for musicol
expression. Bublin, Iielanu: National 0niveisity of Singapoie.
Cliff, Bave. 2uu6. hpB}: An automateu B} with flooishow feeuback. euiteu by K. 0'Baia anu B.
Biown: Kluwei.
Cloete, T., anu C. Scheffei. 2uu8. Benchmaiking of a full-bouy ineitial motion captuie system
foi clinical gait analysis. Conf Proc lFFF Fnq HeJ Biol Soc 2uu8:4S79-82.
Collins, Nick, Chiis Kiefei, Zeeshan Patoli, anu Naitin White. 2u1u. Nusical Exoskeletons:
Expeiiments with a Notion Captuie Suit. In New lnterfoces for Husicol Fxpression
{NlHF). Syuney, Austialia.
Cont, Aishia, Thieiiy Couuys, anu Cyiille Beniy. 2uu4. Real-time gestuie mapping in Pu
enviionment using neuial netwoiks. In ProceeJinqs of tbe 2004 conference on New
interfoces for musicol expression. Bamamatsu, Shizuoka, }apan: National 0niveisity of
Singapoie.
Bean, Katie. 2u11. B} Robot Turns tbe Tobles. Wiieu 2uu1 |citeu }anuaiy 14 2u11j. Available
fiom http:www.wiieu.comcultuielifestylenews2uu1uS4S976.
Bobiian, Chiistophei, anu Fiiiic Bevilacqua. 2uuS. uestuial contiol of music: using the vicon
8 motion captuie system. In ProceeJinqs of tbe 200S conference on New interfoces for
musicol expression. Nontieal, Quebec, Canaua: National 0niveisity of Singapoie.
Booinbusch, Paul. 2uu2. Composeis' views on mapping in algoiithmic composition. 0rq.

78
SounJ 7 (2):14S-1S6.
Bownie, }. Stephen. 2uu8. The music infoimation ietiieval evaluation exchange
(2uuS&nuash;2uu7): A winuow into music infoimation ietiieval ieseaich. Acousticol
Science onJ Tecbnoloqy 29 (4):247-2SS.
Bownie, Naic. 2uu4. Choieogiaphing The Extenueu Agent: peifoimance giaphics foi uance
theatei, Nassachusetts Institute of Technology, Boston.
Fabien uouyon, Peifecto Beiieia, Emilia uomez, Peuio Cano, }oiui Bonaua, Alex Loscos, Xaviei
Amatiiain, anu Xaviei Seiia. 2uu8. Content piocessing of music auuio signals. In SounJ
to sense, sense to sounJ: A stote-of-tbe-ort in SounJ onJ Husic Computinq, euiteu by
Pietio Polotti anu B. Rocchesso. Beilin: Logos veilag Beilin umbB.
Felumeiei, Naik, anu }oseph A. Paiauiso. 2uu4. uiveaway wiieless sensois foi laige-gioup
inteiaction. In CEl '04 extenJeJ obstrocts on Eumon foctors in computinq systems.
vienna, Austiia: ACN.
Felumeiei, Naik, anu }oseph A. Paiauiso. 2uu7. An Inteiactive Nusic Enviionment foi Laige
uioups with uiveaway Wiieless Notion Sensois. Computer Husic }ournol S1 (1):Su-67.
Flckigei, Natthias. 2uu9. Who is Bancing. Recognize Banceis by Neans of Acceleiation
Sensoi. Zuiich: Swiss Feueial Institute of Technology.
Fuiniss, Nauieen. Hotion Copture 2uu4. Available fiom http:web.mit.euucomm-
foiumpapeisfuiniss.html.
Futielle, }oe, anu Stephen Bownie. 2uuS. Inteiuisciplinaiy Reseaich Issues in Nusic
Infoimation Retiieval: ISNIR 2uu2. }ournol of New Husic Reseorcb S2 (2):121-1S1.
uates, Caiiie, anu Siiiam Subiamania. 2uu6. A Lens on Technology's Potential Roles foi
Facilitating Inteiactivity anu Awaieness in Nightclub. Saskatoon, Canaua: 0niveisity of
Saskatchewan.
uates, Caiiie, Siiiam Subiamanian, anu Cail uutwin. 2uu6. B}s' peispectives on inteiaction
anu awaieness in nightclubs. In ProceeJinqs of tbe 6tb conference on Besiqninq
lnteroctive systems. 0niveisity Paik, PA, 0SA: ACN.
uaye, Lalya, Ramia Naz, anu Lais Eiik Bolmquist. 2uuS. Sonic City: the uiban enviionment as
a musical inteiface. In ProceeJinqs of tbe 200S conference on New interfoces for musicol
expression. Nontieal, Quebec, Canaua: National 0niveisity of Singapoie.
uouy, Rolf Inge, anu Alexanuei Refsum }ensenius. 2uu9. BoJy Hovement in Husic lnformotion
Retrievol. 0slo: The 0niveisity of 0slo.
uoto, Suguiu, anu Takahiko Suzuki. 2uu4. The case stuuy of application of auvanceu gestuie
inteiface anu mapping inteiface: viitual Nusical Instiument "Le SupeiPolm" anu
uestuie Contiollei "BouySuit". In ProceeJinqs of tbe 2004 conference on New interfoces
for musicol expression. Bamamatsu, Shizuoka, }apan: National 0niveisity of Singapoie.
uiaham-Rowe, Buncan. 2u11. Clubbers cboose cbip implonts to jump queues. NewScientist
2uu4 |citeu }anuaiy 1S 2u11j. Available fiom
http:www.newscientist.comaiticleunSu22-clubbeis-choose-chip-implants-to-
jump-queues.html.
Bansen, Kjetil Falkenbeig, anu Robeito Biesin. 2uu6. Napping stiategies in B} sciatching. In
ProceeJinqs of tbe 2006 conference on New interfoces for musicol expression. Paiis,
Fiance: IRCAN Centie Pompiuou.
Biomin, Bennis, Nichael Chlauil, Natalie vanatta, Baviu Naumann, Faiooq Anjum, anu Ravi
}ain. 2uuS. coueBL0E: A Bluetooth Inteiactive Bance Club System.
Buck, Tim. 6et Text-To-Screen Fntertoinment ot your live Fvent! 2u1u. Available fiom
http:text-to-scieen.com.
Bunt, Anuy, anu Ross Kiik. 2uuu. Napping Stiategies foi Nusical Peifoimance. In TrenJs ln
6esturol Control 0f Husic, euiteu by N. Wanueiley anu N. Battiei: Iicam - Centie
Pompiuou.
Bunt, Anuy, Naicelo N. Wanueiley, anu Ross Kiik. 2uuu. Towaius a Nouel foi Instiumental

79
Napping in Expeit Nusical Inteiaction. In lnternotionol Computer Husic Conference. San
Fiancisco: Inteinational Computei Nusic Association.
Bunt, Anuy, Naicelo N. Wanueiley, anu Natthew Paiauis. 2uu2. The impoitance of paiametei
mapping in electionic instiument uesign. In ProceeJinqs of tbe 2002 conference on New
interfoces for musicol expression. Bublin, Iielanu: National 0niveisity of Singapoie.
}ehan, Tiistan, anu Beinu Schonei. 2uu1. An Auuio-Biiven, Spectial Analysis-Baseu, Peiceptual
Synthesis Engine. Papei ieau at Auuio Engineeiing Society, at Amsteiuam, The
Netheilanus.
}ensenius, Alexanuei Refsum. 2uu7. ACTI0N - S00NB: Beveloping Nethous anu Tools to Stuuy
Nusic-Relateu Bouy Novement, Bepaitment of Nusicology, 0niveisity of 0slo, 0slo.
Kaisei, ueiiit, uustav Ekblau, anu Linna Bioling. 2uu7. Auuience Paiticipation In Auance-Club
Context: Besign 0f A System Foi Collaboiative Cieation 0f visuals. In NorJic Besiqn
Reseorcb. Stockholm
Kitagawa, Niuoii, anu Biian Winusoi. 2uu8. HoCop for Artists: Workflow onJ Tecbniques for
Hotion Copture. Euiteu by P. Temme. Builington, NA: Focal Piess.
Lam, Bonnie. 2u11. IooJ & Beveroqe. Spiinwise 2uu8 |citeu }anuaiy 1S 2u11j. Available fiom
http:www.spiingwise.comfoou_beveiageinteiactive_touchbai_combines.
Law, E. L. N., L. von Ahn, R. B. Bannenbeig, anu N. Ciawfoiu. 2uu7. Tagatune: A uame foi
Nusic anu Sounu Annotation. Papei ieau at Inteinational Confeience on Nusic
Infoimation Retiieval (ISNIR'u7), at vienna, Austiia.
Law, Euith, Kiis West, Nichael Nanuel, Neit Bay, anu }. Stephen Bownie. 2uu9. Evaluation 0f
Algoiithms 0sing uames: The Case 0f Nusic Taggin. In 10tb lnternotionol Society for
Husic lnformotion Retrievol Conference {lSHlR 2009).
Lee, }in Ba, anu }. Stephen Bownie. 2uu4. Suivey of music infoimation neeus, uses, anu seeking
behaviouis: Pieliminaiy finuings. Papei ieau at In ISNIR Pioceeuings, at Baicelona.
Leitich, Stefan, anu Naikus Toth. 2uu7. PublicB} - Nusic selection in public spaces as
multiplayei game. In AuJio Hostly 2007. Ilmenau, ueimany.
Lesaffie, Nicheline, Liesbeth Be voogut, Naic Leman, Beinaiu Be Baets, Bans Be Neyei, anu
}ean-Pieiie Naitens. 2uu8. Bow potential useis of music seaich anu ietiieval systems
uesciibe the semantic quality of music. }ournol of tbe Americon Society for lnformotion
Science onJ Tecbnoloqy S9 (S):69S-7u7.
Lew, Nichael S., Nicu Sebe, Chabane Bjeiaba, anu Ramesh }ain. 2uu6. Content-baseu
multimeuia infoimation ietiieval: State of the ait anu challenges. ACH Trons.
HultimeJio Comput. Commun. Appl. 2 (1):1-19.
Liuy, Thomas, anu Anuieas Raubei. 2uu9. Nusic Infoimation Retiieval. In EonJbook of
reseorcb on Jiqitol librories: Jesiqn, Jevelopment onJ impoct, euiteu by Y.-L. Theng, S.
Foo, B. uoh anu C. N. }in. New Yoik: IuI ulobal.
Lonsuale, Auam }., anu Auiian C. Noith. 2u11. Why uo we listen to music. A uses anu
giatifications analysis. Britisb }ournol of Psycboloqy 1u2 (1):1u8-1S4.
Naes, Pietei-}an, Naic Leman, Nicheline Lesaffie, anu Nichiel Bemey. 2u1u. Fiom expiessive
gestuie to sounu: The uevelopment of an embouieu mapping tiajectoiy insiue a
musical inteiface. }ournol on HultimoJol 0ser lnterfoces S (1-2):67-78.
Nalloch, }oseph, Stephen Sinclaii, anu Naicelo N. Wanueile. 2uu7. Fiom contiollei to sounu:
Tools foi collaboiative uevelopment of uigital musical instiuments. Papei ieau at
Inteinational Computei Nusic Confeience, at Copenhagen, Benmaik.
Nalozemoff, Alexis }., anu Philippe Bepalle. 2uu9. N0NT Su2 Pioject Repoit: uait Recognition
0sing Acceleiometeis anu Sounu. Nontieal: Ncuill 0niveisity.
Nalt, Nikhail, anu Emmanuel }ouiuan. 2uu8. Zsa.Besciiptois: a libiaiy foi ieal-time
uesciiptois analysis. Papei ieau at Sounu anu Nusic Computing, at Beilin, ueimany.
Nanuelis, }ames, anu Phil Busbanus. 2uu4. Bon't just play it, giow it!: bieeuing sounu
synthesis anu peifoimance mappings. In ProceeJinqs of tbe 2004 conference on New

8u
interfoces for musicol expression. Bamamatsu, Shizuoka, }apan: National 0niveisity of
Singapoie.
Naishall, Naik, anu Naicelo Wanueiley. 2uu6. Evaluation of Sensois as Input Bevices foi
Computei Nusic Inteifaces. In Computer Husic HoJelinq onJ Retrievol, euiteu by R.
Kionlanu-Naitinet, T. voiniei anu S. Ystau: Spiingei Beilin Beiuelbeig.
Noeslunu, Thomas B. 2uuu. Inteiacting with a viitual Woilu thiough Notion Captuie. In
lnteroction in virtuol lnbobiteJ SB WorlJs: Spiingei-veilag.
Noeslunu, Thomas B., Auiian Bilton, anu volkei Kigei. 2uu6. A suivey of auvances in vision-
baseu human motion captuie anu analysis. Computer vision onJ lmoqe 0nJerstonJinq
1u4 (2-S):9u-126.
Ng, K. C. 2uu4. Nusic via motion: tiansuomain mapping of motion anu sounu foi inteiactive
peifoimances. ProceeJinqs of tbe lFFF 92 (4):64S-6SS.
Paik, Chiistine. 2u11. Notion-baseu gaming: Testing the best. ABC, }anuaiy u6, 2u11.
Pelletiei, }ean-Naic. 2uu4. Sounu anu Sight: Composition foi Image Analysis Systems, Institute
of Auvanceu Neuia Aits & Science, uifu, }apan.
Puckette, N., T. Apel, anu B. Zicaielli. 1998. Real-time auuio analysis tools foi Pu anu NSP.
Papei ieau at Inteinational Computei Nusic Confeience, at San Fiancisco:
Inteinational Computei Nusic Association.
Qian, uang, Feng uuo, Touu Ingalls, Loien 0lson, }ouy }ames, anu Thanassis Rikakis. 2uu4. A
gestuie-uiiven Nultimoual Inteiactive Bance System. Papei ieau at IEEE Inteinational
Confeience on Nultimeuia anu Expo, at Taipei
Roetenbeig, Baniel, Benk Luinge, anu Pei Slycke. 2uu9. Xsens NvN: Full 6B0F Buman Notion
Tiacking 0sing Niniatuie Ineitial Sensois. Xsens Technologies.
Rovan, }oseph Butch, Naicelo N. Wanueiley, Shlomo Bubnov, anu Philippe Bepalle. 1997.
Instiumental uestuial Napping Stiategies as Expiessivity Beteiminants in Computei
Nusic Peifoimance. Papei ieau at Kansei - The Technology 0f Emotion Woikshop.
Saltei, Chiistophei, Naiije Baalman, anu Baniel Noouy-uiigsby. 2uu8. Between Napping,
Sonification anu Composition: Responsive Auuio Enviionments in Live Peifoimance. In
Computer Husic HoJelinq onJ Retrievol. Sense of SounJs, euiteu by R. Kionlanu-
Naitinet, S. Ystau anu K. }ensen: Spiingei Beilin Beiuelbeig.
Schmeuei, Anuiew, Auiian Fieeu, anu Baviu Wessel. 2u1u. Best Piactices foi 0pen Sounu
Contiol. In linux AuJio Conference. 0tiecht, NL.
SBC. 2u11. Sustoinoble Bonce Club 2u11 |citeu }anuaiy Su 2u11j. Available fiom
http:www.sustainableuanceclub.com.t=piouucts&p=1.
Skogstau, Stale Anuieas van Boip, Alexanuei Refsum }ensenius, anu Kiistian Nymoen. 2u1u.
0sinq lR 0pticol Horker BoseJ Hotion Copture for Fxplorinq Husicol lnteroction. 0slo:
The 0niveisity of 0slo.
Tahiioglu, Koiay, anu Cumhui Eikut. 2uu7. ClaPB: A testbeu foi contiol of multiple sounu
souices in inteiactive anu paiticipatoiy contexts. Papei ieau at The PuieBata
Convention, at Nontieal, Canaua.
Takahashi, Bean. 2u11. Tbe viJeo qome console wor coulJ enJ in o tbree-woy tie, }une 9, 2u1u
2u1u |citeu 2u12u11 2u11j. Available fiom
http:ventuiebeat.com2u1uu6u9the-state-of-the-viueo-game-console-wai-is-
iemaikably-even.
Tanaka, Atau, anu R. Benjamin Knapp. 2uu2. Nultimoual inteiaction in music using the
Electiomyogiam anu ielative position sensing. In ProceeJinqs of tbe 2002 conference on
New interfoces for musicol expression. Bublin, Iielanu: National 0niveisity of Singapoie.
TangibleInteiaction. Zyqote lnteroctive Bolls. Tangible Inteiaction 2u1u. Available fiom
http:www.tangibleinteiaction.comgalleiyzygote.
Taiabella, Leonello, anu uiaziano Beitini. 2uu4. About the Role of Napping in uestuie-
Contiolleu Live Computei Nusic. In Computer Husic HoJelinq onJ Retrievol, euiteu by

81
0. K. Wiil: Spiingei Beilin Beiuelbeig.
Typke, Rainei, Fians Wieiing, anu Remco C. veltkam. 2uuS. A Suivey 0f Nusic Infoimation
Retiieval Systems. Papei ieau at The Inteinational Society foi Nusic Infoimation
Retiieval, Septembei 11-1S at Lonuon.
0lyate, Ryan, anu Baviu Bianciaiui. 2uu1. The inteiactive uance club: avoiuing chaos in a multi
paiticipant enviionment. Papei ieau at NINE 'u1: Pioceeuings of the 2uu1 confeience
on New inteifaces foi musical expiession.
Wessel, Baviu, anu Natthew Wiight. 2uu2. Pioblems anu Piospects foi Intimate Nusical
Contiol of Computeis. Comput. Husic }. 26 (S):11-22.
Wieczoikowska, Alicja A. 2uuS. Nusic Infoimation Retiieval. In FncyclopeJio of Boto
Worebousinq onJ Hininq, euiteu by }. Wang.
Wieiing, Fians. 2uu7. Can Bumans Benefit fiom Nusic Infoimation Retiieval. In AJoptive
HultimeJio Retrievol: 0ser, Context, onJ IeeJbock, euiteu by S. Naichanu-Naillet, E.
Biuno, A. Ninbeigei anu N. Betyniecki: Spiingei Beilin Beiuelbeig.
Winklei, Touu. 199S. Naking Notion Nusical: uestuie Napping Stiategies foi Inteiactive
Computei Nusic. Papei ieau at Inteinational Computei Nusic Confeience, at San
Fiancisco.
Xsens. 2u11. Xsens Technologies B.v. Xsens NvN 0sei Nanual.
Zone, Synth. 2u11. BoJySyntb |citeu Apiil 2u11j. Available fiom
http:www.synthzone.combsynth.html.




Buy your books fast and straightforward online - at one of worlds
fastest growing online book stores! Environmentally sound due to
Print-on-Demand technologies.
Buy your books online at
www.get-morebooks.com
Kaufen Sie Ihre Bcher schnell und unkompliziert online auf einer
der am schnellsten wachsenden Buchhandelsplattformen weltweit!
Dank Print-On-Demand umwelt- und ressourcenschonend produzi-
ert.
Bcher schneller online kaufen
www.morebooks.de
VDM Verlagsservicegesellschaft mbH
Heinrich-Bcking-Str. 6-8 Telefon: +49 681 3720 174 info@vdm-vsg.de
D - 66121 Saarbrcken Telefax: +49 681 3720 1749 www.vdm-vsg.de