Multisensory Communication and Experience through Multimedia

Conducting a Virtual Orchestra

Sebastien Schertenleib, Mario Gutirrez, Frdric Vexo, and Daniel Thalmann Swiss Federal Institute of Technology in Lausanne
The Virtual Orchestra is a multimodal system architecture based on complex 3D sound environment processing and a compact user interface. The system architecture supports highperformance sound content for VR applications, and the user controls the performance with a magnetically tracked handheld device. ne of the most exciting and rich experiences in the real world is music and all that surrounds it. A music performancebe it a classical music concert performed by a symphonic orchestra or a rock concertalways conveys a range of experiences. Music stimulates multiple senses at once. We hear and feel the sound vibrations in our bodies, and watching the performance complements the experience. As a result, we end up jumping, dancing, or discreetly following the rhythm with our arms or feet. The objective of our work is to give music amateurs the opportunity to conduct a group of musicians and produce a new kind of multimedia experience. To explore this idea, we developed a semi-immersive virtual reality (VR) system, based on a large projection screen, designed to create an entertaining multimedia experience for nonexpert users. We decided to implement a music performance simulator, a VR system where several virtual characters play different musical instruments and adapt their performance according to the users instructions. Although we pay special attention to the realism of the scenarios visual representation, animating the musicians, and the music they produce (3D sound rendering), the conducting interface doesnt try to accurately reproduce the complex interaction of conducting an orchestra. This is left as future work. We provide an interface that partially captures the expressivity of users gestures without forcing them to perform a specic conducting pattern.

cal pieces codified as MIDI files. The user conducts the performance by deciding on the music score to interpret and modifying in real time the orchestras layout as well as the performances tempo and dynamics (volume). The conductor and audience can view and listen to the music ensemble from any position in the 3D environmentwith dynamic acoustic effects. The user doesnt need to be a professional to conduct this orchestra because all the functionalities and options are available through a simple interface: a magnetically tracked handheld device. We faced several challenges in achieving a reasonable level of immersion for the users (the conductor and spectators) in such an environment. In response, we designed and implemented solutions to three main problems: 3D sound rendering, the user interface, and the virtual environment and musicians. (See the Related Work sidebar for other research in these areas.) 3D sound The purpose of a VR system is to reect as best as possible the real world to the human senses. To achieve a sense of presence, VR systems must provide content such as graphics, natural user interfaces, and sound. Three-dimensional sound processing is essential for VR worlds. The goal of 3D sound rendering in our work was to reproduce the musical instruments sound and compute the acoustic effects due to the scenarios physical characteristics. (We explore this element in more detail in the 3D sound rendering section to follow. ) User interface The user interface in a virtual environment (VE) is a key component to creating a sense of presence. Our user interface problem involved how to direct the performance (tempo and dynamics) of the virtual musicians and control the characteristics of their environment. We aimed at maximizing the sense of presence by eliminating complex or cumbersome devices such as head-mounted displays (HMDs), data gloves, and similar peripheralsavoiding the classical VR approach. We chose the alternative of a semi-immersive scenario. Our VE is rendered on a large projection screen where several users can experience the simulation. The immersion was enhanced by 3D images that users view with 3D glasses. Our objective was to create a fully customizable application where system developers and rst-time users could modify, at any time, most of the environment settings such as the camera

Development challenges
The participant in our VR system conducts a virtual orchestra performing in a 3D concert hall. Several autonomous virtual humans play musi-

40

1070-986X/04/$20.00 2004 IEEE

Published by the IEEE Computer Society

Related Work
Implementing an interactive virtual orchestra is a multidisciplinary project involving different research areas. This is an overview of the state of the art on the main research topics involved in our work. sive results, making the virtual characters behaviors more believable for the user. During the last few years, the synthesis and processing of 3D sound in VR applications hasnt received enough consideration. We can explain this with the following reasons: Lack of space to physically locate the required speaker congurations. The idea that a VR systems sound component didnt have the highest priority. The hardware required to compute real-time highquality content on multiple speakers was expensive or nonexistent.

3D sound
When we ask people about their expectations of a VR simulation, they say that at rst they feel attracted by all the visual effects and fancy interaction devicesheadmounted displays (HMDs), data-gloves, and so forth. Sound effects are rarely mentioned as being the most interesting aspects of the experience, but in fact, they make a big difference between a simple 3D application and an immersive experience.1 For a realistic VR application, the sound system is more important than we might think because we use sound to gather information about our surrounding environment,2 rich sound environments help us pay attention to details, 3 and the human brain responds to changes in sound (such as tempo and loudness) and reacts immediately to any sound event. Audition produces more instinctive answers than visual artifacts. 4 In a VR environment, the auditory experience is opposed to the visual one. The user is already prepared to see nonreal images (habituation to TV and cinema). 5 Yet audio is different; experiments have shown that poor sound cues can confuse users and make them ignore any sound in the next few minutes (see http://www.creativelabs.com/).6 A bad or irrelevant sound system reduces the attention to and believability of a simulation as much as good sound systems increase it. Because a listeners expectations about the surrounding world will be directly affected by the sound processing, we need to lter them appropriately. Many VR applications dont implement 3D sound properlyif they use 3D sound processing at all. Generally, VR engines tend to be graphic-centered systems, built on top of the rendering pipeline. However, with current graphics processing unit (GPU) systems, more CPU time is available for computing other components such as physics and 3D sound. 1 Some studies by Nvidia and the Microsoft developers team conducted during the conception of the Xboxs audio hardware7,8 showed that the sound environment is so natural for humans that simulating it in a VR system gives impres-

User interface
There have been several attempts to reproduce the experience of conducting an orchestra. One of the most remarkable is the work presented at Siggraph 99 by Jakub Segen and his colleagues.9 They proposed a visual interface based on optical tracking of conductor gestures (hand and baton tracking). The user controlled the beat, tempo, and volume of the music performed by a MIDI sequencer. They achieved immersion with a projection screen displaying a group of virtual dancers who followed the music. The Personal Orchestra, a system implemented at the House of Music Vienna, 10 is a more recent effort to provide a virtual conductor application. The interface is an infrared-tracked baton to control in real time the tempo and volume of prerecorded music and video from a real orchestra. In both cases, the main interaction device is the conductors baton, tracked by image segmentation in the case of the visual interface, or by infrared signals emitted by the commercially available Buchla Lightning II conductors baton11 used in the Personal Orchestra. Another similar device, the Radio Baton, 12 tracks with radio-frequency signals. Researchers have developed other similar devices, some of which are more complex and detect not only the batons position and the pressure on its handle; these are complemented by additional trackers to acquire the users gestures and capture their expressivity. 13,14 Our work doesnt intend to advance the state of the art on such conducting interfaces like the Radio Baton or the work of Marrin, 14 which usually require professional skills and are designed for expert users. The goal of our interface was to acquire continued on p. 42

JulySeptember 2004 41

continued from p. 41 simple parameters, such as the amplitude and frequency of arm gestures, and provide additional functionalities to interact with the environment. The research weve seen doesnt provide an interface for customizing the environment or choosing a piece to play. The Personal Orchestra10 shows video from a real orchestra, but the repertoire is limited to the preprocessed recordings. The Visual Conductor12 is more flexible because it uses MIDI files, but it has no representation of the musicians (dancers only). Usually, these systems focus on providing a conducting interfacethe scene and orchestra are preset and cant be edited by the user. The Personal Orchestra lets the user select a piece to play and gives some additional information, but the user cant modify any other parameter. We decided to experiment with PDAs as conducting interfaces. Early research on PDAs and VR include the work of Ken Watson and his colleagues.15 They implemented a basic interface for 2D interactions in a CAVE-like system. Recent research has focused on implementing 2D GUIs for basic interaction: camera control, scene management, and interaction between multiple users. 1618 In the touch-space system, 19 a PDA is optically tracked and used to display virtual objects superimposed on the real world. This work shows that a PDA can serve as an efcient tool to reinforce humanhuman contact in a mixed-reality environment. This is an important point to consider, because apart from immersing the users into the simulation, we want them to share their impressions and enjoy the virtual performance together.

set them manually. The virtual drummer was able to execute only precalculated music scores, and there was no way to modify or conduct its performance in real time. Some of the research work at VRlab-EPFL has focused on the problem of virtual humans autonomy, including creating autonomous musicians. Recent research has led to defining a model to represent the content of virtual scenes and focus on the interactions between virtual humans and virtual objects. 22 Researchers have demonstrated this by developing an autonomous virtual pianist (virtual agent) who can interpret virtual scorebooks. 23 The scorebook contains musical dataextracted and preprocessed from a MIDI filethat the virtual pianist can use as input for a learning phase (used to precalculate the pianist gestures) and a subsequent playing phase. The musical input data from the scorebook is also used to synthesize the piano animation and corresponding sound production.

References
1. A. Rollings and E. Adams, Construction and Management Scene, On Game Design, Pearson Education, 2003. 2. G. Lecky-Thompson, Innite Game Universe, Level Design, Terrain, and Sound, vol. 2, Charles River Media, 2002. 3. C. Crawford, On Game Design, New Riders, 2003. 4. C. Carollo, Sound Propagation in 3D Environments, Ion Storm, 2002. 5. Facts And Figures about our TV Habits, TV-Turnoff Networks, 2003 6. J. Blauert, Spatial Hearing: The Psychoacoustics of Human Sound Localization, revised ed., MIT Press, 1997. 7. M. Deloura, Programming Gems 3, Charles River Media, 2002. 8. M. ODonnel, Producing Audio for Halo, MS Press, 2002. 9. J. Segen, J. Gluckman, and S. Kumar, Visual Interface for Conducting Virtual Orchestra, Proc. Intl Conf. Pattern Recognition, vol. 1, IEEE CS Press, 2000. 10. J.O. Borchers, W. Samminger, and M. Muhlhauser, Personal Orchestra: Conducting Audio/Video Music Recordings, Proc. Intl Conf. Web Delivering of Music, Wedelmusic, 2002, p. 93. 11. R. Rich, Buchla Lightning II, Electronic Musician, vol. 12, no. 8, 1996. 12. M. Mathews, The Conductor Program and Mechanical Baton, Current Directions in Computer Music Research, M.V. Mathews and J.R. Pierce, eds., MIT Press, 1991. 13. T. Marrin, Possibilities for the Digital Baton as a General-Purpose Gestural Interface, Proc. ACM-Conf. Human Factors in Computing Systems (CHI 97), 1997. 14. T. Marrin, Inside the Conductors Jacket: Analysis,

Virtual environment and musicians

One of the rst examples of virtual musicians was presented in the Electric Garden at Siggraph 97 by the DIVA-Group. 20 This demo showed four virtual musicians3D human-like articulated models following a real conductor. The musicians movements were determined by inverse kinematics calculation from a MIDI le and a set of constraint rules for each instrument. The animation was coordinated with a synchronization signal generated by a tempo-tracking module. A more recent effort aimed at creating autonomous virtual musicians is the work of Martijn Kragtwijk and his colleagues. 21 The authors presented a set of algorithms to generate 3D animated musicians based on an abstract representation of the musical piece (MIDI les). They implemented a virtual drummer displaying realistic animations of the limbs and sticks before the contact moments. Instead of using motion capture or inverse kinematics to dene the required poses for a given drum-kit conguration, the authors implemented a GUI-based pose editor to

IEEE MultiMedia 42

15.

16.

17.

18.

Interpretation and Musical Synthesis of Interaction Expressive Gesture, PhD thesis, MIT Media Lab, 2000. K. Watsen, D.P. Darken, and W.V. Capps, A Handheld Computer as an Interaction Device to a Virtual Environment, Proc. 3rd Intl Immersive Projection Technology Workshop (IPTW99), 1999. E. Farella et. al., Multi-client Cooperation and Wireless PDA Interaction in Immersive Virtual Environment, Proc. Euromedia Conf., 2003, E. Farella et al., Using Palmtop Computers and Immersive Virtual Reality for Cooperative Archaeological Analysis: the Appian Way Case Study, Proc. Intl Conf. Virtual Systems and Multimedia, Intl Soc. on Virtual Systems and MultiMedia, 2002. L. Hill and C. Cruz-Neira, Palmtop Interaction Methods for Immersive Projection Technology Systems, Proc. Intl Immersive Projection Technology Workshop, 2000..

19. A. Cheok et al., Touch-Space: Mixed Reality Game Space Based on Ubiquitous, Tangible, and Social Computing, , vol. 6, no. 56, Jan. 2002, pp. 430-442. 20. R. Hnninen, L. Savioja, and T. Takala, Virtual Concert Performance - Synthetic Animated Musicians Playing in an Acoustically Simulated Room, Proc. Intl Computer Music Conf., (poster paper), 1996. 21. M. Kragtwijk, A. Nijholt, and J. Zwiers, Implementation of a 3D Virtual Drummer, Computer Animation and Simulation, N. Magnenat-Thalmann and D. Thalmann, eds., Springer Verlag Wien, 2001, pp. 15-26. 22. J.Esmerado, A Model of Interaction between Virtual Humans and Objects: Application to Virtual Musicians, Ph.D. Thesis, No. 2502, EPFL, 2001. 23. J. Esmerado et al., Interaction in Virtual Worlds: Application to Music Performers, Proc. Computer Graphics Intl Conf. (CGI), 2002.

point of view, the musicians position, the piece to play, and so forth. Moreover, we decided to separate the interface from the simulation screen to free the scene from overlaying menus or other GUI components. At the same time, both controlsthe environment settings and conducting interfacewere to remain equally accessible. In this context, the user would be able to play the roles of both the artistic director and conductor. We looked for an interface that remained intuitive enough while letting users execute various tasks through a single interaction device. Our search led us to an area that researchers have just started to explore in the context of VR applications: handheld personal digital assistants (PDAs). Using the PDA not as a complement, but as the main interface to the VE, let us implement a shared simulation where one of the users acts as the scene or artistic director and conductor while others can enjoy the experience as spectators of the virtual concert. Virtual environment and musicians Besides a realistic 3D sound and an intuitive, yet flexible interface, the virtual environment representation constitutes a key simulator component. A convincing animation of autonomous virtual musicians obeying to the conductor in real time contributes to the experience, making it more attractive and believable. One problem with virtual environment and

musicians involves making the virtual entities visually appealing for believability and full user immersion. The virtual musicians had to perform like real musicianswhich called for autonomous animation. We adapted a virtual agent model that lets us edit the performance parameters in real time, in particular the tempo and dynamics. Following the same model and design patterns, we implemented a new musician, a virtual utist. We use both of them in our simulation.

VR system architecture
We implemented the virtual orchestra conducting application using a component-based framework developed at VRlab. (See related literature for a deeper overview of this system.1) The systems software architecture is based on a distributed application model with TCP/IP as the main communication channel bridging the server and client (see Figure 1, next page). Consequently, the software is divided into two main executable modules. The PC server consists of multiple software components that handle communication, including rendering 3D scenes and sound; processing inputs from the external VR devices (handheld and magnetic tracker); animating the 3D models; and in particular, animating complex virtual human models, including skeleton animation and skin and clothes deformation. We can modify the

JulySeptember 2004 43

Figure 1. Architecture of the virtual orchestra VR system based on component patterns.

Simulation runtime engine Service manager Network layer

TCP/IP packages

Events, remote method invocations (RMIs) Data manager Message handler Simulation State Service data controller Animation orders Dedicated service Face orders Simulation Control messages Play orders Python interpreter

Container objects

Read/write skin mesh Read/write skeleton Read visims

Skin deformation service Skeleton animation service

Face deformation service

Read sound samples

Sound service

Scenario manager module and handheld device controller

Read/write camera read Container objects

3D rendering service 3D pointer position orientation

Camera changes Pointed object ID

Scenarios, configurations

Model view matrix Scene status

TCP/IP module

Loading/saving service

Scenarios, configurations Communication module

XML data

VR system database

Handheld Device

Gestures Analyzer

interactive scenarios state with python scripts at runtime. The client (a handheld device) is responsible for acquiring the users interaction with the system through its integrated 2D GUI and the attached sensor (a magnetic tracker). The magnetic tracker information is analyzed by another PC (a gestures analyzer) acting as a second client. We had a two-fold rationale for our choice of the distributed application model. First was our desired system functionalities and second was due to technical requirements during the system developmentthe need to distribute the data processing and components development.

3D sound rendering
The most important properties of a sound source in terms of computer knowledge2,3 are the sources 3D position and orientation in the world; cone of propagation; distance between the listener and source; Doppler effect; volume and frequency; and

IEEE MultiMedia 44

Sound source

Distance

Inside volume

Figure 2. (a) Sound source propagation cone model. (b) Sound source attenuation due to reections and reverberation models.

Transitional volume

Volume = inside volume + specified outside volume

Outside cone Inside cone (a)

Direct 0 Reflections Reverb

Decibels

Decay_time

Reflections_delay Reverb _delay (b)

Time

occlusion, obstruction, and exclusion. By ltering the different sound sources based on the previous settings, the direction parameter becomes the most important property to consider. Otherwise, the system rendering process is deeply limited. In our implementation, we represent the sound propagation with a cone, which gives us exibility to specify the different lters for each sound source (see Figure 2). When obstacles between the listener and a

sound source exist, we must calculate all attenuations due to occlusion, obstruction, and exclusion situations depending on the 3D environments conguration. In general, obstructions occur when both sound source and listener are in the same room. Exclusions occur when the rooms arent completely separated; otherwise, we have an obstruction.2 3D environments The complexity of 3D sound processing

JulySeptember 2004 45

Figure 3. Example of a sound path depending on external events that control the synchronization between the sound rendering and human animation.

Different chunk for the same note Data 1 Data 2 Data 3 Analysis skill 1 Usable data 1

No events occur

Data n External events occur

Analysis skill n

Usable data n

Skill interaction

Synchronization

Object animation

Interaction results

Body animation

IEEE MultiMedia

depends on the environment architecture. We set up some predefined environments for common places such as a church, bathroom, or auditorium. Designers can define the shape of their sound environments by making small modicationsfor example, dening the rooms size. From a behavioral point of view, the air absorption factor acts on the compression of the sound wave, which can distort the original sound source. Obstacles and walls inuence the sound propagation and modify it by adding reverberations, reections, and delays (see Figure 2). Moreover, to increase efciency, we combine different components together to give the system additional information. The rendering pipeline contains knowledge about the environment that we can extract from the Binary Space Partition (BSP) Tree4 or Cosmo 3D Scenegraph (http://www.cai.com/ cosmo/)our viewer is based on the latter library. Then, this information can be used by the physics or sound components for collision or obstacle detection among other operations. Managing complex environments To compute all attenuations from the initial position and orientation of a sound source and the listener, we need to nd the path followed by the wave, resulting from its interaction with

obstacles and walls. For this, we use a customized version of the A* algorithm5 (http://www.iadepot.com), which gives us different points and angles for the objects (see Figure 3). Then we can calculate all attenuations and effects to be set for filtering the wave. However, due to time constraints, we cant perform intensive computations for all sources so our simplified version allows some errors but guarantees a smooth performance. In other words, we dont compute every sound source at each animation frame but check only if any drastic change occurs between two continuous steps. Otherwise, we compute extrapolations using the previous states of the sound source properties. Such operations are excellent candidates to be executed on a multiprocessing architecture. To take advantage of particular systems like PCs with dual or hyper-threading processors,6,7 we have implemented our system using an API (http://www.openmp.org), which increases the number of sound sources we can process in a single frame, based on a ratio of the number of logical CPUs available. Based on the system events, we analyze the data for controlling the human manager who is responsible for synchronizing the sound processing with the human animation (see Figure 2).

46

Playing dynamic music To play music that can be modied in real time, we must be able to record the same track with different transitions and then use this database to choose which sample needs to be played depending on the current events. With our current implementation, we rely on Gigabase (http://www.ispras.ru/~knizhnik/gigabase.html), an object-relational database management system. The idea was to provide a database table based on a bidimensional array. One axis represents the different ways we can play a single chunk of data (like a modication in dynamics), and the second axis represents the time. This required establishing some connections between the visual aspects of the virtual musician and the sound rendering.8 Of course, this approach depends on the quality and quantity of the primitive sound samples. As we want to extend the database without any new recording session, we used MIDI les. Thus, we can manipulate them directly within the soundboard for greater efciency (see http://www. microsoft.com/directx). Synchronizing animation with sound propagation Real-time systems must react fast enough to events to maintain immersion. This is particularly true with sound rendering, which is deeply affected by any processing delay. In the case of the virtual orchestra, we need to synchronize the animation of the musicians with the corresponding sound. Due to real-time restrictions, its impossible to predict state transitions, and thus we cant prepare the right keyframe at a certain time. Carefully managing sound and animation databases is crucial to reducing latency.9 The processing is based on Joaquim Esmerados work.8 We extended his work by adding more exibility and independence on the musician attributes while relying on current hardware. Our system relies on preordering data using statistical information based on Markov chains. Researchers have done some similar work in persistent worlds,10 but they need to quickly transfer small amounts of information through a network from a complex database. In our case, we store the data locally but must provide close to instantaneous access to the current state, depending on external events. This involves managing a fast access multiple-level database. In our implementation, the high level consists of direct interaction with the real database image by making requests to the hard drive that stores the

data. However, as soon as the user selects a music partition, we can upload the full table in the main system memory. Previous work11 has demonstrated that a latency of 50 ms or less is adequate for human beings, and 100 ms is the absolute limit for sound propagation. These constraints prevented us from modifying the partition directly. Because were working with a xed music score, we approached the problem from the opposite side. 12 Instead of trying to follow the animation with the sound processing, we used the sound information to reect and blend the right animation. A selected note within a music score triggers the events allowing the manipulation of the different skeletons bones of a specied musician (see H-Anim, http://www.h-anim.org). Thus, because the system is managed through sound events (see Figure 3), we can avoid delays in the sound processing. This approach degrades the animations smoothness a bit, which is an acceptable loss, because human senses are more affected by sound distortion than by visual artifacts.

Figure 4. Autonomous utist and the system installation with a large projection screen and 5.1 Dolby digital speakers.

Case study
In our case study, we focused on the data management required to maintain the synchronization of sound and animation, which is critical when providing a believable simulation of a virtual orchestra. Figure 4 shows our application running. Our interface has two input channels: the 2D GUI on the PDA screen and a magnetic tracker attached to it (see Figure 5, next page). From the 2D GUI, users have a simplied view of the scene where they can edit each musicians position and orientation in the scene. If the user modies the conductors position and orientation, the scenes point of view in the projection screen is modied to match the virtual conductors new position and orientation. The corresponding modifications in the sound environment are computed and applied by the internal sound component

JulySeptember 2004 47

(a)

presto
2

-4

forte

conducting gestures. To overcome this problem, we attached a magnetic tracker to the PDA. The magnetic sensor lets us acquire the amplitude and frequency of the gestures performed while the user holds the handhelda function of the rotation angles measured by the tracker to represent its orientation. The frequency is mapped into one of the possible values for the tempo, while the gestures average amplitude affects the dynamics (see Figure 5). We calculate the orientation values amplitude and frequency in real time. The orientation values measured by the magnetic tracker are divided into ve regions corresponding to the possible values for the dynamics; the frequency is calculated as a function of the gestures acceleration. The tracker (Ascension Flock of Birds; http://www.ascensiontech.com) is sampled at 60 Hz over a TCP connection. The information is processed by a program running on a dedicated PC gestures analyzer that sends the current tempo and dynamics values through the network (TCP sockets).

n Ha

eld Magnetic dh tracker

-2 2

Conclusions
Were still at the beginning of the research phase, in particular concerning the methods to better acquire the users expressivity in a simple, noncumbersome way. In the future, we plan to perform a formal evaluation of the efciency and user acceptance of the user interface and the immersion the system achieves. We must also carry out further research concerning the synthesis of autonomous musicians, the synchronization of music and animation, and the naturalness of the gestures performed by the virtual charactersour virtual musicians still perform robotic-type movements. However, we can affirm that combining 3D rendering with a high-quality 3D sound and handheld interfaces devices in VR applications produces an entertaining multimedia experience. It opens the possibilities to novel interaction paradigms and other research areas. MM

piano

Ascension Flock of Birds (b)

-2

adagio

Figure 5. (a) The handheld interface and (b) the interpretation of the data from the magnetic tracker. The blue dots represent each musicians position, and the green dot shows the conductor.

unit. Additionally, the PDA lets users play and pause the execution as well as select the scorebook the orchestra will interpret. The orchestra direction window is one of two alternatives available for modifying the performances tempo (largo, adagio, default, allegro, presto) and dynamics (pianissimo, piano, default, forte, fortissimo). The second method uses gestures tracking. The user can choose to work with the sliders, or in the case of the tempo, tap rhythmically with the stylus on the conducting window. The tapping frequency is mapped to one of the available tempo options. The application running on the PDA calculates this operation. If the user stops tapping, the last tempo detected is applied and the PDA sends the information to the main system. The GUI-based tempo and dynamics control doesnt allow for interpreting the user emotions expressed through the arm gestures, the typical

References
1. M. Ponder et al., VHD++ Development Framework: Towards Extendible Component Based VR/AR Simulation Engine Featuring Advanced Virtual Character Technologies, Computer Graphics Intl (CGI), 2003. 2. R. Altman, ed., Sound Theory Sound Practice, Routledge, 1992. 3. J.R. Boer, Game Audio Programming, Charles Rivers Media, 2002.

IEEE MultiMedia 48

4. T. Akenine-Mller and E. Haines, Real-Time Rendering, A.K. Peters, 2002. 5. S. Rabin, IA Game Programming Wisdom, Charles River Media, 2002. 6. R. Gerber, Advanced OpenMP Programming, Intel, 2003. 7. A. Binstock, Multithreading, Hyper-Threading, Multiprocessing: Now, Whats the Difference?, Pacic Data Works, Intel Developer Service, 2003 8. J. Esmerado, A Model of Interaction between Virtual Humans and Objects: Application to Virtual Musicians, Ph.D. Thesis, No. 2502, EPFL, 2001. 9. Y. Arafa et al., Two Approaches to Scripting Character Animation, Proc. Workshop `Embodied Conversational Agents: Lets Specify and Evaluate Them!, AAMAS,, 2002, p. 35. 10. K. Wilson, Game Object Structure, GDC, 2003. 11. M. ODonnel, Producing Audio for Halo, MS Press, 2002. 12. D. Sonnenschein, Sound Design, Michael Weise Production, 2001.

Mario Gutierrez is a research assistant and PhD student in the VRlab at EPFL. His research focuses on Internet and mobile applications, interaction techniques for VEs, and autonomous animation of virtual humans. He has an MS in computer science from the Monterrey Institute of Technology, Toluca Campus, Mexico.

Frederic Vexo is a senior researcher and project leader of the VRlab at EPFL. His research interests include virtual reality, humancomputer interaction, virtual music performance, and standards for multimedia content representation. He has a PhD from Reims University.

Sebastien Schertenleib is a research assistant and PhD student in the VRlab at EPFL. His research interests include VR system architectures for continuous VEs. He is also involved in reconstructing ancient heritage through augmented reality simulations. He has an MS in computer science from the Swiss Federal Institute of Technology in Lausanne.

Daniel Thalmann is a professor and director of the VRlab at EPFL. His research interests include virtual humans, computer animation, and networked VEs. He has a PhD in computer science from the University of Geneva and an Honorary Doctorate from University Paul-Sabatier in France. Contact the authors at {Sebastien.Schertenleib, Mario.Gutierrez, Frederic.Vexo, Daniel.Thalmann}@ ep.ch.

wireless networks SET gigabit Ethernet INDUSTRY enhanced parallel ports FireWire STANDARDS 802.11 token rings
IEEE Computer Society members work together to dene standards like IEEE 802, 1003, 1394, 1284, and many more.

JulySeptember 2004

HELP SHAPE FUTURE TECHNOLOGIES JOIN AN IEEE COMPUTER SOCIETY STANDARDS WORKING GROUP AT

computer.org/standards/
49