AES UK North workshop on capturing and rendering audio for VR

Gavin Kearney (York)

170 Million VR users worldwide – from top end HTC Vive to Google daydream.
Applications in Game Audio, Cinematic VR, Teleconferencing and Social interactions.

Requirements: Congruent audio and video cues.

Acoustic transfer function

Establish head related impulse functions – derive filters – simulate.
For an entire sphere we would need lots of points and to interpolate between them – significant
processing required.

Alternative method: Virtual loudspeaker approach. Get HRFT’s for fixed virtual loudspeaker
locations. Easier to process.

Need to rotate soundfields – easy to do with ambisonic soundfields

‘Scene based audio’ Creating virtual mics in the direction of where loudspeakers will be.

You get timbral differences in different orders of ambisonics – colouration effects.

(An argument that content creators should be using their own HRTF’s). Some spatialization is
enhanced by VR since head tracking movement can help to localise the sound, but the pinnae is
crucial for height.

Sadie binaural database includes a variety of HRTF’s for different virtual loudspeakers setups.
https://www.york.ac.uk/sadie-project/binaural.html

Other tools:
Google VR SDK, Google jump – 360 video for youtube, Facebook spatial workstation.
https://facebook360.fb.com/downloads/spatial-workstation-windows-v2-0-0-beta/
Ambix plugins
http://www.matthiaskronlachner.com/?p=2015

Enda Bates (Trinity College Dublin)

Composer of spatial music.
360 video and spatial music are ideally suited.
Trinity 360 project
https://endabates.wordpress.com/2016/01/06/welcome-to-the-trinity-360-blog/
https://www.youtube.com/watch?v=90UgT1caDCk

Project comparing the differences between ambisonic microphones.

https://secure.aes.org/forum/pubs/conferences/?elib=18317

(Brian Fallon developed first order ambisonic encoder plugin for consumer Zoom H2n recorder
https://endabates.wordpress.com/2016/10/27/zoom-h2n-conversion-plugin/)

Paper ‘Spatial music, virtual reality, and 360 media’

http://www.aes.org/e-lib/browse.cfm?elib=18496
For 360 video there are tensions between where we might want to place the microphone (close to
the instruments) and where we have to place the camera (since with 360 cameras close objects can
produce artifacts). E.g. We might want the camera in an ‘audience’ position, but we would want the
microphone on stage.

Hyunkook Lee (Huddersfield University)

Use cardiod mic arrays for spatial recording.
Directional perception in the horizontal plane relies mainly on binaural cues from Inter channel level
difference / Inter channel time difference (ICLD / ICTD). Cardiod mic arrays can capture these
differences.

http://c4dm.eecs.qmul.ac.uk/events/wimp2/Lee.pdf
http://www.aes.org/e-lib/browse.cfm?elib=18511

Tom Parnell (BBC)

1/3 of the adult population of the UK listens to radio via headphones. Opportunities for binaural
work. Tom outlined a number of experiments that have taken place at the BBC with binaural and
spatial audio.

Radio 4 States of Mind: The sky is wider

Proms concerts recorded with Schoeps ORTF 3D + close spot mics (with delays based on distance
from).
Boulez Anthemes 2 in binaural http://www.bbc.co.uk/programmes/p0471dcv

Nuendo + IOSONO Spatial audio workstation

http://www.iosono-sound.com/professional-audio/

‘The turning forest’ Google daydream VR

http://www.bbc.co.uk/taster/projects/turning-forest
Location recordings made with 20 mic array. These then spatialized through virtual speakers in same
locations. The Audio Definition Model was used to integrate audio assets and metadata with their
spatial locations (http://www.bbc.co.uk/rd/publications/audio-definition-model-software). This was
then integrated through a plugin into the Unity game engine. Exported for VR use in Google
Daydream. Audio as a) ambisonic soundfields that rotate with user and b) spot sfx spatialized in real
time for the interactive elements.

Chris Pike
http://www.bbc.co.uk/rd/people/chris-pike
http://www.bbc.co.uk/rd/projects/360-video-virtual-reality
http://www.bbc.co.uk/rd/blog/2016-05-virtual-reality-sound-in-the-turning-forest