gaudio at aes conference 2016_published ver
TRANSCRIPT
End-to-End VR Audio Solution for Fully Immersive/Interactive Experience
Henney Oh, Ph. D.
2016 AES Int’l Conference on AVAR. Sep 30 - Oct 1, 2016. @ Los Angeles Convention Center
|
This workshop covers
2
• Backgrounds• Immersive and interactive • Binaural rendering• Formats: object-based audio vs. scene-based audio• End-to-end workflow for VR audio: codec-based and app-based
• Solutions• Mixing and mastering with GAUDIO Works• Sol Player, a reference quality VR/360 audio player (API)• GAO, a VR audio format for Ambisonics + object + channel• GAUDIO Craft, a VR/3D audio plug-in for gaming VR• Hands-on and practices
|
GAUDIO LAB is
3
• Founded by guys from MPEG Audio standardization community• Proudly, inventors of the binaural rendering technology for MPEG-H
3D Audio (ISO/IEC 23003-8) • Strong audio engineering forces of 5 Ph.D. and 2 Masters• Now, focus on providing end-to-end solution for whole VR audio
ecosystem
|
What is VR Audio ?
5
Technology or content that enables “being there” experience in respect of
auditory perception
Immersive & Interactive
|
What is interactive ?
7
• 3DOF (Yaw-Pitch-Roll)‣ Head orientation only‣ 360 Video‣ Ambisonics
• 6DOF (3DOF + X-Y-Z)‣ Head orientation + Walk around‣ Full VR‣ Object-based audio
DOF (Degree Of Freedom)
Yaw
Roll
Pitch
Yaw
Roll
Pitch
Y
Z X
Listener’s freedom of movements for interactivity in VR space
|
Loudspeaker or Headphone ?
8
• Hard to remove local room presence (you are “here”)
• Virtual space limited by room size • 1 scene / room
• No local room artifact (block the “here” environment)
• Unlimited virtual space • 1 scene / person
Vs.
Loudspeaker Headphone (earphone)
|
Headphone. then, “binaural”
9
= Having or relating to two ears
widely used to mean “headphone virtualization”
|
Binaural Rendering
10
HRTF (HRIR) Listen at anechoic room
BRIR = HRTF + RIR Listen at real room
Signal processing technology to make sound virtualise in 3D space with headphone representation using HRTF or BRIR
|
Binaural Rendering
11
Engine
=
Binaural rendering
VR audio Automobile
Core technology to enable VR audio Compared to the engine in automobile
Core: Core:
|
VR Audio Format
12
• Channel
• Ambisonics
• Object
Concern all the ecosystem from content creation, delivery and consumption
|
An exemplary VR sound scene
13
Guitar
BellBird
Fountain
You
BGM
Reflections
Diffusion
Direct
Diegetic Sound
Non-diegetic Sound
Scene
|
Channel-based
14
• A traditional scene representation
• Pre-rendered in studio to target channel configuration
• Each channel directly maps to each loudspeaker
• More channel, more immersive
|
Channel to Binaural Rendering
15
Set loudspeaker layout as the source positions and binaural rendering
45 degree315 degree
135 degree225 degree
|
Ambisonics (Scene-based)
16
• Loudspeaker-independent sound-field representation on spherical harmonics
• Captured by B-format microphone
• Easy for 3DOF, NOT suitable for 6DOF Spherical Harmonics
B-format FOA microphone
FOA (First Order Ambisonics)
HOA (Higher Order Ambisonics, 3rd)
|
Ambisonics
17
Non-diegetic stereo (BGM) CANNOT be implemented
|
Ambisonics to Binaural Rendering
18
Ambisonics to Channel Channel to BinauralRotation (3DOF)
Two-step rendering required after rotating sphere to the head orientation
|
Object-based
19
Object 1
Object 2
Object 3Object 4
Object 5 (Stereo)
Send every sound component constructing sound-scene to playback-side
Metadata
To Playback
|
Object to Binaural Rendering
20
Object to Binaural
Straightforward and easy. Directly from object position to the ears
Complexity linearly increases as the number of objects increases
|
Comparative benefit of object-based
21
Directly Binaural Rendering to Headphone
Rotation (3DOF)
Sound Sources (Tracks)
…
Binaural Rendering
Binaural Rendering
Channel Mapping
Channel mapped scene
PlayerProduction
ChannelRendering to
Channel
AmbisonicsSound Sources (Tracks)
Sound Sources (Tracks)
Object
……
…
Rendering to
Ambisonics
|
VR Audio Format Transformation
22
Channel
Object
AmbisonicsO
O
OO
|
Mixed format representation
23
Channel
Ambisonics
Object
Possible and recommended to represent sound scene with the combinations of the 3 formats
|
End-to-end workflow in VR audio
24
Sound acquisition Post-production Transmission Play/Rendering
Cinematic VR
Gaming VR
Codec-based distribution
App-based distribution
Player
+
Content + EngineAuthoring tool
(Game Engine)
Sound Source
Player + EngineContent (format)
Authoring tool(DAW)
Compatibility
Sound source
|
Codec-based vs. App-based
25
Codec-based App-based
Typically in Cinematic VR Gaming VR
What have to deliver Content only Content + Playback engine
Live streaming capability Yes No
Upgradability Difficult Easy
Content distribution platform
VR Players App market per OS
Solution for VR Audio
|
GAUDIO Core - the binaural rendering engine
27
• High quality - low complexity • Pinpoint spatial resolution• All 3 formats (Object, Ambisonics, Channel) • Advanced distance rendering• Source directivity, occulusion, reverberations• Diegetic & non-diegetic selection per each source• HRTF personalisation ready
Key Features
|
GAUDIO Works
28
Cinematic VR content
(360 video + VR audio)
Channel
Object
Ambisonics
VR/360 content creation tool for cinematic VR
Sound acquisition Post-production Transmission Play/Rendering
360 Video
|
GAUDIO Works
29
• Sound powered by GAUDIO Core • Support all formats for In-Out
• Object, Ambisonics, Channel• Intuitive user interfaces
• Realtime automation R/W on 360 Video• 2 independent 3D map-view• Automated listener’s head orientation
• Easy to export• Direct export to 360 video (mp4, mov)• AV lipsyncPro Tools HD
AAX Plug-in
Key Features
|
GAUDIO Works: Release
31
• Beta release today (Oct. 1, 2016)• Download at our website
http://www.gaudiolab.com/• It’s free! (forever)
|
GAUDIO Sol Player / Sol API
32
Enjoy VR
Sound acquisition Post-production Transmission Play/Rendering
VR/360 audio player / player api
Cinematic VR content
(360 video + VR audio)
L
R
|
GAUDIO Sol Player / Sol API
33
• Sound powered by GAUDIO Core
• Support any combination of object, Ambisonics, and/or channel
• Low complexity and low latency
• Support multiple platforms
C D E F G
do re mi fa Sol
Why ‘Sol’ ?
Sol Player = G-Player
Key Features
< Reference Sol Player >
|
GAUDIO Sol API
34
OS
Device
Android Mac Windows iOS (coming soon)
Cardboard Gear VR Oculus Rift HTC Vive
Now available in
|
GAUDIO Sol Player Hands-on (Android version)
35
videolink: https://youtu.be/JDr8uKv3hN0
|
Sol Reference Player: Release
36
• It’s also free! (forever)• Available at
• Android (Google Play) - already
• Gear VR (Oculus store) - under certification
• Oculus Rift (in a month)• HTC Vive (in a month)< Reference Sol Player >
• Stand-alone 360 video with Sol
• Monitoring purpose
• Demo for customer or end-user
|
GAO
37
• Dedicated and optimised format for VR audio content delivery
• Gaudio Ambisonics + Objects
• Interface between Works and Sol
• Deliver all VR audio formats including object, Ambisonics, and channel
• Independent from any compression codecs
Ambisonics Objects
FOA, HOA up to 64 objects
Sound acquisition Post-production Transmission Play/Rendering
|
GAUDIO Works, GAO and Sol
38
Other platforms supporting FOA
FOA
Sound acquisition Post-production Transmission Play/Rendering
|
GAUDIO Craft
39
• Sound powered by GAUDIO Core
• Sound can be simulated and monitored with GAUDIO Works
• Easy to migrate from existing projects
• Intuitive interface for developer easily localise the sounds
• Support major platforms:
Sound acquisition Post-production Transmission Play/Rendering
VR/3D game engine plug-in
Unity3D Unreal
Key Features
|
Conclusions
40
• Covered technical backgrounds and solutions in end-to-end VR audio ecosystem
• Introduced GAUDIO’s proposed solutions for the ecosystem: Works, Sol, GAO, and Craft
• Please let us know your problems, then, we keep supplying technologies aiming at ultimate VR audio experience
• Hands-on demo on the booth !
Sound for VRwww.gaudiolab.com | [email protected] | 501 475 1974
7083 Hollywood Boulevard, Los Angeles, CA
You’re brilliant, We’re hiring !(for LA office)
THANK YOU