and gameplay in 140 synchronization of music€¦ · synchronization of music and gameplay in 140...
TRANSCRIPT
![Page 1: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/1.jpg)
Synchronization of Music and Gameplay in 140
Jakob Schmid, audio, 140
![Page 2: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/2.jpg)
me
Jakob SchmidAudio programmer by day (Playdead)Composer by night
![Page 3: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/3.jpg)
140
Design and programming: Jeppe Carlsen
Visual design: Niels Fyrst, Andreas PeitersenAudio: Jakob Schmid (me)
Hobby project, 3 yearsUnity 3 Free
![Page 4: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/4.jpg)
IGF
IGF award 2013Excellence in Audio- honorable mention, Technical Excellence
Spilprisen 2014Sound of the Year
Nordic Game Award 2014Artistic Achievement
![Page 5: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/5.jpg)
release
140 is out:● Steam● Humble Store
![Page 6: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/6.jpg)
talk
● 140 music and game interaction● 140 audio production● Unity 4 music programming tips
![Page 7: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/7.jpg)
140 demo
![Page 8: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/8.jpg)
music and game interaction
![Page 9: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/9.jpg)
music and game interaction
Goals:● Control game elements from music● Control music from game progression
![Page 10: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/10.jpg)
music and game interaction
Goals:● Control game elements from music● Control music from game progression
![Page 11: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/11.jpg)
game element control
● Play music loop● Use audio time from loop to control game
elements (instead of game time)
![Page 12: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/12.jpg)
game element control
● Get audio time from playing loop● When next musical beat reached, raise
event● Game elements listen for events and trigger
animation on beats
![Page 13: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/13.jpg)
game element control
wait for 16th note #1
start moving
wait for 16th note #8
start moving
1.
2.
3.
4.
![Page 14: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/14.jpg)
tempo
How to get 16th notes from audio time?
![Page 15: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/15.jpg)
tempo
How to get 16th notes from audio time?
16th notes: 140 beat/m * 4 note/beat
= 560 note/m
= 560/60 note/s
60/560 s/note
![Page 16: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/16.jpg)
tempo
How to get 16th notes from audio time?
16th notes: 140 beat/m * 4 note/beat
= 560 note/m
= 560/60 note/s
60/560 s/note
~ A new 16th note every 0.107 s
![Page 17: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/17.jpg)
game element control
Using loops and audio time,we can control game elements from music.
![Page 18: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/18.jpg)
music and game interaction
Goals:● Control game elements from music● Control music from game progression
![Page 19: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/19.jpg)
music and game interaction
Goals:● Control game elements from music● Control music from game progression
![Page 20: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/20.jpg)
unity 3 audio timing
Using relative timing,play sound at audio time T:
AudioSource.Play(delay)
![Page 21: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/21.jpg)
unity 3 audio timing
Using relative timing,play sound at audio time T: now = AudioSettings.dspTime delay = T - now
AudioSource.Play(delay)
![Page 22: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/22.jpg)
unity 3 audio timing
Using relative timing,play sound at audio time T: now = AudioSettings.dspTime delay = T - now
AudioSource.Play(delay)
- Audio is running in a different thread!Timing can be inaccurate: e.g. audio update between getting time and playing sound
![Page 23: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/23.jpg)
what we needed
140 is a precise rhythm game.We needed sample-accurate timing for music!
![Page 24: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/24.jpg)
what we needed
140 is a precise rhythm game.We needed sample-accurate timing for music!
- That means a precision of 0.00002 s
![Page 25: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/25.jpg)
how we did it
Simple solution with sample-accurate timing:
● Loops of fixed length (or a multiple)● Start all loops in same frame, possibly
muted● Stopping a loop is OK, never restart
![Page 26: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/26.jpg)
how we did it
Simple solution with sample-accurate timing:
● Loops of fixed length (or a multiple)● Start all loops in same frame, possibly muted● Stopping a loop is OK, never restart
During game progression:● Control volume/muting and pan● Never change pitch
![Page 27: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/27.jpg)
how we did it
audio sourcecalculate volume and pan from
difference in positionaudio receiver
Simple attenuation and panning for music loops:
![Page 28: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/28.jpg)
music control
By using loops and modifying volume and pan,we can control music from game progress.
![Page 29: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/29.jpg)
audio production
![Page 30: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/30.jpg)
development limitations
● Looped tracks● Fixed tempo, 140 BPM● Fixed key, Cm
![Page 31: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/31.jpg)
inspired by ancient hardware
FM synthesis
samples
![Page 32: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/32.jpg)
toolset limitations
● Ableton Live, no plugins● FM synthesis: Operator● Samples: Simpler
![Page 33: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/33.jpg)
ableton demo
![Page 34: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/34.jpg)
unity music programming tips
![Page 35: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/35.jpg)
unity 4 playback timing
Unity 4 has absolute playback timing! Yay- enables sample accuracy without loops
![Page 36: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/36.jpg)
unity 4 playback timing
Unity 4 has absolute playback timing!- enables sample accuracy without loops
Toolset:
AudioSettings.dspTime
AudioSource.PlayScheduled(time)
![Page 37: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/37.jpg)
audio timing goal
loopLengthdspTime
loopLength
![Page 38: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/38.jpg)
naive solution
Play()
play
dspTime
start:next = now now = AudioSettings.dspTime
![Page 39: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/39.jpg)
naive solution
loopLength
Play()next += loopLength
play
dspTime
next
start:next = now
![Page 40: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/40.jpg)
naive solution
loopLength
Play()next += loopLength
play
dspTime
next
now = next?
start:next = now
![Page 41: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/41.jpg)
naive solution
loopLength
Play()next += loopLength
play
dspTime
next
now = next?Play()
play
start:next = now
![Page 42: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/42.jpg)
naive solution
loopLength
Play()next += loopLength
play
dspTime
next
loopLength
start:next = now
now = next?Play()next += loopLength
play next
![Page 43: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/43.jpg)
problem
Scheduling a sound for ‘now’ is already too late!
Audio is running in a different thread.
![Page 44: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/44.jpg)
solution
Scheduling a sound for ‘now’ is already too late!
Audio is running in a different thread.
- Add buffer time!
![Page 45: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/45.jpg)
unity 4 audio timing
buf
now
dspTime
start:next = now + buf now = AudioSettings.dspTime
![Page 46: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/46.jpg)
unity 4 audio timing
buf
PlayScheduled(next)
playnow
dspTime
start:next = now + buf
![Page 47: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/47.jpg)
unity 4 audio timing
loopLength
buf
PlayScheduled(next)next += loopLength
playnow
dspTime
next
start:next = now + buf
![Page 48: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/48.jpg)
unity 4 audio timing
loopLength
buf
PlayScheduled(next)next += loopLength
PlayScheduled(next)next += loopLength
now
dspTime
next
start:next = now + buf
buf
now + buf > next?
![Page 49: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/49.jpg)
unity 4 audio timing
loopLength
buf
PlayScheduled(next)next += loopLength
PlayScheduled(next)next += loopLength
dspTime
next
start:next = now + buf
play
buf
now + buf > next?PlayScheduled(next)
![Page 50: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/50.jpg)
goal
loopLength
PlayScheduled(next)next += loopLength
PlayScheduled(next)next += loopLength
dspTime
now + buf > next?PlayScheduled(next)
start:next = now + buf
accurate loop length
![Page 51: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/51.jpg)
pseudocode
Start:
buf = 0.1 // as low as possible
next = AudioSettings.dspTime + buf
Update:
now = AudioSettings.dspTime
if(now + buf > next)
audio.PlayScheduled( next )
next += loopLength
![Page 52: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/52.jpg)
audio timing demo
![Page 53: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/53.jpg)
music control demo
www.schmid.dk/gallery/play_scheduled/
- with example C# code
![Page 54: and Gameplay in 140 Synchronization of Music€¦ · Synchronization of Music and Gameplay in 140 Jakob Schmid, audio, 140. me ... Timing can be inaccurate: e.g. audio update between](https://reader033.vdocuments.mx/reader033/viewer/2022050513/5f9d663a8bb0f31849416268/html5/thumbnails/54.jpg)
Synchronization of Music and Gameplay in 140
jakob @ schmid.dk