video game music overview

Game audio

Karen Collins

OUTLINE

– Defining Games Audio– History– Functions of Games Audio – Approaches to composition– diegesis and dynamic audio

Who cares about games?

• Majority of people in western world play games (75% of heads of households (in the United States) play computer or video games)

• average age for a game player is thirty• nearly half of game players are female (55%

male, 43% female). • “Women over the age of 18 represent a greater

portion of the gameplaying population (28%) than boys from ages 6 to 17 (21%).

Who cares about game audio?

• 83% of Adult Gamers listed sound as one of the most important video game console elements

• 47% of Game Console owners (18-25 yr) hook up their game console to a home theatre system

• 48% of Hardcore Gamers said surround sound is a purchase driver for next generation consoles

What is game audio?

1. Dialogue/speech sounds

2. Sound effects

3. Music/score

4. Ambience

5. Interface sounds (menu, etc.)

What’s wrong?

How games are not movies:The problems of games audio

• Linearity Vs. Non-Linearity/ unpredictability, mixing

• Interactivity with player

• Temporality (length)

• Budgets

• Delivery methods/technology

What is Interactive Audio?

• sound events occur in direct reaction to a player’s movements. The player triggers the cue, and can repeatedly activate it, such as by making a character jump up and down.

Interactive Audio:

FootstepsEnemy music cue

What is adaptive audio?

• “adaptive” audio is generally referred to as sound that reacts to transformations in the gameplay environment—such as a change from day to night set by a game’s timer mechanism. Adaptive audio is not directly triggered by a player

Adaptive Audio

Change in music day to nightWolf, crow

What is Dynamic Audio

• Dynamic audio is a broad term to refer to both interactive and adaptive audio

A brief history of game audio

(in 4 minutes!)

History

Functions of Games Sound

1. Commercial functions2. Structural reinforcement3. Immersion/suspension of disbelief4. Semiotics functions5. Narrative functions6. Emotional functions7. Aesthetic functions8. Kinetic functions

1. Commercial functions

– Sell games– Sell artists– Sell soundtracks– Sell movies– Sell other games– Branding: familiarization

Sales, marketing and branding

2. Structural reinforcement

• Mask structure and editing

• Enhance/reinforce structure of game

• Create structure

• Situate player in game matrix

• Indicate change in narrative

• Enhance continuity

Structural reinforcement

3. Immersion/suspension of disbelief

• Add realism: – Emphasize real sounds– Represent location/spatialization

• Create illusion

• Cover external noise

Immersion/suspension of disbelief

4. Semiotic functions

• Sonic symbols, leitmotifs: identify objects and decrease learning curve

• Preparatory function

• Acousmatic sound: Focus attention, identify goals

• Convey meaning, emotion

• Comment

• Add slant or bias

Semiotic Preparatory Function

A test of the semiotics of space/place in Zelda

1. Ranch (farm)2. Windmill3. Fire temple4. Evil Castle underground5. Valley of gypsy thieves6. Flying fairy7. Dawn8. Temple of time9. Ice cavern

ABCDEFGHI

5. Narrative functions

• Advancement of plots• Locate player in storyline• Anchor player in terms of game matrix/level,

etc. • anticipate/foreshadow• Dialogue can be a major event in plot• Access character’s thoughts/reveal character

details• Reveal goals

Narrative: correct response

6. Emotional functions

• Emotional and physiological: manipulate player

• Disturb or enhance gameplay: battle scenes

• Mood induction

• Create intimacy (immersive)

Emotional

7. Aesthetic functions

• Improve our appreciation of the game

• Make the game better/more enjoyable

• Make it “cool”

• Indicate genre/style

• Inter-textual referencing

8. Kinetic functions

• Aerobic exercise: fitness

• Music as motivating factor for movement

• Edutainment for toddlers, etc.

Kinetic Functions

• Dance Dance Revolution, etc.

Problems (and some solutions) of Interactive Audio in Games

Mixing: Problems:-too many sounds

-no “post production” mixing

-unpredictable timings: results in “muddy”,

clash of sounds

Mixing Solutions!

• Reduce amount of sounds

• Control frequencies of sounds

• Assign priorities in code– E.g. dialogue = priority “1”

• Ambience = priority “4”• Gunshot = priority “2”• Music = priority “3”• Footsteps = priority “5”

Problem: technological constraints

• Trying to write songs in a few kB for constrained technology such as mobile phones

Solutions: technological constraints:

• Playing a single sample at different pitches to produce variation.

• The "pitch it up, play it down" technique for saving space.

Sampling to save size

• This game contains only a single "trumpet fall" sound effect

• modify the playback sample rate so that each time you pick up a piece of cheese, the sound is played at a different pitch

• randomly vary the pitch during gameplay, so there's no repeating pattern to it

• The sound tends to mask the transition from one mix to another, helping to create a more seamless audio experience.

• same kind of pitch-shifting effect applied to car horn beeps: (fig 1)

Sampling to save space• Another way to save space is using the "pitch it up, play

it down" technique. • take your original, high-resolution sound effect and

transpose it up an octave, halving the length. • Then convert it to a low-resolution compressed format,

like this: (cat growl, octave up, car start, octave up. Then in the game, play it down an octave: (octave down).

• Although the game sound might be a little crunchy, you've just cut the size of your file in half without losing too much audio fidelity. Obviously, the higher you pitch the sound, the "crunchier" the playback will become, but the technique can be used for custom instruments as well as sound effects, and is particularly effective on the tiny speakers in mobile devices.

• Problem: transitions from cue to cue– No “post” =

unpredictable timings– Abrupt jumps are

jarring

(1) Solution: transitions: Cross fade

• Problem: Multi-player interactivity

• Unscripted events!

• parameter-based music is difficult:– Music cannot be tied to specific events or

locations– Music cannot be tied to specific

parameters

Solutions: multi-player interactivity

• Work with multi-players!

• Asheron’s Call II, Lord of the Rings: players can bring instruments and jam out with each other

Multi-player music

Problem: listener fatigue

• Loops are boring.• Loops are boring.• Loops are boring.• Loops are boring.• Loops are boring.• Loops are boring.• Loops are boring.• Loops are boring.• Loops are boring.• Loops are boring.

Solutions: Listener fatigue

• Learn how to use silence!

• “boredom switch”: timer for set levels to stop

• VARIABILITY!

Dynamic Composition: Variability

1. variable tempo2. variable pitch3. variable volume4. variable DSP5. variable melodies (algorithmic generation)6. Variable harmony (chord arrangements)7. Variable harmony (keys)8. variable mixing 9. Variable sequencing10. variable structure

Variable tempo

Variable tempo

Variable pitch

Variable Pitch

Variable volume

Variable DSP

• Change DSP effects to change music:– E.g. reverb to make it softer, dreamier– Phasing to create a “dazed” effect– Overdrive to make it more aggressive, etc.

Variable melodies

• Ballblazer (1984): the music plays forever, without repeating itself but without straying too far from the original theme. The bassline doesn't vary at all, only the lead line in the higher register is algorithmically varied.

Variable harmony

• Asheron’s Call 2: changes in mode indicate changes in gameplay:– A player standing alone in an area would

hear the background score in Lydian, but as creatures fill the area, the music transitioned to Phrygian. If monsters outnumber player, music transition to diminished mode and quickened drum groove

Variable mixing

• Adjust mix by– Bringing instruments up in mix that were

quiet or silent before– Drop elements from mix

Cheese Racer

• Muting and unmuting multiple tracks in a MIDI file to produce various mixes.– 2 drum loop samples,– three tones: bass, melody and chords

• In the code for the first level, ten different combinations of tracks: percussion + bass, percussion + bass + melody, bass + melody + chords, etc.

(cont..)

Notice how the mix changes every time the mouse gets a piece of cheese. (Fig 2 : exit) (Fig 3)

• Music loop 40 seconds. 10 different combinations: therefore, each level has more than 6 minutes of different music mixes. “Because the mix changes depending on gameplay, the music will never play exactly the same way twice, thereby increasing variation and decreasing "ear fatigue."”

• In fact, there are two more sets of tracks playing two more styles of music, using the same tempo and percussion tracks as the first level.

• Entire game contains almost 20 minutes of various music mixes, using only 68K of compressed sample and MIDI data.

Dynamic Music- example

• Layer of instrumentation added when Mario rides Yoshi in Super Mario World.

Russian Squares• Using layers, each cue

increment represents a puzzle row being cleared by the player.

Variable sequencing• Alter

sequences of music

• Zelda Ocarina: Hyrule field

Variable structure

• Create “transition matrix”: chart out all possible directions for each sequence, and create transitions for marker points/jumps

Transition Matrix: No One Lives Forever

• Music and Arranging by Guy Whitmore and Tobin Buttram

• This music clip moves through all six of the music states for the Ambush theme, demonstrating transitions called from a transition matrix. Time      Music State0:00      6 (combat 2)1:22      transition 1:30      2 (sub-ambient)2:14      transition2:16      4 (suspense)3:10      transition3:12      3 (ambient)4:08      transition4:12      5 (combat 1)5:33      transition     1 (silence)

No One Lives Forever

• EarthOrbit:

• The Ambush theme starts in music state 5 (combat 1), transitions to music state 2 (ambient), then transitions to music state 6 (combat 2) with motifs.

What else can be done?

• Dynamic music offers all kinds of interesting possibilities to have music as a driving force in a game!

Super Mario DS

• New Super Mario Bros DS– Enemies jump and fly

in time to music

Electroplankton

Rez

Vib Ribbon

Levels of dynamic sound

1. non-dynamic, non-diegetic sound2. Adaptive non-diegetic sound3. Interactive non-diegetic sound4. Non-dynamic diegetic sound5. Adaptive diegetic sound6. Interactive diegetic sound7. Kinetic diegetic and non-diegetic

(reminder: Non-Diegetic: Outside the character’s space. Diegetic: “real sounds”, inside character’s space)

Non-dynamic, non-diegetic sound

• non-dynamic, linear music or sound found most frequently in the cinematic sequences (movie sequences in which the player’s functions are stopped while a short clip to advance the plot is played). In other words, the player has no control over the possibility of interrupting the music (short of resetting/turning the game off).

Non-dynamic, non-diegetic sound

Adaptive non-diegetic sound

• Sound events occurring in reaction to gameplay, but which are unaffected by the player’s direct movements

• The player cannot re-trigger these events.

Adaptive non-diegetic sound

Interactive non-diegetic sound

• Sound events occurring in the character’s space

• The player can re-trigger these events.

Interactive non-diegetic sound

Non-dynamic diegetic sound

• In non-dynamic diegetic audio, the sound event occurs in the character’s space, but the character does not directly interact with it.

Non-dynamic diegetic sound

Adaptive diegetic sound

• Diegetic sounds adapt to gameplay environment (e.g. night to day)

Adaptive Diegetic Sounds

Interactive diegetic sounds

• occur in the character’s space and with which the player’s character can interact. The player instigates the audio cue, but does not necessarily affect the sound of the event once the cue is triggered.

Interactive Diegetic Sounds

Kinetic interactive Sound

• Player interacts with the sound by mimicking the sound/image kinetically– Guitar Hero– Donkey Konga