1

Digital Sound for the Cinema

Granular Synthesis, Delay-based effectsTheatre and Cinema sound and Sonic

Interaction Design

Sandra Paulettosandra.pauletto@york.ac.uk

Granular synthesis

• Introduced and explored by Iannis Xenakis(Formalised Music, 1971) in the 1970s and CurtisRoads (Microsound)

• The technique is based on the idea of creatingcomplex sounds by combining many very shortacoustic events called grains

• The grains should have a duration less than50/100ms, typically 10 to 30ms

• The density of grains can vary between hundreds andthousands per second

• Examples running water and footsteps

2

Grain parameters• Signal origin: synthesised or from a

recording (granular synthesis becomes asound processing effect)

• Duration: of the single grain• Frequency: of the wave contained in the

grain• Amplitude envelope: a sine, a gaussian

(bell-shaped), an ADSR, a window (e.g.hanning), a very short pulse

• Spatial localisation: spatial position in thestereo or surround field

3

The spacing between grain is uniform or slowlychanging

A pitch is perceivedFor example: the duration of each grain is 20ms

– just after one grain disappears another comes in– there will be 50 grains per second– i.e. the sequence of 50 grains is perceived as a periodic

waveform of 50Hz– grains are spaced by less than 20ms -> pitch increases

• Vibrato can be created by slowly oscillating thespacing between the grains

• Glissandi can be created if the spacing betweengrains changes slowly and continuously over time

Synchronous granular synthesis

Examples from Vocal Transformer in Logic

Asynchronous granular synthesis• All the parameters can be randomised: in particular

the spacing between grains, but also wave, duration,etc.

• The sound designer can control the parameters of therandomisatione.g. grain duration is between 10 to 20 ms;– the density of grains per second is between 10 and 500;– etc.

• The resulting sound will not have a clear pitchbecause the spacing is randomised

• This is a way to create clouds (Curtis Roads) ortextures of sound

4

Clouds

Granulation of sampled sounds

Example from GrooveShifter

Example fromVocal transformer

5

Blue Velvet, 1986Dir: David Lynch

6

Delay-based effects• A delay takes an audio signal and plays it back after a

specified delay time• Delay time: can range from several milliseconds to a

few seconds• Digital delays store samples in the memory of the

computer and they output them after the delay time haselapsed

• Various effects can be created with delays: chorus,flanger, phaser, echo, reverb, etc.

• The delayed sound is usually mixed with the originalsound and the levels of unprocessed (dry) andprocessed (wet) signals can be controlled independently

A simple delay diagram

0_G01 example

Delay time less than50ms: filtering effect

Delay time higherthan 50ms: echoeffect

7

Delay with feedback• Route some of the output of the delay

back into the input of the delay

• The feedback is multiplied by its gaincontrol < 1 (to avoid overload)

• Now the sound is repeated many times andeach time it becomes lower in level

– Delay times < 50ms -> filtering effect calledComb filter

– Delay times > 50ms -> fading echoes

Delay with feedback

2-G02 example

8

Multitap delay

• If a delay has more than 1 output it is known as amulti-tap delay

3-Multitap example

Various delay time settings• Reverb-like effect: many delays with feedbacks

• Chorus-like effect: a delay with feedback with adelay time of 20-100ms creates a doubling effect asif two sounds were played in unison

• Echo: delay time over 50ms

• A composition tool for building rhythms: if thedelay time is over 100ms then the repetition isevident and can be used to build rhythms

9

The Eternal Sunshine of the SpotlessMind, 2004

Dir: M. Gondry

10

Delay-based vibrato• The delay time, rather than being fixed, is

modulated• Using an LFO - usually triangular or

sinusoidal (sinusoids sound more natural)

DELAY

LFOINPUT OUTPUT

4-Variable delay example

Chorus effect• More than one instrument playing at the

same time– Adds thickness– Makes the sound richer

• In the real world sounds are not played inprecise synchronisation– delays between the sounds

• In addition, there can be a slight detuningbetween the pitches of the sounds– variable delay time as shown with the vibrato

11

Multi voice chorus

5-Multichorus example

THX1138, 1971Dir: G. Lucas

Sound Designer: Walter Murch

12

Reverberation

sound source

listener

direct sound

reflections

room walls

13

THX1138, 1971Dir: George Lucas

Sound Designer: Walter Murch

14

ReflectionsSound reflections from walls and objects are:1. Delayed with respect to the direct sound because

they have to travel a longer distance2. Weaker in loudness because walls and objects will

normally absorb some of the sound energy3. Lower in frequency: the high frequencies are

absorbed more quickly (high frequency damping)– The reflections of a sound are what we call the

reverberation or reverb– From the reverb of a room we can understand if the room is

big, small, and if there is reflective or damping material inthe room

– The delay times are short (< 50ms), i.e. we cannot heardistinct copies of the sound

Different parts of a reverb• Early reflections:

– just after the direct sound– well defined directional reflections– depend on the shape and size of the room– depend on the position of listener and sound source

• Diffuse reverberation or late reflections:– random reflections– more difficult to relate to the characteristics of the room– diffuse reverberation seems to relate to the size of the room and in good

concert halls it decays exponentially

Impulseresponse

15

Reverberation timeReverb time: amount of time it takes for the sound pressure

level or intensity to decay by 60dBsReverb time is a measure of:• the size of the room• the surfaces of the room

– Reflective materials: concrete, tile floors, brick walls andwindows

-> reverb time increases– Absorptive materials: curtains, heavy carpets, presence of

people

-> reverb time decreasesMedium concert hall reverb times: 1.5, 2 secs

Digital reverb: Schroeder’s reverb

To complete the reverb effect the direct sound needs tobe added reverb example

16

Comb filterImpulseresponse

Frequency response

All-pass filter

Feedback gain = g

Delay time: around5ms

The frequencies pass through thefilter

There will be a frequency-dependent phase shift

Impulseresponse

17

Variation on the Schroeder reverb1) High frequencies damping, i.e. lower frequencies have longer

reverb times– A low pass filter can be placed in the feedback path of the comb

filters, i.e. the reflected sound loses high frequencies (Dodge andJerse, p. 302)

– This helps making the sound of the reverb less “metallic”

2) To model the earlyreflections, a multitap delayline can be added at thebeginningNormally the first 80ms of theresponse is realised with themultitap delay

Blade Runner, 1982Dir: Ridley Scott

18

Sonic Interaction Design and its relationto Film and Theatre sound design

• Workshop on Sonic Interaction Design at TheUniversity of York, 2009• A call for sound design for a theatre scene• “The New Anny”, Martin Crimp (from ‘Attempts onher life”)• Prof. Stephen Barrass sound design• Theatre performance created at York semi-improvised• Directed by Prof. Mary Luckhurst

Pauletto, S., Hug, D., Barrass, S., Luckhurst, M. IntegratingTheatrical Strategies into Sonic Interaction Design, Proceedingsof AudioMostly Conference, 2010

19

Listening modes• Causal: the most common, consists of listening to a sound in

order to gather information about its cause (or source). Do notoverestimate the accuracy and potential, its capacity to furnishsure, precise data. It is the most common mode but also themost easily influenced and deceptive (Michel Chion)

-> storytelling

• Reduced: this mode focuses on the traits of the sound itself,independent of its cause and of its meaning (Pierre Schaeffer)

-> music, overall sound style

• Acousmatic: listening to a sound without seeing its cause(e.g. off-screen sound). We are looking for a cause using causallistening, but we end up searching in traits usually used byreduced listening. (Pierre Schaeffer/Michel Chion)

-> surprise

Listening modes

• Semantic: refers to a code or a language to interpret amessage. Listening to speech requires this mode (Michel Chion)

-> the importance of the voice

• Referential: consists of being aware of or affected by thecontext of the sound, linking not only to the source butprincipally to the emotional and dramatic meaning (DavidSonnenschein)

-> context, drama, emotion

20

“There is no soundtrack”(Chion)

• Synchresis: combination of synchroniation andsynthesis. The spontaneous and irresistible weldproduced between a particular auditory phenomenonand visual phenomenon when they occur at the sametime

• Magnetisation: the sound coming from thespeakers is attributed to the objects on the screen

The New Anny: sound design only

21

Creating a performance starting from sound was aliberating process for the director and the actor

The process allowed an in depth discussion of thelistening mechanisms at play when:- originating an idea, i.e. in rehearsal- when listening to sound alone- when listening while watching an interaction- it highlighted different perspectives:

the actor/user, the sound designer and interactiondesigner/director, the audience

Summary of main conclusions

22

Synchronicity and spatial relationships between thesound and the object/action are very important aspects thatallow us to believe a sound (plausible or not) to beproduced by a certain action or object

The meaning of sound in itself is not necessarily a staticgiven and changes in the interactive condition or mayeven become meaningful in this condition only

Many levels of interplay between sounds andinteractions were observed: some combination wasconsidered very neutral and easily accepted, on the otherhand, if a sound was perceived as very unrealistic as asound of a particular interaction then the effect was usuallycomical and surprising

Tendency to always attribute a meaning/interpretation

23

• Chion, M. Audio-vision, 1994

• Sonneschein, D. Sound Design, 2001

• Dodge, C. & Jerse, T. A. (1997) Computer Music:synthesis composition and performance, Publisher:Schirmer

• Roads, C. (1996) Computer Music Tutorial, MIT Press• Roads, C. (2004) Microsound, MIT Press• Farnell, A. (2009) Designing Sound, MIT Press• Lynn, P. A., and Fuerst, W. (1998) Introductory Digital

Signal Processing with Computer Applications, JohnWiley and Sons Ltd.

Resources

• Puckette, M. The Theory and Technique of Electronic Music,2007, available at http://crca.ucsd.edu/~msp/techniques.htm

• Pure Data documentation:http://crca.ucsd.edu/~msp/Pd_documentation/

• Chafe, C. (1999) A short history of digital sound synthesis bycomposers in the USA,ccrma.stanford.edu/~cc/lyon/historyFinal.pdf

• Smith, J. (1991) Viewpoints of the history of digital synthesis,CCRMA

http://www-ccrma.stanford.edu/~jos/kna/

• Pauletto, S., Hug, D., Barrass, S., Luckhurst, M. IntegratingTheatrical Strategies into Sonic Interaction Design,Proceedings of AudioMostly Conference, 2010

Resources