history and practice of digital sound synthesis

85
Julius Smith AES-2006 Heyser Lecture – 1 / 84 History and Practice of Digital Sound Synthesis Julius Smith CCRMA, Stanford University AES-2006 Heyser Lecture October 6, 2006

Upload: trinhthien

Post on 30-Dec-2016

220 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: History and Practice of Digital Sound Synthesis

Julius Smith AES-2006 Heyser Lecture – 1 / 84

History and Practice of Digital Sound Synthesis

Julius SmithCCRMA, Stanford University

AES-2006 Heyser Lecture

October 6, 2006

Page 2: History and Practice of Digital Sound Synthesis

Overview

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 2 / 84

Page 3: History and Practice of Digital Sound Synthesis

Outline

Overview

• Outline

• CCRMA Perspective

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 3 / 84

Digital sound synthesis approaches in approximate historical order:

• Wavetable (one period)• Subtractive• Additive• Frequency Modulation (FM)• Sampling• Spectral Modeling• Physical Modeling

Some connections with audio coding will be noted

Emphasis:

• Sound examples• Block diagrams• Historical notes

Page 4: History and Practice of Digital Sound Synthesis

CCRMA Perspective

Overview

• Outline

• CCRMA Perspective

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 4 / 84

The Knoll, Stanford University

Page 5: History and Practice of Digital Sound Synthesis

Early Digital Sound Synthesis

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 5 / 84

Page 6: History and Practice of Digital Sound Synthesis

Wavetable Synthesis in Music I-V (1957-1969)

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 6 / 84

Page 7: History and Practice of Digital Sound Synthesis

Music V Scripting Language (“Note Cards”)

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 7 / 84

• Essentially Supported in MPEG-4 Structured Audio OrchestraLanguage (SAOL) (Music V → csound→ SAOL)

• “Encoding sounds” as “instruments” is hard, in general

Page 8: History and Practice of Digital Sound Synthesis

Kelly-Lochbaum Vocal Tract Model

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 8 / 84

k1 − k1

1 − k1

1 + k1

R1

z− 21

z− 21

z− 21

z− 21

…kM − kM

1 − kM

1 + kM

RM

Glottal PulseTrain or Noise

e(n)SpeechOutput

y(n)

(UnusedAllpassOutput)

Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical)

…e(n) y(n)

John L. Kelly and Carol Lochbaum (1962)

Page 9: History and Practice of Digital Sound Synthesis

Sound Example

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 9 / 84

“Bicycle Built for Two”: (WAV) (MP3)

• Vocal part by Kelly and Lochbaum (1961)• Musical accompaniment by Max Mathews• Computed on an IBM 704• Based on Russian speech-vowel data from Gunnar Fant’s book• Probably the first digital physical-modeling synthesis sound

example by any method• Inspired Arthur C. Clarke to adapt it for “2001: A Space Odyssey”

— the computer’s “first song”

Page 10: History and Practice of Digital Sound Synthesis

Classic Additive-Synthesis Analysis (Heterodyne Comb)

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 10 / 84

John Grey 1975 — CCRMA Tech. Reports 1 & 2(CCRMA “STANM” reports — available online)

Page 11: History and Practice of Digital Sound Synthesis

Classic Additive-Synthesis (Sinusoidal Oscillator Envelopes)

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 11 / 84

John Grey 1975 — CCRMA Tech. Reports 1 & 2(CCRMA “STANM” reports — available online)

Page 12: History and Practice of Digital Sound Synthesis

Classic Additive Synthesis Diagram

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 12 / 84

noise

FIR

A1(t) A2(t) A3(t) A4(t)f1(t) f2(t) f3(t) f4(t)

Σ

y(t) =4

i=1

Ai(t) sin

[∫ t

0

ωi(t)dt + φi(0)

]

Page 13: History and Practice of Digital Sound Synthesis

Classic Additive-Synthesis Examples

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 13 / 84

• Bb Clarinet• Eb Clarinet• Oboe• Bassoon• Tenor Saxophone• Trumpet• English Horn• French Horn• Flute

• All of the above• Independently synthesized set

(Synthesized from original John Grey data)

Page 14: History and Practice of Digital Sound Synthesis

Frequency Modulation (FM) Synthesis

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 14 / 84

FM synthesis is normally used as a spectral modeling technique.

• Discovered and developed (1970s) by John M. Chowning(CCRMA Founding Director)

• Key paper: JAES 1973 (vol. 21, no. 7)• Commercialized by Yamaha Corporation:

DX-7 synthesizer (1983) OPL chipset (SoundBlaster PC sound card) Cell phone ring tones

• On the physical modeling front, synthesis of vibrating-stringwaveforms using finite differences started around this time:Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)

Page 15: History and Practice of Digital Sound Synthesis

FM Formula

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 15 / 84

x(t) = Ac sin[ωct + φc + Am sin(ωmt + φm)]

where

(Ac, ωc, φc) specify the carrier sinusoid(Am, ωm, φm) specify the modulator sinusoid

Can also be called phase modulation

Page 16: History and Practice of Digital Sound Synthesis

Simple FM “Brass” Patch (1970–)

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 16 / 84

Jean-Claude Risset observation (1964–1969):Brass bandwidth ∝ amplitude

A F

FA

Out

fc = f0

fm = f0

g

Page 17: History and Practice of Digital Sound Synthesis

FM Harmonic Amplitudes (Bessel Function of First Kind)

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 17 / 84

Harmonic number k, FM index β:

0

2

4

6

8

10

05

1015

2025

30

−0.5

0

0.5

1

Order k

Argument β

J k(β)

Page 18: History and Practice of Digital Sound Synthesis

Frequency Modulation (FM) Examples

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 18 / 84

All examples by John Chowning unless otherwise noted:

• FM brass synthesis

Low Brass example Dexter Morril’s FM Trumpet

• FM singing voice (1978)Each formant synthesized using an FM operator pair(two sinusoidal oscillators)

Chorus Voices Basso Profundo

• Other early FM synthesis

Clicks and Drums Big Bell String Canon

Page 19: History and Practice of Digital Sound Synthesis

FM Voice

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 19 / 84

FM voice synthesis can be viewed as compressed modeling ofspectral formants

Frequency f0

Mag

nitu

de

Carrier 1 Carrier 2

Carrier 3

Modulation Frequency (all three)

Page 20: History and Practice of Digital Sound Synthesis

Sampling Synthesis History

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 20 / 84

• 1979 - Fairlight Computer Music Instrument - 8-bit

First commercial sampler Eight voices, 8 bits, 64 KB (4 sec) RAM, 16 kHz (mono) Editing, looping, mixing One could draw waveforms and additive-synthesis amplitude

envelopes (for each harmonic) with a light pen $25,000–$36,000!

• 1981 - E-mu Systems Emulator

First “affordable” sampler ($10,000) Eight voices, 128 K RAM, 8-bit, 80 lb.

• 1986 - Ensoniq Mirage

Breakthrough price-point ($1695) Eight voices, 144 K RAM, 8-bit

Page 21: History and Practice of Digital Sound Synthesis

Modern Sampled Piano

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 21 / 84

Example: 1

• 40 Gigabytes on ten DVDs (three sampled pianos)• Every key sampled• 4–10 “velocity layers”• Separate recordings with soft pedal down• Separate “release” recordings, for multiple striking velocities

1Synthogy Ivory, $349 (Electronic Musician, October 2006)

Page 22: History and Practice of Digital Sound Synthesis

Fundamental Problem with Sampling Synthesis

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 22 / 84

Piano timbre is determined by

• key number (1 byte)• key velocity (2 bytes more than enough)• pedal state (1 bit [or byte] per pedal)

Piano control is relatively low-dimensional:

• Less than six bytes of information per note played• No continuous controls (typically)• Ratio of total sampled data to one note of control data

≈ one billion

Page 23: History and Practice of Digital Sound Synthesis

Now consider bowed strings

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 23 / 84

Control parameters:

• Left-hand finger position(s)• Left-hand vibrato• Bow velocity• Bow force• Bow position• Bow angle• Shoulder damping• Instrument orientation• Player motion (within a room)

Page 24: History and Practice of Digital Sound Synthesis

Difficulty of sampling bowed strings

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 24 / 84

• Bowed-string control is infinite-dimensional in principle• Many time-varying functions — “gestures”

(we counted more than 10)• Complete sampling of bowed strings on the level of pianos has

apparently never been done• Rule-driven navigation of the most useful recorded playing

regimes has worked well (e.g., Synful Orchestra)• Model-based approaches greatly reduce data requirements:

Spectral models (inspired by sound perception) Physical models (model the sound source)

Page 25: History and Practice of Digital Sound Synthesis

Spectral Modeling Synthesis(Historical Summary)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 25 / 84

Page 26: History and Practice of Digital Sound Synthesis

Classic Vocoder Analysis & Resynthesis (Dudley 1939)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 26 / 84

f

A

f

A

f

A

Analysis Synthesis

Data Compression,Transmission,Storage,Manipulation,Noise reduction, ...

Processing

magnitude, ormagnitude andphase extraction

x(t)

x0(t)

x1(t)

xN−1

x(t)

x0(t)

x1(t)

xN−1

Page 27: History and Practice of Digital Sound Synthesis

Phase Vocoder Channel Model

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 27 / 84

ωk ω0

Channel FilterResponse

ak

∆ωk

Analysis Model Synthesis Model

ak(t) ωk+∆ωk(t)

A F

Out

Sine Osc

• Early “channel vocoder” implementations (hardware) onlymeasured amplitude ak(t) (Dudley 1939)

• The “phase vocoder” (Flanagan and Golden 1966) added phasetracking in each channel

• Portnoff (1976) developed the FFT phase vocoder,which replaced the heterodyne comb in computer-musicadditive-synthesis analysis (James A. Moorer)

• Inverse FFT synthesis (Rodet and Depalle 1992) gave fastersinusoidal oscillator banks

Page 28: History and Practice of Digital Sound Synthesis

Amplitude and Frequency Envelopes

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 28 / 84

ak(t)

∆ωk(t) = φk(t)

0

t

t

Page 29: History and Practice of Digital Sound Synthesis

Channel Vocoder Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 29 / 84

• Original

• 10 channels, sine carriers• 10 channels, narrowband-noise carriers

• 26 channels, sine carriers• 26 channels, narrowband-noise carriers• 26 channels, narrowband-noise carriers, channels reversed

• Phase Vocoder: Identity system in absence of modifications

• The FFT Phase Vocoder next transitioned to the Short-TimeFourier Transform (STFT) (Allen and Rabiner 1977)

Page 30: History and Practice of Digital Sound Synthesis

Tracking Spectral Peaks in the Short-Time Fourier Transform

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 30 / 84

dB magPeaktracking

QuadraticPeakInterpolation

Frequencies

Amplitudes

Phases

window w(n)

s(t)

atan

FFT

• STFT peak tracking at CCRMA: mid-1980s (PARSHL program)• Motivated by vocoder analysis of piano tones• Influences: STFT (Allen and Rabiner 1977),

ADEC (1977), MAPLE (1979)• Independently developed for speech coding by McAulay and

Quatieri at Lincoln Labs

Page 31: History and Practice of Digital Sound Synthesis

Example Spectral Trajectories

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 31 / 84

f

t

Page 32: History and Practice of Digital Sound Synthesis

Sines + Noise Synthesis (1989)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 32 / 84

PSfragA1(t) A2(t) A3(t) A4(t)f1(t) f2(t) f3(t) f4(t)

∑white noiseu(t) filter(t)

ht(τ)

y(t) =4

i=1

Ai(t) cos[

∫ t

0ωi(t)dt + φi(0)

]

+ (ht ∗ u)(t)

Page 33: History and Practice of Digital Sound Synthesis

Sines + Noise Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 33 / 84

Xavier Serra 1989 thesis demos (Sines + Noise signal modeling)

• Piano

Original Sinusoids alone Residual after sinusoids removed Sines + noise model

• Voice

Original Sinusoids Residual Synthesis

Page 34: History and Practice of Digital Sound Synthesis

Musical Effects with Sines+Noise Models (Serra 1989)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 34 / 84

• Piano Effects

Pitch downshift one octave Pitch flattened Varying partial stretching

• Voice Effects

Frequency-scale by 0.6 Frequency-scale by 0.4 and stretch partials Variable time-scaling, deterministic to stochastic

Page 35: History and Practice of Digital Sound Synthesis

Cross-Synthesis with Sines+Noise Models (Serra 1989)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 35 / 84

• Voice “modulator”

• Creaking ship’s mast “carrier”• Voice-modulated creaking mast• Same with modified spectral envelopes

Page 36: History and Practice of Digital Sound Synthesis

Sines + Transients Sound Examples (Serra 1989)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 36 / 84

In this technique, the sinusoidal sum is phase-matched at thecross-over point only (with no cross-fade).

• Marimba

Original Sinusoidal model Original attack, followed by sinusoidal model

• Piano

Original Sinusoidal model Original attack, followed by sinusoidal model

Page 37: History and Practice of Digital Sound Synthesis

Multiresolution Sines + Noise + Transients (Levine 1998)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 37 / 84

Why Model Transients Separately?

• Sinusoids efficiently model spectral peaks over time• Filtered noise efficiently models spectral residual vs. t• Neither is good for abrupt transients in the waveform• Phase-matched oscillators are expensive• More efficient to switch to a transient model during transients• Need sinusoidal phase matching at the switching times

Transient models:

• Original waveform slice (1988)• Wavelet expansion (Ali 1996)• MPEG-2 AAC (with short window) (Levine 1998)• Frequency-domain LPC

(time-domain amplitude envelope) (Verma 2000)

Page 38: History and Practice of Digital Sound Synthesis

Time Scale Modification of Sines + Noise + Transients Models

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 38 / 84

transien

ts

transien

ts

transien

ts

transien

ts

sines +noise

sines +noisesines +noise

sines +noise

sines +noise

sines +noise

sines +noise

sines +noise

sines +noise

original signal

time-scaledsignal

time

Time-Scale Modification (TSM) becomes well defined :

• Transients are translated in time• Sinusoidal envelopes are scaled in time• Noise-filter envelopes also scaled in time• Dual of TSM is frequency scaling

Page 39: History and Practice of Digital Sound Synthesis

Sines + Noise + Transients Time-Frequency Map

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 39 / 84

0 50 100 150 200 250−1

−0.5

0

0.5

1

time [milliseconds]

ampl

itude

0 50 100 150 200 2500

2

4

6

8

10

12

14

16

freq

uenc

y [k

Hz]

(Levine 1998)

Page 40: History and Practice of Digital Sound Synthesis

Corresponding Analysis Windows

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 40 / 84

0 50 100 150 200 250time [milliseconds]

ampl

itude

tran

sien

thi

gh o

ctav

em

iddl

e oc

tave

low

oct

ave

Page 41: History and Practice of Digital Sound Synthesis

Quasi-Constant-Q (Wavelet) Time-Frequency Map

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 41 / 84

0 50 100 150 200 250

−2

−1

0

1

2

x 104

time [milliseconds]

ampl

itude

0 50 100 150 200 2500

2

4

6

8

10

12

14

16

freq

uenc

y [k

Hz]

Page 42: History and Practice of Digital Sound Synthesis

Bark-Band Noise Modeling at High Frequencies (Levine 1998)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 42 / 84

0 500 1000 1500 2000 2500 3000 3500 4000 450030

35

40

45

50

55

60

65

70

75

80

85

frequency [Hz]

mag

nitu

de [d

B]

Page 43: History and Practice of Digital Sound Synthesis

Amplitude Envelope for One Noise Band

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 43 / 84

0 50 100 150 200 250 300 35040

50

60

70

80

Orig

inal

Mag

. [dB

]

0 50 100 150 200 250 300 35045

50

55

60

65

70

75

80

time [milliseconds]

LSA

Mag

. [dB

]

For more information, see Scott Levine’s thesis.22http://ccrma.stanford.edu/~scottl/thesis.html

Page 44: History and Practice of Digital Sound Synthesis

Sines + Noise + Transients Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 44 / 84

Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)(http://ccrma.stanford.edu/~scottl/thesis.html)

Mozart’s Le Nozze di Figaro

• Original• Compressed using MPEG-AAC at 32 kbps• Compressed using sines+transients+noise at 32 kbps

• Multiresolution sinusoids alone• Residual Bark-band noise• Transform-coded transients (AAC)• Bark-band noise above 5 kHz

Page 45: History and Practice of Digital Sound Synthesis

Rock Example

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 45 / 84

Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)(http://ccrma.stanford.edu/~scottl/thesis.html)

“It Takes Two” by Rob Base & DJ E-Z Rock

• Original• MPEG-AAC at 32 kbps• Sines+transients+noise at 32 kbps

• Multiresolution sinusoids• Residual Bark-band noise• Transform-coded transients (AAC)• Bark-band noise above 5 kHz

Page 46: History and Practice of Digital Sound Synthesis

Time Scale Modification using Sines + Noise + Transients

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 46 / 84

Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)(http://ccrma.stanford.edu/~scottl/thesis.html)

Time-Scale Modification (pitch unchanged)

• S+N+T time-scale factors [2.0, 1.6, 1.2, 1.0, 0.8, 0.6, 0.5]

S+N+T Pitch Shifting (timing unchanged)

• Pitch-scale factors [0.89, 0.94, 1.00, 1.06, 1.12]

Page 47: History and Practice of Digital Sound Synthesis

Spectral Modeling History Highlights

Julius Smith AES-2006 Heyser Lecture – 47 / 84

• Fourier’s theory (1822)• Teleharmonium (1906)• Hammond organ (1930s)• Channel Vocoder (1939)• Phase Vocoder (1966)• “Additive Synthesis” (1969)• FFT Phase Vocoder (1976)• Sinusoidal Modeling

(1977,1979,1985)• Sines+Noise (1989)• Sines+Transients (1989)• Sines+Noise+Transients (1998)

Perceptual audio coding:

• Princen-Bradley filterbank (1986)• K. Brandenburg thesis (1989)• Auditory masking usage• Dolby AC2• Musicam• ASPEC• MPEG-I,II,IV

(incl. S+N+T “parametric sounds”)

Page 48: History and Practice of Digital Sound Synthesis

Future Prospects

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 48 / 84

Observations:

• Sinusoidal modeling of sound is “Unreasonably Effective”• Basic “auditory masking” discards ≈ 90% information• Interesting neuroscience observation:

“... most neurons in the primary auditory cortex A1are silent most of the time ...”

(from “Sparse Time-Frequency Representations”, Gardner andMagnesco, PNAS:103(16), April 2006)

• What is a true and correct “psychospectral model” for sound?

The cochlea of the ear is a real-time spectrum analyzer How is the “ear’s spectrogram” represented at higher levels?

Page 49: History and Practice of Digital Sound Synthesis

Physical Modeling Synthesis(Historical Summary)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 49 / 84

Page 50: History and Practice of Digital Sound Synthesis

Kelly-Lochbaum Vocal Tract Model

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 50 / 84

k1 − k1

1 − k1

1 + k1

R1

z− 21

z− 21

z− 21

z− 21

…kM − kM

1 − kM

1 + kM

RM

Glottal PulseTrain or Noise

e(n)SpeechOutput

y(n)

(UnusedAllpassOutput)

Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical)

…e(n) y(n)

John L. Kelly and Carol Lochbaum (1962)

Page 51: History and Practice of Digital Sound Synthesis

Digital Waveguide Models (1985)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 51 / 84

Lossless digital waveguide∆= bidirectional delay line

at some wave impedance R

z−N

z−N

R

Useful for efficient models of

• strings• bores• plane waves• conical waves

Page 52: History and Practice of Digital Sound Synthesis

Signal Scattering

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 52 / 84

Signal scattering is caused by a change in wave impedance R:

z−N

z−N

k1 − k1

1 − k1

1 + k1

z−N

z−N

R1 R2

RRRR

k12

121 +

−=

If the wave impedance changes every spatial sample, theKelly-Lochbaum vocal-tract model results.

Page 53: History and Practice of Digital Sound Synthesis

Ideal Plucked String (Displacement Waves)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 53 / 84

(x = 0) (x = L)

y (n+N/2)

-1“Bridge”

y (n)+

“Nut”

-y (n)-

-1

y (n-N/2)+

(x = Pluck Position)

• Load each delay line with half of initial string displacement• Sum of upper and lower delay lines = string displacement

Page 54: History and Practice of Digital Sound Synthesis

Ideal Struck String (Velocity Waves)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 54 / 84

(x = 0) (x = L)

v (n+N/2)

-1“Bridge”

v (n)+

“Nut”

-v (n)-

-1

v (n-N/2)+

(x = Hammer Position)

c

c

Hammer strike = momentum transfer = velocity step:

mhvh(0−) = (mh + ms)vs(0+)

Page 55: History and Practice of Digital Sound Synthesis

Karplus-Strong (KS) Algorithm (1983)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 55 / 84

N samples delayOutput y (n)+

z 1-

1/2

1/2

y (n-N)+

• Discovered (1978) as “self-modifying wavetable synthesis”• Wavetable is preferably initialized with random numbers

Page 56: History and Practice of Digital Sound Synthesis

Karplus-Strong Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 56 / 84

• “Vintage” 8-bit sound examples:

• Original Plucked String: (WAV) (MP3)• Drum: (WAV) (MP3)• Stretched Drum: (WAV) (MP3)

Page 57: History and Practice of Digital Sound Synthesis

EKS Algorithm (Jaffe-Smith 1983)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 57 / 84

Hβ(z)

Hρ(z) Hs(z)

HL(z)Hp(z)

Hd(z)

z−N

N = pitch period (2× string length) in samples

Hp(z) =1 − p

1 − p z−1= pick-direction lowpass filter

Hβ(z) = 1 − z−βN = pick-position comb filter, β ∈ (0, 1)

Hd(z) = string-damping filter (one/two poles/zeros typical)

Hs(z) = string-stiffness allpass filter (several poles and zeros)

Hρ(z) =ρ(N) − z−1

1 − ρ(N) z−1= first-order string-tuning allpass filter

HL(z) =1 − RL

1 − RL z−1= dynamic-level lowpass filter

Page 58: History and Practice of Digital Sound Synthesis

STK EKS Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 58 / 84

• Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, andothers — distributed by CCRMA:Google search: STK ToolKit

STK Plucked String: (WAV) (MP3)

• Plucked String 1: (WAV) (MP3)• Plucked String 2: (WAV) (MP3)• Plucked String 3: (WAV) (MP3)

Page 59: History and Practice of Digital Sound Synthesis

EKS Sound Example (1988)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 59 / 84

Bach A-Minor Concerto—Orchestra Part: (WAV) (MP3)

• Executed in real time on one Motorola DSP56001(20 MHz clock, 128K SRAM)

• Developed for the NeXT Computer introduction at DaviesSymphony Hall, San Francisco, 1988

• Solo violin part was played live by Dan Kobialka of the SanFrancisco Symphony

Page 60: History and Practice of Digital Sound Synthesis

Digital Waveguide Single Reed, Cylindrical Bore Model (1986)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 60 / 84

Bell

MouthPressure

EmbouchureOffset

Reed to Bell Delay( )npm

2

BoreReed

ReflectionFilter

OutputFilter

Bell to Reed Delay

( )np+b

( )np−b

-

-h∆

+

hm

-

Reed Table

Digital waveguide clarinet

• Control variable = mouth half-pressure• Total reed cost = two subtractions, one multiply, and one table

lookup per sample

Page 61: History and Practice of Digital Sound Synthesis

Digital Waveguide Wind Instrument Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 61 / 84

• STK Clarinet: (WAV) (MP3)Google search: STK clarinet

Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, andothers — distributed by CCRMA:Google search: STK ToolKit

• Staccato Systems Slide Flute(based on STK flute, ca. 1995): (WAV) (MP3)

• Yamaha VL1 “Virtual Lead” synthesizer demos (1994):

• Shakuhachi: (WAV) (MP3)• Oboe and Bassoon: (WAV) (MP3)• Tenor Saxophone: (WAV) (MP3)

Page 62: History and Practice of Digital Sound Synthesis

Digital Waveguide Bowed Strings (1986)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 62 / 84

Bridge-Body

Bow Force

Bow to Bridge Delay

String

ReflectionFilter

BodyFilter

Bridge to Bow Delay

Nut to Bow Delay

Bow to Nut Delay

-1

String BowNut Air

Bow Velocity

-

v+ls,

v+rs,

v∆+ ρ

v−ls,

v−rs,

- *vb

Bow Table

• Reflection filter summarizes all losses per period(due to bridge, bow, finger, etc.)

• Bow-string junction = memoryless lookup table(or segmented polynomial)

Page 63: History and Practice of Digital Sound Synthesis

“Electric Cello” Sound Examples (Peder Larson)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 63 / 84

• Staccato Notes: (WAV) (MP3)(short strokes of high bow pressure, as from a bouncing bow)

• Bach’s First Suite for Unaccompanied Cello: (WAV) (MP3)

Page 64: History and Practice of Digital Sound Synthesis

Soft Clipper

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 64 / 84

f(x) =

−2

3, x ≤ −1

x −x3

3, −1 ≤ x ≤ 1

2

3, x ≥ 1

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8x=−1:0.01:1; plot([−(2/3)*ones(1,100), x−x.3/3, (2/3)*ones(1,100)])

x(n)

f(x(n

))

Page 65: History and Practice of Digital Sound Synthesis

Amplifier Distortion + Amplifier Feedback

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 65 / 84

Sullivan 1990

GainFeedbackAmplifier

...

Pre-distortion output level

Pre-distortion gain

Output Signal

Distortion output level

Nonlinear Distortion

Amplifier Feedback Delay

String 1

String N

Distortion output signal often further filtered by an amplifier cabinetfilter, representing speaker cabinet, driver responses, etc.

Page 66: History and Practice of Digital Sound Synthesis

Distortion Guitar Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 66 / 84

(Stanford Sondius Project, ca. 1995)

• Distortion Guitar: (WAV) (MP3)• Amplifier Feedback 1: (WAV) (MP3)• Amplifier Feedback 2: (WAV) (MP3)

Page 67: History and Practice of Digital Sound Synthesis

Commuted Synthesis of Acoustic Strings (1993)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 67 / 84

Trigger Outpute(t) s(t) y(t)

ResonatorStringExcitation

Schematic diagram of a stringed musical instrument.

Trigger OutputResonator StringExcitation

Equivalent diagram in the linear, time-invariant case.

AggregateExcitation

a(t)String

x(t)OutputTrigger

Use of an aggregate excitation given by the convolution of originalexcitation with the resonator impulse response.

Page 68: History and Practice of Digital Sound Synthesis

Commuted Components

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 68 / 84

AggregateExcitation

a(t)String

x(t)OutputTrigger

“Plucked Resonator” driving a String.

y(t)BridgeCoupling

s(t) GuitarBody

AirAbsorption

RoomResponse Output

Possible components of a guitar resonator.

Page 69: History and Practice of Digital Sound Synthesis

Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 69 / 84

Electric Guitar (Pick-Ups and/or Body-Model Added) (StanfordSondius Project → Staccato Systems, Inc. → ADI, ca. 1995)

• Example 1: (WAV) (MP3)• Example 2: (WAV) (MP3)• Example 3: (WAV) (MP3)• Virtual “wah-wah pedal”: (WAV) (MP3)

STK Mandolin

• STK Mandolin 1: (WAV) (MP3)• STK Mandolin 2: (WAV) (MP3)

Page 70: History and Practice of Digital Sound Synthesis

Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 70 / 84

More Recent Acoustic Guitar

• Bach Prelude in E Major: (WAV) (MP3)• soundexamplewavBach silenceLoure in E Major: (WAV) (MP3)

Virtual performance by Dr. Mikael Laurson, Sibelius Institute

Virtual guitar by Helsinki Univ. of Tech., Acoustics Lab3

3http://www.acoustics.hut.fi/

Page 71: History and Practice of Digital Sound Synthesis

Commuted Synthesis of Linearized Violin

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 71 / 84

e(n) x(n)s(n)Amplitude(n)Frequency(n)

x(n)Amplitude(n)Frequency(n)

a(n)

a(n) x(n)Output

Impulse-ResponseTrain

Amplitude(n)Frequency(n)

a)

b)

c)

Output

Output

String

Stringe(n)

String Resonator

Resonator

ImpulseTrain

ImpulseTrain

• Assumes ideal Helmholtz motion of string• Sound Examples (Stanford Sondius project, ca. 1995):

Bass: (WAV) (MP3) Cello: (WAV) (MP3) Viola 1: (WAV) (MP3) Viola 2: (WAV) (MP3)

Violin 1: (WAV) (MP3) Violin 2: (WAV) (MP3) Ensemble: (WAV) (MP3)

Page 72: History and Practice of Digital Sound Synthesis

Commuted Piano Synthesis (1995)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 72 / 84

Hammer-string interaction pulses (force):

5 10 15 20Time

0.1

0.2

0.3

0.4

0.5

Force

Page 73: History and Practice of Digital Sound Synthesis

Synthesis of Hammer-String Interaction Pulse

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 73 / 84

Time Time

LowpassFilter

Impulse Impulse Response

• Faster collisions correspond to narrower pulses(nonlinear filter )

• For a given velocity, filter is linear time-invariant

• Piano is “linearized” for each hammer velocity

Page 74: History and Practice of Digital Sound Synthesis

Multiple Hammer-String Interaction Pulses

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 74 / 84

Superimpose several individual pulses:

Impulse 1 LPF1

LPF2

LPF3

+Impulse 2

Impulse 3

StringInput

Time

Force

0

δ1

δ2

δ3

Page 75: History and Practice of Digital Sound Synthesis

Multiple Hammer-String Interaction Pulses

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 74 / 84

Superimpose several individual pulses:

Impulse 1 LPF1

LPF2

LPF3

+Impulse 2

Impulse 3

StringInput

Time

Force

0

δ1

δ2

δ3

As impulse amplitude grows (faster hammer strike), output pulsesbecome taller and thinner, showing less overlap.

Page 76: History and Practice of Digital Sound Synthesis

Complete Piano Model

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 75 / 84

Natural Ordering:

LPF1

LPF2

LPF3

+ StringOutput

Sound Board& Enclosure

TappedDelayLine

δ1

δ2

δ3

ImpulseGener-

ator

δ1

vc

Trigger

Commuted Ordering:

LPF1

LPF2

LPF3

+ String OutputTappedDelayLineTrigger

vc

Sound Board& Enclosure

Impulse Response

• Soundboard and enclosure are commuted• Only need a stored recording of their impulse response• An enormous digital filter is otherwise required

Page 77: History and Practice of Digital Sound Synthesis

Piano and Harpsichord Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 76 / 84

(Stanford Sondius Project, ca. 1995)

• Piano: (WAV) (MP3)• Harpsichord 1: (WAV) (MP3)• Harpsichord 2: (WAV) (MP3)

Page 78: History and Practice of Digital Sound Synthesis

More Recent Harpsichord Example

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 77 / 84

• Harpsichord Soundboard Hammer-Response: (WAV) (MP3)• Musical Commuted Harpsichord Example: (WAV) (MP3)

Reference:

“Sound Synthesis of the Harpsichord Using a ComputationallyEfficient Physical Model”,

by Vesa Valimaki, Henri Penttinen, Jonte Knif, Mikael Laurson,and Cumhur Erkut

JASP-2004

Google search: Harpsichord Sound Synthesis

Page 79: History and Practice of Digital Sound Synthesis

Physical Modeling in Audio Coding

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 78 / 84

Spectral modeling synthesis is finding application in audio coding.Can physical modeling synthesis be used as well?

• MPEG-4/SAOL already supports essentially all sound synthesismethods

• Ability to encode sounds automatically is limited

Codebook-Excited Linear Prediction (CELP) is a successfulsource-filter model (not quite physical)

There are many isolated examples of model-fitting torecorded data

Good model-based denoising results have been obtained Coder problem much harder when many sources are mixed

Page 80: History and Practice of Digital Sound Synthesis

Best Known Model-Based Audio Coders

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 79 / 84

A “cover band” can put together a very convincing facsimile ofpopular music performance

JOS high-school band “Bittersweet”

Page 81: History and Practice of Digital Sound Synthesis

Future Physical Modeling in Audio Coding?

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 80 / 84

A “Cover Band” Approach to Model-Based Audio Coding:

1. Recognize individual “audio streams” in a mix (CASA)(“I hear a trap set, electric bass, Fender Rhodes, and a strat”)

2. For each stream, calibrate its model heuristically(“Here is what I hear the bass part doing: ...”)

3. Fine-tune the synthetic mix to the real mix(joint “maximum likelihood estimation”)

Features of “Cover-Band Coding” (CBC):

• The “playing experience” of each “virtual performer” preventsartifacts — “musically unreasonable” parameters are madeunlikely (“Bayesian priors”)

• An incorrect instrument must “imitate” its assigned stream• New arrangements can be synthesized by deliberately choosing

a new ensemble!

Page 82: History and Practice of Digital Sound Synthesis

Summary

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 81 / 84

Page 83: History and Practice of Digital Sound Synthesis

Summary

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 82 / 84

We have reviewed a “CCRMA-centric slice” of the history of digitalsound synthesis (usually starting with results from Bell Labs):

• Wavetable (one period)• Subtractive• Additive• FM• Sampling• Spectral Modeling• Physical Modeling (more in tomorrow’s 4:30 PM masterclass)• Connections to audio coding

Page 84: History and Practice of Digital Sound Synthesis

Sound Acknowledgment

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Sound Acknowledgment

Julius Smith AES-2006 Heyser Lecture – 83 / 84

Page 85: History and Practice of Digital Sound Synthesis

Sound Acknowledgment

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Sound Acknowledgment

Julius Smith AES-2006 Heyser Lecture – 84 / 84

Thanks to Emu / Creative Labs for providing a superb-qualityexternal D/A converter for this talk (an E-Mu 0404/USB)