large-scale physical modeling synthesis

42
Large-scale Physical Modeling Synthesis Stefan Bilbao Acoustics and Fluid Dynamics Group / Music University of Edinburgh NUI Maynooth, 2008 1. Abstract sound synthesis and physical modeling 2. Physical modeling techniques 3. Direct simulation 4. Implementation Issues

Upload: bendek

Post on 25-Feb-2016

113 views

Category:

Documents


1 download

DESCRIPTION

Large-scale Physical Modeling Synthesis. NUI Maynooth, 2008. Stefan Bilbao Acoustics and Fluid Dynamics Group / Music University of Edinburgh. Abstract sound synthesis and physical modeling Physical modeling techniques Direct simulation Implementation Issues. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Large-scale Physical Modeling Synthesis

Large-scale Physical Modeling Synthesis

Stefan BilbaoAcoustics and Fluid Dynamics Group / MusicUniversity of Edinburgh

NUI Maynooth, 2008

1. Abstract sound synthesis and physical modeling2. Physical modeling techniques3. Direct simulation4. Implementation Issues

Page 2: Large-scale Physical Modeling Synthesis

Digital Sound Synthesis: Motivations

Various goals: Achieving complete parity with sound produced by

existing instruments… Creating new sounds and instruments…

The goal determines the particular methodology and choice of technique…and the computational complexity as well as implementation details!

Main groups of techniques: Sampling synthesis Abstract techniques Physical Modeling

Page 3: Large-scale Physical Modeling Synthesis

Abstract Digital Sound Synthesis

Origins…early computer hardware/software design, speech (Bell Labs, Stanford)

Basic operations: delay lines, FFTs, low-order filtersSome difficulties: Sounds produced are

often difficult to control… betray their origins, i.e., they sound synthetic. BUT: may be very efficient!

Amplitude Frequency

Sinusoid FM output

Carrier

ModulatorVariable rate read

Table of data

Wavetable Synthesis (1950s)

Oscillators/Additive Synthesis (1960s)

FM Synthesis(1970s)

Others:

SubtractiveWaveshapingGranular…

Page 4: Large-scale Physical Modeling Synthesis

Physical Modeling

Physical models: based of physical descriptions of musical

“objects” can be computationally demanding… potentially very realistic sound control parameters: few in number, and

perceptually meaningful digital waveguides, modal synthesis, finite

difference methods, etc.

Page 5: Large-scale Physical Modeling Synthesis

Linearity and Nonlinearity A nonlinear system is best defined as a system which is not

linear (!) A linear system, crudely speaking: a scaling in amplitude of the

excitation results in an identical scaling in amplitude of the response

Many interesting and useful corollaries… Many physical modeling techniques are based on this

simplifying assumption…

ResonatorExcitationGestural data (control rate)

Sound output (audio rate)

Strongly nonlinear Linear (to a first approximation)

Page 6: Large-scale Physical Modeling Synthesis

Digital Waveguides (J. O. Smith, CCRMA, Stanford, 1980s--present)

A delay-line interpretation of 1D wave motion:

Useful for: strings/acoustic tubes

Waves pass by one another without interaction

Extremely efficient…almost no arithmetic!

See (Smith, 2004) for much more on waveguides…

Rightward traveling wave

Leftwardtraveling

wave

Add waves at listening point for output

Page 7: Large-scale Physical Modeling Synthesis

Products Using Waveguide SynthesisProducts Using Waveguide Synthesis

Physical modeling synthesizers Yamaha VL-1 & VL-7, 1994 Korg Prophecy, 1995

Sound cards Creative Sound Blaster AWE64 Creative Sound Blaster Live

Technology patented by Stanford University and Yamaha

Yamaha VL-1

(Sound examples from: http://www-ccrma.stanford.edu/~jos/waveguide/Sound_Examples.html)

(This slide courtesy of Vesa Valimaki, Helsinki University of Technology, 2008.)

Page 8: Large-scale Physical Modeling Synthesis

Waveguide Stringed Instruments (Helsinki University of Technology, Department of Acoustics and Signal Processing)

Sound examples:

Full harpsichord synthesis (Valimaki et al., 2004)

Guitar modeling (Valimaki et al., 1996)

See http://www.acoustics.hut.fi/~vpv/ for many other sound examples/related publications

Page 9: Large-scale Physical Modeling Synthesis

Modal Synthesis (Adrien et al., IRCAM, 1980s--present)

Vibration is decomposed into contributions from various modes, which oscillate independently, at separate frequencies

Basis for Modalys/MOSAIC synthesis system (IRCAM)

Sound output

Page 10: Large-scale Physical Modeling Synthesis

Limitations Digital waveguides:

work well in 1D, but do not extend well to problems in higher dimensionsCannot handle nonlinearities:

Linear String Nonlinear String

Cannot extract efficient delay-line structures… BUT: when waveguides may be employed, they are far

more efficient than any other technique!

Page 11: Large-scale Physical Modeling Synthesis

Limitations Modal synthesis

Not computationally efficientIrregular geometries huge memory costs (storage of modes)Also cannot handle distributed nonlinearities:

Linear Plate Nonlinear (von Karman) Plate These methods are extremely useful, as first approximations…

Page 12: Large-scale Physical Modeling Synthesis

Observations

These methods can be efficient, but:They are really “physical interpretations” of

abstract methods:Wavetable synthesis waveguides Additive synthesis modal synthesis

Can deal with some simple physical models this way, but not many.

Page 13: Large-scale Physical Modeling Synthesis

Physical Modelling Synthesis: Time-domain Methods

System of equations

Numerical method

(recursion)

Musical instrument

Output waveform

Finite Difference Methods

Finite Element Methods

Spectral/PseudospectralMethods

Methods are completely general— no assumptions about behaviour Vast mainstream literature, 1920s to present.

Page 14: Large-scale Physical Modeling Synthesis

FD schemes as recursions

seconds/1 sf

seconds/1 sf

All time domain methods operate as recursions over values on a grid Recursion updated at a given sample rate fs Typical audio sample rates:

32000 Hz 44100 Hz 48000 Hz 96000 Hz

Page 15: Large-scale Physical Modeling Synthesis

FD schemes as recursions

Solution evolves over time

Output waveform is read from a point on the grid

Entire state of object is computed at every clock tick

Page 16: Large-scale Physical Modeling Synthesis

Sound Examples: Nonlinear Plate

Under struck conditions, a wide variety of possible timbres:

Under driven conditions, very much unlike plate reverberation…

Nonlinear plate vibration the basis for many percussion instruments: cymbals, gongs, tamtams

Audible nonlinear phenomena: subharmonic generation, buildup of high-frequency energy, pitch glides

Page 17: Large-scale Physical Modeling Synthesis

FD Cymbal ModelingCymbals: an interesting synthesis problem:• Simple PDE description• Regular geometry• Highly nonlinear

Time-domain methods are a very good match…

A great example of a system which is highly nonlinear…linear models do not do justice to the sound!

Linear model Nonlinear model

Difference methods really the only viable option here…

Page 18: Large-scale Physical Modeling Synthesis

FPGA percussion instrument (R. Woods/K. Chuchasz, Sonic Arts Research Centre/ECIT, Queen’s University Belfast)

Page 19: Large-scale Physical Modeling Synthesis

FD Wind Instruments

Wind instrument models:Also very easily approached using FD methods…

Clarinet

Saxophone

Squeaks!

BUT: for simple tube profiles (cylindrical, conical), digital waveguides are far more efficient!

Page 20: Large-scale Physical Modeling Synthesis

FD Modularized Synthesis: Coupled Strings/Plates/Preparation Elements

String/soundboard connection

Prepared plate Bowed plate

Spring networks

A complex nonlinear modular interconnection of plates, strings, and lumped elements…

Page 21: Large-scale Physical Modeling Synthesis

FD Plate Reverberation Physical modeling…but not for synthesis! Drive a physical model with an input

waveform In the linear case: classic plate reverberation

(moving input, pickups)

Page 22: Large-scale Physical Modeling Synthesis

RenderAIR: FD Room Acoustics SimulationRenderAIR: FD Room Acoustics Simulation (D. Murphy, S. Shelley, M. Beeson, A. Moore, A. Southern, University of York, UK)

Audio bandwidth 3D models = High Memory/High Computation load. Possible Solutions? Uses Collada (Google Earth/Sketchup) format geometry files. “Grows” a mesh to fit the user defined geometry. Mesh topology/FDTD-Scheme plug-ins for speed of development. Contact and related publications: Damian Murphy, University of York, UK

[email protected] http://www-users.york.ac.uk/~dtm3/research.html

Page 23: Large-scale Physical Modeling Synthesis

Musical Example

Untitled (2008)---Gordon Delap

stereo, realized in Matlab(!)

Page 24: Large-scale Physical Modeling Synthesis

A general family of systems in musical acoustics A useful (but oversimplified) model problem:

Parameters: d: dimension (1,2, or 3) p: stiffness (1 or 2) c: ‘speed’ V : d-dim. ‘volume’

0222

2

uctu p

ddVx

p\d 1 2 3 1

2

strings acoustic tubes

membranes room acoustics

bars plates

Page 25: Large-scale Physical Modeling Synthesis

Computational Cost: A Rule of Thumb Result: bounds on both memory requirements, and the operation count:

pds

cfV

/

pds

s cfVf

/

# memory

locations# arithmetic

ops/sec

Some points to note here: As c decreases, or as V becomes larger, the “pitch” decreases and computation increases: low-pitched sounds cost more… Complexity increases with dimension (strongly!) Complexity decreases with stiffness(!)The bound on memory is fundamental, regardless of the method employed…

Page 26: Large-scale Physical Modeling Synthesis

Computational Costs

106 107 108 109 1010 1011 1012 1013 1014 1015 1016 1017

Arithmetic operations/second, at 48 kHz:

Large acoustic spaces

Plate reverberation

Wind instruments

Single string Bass drum

Small-mediumacoustic spaces

Great variation in costs…

Full piano

Approx. limit of present realtime performance on commercially available desktop machines

Page 27: Large-scale Physical Modeling Synthesis

Difficulties: Numerical Stability

For nonlinear systems, even in isolation, stability is a real problem.

Solution can become unstable very unpredictably…

Problems for composers, and, especially: live performers!

Even trickier in fixed-point arithmetic.

Page 28: Large-scale Physical Modeling Synthesis

Parallelizability: Modal Synthesis

Each mode evolves independently of the others:

Result: independent computation for each mode (zero connectivity) Obviously an excellent property for hardware realizations.

Each mode behaves as a “two-pole” filter:

Page 29: Large-scale Physical Modeling Synthesis

Parallelizability: Explicit finite difference methods

A useful type of scheme: explicit

Each unknown value calculated directly from previously computed values at neighboring nodes

“local” connectivity…

Useful for linear problems…

Unknown

Known

Update point

sf/1

sf/1

Page 30: Large-scale Physical Modeling Synthesis

Parallelizability: Implicit finite difference methods

Other schemes are implicit…

Unknowns coupled to one another (locally)

Useful for nonlinear problems…(stability!)

Unknown

Known

Update group of points

sf/1

sf/1

Page 31: Large-scale Physical Modeling Synthesis

Parallelizability: Sparse matrix representations Can always rewrite explicit updates as

(sparse) matrix multiplications:

= nx1nx

State transition matrix Last stateNext state

Sparse, often structured (banded, near Toeplitz)

Size N by N, where N is the number of FD grid locations. NNZ entries: O(N)

Page 32: Large-scale Physical Modeling Synthesis

Parallelizability: Sparse matrix representations Can sometimes write implicit updates as (sparse) linear

system solutions:

= nx1nx

Last stateNext state

Many fast methods available: Iterative… Thomas-type for banded matrices FFT-based for near-Toeplitz

Different implications regarding parallelizability!

Page 33: Large-scale Physical Modeling Synthesis

I/O: Modal Methods Modal representations are non-local: input/output at a given location requires

reading/writing to all modes:

Excitation point

Readout point

Location-dependent expansion coefficients

Input

Output

Location-dependent expansion coefficients

Expansion coefficients calculated offline! Must be recalculated for each separate I/O location Multiple outputs: need structures running in parallel…

Page 34: Large-scale Physical Modeling Synthesis

I/O: Finite Difference Methods Finite difference schemes are essentially local: Input/output is very straightforward: insert/read values directly

from computed grid… O(1) ops/time step

Connect excitation element/insert sample

Read value

Page 35: Large-scale Physical Modeling Synthesis

I/O: Finite Difference Methods Multichannel I/O is very simple… No more costly than single channel!

Connect excitation element/insert sample

Read values

Read/write over trajectory

Moving I/O also rather simple Interpolation (local) required…

Page 36: Large-scale Physical Modeling Synthesis

Boundary conditions

Updating over interior is straightforward…

Need spcialized updates at boundary locations…

…as well as at coordinate boundaries

Page 37: Large-scale Physical Modeling Synthesis

Modularized synthesis

Idea: allow instrument designer (user) to connect together components at will:

Basic object types: Strings Bars Plates Membranes Acoustic tubes Various excitation

mechanisms Need to supply connection

details (locations, etc.)

Object 1

Object 2

Object 3

Page 38: Large-scale Physical Modeling Synthesis

Challenges: Modular Stability

Easy enough to design stable simulations for synthesis for isolated objects…

Even for rudimentary systems, problems arise upon interconnection:

Stable Connection Unstable Connection

Mass/spring system

Ideal String

For more complex systems, instability can become very unpredictable…

Page 39: Large-scale Physical Modeling Synthesis

Energy based Modular Stability Key property underlying all physical models is energy. For a system of lossless interconnected objects, each has an associated stored energy H:

Each energy term is non-negative, and a function only of local state variables---can bound solution size:

Numerical methods: assure same property in recursion in discrete time, i.e.,

constant0 HH p

H1

H2H3

H4

p

pHHHdtd 0

p

np

nn HHH constant Need to ensure positivity in discrete time…

Page 40: Large-scale Physical Modeling Synthesis

Energy: Coupled Strings/Soundboard/Lumped Elements System

Soundboard EnergyEnergy of Prepared ElementsEnergy of StringsTotal Energy

Can develop modular numerical methods which are exactly numerically conservative…

A guarantee of stability… A useful debugging feature! Returning to the plate/string/prepared elements system,

time

Page 41: Large-scale Physical Modeling Synthesis

Concluding remarks Digital waveguides:

Ideal for 1D linear uniform problems: ideal strings, acoustic tubes

Extreme efficiency advantage… Modal synthesis:

Apply mainly to linear problems Zero connectivity I/O difficulties (non-local excitation/readout) Possibly heavy precomputation Good for static (i.e., non-modular) configurations

FD schemes Apply generally to nonlinear problems Local connectivity Stability difficulties I/O greatly simplified Minimal precomputation Flexible modular environments possible

Page 42: Large-scale Physical Modeling Synthesis

References General Digital Sound Synthesis:

C. Roads, The Computer Music Tutorial, MIT Press, Cambridge, Massachusetts,1996. R. Moore, Elements of Computer Music, Prentice Hall, Englewood Cliffs, New Jersey,

1990. C. Dodge and T. Jerse, Computer Music: Synthesis, Composition and Performance,

Schirmer Books, New York, New York, 1985. Physical Modeling (general)

V. Valimaki and J. Pakarinen and C. Erkut and M. Karjalainen, Discrete time Modeling of Musical Instruments, Reports on Progress in Physics, 69, 1—78, 2005.

Special Issue on Digital Sound Synthesis, IEEE Signal Processing Magazine, 24(2), 2007. Digital Waveguides

J. O. Smith III, Physical Audio Signal Procesing, draft version, Stanford, CA, 2004. Available online at http://ccrma.stanford.edu/~jos/pasp04/

V. Välimäki, J. Huopaniemi, M. Karjalainen, and Z. Jánosy, “Physical modeling of plucked string instruments with application to real-time sound synthesis,” J. Audio Eng. Soc., vol. 44, no. 5, pp. 331–353, May 1996.

V. Välimäki, H. Penttinen, J. Knif, M. Laurson, and C. Erkut, “Sound synthesis of the harpsichord using a computationally efficient physical model,” EURASIP Journal on Applied Signal Processing, vol. 2004, no. 7, pp. 934–948, June 2004.

Modal Synthesis D. Morrison and J.-M. Adrien, MOSAIC: A Framework for Modal Synthesis, Computer

Music Journal, 17(1):45—56, 1993. Finite Difference Methods

S. Bilbao, Numerical Sound Synthesis, John Wiley and Sons, Chichester, UK, 2009 (under contract).