joe berkovitz - dynamic audio synthesis on the flash flex platform
DESCRIPTION
code coverage measurement tools (Flexcover has gotten a lot of attention this year)visual editor frameworks (I'm just about to release a new open source project called Moccasin that distills a lot of Noteflight's editing infrastructure into a general framework)TRANSCRIPT
![Page 1: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/1.jpg)
Dynamic Audio Synthesis
on the Flash/Flex Platform
Joe Berkovitz
![Page 2: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/2.jpg)
Overview
• Bit o’ background
• Flash 9 sound capabilities
• The Flash 10 Sound API
• StandingWave audio synthesis library
• New avenues for Flash audio and music
![Page 3: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/3.jpg)
About me
![Page 4: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/4.jpg)
Samples ‘n’ frames ‘n’ acoustics
![Page 5: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/5.jpg)
Player 9 Sound API
• Sound objects can be loaded from URLs...
• ...or embedded as assets
• Load ‘em, play ‘em, change volume or stereo pan – that’s about it!
• Unpredictable latency, synchronization
![Page 6: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/6.jpg)
Player 9 Audio// loading a sound over the networkvar s:Sound = new Sound();s.addEventListener(Event.COMPLETE, handleComplete);s.load(new URLRequest(“TheBomb.mp3”);
function handleComplete(e:Event) { Sound(event.target).play();}
![Page 7: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/7.jpg)
Player 9 Audio// loading a sound from a library asset[Embed(source=”TheBomb.mp3”)]private static var MySoundAsset:Class;
function playSound():void{ var s:Sound = new MySoundAsset(); var sc:SoundChannel = s.play(); sc.soundTransform = new SoundTransform(0.5 /*volume*/); // set other transform properties... // examine playback position trace(sc.position);}
![Page 8: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/8.jpg)
What we want to do...
• Dynamically compute sounds from scratch
• Process samples on the fly to shift pitch, add effects, equalization...
• Play more than 32 sounds at the same time
• Compute continuous sound output
![Page 9: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/9.jpg)
FP10 to the rescue
• Synchronization and continuity: SAMPLES_CALLBACK event
• Access to codecs: Sound.extract()
• The Vector data type
• Adobe Pixel Bender runtime
• Alchemy
![Page 10: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/10.jpg)
Dynamic Audio Outputvar s:Sound = new Sound();s.addEventListener(SampleDataEvent.SAMPLE_DATA, handleSampleData);var sc:SoundChannel = s.play();
function handleSampleData(e:SampleDataEvent) { for (i = 0; i < /*number of frames*/ ; i++) { e.data.writeFloat(/* left channel */); e.data.writeFloat(/* right channel */); }}
![Page 11: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/11.jpg)
Codec Access[Embed(source=”TheBomb.mp3”)]private static var MySoundAsset:Class;
function getSoundSamples(numFrames:uint):void{ var s:Sound = new MySoundAsset(); var bytes:ByteArray = new ByteArray(); s.extract(bytes, numFrames, _position);
bytes.position = 0; for (var i:Number = 0; i < numFrames; i++) { ... = bytes.readFloat(); // Left channel ... = bytes.readFloat(); // Right channel }}
![Page 12: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/12.jpg)
Some things to know
• Between 2048 and 8192 samples per event
• The fewer samples you provide, the lower the latency (time before a sample is heard)
• Initial latency > steady-state latency
• More samples/callback = more stability
• Don’t take too long to generate samples!
![Page 13: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/13.jpg)
Basic Audio Demo
![Page 14: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/14.jpg)
Going beyond theFP10 sound API
• Goal: to work at a higher level than samples
• High level concepts: Sources, Filters
• Play back timed, overlapping sequences of sonic or musical events: Performances
• Create a modular, extensible system
![Page 15: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/15.jpg)
The StandingWave Project
• Open source AS3 library
• Outgrowth of Noteflight
• Requires Player 10
• Enables new class of audio applications
![Page 16: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/16.jpg)
Audio Processing Concepts
Sample
Descriptor: 44.1 kHz Stereo
left[0]
left[1]
left[2]
left[3]
right[0]
right[1]
right[2]
right[3]
... ...
![Page 17: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/17.jpg)
SampleSample
Audio Processing Concepts
AudioPlayerSource
raw samples,algorithms...
Samples
![Page 18: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/18.jpg)
Audio Processing Concepts
AudioPlayerFilterSource
transformations
SampleSampleSamplesSampleSampleSamples
![Page 19: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/19.jpg)
Audio Processing Concepts
Filter
AudioPlayer
Source
FilterSource
FilterSource
Perf
orm
ance
starts at 0.0 sec
starts at 0.2 sec
starts at 0.7 sec
(...continues...)
timed sequenceof sounds
![Page 20: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/20.jpg)
Audio Processing Concepts
Filter
AudioPlayer
Source
FilterSource
FilterSource
Perf
orm
ance
starts at 0.0 sec
starts at 0.2 sec
starts at 0.7 sec
...
AudioPerformer
SampleSampleSamples
SampleSampleSamples
![Page 21: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/21.jpg)
Audio Processing Concepts
Filter AudioPlayer
Source
FilterSource
FilterSource
Perf
orm
ance
starts at 0.0 sec
starts at 0.2 sec
starts at 0.7 sec
(...continues...)
FilterAudioPerformer
![Page 22: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/22.jpg)
IAudioSourcepublic interface IAudioSource{ function get descriptor():AudioDescriptor; function get frameCount():Number;
function get position():Number; function resetPosition():void;
function getSample(numFrames:Number):Sample;
function clone():IAudioSource;} implementations: Sample, SineSource, SoundSource...
![Page 23: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/23.jpg)
IAudioFilterpublic interface IAudioFilter extends IAudioSource{ function get source():IAudioSource; function set source(s:IAudioSource):void;} implementations: EnvelopeFilter, EchoFilter, BandpassFilter...
![Page 24: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/24.jpg)
IPerformancepublic interface IPerformance{ function getElementsInRange (start:Number, end:Number) :Vector.<PerformanceElement>;} implementations: ListPerformance...
![Page 25: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/25.jpg)
Noteflight: A Music Notation Editor
in Flash
![Page 26: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/26.jpg)
What’s missing?
• Direct microphone input
• More codec libraries
• Fully functional and supported audio processing in PixelBender/Alchemy
![Page 27: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/27.jpg)
Where to get the StandingWave library
http://code.google.com/p/standingwave
![Page 28: Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform](https://reader034.vdocuments.mx/reader034/viewer/2022051610/548349e2b479590a0d8b4a04/html5/thumbnails/28.jpg)
Places and Projects
http://joeberkovitz.com
http://www.hobnox.com
http://labs.adobe.com/wiki/index.php/Alchemy:Libraries
http://www.getmicrophone.com/
http://void.andre-michelle.com