music visualizer with javafx, jmf + java sound · music visualizer with javafxtm, jmftm + javatm...

Music VisualizerWith JavaFXTM,JMFTM + JavaTM SoundLucas JordanEffectiveUIDenver - Rochester - Vancouver

2

IntroductionJavaFX Launch> Sun came to us

“You tell us what you should make”> Music Visualizer - Synethesiator

Fancy UI Complex computation Music is hip (right?) Perfect for JavaFX

What is a music Visualizer and is JavaFX a reasonable platform for this?

What is an Audio Visualizer> Digital Audio

Sound is air pressure on your eardrum The pressure is proportional to the movement of a speaker The movement is proportional to the voltage in a wire The voltage is proportional to a byte in your application A byte of sound

> Digital Graphics Take that byte of sound Display a graphic proportional to the byte

> Now you are seeing something proportional to what you are hearing!

Examples

5

How do we get JavaFX to do this?> Playing audio with java

javafx.scene.media.* java.applet.AudioClip sun.audio.* java.sound.sampled.* javax.media.* Quicktime?

But which give us access to the audio data as it is played?

6

How do we get JavaFX to do this?> JavaFX can’t do this on its own

JMF or Java Sound required> But it can when there is a J2SE stack

Desktop Web Probably the fancier phones soon

7

How do we get JavaFX to do this?> JMF - Java Media Framework

Kind of ancient Really flexible Poor codec support / we are cheap Insane deployment story

Originally designed as an extension to the JVM> Java Sound

Poor codec support JavaFX can play MP3s, but Java out of the box can’t? Find third party jars

Built into the JVM

8

Getting at the audio data (JMF)> JMF requires a lot of code to set up> Here are the highlights on setting it up

To simply play audio Player player = Manager.createPlayer(url); We can’t use a Player

Implement Effect Register your Effect with the Manager Create a Processor from the Manager

A Processor extends Player Plug-in your Effect

9

Getting at the audio data (JMF)> The Effect interface

Extends Codec Gets inserted after the audio is decoded by a codec Has lots of other methods, process is what is interesting to us

public int process(Buffer inputBuffer, Buffer outputBuffer){//Copy inputBuffer into outputBuffer//Look at inputBuffer, do something interesting return BUFFER_PROCESSED_OK;}

10

Getting at the audio data (Java Sound)> Steps to get at the byte of sound

Create an AudioInputStream of a known AudioFormat Create a SourceDataLine Create a Thread

Reads from AudioInputStream Does something interesting Write to SourceDataLine

> We want a known format because we don’t want to write a spectrum analyzer for each possible encoding, sample rate, etc

11

Getting at the audio data (Java Sound)> The code to set up the audio input and outputAudioInputStream originalInputStream = AudioSystem.getAudioInputStream(inputStream);

//Format of data read inAudioFormat originalFormat = originalInputStream.getFormat();

//Normalize the data to something we knowdecodedFormat = new AudioFormat(AudioFormat.Encoding.PCM_SIGNED,

originalFormat.getSampleRate(), … );

//Input stream of the normalized format.AudioInputStream decodedInputStream = AudioSystem.getAudioInputStream(decodedFormat,

originalInputStream);

SourceDataLine line = (SourceDataLine)AudioSystem.getLine(info);

line.open(decodedFormat);line.start();

12

Getting at the audio data (Java Sound)> Thread to play the audio

Runnable.run() dis is a decodedInputStream

byte[] data = new byte[4096]int nBytesRead;while (nBytesRead = dis.read(data, 0, data.length) != -1) {

//Do Something Interesting! line.write(data, 0, nBytesRead);}

13

Doing Something Interesting Considerations> Sample Rate

Audio processing thread updates too fast for JavaFX A Spectrum Analysis or other FFT code might want more data then just one

read-write loop We want to accumulate audio bytes in the audio processing thread Create another thread which produces a final value at a more reasonable

rate We want to accumulate audio bytes in the audio processing thread

> Interacting With JavaFX Presenting our processed value in a JavaFX way We want to bind to the value JavaFX wrapper class Observable/Observer

14

Java and JavaFX classes> Create a Java class SoundDetailspublic class SoundDetails extends Observable implements Runnable{

private byte[] accumulatedData; private float currentLoudness;

public void run() { while(true){ try { Thread.sleep(1000 / 33); currentLoudness = calculateLoudness(accumulatedData); setChanged(); notifyObservers(); } catch (InterruptedException ex) { //log or something } } }}

15

Java and JavaFX classes> Create a class SoundDetailsFX

public class SoundDetailsFX extends Observer{ public-init var soundDetails:SoundDetails; public-read var currentLoudness:Number; init{ soundDetails.addObserver(this); } override function update(observable: Observable, arg: Object) { currentLoudness = soundDetails.getCurrentLoudness(); }}

16

Java and JavaFX classes> Finally, doing something interesting

Where sdfx is a SoundDetailsFX object

var rect = Rectangle{ width: 10 height: bind sdfx.currentLoudness fill: Color.ANTIQUEWHITE }

17

Recap> Exposed bytes of sound as they are being played

JMF or Java Sound> Accumulated and processed those bytes

Processing Thread> Wrapped the result of that process in a JavaFX object

Observable/Observer> Used a JavaFX programming paradigm to display a graphic

bind

18

Resources> Example Code

http://lucasjordan.com/audiovisualizer> JMF

http://java.sun.com/javase/technologies/desktop/media/jmf/> Java Sound

http://java.sun.com/products/java-media/sound/> Tritonus: Open Source Java Sound

http://www.tritonus.org/> EffectiveUI

http://effectiveui.com

19

Lucas Jordanlucasjordan@gmail.comlucas.jordan@effectiveui.com

effectiveui.comlucasjordan.com