digital sound as computer science
DESCRIPTION
Digital Sound as Computer Science. Jennifer Burg. CPATH Workshop Series: “Revitalizing Computer Science Education Through the Science of Digital Media” Workshop 2, July 28 and 29, 2008 Wake Forest University. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/1.jpg)
Digital Sound as Computer ScienceDigital Sound as Computer Science
CPATH Workshop Series: “Revitalizing Computer Science Education
Through the Science of Digital Media”Workshop 2, July 28 and 29, 2008
Wake Forest University
This work was funded by National Science Foundation CPATH grant CCF 0722261, Jennifer Burg PI, Conrad Gleber Co-PI
Jennifer Burg
![Page 2: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/2.jpg)
National Science Foundation CPATH National Science Foundation CPATH GrantGrant “Revitalizing Computer Science Education through the
Science of Digital Media” Jennifer Burg, PI, Wake Forest University Conrad Gleber, Co-PI, La Salle University Three years (Aug. 2007 – July 2010), seven workshops Each workshop
A special digital media topic One speaker from a related academic discipline One speaker from a related business or industry What would they like to see in a computer science major working for or
with them?
![Page 3: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/3.jpg)
Workshop SeriesWorkshop SeriesHost Location Topic DateConrad Gleber, Dir. of Digital Arts and Multimedia Design, Dept. of Math. and Computer Science
La Salle University, Philadelphia, PA
Algorithms, scripting, and programming for visual art
May 29 and 30, 2008
Jennifer Burg, Associate Professor of Computer Science
Wake Forest University, Winston-Salem, NC
Digital sound July 28 and 29, 2008
Gail Rubini, Prof. of Design, College of Visual Arts, Theatre, and Dance and Ken Balfauf, Dir. Program for Interdisciplinary Computing
Florida State University, Tallahassee, FL
Visualization April 2009
Michael Mateas, Assistant Prof. of Computer Science
University of California Santa Cruz
TBA Tentatively May 2009
Michael Niederman, Chair, Television Department
Columbia College, Chicago, IL
Digital media in television production
Tentatively August 2009
Cher Cornett, Dir. Digital Media Center
East Tennessee State University, Johnson City, TN
Game programming Tentatively May 2010
Gerald Gannod, Assoc. Prof. of Comp. Sci. and Systems Analysis
Miami University, Oxford, OH
TBA Tentatively August 2010
![Page 4: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/4.jpg)
Digital Sound Production WorkshopDigital Sound Production Workshop
June 2 to July 25, 2008 Students of music and computer science working
together Interdisciplinary collaborative projects Funded by National Science Foundation CCLI grant
“Linking Science, Art, and Practice through Digital Sound,” Jennifer Burg, PI; Jason Romney, Co-PI
![Page 5: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/5.jpg)
![Page 6: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/6.jpg)
![Page 7: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/7.jpg)
![Page 8: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/8.jpg)
![Page 9: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/9.jpg)
Goals in this workshop from the PIs’ Goals in this workshop from the PIs’ perspectiveperspective Consider in what ways digital sound is legitimately
part of the computer science curriculum Explore concepts, assignments, experiments, and
exercises that are interesting to students because they bring together science, art, and practice
Figure out where these can be plugged into the computer science curriculum
![Page 10: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/10.jpg)
Goals in this workshop from the Goals in this workshop from the participants’ perspectiveparticipants’ perspective Consider how these ideas shed light on your own
work Make contacts with colleagues who share your
interests Eat well
![Page 11: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/11.jpg)
What does the study of digital sound What does the study of digital sound entail?entail? Physics
sound waves, acoustics, resonance Engineering and digital signal processing (DSP)
Sound card, microphones, speakers, hardware sound processors, cables Mathematics
Trigonometry, logarithms, complex numbers, summations, integrals, transforms Music
Fundamental frequencies, harmonics, octaves Algorithms
Transforms, filters, compression
![Page 12: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/12.jpg)
What makes digital sound a suitable What makes digital sound a suitable topic within computer science?topic within computer science? It’s based on digital encoding and manipulation of
digital data. It’s “applied” computer science, which is what
makes it interesting.
![Page 13: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/13.jpg)
Topics in Digital Sound as Computer ScienceTopics in Digital Sound as Computer Science
Sound waves and acoustics Analog vs. digital representations, digital encoding,
sampling and quantization Decibels for measuring amplitude Frequencies related to pitch, complex waveforms Implications of sampling rate: the Nyquist theorem and
aliasing Implications of bit depth: quantization error, SQNR,
dynamic range, dithering, noise shaping, dynamic compression and expansion
![Page 14: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/14.jpg)
Topics in Digital Sound as Computer ScienceTopics in Digital Sound as Computer Science
MIDI compared to digital audio MIDI message formats and protocols MIDI samplers vs. synthesizers Sound wave synthesis
![Page 15: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/15.jpg)
Topics in Digital Sound as Computer ScienceTopics in Digital Sound as Computer Science
Hardware for sound processing Sound cards; ADCs and DACs; connection types;
cables; microphones, speakers and monitors; frequency response of microphones, speakers, and monitors
Software for sound processing Audition, Audacity, Sound Forge, Logic, Pro Tools,
Reason, Cakewalk Music Creator and Sonar, etc. MATLAB Chuck
![Page 16: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/16.jpg)
Topics in Digital Sound as Computer ScienceTopics in Digital Sound as Computer Science
Fourier analysis, frequency components, the Fourier transform, windowing functions
Filters (FIR and IIR), EQ, types of filters (shelf, low-pass, high-pass, bandpass, bandstop)
Special effects, e.g. reverb, autotuning, vocoding Data rate, data compression, psychoacoustical
models for compression, frequency masking
![Page 17: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/17.jpg)
As is true with most topics in computer As is true with most topics in computer science, you can approach digital sound at science, you can approach digital sound at different levels of abstractiondifferent levels of abstraction Mathematical/algorithmic – pencil and paper,
chalkboard, and calculator Low-level programming (e.g. C under Linux) Chuck MATLAB Audition, Audacity, Sound Forge, Logic, Pro Tools,
Reason, various plugins, etc. Sample editor vs. track editor Combining digital audio and MIDI
![Page 18: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/18.jpg)
Frequency Components of Frequency Components of Sound WavesSound Waves Generate notes C4, E4, and G4. Add the waves and look
at the result. In Audition In MATLAB
A single-frequency wave is a single pitch. Voices and instruments don’t produce single pitches.
They have harmonics. Sound in music and nature are complex waveforms with
frequency components.
![Page 19: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/19.jpg)
Notes, Octaves, and FrequenciesNotes, Octaves, and Frequencies Let f1 and f2 be the frequencies of two notes where the
second is an octave above the first. Then f2 = 2*f1. There are 12 notes in an octave. Find x such that f2 =2*f1 = ((((((((((((f1*x)*x)*x)*x)*x)*x)*x)*x)*x)*x)*x)*x) 2*f1= f1*x12
2 = x12
x = 1.0595 Thus if fa and fb are the frequencies of two consecutive
notes, then fb = 1.0595 * fa.
12 2
![Page 20: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/20.jpg)
Nyquist Theorem and AliasingNyquist Theorem and Aliasing The sampling rate must be more than twice the
frequency of the highest frequency component of the sound being sampled. Otherwise you can have aliasing. A frequency component comes out lower than it should
be. Demonstrated in Audition
![Page 21: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/21.jpg)
How is amplitude measured?How is amplitude measured?
depthbit the is where
2value sample
log20
:scale full Decibels
-1n10
n
dBFS
hearing of threshold the of pressure airmeasured being sound of pressure air
log20
:pressure sound on based Decibels
10dbSPL
In Audition
![Page 22: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/22.jpg)
The Effect of Bit Depth in QuantizationThe Effect of Bit Depth in Quantization Rounding to discrete quantization levels causes
error. The error is itself a wave. In MATLAB
Signal to quantization noise ratio (SQNR) and dynamic range are also measured in decibels.
depthbit the is where
)2(log20 10
nSQNR n
![Page 23: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/23.jpg)
Audio DitheringAudio Dithering Add a random amount between -1 and 1 (scaled to
the bit depth of the audio file) to each sample before quantizing.
There will be fewer consecutive samples that round to the same amount. Rounding to 0 is the worst thing, causing breaks.
Demonstration In Audition In MATLAB
![Page 24: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/24.jpg)
Noise ShapingNoise Shaping Raise error wave above the Nyquist frequency. Do this by making the error go up if it was previously
down and down if it was previously up. This raises the error wave’s frequency.
The amount added to a sample depends on the error in previous sample.
iii
ii
iiii
outFinFEinFoutF
cEDinFinF
__
____ 1
![Page 25: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/25.jpg)
Digital FiltersDigital Filters Infinite impulse response (IIR) vs. finite impulse
response (FIR) filters Filtering in the time domain by means of
convolution
Click to animate
![Page 26: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/26.jpg)
Digital FiltersDigital Filters Filtering in the frequency domain by means of the
Fourier transform
![Page 27: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/27.jpg)
Digital FiltersDigital Filters
![Page 28: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/28.jpg)
Creating FiltersCreating Filters
frequency response graph
![Page 29: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/29.jpg)
Creating FiltersCreating Filters
frequencyresponsegraphs
![Page 30: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/30.jpg)
Creating a Low-Pass FilterCreating a Low-Pass Filter
FrequencyResponse(frequencydomain)
ImpulseResponse(timeDomain)
![Page 31: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/31.jpg)
Creating a Low-Pass FilterCreating a Low-Pass Filter
![Page 32: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/32.jpg)
Creating a Low-Pass FilterCreating a Low-Pass Filter You can do it yourself in MATLAB:
Create the filter using the given function, sin(2fc)/n Read in an audio clip Since this is a filter in the time domain, convolve audio
clip with the filter Listen to the result Graph the frequencies of filtered clip against the
unfiltered clip. (Do this by taking the Fourier transform of each first.)
See the demonstration and worksheet for details.
![Page 33: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/33.jpg)
Creating FIR and IIR Filters with MATLAB’s Digital Creating FIR and IIR Filters with MATLAB’s Digital Signal Processing ToolboxSignal Processing Toolbox>> lowA = wavread('440.wav');>> highA = wavread('880.wav');>> highest = wavread('2000.wav');>> mixed = (lowA+highA+highest) / 3;>> [a,b] = butter(6,1000/4000);>> output = filter(a,b,mixed);>> ideal = (lowA+highA) / 2;>> wavplay(ideal,8000);>> wavplay(output,8000);>> hold on>> plot(ideal);>> plot(output,'red');>> axis([1 50 -1 1])Demonstration
![Page 34: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/34.jpg)
Filter Visualization Tool in MATLABFilter Visualization Tool in MATLAB MATLAB also has a Filter Visualization Tool that
lets you set zeros and poles for a filter and see the frequency, phase, and impulse responses.
Demonstration
![Page 35: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/35.jpg)
C Programs for Digital Sound and MIDIC Programs for Digital Sound and MIDI
#include <stdio.h>#include <string.h>#include <linux/soundcard.h>#include <unistd.h>#include <fcntl.h>/*CTRL-Break out of program*/int main(){ char* device00 = "/dev/midi" ; unsigned char data[3]; unsigned char byte1, byte2, byte3; int fd fd = open(device00, O_RDONLY, 0); if (fd < 0) { printf("Error: cannot open %s\n", device00); } else printf("Opened dev/midi00\n"); byte1 = byte2 = byte3 = -1;
while (1) { read(fd, data, sizeof(data)); if (!(data[0] == byte1 && data[1] == byte2 && data[2]
== byte3) { printf("%d ", data[0]); printf("%d ", data[1]); printf("%d\n", data[2]); byte1 = data[0]; byte2 = data[1]; byte3 = data[2]; } } return 0;}
Reading Messages from dev/midi
![Page 36: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/36.jpg)
#include <unistd.h>#include <fcntl.h>#include <sys/types.h>#include <sys/ioctl.h>#include <stdlib.h>#include <stdio.h>#include <linux/soundcard.h>#define LENGTH 3#define RATE 8000#define SIZE 8#define CHANNELS 1unsigned char buf[4*LENGTH*RATE*SIZE*CHANNELS/8];int main(){ int fd, arg, status; int i, j, k, begin, end, bufEnd; char temp; printf("Size of buffer is %d\n", sizeof(buf)); fd = open("/dev/dsp", O_RDWR, 0); if (fd < 0) { perror("Opening /dev/dsp failed\n"); exit(1); } arg = SIZE; status = ioctl(fd, SOUND_PCM_WRITE_BITS, &arg); if (status == -1) perror("Unable to set sample size\n");
Reading from and Writing to the Sound Card
![Page 37: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/37.jpg)
arg = CHANNELS; status = ioctl(fd, SOUND_PCM_WRITE_CHANNELS, &arg); if (status == -1) perror("Unable to set number of channels\n"); arg = RATE; status = ioctl(fd, SOUND_PCM_WRITE_RATE, &arg); if (status == -1) perror("Unable to set sampling rate\n");
status = read(fd, buf, 1); while (buf[0] >= 125 && buf[0] <= 131) { status = read(fd, buf, 1); } printf("%d ",buf[0]); printf("broke the silence\n");
Reading from and Writing to the Sound Card
![Page 38: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/38.jpg)
for (i = 0; i <= 3; i++) { //printf("Say something\n"); status = read(fd, buf+(24000*i), 24000); if (status != 24000) perror("Read wrong number of bytes\n"); begin = 24000*i; end = begin + 12000; bufEnd = begin + 24000 - 1; for (j = begin, k = 0; j < end; j++, k++) { temp = *(buf+j); *(buf+j) = *(buf + bufEnd - k); *(buf+j) = temp; } } status = ioctl(fd, SOUND_PCM_SYNC, 0); if (status == -1) perror("SOUND_PCM_SYNC failed\n"); printf("You said \n"); status = write(fd, buf, sizeof(buf)); if (status != sizeof(buf)) perror("Wrote wrong number of bytes\n");}
Reading from and Writing to the Sound Card
![Page 39: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/39.jpg)
Creating a Vocoder in MATLABCreating a Vocoder in MATLAB
From http://www.paia.com/ProdArticles/vocodwrk.htm
![Page 40: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/40.jpg)
Creating a Vocoder in MATLABCreating a Vocoder in MATLABfunction output = vocoder(input1, input2, s, window)
h = hanning(window)';input1 = input1';input2 = input2';q=(s-window);output = zeros(1,s);fftdata = zeros(1,s);input1fft = zeros(1,s);input2fft = zeros(1,s);for i=1:window/4:q b = i+window-1; input1partfft = fft(input1(i:b).*h); input2partfft = fft(input2(i:b).*h); input1fft(i:b) = input1fft(i:b) + abs(input1partfft); input2fft(i:b) = input2fft(i:b) + abs(input2partfft); mult = input1partfft.*input2partfft; fftdata(i:b) = fftdata(i:b)+ abs(fft(mult)); output(i:b) = output(i:b)+ifft(mult);endoutput = output/max(output);
Demonstration
![Page 41: Digital Sound as Computer Science](https://reader033.vdocuments.mx/reader033/viewer/2022051402/56816835550346895dddee23/html5/thumbnails/41.jpg)
Questions for this workshopQuestions for this workshop How do we relate the science to art and practice? How much science does the artist/practitioner
need? Where are the points where knowing the science
results in better work? How do we change the computer science
curriculum to retain the science but relate it more interestingly to art and practice?