Music: Broken Symmetry, Geometry, and Complexity

Download Music: Broken Symmetry, Geometry, and Complexity

Post on 02-Jan-2017




0 download

Embed Size (px)


<ul><li><p>Music: Broken Symmetry,Geometry, and ComplexityGary W. Don, Karyn K. Muir, Gordon B. Volk, James S. Walker</p><p>The relation between mathematics andmusic has a long and rich history, in-cluding: Pythagorean harmonic theory,fundamentals and overtones, frequencyand pitch, and mathematical group the-</p><p>ory in musical scores [7, 47, 56, 15]. This articleis part of a special issue on the theme of math-ematics, creativity, and the arts. We shall exploresome of the ways that mathematics can aid increativity and understanding artistic expressionin the realm of the musical arts. In particular, wehope to provide some intriguing new insights onsuch questions as:</p><p> Does Louis Armstrongs voice sound like histrumpet?</p><p> What do Ludwig van Beethoven, Ben-ny Goodman, and Jimi Hendrix have incommon?</p><p> How does the brain fool us sometimeswhen listening to music? And how havecomposers used such illusions?</p><p> How can mathematics help us create newmusic?</p><p>Gary W. Don is professor of music at the Universityof Wisconsin-Eau Claire. His email address is</p><p>Karyn K. Muir is a mathematics major at the State Uni-versity of New York at Geneseo. Her email address</p><p>Gordon B. Volk is a mathematics major at the Universi-ty of Wisconsin-Eau Claire. His email address is</p><p>James S. Walker is professor of mathematics at the Uni-versity of Wisconsin-Eau Claire. His email address</p><p> Melody contains both pitch and rhythm. Isit possible to objectively describe their con-nection?</p><p> Is it possible to objectively describe the com-plexity of musical rhythm?</p><p>In discussing these and other questions, we shalloutline the mathematical methods we use andprovide some illustrative examples from a widevariety of music.</p><p>The paper is organized as follows. We first sum-marize the mathematical method of Gabor trans-forms (also known as short-time Fourier trans-forms, or spectrograms). This summary empha-sizes the use of a discrete Gabor frame to performthe analysis. The section that follows illustratesthe value of spectrograms in providing objec-tive descriptions of musical performance and thegeometric time-frequency structure of recordedmusical sound. Our examples cover a wide rangeof musical genres and interpretation styles, in-cluding: Pavarotti singing an aria by Puccini [17],the 1982 Atlanta Symphony Orchestra recordingof Coplands Appalachian Spring symphony [5],the 1950 Louis Armstrong recording of La Vie enRose [64], the 1970 rock music introduction toLayla by Duane Allman and Eric Clapton [63], the1968 Beatles song Blackbird [11], and the Re-naissance motet, Non vos relinquam orphanos,by William Byrd [8]. We then discuss signal syn-thesis using dual Gabor frames, and illustratehow this synthesis can be used for processingrecorded sound and creating new music. Then weturn to the method of continuous wavelet trans-forms and show how they can be used togetherwith spectrograms for two applications: (1) zoom-ing in on spectrograms to provide more detailedviews and (2) producing objective time-frequency</p><p>30 Notices of the AMS Volume 57, Number 1</p></li><li><p>portraits of melody and rhythm. The musical il-lustrations for these two applications are from a1983 Aldo Ciccolini performance of Erik SatiesGymnopdie I [81] and a 1961 Dave Brubeck jazzrecording Unsquare Dance [94]. We conclude thepaper with a quantitative, objective descriptionof the complexity of rhythmic styles, combiningideas from music and information theory.</p><p>Discrete Gabor Transforms: SignalAnalysisWe briefly review the widely employed method ofGabor transforms [53], also known as short-timeFourier transforms, or spectrograms, or sono-grams.The firstcomprehensive effort inemployingspectrograms in musical analysis was Robert Co-gans masterpiece, New Images of Musical Sound[27] a book thatstilldeservesclose study. A morerecent contribution is [62]. In [37, 38], Drfler de-scribes the fundamental mathematical aspects ofusing Gabor transforms for musical analysis. Othersources for theory and applications of short-timeFourier transforms include [3, 76, 19, 83, 65].Thereis also considerable mathematical background in[50, 51, 55], with musical applications in [40]. Us-ing sonograms or spectrograms for analyzing themusic of birdsong is described in [61, 80, 67]. Thetheory of Gabor transforms is discussed in com-plete detail in [50, 51, 55] from the standpoint offunction space theory. Our focus here, however,will be on its discrete aspects, as we are going tobe processing digital recordings.</p><p>The sound signals that we analyze are all dig-ital, hence discrete, so we assume that a soundsignal has the form {f (tk)}, for uniformly spacedvalues tk = kt in a finite interval [0, T]. A Gabortransform of f , with window functionw , is definedas follows. First, multiply {f (tk)} by a sequence ofshifted window functions {w(tk")}M"=0, produc-ing time-localized subsignals, {f (tk)w(tk")}M"=0.Uniformly spaced time values, {" = tj"}M"=0, areused for the shifts (j being a positive integergreater than 1). The windows {w(tk ")}M"=0 areall compactly supported and overlap each other;see Figure 1. The value of M is determined bythe minimum number of windows needed to cover[0, T], as illustrated in Figure 1(b). Second, becausew is compactly supported, we treat each subsignal{f (tk)w(tk ")} as a finite sequence and applyan FFT F to it. This yields the Gabor transform of{f (tk)}:(1) {F{f (tk)w(tk ")}}M"=0.We shall describe (1) more explicitly in a moment(see Remark 1 below). For now, note that becausethe values tk belong to the finite interval [0, T],we always extend our signal values beyond theintervals endpoints by appending zeros; hencethe full supports of all windows are included.</p><p>(a) (b) (c)</p><p>Figure 1. (a) Signal. (b) Succession of windowfunctions. (c) Signal multiplied by middlewindow in (b); an FFT can now be applied tothis windowed signal.</p><p>The Gabor transform that we employ uses aBlackman window defined by</p><p>w(t) =</p><p>0.42+ 0.5 cos(2t/) +0.08 cos(4t/) for |t| /2</p><p>0 for |t| &gt; /2for a positive parameter equaling the widthof the window where the FFT is performed. InFigure 1(b) we show a succession of these Black-man windows. Further background on why we useBlackman windows can be found in [20].</p><p>The efficacy of these Gabor transforms is shownby how well they produce time-frequency portraitsthat accord well with our auditory perception,which is described in the vast literature on Ga-bor transforms that we briefly summarized above.In this paper we shall provide many additionalexamples illustrating their efficacy.</p><p>Remark 1. To see how spectrograms display thefrequency content of recorded sound, it helps towrite the FFT F in (1) in a more explicit form.The FFT that we use is given, for even N, by thefollowing mapping of a sequence of real numbers{am}m=N/21m=N/2 :</p><p>(2) {am}F!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!</p><p>{A =</p><p>1N</p><p>N/21</p><p>m=N/2am e</p><p>i2m/N},</p><p>where is any integer. In applying F in (1), wemake use of the fact that each Blackman windoww(tk ") is centered on " = j"t and is 0 fortk outside of its support, which runs from tk =(j" N/2)t to tk = (j" + N/2)t and is 0 attk = (j"N/2)t . So, for a given windowing spec-ified by ", the FFT F in (2) is applied to the vector(am)</p><p>N/21m=N/2 defined by(</p><p>f (tk)w([k j"]t))j"+N/21k=j"N/2</p><p>.</p><p>In (2), the variable corresponds to frequenciesfor the discrete complex exponentials ei2m/N</p><p>used in defining the FFT F . For real-valued data,such as recorded sound, the FFT values A satisfythe symmetry condition A = A , where A isthe complex conjugate ofA . Hence, no significantinformation is gained with negative frequencies.Moreover, when the Gabor transform is displayed,the values of the Gabor transform are plotted as</p><p>January 2010 Notices of the AMS 31</p></li><li><p>squared magnitudes (we refer to such plots asspectrograms). There is perfect symmetry at and , so the negative frequency values are notdisplayed in the spectrograms.</p><p>Gabor FramesWhen we discuss audio synthesis, it will be im-portant to make use of the expression of Gabortransforms in terms of Gabor frames.An importantseminal paper in this field is [92]. A comprehensiveintroduction can be found in [48]. We introduceGabor frames here because they follow naturallyfrom combining (1) and (2). If you prefer to seemusical examples, then please skip ahead to thenext section and return here when we discusssignal synthesis.</p><p>Making use of the description of the vector (am)given at the end of Remark 1, we can express (1)as</p><p>(3)1N</p><p>k1</p><p>k=k0f (kt)w([k j"]t) ei2[kj"]/N.</p><p>In (1), the values k0 and k1 are the lower and upperlimits of k that are needed to extend the signalvalues beyond the endpoints of [0, T]. We havealso made use of the fact that w([k j"]t) = 0for k j" N/2 and for k j" +N/2.</p><p>We now define the functions G",(k) for =N/2, . . . ,N/2 1 and " = 0, . . . ,M as</p><p>(4) G",(k) =1Nw([k j"]t) ei2[kj"]/N</p><p>and write C", for the Gabor transform values thatare computed by performing the computation in(3):</p><p>(5) C", =k1</p><p>k=k0f (tk)G",(k).</p><p>We have suppressed the dependence on j since itis a fixed value, specifying the amount of shiftingfor each successive window (via " = tj"). It isset in advance and does not change during theanalysis procedure using {G",}.</p><p>The significance of Equation (5) is that each Ga-bor transform value C", is expressed as an innerproduct with a vector</p><p>(G",(k)</p><p>). Hence, the entire</p><p>arsenal of linear algebra can be brought to bearon the problems of analyzing and synthesizingwith Gabor transforms. For instance, the vectors{G",} are a discrete frame. The theory of framesis a well-established part of function space theory,beginning with the work of Duffin and Schaefferon nonharmonic Fourier series [45, 99] throughapplications in wavelet theory [33, 18, 58] as wellas Gabor analysis [50, 51, 55, 24].</p><p>To relate our discrete Gabor frame to this bodyof standard frame theory, and to provide an ele-mentary condition for a discrete Gabor frame, we</p><p>require that the windows satisfy</p><p>(6) A M</p><p>"=0w2(tk ") B</p><p>for two positive constants A and B (the frameconstants). The constants A and B ensure numeri-cal stability, including preventing overflow duringanalysis and synthesis. The inequalities in (6)obviously hold for our Blackman windows whenthey are overlapping as shown in Figure 1(b). Us-ing (4) through (6), along with the Cauchy-Schwarzinequality, we obtain (for K := k1 k0 + 1):</p><p>(7)M,N/21</p><p>"=0,=N/2|C", |2 (KB) f2,</p><p>where f is the standard Euclidean norm: f2 =k1k=k0 |f (tk)|2. When we consider Gabor transform</p><p>synthesis later in the paper, we shall also find that</p><p>(8) (A/K) f2 M,N/21</p><p>"=0,=N/2|C", |2.</p><p>So we have (using B1 := A/K and B2 := KB):</p><p>(9) B1f2 M,N/21</p><p>"=0,=N/2|C",|2 B2f2 ,</p><p>a discrete version of standard analysis operatorbounds in function space theory. By analysis oper-ator, we mean the Gabor transform G defined by{f (tk)}</p><p>G!!!!!!!!!!!!!!!!! {C",}.</p><p>Remark 2. (1) It is not necessary to use a singlefixed window w to perform Gabor analysis. Forexample, one could use windows w"(tk ") foreach ", provided A </p><p>M"=0w</p><p>2" (tk ") B is</p><p>satisfied. Doing so allows for more flexibility inhandling rapid events like drum strikes [100],[37, Chap. 3], [39]. (2) The time values {tk} donot need to be evenly spaced (this is called non-uniform sampling) [50, 51]. An application usingnonuniform sampling is described in Example 12.</p><p>Musical Examples of Gabor AnalysisWe now discuss a number of examples of usingspectrograms to analyze recorded music. Our goalis to show how spectrograms can be used to pro-vide another dimension, a quantitative dimension,to understanding the artistry of some of the greatperformances in music. For each of the displayedspectrograms, you can view a video of the spec-trogram being traced out as the music plays bygoing to this webpage:</p><p>(10)</p><p>Viewing these videos is a real aid to understand-ing how the spectrograms capture importantfeatures of the music. The website also providesan online bibliography and links to the software</p><p>32 Notices of the AMS Volume 57, Number 1</p><p></p></li><li><p>we used. Another site with advanced software fortime-frequency analysis is [69].</p><p>As we describe these examples, we shall brieflyindicate how they relate to the art of music and itscreation. While spectrograms are extremely use-ful for quantitative analysis of performanceandperformance itself is an act of artistic creationweshall also point out some ways in which spectro-grams can be of use in aiding the creative processof musical composition. When we finish discussingthese examples, we will summarize all of theseobservations on musical creativity.</p><p>Example 1 (Pavarottis vocals). Our first exampleis a spectrogram analysis of a 1990 recordingof the conclusion of Puccinis aria, Nessun Dor-ma, featuring a solo by Luciano Pavarotti [17].See Figure 2. This spectrogram shows the high-amplitude vibrato (oscillations of pitch) thatPavarotti achieves. Using the spectrogram, wecan measure quantitatively the amplitudes in thevibrato. This illustrates an interesting creativeapplication of spectrograms. Because they can bedisplayed in real time as the singer is performing,they can be used to help performers to analyzeand improve their vibrato, both in amplitude andsteadiness. One affordable package for real-timespectrogram display can be found at [1]. Noticealso that Pavarotti is able to alter the formantstructure (the selective amplification of differentfrequency bands) of the same word in the lyrics,which produces a clearly audible change in bright-ness (timbre) in the sound. Using spectrograms tostudy formants is usually the domain of linguis-tics [75]. But here we see that formantsthrougha real-time adaptation of selective frequency am-plification (selective resonance)play a role in themagnificent instrumentation that is possible withthe human voice.1</p><p>Example 2 (Contrasting choral and solo voicings).Our second example is another passage fromthe same recording of Nessun Dorma [17], con-taining a choral passage and the introduction ofPavarottis concluding solo. See Figure 3. We cansee in the spectrogram that there is a notablecontrast between the sinuous, blurred vibrato ofthe chorus in the beginning of the passage ver-sus the clearer vibrato of Pavarottis solo voice,which is centered on elongated, single pitches.The blurring of the vibrato of the chorus is dueto reverberation (persistent echoing). We candescribe this with the following model:</p><p>(11) f (tk) =J</p><p>j=0g([k jm]t)h(j),</p><p>1There is some speculation that the human vocal appa-ratus actually evolved first in order to sing, rather thanspeak [73].</p><p>Figure 2. Spectrogram from a recording ofNessun Dorma with Luciano Pavarotti.Pavarottis large-amplitude vibrato is clearlyvisible and measurable. The differingformants (selective amplification of differentfrequency bands) for different vocalizations ofthe lyrics (printed below the spectrogram) arealso clearly displayed. Pavarotti changes theformants for the first two vocalizations ofvincer, and this is clearly audible as a changeof brightnes...</p></li></ul>