identifying “cover songs” with beat-synchronous chroma

Identifying Cover Songs - Ellis & Poliner 2007-04-20 - /161

1. Cover Songs2. Chroma Features3. Beat Tracking4. Matching Cover Songs

Identifying “Cover Songs”with Beat-Synchronous

Chroma FeaturesDan Ellis and Graham Poliner

Laboratory for Recognition and Organization of Speech and Audio Dept. Electrical Eng., Columbia Univ., NY USA

{dpwe,graham}@ee.columbia.edu http://labrosa.ee.columbia.edu/

http://labrosa.ee.columbia.edu

http://labrosa.ee.columbia.edu


Cover Songs• “Cover Songs” = reinterpretation of a piece

different instrumentation, characterno match with “timbral” features

• Need a different representation!beat-synchronous chroma features

2

Let It Be - The Beatles Let It Be - Nick Cave

time / sectime / sec

freq / k

Hz

Let It Be / Beatles / verse 1

2 4 6 8 100

1

2

3

4

freq / k

Hz

chro

ma

chro

ma

0

1

2

3

4Let It Be / Nick Cave / verse 1

2 4 6 8 10

Beat-sync chroma features

5 10 15 20 25 beats

Beat-sync chroma features

5 10 15 20 25 beats

A

C

D

F

G

A

C

D

F

G


Chroma Features• Chroma features map spectral energy

into one canonical octavei.e. 12 semitone bins

• Can resynthesize as “Shepard Tones”all octaves at once

3

Piano scale

2 4 6 8 10 100 200 300 400 500 600 700

2 4 6 8 10 time / sec

time / sec

freq

/ k

Hz

0

1

2

3

4

freq / k

Hz

0

1

2

3

4

time / frames

ch

rom

a

A

C

D

F

G

Piano chromatic scale

Shepard tone resynth

IF chroma


Calculating Chroma Features• Method 1: Map every STFT bin

blurs non-tonal energy

• Method 2: Map only STFT peaksstill blurry at low frequencies

• Method 3: Instantaneous Frequency /tescapes frequency resolution limit

4

50 100 150 200 fft bin 2 4 6 8 10 time / sec 50 100 150 200 250 300 time / frame

freq /

kH

z

0

1

2

3

4

chro

ma

A

C

D

F

G

chro

ma

A

C

D

F

G

50 100 150 200 fft bin 2 4 6 8 10 time / sec 50 100 150 200 250 300 time / frame

freq /

kH

z

0

1

2

3

4

chro

ma

A

C

D

F

G

chro

ma

A

C

D

F

G

2 4 6 8 10 time / sec 50 100 150 200 250 300 time / frame

freq /

kH

z

0

1

2

3

4

chro

ma

A

C

D

F

G

chro

ma

A

C

D

F

G

0 2000 4000

→( )


Beat Tracking (1)• Goal: One feature vector per ‘beat’ (tatum)

for tempo normalization, efficiency

• “Onset Strength Envelope”sumf(max(0, difft(log |X(t, f)|)))

• Autocorr. + window → global tempo estimate

5

10203040

freq

/ mel

0

0

5 10 15time / sec

0 100 200 300 400 500 600 700 800 900lag / 4 ms samples

1000

0

168.5 BPM


Beat Tracking (2)• Dynamic Programming finds beat times {ti}

optimizes i O(ti) + i W((ti+1 – ti – p)/)where O(t) is onset strength envelope (local score)W(t) is a log-Gaussian window (transition cost)p is the default beat period per measured tempoincrementally find best predecessor at every timebacktrace from largest final score to get beats

6

C*(t) = γ O(t) + (1–γ)max{W((τ – τp)/β)C*(τ)}τ

P(t) = argmax{W((τ – τp)/β)C*(τ)}τ

tτ

O(t)

C*(t)


Beat Tracking Results• DP will bridge gaps (non-causal)

there is always a best path ...

• 2nd place in MIREX 2006 Beat Trackingcompared to McKinney & Moelants human data

7

10

20

30

40

0 5 10 150

20

40

freq / B

ark

band

Subje

ct #

time / s

test 2 (Bragg) - McKinney + Moelants Subject data

182 184 186 188 190 192 time / sec

10

20

30

40

freq / B

ark

band

Alanis Morissette - All I Want - gap + beats


Beat-Synchronous Chroma Features• Beat + chroma features / 30ms frames→ average chroma within each beatcompact; sufficient?

! "# "! $# $! %# %!

$

&

'

(

"#

"$

"#

$#

%#

&#

$

&

'

(

"#

"$

# ! "# "!)*+,-.-/,0

)*+,-.-1,2)/

34,5-.-6,7

89/,)-/)4,9:);

0;48+2-1*9/

0;48+2-1*9/

#


Matching (1): Little Fragments• Cover versions may change song structure

multiple local matches at different alignments

• Match query and target as many small pieces?

9

how big are the pieces?

how do we combine individual scores?

do we have all day?

100 200 300 400 500 beats

100 200 300 400 500 beats

G

EDC

A

chro

ma b

ins

G

EDC

A

chro

ma b

ins

extract

cross-correlate

Query

Candidate


Matching (2): Global Correlation• Cross-correlate entire beat-chroma matrices

... at all possible transpositionsimplicit combination of match quality and duration

• One good matching fragment is sufficient...?

10

100 200 300 400 500 beats @281 BPM

-500 -400 -300 -200 -100 0 100 200 300 400 skew / beats

-5

0

+5

G

E

D

C

A

ch

rom

a b

ins

G

E

D

C

A

ch

rom

a b

ins

ske

w /

se

mito

ne

s

Elliott Smith - Between the Bars

Glen Phillips - Between the Bars

Cross-correlation


Filtered Cross-Correlation• Raw correlation not as important as precise

local matchlooking for large contrast at ±1 beat skewi.e. high-pass filter

11

-500 -400 -300 -200 -100 0 100 200 300 400

0

0.2

0.4

0.6

-500 -400 -300 -200 -100 0 100 200 300 400 skew / beats

skew / beats

-5

0

+5

ske

w /

se

mito

ne

s Cross-correlation

Cross-correlation @ skew = +2 semitones

raw

filtered


Results (1): Ellis 23 set• 23 pairs of cover songs from uspop2002 +...

one correct match per query

12

Test

Qu

ery

Ab Ad Al Am Be Be Bl Ca Ce Cl Co Co Da En Fa Go Go Gr Hu I_ I_ Le Ta

Abracadabra/sugar_ray

Addicted_To_Love/tina_turner

All_Along_The_Watchtower/dave_matthews_band

America/simon_and_garfunkel

Before_You_Accuse_Me/eric_clapton

Between_The_Bars/glen_phillips

Blue_Collar_Man/styx

Caroline_No/brian_wilson

Cecilia/simon_and_garfunkel

Claudette/roy_orbison

Cocaine/nazareth

Come_Together/beatles

Day_Tripper/cheap_trick

Enjoy_The_Silence/tori_amos

Faith/limp_bizkit

God_Only_Knows/brian_wilson

Gold_Dust_Woman/sheryl_crow

Grand_Illusion/styx

Hush/milli_vanilli

I_Can_t_Get_No_Satisfaction/rolling_stones

I_Love_You/faith_hill

Let_It_Be/nick_cave

Take_Me_To_The_River/annie_lennox

Cover Songs - dpwe23 - 12/23 correct


Results (2): MIREX 06• Cover song contest

30 songs x 11 versions of each (!)(data has not been disclosed)# true covers in top 108 systems compared (4 cover song + 4 similarity)

• Found 761/3300= 23% recallnext best: 11%guess: 3%

13

MIREX 06 Cover Song Results:# Covers retrieved per song per system

CS DE KL1 KL2 KWL KWT LR TP

cover song systems similarity systems

so

ng

-se

t (e

ach

ro

w is o

ne

qu

ery

so

ng

)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

2

0correct

matchesretrieved

4

6

8


Where are the matches?• Look inside global cross-correlation to find

matching fragments...xcorr = t f (C1(t, f)⋅C2(t, f)) - view along time

14

Let It Be / Beatles (beats 11-441)

50 100 150 200 250 300 350 400

0 50 100 150 200 250 300 350 400

-0.2

0

0.2

0.4

Let It Be / Nick Cave (beats 13-443)

50 100 150 200 250 300 350 400 time / beats

time / beats

time / beats

ch

rom

a

A

C

D

F

G

chro

ma

A

C

D

F

G


What are the mistakes?• False reject - missed true match

cover version is too different, beat tracking wrong ...

• False alarm - invalid match“Cocaine” (Clapton) vs. “Satisfaction” (Stones)

15

Eric Clapton - Cocaine - beats 17:1027

100 200 300 400 500 600 700 800 900 1000

Rolling Stones - Satisfaction - beats 1:1011

100 200 300 400 500 600 700 800 900 1000

0 100 200 300 400 500 600 700 800 900 1000-2

-1

0

1

2

chroma

A

C

D

F

G

chroma

A

C

D

F

G


Conclusions and Future Work• Beat-synchronous chroma features

are successful for matching cover songscaptures melody-harmony, not instruments

• Further uses:Beat-chroma fragments as musical building blockse.g. VQ over large body of musicfind recurrent motifsartist identification?

• Code available! Google “matlab cover song”

16

identifying “cover songs” with beat-synchronous chroma

Documents