Slide 1

1 The Representation, Indexing and Retrieval of Music Data at NTHU Arbee L.P. Chen National Tsing Hua University Taiwan, R.O.C. http://www.cs.nthu.edu.tw/~alpchen

Slide 2

2 Outline Content-based media data retrieval Music data retrieval Features of music data Feature indexing and matching Prototypes Reference

Slide 3

3 Content-based Media Data Retrieval Representation of media contents features Feature extraction from media data Feature indexing Query interface

Slide 4

4 Content-based Media Data Retrieval Matching query features against the feature index approximate/partial matching similarity measure precision: how many of the answers are in fact correct recall: how many of the correct answers are in fact retrieved relevance feedback

Slide 5

5 Music Data Retrieval: System Architecture

Slide 6

6 Features of Music Data

Slide 7

7 Static music information The intrinsic music characteristics of music objects Key, beat, and tempo E.g., the Beethoven Symphony No. 5, Op. 67, C minor, 4/4, Allegro con brio Acoustical features Loudness, pitch, duration, bandwidth and brightness Can be computed and represented as numerical values

Slide 8

8 Features of Music Data Thematic features Themes, melodies, rhythms, and chords Can be derived from the staff information of a music object Melody The melody of a song is the sequence of the pitches of all notes in the songs E.g., the melody of the theme of the Beethoven s Symphony No.5 is sol sol sol mi fa fa fa - re

Slide 9

9 Features of Music Data Rhythm The rhythm of a song is the sequence of the durations of all notes in the songs E.g., the rhythm of the theme of the Beethoven s Symphony No.5 is 1/2-1/2-1/2-2-1/2-1/2-1/2-4 Chord A chord consists of three (root, third, and fifth) or more notes which sound together in harmony

Slide 10

10 Features of Music Data Coding scheme: a music object a sequence of music segments music segment = (segment type, segment duration, segment pitch) four segment types: (type A), (type B), (type C), and (type D)

Slide 11

11 Features of Music Data For example, the sequence of music segments: (B,3,-3) (A,1,+1) (D,3,-3) (B,1,-2) (C,1,+2) (C,1,+2) (C,1,+1)

Slide 12

12 music segment = (type, duration, pitch)

Slide 13

13 Features of Music Data Repeating Pattern A sequence of notes appearing more than once in the music object Efficient content-based retrieval Semantics-rich representation Extracting repeating patterns Tree-based approach Matrix-based approach

Slide 14

14 Features of Music Data Experiment 1

Slide 15

15 Features of Music Data Dissimilarity of melody strings

Slide 16

16 Features of Music Data Dissimilarity of repeating patterns

Slide 17

17 Features of Music Data Experiment 2

Slide 18

18 Features of Music Data Validity of classes

Slide 19

19 Finding Repeating Patterns: Tree-based Approach Construct an RP-tree for RP s with lengths 2 n, n 0, 1,... S = ABCDEFGHABCDEFGHIJABC

Slide 20

20 Finding Repeating Patterns: Tree-based Approach Length 1 {A, 3, (1, 9, 19)} {B, 3, (2, 10, 20)} {C, 3, (3, 11, 21)} {D, 2, (4, 12)} {E, 2, (5, 13)} {F, 2, (6, 14)} {G, 2, (7, 15)} {H, 2, (8, 16)}

Slide 21

21 Finding Repeating Patterns: Tree-based Approach Length 2 {AB, 3, (1, 9, 19)} = {A, 3, (1, 9, 19)} 0 {B, 3, (2, 10, 20)} {BC, 3, (2, 10, 20)} = {B, 3, (2, 10, 20)} 0 {C, 3, (3, 11, 21)} {CD, 2, (3, 11)} = {C, 3, (3, 11, 21)} 0 {D, 2, (4, 12)}

Slide 23

23 Finding Repeating Patterns: Tree-based Approach

Slide 24

24 Finding Repeating Patterns: Tree-based Approach Prune trivial patterns of length 2 n, n = 0, 1, Let X be an RP of S, Y a substring of X, and Z a substring of Y If freq(X) = freq(Z), Y is trivial

Slide 25

25 Finding Repeating Patterns: Tree-based Approach Length 1 {ABCDEFGH, 2, (1, 9)} {ABCD, 2, (1, 9)} {BCDE, 2, (2, 10)} {CDEF, 2, (3, 11)} {DEFG, 2, (4, 12)} {EFGH, 2, (5, 13)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)} {CD, 2, (3, 11)} {DE, 2, (4, 12)} {EF, 2, (5, 13)} {FG, 2, (6, 14)} {GH, 2, (7, 15)}

Slide 27

27 Finding Repeating Patterns: Tree-based Approach Length 4 {ABCDEFGH, 2, (1, 9)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)}

Slide 28

28 Finding Repeating Patterns: Tree-based Approach Generate all patterns of lengths 2 n, n 0, 1,... {ABCDEFGH, 2, (1, 9)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)} {ABC, 3, (1, 9, 19)} order-1 string-join AB 1 BC = ABC