1

Content-based Content-based Music Information Music Information

RetrievalRetrieval

Student: DENG Jie. Supervisor: Prof. LEUNG, Clement

Department of Computer ScienceHong Kong Baptist University

March, 2010

2

Outline Outline

What will be Covered: Introduction A Brief Review of Music MIR in the Real World and Challenges Current Content-based MIR Key

Techniques Evaluation of MIR Conclusion and Future Work

3

IntroductionIntroduction

Music Information Retrieval (MIR) is the interdisciplinary science of retrieving information from music.

– Mainly based on three-filed subjects: traditional information retrieval, musicology and digital audio.

Content-based MIR is the science of extracting features from musical content, such as melody, rhythm and tempo and so on to facilitate tasks such as analysis and music retrieval.

Aim:– To better understand “music” in the music work– To really search music by “music”

4

Music Channels

Chart Shows

Record store

Mate’s recommendation

Motivation - Music Motivation - Music DiscoveryDiscovery

+Gigs

Radio

5

Music IR Application Music IR Application Digital Music Libraries / Sound Archives

– Seeking for content-based access music libraries– Combined with metadata search of existing

catalogues Music Education

– Voice or instrumental teaching Music Related Legal and Copyright

– Is the creative content of this music work based on something for which others hold the rights?

Musicology– Is this piece of music work similar to any other

works?– Dose any part of this piece closely resemble any

part of any other works?– Is this piece of music work is based on others?

6

A Brief Review of MusicA Brief Review of Music

Music Concepts Three basic features of a musical sound

– Pitch– Intensity / Dynamics– Timbre / Tone color

There are many other terms describing music– Tempo– Tonality– Time Signature– Key Signature

7

A Brief Review of MusicA Brief Review of Music

Music Characteristics Music can be defined as the art of disposing and producing sounds and silences in time

– Has horizontal and vertical dimensions The main dimensions of music can be used for music retrieval (Reference Nicola Orio)

– Timbre : Quality of the produced sound– Orchestration : Sources of sound production– Acoustics : Quality of the recorded sound– Rhythm : Patterns of sound onsets– Melody : Sequence of notes– Harmony : Sequence of chords– Structure : Organization of the musical work

8

A Brief Review of MusicA Brief Review of Music Music Representation

– Visual (musical scores, manuscripts)– Aural (digital music)– Text– Hybrid (visual representation of an audio music file )

__________________________________________________________________________________

E---------------0---------3-3------3--1-1-------------3-1-1---------------- B---1-------------3-2-----2-2------2--3-3-----3-2---2---3-3------------1--- G---0--0--0h1-------0-----0-0------0--2-2---2---0-0-----2-2------------2--- D---2--2------------------------------0---0-------------0----0-1-2-3-3---3- A-3-------------------0-0-----0--0--------------0-------------------------- E-------------0------------------------------------------------------------

Common Music Notatio

n

Tablature

Example: Visual Representation

9

MIR in the Real World MIR in the Real World

There are mainly three-category USERS in MIR

“Professional”“Amateur” “Academic”

Just about anyone!

Librarians

Publishers

Producers

Performers

Composers

Lawyers ...

Vast numbers Very many

Musicologists

Educators

Significant numbers

10

MIR in the Real World MIR in the Real World Common Music Data and Format Audio recordings

– Sampled sound– Wave, MP3, AAC, etc.

Symbolic recordings– Abstract musical instructions, MusicXML– Scores, MIDI, Humdrum, etc.

11

MIR in the Real World MIR in the Real World Overview of some existing music search systems

Search by music related metadata: (artists, albums, tracks, music reviews, new release, etc.) Yahoo! Music and Allmusic are the examples of this search type

Search by music lyrics: Lyrics.com and SongLyrics.com

Music Media Management and Track Identification: Identify metadata for music tracks, for example Gracenote and MusicIP

Recommend similar music: by mining some music feature elements (melody, rhythm, tone color, etc) to recommend user some similar music

Recommend personalized music: by mining some users’ information to recommend them some their favorite music

12

Challenges in MIRChallenges in MIR Began in the 1950’s, still an emerging discipline Subjectivity and Versioning Many levels of music knowledge Lack of bibliographic control and data quality________________________________________________________________________

Signal Processing Machine Learning Human Computer Interaction

HearingRepresentation

UnderstandingAnalysis

ReactingInteraction

MIR Pipeline

13

A Simplified MIR MapA Simplified MIR Map

integration of audio visual, symbolic and textual data

This very schematic diagram highlights trends

Extracted or produced information

Actions

External data

14

Basic Steps of Content-based MIRBasic Steps of Content-based MIR

Representation of music contents– Features: melody, rhythms, etc.

Feature extraction from music data Feature indexing Query interface Matching query features against the

feature index

15

Representation of Music Content – Most music features used to represent

music are always melody.– Rhythm feature only consider the rhythm

omitting the melody.– Melody contour method uses three

characters to express the contour of melody.

Content-based MIRContent-based MIR

16

Content-based MIRContent-based MIR

Feature extraction from music data– There are two category algorithms: time

domain (Autocorrelation function, Average magnitude difference function and Simple inverse filter tracking) and frequency domain models (Spectrum and Cepstrum)

– Common extracted tools in the following:• Short-term Fourier Transform features (FFT)• Mel-Frequency Cepstral Coefficients (MFCC)• Daubechies Wavelet Coefficient Histogram (DWCH)

– Pitch is the main feature extracted in practice

17

Content-based MIRContent-based MIR

Feature Indexing – Index terms: play a similar role of words in

textual documents.– Sequence matching techniques: consider both

the query and the documents as sequences of symbols and model the possible difference between them.

– Geometric methods: cope with polyphonic scores and also exploit the properties of continuous distance measures.

– Based on the above methods, there are mainly three category music search on the melody feature.

• Melodic retrieval based on index terms (N-grams)• Melodic retrieval based on sequence matching • Melodic retrieval based on geometric methods

18

Content-based MIRContent-based MIR

Query Interface – Query by text (keywords: album, artist, track,

etc.)– Query by aural (singing or humming)

• Wave input (sing the whole or part of the songs)• Music notes segmentation• Thematic melodies are extracted, translated into

text representations of intervals, pith, and harmony

• Comparison procedure– Query by tapping

• Wave input by tapping• Compute the duration of each note• Similarity comparison

19

Content-based MIRContent-based MIR Matching Query Features against the

Feature Index – Approximate/Partial matching– Similarity measure (MFCC, GMM, KNN)– Precision: how many of the answers are in fact correct– Recall: how many of the correct answers are in fact retrieved– Relevance feedback

Vector space model– Documents and queries are presented by vectors– Each element in a vector is determined by an indexing

scheme (N-grams or others)– The value of each element is determined by a weight scheme– The similarity between document di and query qj :

20

Evaluation of MIREvaluation of MIR

The community has established an array of software tools to support this work – see http://music-ir.org/evaluation

In 2004, Audio Description Contest first attempted to build comparative benchmark of MIR algorithms.

Downie has already given us the foundations and future of the scientific evaluation of MIR systems.

Traditional information retrieval evaluation can also be adopted in MIR, for example precision and recall measures.

21

ConclusionConclusion Music is a complicated art form of information and requires special retrieval systems MIR technology is improving, but the real application is still lacking Basic music concepts and characteristics Basic steps and models of MIR Current Content-based MIR Key Techniques Scientific evaluation of MIR___________________________________________________________

Mining the semantic information in multimedia, especially in digital audio music, and then propose a comprehensive and adaptive method to automatically analysis and retrieve the high level semantic information of music, for example, emotion, mood, and style, etc.

Future WorkFuture Work

22

Related Related Researches and ProjectsResearches and Projects

ISMIR since 2000– International Symposium on Music Information

Retrieval WOCMAT since 2005

– Workshop On Computer Music and Audio Technology

Digital archive application– Data mining in digital music archive

Free music audio, sound processing tools and music-related visualization and mining tools

– http://www.music-ir.org/evaluation/tools.html Music IR evaluation since 2005

– http://www.music-ir.org/mirexwiki/index.php/Main_Page– Test collection: music documents, query sets, and

judgment– Major handle: copyright issue

23

ReferenceReference

• Michael S. LEW, Nicu Sebe. Content-Based Music Information Retrieval: Current Directions and Feature Challenges. Proceedings of the IEEE, April 2008

• Nicola Orio. Music Retrieval: A Tutorial and Review. Foundations and Trends in Information Retrieval, Volume1, Issue 1, Pages 1-96, 2006.

• J. T. Foote, "An Overview of Audio Information Retrieval." In ACM-Springer Multimedia Systems, vol. 7 no. 1, pp. 2-11, ACM Press/Springer-Verlag, January 1999

• Remco C. Veltkamp, Frans Wiering, Rainer Typke. Content Based Music Retrieval. In B. Furht (Ed.), Encyclopedia of Multimedia. Springer, 2006.

• Giovanna Neve, Nicola Orio: A Comparison of Melodic Segmentation Techniques for Music Information Retrieval. ECDL 2005: 49-56.

• Hwei-Jen Lin, Hung-Hsuan Wu. Efficient geometric measure of music similarity. Information Processing Letters, Volume 109, Issue2, Page 116-120, 2008.

• Iman S. H. Suyoto, Alexandra L. Uitdenbogerd, and Falk Scholer. Searching Musical Audio Using Symbolic Queries.IEEE. Transactions onAudio, Speech and Language Processing, 16(2):372–381, 2008.

• The Scientific Evaluation of Music Information. Retrieval Systems: Foundations and Future. Computer Music Journal, Computer Music Journal, 28:2, pp. 12–23, Summer 2004.

• Michael Fingerhut. Real music libraries in the virtual future: for an integrated view of music and music information. Digitale bibliotheken voor muziek, 2005.

24

Thank you!

Questions?