MUSIC INFORMATION

RETRIEVAL

SpeakerAssociate Professor Ning-Han Liu

What’s MIR Music information retrieval (MIR) is

the interdisciplinary science of retrieving information from music.

MIR is a small but growing field of research with many real-world applications.

Applications of MIR Recommender systems Track separation and instrument

recognition Automatic score creation Automatic categorization Music generation

Methods used in MIR-Data Source Music Content

Symbolic Data Formats○ Scores○ MIDI

Digital Audio Formats○ WAV○ Mp3

MetadataSong nameSingerPerformertags

Methods used in MIR-Feature Representation Mel-Frequency Cepstral Coefficient

(MFCC)is a measure of the timbre of a piece of

music. Chords Harmonics Melody Main pitch Beats per minute

Methods used in MIR-Statistics and Machine Learning Music analysis and knowledge

representation Classification, clustering and modeling

Famous Application iTunes: is a media player computer program, used for playing,

downloading, saving, and organizing digital music and video files on desktop or laptop personal computers.

Famous Application KKBOX

Famous Application Nike+iPod

Previous research issue I Query music efficiently

Music representation○ ‘U’,’D’,’S’○ Rhythm○ Chords

Index structure○ Link list○ Suffix tree

Query process○ Exact matching○ Approximate matching

Previous research issue I String Matching

Exact String Matching○ E.g. sol-mi-mi-fa-re-re-do-re-mi-fa-sol-sol-sol○ Query: do-re-mi○ To find whether data contain the query melody

stringAlgorithms: KMP, Boyer-MooreProblem

○ Cannot expect users to precisely specify music query

○ Need to retrieve all entire melody string

Previous research issue I String Matching

Approximate String Matching○ Edit distance is often used as similarity

measure○ E.g. sol-mi-mi-fa-re-re-do-re-mi-fa-sol-sol-sol○ Query: do-mi matching do-re-mi with edit

distance=1

Previous research issue I String Matching Algorithm Using Index

Arbee L.P. Chen (ICMCS 1999)

Previous research issue I Approximate Matching

Previous research issue I Query by Music Segments

Arbee L.P. Chen (ICME 2000)

Previous research issue I Query by Music Segments

Index: Suffix Tree

Previous research issue II Music Recommendation

Content based filtering (CBF)○ predicts user preferred data items by matching the

representations of the data items relevant to the userCollaborative filtering (CF)

○ uses the correlations between users on the basis of their ratings to predict items for users

Mixed method○ Combine CBF and CF

Previous research issue II A music recommendation system based on music data

grouping and user interests Arbee (ICME 2000)

Previous research issue II

Novel research topic I Playlist Generation

An Intelligent Music Playlist Generator based on the Time Parameter with Artificial Neural Networks (Liu, 2010)

time

time

Hip Hop Rock&Roll Blues

Slow tempo

Fast tempo

Medium tempo

Slow tempo

07:00 08:00 12:00

07:00 07:30 09:00 12:30

User A

User B

N.H. Liu, S.J. Hsieh, C.F. Tsai, "An Intelligent Music Playlist Generator based on the Time Parameter with Artificial Neural Networks," Expert systems with applications. Volume 37, Issue 4, April (2010), Pages 2815-2825 . (SCI, impact factor=2.9, EI)

The structure of the mixed artificial neural networks

Novel research topic I

Input Layer

Hidden Layers

Output Layer

Long term ANN

Short term ANN

Fusion functionOutput

Music Item Mx

Feature 1Feature 2

.

.

.Feature n

OutputANN1

OutputANN2Time parameter t

Novel research topic II JoMP: A Mobile Music Player Agent for Joggers

based on User Interest and Pace (Liu, 2009)

Music Player

Pace DetectionModule

Smart Phone

Music Selection Module

User Feedback Module

User ProfileMusic Filtering Module I

(Depends on user interesting)

Music Database

Music Filtering Module II(Depends on user status)

Music Pool for Running

JoMP Server

Web Based Music Player with Feedback Mechanism

Music Web

Server

User Client

Pace Prediction

Module

N.H. Liu, H.Y. Kung, "JoMP: A Mobile Music Player Agent for Joggers based on User Interest and Pace," IEEE Transactions on Consumer Electronics, Volume 55, No. 4, Nov, (2009 (SCI, EI)

Novel research topic II System prototype

Novel research topic III Intelligent Music Player for Bike Sport Using

EEG & GPS Sensors Brainwave earphone (NeuroSky™ Minset) and wearing

method.

EEG Signal rhythm Frequencyα waves 8~13 Hzβ waves 14~30 Hzγ waves > 21 Hz δ wave 0.5~3 Hzθ waves 4~7 Hz

N.H. Liu, H.M. Hsu, H.C. Chu, and S.H. Hsu "Intelligent Music Player for Bike Sport Using Electroencephalogram and Global Positioning System Sensors" Sensor Letters, Volume 11, Number 5, pp. 772-780 (2013) (SCI).

Novel research topic III The system is based on the Fuzzy Inference System (FIS),

combined with EEG and GPS sensors to measure the sport data of cyclists.

EEG sensor

Tablet PCwith GPS

Novel research topic IV Intelligent car audio system in cloud

N.H. Liu "Design of an Intelligent Car Radio and Music Player System," accepted by Multimedia Tools and Applications. (2013).(SCI, IF=1.014) DOI: 10.1007/s11042-013-1467-z

Novel research topic V Car audio system with doze prevention

Novel research topic V Car audio system with doze prevention

N.H. Liu, C.Y. Chiang, H.M. Hsu, "Improving Driver Alertness through Music Selection Using a Mobile EEG to Detect Brainwaves"  Sensors, 13(7):8199-8221 (2013)(SCI, IF=1.953).

Novel research topic V Car audio system with doze prevention

Novel research topic VI Distance estimation methods

Methods to calculate a personalized distance measure between different pieces of music based on user preferences.

The questions ask the user to rate the similarity between songs selected by the system.

N.H. Liu "Comparison of Content-based Music Recommendation Using Different Distance Estimation Methods," Applied Intelligence, Volume 38, issue 2, pp. 160-174, Mar. (2013).(SCI, IF=1.853)

Novel research topic VI Weighted squared Euclidean distance function

generated from maximum likelihood estimation2

1)()Q,(X

d

nnnn qxwD

pi

i

Mm

rip QWmPQWMP ),|(),|(

Q),|( max arg' WMPW pW

d

pppp w

WMPwWMP

wWMP

WWMP)Q,|(

,...,)Q,|(

,)Q,|(

}),Q,|({21

pi

pi

Mmnnii

Mmi

nn qmr

r

ww 2,

'

)(

troundin ofdimension th -n of valueweighted-re theis where, ',

1

',

WwT

ww tn

T

ttn

n

Novel research topic VI Weighted squared Euclidean distance function

generated from genetic algorithm

QSQ MSix

Wi qix

QWDifW ),|(S min arg'

bit0 bit1 bit15 bit0 bit1 bit15 bit0 bit1 bit15

w1 w2 wd

Chromosome W

Define a fitness function to evaluate the chromosome performance

QSQ MSix

i qix

QW

Wf

1),|Dif(S

1)(

Novel research topic VI Distance function generated by genetic programming

QSq MSixx

Wi qix

QSFSDifF ),(|( min arg'

The fitness function is defined as follows:

QSQ MSixx

i qix

QSF

Wf

1),(|Dif(S

1)(

Novel research topic VII Music Selection Interface for Car Audio System

In the architecture, users are able to select a genre of music or a playlist from through a 2D interface. Self-organizing map, depending on a personalized distance function and music contents, is utilized to map music tracks to the interface.

Novel research topic VII Prototype in car

N.H.Liu, "Music Selection Interface for Car Audio System Using SOM with Personal Distance Function,"  EURASIP Journal on Audio, Speech, and Music Processing,  2013:20. (2013) (SCI)

Novel research topic VIII Query by Singing/Humming

When a user cannot remember the title of a song, or its related details, the most direct and convenient method to search for the song is by humming a section of it.

The background of the user often influences the genres of the songs being searched.

We use the information from a user’s search history, as well as the properties of genres common to users with similar backgrounds, to estimate the genre or style the current user may be interested in based on a probability calculation.

Novel research topic VIII System flow chart

Humming Query

Melody Representation

Query Processing

Users Query History

Melody Processing

Melody Representation

MatchProcessing

Rank List

New Rank List

Music Database Music

Profile

Re-ranking by Hybrid

Recommendation Method

User

Phase I Phase II

Novel research topic VIII The Singing/Humming signals process

NS0

NS1 NS2

NS3 NS4 NS5 NS6 NS7

NS8

NS9 NS10 NS11 NS12

A2# (116.54Hz)

F2# (92.499Hz)

F2# (92.499Hz)NS0

NS1

NS2

Twinkle, Twinkle, Little Star

Pitch String: 67 64 64 65 62 62 60 62 64 65 67 67 67

Contour String: -3 0 +1 -3 0 -2 +2 +2 +1 +2 0 0

Novel research topic VIII Ranking by User’s Preference

Novel research topic VII Ranking through Similar Users’ Records

)()()|(

)|(APCPCAP

ACP iii

naïve Bayesian prediction

6

1

)|()|(m

imi CaPCAP

Novel research topic VII User Interface

N.H. Liu, "Effective Results Ranking for Mobile Query by Singing/Humming Using a Hybrid Recommendation Mechanism," IEEE Transactions on Multimedia, Aug (2014)(SCI, IF=1.754)