camerata at mediaeval 2014 - extracting answer passages from classical music scores using natural...
TRANSCRIPT
C@MERATA at MediaEval 2014 - Extracting Answer Passages from
Classical Music Scores using Natural Language Descriptions
Richard Sutcliffe, University of Essex Tim Crawford, Goldsmiths, University of London
Chris Fox, University of Essex Eduard Hovy, Carnegie-Mellon University
Deane Root, University of Pittsburgh
2
Outline Background
Task Design
Passages and Divisions
Query Types and Music Scores
Gold Standard
Evaluation Metrics
XML Formats
Baseline System
Campaign this Year
Results
Conclusions
4
C@merata Task There is a series of questions with required answers. Provided Question:
• A short noun phrase in English referring to musical features in a score,
• A short classical music score in MusicXML. Required Answer:
• The location(s) in the score of the requested musical feature. We call each such location a passage.
5
Motivation When studying musicological analyses of works of western classical art music there are frequent references to relevant passages in the printed score. Our work if successful could enable such references to be associated with passages automatically, hence facilitating the scholarly work of musicologists and musicians, both professional and amateur. It could also enable passages to be found interactively on the basis of an input natural language phrase.
6
Organisers Richard Sutcliffe, University of Essex (has worked on CLEF QA evaluations for ten years). Tim Crawford, Goldsmiths, University of London (lutenist with the Parley of Instruments, musicologist specialising in electronic editions of works by Henry Purcell, Silvius Leopold Weiss etc). Eduard Hovy, Carnegie-Mellon University (has worked extensively on NLP and QA at TREC, MUC, TIPSTER, CLEF, Machine Reading etc). Deane Root, University of Pittsburgh (Editor-in-Chief of Grove Online - a comprehensive source of information on western classical music) Chris Fox, University of Essex (has worked on NLP for many years, also has a thorough knowledge of music theory).
7
Background Originally there was Information Retrieval along with the SIGIR conference, from 1978 onwards. Interest developed in Music Information Retrieval, leading to the ISMIR conference, from 2000 onwards. TREC IR evaluations began in 1992. CLEF multi-lingual and cross-lingual evaluations began in 2000. MIREX evaluations began in 2005. They are run by the University of Illinois at Urbana-Champaign. Focus on low-level tasks like melodic similarity, mostly with audio data.
8
Background cont. Question Answering started at CLEF in 2003. In 2011, QA4MRE task started. 12 documents on 3 topics (Aids, Climate Change, Music and Society), ten MCQ questions on each. Used TED film transcripts. In 2012, QA4MRE had same topics and used various sources for the music topic, including 1911 Encyclopaedia Britanica and Wikipedia. In 2013, QA4MRE used articles from Grove Online for the music topic: ‘Johann Baptist Cramer’, ‘Electronic Dance Music’, ‘Film Music - Hollywood’, ‘Disciplines of Musicology - Analytic Traditions’.
9
Task Design There are 200 queries. We saw earlier that we have: Provided Question:
• A short noun phrase in English referring to musical features in a score,
• A short classical music score in MusicXML. Required Answer:
• The location(s) in the score of the requested musical feature. We call each such location a passage.
10
Example Query
t: pitch_and_length
q: semiquaver G#
s: corelli_allegro_tr_clementi.xml
[ 4/4, 4, 2:5-2:5 ]
[ 4/4, 4, 5:15-5:15 ]
[ 4/4, 4, 7:3-7:3 ]
...
11
Specifying Passages in a Score We specify a passage as follows:
• a time signature, • a divisions value • a start bar and beat • an end bar and beat Example:
[4/4,1,1:1-2:4] time signature is 4/4. divisions value is 1 so we count in crotchets (quarter notes) Passage starts in bar 1 before the first crotchet (i.e. 1:1) Passage ends in bar two after the fourth crotchet (i.e. 2:4) Thus passage consists of the two complete bars (measures) numbered one and two.
12
What is a Divisions Value? The number of beats into which we divide a crotchet. A suitable value depends on what we wish to demarcate. Working in whole crotchets: Divisions = 1
Working in quavers: Divisions = 2
Working in quaver triplets: Divisions = 3
Working in semiquavers or quavers or triplet quavers: Divisions = 12 !!
With Divisions = 12:
one crotchet = 12 beats
one quaver = 6 beats
one quaver triplet = 4 beats
one semiquaver = 3 beats
Note: MusicXML uses the divisions concept and we got the idea from there.
13
Query Types In C@merata there are twelve types of query:
Type No Example
simple_pitch 30 G5 simple_length 30 dotted quarter note
pitch_and_length 30 D# crotchet perf_spec 10 D sharp trill stave_spec 20 D4 in the right hand word_spec 5 word "Se" on an A flat
followed_by 30 crotchet followed by semibreve melodic_interval 19 melodic octave
harmonic_interval 11 harmonic major sixth cadence_spec 5 perfect cadence
triad_spec 5 tonic triad texture_spec 5 polyphony
All 200
14
MusicXML Scores
Work Staves Scoring Lang
bach_cello_suite_1_bwv1007_prelude 1 vc Amer. bach_chorale_24835b3 4 SATB Eng.
bach_chorale_507b 4 SATB Amer. bach_minuet_in_g_bwv_anh114 2 hpd Amer.
carissimi_o_felix_anima 3 SAB Eng. charpentier_te_deum_preludium 5 2 vn, va, 2 vc Amer.
corelli_allegro_tr_clementi 2 hpd Eng. cutting_galliard_11 1 lute Eng.
dowland_earl_of_essex_measure 1 A Amer. lassus_psalm_50 3 SAB Eng.
lully_andante 3 2 vn, vc Amer. monteverdi_lasciatemi_morire 5 SSATB Amer. purcell_fie_nay_prithee_zd10 3 SSA Eng.
scarlatti_a_se_florindo 3 S, hpd Amer. scarlatti_k466 2 hpd Eng.
tallis_all_praise_to_thee 1 A Eng. telemann_taenzchen 2 hpd Amer.
telemann_twv33_21_tres_vite 2 hpd Eng. vivaldi_concerto_rv299_largo 3 vc, hpd Eng.
weiss_sonata_34_prelude 2 lute Amer.
15
Source of MusicXML Scores Public domain scores were obtained from sites such as musescore.com. Most of these are transcribed by amateurs. MusicXML is an interchange format. Thus, most score-writing programs e.g. Finale or Sibelius can export to MusicXML (and input from it).
17
Example Query 1 (from earlier)
t: pitch_and_length
q: semiquaver G#
s: corelli_allegro_tr_clementi.xml
[ 4/4, 4, 2:5-2:5 ]
[ 4/4, 4, 5:15-5:15 ]
[ 4/4, 4, 7:3-7:3 ]
...
18
Example Query 2
t: followed_by
q: G sharp followed by B
s: corelli_allegro_tr_clementi.xml
[ 4/4, 4, 2:5-2:6 ]
[ 4/4, 4, 5:15-5:16 ]
[ 4/4, 4, 8:11-8:12 ]
...
19
Example Query 3
t: melodic_interval
q: rising melodic octave
s: corelli_allegro_tr_clementi.xml
[ 4/4, 4, 1:1-1:8 ]
[ 4/4, 4, 4:1-4:8 ]
[ 4/4, 4, 4:9-4:16 ]
...
20
Example Query 4
t: triad_spec
q: tonic triad
s: bach_chorale_507b.xml
[ 4/4, 1, 0:1-0:1 ]
[ 4/4, 1, 7:1-7-1 ]
21
Creation of the Gold Standard of 200 Questions Stage 1
20 Baroque scores were collected. 4 on 1 stave, 6 on 2 staves etc. 10 questions set on each score. 10 scores: English terminology, 10 scores: American terminology. Mix of question types set against each score. For each question, answers were found in the score. Gradually built up the required question-type distribution. Questions were recorded in C@merata ascii format.
22
Creation of the Gold Standard (cont.) Stage 2 Answers from Stage 1 were carefully checked. (Precision was good, Recall was sometimes lacking.) Stage 3
Questions in C@merata ascii format automatically converted into C@merata XML format. Stage 4
Test questions automatically extracted from Gold Standard (minus the answers).
23
Evaluation of a Returned Passage Passage is beat correct if it starts at exactly the correct beat (as specified by the divisions value) in the correct start bar and also ends at the correct beat in the end bar. - Useful for applications of results which are themselves automatic. Passage is measure correct if it starts in the bar where the requested feature starts and ends in the bar where the requested feature ends. - Sufficient in many cases for humans to use.
24
Example Query 1: Beat correct vs. Measure Correct
t: pitch_and_length
q: semiquaver G#
s: corelli_allegro_tr_clementi.xml
[ 4/4, 4, 2:5-2:5 ]
[ 4/4, 4, 5:15-5:15 ]
[ 4/4, 4, 7:3-7:3 ]
...
25
Beat Precision (BP) is the number of beat correct passages returned by a system divided by the number of passages (correct or incorrect) returned. Beat Recall (BR) is the number of beat correct passages returned by a system divided by the total number of answer passages known to exist. Measure Precision (MP) is the number of measure correct passages returned by a system divided by the number of passages (correct or incorrect) returned. Measure Recall (MR) is the number of measure correct passages returned by a system divided by the total number of answer passages known to exist.
26
XML Question Format <questions task="camerata" year="2014" task_no="1"
organisation="University of Essex"
group="LAC Group">
<question number="001" music_file="f01.xml" divisions="4">
<text>harmonic perfect fifth</text>
<answer>
</answer>
</question>
</questions>
27
XML Answer Format (for Runs)
<questions task="camerata" year="2014" task_no="1"
runtag="lacg01" organisation="University of Essex"
group="LAC Group">
<question number="001" music_file="f01.xml" divisions="4">
<text>harmonic perfect fifth</text>
<answer>
<passage start_beats="4" start_beat_type="4"
end_beats="4" end_beat_type="4"
start_divisions="4" end_divisions="4"
start_bar="2" start_offset="1"
end_bar="2" end_offset="4" />
<passage start_beats="4" start_beat_type="4"
end_beats="4" end_beat_type="4"
start_divisions="4" end_divisions="4"
start_bar="7" start_offset="5"
end_bar="7" end_offset="8" />
<passage start_beats="4" start_beat_type="4"
end_beats="4" end_beat_type="4"
start_divisions="4" end_divisions="4"
start_bar="9" start_offset=1"
end_bar="9" end_offset=8" />
</answer>
</question>
</questions>
28
Baseline System To encourage participation we created a baseline C@merata system Written in Python 27 Uses excellent Music21 of Michael Scott Cuthbert (Will convert MusicXML into a tractable form and manipulate it) Handles all the basic input of questions, processing and output Shows how to use Music21 Actual question-answering component is skeletal
29
Architecture of Baseline System 1. Analyse the input xml file and extract questions and score files 2. For each question:
• Parse question using Stanford Parser • Determine question type (similar to normal QA) • Parse the score file using Music21 • Based on question type, search for answer passages in score 3. Write out answers to output xml file
30
Campaign this Year Fri 24.03.14 Participant registration Fri 02.05.14 Release of training data Mon 16.06.14 - Fri 20.06.14 Systems answer test questions Fri 27.06.14 Publication of results Thu 16.10.14 - Fri 17.10.14 MediaEval Workshop in Barcelona
31
Participants Runtag Leader Affiliation Country
CLAS Stephen Wan CSIRO Australia DMUN Tom Collins De Montfort University England OMDN Donncha Ó Maidín University of Limerick Ireland TCSL Nikhil Kini Tata Consultancy Services India UNLP Kartik Asooja NUI Galway Ireland
32
Results
Run BP BR BF MP MR MF
CLAS01 0.713 0.904 0.797 0.764 0.967 0.854
DMUN01 0.372 0.712 0.489 0.409 0.784 0.538
DMUN02 0.380 0.748 0.504 0.417 0.820 0.553
DMUN03 0.440 0.868 0.584 0.462 0.910 0.613
LACG01 0.135 0.101 0.116 0.188 0.142 0.162
OMDN01 0.415 0.150 0.220 0.424 0.154 0.226
TCSL01 0.633 0.821 0.715 0.652 0.845 0.736
UNLP01 0.113 0.516 0.185 0.155 0.703 0.254
UNLP02 0.290 0.512 0.370 0.393 0.692 0.501
Maximum 0.713 0.904 0.797 0.764 0.967 0.854
Minimum 0.113 0.150 0.185 0.155 0.154 0.226
Average 0.420 0.654 0.483 0.460 0.734 0.534
33
Summary of All Results Based on Beat F-score (BF) ranking is: CLAS01 (0.797), TCSL01 -(0.715), DMUN03 (0.584), UNLP02 (0.370), OMDN01 (0.220), LACG01 (0.116) All submitted runs beat the baseline Best scores were quite high, so task was perhaps quite easy
34
Results by Question Type
Type BP BR BF MP MR MF
simple_pitch 0.645 0.736 0.677 0.685 0.787 0.720
simple_length 0.780 0.846 0.810 0.830 0.906 0.864
pitch_and_length 0.662 0.726 0.644 0.719 0.803 0.710
perf_spec 0.339 0.547 0.339 0.350 0.582 0.352
stave_spec 0.408 0.682 0.508 0.432 0.732 0.540
word_spec 0.487 0.771 0.520 0.487 0.771 0.520
followed_by 0.291 0.518 0.278 0.351 0.716 0.355
melodic_interval 0.396 0.417 0.402 0.471 0.501 0.481
harmonic_interval 0.185 0.207 0.188 0.269 0.329 0.281
cadence_spec 0.071 0.141 0.093 0.171 0.297 0.214
triad_spec 0.081 0.125 0.095 0.124 0.171 0.138
texture_spec 0.060 0.109 0.075 0.072 0.141 0.092
35
Summary of Question Type Results Ranking by question type using BF: simple_length (0.810), simple_pitch (0.677), pitch_and_length (0.644), word_spec (0.520), stave_spec (0.508), melodic_interval (0.402), perf_spec (0.339), followed_by (0.278), harmonic_interval (0.188), triad_spec (0.095), cadence_spec (0.093), texture_spec (0.072) Not all question types are equally easy First few types in the list are quite easy Last three or four types in the list are very difficult. melodic_interval easier than harmonic_interval
36
Conclusions Results are generally high So task was rather easy Some question types were easier than others Technical aspects (data formats, passages, evaluation etc) worked out well Need more complex queries for a second campaign Participants are interested in continuing
41
Q: passage in common time A: [4/4,1,1:1-4:4] Q: interval of a melodic sixth A: [4/4,4,1:1-1:2], [4/4,4,1:8-1:9] [4/4,4,1:9-1:10], [4/4,4,2:1-2:2], [4/4,4,2:8-2:9], [4/4,4,2:9-2:10], [4/4,4,3:1-3:2], [4/4,4,3:8-3:9], [4/4,4,3:9-3:10] Q: second A: None, because a second is harmonic by default Q: C followed by Eb A: [4/4,4,1:5-1:6], [4/4,4,1:13-1:14], [4/4,4,4:1-4:2], [4/4,4,4:9-4:10] Q: C followed by Eb in the bass clef A: [4/4,4,4:1-4:2], [4/4,4,4:9-4:10] Q: semiquaver E natural / sixteenth note E natural A: [4/4,4,2:3-2:4]
42
Work: J.S. Bach, Suite No. 3 in C Major for Cello, BWV 1009, Sarabande Extract:
Q: harmonic interval of a minor third A: [3/4,2,208:1-208:1] Q: minor third A: [3/4,2,208:1-208:1] (thirds are harmonic by default) Q: dotted quaver / dotted eighth note A: [3/4,4,206:5-206:7], [3/4,4,207:5-207:7], [3/4,4,208:5-208:7] Q: harmonic perfect fifth A: [3/4,1,206:1-206:1], [3/4,1,207:1-207:1], [3/4,4,208:5-208:7] (in last passage, C and G should be dotted but were not in the original) Q: simultaneous harmonic perfect fifth and harmonic eleventh A: [3/4,4,208:5-208:7] (same point, C and G are assumed dotted in this example)
43
Work: J. Dowland, King of Denmark's Galliard, P 40 Extract:
Q: perfect cadence
A: [3/4,1,3:3-4:3] (We are assuming the cadence continues until the end of the bar) Q: four consecutive quavers / four consecutive quarter notes A: [3/4,2,2:3-2:6] Q: dotted minim in the bass / dotted half note in the bass A: [3/4,1,2:1-2:3] Q: harmonic fourth A: [3/4,1,1:1-1:1], [3/4,1,1:2-1:3], [3/4,1,4:1-4:3] (Note that there are two instances in bar (measure) 1 because the chord is played twice)
44
Work: J. Dowland, Pauana Dulandi, P 86 Extract:
Q: consecutive 5ths in the bass A: [4/4,1,2:1-2:4] Q: semibreve Bb in the treble clef / whole note Bb in the treble clef A: [4/4,1,1:1-1:4], [4/4,1,4:1-4:4] Q: octave followed two bars later by another octave
A: [4/4,1,1:1-3:4] (assumed harmonic) Q: Vc triad A: [4/4,1,2:3-2:3]
45
Work: G.F. Handel, Messiah, HG xlv, "And the Glory of the Lord" Extract:
Q: crotchet rest / quarter note rest A: [3/4,1,1:1-1:1] Q: dotted crotchet followed by three quavers / dotted quarter note followed by three eighth
notes A: [3/4,1,5:1-5:3], [3/4,1,6:1-6:3] Q: four hemidemisemiquavers / four sixty-fourth notes A: [3/4,4,3:9-3:9] Q: three quavers in a row / three eighth notes in a row A: [3/4,2,5:4-5:6], [3/4,2,6:4-6:6]
46
Work: G.F. Handel, Messiah, HG xlv, Overture Extract:
Q: D sharp crotchet / D sharp quarter note
A: [4/4,1,22:1-22:1], [4/4,1,25:1-25:1] Q: D natural quaver / D natural eighth note
A: [4/4,2,23:3-23:3], [4/4,2,24:4-24:4] Q: open VII triad in the first inversion A: [4/4,1,23:3-23:3] Q: melodic fourth in the bass clef A: [4/4,1,24:1-24:2]
48
Q: eight staccato notes in succession A: [4/4,1,17:1-17:4] Q: augmented melodic fourth A: [4/4,2,19:3-19:4], [4/4,2,20:4-20:5] Q: third
A: [4/4,4,17:13-17:13], [4/4,2,19:2-19:2], [4/4,2,19:4-19:4], [4/4,4,19:9-19:9], [4/4,4,20:9-20:9] (these are assumed to be harmonic and can thus be across parts) Q: a quaver, then a major third / an eighth note, then major third
A: [4/4,2,19:1-19:2], [4/4,2,19:3-19:4] (third assumed to be harmonic and to follow the quaver immediately) Q: change from bass to treble clef A: [4/4,1,17:2p] (recall that this means the clef change is immediately after 17:2)
49
Work: D. Scarlatti, Keyboard Sonata in D major, K 430 Extract:
Q: three fourths
A: [3/8,2,56:1-56:3] (assumed consecutive) Q: treble clef F natural A: [3/8,2,62:2-62:2] Q: melodic octave
A: [3/8,4,57:1-57:2], [3/8,4,57:2-57:6] (melodic must be stated, otherwise it is harmonic) Q: harmonic 5th followed by harmonic 4th A: [3/8,2,58:1-58:2], [3/8,2,62:1-62:2]
50
Architecture of Baseline System We have built a very basic system to perform the task: 1. Analyse the input xml file and extract questions and score files 2. For each question:
• Parse question using Stanford Parser • Determine question type (similar to normal QA) • Parse the score file using Music21 • Based on question type, search for answer passages in score 3. Write out answers to output xml file
51
Linguistic Observations Queries are noun phrases Head noun group is the main feature (e.g. F#) PP modifiers qualify this (e.g. in the bass clef) A lot of terminology is used (e.g. semiquaver, soprano part, forte bar) Search based on query classification will work at least for simple examples.
52
Summary We are trying to find ways to link natural language descriptions of music to musical scores. Starting with the C@merata task, we are working with very simple tasks to develop the technology. Then we will progress to more complex tasks which are of genuine interest to musicologists. In due course, we aim to tie passages in musicological texts such as in Grove Online to the corresponding music scores.
53
Music Extracts symphony opening with a horn call over shimering strings Bruckner 4th Symphony http://www.youtube.com/watch?v=J8t1TzN0RRY Start: 0:00 End: 0:44 symphony closing with six unison chords Sibelius 5th Symphony http://www.youtube.com/watch?v=nkzrSZKA4cM Start: 9:36 End: 9:56 Nielsen 4th Symphony, Finale (shows two timpanists well) http://www.youtube.com/watch?v=yXDe1hj4HBo Start: 32:10 End: 32:48
54
Extracts Not Used symphony opening with two octave descending arpeggio Beethoven 9th Symphony string quartet opening including double stopping on all four instruments Beethoven op 127 symphony featuring a battle between two timpanists Nielsen 4th Symphony, Finale (not so good) http://www.youtube.com/watch?v=sD9I-UiYfW8 Start: 6:30 End: 7:08
55
Grove’s Dictionary of Music and Musicians • 1879-1889 George Grove, civil engineer, music administrator, writer and then Director of
Royal College of Music, wrote A Dictionary of Music and Musicians in four volumes. • 1904-910 Fuller Maitland edited the second edition - Grove's Dictionary of Music and
Musicians - in five volumes. • 1927 Henry Colles edited the third edition - in five volumes. • 1940 Henry Colles edited the fourth edition - in seven volumes. • 1954 Eric Blom edited the fifth edition - in nine volumes.
56
• 1980 Stanlie Sadie edited the sixth edition - The New Grove Dictionary of Music and
Musicians - in twenty volumes. Contained 22,500 articles and 16,500 biographies. • 2001 Stanlie Sadie edited the seventh edition - in 29 volumes. This was also available
online. • 2009 Deane Root was appointed editor of Grove Music Online. By this time it contained
more than 50,000 articles. Grove is now considered to be the most comprehensive and scholarly source of information on Western Classical Art Music which exists. It is used daily by musicians and musicologists worldwide.
57
What might our queries be like in future? We will show some examples:
• A natural language phrase describes a musical feature.
• A regular expression or other pattern could not capture it.
59
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic"
60
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra'
61
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra' A: Jean Sibelius, Symphony No. 5
62
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra' A: Jean Sibelius, Symphony No. 5 Q: 'symphony featuring a battle between two timpanists'
63
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra' A: Jean Sibelius, Symphony No. 5 Q: 'symphony featuring a battle between two timpanists' A: Carl Nielsen, Symphony No. 4 "The Inextinguishable"
64
What are the Characteristics of these Queries? Not very long Not that specific ('over', 'featuring') Use musical terms ('horn', 'strings', 'stacatto', 'chords', 'full orchestra', 'symphony', 'timpani') Also use non-musical terms, interpreted in a musical way ('shimmering', 'battle')
65
What are the Characteristics of these Queries? Not very long Not that specific ('over', 'featuring') Use musical terms ('horn', 'strings', 'stacatto', 'chords', 'full orchestra', 'symphony', 'timpani') Also use non-musical terms, interpreted in a musical way ('shimmering', 'battle') Nevertheless, experts can answer them!
66
Table 4. Results for simple_pitch Questions
Run BP BR BF MP MR MF
CLAS01 0.959 0.972 0.965 0.982 0.995 0.988
DMUN01 0.717 0.674 0.695 0.790 0.743 0.766
DMUN02 0.729 0.729 0.729 0.798 0.798 0.798
DMUN03 0.955 0.972 0.963 0.968 0.986 0.977
LACG01 0.000 0.000 0 0.200 0.028 0.049
OMDN01 0.000 0.000 0.000 0.000 0.000 0.000
TCSL01 0.959 0.963 0.961 0.982 0.986 0.984
UNLP01 0.422 0.789 0.550 0.478 0.894 0.623
UNLP02 0.422 0.789 0.550 0.478 0.894 0.623
Maximum 0.959 0.972 0.965 0.982 0.995 0.988
Minimum 0.000 0.000 0.000 0.000 0.000 0.000
Average 0.645 0.736 0.677 0.685 0.787 0.720
67
Table 5. Results for simple_length Questions
Run BP BR BF MP MR MF
CLAS01 0.904 0.988 0.944 0.915 1.000 0.956
DMUN01 0.858 0.852 0.855 0.879 0.874 0.876
DMUN02 0.863 0.889 0.876 0.884 0.911 0.897
DMUN03 0.955 0.985 0.970 0.967 0.997 0.982
LACG01 0.529 0.197 0.287 0.620 0.231 0.337
OMDN01 0.408 0.471 0.437 0.419 0.483 0.449
TCSL01 0.979 0.988 0.983 0.991 1.000 0.995
UNLP01 0.636 0.797 0.707 0.791 0.991 0.880
UNLP02 0.636 0.797 0.707 0.791 0.991 0.880
Maximum 0.979 0.988 0.983 0.991 1.000 0.995
Minimum 0.408 0.471 0.437 0.419 0.483 0.449
Average 0.780 0.846 0.810 0.830 0.906 0.864
68
Table 6. Results for pitch_and_length Questions
Run BP BR BF MP MR MF
CLAS01 0.860 0.937 0.897 0.895 0.975 0.933
DMUN01 0.653 0.785 0.713 0.721 0.867 0.787
DMUN02 0.663 0.823 0.734 0.730 0.905 0.808
DMUN03 0.760 0.943 0.842 0.770 0.956 0.853
LACG01 0.157 0.196 0.174 0.172 0.215 0.191
OMDN01 0.714 0.032 0.061 0.714 0.032 0.061
TCSL01 0.723 0.892 0.799 0.754 0.930 0.833
UNLP01 0.460 0.696 0.554 0.582 0.880 0.701
UNLP02 0.460 0.696 0.554 0.582 0.880 0.701
Maximum 0.86 0.943 0.897 0.895 0.975 0.933
Minimum 0.460 0.032 0.061 0.582 0.032 0.061
Average 0.662 0.726 0.644 0.719 0.803 0.710
69
Table 7. Results for perf_spec Questions
Run BP BR BF MP MR MF
CLAS01 1.000 0.862 0.926 1.000 0.862 0.926
DMUN01 0.407 0.379 0.393 0.444 0.414 0.428
DMUN02 0.407 0.379 0.393 0.444 0.414 0.428
DMUN03 0.741 0.690 0.715 0.741 0.690 0.715
LACG01 0.000 0.000 0.000 0.000 0.000 0.000
OMDN01 0.000 0.000 0.000 0.000 0.000 0.000
TCSL01 0.066 0.897 0.123 0.066 0.897 0.123
UNLP01 0.045 0.586 0.084 0.053 0.690 0.098
UNLP02 0.045 0.586 0.084 0.053 0.690 0.098
Maximum 1.000 0.897 0.926 1.000 0.897 0.926
Minimum 0.000 0.000 0.000 0.000 0.000 0.000
Average 0.339 0.547 0.339 0.350 0.582 0.352
70
Table 8. Results for stave_spec Questions
Run BP BR BF MP MR MF
CLAS01 0.568 1.000 0.724 0.568 1.000 0.724
DMUN01 0.534 0.840 0.653 0.568 0.893 0.694
DMUN02 0.534 0.840 0.653 0.568 0.893 0.694
DMUN03 0.619 0.973 0.757 0.619 0.973 0.757
LACG01 0.165 0.240 0.196 0.174 0.253 0.206
OMDN01 0.000 0.000 0.000 0.000 0.000 0.000
TCSL01 0.661 0.987 0.792 0.661 0.987 0.792
UNLP01 0.173 0.440 0.248 0.230 0.587 0.331
UNLP02 0.173 0.373 0.236 0.241 0.520 0.329
Maximum 0.661 1.000 0.792 0.661 1.000 0.792
Minimum 0.000 0.000 0.000 0.000 0.000 0.000
Average 0.408 0.682 0.508 0.432 0.732 0.540
71
Table 9. Results for word_spec Questions
Run BP BR BF MP MR MF
CLAS01 1.000 1.000 1.000 1.000 1.000 1.000
DMUN01 0.750 0.750 0.750 0.750 0.750 0.750
DMUN02 0.750 0.750 0.750 0.750 0.750 0.750
DMUN03 1.000 1.000 1.000 1.000 1.000 1.000
LACG01 0.044 0.250 0.075 0.058 0.333 0.099
OMDN01 0.000 0.000 0.000 0.000 0.000 0.000
TCSL01 0.261 1.000 0.414 0.261 1.000 0.414
UNLP01 0.067 0.833 0.124 0.067 0.833 0.124
UNLP02 0.067 0.833 0.124 0.067 0.833 0.124
Maximum 1.000 1.000 1.000 1.000 1.000 1.000
Minimum 0.000 0.000 0.000 0.000 0.000 0.000
Average 0.487 0.771 0.520 0.487 0.771 0.520
72
Table 10. Results for followed_by Questions
Run BP BR BF MP MR MF
CLAS01 0.748 0.859 0.800 0.830 0.953 0.887
DMUN01 0.090 0.797 0.162 0.093 0.820 0.167
DMUN02 0.092 0.820 0.165 0.094 0.844 0.169
DMUN03 0.094 0.844 0.169 0.096 0.859 0.173
LACG01 0.003 0.008 0.004 0.068 0.156 0.095
OMDN01 0.567 0.133 0.215 0.567 0.133 0.215
TCSL01 0.733 0.688 0.710 0.842 0.789 0.815
UNLP01 0.000 0.000 0.000 0.025 0.695 0.048
UNLP02 0.000 0.000 0.000 0.260 0.633 0.369
Maximum 0.748 0.859 0.800 0.842 0.953 0.887
Minimum 0.000 0.000 0.000 0.025 0.133 0.048
Average 0.291 0.518 0.278 0.351 0.716 0.355
73
Table 11. Results for melodic_interval Questions
Run BP BR BF MP MR MF
CLAS01 0.660 0.837 0.738 0.699 0.886 0.781
DMUN01 0.528 0.545 0.536 0.724 0.748 0.736
DMUN02 0.521 0.610 0.562 0.701 0.821 0.756
DMUN03 0.562 0.659 0.607 0.736 0.862 0.794
LACG01 0.000 0.000 0.000 0.158 0.024 0.042
OMDN01 0.000 0.000 0.000 0.000 0.000 0.000
TCSL01 0.894 0.683 0.774 0.904 0.691 0.783
UNLP01 0.000 0.000 0.000 0.000 0.000 0.000
UNLP02 0.000 0.000 0.000 0.000 0.000 0.000
Maximum 0.894 0.837 0.774 0.904 0.886 0.794
Minimum 0.000 0.000 0.000 0.000 0.000 0.000
Average 0.396 0.417 0.402 0.471 0.501 0.481
74
Table 12. Results for harmonic_interval Questions
Run BP BR BF MP MR MF
CLAS01 0.158 0.429 0.231 0.353 0.957 0.516
DMUN01 0.415 0.386 0.400 0.585 0.543 0.563
DMUN02 0.415 0.386 0.400 0.585 0.543 0.563
DMUN03 0.492 0.457 0.474 0.631 0.586 0.608
LACG01 0.091 0.014 0.024 0.273 0.043 0.074
OMDN01 0.000 0.000 0.000 0.000 0.000 0.000
TCSL01 0.000 0.000 0.000 0.000 0.000 0.000
UNLP01 0.000 0.000 0.000 0.000 0.000 0.000
UNLP02 0.000 0.000 0.000 0.000 0.000 0.000
Maximum 0.492 0.457 0.474 0.631 0.957 0.608
Minimum 0.000 0.000 0.000 0.000 0.000 0.000
Average 0.185 0.207 0.188 0.269 0.329 0.281
75
Table 13. Results for cadence_spec Questions
Run BP BR BF MP MR MF
CLAS01 0.238 0.625 0.345 0.286 0.750 0.414
DMUN01 0.083 0.125 0.100 0.333 0.500 0.400
DMUN02 0.083 0.125 0.100 0.333 0.500 0.400
DMUN03 0.167 0.250 0.200 0.417 0.625 0.500
LACG01 0.000 0.000 0.000 0.200 0.125 0.154
OMDN01 0.000 0.000 0.000 0.000 0.000 0.000
TCSL01 0.000 0.000 0.000 0.000 0.000 0.000
UNLP01 0.000 0.000 0.000 0.000 0.000 0.000
UNLP02 0.000 0.000 0.000 0.000 0.000 0.000
Maximum 0.238 0.625 0.345 0.417 0.75 0.500
Minimum 0.000 0.000 0.000 0.000 0.000 0.000
Average 0.071 0.141 0.093 0.171 0.297 0.214
76
Table 14. Results for triad_spec Questions
Run BP BR BF MP MR MF
CLAS01 0.348 0.727 0.471 0.391 0.818 0.529
DMUN01 0.100 0.091 0.095 0.200 0.182 0.191
DMUN02 0.100 0.091 0.095 0.200 0.182 0.191
DMUN03 0.100 0.091 0.095 0.200 0.182 0.191
LACG01 0.000 0.000 0.000 0.000 0.000 0.000
OMDN01 0.000 0.000 0.000 0.000 0.000 0.000
TCSL01 0.000 0.000 0.000 0.000 0.000 0.000
UNLP01 0.000 0.000 0.000 0.000 0.000 0.000
UNLP02 0.000 0.000 0.000 0.000 0.000 0.000
Maximum 0.348 0.727 0.471 0.391 0.818 0.529
Minimum 0.000 0.000 0.000 0.000 0.000 0.000
Average 0.081 0.125 0.095 0.124 0.171 0.138
77
Table 15. Results for texture_spec Questions
Run BP BR BF MP MR MF
CLAS01 0.182 0.500 0.267 0.273 0.750 0.400
DMUN01 0.100 0.125 0.111 0.100 0.125 0.111
DMUN02 0.100 0.125 0.111 0.100 0.125 0.111
DMUN03 0.100 0.125 0.111 0.100 0.125 0.111
LACG01 0.000 0.000 0.000 0.000 0.000 0.000
OMDN01 0.000 0.000 0.000 0.000 0.000 0.000
TCSL01 0.000 0.000 0.000 0.000 0.000 0.000
UNLP01 0.000 0.00 0.000 0.000 0.000 0.000
UNLP02 0.000 0.000 0.000 0.000 0.000 0.000
Maximum 0.182 0.500 0.267 0.273 0.750 0.400
Minimum 0.000 0.000 0.000 0.000 0.000 0.000
Average 0.060 0.109 0.075 0.072 0.141 0.092