geber ramalho automatic music accompaniment: a 10-year research program

Geber Ramalho

Automatic Music Accompaniment:

a 10-year research program

Jam Session

Geber Ramalho – SBCM’03

3Why to build such systems?

Arrangement assistant (Rhythm session)

Rehearsal partner

Performance (Stage) partner

Tutoring system

In other markets, such as computer games


4Roadmap

Music Accompaniment

Fragment reuse approach

Research retrospective• ImPact System: jazz bass lines

• Chord Prediction

• D’Accord Guitar

Research perspectives

The Task

ReasoningReasoning

Knowledge:

- Music theory, phrases, ...- skills ...

Goals:- ??

Actions:- Choose and play chords, melody,rhythms

executionexecutionperceptionperception

Chords: ... A7(b9 | Cm7 | F7 | Fm7 | Bb7 | Ebmaj7 . . .

Solo: ...

Piano: ...

Drums: ...

Bass: ...

Agent


6Challenges

Few criteria for evaluating a lot of possible solutions

Knowledge acquisition: hard to justify note by note

Collective creation in real time• Fast and coordinated reactions and planning


7

Fragment Reuse Approach

The Most Promising Approach


8Main Approaches

Creating from scratch note by note• Grammars, Logical Rules, Probability,… for choosing notes

– Ex.: Jonhson-Laird´s, Cybernetic-Composer, etc.

Fragments reuse• Creating new accompaniments by appending music

fragments, retrieved from a library, to one another– Ex.: Band-in-a-Box, ImPact, Hodgson’s, etc.


9From scratch note by note

Probabilistic derivational tree (mostly used)• Or markovian chain approaches

Grammars or rules• Use notes from the local tonality

• Possibly combined with random choices “prefer short notes at the beginning of the phrase”

Constraint-based programming• Filter-out bad notes or phrases

Etc.

.

35

65

50

50

2560

15

100

100

100

100

100


10

Melodic and Rhythmic Fragments

Dm7 A7

Miles Davis:

Important role in creation (specially in Jazz)

• Learning by imitation

• Pattern-oriented musicology


11

Melodic and Rhythmic Fragments

Minimize knowledge acquisition effort

• They are available (scalability)

• Adequate level of granularity

• Embody a knowledge hardly formalized

Improves expressiveness

Cm7(b5) F7

Em7(b5) A7

stepwise, dissonant, low density, ascending, etc.

Chord-based, consonant, medium density, etc.


12Fragments Reuse

Idea (Case-Based Reasoning)• Solve new problems (e.g., chord sequence) adapting

solutions (e.g., musical fragments) used in previous problems (e.g., played before)

Questions• How long should be the fragments?

• How to represent them?

• How to index them?

• Why to choose a particular fragment?

• How to adapt a retrieved fragment to the new context?


13Questions

Fragments Length • Plausibility (variable length) vs. Simplicity (fixed length)

• Reactivity (short) vs. Recurrence (long)

Fragment indexation (attributes)• Context

– Underlying chords, position, tempo,...

• Musical properties – Dissonance, density, range, melodic contour,…

• Time window (how far to go to the past?)

|Em7(b5) | A7(b9) | Cm7 | F7 | Fm7 ...


14Questions

Fragment Choice• Query Q = (C, D) ,

– where C = {c1,...,ck} context descriptionD = {d1,...,dj} desired musical

properties

• Retrieval function F(Q,L), where L is the library– Random– Based on similarity– Based on weights given by the user


15

The Impact System

PhD work


16Main Loop

1) Segment the given Chord Grid into “Chord Chunks”e.g., II-, II-V-I, IV-II-V-I,…

2) Determine the desired “musical properties” of the segment

a - Activate PACTs (intentions)

b - Select the PACTs concerning the segment;

c – Assemble these PACTs to compose the query

3) Retrieve a fragment and adapt it


17Rules for Activating PACTs

Examples

• IF the agent is playing at the theme exposition, THEN “play chord notes until the next section”

• IF the next chord is highly dissonant, THEN “use chromatic scale during it”

• IF the drummer is playing a RIFF, THEN “choose a similar rhythm”

Case RetrievalCase


19

Ornithology

Autumn Leaves

Two Examples


20

What if the Chord Grid is unknown?

MSc work (S. Cunha)


21

Given the chord sequence (and a melody) played so far, which is the next chord?

No precise rules for chord prediction• It depends on the music style

• the musicians themselves

• the very song being played, ...

How to extract structural information from songs?

Chord Prediction


22Our approach on chord prediction

Previous approach (Thom & Dannemberg 95)• n-gram probabilistic models algorithm

• Error rate 42%, for songs in C tonality and without prediction of chords duration

Our Hybrid system• Prediction technique: neural networks (=> prior knowledge on

typical chord sequences)

• Online adaptation: knowledge based (=> sequence tracking rules)

Chords SequenceTracker

NeuralNet next

chord


23Results

On Jazz standards in their original tonality

• MLP-backpropagation: error rate = 11,2%

• Depending on the song, error rate decreases 3% per repetition in average

Sound Example: So Nice

1 Cm7 2 Dm7b5 G7 3 Cm7 4 Cm7Am7 Em

5 Bbm7Bbm7 Em7

6 Eb7 7 AbMaj7 8 AbMaj7

9 Dm7b5 10 G7 11 Dm7b5 G7 12 Cm7

13 Am7b5Bm

14 Ab7 15 Dm7b5 16 G7Am

17 Cm7 18 Dm7b5 G7 19 Cm7 20 Cm7

21 Bbm7 22 Eb7 23 AbMaj7 24 AbMaj7

25 AF

26 EbMaj7 27 Dm7b5 G7Dm7b5

28 Cm7

29 F7 30 Fm7 Bb7Fm7

31 EbMaj7 21 Dm7b5 G7

1st execution

7 erros (in red): 4, 5, 13, 16, 25, 27, 30


1 Cm7 2 Dm7b5 G7 3 Cm7 4 Cm7

5 Bbm7Bbm7 Em7

6 Eb7 7 AbMaj7 8 AbMaj7

9 Dm7b5 10 G7 11 Dm7b5 G7 12 Cm7

13 Am7b5 14 Ab7 15 Dm7b5 16 G7Am

17 Cm7 18 Dm7b5 G7 19 Cm7 20 Cm7


25 A 26 EbMaj7 27 Dm7b5 G7Dm7b5

28 Cm7

29 F7 30 Fm7 Bb7Fm7

31 EbMaj7 21 Dm7b5 G7

2nd execution

4 erros (in red): 5, 16, 27, 30


3rd execution

0 erros!

1 Cm7 2 Dm7b5 G7 3 Cm7 4 Cm7


9 Dm7b5 10 G7 11 Dm7b5 G7 12 Cm7

13 Am7b5 14 Ab7 15 Dm7b5 16 G7

17 Cm7 18 Dm7b5 G7 19 Cm7 20 Cm7


25 A 26 EbMaj7 27 Dm7b5 G7 28 Cm7

29 F7 30 Fm7 Bb7 31 EbMaj7 21 Dm7b5 G7


27

D’Accord Guitar Project(www.daccord.com.br)

2 researchers (Geber Ramalho e Carlos Sandroni)

2 Ph.D. students (Giordano Cabral e Ernesto Lima)

1 M.Sc. Student (Márcio Dahia)

6 undergrad students (Hugo, Bel, Tiago,...)

Partitions vs. Ciphered Songs

D’Accord Guitar• Idea: show intuitively and precisely how to play


29Projeto D’accord

3 Modules• D’Accord Guitar Player

• D’Accord Guitar Editor

• D’Accord Guitar Browser


30Main Challenges

File Format (synchronize information)

Precise scheduling

Cipher interpretation

Solo transcription

Transposition

Cipher Interpretation

Challenges• various positions for the same note and various fingers for

the same position

=> hundreds of possible positions for the same chord

• Theoretical, cultural, personel and anatomic constraints: how to take them into account?


32Cipher Interpretation: steps

Valid chords• Identify valid notes in the fret board

• Identify valid positions in the fret board (according to music theory)

• Identify valid fingerings

Evaluate each chord and chord transition to find the best path

• Multi-criteria decision

Cipher Interpretation

For a single chord• Facility (anatomy/rules)

• Flexibility (anatomy/rules): to play other notes

• Frequency (cultural/data base): how classic?

• Sonorority (cultural/data base): aesthetic

For a chord transition• Facility (anatomy/rules)

• Bass line (musical theory/rules): voice leading

• Melodic (soprano) line (musical theory/rules): voice leading


34Search (Dynamic Programming)


35Results

More than 40.000 users (beta test)

95% considered D’Accord Guitar “good” or “excellent”

However, the right hand was not that good!!• Too repetitive


36

D´Accord Guitar: rhythmic accompaniment

From scratch via a computer keyboard

• good quality

• time-consuming

Apply a rhythmic pattern to all chords

• bad quality

• quite fast


37Cyber-João

Adapt the Impact approach will require• New musical fragments

• New attributes and

• New rules

Due to the changes• Bass to Guitar

• Melody to Rhythm

• Jazz to Bossa Nova

João Gilberto’s records• Manually identified some

rhythmic patterns


38Cyber-João

Examples• Chega de Saudade

• Desafinado

Question• Is the approach scalable?


39VexPat

A Pattern Extraction tool• Identify similar recurrent music material (melody, rhythm, …)


40

Research Perspectives around D’Accord Guitar Project

Projects

MIDIGuitar

MIDIGuitar

CDsCDs Dataacquisition

Symbolic(MIDI) data

Environment/ToolsPattern Extraction

VisualizationMachine Learning

Pre-processing(meta-data inductionand noise filtering)

New Symbolic data

Basic D’Accord Guitar

(+ Cyber-João)

D’Accord GuitarPersonalization

Automatic MusicAnalysis

Prototypedeveloped


42Musicological Analysis of Brazilian Guitar

There are lots of knowledge in harmony, melody, counterpoint,…

However, we know not too much about • Rhythm in general and

• (non-classic) guitar in particular


43Musicological Analysis of Brazilian Guitar

Some open questions• What is a rhythmic pattern, considering short (millisecond)

time variations?

• How performance modifies the standard rhythm?

• Are there rhythmic patterns for a given style and/or performer?

• Is there a way to formalize pattern choices? In which situations are they used?

– Harmonic, melodic, location, tempo…– Previous/next patterns (grammar)…

• How frequently are they used?

• Are there variations of these patterns?


44Final remarks

Automatic music accompaniment• is a challenging domain

• requires different approaches, techniques and tools

• can be useful for different applications

More 10 years…

geber ramalho automatic music accompaniment: a 10-year research program

Documents