advanced functional programming in industry

8/10/2019 Advanced Functional Programming in Industry

1/84

Advanced Functional Programming in Industry

Jose Pedro Magalhaes

November 21, 2014London, United Kingdom

Jose Pedro Magalhaes Advanced Functional Programming in Industry, FP Days 2014 1 / 46


2/84

Introduction

Haskell: a statically typed, lazy, purely functional language

Modelling musical harmony using Haskell

Applications of a model of harmony: Musical analysis Finding cover songs Generating chords and melodies Correcting errors in chord extraction from audio sources Chordifya web-based music player with chord recognition



3/84

Demo: Chordify

Demo:

http://chordify.net

http://chordify.net/http://chordify.net/


4/84

Table of Contents

Harmony

Haskell

Harmony analysis

Harmonic similarity

Music generation

Chord recognition: Chordify



5/84

What is harmony?

IVIVI V/V

C F D G7 7

C

{Ton TonSDom Dom

Harmony arises when at least two notes sound at the same time

Harmony induces tension and release patterns, that can be describedby music theory and music cognition

The internal structure of the chord has a large influence on theconsonance or dissonance of a chord

The surrounding context also has a large influence



6/84

What is harmony?

IVIVI V/V

C F D G7 7

C

{Ton TonSDom Dom

Harmony arises when at least two notes sound at the same time

Harmony induces tension and release patterns, that can be describedby music theory and music cognition

The internal structure of the chord has a large influence on theconsonance or dissonance of a chord

The surrounding context also has a large influence

Demo: how harmony affects melody



7/84

Simplified harmony theory I

A chordis a group of tones separated by intervals of roughly thesame size.

All music is made out of chords (whether explicitly or not).

There are 12 different notes. Instead of naming them, we numberthem relative to the first and most important one, the tonic. So weget I, II, II . . . VI, VII.

A chord is built on a root note. So I also stands for the chord builton the first degree, V for the chord built on the fifth degree, etc.

So the following is a chord sequence: I IV II

7

V

7

I.



8/84

Simplified harmony theory II

Models for musical harmony explain the harmonic progression in music:

Everything works around the tonic(I).

The dominant(V) leads to the tonic.

The subdominant(IV) tends to lead to the dominant.

Therefore, the I IV V I progression is very common. There are also secondary dominants, which lead to a relative tonic.

For instance, II7 is the secondary dominant of V, and I7 is thesecondary dominant of IV.

So you can start with I, add one note to get I7, fall into IV, change

two notes to get to II7, fall into V, and then finally back to I.



9/84

An example harmonic analysis

IVIVI V/V

C F D G7 7

C

{Ton TonSDom Dom

Piece

PT

T

I

C

PD

D

D

V/V

V7

G:7

II7

D:7

S

IV

F

PT

T

I

C



10/84

Why are harmony models useful?

Having a model for musical harmony allows us to automaticallydetermine the functional meaning of chords in the tonal context.The model determines which chords fit on a particular moment in asong.



11/84

Why are harmony models useful?

Having a model for musical harmony allows us to automaticallydetermine the functional meaning of chords in the tonal context.The model determines which chords fit on a particular moment in asong. This is useful for:

Musical information retrieval (find songs similar to a given song) Audio and score recognition (improving recognition by knowing

which chords are more likely to appear)

Music generation (create sequences of chords that conform to themodel)



12/84

Table of Contents

Harmony

Haskell

Harmony analysis

Harmonic similarity

Music generation




13/84

Why Haskell?

Haskell is a strongly-typed pure functional programming language:

Strongly-typed All values are classified by their type, and types areknown at compile time (statically). This gives us strongguarantees about our code, avoiding many common

mistakes.

Pure There are no side-effects, so Haskell functions are likemathematical functions.

Functional A Haskell program is an expression, not a sequence ofstatements. Functions are first class citizens, and explicitstate is avoided.



14/84

Notes

dataRoot =A | B | C | D | E | F | GtypeOctave=Int

dataNote =NoteRoot Octave



15/84

Notes

dataRoot =A | B | C | D | E | F | GtypeOctave=Int

dataNote =NoteRoot Octave

a4, b4, c4, d4, e4, f4, g4 ::Note

a4 =Note A 4b4 =Note B 4c4 =Note C 4d4 =Note D 4e4 =Note E 4

f4 =Note F 4

g4 =Note G 4



16/84

Melody

typeMelody= [Note]

cMajScale ::MelodycMajScale = [c4, d4, e4, f4, g4, a4, b4]



17/84

Melody

typeMelody= [Note]


cMajScaleRev ::Melody

cMajScaleRev = reverse cMajScale



18/84

Melody

typeMelody= [Note]


cMajScaleRev ::Melody

cMajScaleRev = reverse cMajScale

reverse :: [ ] [ ]reverse [ ] = [ ]reverse (h : t) = reverse t ++ [ h]

(++) :: [ ] [ ] [ ](++) =. . .


T i i


19/84

Transposition

Transposing a melody one octave higher:

octaveUp ::Octave OctaveoctaveUp n = n + 1

noteOctaveUp ::Note NotenoteOctaveUp (Noter o) =Noter (octaveUp o)

melodyOctaveUp ::Melody MelodymelodyOctaveUp m = map noteOctaveUp m


G i l i


20/84

Generation, analysis

Building a canon from a melody:

canon ::Melody Melodycanon m = m ++ canon m


G ti l i


21/84

Generation, analysis

Building a canon from a melody:

canon ::Melody Melodycanon m = m ++ canon m

Is a given melody in C major?

root ::Note Rootroot (Noter o) = r

isCMaj ::Melody BoolisCMaj = all (cMajScale)map root


D t il l ft t


22/84

Details left out

We have seen only a glimpse of music representation in Haskell.

Rhythm

Accidentals

Intervals

Voicing

. . .

A good pedagogical reference on using Haskell to represent music:http://di.uminho.pt/~jno/html/ipm-1011.html

A serious library for music manipulation:http://www.haskell.org/haskellwiki/Haskore


Table of Co te ts
http://di.uminho.pt/~jno/html/ipm-1011.htmlhttp://di.uminho.pt/~jno/html/ipm-1011.htmlhttp://di.uminho.pt/~jno/html/ipm-1011.htmlhttp://www.haskell.org/haskellwiki/Haskorehttp://www.haskell.org/haskellwiki/Haskorehttp://di.uminho.pt/~jno/html/ipm-1011.html


23/84

Table of Contents

Harmony

Haskell

Harmony analysis

Harmonic similarity

Music generation



Application: harmony analysis


24/84

Application: harmony analysis

Parsing the sequenceGminC7 Gmin C

7 FMaj D7 G7 CMaj:

Piece

PT

T

IC:maj

PD

D

DD

V7

G:7

V/V

II7

D:7

S

IV

F:maj

V/IV

I7

C:7

V/I

Vmin

G:min

SIV

ins

V/IV

I7

C:7

V/I

Vmin

G:min


Table of Contents


25/84

Table of Contents

Harmony

Haskell

Harmony analysis

Harmonic similarity

Music generation



Application: harmonic similarity


26/84

Application: harmonic similarity

A practical application of a harmony model is to estimate harmonicsimilarity between songs

The more similar the trees, the more similar the harmony

We dont want to write a diff algorithm for our complicated model;

we get it automatically by using a generic diff The generic diff is a type-safe tree-diff algorithm, part of a students

MSc work at Utrecht University

Generic, thus working for any model, and independent of changes tothe model


Table of Contents


27/84

Table of Contents

Harmony

Haskell

Harmony analysis

Harmonic similarity

Music generation



Application: automatic harmonisation of melodies


28/84

Application: automatic harmonisation of melodies

Another practical application of a harmony model is to help selecting

good harmonisations (chord sequences) for a given melody:

IV

III

V

V

I

III

IV

III

II

We generate candidate chord sequences, parse them with the harmonymodel, and select the one with the least errors.


Visualising harmonic structure


29/84

Visualising harmonic structure

Piece

Phrase

Ton

I: Maj

C: Maj

Dom

Dom

V: Dom7

G: Dom7

II: Dom7

D: Dom7

Sub

IV: Maj

F: Maj

III: Min

E: Min

Ton

I: Maj

C: Maj

You can see this tree as having been produced by taking the chords ingreen as input. . .


Generating harmonic structure


30/84

Generating harmonic structure

Piece

Phrase

Ton

I: Maj

C: Maj

Dom

Dom

V: Dom7

G: Dom7

II: Dom7

D: Dom7

Sub

IV: Maj

F: Maj

III: Min

E: Min

Ton

I: Maj

C: Maj

You can see this tree as having been produced by taking the chords ingreen as input. . . or the chords might have been dictated by the structure!


A functional model of harmony


31/84


PieceM [PhraseM] (M {Maj, Min})




32/84



PhraseM TonM DomM TonM| DomM TonM




33/84

y



TonMaj IMajTonMinI

m

Min




34/84

y



TonMaj IMajTonMinI

m

Min

DomM V7M

| VM

| VII0M| SubM DomM

| II7M V7M




37/84

A naive datatype encoding musical harmony:

dataPiece=Piece[ Phrase ]

dataPhrasewherePhraseIVI::Ton Dom Ton PhrasePhraseVI :: Dom Ton Phrase


Now in HaskellI


38/84




dataTonwhereTonMaj::Degree Ton

TonMin::Degree Ton


Now in HaskellI


39/84





TonMin::Degree Ton

dataDom whereDomV7 ::Degree DomDomV ::Degree DomDomVII0 ::Degree Dom

DomIVV::SDom Dom DomDomIIV ::Degree Degree Dom


Now in HaskellI


40/84





TonMin::Degree Ton

dataDom whereDomV7 ::Degree DomDomV ::Degree DomDomVII0 ::Degree Dom

DomIVV::SDom Dom DomDomIIV ::Degree Degree Dom

dataDegree=I | II| III . . .


Now in HaskellII


41/84

A GADT encoding musical harmony:

dataMode=MajMode|MinMode

dataPiece(::Mode)where

Piece::[ Phrase ] Piece


Now in HaskellIIG


42/84





dataPhrase(::Mode)wherePhraseIVI::Ton Dom Ton PhrasePhraseVI :: Dom Ton Phrase


Now in HaskellIIA GADT di i l h


43/84






dataTon(::Mode)whereTonMaj::SD I Maj Ton MajModeTonMin::SD I Min Ton MinMode


Now in HaskellIIA GADT di i l h


44/84






dataTon(::Mode)whereTonMaj::SD I Maj Ton MajModeTonMin::SD I Min Ton MinMode

dataDom(::Mode)whereDomV7 ::SD V Dom

7 Dom

DomV ::SD V Maj Dom DomVII0 ::SD VII Dim Dom DomIVV::SDom Dom Dom DomIIV ::SD II Dom

7 SD V Dom7 Dom


Now in HaskellIII


45/84

Scale degrees are the leaves of our hierarchical structure:

dataDiatonicDegree=I | II| III | IV |V | VI | VIIdataQuality =Maj | Min | Dom7 |Dim

dataSD(::DiatonicDegree) (::Quality)whereSurfaceChord::ChordDegree SD


Now in HaskellIII


46/84

Scale degrees are the leaves of our hierarchical structure:

dataDiatonicDegree=I | II| III | IV |V | VI | VIIdataQuality =Maj | Min | Dom7 |Dim

dataSD(::DiatonicDegree) (::Quality)whereSurfaceChord::ChordDegree SD

Now everything is properly indexed, and our GADT is effectivelyconstrained to allow only harmonically valid sequences!


Generating harmony


47/84

Now that we have a datatype representing harmony sequences, how dowe generate a sequence of chords?


Generating harmony


48/84


QuickCheck! We giveArbitraryinstances for each of the datatypes in our

model.


Generating harmony


49/84


QuickCheck! We giveArbitraryinstances for each of the datatypes in our

model.

. . . but we dont want to do this by hand, for every datatype, and to haveto adapt the instances every time we change the model. . . so we usegeneric programming:


Generating harmony


50/84

Now that we have a datatype representing harmony sequences, how do

we generate a sequence of chords?

QuickCheck! We giveArbitraryinstances for each of the datatypes in ourmodel.


gen :: .(Representable, Generate(Rep))Gen


Generating harmony


51/84

Now that we have a datatype representing harmony sequences, how do

we generate a sequence of chords?

QuickCheck! We giveArbitraryinstances for each of the datatypes in ourmodel.


gen :: .(Representable, Generate(Rep))[ (String,Int) ] Gen


Examples of harmony generation


52/84

testGen :: Gen(Phrase MajMode)

testGen = gen [("Dom_IV-V", 3), ("Dom_II-V", 4)]example :: IO ()example =letk =Key(Note C)MajMode

insample testGen>>= mapM (printOnKey k)


Examples of harmony generation


53/84

testGen :: Gen(Phrase MajMode)

testGen = gen [("Dom_IV-V", 3), ("Dom_II-V", 4)]example :: IO ()example =letk =Key(Note C)MajMode

insample testGen>>= mapM (printOnKey k)

>example[C: Maj, D: Dom7, G: Dom7, C: Maj][C: Maj, G: Dom7, C: Maj][C: Maj, E: Min, F: Maj, G: Maj, C: Maj]

[C: Maj, E: Min, F: Maj, D: Dom7, G: Dom7, C: Maj][C: Maj, D: Min, E: Min, F: Maj, D: Dom7, G: Dom7, C: Maj]


Generating a melody for a given harmony


54/84

We then generate a melody in 4 steps:

1. Generate a list of candidate melody notes per chord;

2. Refine the candidates by filtering out obviously bad candidates;

3. Pick one focal candidate melody note per chord;

4. Embellish the candidate notes to produce a final melody.


Generating a melody for a given harmony


55/84

We then generate a melody in 4 steps:

1. Generate a list of candidate melody notes per chord;

2. Refine the candidates by filtering out obviously bad candidates;

3. Pick one focal candidate melody note per chord;

4. Embellish the candidate notes to produce a final melody.

These four steps combine naturally using plain monadic bind:

melody ::KeyState MyState Songmelody k = genCandidates>>= refine>>= pickOne>>= embellish

>>= returnSongk


Example I


56/84

&?

#

n

Phrase

Ton

I: Maj

C: Maj

Dom

Dom

V: Dom7

G: Dom7II: Dom

7

D: Dom7

Sub

IV: Maj

F: Maj

III: Min

E: Min

Ton

I: Maj

C: Maj


Example II


57/84

?#

## n#

Phrase

Ton

I: Min

E: Min

Dom

Dom

Dom

V: Dom7

B: Dom7

II: Dom7

F: Dom7

Sub

IV: Min

A: Min

Sub

IV: Min

A: Min

Ton

I: Min

E: Min


Table of Contents


58/84

Harmony

Haskell

Harmony analysis

Harmonic similarity

Music generation



Back to Chordify: chord recognition


59/84

Yet another practical application of a harmony model is to improve chordrecognition from audio sources.

0.92 C 0.96 EmChord candidates 0.94 Gm 0.97 C

1.00 C 1.00 G 1.00 EmBeat number 1 2 3

How to pick the right chord from the chord candidate list? Ask theharmony model which one fits best.


Chordify: architecture


60/84

Frontend Reads user input, such as YouTube/Soundcloud/Deezer links, or files Extracts audio Calls the backend to obtain the chords for the audio Displays the result to the user Implements a queueing system, and library functionality

Uses PHP, JavaScript, MongoDB


Chordify: architecture
http://www.lilypond.org/http://hackage.haskell.org/package/HarmTrace


61/84

Frontend Reads user input, such as YouTube/Soundcloud/Deezer links, or files Extracts audio Calls the backend to obtain the chords for the audio Displays the result to the user Implements a queueing system, and library functionality

Uses PHP, JavaScript, MongoDB Backend

Takes an audio file as input, analyses it, extracts the chords The chord extraction code uses GADTs, type families, generic

programming (see theHarmTracepackage on Hackage) Performs PDF and MIDI export (usingLilyPond) Uses Haskell, SoX, sonic annotator, and is mostly open source


Chordify: numbers
http://hackage.haskell.org/package/HarmTracehttp://www.lilypond.org/http://www.lilypond.org/http://hackage.haskell.org/package/HarmTrace


62/84

Online since January 2013

Top countries: US, UK, Germany, Indonesia, Canada

Views: 3M+ (monthly)

Chordified songs: 1.5M+

Registered users: 200K


How do we handle these visitors?


63/84

Single VPS, 6 Intel Xeon cores, 24GB RAM, 500GB SSD, 2TB harddrive




64/84


Single server hosts both the web and database servers




65/84



Can easily handle peaks of (at least) 700 visitors at a time




66/84




Chordifying new songs takes some computing power, but most songsare in the database already




67/84





Queueing system for busy periods




68/84





Queueing system for busy periods

Infrastructure costs are minimal


Frontend (PHP/JS) and backend (Haskell) interaction


69/84

Frontend receives a music file, calls backend with it




70/84


Backend computes the chords, writes them to a file:




71/84


Backend computes the chords, writes them to a file: 1;D:min;0.232199546;0.615328798

2;D:min;0.615328798;0.998458049

...




72/84



2;D:min;0.615328798;0.998458049

...

Frontend reads this file, updates the database if necessary, andrenders the result




73/84



2;D:min;0.615328798;0.998458049

...

Frontend reads this file, updates the database if necessary, andrenders the result

Backend is open-source (and GPL3); only option is to run it as astandalone executable


The importance of the UI


74/84

Lets have a look at four different online services giving you the chordsfor a song (Radioheads Karma Police).


The importance of the UIChordie


75/84


The importance of the UIRiffstation


76/84


The importance of the UIYoutab.me


77/84


The importance of the UIChordify


78/84


Logistics of an internet start-up


79/84

Chordify is created and funded by 5 people




80/84


If you can do without venture capital, do it!




81/84



You might end up doing more than just functional programming,though: Deciding on what features to implement next Recruiting, interviewing, dealing with legal issues related to

employment Taxation (complicated by the fact that we sell worldwide and support

multiple currencies) User support Outreach (pitching events, media, this talk, etc.)




82/84



You might end up doing more than just functional programming,though: Deciding on what features to implement next Recruiting, interviewing, dealing with legal issues related to

employment Taxation (complicated by the fact that we sell worldwide and support

multiple currencies) User support Outreach (pitching events, media, this talk, etc.)

But its fun, and you learn a lot!


Summary


83/84

Musical modelling with Haskell: A model for musical harmony as a Haskell datatype

Makes use of several advanced functional programming techniques,such as generic programming, GADTs, and type families

When chords do not fit the model: error correction

Harmonising melodies

Generating harmonies

Recognising harmony from audio sources

Transporting academic research into industry


Play with it!


84/84

http://chordify.net

http://hackage.haskell.org/package/HarmTrace

http://hackage.haskell.org/package/FComp

http://chordify.net/http://hackage.haskell.org/package/HarmTracehttp://hackage.haskell.org/package/FComphttp://hackage.haskell.org/package/FComphttp://hackage.haskell.org/package/HarmTracehttp://chordify.net/

advanced functional programming in industry

Documents