reducing confounding factors in automatic acoustic
TRANSCRIPT
Reducing confounding factors inautomatic acoustic recognition
of individual birds
Dan Stowell
Machine Listening LabCentre for Digital Music
[email protected] Acoustic recognition of birds 1 / 31
versus
Machine listening and bird sounds - why?
[email protected] Acoustic recognition of birds 3 / 31
Machine listening and bird sounds - why?
I Changes in populations, in migration patterns—monitoring is important
I Intrusive vs. passive monitoring—behavioural impact of catching/ringing birds
I Many birds are most easily observed by soundI Manual (volunteer) monitoring common, but not scalable
[email protected] Acoustic recognition of birds 4 / 31
In this talk...
Classification-based approaches to:
0. Bird species recognition
1. Bird sound detection (presence/absence)
2. Bird individual ID
(By the way: we do more than just classification!)
[email protected] Acoustic recognition of birds 5 / 31
Species classification of bird sounds
In 2014: feature-learning approach to bird sound recognition
Dataset Location Total duration Num items Num classes Labelling
lifeclef Brazil 77.8 hours (12M frames) 9688 501 singlelabel
mfc
c-m
s
mfc
c-m
axp
mfc
c-m
odul
mels
pec-
ms
mels
pec-
maxp
mels
pec-
modul
mels
pec-
kfl1
-ms
mels
pec-
kfl2
-ms
mels
pec-
kfl3
-ms
mels
pec-
kfl4
-ms
mels
pec-
kfl8
-ms
mels
pec-
kfl4
pl8
kfl4
-ms50
60
70
80
90
100A
UC
(%
)
Feature learning
lifeclef2014 Classifier: multilabel
[email protected] Acoustic recognition of birds 6 / 31
Bird species classification: Warblr
‘Warblr’ app – for Android and iOS
[email protected] Acoustic recognition of birds 7 / 31
Bird species classification: WarblrOver 45,000 recordings submitted to our database (≈ 80/day)
8 6 4 2 0 2 4
50
51
52
53
54
55
56
57
58
Submission geolocations
[email protected] Acoustic recognition of birds 8 / 31
Some of our users...
[email protected] Acoustic recognition of birds 9 / 31
Some of our users...
[email protected] Acoustic recognition of birds 9 / 31
Part 1: Bird Audio Detection challenge
Many projects need reliable detection of bird soundse.g. in long unattended recordings
But existing methods are not robust, not general-purpose enough,and need lots of manual tweaking/post-processing
[email protected] Acoustic recognition of birds 11 / 31
Bird Audio Detection challenge
We designed the Bird Audio Detection challenge
Dev set 1: 10k items,crowdsourced audiofrom around the UK(Warblr phone app)
Dev set 2: 7k items,crowdsourced audiofrom misc field recordings
Testing set: 10k items,remote monitoring,Chernobyl Exclusion Zone
[email protected] Acoustic recognition of birds 12 / 31
Bird Audio Detection challenge
I Training/testing sets differ in:I locationI recording eqptI speciesI class balanceI background soundsI time of dayI time of yearI weatherI ...
I How is a classifier meant to work in such mismatchedconditions???
[email protected] Acoustic recognition of birds 13 / 31
Bird Audio Detection challenge: outcomes
I 30 teams submitted
I Strong results (up to 89% AUC)
I Domain adaptation strategiesI Pseudo-labelling, test mixingI Though not always needed
[email protected] Acoustic recognition of birds 14 / 31
So why do we evaluate using matched conditions?
I To study the classifier’s behaviour
I Sometimes a practical application is in matched conditions
I Pragmatic reasons: only one dataset available; free choice ofbootstrap/n-fold crossvalidation
I ...because our algorithms aren’t good enough at avoidingconfounds?
[email protected] Acoustic recognition of birds 15 / 31
Machine learning workflow
train → validate → test
[email protected] Acoustic recognition of birds 16 / 31
Machine learning workflow
train → validate → test → reallytest
[email protected] Acoustic recognition of birds 16 / 31
Part 2: Identifying individual bird ID
Motivation: reduce intrusive monitoring(capturing/tagging/ringing)
Many birds do have individual signature
[email protected] Acoustic recognition of birds 18 / 31
Identifying individual bird ID
Data collection:
[email protected] Acoustic recognition of birds 19 / 31
Identifying individual bird ID
Data collection:
[email protected] Acoustic recognition of birds 19 / 31
Identifying individual bird ID
Data collection:
Bird ID: categorical label.Is this the “same” task as species classification?
[email protected] Acoustic recognition of birds 19 / 31
Identifying individual bird ID
[email protected] Acoustic recognition of birds 20 / 31
Identifying individual bird ID
[email protected] Acoustic recognition of birds 20 / 31
Making use of silence (1)
Training set:
Testing set:
[email protected] Acoustic recognition of birds 21 / 31
Making use of silence (1)
[email protected] Acoustic recognition of birds 21 / 31
Making use of silence (1)
Training set:
Testing set:
[email protected] Acoustic recognition of birds 21 / 31
Making use of silence (1)
Training set:
Testing set:
[email protected] Acoustic recognition of birds 21 / 31
Analogy: the ‘album effect’ in music artist ID
Training set:Express Yourself Bad
[email protected] Acoustic recognition of birds 22 / 31
Analogy: the ‘album effect’ in music artist ID
Training set:Express Yourself Bad
Testing set:Like a Prayer Smooth Criminal
[email protected] Acoustic recognition of birds 22 / 31
Analogy: the ‘album effect’ in music artist ID
Training set:Express Yourself Bad
Testing set:Like a Prayer Smooth Criminal
[email protected] Acoustic recognition of birds 22 / 31
Analogy: the ‘album effect’ in music artist ID
Training set:Express Yourself Bad
Testing set:Like a Prayer Smooth Criminal
[email protected] Acoustic recognition of birds 22 / 31
Territorial birds: the territory is the ‘album’
standard aug30
40
50
60
70
80
90
100
AUC
(%)
mel spec features
little owl cross-yearpipit within-yearpipit across-yearchiff chaff within-yearchiff chaff across-year
standard aug30
40
50
60
70
80
90
100
AUC
(%)
skfl features
little owl cross-yearpipit within-yearpipit across-yearchiff chaff within-yearchiff chaff across-year
[email protected] Acoustic recognition of birds 23 / 31
Making use of silence (2)
Data augmentation of the TESTING set (adversarial)
Measure the ‘distractability’ of the classifierwhen mismatched silence is added
Measure RMSE in classifier decisions
[email protected] Acoustic recognition of birds 24 / 31
Making use of silence (2)
0 2 4 6 8 10 12Estimated
PC1101
PC1102
PC1103
PC1104
PC1105
PC1106
PC1107
PC1108
PC1109
PC1110
PC1111
PC1112
PC1113
True
Confusion matrix: linhart2015marcutday1day2_melspec-kfl4pe8kfl4-ms_nr05_pk0_heq0_pool0_rfall_max
1
0
1
2
3
4
5
[email protected] Acoustic recognition of birds 25 / 31
Making use of silence (3)
Data augmentation of the TRAINING set
...
Each item gets new versions with added silence from each class
[email protected] Acoustic recognition of birds 26 / 31
Making use of silence (4)
Finally we can add a new wastebasket class
...
NB not using the a/b labels here
[email protected] Acoustic recognition of birds 27 / 31
Conclusions
Outdoor bird sound recognition is tricky:
I The sounds (classes) are highly variable
I Many potential confounding factorsfor black-box ML
1. Bird Audio Detection Challenge:I Good detection, even in strongly mismatched conditionsI Adaptation methods useful—though, not always needed?
2. Recognising individual bird ID:I Strong recognition possible (depending on species)I Silence is surprisingly useful for sound recognition!
Generally: make more use of mismatched-condition testing
[email protected] Acoustic recognition of birds 30 / 31
Thank youCollaborators:
1. Bird Audio Detection Challenge:Mike Wood (U of Salford), Yannis Stylianou (U of Crete),Herve Glotin (U of Toulon), IEEE Signal Processing Society
2. Recognising individual bird ID:Pavel Linhart (Adam Mickiewicz U / Praha U)
Machine Listening Lab:
[email protected] Acoustic recognition of birds 31 / 31