improving speech recognition with embodied cognition and behaviour-based robotics

Improving Speech Recognitionwith Embodied Cognition

and Behaviour-based Robotics

Improving Speech Recognitionwith Embodied Cognition

and Behaviour-based Robotics

Jorge Davila-Chacon

University of Hamburg - Knowledge Technology

www.informatik.uni-hamburg.de/WTM/

Spotify ML Meetup – November 3rd 2014

MotivationMotivation

Jorge Davila-Chacon Bio-Inspired SSL for Robot ASR 2

• Why is bio-inspired SSL interesting / useful?

Neurobotic ExperimentsNeurobotic Experiments


Virtual Reality LabVirtual Reality Lab


Bauer, J., Davila-Chacon, J., Strahl, E., Wermter, S. Smoke and Mirrors — Virtual Realities for Sensor Fusion Experiments in Biomimetic Robotics. In: Multisensor Fusion and Integration for Intelligent Systems, 2012

Neurobotic ExperimentsNeurobotic Experiments

Jorge Davila-Chacon 5Bio-Inspired SSL for Robot ASR

Bio-Inspired Sound Source LocalisationBio-Inspired Sound Source Localisation


ITD

ILD

ITDs fromLow Frequencies

ITDs fromLow Frequencies

ILDs fromHigh Frequencies

ILDs fromHigh Frequencies

Spatial cues allow sound source localisation:

• Interaural Time Difference (ITD)• Interaural Level Difference (ILD)

Spatial cues allow sound source localisation:

• Interaural Time Difference (ITD)• Interaural Level Difference (ILD)

Same frequency component

Same frequency component


ITDs extracted in Medial Superior Olive (MSO)

ITDs extracted in Medial Superior Olive (MSO)

• AVCN - Anterior Ventral Cochlear Nucleus

• AN - Auditory Nerve

• IC – Inferior Colliculus

Interaural Time DifferencesNeuroanatomy

Interaural Time DifferencesNeuroanatomy


Interaural Time DifferencesComputational Principle

Interaural Time DifferencesComputational Principle


ILDs extracted in Lateral Superior Olive (LSO)

ILDs extracted in Lateral Superior Olive (LSO)

• MNTB - Medial Nucleus of the Trapezoid Body

• IC – Inferior Colliculus

Interaural Level DifferencesNeuroanatomy

Interaural Level DifferencesNeuroanatomy



Output ofMSO and LSO integrated in

IC

Output ofMSO and LSO integrated in

IC

J. Dávila-Chacón, S. Heinrich, J. Liu, S. Wermter. Biomimetic Binaural Sound Source Localisation with Ego-Noise Cancellation. International Conference on Artificial Neural Networks, 2012.



MLP

IC

IC



J. Dávila-Chacón, S. Magg, J. Liu, S. Wermter. Neural and Statistical Processing of Spatial Cues for Sound Source Localisation. International Joint Conference on Neural Networks, 2013.



Simple IC outputSimple IC output



Complex IC outputComplex IC output



Static SSLStatic SSL

Dynamic SSL

Dynamic SSL

Feed forwardneural network

Robotic Automatic Speech RecognitionRobotic Automatic Speech Recognition


Platforms used for ASR: iCub and Soundman

Platforms used for ASR: iCub and Soundman



J. Dávila-Chacón, J. Twiefel, J. Liu, S. Wermter. Improving Humanoid Robot Speech Recognition with Sound Source Localisation. International Conference on Artificial Neural Networks, 2014.

Binary measure - Static ASRBinary measure - Static ASR



Continuous measure - Static ASR

Continuous measure - Static ASR


● Robotics as a “sandbox” for learning ML

● Neuroscience provides clues for computational principles

● Embodiment• iCub allows computation of spatial cues

• Interaction with environment can reduce noise

● Signal processing with ANN• Spiking ANN are an effective representation of spatial cues

• Bayesian integration important for dimensionality reduction

• Softmax Neural layer robust to ego-noise and reverberation

ConclusionConclusion


Future WorkFuture Work

● Neural SSL• Integrate GPU version of MSO and LSO

• Propagation of probabilities through time

• From discrete to continuous

● Integration with vision• From supervised to unsupervised SSL

• Possible extension to sensorimotor contingencies• Vision to select between multiple sound sources

• Vision for speech segregation


Thank you for your attention.

[email protected]

LinkedIn: Jorge Davila Chacon

• J. Liu, D. Perez-Gonzalez, A. Rees, H. Erwin, S. Wermter. A biologically inspired spiking neural network model of the auditory midbrain for sound source localisation. Neurocomputing (2010)

• J. Davila-Chacon, S. Heinrich, J. Liu, and S. Wermter. Biomimetic binaural sound source localisation with ego-noise cancellation. International Conference on Artificial Neural Networks (2012)

• J. Bauer, J. Davila-Chacon, E. Strahl, S. Wermter. Smoke and Mirrors — Virtual Realities for Sensor Fusion Experiments in Biomimetic Robotics. Multisensor Fusion and Integration for Intelligent Systems (2012)

• J. Davila-Chacon, S. Magg, J. Liu, S. Wermter. Neural and Statistical Processing of Spatial Cues for Sound Source Localisation. International Joint Conference on Neural Networks (2013)

• J. Dávila-Chacón, J. Twiefel, J. Liu, S. Wermter. Improving Humanoid Robot Speech Recognition with Sound Source Localisation. International Conference on Artificial Neural Networks (2014)


mailto:[email protected]

AppendixAppendix

Best performances with clustering layer


AppendixAppendix

Bayesian IC model


AppendixAppendix

Levenshtein distance



improving speech recognition with embodied cognition and behaviour-based robotics

Documents

medial superior olive

lateral superior olive

multisensor fusion

hamburg knowledge

biomimetic robotics

mirrors virtual realities

sensor fusion experiments

intelligent systems