爀吀愀氀欀椀猀搀攀猀椀最渀攀搀琀漀戀攀昀漀爀琀栀攀 … roy...

Earphones: The Next Computing Platform

Romit Roy Choudhury

after Smartphones

Presenter

Presentation Notes

Pleasure to be here Body clock time is 2am … Talk is in 2 parts Talk is designed to be for the broad audience … so almost everything should be accessible. So if you don’t get something, please ask. That way, we will also have some interactions. I certainly don’t want to be the only person speaking.

Nura

Bose

Nokia

Apple

Samsung

Dolby, Oculus

Nura

Bose

Nokia

Apple

Samsung

Dolby, Oculus

… just the tip of the iceberg

Mic IMUSpkr. BLE

Sensors, hardware

Mic IMUSpkr. BLE Vibration Optical Neural ArraysIn-Ear

Sensors, hardware

Mic IMUSpkr. BLE Vibration Optical Neural ArraysIn-Ear

Acoustics,Speech

Wireless(BLE, WiFi, 60GHz)

Multi-modal Sensing

Embedded Systems

Machine Learning

Sensors, hardware

Primitives, Algorithms

Mic IMUSpkr. BLE

Sensors, hardware

Acoustic

Health Sense

Security

Navigate me

Acoustics,Speech

Wireless(BLE, WiFi, 60GHz)

Multi-modal Sensing

Embedded Systems

Machine Learning

Primitives, Algorithms

Vibration Optical Neural ArraysIn-Ear

Translate

2005 2015 20232018

But why Earables? Why now?

Reading and typing, a cognitive re-focus

Voice + hearing seamless for humans


1


Phone, watch, fitbit = quantified lower body


Earables gateway to upper body, head


1

2


Socially well accepted (unlike, Google Glass)

Phone, watch, fitbit = quantified lower body


Earables gateway to upper body, head

Takes off one important risk factor


1

2

3

But what are hurdles … show-stoppers?

Many Hurdles …

Energy

Many Hurdles …

1

Energy

Discomfort

Many Hurdles …

1

2

Energy

Discomfort

Privacy / Shy

Many Hurdles …

1

2

3

Energy

Discomfort

Privacy / Shy

Health, radiation

Many Hurdles …

1

2

3

4

Energy

Discomfort

Privacy / Shy

Health, radiation

Many Hurdles …but not Insurmountable

1

2

3

4

Contact Mic.

Hollow

Whisper Zone

Low Frequency

We are building a

software libraryfor Earable Computing

With particular interest in the more challenging problems …

We are building a


Voice assistants

Motion tracking

In-Ear Anatomy (health, security)

Face/Jaw Activity (eating, drinking)

Spatial acoustics

Noise Cancellation

Sensor and Hardware Layer

Beamforming to Speech

Voice assistants

Motion tracking



Spatial acoustics

Noise Cancellation



We are building a


Voice assistants Low SNR Speech Recognition (whisper)

Voice assistants Low SNR Speech Recognition (whisper)

WhisperDecoder ASR

Voice assistants

Motion tracking



Spatial acoustics

Noise Cancellation



Spatial Acoustics

Different propagation delays at two ears

Spatial Acoustics


Inject artificial delays as if sounds coming from a specific direction

Spatial Acoustics


But delays are not unique along hyperbola


Spatial Acoustics




Personal Transfer Function

3D Gazing Direction

Spatial Acoustics




Personal Transfer Function

FollowMe

Starbuckshere

Acoustic Augmented Reality (AAR)

Spatial Acoustics

Voice Escort

Voice assistants

Motion tracking



Spatial acoustics

Noise Cancellation



We are building a


Motion tracking Indoor localization


Body/skeleton is a natural low pass filter


LocationTracking

Lower body IMU (smartphone)

Head IMU (earphone)

Kinematic Motion models

Body/skeleton is a natural low pass filter

Voice assistants

Motion tracking



Spatial acoustics

Noise Cancellation



We are building a


Beamforming to Speech Estimate Angle of Arrival (AoA)

Beamforming to Speech Estimate Angle of Arrival (AoA)

AoA Estimate.

.

.

Path #2(1st Echo)

Path #3(2nd Echo)

Path #1(Direct Path)

Estimate Angle of Arrival (AoA) Beamforming to Speech

Path #2(1st Echo)

Path #3(2nd Echo)

Path #1(Direct Path)

A B C D E F G H I JA B C D E F G

……A B C …

A B C D E F G H I JA B C D E. …

…A B C …

Iterative Align and Cancel (IAC)

Algorithm

Voice assistants

Motion tracking



Spatial acoustics

Noise Cancellation



We are building a


Noise Cancellation

Ear Drum

Noise

Anti-Noise

Bone Conduction

Speaker

NoiseCancelation

Cancellation with Hollow earbuds

Face/jaw activity

Different eating, drinking, tooth brushing activity

Activities = Eating, Drinking, Brushing

In-ear health sensing Ear Impulse Response (EIR)

Voice assistants

Motion tracking



Spatial acoustics

Noise Cancellation



We are building a


Questions thus far …

Part II

Design a hollow earable

that can still cancel noise

Existing Solutions

This Project

Existing SolutionsBetter Noise Cancellation

9dB Better

This Project

Existing Solutions

Better Form Factor(Hollow)

Better Noise Cancellation

Comparable Performance

9dB Better

Wireless Networking +

Our System: MUTE

Noise Cancellation

1. Sound starts

Alice

1. Sound starts

2. IoT relay forwards sound over wireless

Alice

1. Sound starts

2. IoT relay forwards sound over wireless

Alice

Wireless radios travel a million times faster than sound

1. Sound starts

2. IoT relay forwards sound over wireless 3. Wireless arrives at

ear-device earlier

Alice

1. Sound starts


ear-device earlier

4. Actual sound arrives later

Alice

1. Sound starts


ear-device earlier

Lookahead

4. Actual sound arrives later

Alice

Lookahead allows us to cover ears in time

Analogy: Light travels much faster than sound

MUTE: Leverage lookahead for noise cancellation

Talk Outline

How can MUTE leverage lookahead?

Wideband Cancellation

Non-Causal Filtering

Sound Source Profiling

Timing Gain

Signal Processing Gain

Application-Specific Gain

Noise Cancelling Headphones-- What is inside?

Noise Cancellation Primer

Ref. Mic. DSP Anti-NoiseSpeaker


Ref. Mic. DSP Anti-NoiseSpeaker

Feedback Mic.

(Error Feedback)

How do they perform today?

500 1000 1500 2000 2500 3000 3500

Frequency (Hz)

-30

-20

-10

0

Can

cella

tion

(dB)

Bose_Active

Active noise cancellation only at low frequencies

Bose QC35

Good

Bad

500 1000 1500 2000 2500 3000 3500

Frequency (Hz)

-30

-20

-10

0

Can

cella

tion

(dB)

Bose_Active


Bose QC35

Good

Bad

Sound absorbing material blocks high frequencies


500 1000 1500 2000 2500 3000 3500

Frequency (Hz)

-30

-20

-10

0

Can

cella

tion

(dB)

Bose_Active

Bose_Overall

Sound absorbing material blocks high frequencies

Bose QC35

Good

Bad

Why does noise cancellation not work at higher frequencies?

Let’s look into the headphone again.

Ref. Mic. Fb. Mic.DSP Speaker

Time


𝑡𝑡1

𝑡𝑡2

Time


𝑡𝑡2

Time

𝑡𝑡1


𝑡𝑡1

𝑡𝑡2

Time budget (lookahead):𝑡𝑡2 − 𝑡𝑡1 ≈ 30 μs

1 cm

Time


𝑡𝑡1

𝑡𝑡2

𝑡𝑡3


Time


𝑡𝑡1

𝑡𝑡2

𝑡𝑡3


Time

Time lag: Δ𝑡𝑡 = 𝑡𝑡3 − 𝑡𝑡2


Time

High Frequency:

Anti-Noise

Noise

Residual Error

Low Frequency:

Anti-Noise

Noise

Residual Error


Time

500 1000 1500 2000 2500 3000 3500

Frequency (Hz)

-30

-20

-10

0

Can

cella

tion

(dB)

Bose_Active

Let’s now look at noise cancellation in MUTE ...


𝑡𝑡1

𝑡𝑡2

Time

Fb. Mic.DSP Speaker

𝑡𝑡2

𝑡𝑡1

Ref. Mic.

Time

Fb. Mic.DSP Speaker

𝑡𝑡1

𝑡𝑡2

Ref. Mic.

Time

Fb. Mic.DSP Speaker

𝑡𝑡2

Ref. Mic.𝑡𝑡1

1 m

Time budget (lookahead):𝑡𝑡2 − 𝑡𝑡1 ≈ 3 ms(100X larger)

Time

Fb. Mic.DSP Speaker

𝑡𝑡2

Ref. Mic.𝑡𝑡1


Time

Fb. Mic.DSP SpeakerRef. Mic.𝑡𝑡1


Cancel high frequencies

Anti-Noise

Noise

Residual Error

Time

Talk Outline





Timing Gain




Ref. Mic. Feedback Mic.

DSP



DSP

𝑓𝑓(•)



DSP

𝑔𝑔(•)

𝑓𝑓(•)



𝑓𝑓(•)

𝑔𝑔(•)

−𝑓𝑓−1 ∗ 𝑔𝑔


Future PastCurrent

Channel Impulse Response



Future PastCurrent

Today’s Headphones (Causal)

MUTE (Non-Causal)

Channel Impulse Response

Lookahead

“Lookahead” Non-Causal Filtering Better Cancellation


Talk Outline





Timing Gain



Alice Bob Alice

With lookahead …

Alice Bob Alice

Talk Outline





Timing Gain



Implementation & Evaluation

MUTE Platform

Noise Source

MUTE PlatformIoT Relay

Noise Source

MUTE Platform

Wireless Receiver

Noise SourceIoT Relay

MUTE Platform

DSP Processor

Wireless Receiver


MUTE Platform

Feedback Microphone

Wireless Receiver


DSP Processor

MUTE Platform

Anti-Noise SpeakerWireless Receiver


DSP Processor

Feedback Microphone

MUTE Platform

MUTE (Hollow)

Evaluation

Goal: Comparable Performance

MUTE (Hollow) Ear Blocking Headphone

500 1000 1500 2000 2500 3000 3500

Frequency (Hz)

-40

-30

-20

-10

0

Can

cella

tion

(dB)

Bose

MUTE (Hollow)

2. MUTE hollow design (comfort)1. Bose headphone

Evaluation

Goal: Better Performance

MUTE (Non-Hollow) Ear Blocking Headphone

500 1000 1500 2000 2500 3000 3500

Frequency (Hz)

-40

-30

-20

-10

0

Can

cella

tion

(dB)

Bose

MUTE (Hollow)

MUTE (Non-Hollow)

2. MUTE hollow design (comfort)3. MUTE non-hollow design (performance)

1. Bose headphone

+

Zooming Out

Wireless Signal

MUTE Tabletop Relay

DSP

MUTE Tabletop Relay

Noise Cancellation as an Edge Service

DSP

Noise Cancellation as an Edge Service

DSP

Anti-Noise for Alice

Feedback

MUTE Enabled Noise Sources

Conclusion

Acoustic Data Plane

Wireless Control Plane

MUTETabletop Relay

MUTEEdge Service

MUTENoise Source

AcousticNoise

Cancellation

Conclusion

AcousticNoise

Cancellation

On-Ear Machine Learning

Acoustic Augmented

Reality…

MUTETabletop Relay

MUTEEdge Service

MUTENoise Source

Acoustic Data Plane

Wireless Control Plane

Closing Thoughts …

The Wearable Market Projections

WiFore: https://www.nickhunn.com/wp-content/uploads/downloads/2014/08/The-Market-for-Smart-Wearables.pdfhttps://www.nickhunn.com/wp-content/uploads/downloads/2014/08/The-Market-for-Smart-Wearables.pdf

https://www.nickhunn.com/wp-content/uploads/downloads/2014/08/The-Market-for-Smart-Wearables.pdf

https://www.nickhunn.com/wp-content/uploads/downloads/2014/08/The-Market-for-Smart-Wearables.pdf

Voice assistants

Motion tracking



Spatial acoustics

Noise Cancellation



We are building a


Thanks to my excellent

students and collaborators …

Thanks to all of youJay PrakashVisiting Student

爀吀愀氀欀 椀猀 搀攀猀椀最渀攀搀 琀漀 戀攀 昀漀爀 琀栀攀 … roy...

Documents

爀吀愀氀欀椀猀搀攀猀椀最渀攀搀琀漀戀攀昀漀爀琀栀攀 … roy...