Title slide with replaceable image

Cognition & AmplificationDJS@Oticonusa.comDonald J. Schum, PhDVice President, Audiology & Professional RelationsOticon, Inc.

1Speech Understanding:A Cognitive Process

2Real-timeSpeech Understanding More than word recognitionThe ability to extract meaningful information from on-going conversationOn-going, real-timeNormally effortlessExternally Paced3Multiple Cue Speech ProcessingSpeech UnderstandingAcoustic InformationSituationalCuesSentenceStructurePhonemic CuesWordFrequencyStressPatternGesture4XXXAuditory Scene Analysis& Stream SegregationTask:Getting target speech out of a mixed signal7Noise?Complex EnvironmentsMultiple talkersMovementStable, non-speech sourcesUnstable, non-speech sourcesDistractionsShifting focusA little bit of everythingHow good is the brain?Before caller I.D. . . .Mind Reading . . .Party survival skills. . .Streaming ExamplesMixed Same Voice (Female)Mixed Same Voice (Male)Mixed Different Voices

14Natural Voice Separation & Tracking:Fundamental FrequencyHarmonic StructureSupra-segmentalsVisual CuesLinguisticsLoudnessLocationTimbreRateetc.

15How does the brain get the job done?16The brain strives to organize17

Bregman, 198118

Bregman, 198119

Kubovy & Van Valkenburg, 200120Localization helps to take stimulation in early perception and group into specific auditory streams . . . Spatial organization puts all of these streams in relation to one another, allowing for conscious decisions as to what to pay attention to and what to ignoreAudibility Model21Picket FenceD. Brungart & colleaguesWright-Patterson AFB22TimeLevel23TimeLevel24TimeLevel25TimeLevel26TimeLevel27TimeLevel28TimeLevel29Ignoring takes effortLinguistic competition: Hardest to suppress31

T.M.B.32Communication Difficulties in the ElderlyHearing is not independent of the rest of the nervous system

33Which Skills Are Affected?motor skillssensory sensitivity & acuityshort term memorysensory-motor reaction timeprocessing & decision speedselective attentionWhich Skills are Retained?Long-term memory (recall)intelligencelinguistic skillsNormal Aging34Neural SlowingReal-timeSpeech Understanding More than word recognitionThe ability to extract meaningful information from on-going conversationOn-going, real-timeNormally effortlessExternally Paced36Sensorineural Hearing LossLoss of the ability to organize sound

Multiple Cue Speech ProcessingSpeech UnderstandingAcoustic InformationSituationalCuesSentenceStructurePhonemic CuesWordFrequencyStressPatternGesture38Efficient Speech ProcessingAnticipatory ProcessingSortingIdentify potential candidates. . .. . .eliminate what does not fit

Natural Voice Separation & Tracking:

Fundamental FrequencyHarmonic StructureSupra-segmentalsVisual CuesLinguisticsLoudnessLocationTimbreRateetc.

40Listening is normally automatic, effortless.Normals can listen without having to work at it unless conditions get quite difficult. Refer to implicit/explicit idea but without using the terms too much. 41Speech Understanding & Listening EffortPart of the Agil story is effect on Listening Effort.42

HighLowNormal Hearing43

HighLowNormal HearingSNHL44

HighLowUnderstandingEffortHighLowEffort45

HighLowUnderstandingEffortHighLowEffortSNHL46

HighLowUnderstandingEffortHighLowEffortSNHL4748

SNHL: many more listening situations require effort to hear and understandNormals can listen without having to work at it unless conditions get quite difficult. Refer to implicit/explicit idea but without using the terms too much. 49Use of Context by the Elderly(Schum & Matthews, 1992)High Context:The watchdog gave a warning growlLow ContextJohn was talking about the growlLow Context ScoreHigh Context ScoreScores below the line:Poor use of Contextual CuesSchum & Matthews, 1992

No inputthousandsg. . .hundredsgr. . .hundredgro. . .dozensgroc. . .handfulgrocer. . .a fewgroceriesoneSpeech Processing: No ContextSay the word . . .

Input Signal Possible CandidatesNo inputdozensb. . .a fewba. . .a couplebas. . .onebask. . . basket. . . basketba Speech Processing: ContextMy favorite sport is. . .Input Signal Possible CandidatesTwo-channel Monitoring(Schum, 1994)Primary Task:John was talking about the growlSecondary Task:Acknowledge screen color change

Difficulty of Primary TaskReaction TimeSchum, 1994

Schum, 1994

Do cognitive declines affect basic speech understanding?57

58Lunner, 2003S/N-8 -6 -4 -2 0 2 4 6 80 20 40 60 80 100Memory Span (% correct)Relationship between cognitive abilities and speech understanding in noise59Lunner, 2003S/N-8 -6 -4 -2 0 2 4 6 8600 1200 1800 2400 3000 3600Lexical Decision Speed (ms.)Relationship between cognitive abilities and speech understanding in noise60Does that mean a hearing aid will not be useful for the elderly patient?61

62Davis, 2003Speech Understanding (% correct)50 60 70 80 90U A U A U A U A Low/Low Low/High High/Low High/HighCognitive Performance (2 measures)Relationship between cognitive abilities and hearing aid benefit63

Brain vs. Machine64Machine (computer) Approaches

Subtractive in NatureImprove S/N whenever possibleLocation-based Noise ReductionWhen does directionality work?When does directionality work?Talker in front, closeNoise present & from back/sidesReverberation not excessiveSpectral Noise ReductionWhat does it do?

What does it not do?Spectral Noise ReductionAcceptance of noisy environments

No effect on S/NAutomatic systems are Criterion DependentWDRCWDRC: Objectives in Noise125 250 500 1000 2000 4000 80000

20

40

60

80

100120Frequency (Hz.)dB HLUCLHTLAudibility7525

Please jot down how much change I need.p l ea s j o t d ow n how m u ch ch a n ge I n ee d

Notice that each phoneme changes over time.76

Please jot down how much change I need.p l ea s j o t d ow n how m u ch ch a n ge I n ee dAssumed effect of fast acting compressionAssumed effect: HA finds soft phonemes and boosts level without changing morphology. 77

Please jot down how much change I need.p l ea s j o t d ow n how m u ch ch a n ge I n ee dHand Edit (assumed effect of fast acting compression)

Please jot down how much change I need.p l ea s j o t d ow n how m u ch ch a n ge I n ee dActual effect of Fast Acting Compression

But: HA does not recognize speech . . . Recognizes acoustic energy. True effect of FA compression is to boost all periods of soft energy . . . Morphology of the waveform is quickly lost.79SpeechNoiseGain IncreaseGain IncreaseSpeech in noise, but a good S/N80Effect of Fast Compression: Cognitive ConcernSpeechNoiseGain IncreaseGain IncreaseIn softer periods of speech, noise balloons up81

Input 40 ms 160 ms 640 ms 2560 ms

Same signal just processed with different release times82Brain-inspired solutionsWho else wants to know?84Auditory Scene Analysis& Stream SegregationXXX ConceptNatural Voice Separation & Tracking:Fundamental FrequencyHarmonic StructureSupra-segmentalsVisual CuesLinguisticsLoudnessLocationTimbreRateetc.

86Automatic Speech Recognition

Voice Recognition87Automatic Speech Recognition

Voice Recognition88

Voice Recognition

90The waveform created by the vibrating vocal folds with energy at each harmonic (at a smoothly falling intensity rate) is then shaped by the configuration of the articulators (mostly the tongue) inside the mouth. This is referred to as the formant structure, with the most prominent formants being F1, F2 & F3. However, even though the relative intensity pattern is changed due to the formant structure, there is still energy at each harmonic. (For those of the most nerdy of you, this is called the source-filter theory of speech.)

ABCD1234How would it help?Automatic Speech Recognition

Voice Recognition94Automatic Speech Recognition:

How good is it?95How good is it?0 dB S/N +5 dB S/N +10 dB S/N +15 dB S/N +20 dB S/NBabble

961. The boy fell from the window 2. The wife helped her husband. 3. Big dogs can be dangerous.4. Her shoes were very dirty. 5. The player lost a shoe. 6. Somebody stole the money. 7. The fire was very hot. 8. Shes drinking from her own cup. 9. The picture came from a book. 10. The car was going too fast.The boy know from the wife helped her husband and gods in the game or shoes. Her player lost issue somebody still for my player was very he's drinking from her own secure gain from the car was going to that+5 dB S/N Babble971. The boy fell from the window 2. The wife helped her husband. 3. Big dogs can be dangerous.4. Her shoes were very dirty. 5. The player lost a shoe. 6. Somebody stole the money. 7. The fire was very hot. 8. Shes drinking from her own cup. 9. The picture came from a book. 10. The car was going too fast.To myself from the windowthe life helped her husband big dogs can be dangerous her shoes were very dirty the player lost a shoe somebody stole the moneythe fire was very hot cheese drinking from her own copthe picture came from a bookthe car was going too fastSentences in Quiet98

Davis, 2002How would it help?Brain & Computer work in very different ways101Back to the Present:

Can we compensate for declining cognitive skills?102Brain TrainingCompensation StrategiesPatient Education & Realistic ExpectationsCompensation StrategiesPatient Education & Realistic ExpectationsExcellent Speech Processing (preserve information)Compensation StrategiesPatient Education & Realistic ExpectationsExcellent Speech Processing (preserve information)Signal-to-Noise ratio improvementCompensation StrategiesPatient Education & Realistic ExpectationsExcellent Speech Processing (preserve information)Signal-to-Noise ratio improvementFully Automatic DesignCompensation StrategiesPatient Education & Realistic ExpectationsExcellent Speech Processing (preserve information)Signal-to-Noise ratio improvementFully Automatic DesignPacing & complexity of message

Seminars in Hearing, 2001109

110

Wingfield & Tun, 2001111Compensation StrategiesPatient Education & Realistic ExpectationsExcellent Speech Processing (preserve information)Signal-to-Noise ratio improvementFully Automatic DesignPacing & complexity of message Clear SpeechClear Speech: Research HistoryPicheny, Braida, & Durlach: Mid 1980scoined the term, contrasted to conversational style speechdocumented acoustic changesdocumented intelligibility improvement113Acoustic ChangesSlower rate of speechMore frequent and longer pausesLonger phoneme durations (consonants & vowels)More released word-final stopsGreater differentiation of vowelsImproved Consonant/Vowel ratioEffect? 114Acoustic ChangesSlower rate of speechMore frequent and longer pausesLonger phoneme durations (consonants & vowels)More released word-final stopsGreater differentiation of vowelsImproved Consonant/Vowel ratioEffect? 15-20% WR improvement115 Tuborg is a famous Danish beer. So is Carlsberg.Conversational ClearSlower RateHigher LevelMore & Longer Pauses

116 So is Carlsberg.Conversational ClearLonger, more intense consonants (improved C/V)

117Compensation StrategiesPatient Education & Realistic ExpectationsExcellent Speech Processing (preserve information)Signal-to-Noise ratio improvementFully Automatic DesignPacing & complexity of message Clear SpeechFollow-up ProgramsNature of Aging plus Sensorineural Hearing Loss:

Loss of the ability to organize soundAging & Speech Understanding in Complex EnvironmentsDJS@Oticonusa.comDonald J. Schum, PhDVice President, Audiology & Professional RelationsOticon, Inc.

120