multimedia concepts and applications multimedia concepts and applications affect sensing in speech:...

Multimedia Concepts and Applications


Affect Sensing in Speech: Studying Fusion of Linguistic and Acoustic Features

Alexander Osherenko, Elisabeth André, Thurid VogtUniversity of Augsburg


Multimedia Concepts and ApplicationsAffect Sensing

• Acoustic information• Linguistic information (lexical, stylometric,

deictic)


Multimedia Concepts and ApplicationsFusion

• Decision-level• Feature-level


Multimedia Concepts and ApplicationsResearch Questions

• Fusion• Context• Decision-level vs. feature-level


Multimedia Concepts and ApplicationsExperimental Setting

• SAL corpus, 574 turns, 5 classes• Decision-level using majority, feature-level –

fusing features• Data: 2 stages (history 0 and history 7)– Acoustic modality - 2 (discrete/continuous)

acoustic datasets (A)– Lingustic modality - 29 lexical (L), 31 stylometric

(S), 63 deictic datasets (D)



• Tree– Nodes – features from

particular modalities (A, L,S, D)

– Values

• Maximal recall value• Maximal multimodality

value• Dotted arcs

Results‘ representation



• Best results: 64.2% (history 7) and 44.2% (history 0)

• Significant improvement through context• Insignificant improvement through fusion (about 2%)• Maximal multimodality value (76.5%)

Decision-level Fusion Before Discretization



• Best results: 66.0% (history 7) and 49.0% (history 0)

• Significant improvement through context• Insignificant improvement through fusion (about 2%)• Maximal multimodality value (77.8%)

Decision-level Fusion After Discretization



• Best results: 62.8% vs. 64.2% (history 7) and 46.7% vs. 44.2% (history 0)

• Significant improvement through context• Insignificant improvement through fusion (about 2%)

Feature-level Fusion Before Discretization



Feature-level Fusion After Discretization

• Best results: 67.5% vs. 64.9% (history 7) and 52.8% vs. 45.9% (history 0)

• Significant improvement through context• Insignificant improvement through fusion (about 2%)


Multimedia Concepts and ApplicationsDiscussion

• Role of context• Role of discretization• Fusion?


Multimedia Concepts and ApplicationsFuture work

• New modalities• Weighting

multimedia concepts and applications multimedia concepts and applications affect sensing in speech:...

Documents