Robotic Social Therapy on Children with Autism

Preliminary Evaluation Through Multi Parametric Analysis

Daniele Mazzei, University of Pisa

SocialCom2012First International Workshop on Wide Spectrum Social Signal Processing Amsterdam 3th September 2012

ASDs and Robotics• One of the main difficulties in subjects

with autism spectrum disorders (ASDs) is their inability to understand and analyze the emotional state of their interlocutor.

• Recent research shows that ASDs perceive robots not as machines, but as their artificial partners

R. Picard, “Future affective technology for autism and emotion communication,” Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 364, pp. 3575–3584, dic 2009

The FACET Hypothesis

A THERAPIST-GUIDED SOCIAL INTERACTION BETWEEN A ROBOT ABLE TO EXPRESS EMOTIONS AND

ASDS CAN HELP THEM TO IMPROVE THEIR SOCIAL SKILLS.

IDIA project has been founded by Italian Ministry of Health

The FACE of Autism, Mazzei et. All, ROMAN2010, Viareggio Italy, Sept. 2010

This hypothesis strongly depends on the capability of the treatment to convey affective stimuli involving

the subjects

What we need?

Interaction protocol

FACE RobotHIPOP

acquisition platform

FACET (FACE THERAPY)

The FACE Android

Happiness

Anger

Sadness

Disgust

Fear

Surprise

The FACE Android

The FACET protocol

Subject – Robot/Therapist interaction

Phase 2:Familiarization with the FACET room and the

android

Phase 3:Exposition and

interpretation of FACE’s

expressions

Phase 4:Exposition and

interpretation of the therapist’s

expressions

Phase 5Shared

attention

20 min0 min

Phase 1:Baseline

recording (5 min)

Phase 6:Free Play

5 ASDs (all males) with QI higher than 805 normally developing N.Dev (11 males and 4 females)Age: 6-12 years

Multi input analysis method

• ECG filtered and HRV extracted

• HRV signal was divided in 6 phases according to the therapists’ annotations

• Features extracted with Kubios

[Kubios] M. P. Tarvainen, et all, “Kubios hrv-a software for advanced heart rate variability analysis,” in 4th European Conference of the International Federation for Medical and Biological Engineering, IFMBE Proceedings 2009, vol. 22, 2009, pp. 1022–1025.

Physiological signal processing: ECG

Physiological signal processing: EDR• Signal trend removed (only phasic component)• Low pass filters at 2 and 0.2 Hz obtaining 2 signals• Derivative signals extraction • Division in phases

Extracted Features:• Area Under the

Curve • Mean Amplitude• Number of

Peaks

Data analysis• Each feature was normalized by subtracting

the correspondent baseline phase value• Three analysis steps:

1. Assessing the homogeneity of the two groups (ASDs, Control)

2. Identifying statistical significant differences between protocol phases

3. Classifying populations and phases automatically

Group homogeneity assessmentKruskal-Wallis Test

In general: EDR features p-value > 0.7 and HRV features p-value > 0.05

Phases statistical differences analysis Mann-Whitney test

EDR features statistically discriminate

phase 2 and 3!p-value < 0.05

HRV features do not

discriminate phase 2 and 3p-value > 0.05

Classification• Features dataset reduced using PCA• Selected the first 15 principal components

that describe 90% of the variance• Pattern recognition algorithm based on the

K-Nearest Neighborhood non-parametric classifier

• Supervised classifier

Population classification

Normal developing and ASD subjects population classification

Norm. Dev ASD

Norm. Dev 92.50 ± 12.49 6.67 ± 7.46

ASD 7.50 ± 12.49 93.33 ± 7.46

Classification percentage > 92%

Physiological signals acquired during the interaction with FACE allow to classify ASDs and N.Dev!

Other subjects could be classified in blind using this trained classifier

Phases classification• Only phase 2 and 3 could be classified

ASD subjects phase 2 and 3 classificationPhase 2 Phase 3

Phase 2 89.7436 ± 8.58 14.5299 ± 7.35

Phase 3 10.2564 ± 8.58 85.4701 ± 7.35

Normal developing subjects phase 2 and 3 classification

Phase 2 65.50 ± 21.90 38.5000 ± 24.55

Phase 3 33.50 ± 21.90 61.5000 ± 24.55

In ASD population phases recognized with percentage > 85%

In N.Dev population phases recognized with percentage > 65%

FACET protocol phase 2 and 3 are able to induce different psycho-physiological reactions in ASDs but not in N.Dev!

Behavioral analysis• All ASDs followed FACE

during shared attention task

• 55% of ASDs established spontaneous conversation with FACE ASD N.Dev

0%10%20%30%40%50%60%70%80%90%

100%100%

60%

Shared Attention Success

ASD N.Dev0%

10%

20%

30%

40%

50%

60%

55%

30%

Spontaneus conversation with FACE

FACE is able to trigger ASDs attention

ASDs are more attracted by FACE than N.Dev

Expressions labeling• Happiness, Anger and Sadness well labeled • Difficulties in Fear, Disgust and Surprise

recognition

FACE and therapist expressions induce similar results

Facial expressions labeling difficulties can be related to the subjects’ age in accordance with literature*

S. Widen and J. Russell, “Children acquire emotion categories gradually”, Cognitive Development, vol. 23, no. 2, pp. 291–312, 2008

Conclusions• FACET is well accepted by ASDs• Able to induce different reactions in

ASD and N.Dev subjects• Protocol able to induce in ASDs

different reactions among phases 2 and 3

• Well designed for triggering attention in ASDs

Conclusions• Thanks to its predictable

and stereotyped nature FACE perfectly fits ASDs behavioral attitude

• EDR may be a good candidate for ASD treatment protocols and therapies evaluation

© Fondazione ARPA Pictures by Enzo Cei

Future Works

• On going experiments on control and ASD subjects• More tests of the FACET and HIPOP

hardware/software infrastructure• Use of the FACET platform to perform generic studies

on human-robot empathic links

Thanks For Your Attention

Questions?

mazzei@di.unipi.it

www.faceteam.it

CEEDs · The Collective Experience of Empathic Data SystemsProject number: 258749 Call identifier: FP7-ICT-2009-5

IDIA and FACET conclusions• FACET protocol is able to

evoke different reactions in normally developing and ASDs subjects

• Facial expressions Labeling difficulties in accordance with littirature1

• Thanks to its predictable and stereotyped nature FACE perfectly fits ASDs behavioral attitude

[1] S. Widen and J. Russell, “Children acquire emotion categories gradu-ally,” Cognitive Development, vol. 23, no. 2, pp. 291–312, 2008

© Fondazione ARPA Pictures by Enzo Cei

Multi input analysis method

Complex Social Behavior Analysis

Self reports

psycho-physiological

signals

Therapist behavioral

annotations

• 2 Therapists in the control room annotate separately subject’s conversation, answers and relevant actions

• The therapist in the FACET room quick annotates relevant subjects actions

• The three therapist use FACET videos to identify phases time references.

• Videos are used to annotate subjects answers to facial expressions labeling tasks

• Videos are used to annotate subjects reactions to shared attention task and conversations with FACE and psychologist

• Multi parameter comparison allows to infer complex subject behaviors and reactions

Physiological signal analysis: ECG• ECG:

– Moving average– QRS identification through Pan-Tompkins

algorithm and R peak extraction– RR intervals (tachogram) calculation

• EDR:– Moving average for trend extraction (tonic

component)– De-trend (Only phasic component is considered)– Low passed at 2 and 0.2 Hz (two filtered signals

are obtained)

FACET platform