Development Virtual Environments for fMRI Development Virtual Environments for fMRI Socially to Interact with a virtual avatar Socially to Interact with a virtual avatar ; a pilot study; a pilot study

Hyeongrae, Lee

Dept. of Biomedical Engineering, Hanyang University,

Seoul, Korea

IntroductionIntroduction

3

Introduction

• The social virtual environments (VEs)The social virtual environments (VEs)

VEsVEs Virtualavatar

Virtualavatar Social VEsSocial VEs

4

Introduction

Schizophrenia

• The research and therapy using the social VEs The research and therapy using the social VEs

AlcoholismSocial phobia

Social VEsSocial VEsNeuroscienceNeuroscience

5

Displaydevice

Trackingdevice

Tactiledevice

Conventional device

…

Introduction

• The functional MRI environments

MR room

High magnetic and electronic fields

For social VEs, the other method is needed!!!

6

Introduction

•In this study, we purpose that In this study, we purpose that

1.1.development the social VEs for social cognitive neuroscience development the social VEs for social cognitive neuroscience

study using MR compatible devices.study using MR compatible devices.

2.2.a pilot study performed on the developed social VEs.a pilot study performed on the developed social VEs.

MethodsMethods

8

Outside of magnet roomInside of magnet room

Real-timevideo image

Marker detectionon the hand

Computer

Filter panel

Calculatemaker position

Apply hand positionto virtual environments

MR compatible HMD

MR compatible camera

marker

development the social VEs

• fMRI environment for social interactionfMRI environment for social interaction

9

development the social VEs

• Contents for the social VEsContents for the social VEs

(b) Acceptance

(a) Rejection

10

development the social VEs

11

A pilot study A pilot study

Normal groupNormal group

( n = 12 )( n = 12 )

AgeAge 20.92 (SD: 1.78)20.92 (SD: 1.78)

Sex (M:F)Sex (M:F) 5 : 75 : 7

• ParticipantsParticipants

12

A pilot study

• fMRI experiment design

13

A pilot study

• Questionnaires of social realism about the other avatar

(1) How real the other avatar’s action was?(2) Did you feel that you really shook hands with (or was

rejected by) the virtual other ?

• Questionnaires of social cognition about the other avatar

(1) Did you feel positive feeling about the other avatar’s response?

(2) Did you feel negative feeling about the other avatar’s response?

(3) Did you think that other avatar’s response was suitable in the situation?

(4) Did you feel that the other avatar rejected your offer?(5) Did you feel that the other avatar hated you or had

negative emotion about you?(6) Did you feel that the other avatar took threatening

attitude to you?(7) Did you feel that the virtual other ignored or excluded

you?(8) Did you feel that the virtual other was aware of you?

• Questionnaire (5 point response scale )Questionnaire (5 point response scale )

14

A pilot study

• fMRI data acquisitionfMRI data acquisition - The functional MRI conducted with 1.5T machine (GE Medical System).

• BOLD (Blood Oxygenate Level Dependent)

•EPI sequence, Gradient Echo•64x64x30 matrix with 3.75x3.75x5-mm spatial resolution, FOV: 240mm,•Slice thickness: 5mm, TR: 2s, TE: 14.3, FA=90, # of slices: 30

• High resolution anatomical images

•Fast Spoiled Gradient Echo sequence•256x256x116 matrix with 0.94x0.94x1.50-mm spatial resolution, FOV: 240mm,•Slice Thickness: 1.5mm, TR: 8.5s, TE: 1.8s, FA: 12, # of slices: 116).

15

CoregistrationCoregistration

Temporal NormalizationTemporal Normalization

3D construction3D construction

Preprocessing

Preprocessing

Temporal Smoothing(3-point Low pass filter )

Temporal Smoothing(3-point Low pass filter )

Spatial Normalization(MNI N27 templete,

2x2x2mm3)

Spatial Normalization(MNI N27 templete,

2x2x2mm3)

Spatial Smoothing(Gaussian filter, 8 mm

FWHM)

Spatial Smoothing(Gaussian filter, 8 mm

FWHM)

- Analysis of Functional NeuroImages (AFNI, Ver. 2006_06_30_1332)- subjects whose haed movements are more than 1mm are excluded

A pilot study

Statistical Analysis

Statistical Analysis

General Linear ModelAnalysis in each subjectGeneral Linear Model

Analysis in each subject

Group Analysis

Group Analysis

Paired t-test within group(uncorrected P < 0.001)

Paired t-test within group(uncorrected P < 0.001)

• fMRI analysisfMRI analysis

Results & Results & DiscussionsDiscussions

17

- Social realism about other avatars

Results & Discussions

(1) How real the other avatar’s

action

(2) really shake hands with (or

was rejected by) the virtual

other ?

• The questionnaire results.The questionnaire results.

- 1 point : not at all 5 point : very much

18

- social cognition about other avatars

Results & Discussions

• The questionnaire results.The questionnaire results.

(1) positive feeling(2) negative feeling(3) other avatar’s response was

suitable(4) the other avatar rejected

participant offer(5) hated participant or had

negative emotion(6) threatening attitude(7) ignored or excluded participant(8) awareness of participant

- 1 point : not at all 5 point : very much

19

Results & Discussions

uncorrected P <0.001, cluster size > 80 microliter

• Acceptance > Rejection

L thalamus (-11, -17, 8)R insula lobe (37, -3, 0)

• Rejection > Acceptance

L prefrontal gyrus (-17, 53, 20) R rolandic operculum (37, -3, 20)

L superior temporal sulcus (-37, -31, 6)L cerebelum (VI) (-25, -45, -22)

20

Results & Discussions

• Acceptance > Rejection

L thalamus

R insula lobe

• mirroring other’s emotion related to empathy

• Social cognition related friendship and teamwork

21

Results & Discussions

• Rejection > Acceptance

L prefrontal gyrus

L superior temporal sulcus

• understanding other’s mental states• Social emotion (ex. embarrassment)

• Social emotion (ex. embarrassment)

22

In this study, we develop the VEs for social cognitive In this study, we develop the VEs for social cognitive neuroscience study that it is possible to interact with the neuroscience study that it is possible to interact with the virtual avatars. virtual avatars.

This social VEs would provide social environment to This social VEs would provide social environment to patients with deficit in social cognition, such as patients patients with deficit in social cognition, such as patients with schizophrenia, social phobia and etc., even though just with schizophrenia, social phobia and etc., even though just only handshake with other.only handshake with other.

The social interaction methods using maker would be The social interaction methods using maker would be practicable in various social cognitive neuroscience studies.practicable in various social cognitive neuroscience studies.

Summary

23

Prof.Prof.

Ph.D Ph.D candidatecandidate

M.S candidateM.S candidate

Hyeongrae Lee Neuro-mechanism On Virtual Environment Lab.

Dept of Biomedical Engineering In Hanyang University

E-mail: hensgi@bme.hanyang.ac.krHomepage: http://bme.hanyang.ac.kr

FAX: +82-2-2296-5943 Tel: +82-2-2290-0691