field informatics human sensing accms, kyoto university yuichi nakamura 1/46 copyright (c) 2010...

Field InformaticsHuman Sensing

ACCMS, Kyoto University

Yuichi Nakamura

1/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.

Introduction to Field Informatics Chapter4

Human Sensing:Measuring Human Activities and Social Actions

• Human Activities– simple activities

• walking, eating, house keeping, etc.• simple tasks in daily life

– social actions• conversation, meeting, lectures, etc.• tasks with other people

– others• peoples in a panic

• Measuring what and how?• How store and retrieve the data?


Overview: Purpose of Human Sensing

(a) External information– who did what, how,....

(b) Internal information– thought, intention, feeling,...– physiological conditions

(c) Communication– communicated information– communication intention


Overview: Information on Humans

(a) External information– bodily movements, behaviors, ...– bodily characteristics

(appearance ， sweat ， smell ， etc.)(b) Internal information

– psychological conditions (tension ， fear ， emotion ， comfort/discomfort ， etc.)

– physiological conditions (table 4-3)(c) Communication information

– verbal/non-verbal communication– interpersonal contact, interpersonal

distance, mutual interaction with group


Overview: Tools for Human Sensing

(a) External information– physical sensors– esp. non-invasive, non-intrusive

sensors

(b) Internal information– physiological sensors– brain measurements– introspection, reflection

(c) Communication information– physical sensors– ethnography, ethnomethodology


Information on Humans (examples)

(a) External conditions– bodily movements, behaviors, ...– bodily characteristics

(appearance ， sweat ， smell ， etc.)

(b) Internal conditions– psychological conditions (strain ， fear ，

emotion ， pleasant ， etc.)– physiological conditions

(c) Communication conditions– verbal communication– non-verbal communication


Human positions and movements

• image sensors

• magnetic sensors• ultrasonic wave sensors• RFID


Human Position

• Image sensors– real-time tracking– frequently used for

security and surveillance purpose

• Fish-eye lens and omni-directional cameras– omni-directional– low spatial resolution


Face Detection

• Image sensors with image recognition software.– face detection– face identification

• Many embedded system, e.g., digital camera.


3D Measurements

• Stereo Vision

Multiple Stereo Video CameraStereo Still Camera


Real-time Stereo Machine

• 1995 Carnegie Mellon University


Magnetic Sensors (Motion Capture)

• Comparing with image sensors– better accuracy– no occlusion effect

• Characteristics– ▲ cost, size– × non-intrusive– ◎ accuracy– ◎ occlusion– ◎ lighting– ×other constrains (affected by metal)


Eye Tracking, Gaze Tracking

• Head mount type– measuring eye ball

direction by projecting infrared light

• Table mount type– measuring the pupil

position by video camera(s)


Eye/Gaze Tracking

• Gazing properties– a sequence of fixations– order, duration– movements, saccades

• Internal conditions– intention, attention,

interest

• Object’s characterisitics– features– characteristics




(appearance ， sweat ， smell ， etc.)(b) Internal conditions


– physiological conditions (table 4-3)(c) Communication conditions




Physiological Conditions (examples)

• electrocardiogram (ECG), heart rate, blood pressure, pulse pressure, O2/CO2 concentration in the blood

• breathing rate ， O2/CO2 concentration in breath

• electrooculogram (EOG) ， blink ， pupil size ， focus

• electromyography (EMG) ， evoked electromyography

• skin potential activity, flicker value, body temperature, facial skin temperature, perspiration ， etc.

• electroencephalogram (EEG), magnetoencephalograpy (MEG), functional mgnetic resonance imaging (fMRI) ，near infrared spectroscoping topography (NIRS)


Measuring Brain Activity

• electric activities of neurons

• magnetic field caused by electric activities

• blood flow and brain metabolism


Electroencephalogram

• electrical alterations in accordance with neural activity

• small potential changes on the scalp

－

－５０

０

Ｉ１

Ｉ 2

－５０

－１００

Ｉ１

Ｉ 2

-5０

－５０

-


Measuring Method

• 10-20 system ： distances between adjacent electrodes are either 10% or 20% of the total front-back or right-left distance of the skull. 鼻根部

Nasion

後頭結節Inion

左耳右耳


Brain Waves• delta wave δ ： 1 ～ 3Hz • theta wave θ ： 4 ～ 7Hz • alpha wave α ： 8 ～ 13Hz • beta wave β ： 14 ～ 30Hz • gamma wave γ ： 30 ～ 64Hz 　　　　• omega wave ω ： 64 ～ 128Hz • rho wave ρ ： 128-512Hz • sigma wave σ ： 512-1024Hz

sleep

relaxed

active

exited


Near Infra-Red Spectroscopic Topography (NIRS)

• Near infrared light (around 800nm) is projected and traverses the scalp and skull

• Reflectance from the brain are measured on the scalp

• Brain metabolism can be measured by the ratio of oxidized hemoglobin and deoxidized hemoglobin

Near Infrared Light

Brain

Scalp

Skull


MagnetoEncepharoGraph (MEG)

• Magnetic field arising from neural electrical activity

• Large-scale system with high-performance probes

• high temporal resolution

• noise elimination is a serious problem


functional Magnetic Resonance Imaging ( ｆ MRI)

• Brain activity (blood flow, metabolism)

• Magnetic resonance difference between oxidized hemoglobin and deoxidized hemoglobin


ElectroMyoGraphy (EMG)

• Membrane potential changes in muscle contraction

muscle fiber

nerve muscle connection

muscle

motor nerve


surface EMG

AmplifierElectrodes

• Electric potential changes ( 10mV) ≦ on skin surface

• Electrodes and relatively simple electronic circuits

• problem: noise elimination, MU estimation


０．５ sec

1 ｍ v

Multi Channel Measurement


Example of EMG signal

０．５ sec

1 ｍ v

Condition 1

Condition 2

similar motions with different conditions


Subjective or introspective analysis for psychological conditions

• Conversation Analysis

• Protocol Analysis

• Narrative Analysis


Protocol Analysis

Ask a subject to tell anything which comes up to the subject’s mind, and analyze the internal process of the subject’s.

1: Which one?2: I got it!3: Difficult to find, ...4: Hmm, push it, ... really?5: I’m afraid all are gone...


Protocol Analysis

• think aloud method: speak synchronously what the subject thinks during actions

• retrospective report method: explain actions after it is finished

• In both methods, actions are takes in a video or some recording devices, and those data are minutely analyzed afterwards.


ref. Narrative Analysis

• The subject reconstructs real experiences as a personal story

• Originated from narrative therapy• Linear causality is a dominant feature of

narrative structure • A subject is prompted story telling by a

question addressing what to tell.– Type1: as less interruption as possible– Type2: guided by appropriate questions




(appearance ， sweat ， smell ， etc.)(b) Internal conditions


– physiological conditions (table 4-3)(c) Communication conditions




Sensing of Communication Conditions

• Nonverbal information– 70 ～ 80 ％ of information is carried

through nonverbal behaviors

• Various kinds of nonverbal information– attitude, behaviors– body characteristics– perspiration ， smell– clothes, accessories– others


ref. distance communication

• Asynchronous communication– e-mail– Web

• Realtime communication– chat– video conference– distance lecture


Key Points on Human Sensing

• Objective measurements– physical sensors as much

as possible– non-invasive, non-intrusive

sensors• Multiple sensors

– synchronization– large amount of data

• Data handling– indexing– browsing– retrieval


Some examples

• Smart meeting recording

• Lifelog

• Data Browsing


Smart Meeting Recorder• Non-intrusive sensing and recording

– tracking each person from entering room to sitting down

– tracking each person’s face while talking

• Video capturing with typical picture compositions

カメラ制御コンポーネント

観測カメラ：人物の位置検出

制御指令・映像選択コンポーネント

撮影カメラ：首振りカメラによって追跡撮影映像切替器

MPEG エンコード， HDD に録画

pan/tilt 制御

映像を提示• Two types of cameras

– sensing camera– contents capturing

camera• Contents capturing

cameras are guided by the sensing camera.


Sensing camera:detecting participants positions

control, selecting views

capturing camera: tracking and capturing

video switching

MPEG encoding

pan/tilt control with tracking a face


Automatic Editing Examples


Smart Meeting Browser• Smart meeting browser with realtime meeting

capture• Toward realtime meeting support


Lifelog (personal experience log)

video,sound,location,temperature,time,etc.

experiences

large amount of logs

automatic indexingstructure analysisefficient retrieval

browsing

memory aidseducation supportdisability supporthuman factor analysis


• Browsing– skimming

– gathering related actions

used it here

took it herecame into a room

went out a room

Lifelog (Personal View Records)


Browsing

Browsing for indoor activities browsing by related events


Data Indexing

• Meta-data– author, title, date, keywords, etc.– index, tag, ...

• Automated indexing by video, audio, and text processing.

• Examples– XML– MPEG7– ANVIL


ANVIL


Example of Index<?xml version="1.0" encoding="ISO-8859-1"?><annotation> <head> <specification src="C:\Documents and Settings\nakamine\kijou_sagyou.spec" /> <video src="C:\Documents and Settings\nakamine\fish_nagai.mov" /> <info key="coder" type="String"> Our server </info> <bookmark name="scene01" time="73.43333" /> <bookmark name="scene02" time="234.89999" /> (...) <bookmark name="scene07" time="852.73334" /> <bookmark name="scene08" time="900.59998" /> </head> <body> <track name="situation" type="primary"> <el index="0" start="71.13333" end="203.5"> <attribute name="token">scene01</attribute> </el> <el index="1" start="203.5" end="234.33333"> <attribute name="token">show a sample</attribute> </el> <el index="2" start="234.89999" end="294"> <attribute name="token">scene02</attribute> </el>( 以下省略 )
