thesis research proposal supporting sensor fusion …whd/contextaware/presentationslides.pdf ·...

��

��

��

��

Thesis Research ProposalSupporting Sensor Fusion for

Context-Aware Computing

Thesis Research ProposalThesis Research ProposalSupporting Sensor Fusion for

Context-Aware Computing

Huadong WuThe Robotics Institute

Carnegie Mellon University

��

��

��

��

Sensing hardware: cameras, microphones, etc.Environment situation: people in the meeting room, objects around a moving car, etc.

humans understand

context naturally &effortlessly

Toward Context UnderstandingToward Context Understanding

Identification, representation, and understanding of contextAdapt behavior to context

traditionalsystem

ContextToolkitsystem

Toolkit+ SensorFusion

sensorsensor sensor

sensor

�� !!

��

��

��

Expectations of the Proposed SystemExpectations of the Proposed System

context AI rules

Widget

sensor

Widget

sensor

e.g. Dempster-Shafer Belief Combination

Sensor fusion mediator

Observations & Hypotheses

Dynamic ConfigurationTime interval: T

Sensor listUpdating flag

… …

Expected Performance BoostExpected Performance Boost

1. Uncertainty & ambiguityrepresentation to user applications

2. Information consolidation &conflict resolving for users

3. Adaptive sensor fusion support switch to suitable algorithms

4. Robust to configuration change — and for some to die gracefully

5. Situational description support— using more & complex context

�� ""

��

��

��

What is “Supporting Sensor Fusion” about?

What is “Supporting Sensor Fusion” about?

→ ⋅)( mapping info f

sensory data

ms

s

s

�2

1

nx

x

x

�2

1

context facts/events

SF( )SFSF( )

• Sensor Fusion:

• Supporting Sensor Fusion:

1. Define context status vector & sensory data vector

2. Seek appropriate information mapping function

3. Specify Software architecture and provide software modules/libraries to implement the mapping

�� ##

��

��

��

Network connected ubiquitous computing

What is “Context-Aware Computing” about?

What is “Context-Aware Computing” about?

-car: web-browser and

information center on wheelshouse: intelligent home automation

Smart office

�� $$

��

��

��

Context-Sensing: the Heart of Context-Aware Computing

Context-Sensing: the Heart of Context-Aware Computing

��

��

��

context inform

ation tagging

presen

tation

execu

tion

•Context is any information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and application themselves.

•A system is context-aware if it uses context to provide relevant information and/or services to the user, where relevancy depends on the user’s task

— Dey et al., “Towards a Better Understanding of Context and Context-awareness”, CHI 2000

�� %%

��

��

��

Context-Sensing Frustration & Opportunity Context-Sensing Frustration & Opportunity

sound processing: speaker recognition, speaking understanding

image processing: face recognition, object recognition, 3-D object measureing

location, altitude, speed, orientation

ambient environment

personal physical state: heart rate, respiration rate, blood pressure, blink rate, Galvanic Resistance, body temperature, sweat

microphones cameras, infra-red sensors

GPS, DGPS, serverIP, RFID, gyro, accelerometers, dead-reckoning

network resource, thermometer, barometer, humidity sensor, photo-diode sensors, accelerometers, gas sensor

biometric sensors: heart-rate/blood-pressure/GRS, temperature, respiration, etc., …

information: sensors

space time socialoutside nowactivity schedule

inside history

context

self, family, friends, colleague, acqauintance, etc.

⋅⋅⋅

⋅⋅⋅⋅⋅⋅

=

mnmnn

m

m

n sensor

sensor

sensor

fff

fff

fff

�

�

��

�

�

�2

1

21

22221

11211

2

1

)()()(

)()()(

)()()(

?

?

?

�� &&

��

��

��

Context-Sensing Methodology (1): Sensory Data to Context MappingContext-Sensing Methodology (1): Sensory Data to Context Mapping

context

observations& hypotheses

sensory output

�� ''

��

��

��

moodagitation / tiredness

stress concentration preferencesphysical information

merry, sad, satisfy…

nervousness focus of attention

habits, current name, address, height, weight, fitness, metablism, etc.

inside (personal information, feeling & thinking, emotional)

Starting Point: Context Classification and Modeling

Starting Point: Context Classification and Modeling

work body visionaural (listen/talk)

hands

task ID drive, walk, sit, …

read, watch TV, sight-seeing, …, people: eye contact

content: work, entertainment, living chore, etc., …

type, write, use mouse, etc., …

interruptable…

activity

location proximity time people audiovisualcomputing & connectivity

city, altidude, weather (cloudyness, rain/snow, temperature, humidity, barometer pressure, forecast), location and orientation (absolute, relative to

close to: building (name, structure, facilities, etc. knowledge), room, car, devices (function, states, etc.), …, vicinity temperature, humidity, vibration,

day, date individuals or group (e.g. audience of a show, attendees in a cock-tail party): people interaction, casual chatting, formal meeting, eye contact, attention arousing; non

human talking (information collection), music, etc.; in-sight objects, surrounding scenery

computing environment (processing, memory, I/O, etc., hardware/software resource & cost), network connectivity, communication bandwidth, communication costchange:

travelling, speed, heading,

change: walking/running speed, heading

time of the day: office hour, lunch time, …, season of a year, etc.

interruption source: imcoming calls, encounting, etc., …

noise-level, brightness

history, schedule, expectation

social relationship

outside (environment)

��

��

��

��

• Sensors are highly distributed • System configuration is often highly dynamic in mobile

environment• Economical limitations on sensor selection• Measurement resolution and accuracy requirements be

commensurate to human perception capabilities• Semantic description required

• Multiple sensor modalities and context informationsupport

Sensor Fusion Special RequirementsSensor Fusion Special Requirements

��

��

��

��

Separate Context from Context-Sensing Implementation

Separate Context from Context-Sensing Implementation

Application Application Application

Context Information

Sensor Sensor Sensor

Context Component Architecture

Context Blackboard Architecture

��

��

��

��

Georgia Tech Context-Aware Toolkit System Architecture

Georgia Tech Context-Aware Toolkit System Architecture

WidgetWidget

Service

Sensor Sensor

AggregatorInterpreter

Interpreter

DiscovererContext

Architecture

Application Application

�� ! !

��

��

��

Object Hierarchy and Subclass Relationship in Context ToolkitObject Hierarchy and Subclass Relationship in Context Toolkit

BaseObject

Interpreter Discoverer

Aggregator

Widget

Service

�� " "

��

��

��

Georgia Tech Context Toolkit System:Georgia Tech Context Toolkit System:

1. Context specification2. Separation of concerns

and context handling3. Context interpretation4. Transparent distributed

communications5. Constant availability of

context acquisition6. Context storage7. Resource discovery

1. No intrinsic support for sensing uncertainty indication

2. No sensor fusion support

3. Unnecessary difficult to develop further when sensors’ pool is large

4. No common-knowledge or world-modeling context support

BenefitsBenefits LimitationsLimitations

�� # #

��

��

��

1. Complementary: incomplete data to more complete model

2.2. Competitive:Competitive:to reduce uncertaintiesto reduce uncertainties

3. Cooperative (cueing):data A depends on data B

Sensor Fusion TechnologySensor Fusion Technology

Context ToolkitContext Toolkit Needed PropertiesNeeded Properties

Who is there???

�� $ $

��

��

��

Context Toolkit Extension’s SolutionContext Toolkit Extension’s Solution

WidgetWidgetService

Sensor Sensor

AggregatorInterpreter

Interpreter

DiscovererContext

Architecture

Application Application

Situation Abstraction

�� % %

��

��

��

user contextdatabase

appliance

embedded OS

Context-Sensing Methodology (2): Supporting with Context Aggregation

Context-Sensing Methodology (2): Supporting with Context Aggregation

database sever

gatewayInternet

Intranet

appliance

embedded OS

sensor

smart sensor node

sensor

smart sensor node

sensors

applicationssensor fusion

sensor

smart sensor node

appliance

embedded OS

site contextdatabase

context serverhigher-levelsensor fusion

applicationslower-level

sensor fusion

sensors

�� & &

��

��

��

Context-Sensing Methodology (3): Competitive Sensor Fusion SupportContext-Sensing Methodology (3):

Competitive Sensor Fusion Support

context

observations& hypotheses

sensory output

�� ' '

��

��

��

Why Competitive Sensor Fusion? Why Dempster-Shafer Theory?

Why Competitive Sensor Fusion? Why Dempster-Shafer Theory?

#1. complementary

#2. competitive

#3. cooperativeParametric template,Figures of merit,Syntactic pattern recognition… …

Logical templateAI rule-based reasoning,Heuristic inferenceNeural network… …

Local minimal problem, results cannot be easily explained, not suitable for dynamic

configuration of sensors

It doesn’t make sense that a person is assigned as “0.6 membership of user A”, “0.7

membership of user B”, and “0.9 membership of either user A or B”

“cannot distinguish between lack of belief and disbelief”, cannot address a problem like “its

likely either user A or user B”

Though big improvement over Classic Inference method, still not powerful enough to

reason at fine granularity

Priori knowledge and pdf are required to combine multiple sensor outputs, priori

assessments are not used, do not have enough reasoning power

Flexible, powerful, no pdf needed, cheap computational cost in classification process

No pdf required, very cheap in computation

Likelihood of a hypothesis is updated using a previous likelihood estimation and additional

evidence

Associate pdf with confidence estimation, and provide a way to predict the result probabilities

of their boolean combinations

Sensor i: Pi( x detected | x appeared )Simple & effective for “x vs. ¬x” problems

Fuzzy Logic

Neural Network

Bayesian Network

Voting Fusion

Classic Inference

��

��

��

��

Competitive Sensor Fusion with Dempster-Shafer Theory

Competitive Sensor Fusion with Dempster-Shafer Theory

•Frame of discernment : {{user A}, {user B}, {user A or B}, {neither user A or B}}

•

•

• Updated belief

∑Θ⊆

=E

EmABelief )()(

∑=∩

−=−=φAE

EmABeliefAtyPlausibili )(1)(1)(

∑

∑=⊕

=∩

=∩

−φjEiA

ji

AjEiAji

EmAm

EmAm

Amm)()(1

)()(

2121

21

)(

��

��

��

��

Competitive Sensor Fusion Architecture Support: Information Updating

Competitive Sensor Fusion Architecture Support: Information Updating

Widget

sensor

Widget

sensor

context AI rules

Dempster-Shafer Belief Combination



lower-levelsensor fusion

algorithm selection

Dynamic ConfigurationTime interval: T

Sensor listUpdating flag

… …

��

��

��

��

DynamicContext

Database

System Architecture to Support Sensor Fusion

System Architecture to Support Sensor Fusion

1. Support sensor fusion with information mapping

2. Facilitate sensor fusion: applying AI more easily

3. Further separate context & context acquisition: sensors’ dynamic configuration

4. Easier to use complex context

user-mobile computer

site context database server

site context server

Widget

sensor

Widget

sensor

Widget

sensor

Widget

sensor

SF mediator

Aggregator

SF mediator

Aggregator

context data

ResourceRegistry

context data

ResourceRegistry

OtherAI algorithms

Interpreter

application application

OtherAI algorithms

Interpreter

AI algorithms

Performance Boost

Performance Boost

Discoverer

Dempster-Shafer rule

�� !�!

��

��

��

Primary Concept-Demonstration System

Primary Concept-Demonstration System

camera

infrared camera

motion detector

microphone

fingerprint reader

microphone

IR/RF badge

�� "�"

��

��

��

……

Preference-table-user[Hd]

Preference

144 lb ( = 4 lb)Weight

5’6” ( = 0.5” )Height

Huadong Wu ( =1.0)Name


……


Preference

144 lb ( = 4 lb)Weight

5’6” ( = 0.5” )Height

Huadong Wu ( =1.0)Name

Background-table-user[Hd]

……


Preference

NSH 41029:06AM-10:55AM

Place Time

Huadong WuName

History-table-user[Hd]

Primary Demo System: Context Information Architecture

Primary Demo System: Context Information Architecture

6 ( > 0.5)Detected people #

User-tableDetected users

Current

Device-tableDevices

60 db ( = 6 db)Noise level

Brightness gradeLight condition

72 ºF ( = 3 ºF)Temperature

Area-tableArea

Room-table: NSH A417

User-tableDetected user

……


Device-tableDevices




NSH A417Of room

Inside Area

History-table-user[Chris]

10:45AM, 06/06/2001

Activity-table-user[Chris]

[0.4, 0.9]InsideBackground-table-user[Chris]

Chris

…

Background-table-user[Alan]

Background-table-user[Mel]

Background-table-user[Hd]

Background

…

History-table-user[Alan]

History-table-user[Mel]

History-table-user[Hd]

history

…

2:48PM, 06/06/2001

11:48AM, 06/06/2001

10:32AM, 06/06/2001

First detected

…

Activity-table-user[Alan]

Activity-table-user[Mel]

Activity-table-user[Hd]

Activity

…

Inside

Entrance

Entrance

Place

……

[0.9, 0.98]Alan

[0.3, 0.7]Mel

[0.5, 0.9]Hd

ConfidenceName

User-table

……

User-tableDetected user


Device-tableDevices




NSH A417Of room

Entrance Area

�� #�#

��

��

��

Primary Demo System: from Sensor Observation to Context InformationPrimary Demo System: from Sensor Observation to Context Information

…

2

2

2

0

0

0

0

Four UsersABCD, …

2+

2

2

1

1

0

0

Three UsersABC, …

2+

2

2

1

1

0

0

Two UsersAB, AC, …

2+22+1FR + IC

2+121Active Badge

2222Camera

22+21Infrared Camera (IC)

2+12+0Fingerprint Reader (FR)

1000Microphones

0012Motion Detector

Single UserA? B? C? …

Number of People

People There?

Moving Objects

−−−

−−−−−−

=

))(,),2(),((

))(,),2(),((

))(,),2(),((

)(

)(

)(

2212

1111

2

1

δδδ

δδδδδδ

qtststsf

ptststsf

ltststsf

ts

ts

ts

nmnmnmm

nnn

nnn

nm

n

n

�

�

�

�

�

→

⋅

n

f

m x

x

x

s

s

s

��2

1

)( mappingn informatio2

1

�� $�$

��

��

��

Primary Demo System: System Implementation

Primary Demo System: System Implementation

context database

site contextdatabase

sever

thermometer

smart sensornode

gateway

Internet

appliancesembedded OS

LAN

sensor fusionsignal processing

higher-levelsensor fusionweb server

database server

low-levelsensor fusion

systemmaintenance

mic

motiondetector

IR camera (future expansion)

camera

fingerprintreader

lightscontrol(future expansion)

applications

applications

wearablesensors

(wireless connection)

site contextsever

user mobilecomputer

�� %�%

��

��

��

Secondary Concept-demo SystemSecondary Concept-demo System

Widget

sensor

Widget

sensor

context AI rules

Dempster-Shafer Belief Combination



lower-levelsensor fusion

algorithm selection

Time interval: TSensor list

Updating flag… …

Local Vehicle world Map (LVM)

�� &�&

��

��

��

Secondary Demo System: Context and Jay Gowdy’s Sensor Fusion ArchitectureSecondary Demo System: Context and

Jay Gowdy’s Sensor Fusion Architecture

Hypothesis Pool

Hypothesis Generator

unused observations

new hypotheses

Sensor Fusion ProcessDEM GUI

Radar Lane Tracker Laser Striper

objects lanes objects, boundaries

Sensor Fusion ProcessSensor Fusion Process1. Sensors produce observations2. Observations fed to “Hypothesis Pool”3. Hypotheses strengthen with new data,

weaken in absence of data4. Unused observations generate new

hypotheses

Context InformationContext Information• Hypotheses from sensor fusion

(sensed objects)• Vehicle’s status (pose, speed, etc.) • Environment (downtown vs. highway,

road conditions, etc.)• Other, such as weather, time, driver’s

physiology status, etc.

�� '�'

��

��

��

Secondary Demo System: Using “Additional” context informationSecondary Demo System: Using “Additional” context information

Environment conditions+ Vehicle speed constrains

To detect erroneous object-number assignment

To kick out outline points before doing Kalman filtering

�� !�!�

��

��

��

Working PlanWorking Plan� By May 15, 2001: building system computational architecture� Digest Georgia Tech’s Context-aware Toolkit, set up a context-aware system architecture

using the Context-aware Toolkit, make sure system components communicate properly� Set up database server and web server services, preliminary experiments on context

information repository mechanism

� By December 15, 2001: sensor network development� Choose sensors or do software simulation experiments on simple-case sensory row data

to context information mapping: function-enhanced widgets programming� Adopt or evaluate the IEEE 1451 smart sensing network standards on some sensors� Create or adjust context information repository mechanism using dynamic context

database services

� By May 15, 2002: sensor fusion study� Context taxonomy and knowledge presentation study, sensor fusion mediator programming � Implement context information structure model for given application scenario, stipulate and

program sensor-to-context mapping mechanism: test and improve widgets & the sensor fusion mediator

� Programming to implement Dempster-Shafer evidence combination mechanism for sensor fusion mediator

� By December 15, 2002: system refinement and evaluation� System architecture refinement and evaluation study, given the application scenario,

evaluate performance improvement with respect to sensor fusion and system scalability � System documentation, PhD thesis and defense

�� ! !

��

��

��

• Define context architecture — context description model about the real world

• Mapping sensory data into the context model with system architecture support

Sensor Fusion for Context-Aware Computing: Summary

Sensor Fusion for Context-Aware Computing: Summary

�� !�!�

��

��

��

The Big Picture: How the Idea Works?The Big Picture: How the Idea Works?C

ompl

emen

tary

Sen

sor

Fus

ion

Coo

pera

tive

Sen

sor

Fus

ion

Com

peti

tive

Sen

sor

Fus

ion

Temporal dimensionP

hysi

cal d

imen

sion

fuse multiple measurements from the same sensor

--- at lower-level

fuse multiple sensors’ output (with Dempster-Shafer theory)

--- at higher-level

�� !!!!

��

��

��

• Demonstrate that under the guidance of well-defined context information architecture, synergistic interactions between sensor fusion and context information can greatly support and promote each other

• Propose and demonstrate the idea of supporting generalize-able sensor fusion processes in translating sensory data to semantic representations in an information structure

• Improve the system performance and adaptability of the quite well acclaimed Georgia Tech Context Toolkit systems

Expected ContributionsExpected Contributions

thesis research proposal supporting sensor fusion …whd/contextaware/presentationslides.pdf ·...

Documents