1

Assisted Cognition

Henry Kautz, Oren Etzioni, & Dieter FoxUniversity of Washington

Department of Computer Science & Engineering

2

An Epidemic of Alzheimer’s

4.6 million people in the US with Alzheimer’s16 million people by 2050Today costs $100 billion @ year for care

Additional $61 billion in lost productivity from family members

$ ½ Trillion total cost by 2050!Projections even worse for Japan and Europe

3

4

Cognition in ContextCan often compensate for physical disabilities by change in environment

WheelchairsRedesigned appliances

Cognitive competence also depends on environment

Can you cook dinner, given a dead animal, a stone knife, and set of flints?

5

ProblemCaregiver burnout

½ of all family caregivers suffer depression“The 36 Hour Day”

Far too few professional caregivers to provide constant 1-on-1 help in institutional settings

Already a nationwide shortage of good staff

6

Vision

Understanding human behavior from low-level sensory data

Using commonsense knowledgeLearning individual user models

Actively offering prompts and other forms of help as neededAlerting human caregivers when necessary

Computer systems that improve the independence and safety of people suffering from cognitive limitations by…

7

Data Interpretation Food Chain

Movement

Intentions

Behavior

Interventions

8

Model-Based Interpretation

General approach: build a probabilistic model of– Common user goals– Plans (complex behaviors) that achieve those goals

• Feasibility constraints • Temporal constraints• Failure (abnormality) modes

– How simple behaviors are sensed

Run model “backwards” to interpret sensed data

9

Activity Compass Zero-configuration personal

guidance system Learns model of user’s

travel on foot, by public transit, by bike, by car

Predicts user’s next destination, offers proactive help if lost or late

Integrates user data with external constraints Maps, bus schedules,

calendars, … EM approach to clustering

& segmenting data

The Activity Compass Don Patterson, Oren Etzioni, and Henry Kautz (2003)

10

Scenario IJoe gets off the bus on the way to the community center.He can’t remember which way to walk.He consults his Activity Compass. It has predicted his destination based on past behavior, and guides him to the community center.

11

Scenario IIJoe has a regular physical therapist’s appointment at 2 pm.Compass alerts Joe that he will need to leave home in 10 minutes to catch the usual bus there.Joe fails to leave in time. Compass asks whether goal of going to physical therapist is valid.If answer is affirmative, Compass creates alternative plan to get Joe there as soon as possible.

12

Minimalist User Interface

13

Data Source: Elite Care Residences

Elite Care is the realization of our dream to provide a fundamentally different approach to assisted living. Extended Family Residences promote a family lifestyle of close staff-resident relationships, meaningful community-building activities, and physical and mental achievement while providing assistance when needed… Technology helps us maintain our family environment by allowing us to the focus on residents…

- Lydia Lundberg & Bill Reed

14

Activity of daily living monitor & prompter

Foundations of Assisted Cognition Systems. Kautz, Etzioni, Fox, Weld, and Shastri, 2003

15

Recognizing unexpected events using online model selection

User errors, abnormal behavior

Select model that maximizes likelihood of data: Generic model User-specific model Corrupt (impaired) user

model Neurologically-plausible

corruptions Repetition Substitution Stalling

Fox, Kautz, & Shastri (forthcoming)

fill kettleput kettleon stove

fill kettleput kettleon stove

put kettlein closet

16

Technical Foundations

Hidden Markov models– Mathematical framework for describing processes

with hidden state that must be inferred from observations

Hierarchical plan networks– Represents how a task can be broken down into

subtasksHierarchical hidden Markov models*– Key to climbing food-chain!

* Precisely speaking: factorial hierarchical hidden semi-Markov models

17

General Architecture

Ubiquitous computing infrastructureSensors – position, motion, sound, visionOutput – speech, graphics, robotsPortable wireless computing devices