Technologies to Improve Quality of Life for People with Cognitive Disabilities

Henry KautzDepartment of Computer Science

University of RochesterRochester, NY

Assisted Cognition Systems

Henry KautzDepartment of Computer Science

University of RochesterRochester, NY

Vision

• Understanding human behavior from sensor data

• Actively prompting, warning,and advising

• Alerting caregivers as necessary

Computer systems that improve the independence and well-being of people suffering from cognitive disabilities by…

Foundations• Pervasive sensing

• Movement, location, object manipulation, physiology• On user or embedded in the environment

• AI / machine learning• Model everyday life• Adapt to individual's behavior and goals

• Human computer interaction• Keep focus on user• Ease of use

Outline

• Designing an adaptive wayfinding system for cognitively impaired users

• Creating a context-aware task prompting system

• Challenges in research on assisted cognition systems

Outline

• Designing an adaptive wayfinding system for cognitively impaired users– U Washington Computer Science

• Alan Liu, Gaetano Borriello

– U Rochester Computer Science• Henry Kautz

– U Washington Rehabilitation Medicine• Kurt Johnson, Mark Harniss, Pat Brown

• Creating a context-aware task prompting system• Challenges in research on assisted cognition

systems

7

Wayfinding Difficulties

• Alice doesn’t go to new places by herself because even with practice, she still has trouble remembering directions.

• Bob gets lost easily, and cannot use a map to orient himself and figure out where he is or where to go.

• Carol takes a regular route between home and work, but one day a road along the way is closed and she doesn’t know how to adapt.

GPS Systems

• Why aren't regular GPS systems adequate?• "One size fits all" route planning– Minimize distance or time when driving

• "One size fits all" presentation of directions– Map and turn-by-turn directions

• What makes a route safe and easy to follow by a cognitively impaired user?

9

Research Challenge

• Individuals with cognitive impairment can have vastly different abilities and disabilities– Designing for a "universe of one" (Kintsch & DePaula)

• However:– Can we uncover common features that lie behind

successful solutions?– Can we incorporate these features into a general

system that adapts to the individual?

Research Path

• Formative usability studies– Studied direction-following capabilities and priorities– Potential users interacting with prototypes in situ

• Landmark-based directions– Incorporated landmarks into wayfinding– Balancing cognitive effort and attention to environment

• Modeling individual variation– Implemented framework to generate customized and

adaptable directions– Measure impact of adaptation on users

`

Liu et al. ASSETS 2006, PervasiveHealth 2009

Formative Studies• Indoor and outdoor wayfinding using a

"Wizard of Oz" system• Wide variation in user preference and

expectation of directions– Modality (image, audio, text)– Complexity– Timing

• Attention demand for turn-by-turn directions problematic

• Alternative: landmark-based directions

Landmark-Based Directions• Landmark-based directions– Encourage attention to world– Orient user– More natural

• What are features of good landmarks?

• 9 study participants– TBI, stroke, MS, Dev Dis.

A.L. Liu, H. Hile, H. Kautz, G. Borriello, P.A. Brown, M. Harniss, K. Johnson, PervasiveHealth 2009

Landmark Selection

• Leverage collections of geo-tagged photos– Heuristic: more popular photos more useful

• Can augment / warp photos to make them more closely match user's perspective

Results

• 180 total directions given, 150 correctly followed– Much variation between participants

• Features of most successful landmark-based directions:– In perspective– Close– Path-aligned

• Making turns in relation to landmark required more cognitive effort– Turn-based directions should be user-centric

Modeling Individual Variation

• Studies showed how to create a wayfinding system that was usually better for most users

• However: system was still far from optimal for many users

• Can we design a system that adapts to the individual user?

Adaptation as Optimization

• Approach: for each user, learn a function:

• Generating directions = finding optimal solution to a sequential decision problem

• Issue: scarcity of user-specific training data– Solution: use data from all users, but weigh

particular user's data more heavily

Direction Features × Location Features

→ Probability Success in Following Direction

17

Adaptation Study

• 5 participants• Compared performance on– Route generated with generic

model– Route generated with

personalized model• Qualitative result: personalized

routes considered easier– Need larger, longer study to

quantify significance

18

Summary

• Studied wayfinding with real potential users– Learned range of abilities and preferences– Informed technological requirements

• Incorporated landmarks as wayfinding option– Explored landmark dimensions that affect usability

• Proof-of-concept of automatic adaptation– Creating a personal model using data from both

the individual and the larger user base

Outline

• Designing an adaptive wayfinding system for cognitively impaired users

• Creating a context-aware task prompting system– Attention Control Systems: Richard Levinson– University of Rochester: Henry Kautz– VA Palo Alto: Dr. Harriet Zeiner– UC Santa Cruz: Beth Ann Hockey

• Challenges in research on assisted cognition systems

• PEAT: handheld-based scheduling and cueing system for persons with executive function impairment (Attention Control Systems)– Multimedia prompts to start and end tasks– Assists with task (re)scheduling, by reasoning about

deadlines, durations, and priorities

PEAT

Problems with Prompting Systems

• Require input from user whenever a task begins or ends– Cost of use may discourage users

• Prompts may be out of sync– Self-initiation confuses system– Encourages dependency

• No support for contingent prompts– Remind the user to take her walker when she goes

outside

Integrating Sensing and Prompting

• Goal: use information from sensors to improve prompting

• Determine user's current activity from– Location: Home automation motion sensors, GPS– Objects user is handling: Intel Research Seattle RFID

reader bracelet– Schedule of expected activities

Context-Aware PEAT

• R&D funded by grants from DARPA, Office of Defense, and NIH

• Platform: Android cell phone– Internal sensors: GPS, acceleration– Home base station (laptop) for setup

and relaying data from Insteon home automation sensors and Intel Research RFID reader

– State estimator: employs probabilistic algorithms to infer user activity and learn user model

State Estimator

Cellphone

Wireless connectionsGPS

State DB

Daily Activity Planning & Cueing

PEAT User Interface

PEAT

RFID reader bracelet

WiFi Relay

RF

ZigBee

WiFi

Home sensors

User

Reduced Prompting

• Prompting cancelled if user self-initiates (or self-terminates) task– Reduces confusion / out of sync prompts– Encourages, rather than punishes, self-initiation

• Can system do even more to encourage self-initiation?– Perhaps: prompt later than scheduled time– If prompt is too late and user does

not self-initiate, task will fail– What is the optimal time

for system to prompt?

Optimal Prompting

• System can be configured to use a fixed prompting schedule (e.g. 5 minutes before scheduled time) or an adaptive schedule

• Caregiver specifies rewards associated with– Successful (timely) task completion– User self-initiation of task

• System– Learns model of user self-initiation– Computes prompting time that maximizes expected

reward

Learning the User Model

• User model includes probability distribution over how long it takes user to self-initiate– Technical challenge: how to estimate this

parameter quickly and accurately?– Can use statistical techniques developed for

reliability analysis

System Trial

• In-home trials by Palo Alto VA start next month

• Help regularize patients' daily schedule for– Sleep– Meals– Exercise– Behavioral therapy

"homework"

Outline

• Designing an adaptive wayfinding system for cognitively impaired users

• Creating a context-aware task prompting system

• Challenges in research on assisted cognition systems

Point versus General Solutions

• “[A] tension exists between satisfying the need for customized solutions that address the immediate needs of users, and identifying generalizable results that can be used to achieve long-term research goals.”

[Generalizability in Research with Cognitively Impaired Individuals, Moffatt, Findlater, Allen, 2006]

Evaluating with a Universe of One

• “Twenty-eight papers presenting 25 studies were reviewed. The total number of participants was 423. Most identified papers described case reports or non-randomized clinical trials. Only one randomized controlled trial was identified.”

[Efficacy and usability of assistive technology for patients with cognitive deficits: a systematic review,

Joode, Heugten, Verhey, & Boxtel, 2010]

The Training Hurdle

• “The [studies] that do describe the exact time spent on training show a substantial amount of variation (from 30 minutes up to 9 hours).”

[Efficacy and usability of assistive technology for patients with cognitive deficits: a systematic review,

Joode, Heugten, Verhey, & Boxtel, 2010]

Promise of Context-Aware Systems

Medication Prompting:• “The context-based prompting resulted in significantly

better adherence (92.3%) as compared to time-based (73.5%) or no prompting (68.1%) conditions (p < 0.0002, χ2 = 17.0). In addition, subjects had better adherence in the morning than in the evening. We have shown in this study that a system that generates reminders at an opportune time to take the medication significantly improves adherence.”

[A Study of Medication-Taking and Unobtrusive, Intelligent Reminding, Hayes, Cobbinah, Dishongh, Kaye, Kimel, Labhard, Leen, Lundell,

Ozertem, Pavel, Philipose, Rhodes, & Vurgun, 2009]

Pitfalls of Context-Aware Systems

• “The participant was building a mental model of how the system behaved, and his model did not map well onto the behavior of the system. The problem was especially pronounced when an activity was not detected properly. ... The participant further admitted to changing his behavior in order to ‘fool‘ the system, but since he didn’t understand how it worked, this behavior was counter-productive.

[User-adaptive Reminders for Home-based Medical Tasks, Kaushik, Intille, Larson, 2008]