Activity Recognition Using Commonsense

Reasoning

Pallavi Kaushik and Emmanuel Munguia Tapia

The pain

• The aging

–Potential loss of independence

–Fear

–Extraordinary costs

• The caregivers

–Difficult decisions

–Family disruption

–Extraordinary costs (money and time)

• Traditional medical system often of little help

Limited options

• Aging parent moves in with family

– Tensions

– Space/renovations

– Cost

• Family moves in with aging parent

– Rarely practical

• Home care

• Assisted living

All options exacerbated by distance

The extraordinary cost of care…Nursing

homes

Assisted

living

facilities

Independent

living

facilities

Home

care, 1

visit/day

Adult day

care

Emergency

response

service

Medication

reminder

service

Yearly $41,975 $13,140 -

$64,970

$13,140 -

$64,970

$31,025 $3,650 -

$18,250

$12,775 -

$18,250

$250

One vision: ActivityLink

Providing peace of mind communication

Gadgets for peace of mind?

The single aging parent

… and then neighbor concerns

• Neighbor tells adult son that she hasn’t

noticed his mother gardening lately, and son

sees that garden is untended

Dignified piece of mind: a

standout

ActivityLink available at home

stores

Hundreds of sensors: a kit of

parts

Just a few hundred dollars and $20 a month

Installation elder’s home

Stick-ems easy, wireless …

everyone helps.

Sensors installed in home of

adult children

Devices custom & invisible

Install in just a few hours

System models overall activity

Softwaredetects

“important”

(3-ring)changes

Change detected

“Something’s changed … give her a call”

System lifestyle benefits

• Conversations don’t always start with…

– “Are you still seeing friends?”

– “Are you still taking your medication?”

– “Are you still getting exercise?”

• Easy install, no maintenance

• Sensors “invisible”: not stigmatizing

• Batteries replaced casually, once a year

• Both households have system and use it

Infrastructure enables other

applications

• Fun

• Communication

• New proactive applications for keeping people:

– Active

– Mindful

– Empowered

Current work: proactive health

• Switch/bend sensors

– Doors

– Cabinets

– Drawers

– Thresholds

– Appliances

– Objects

• Wearable sensors

– Accelerometers

– Heart rate monitor

– Self report

• Multi-purpose sensors

– People-locator tags

– Auditory sensors

– Optical sensors

Activity recognition

Eating meals

Talking

Sleeping patterns

Taking medications

Cleaning

Cooking

…

newML algorithms

Detect change in activity;Motivate behavior changes

healthapplications

Best bet: link advice with activity

• Simple messages

• Right time

• Right place

• Non-disruptive

• Big impact

– 20% shown for energy

– Substantial gains for preventative medicine

Requires computational sensing

Requires “pixels where you are”

Requires attention to UI design

Objectives

• Include end-user in the system loop

• Use common sense to generate human

models of activity.

• Use these models as prior information in ML

algorithms to reduce number of training

examples required for classification.

• Use common sense reasoning to infer what

objects might be inside drawers and cabinets

Common sense attributes

Conventional Machine Learning

Using Common sense

How does it work?

GUI for generating the activity model

Example models generated (OMICS)

Bathing

shower

spray

cleaner

sponge

water

brush

drain

agent

tub

wash

bathtub

Laundry

clothes

washing

machine

hanger

clothing

laundry

washer

fold

piece

closet

gather

Preparing dinner

pot

microwave

soup

stove

gas

pan

heat

table

Container

refrigerator

timer

(Objects used and probability)

Using Google to extract time and

room information

Room examples

Time probabilities examples

Model->NB classifierClass entering-the-house: Prior probability = 0.16

door: Discrete Estimator. Counts = 10 90 (Total = 100)

cabinet: Discrete Estimator. Counts = 95 5 (Total = 100)

drawer: Discrete Estimator. Counts = 95 5 (Total = 100)

couch: Discrete Estimator. Counts = 95 5 (Total = 100)

sofa: Discrete Estimator. Counts = 95 5 (Total = 100)

table: Discrete Estimator. Counts = 95 5 (Total = 100)

chair: Discrete Estimator. Counts = 95 5 (Total = 100)

light-switch: Discrete Estimator. Counts = 95 5 (Total = 100)

lamp: Discrete Estimator. Counts = 95 5 (Total = 100)

closet: Discrete Estimator. Counts = 95 5 (Total = 100)

window: Discrete Estimator. Counts = 10 90 (Total = 100)

faucet: Discrete Estimator. Counts = 95 5 (Total = 100)

stove: Discrete Estimator. Counts = 95 5 (Total = 100)

trash-can: Discrete Estimator. Counts = 95 5 (Total = 100)

Looking around with Common sense

Testing over real data

System output examples

12:00a 1:15a 2:31a 3:47a 5:03a 6:18a 7:34a 8:50a 10:06a 11:22a 12:37p 1:53p 3:09p 4:25p 5:41p 6:56p 8:12p 9:28p 10:44p 12:00p

92 Office/study Light-sw itch

82 Office/study Draw er

75 Bedroom Draw er

71 Bedroom Draw er

62 Bedroom Draw er

146 Bedroom Draw er

108 Bedroom Light-sw itch

95 Kitchen Light-sw itch

94 Kitchen Burner

91 Kitchen Refrigerator

84 Kitchen Draw er

80 Kitchen Cabinet

78 Kitchen Draw er

73 Kitchen Cabinet

72 Kitchen Cabinet

70 Kitchen Dishw asher

66 Kitchen Cabinet

55 Kitchen Cabinet

54 Kitchen Door

53 Kitchen Cabinet

143 Kitchen Microw ave

137 Kitchen Freezer

135 Kitchen Draw er

120 Kitchen Light-sw itch

105 Kitchen Light-sw itch

140 Foyer Door

104 Foyer Light-sw itch

93 Bathroom Show er-faucet

88 Bathroom Sink-faucet---cold

68 Bathroom Sink-faucet---hot

67 Bathroom Cabinet

58 Bathroom Medicine-cabinet

57 Bathroom Medicine-cabinet

130 Bathroom Door

101 Bathroom Light-sw itch

100 Bathroom Toilet-Flush-

Thursday, 3/27/2003

Thursday, 3/27/2003

12:00a 1:15a 2:31a 3:47a 5:03a 6:18a 7:34a 8:50a 10:06a 11:22a 12:37p 1:53p 3:09p 4:25p 5:41p 6:56p 8:12p 9:28p 10:44p 12:00p

unknown

preparing lunch

preparing dinner

laundry

cleaning

bathing

entering the house

What is difficult about testing the

models over real data?

• Need to map activity models to user activities

one-to-one

laundry->doing laundry

one-to-multiple

preparing meal->preparing breakfast

preparing meal->preparing lunch

preparing meal->preparing dinner

multiple-to-one

cleaning bathroom->cleaning

cleaning kitchen->cleaning

cleaning study->cleaning

How to do it automatically?

What is difficult about testing the

models over real data?

• Need to map model objects to sensor objects

Sensor -> Model

Shower->Faucet

Stood->Chair

Dishwashing liquid->detergent

• Not all objects in house have sensors, particularly small ones

How to propagate probabilities robustly?

Drawer->Objects_Inside

Things we have learned

• Assigning fixed probabilities to used objects is problematic. Improve probs from google or OMICS.

• Google probabilities for time and room are not great.

• Activities such as cleaning are problematic.

• Collapse breakfast, lunch and dinner into a single preparing meal class.

• String matching is a problem

• Similarity functions implemented need to be improved

Future work

• Extract probability of objects from common sense

• Extract binary features over objects e.g. pot_stove, stove_fridge

• Experiment with other ways to extract room and time information

• Explore new ways to test the models and do the activities/objects mapping