spatio-temporal and context reasoning in smart homes sook-ling (linda) chua stephen marsland, hans...
TRANSCRIPT
Spatio-Temporal and Context Reasoning
in Smart Homes
Sook-Ling (Linda) Chua
Stephen Marsland, Hans W. Guesgen
School of Engineering and Advanced TechnologyMassey University, New Zealand
COSIT 2009
T
The world is aging
- have we noticed?
Source: United Nations (2007)
h e S i t u a t i o n . . .
The populations of the world are aging
Source: United Nations (2007)
T h e S i t u a t i o n . . .
The populations of the world are aging
Source: United Nations (2007)
T h e S i t u a t i o n . . .
The populations of the world are aging
Source: United Nations (2007)
T
< 10%
> 25%
h e S i t u a t i o n . . .
T People choose to stay in their own homes
as long as possible and remain independent
h e S i t u a t i o n . . .
T
Aging
Physical disability Cognitive impairment
diminished sense and touch slower ability to react poor vision, hearing problems memory problems
leads to
People choose to stay in their own homes
as long as possible and remain independent
h e S i t u a t i o n . . .
Supporting inhabitant’s daily activities
T h e S i t u a t i o n . . .
“Smart Homes”
Figure extracted from: http://www.dreamhomesmagazine.com/
T h e S i t u a t i o n . . .
To react intelligently, the smart home needs to:
(1) recognise inhabitant’s behaviour
(2) perform reasoning
spatio-temporal information
contextual information
sensor output
“Tokens”
B
Figure extracted from: http://www.dreamhomesmagazine.com/
The Smart Home
e h a v i o u r R e c o g n i t i o n
“Tokens” The direct representation of current sensor states being triggered
E.g. of a sequence of tokens from the sensors
Date Activation Time Activation Room Object Type Sensor State
16/6/2008 18:05:23 Living room Television Off
16/6/2008 18:08:19 Living room Curtain Closed
16/6/2008 18:09:48 Kitchen Light On
16/6/2008 18:10:35 Kitchen Cupboard Open
16/6/2008 18:25:06 Kitchen Fridge Open
16/6/2008 19:00:02 Laundry Washing Machine
On
B e h a v i o u r R e c o g n i t i o n
“Tokens” The direct representation of current sensor states being triggered
E.g. of a sequence of tokens from the sensors
Date Activation Time Activation Room Object Type Sensor State
16/6/2008 18:05:23 Living room Television Off
16/6/2008 18:08:19 Living room Curtain Closed
16/6/2008 18:09:48 Kitchen Light On
16/6/2008 18:10:35 Kitchen Cupboard Open
16/6/2008 18:25:06 Kitchen Fridge Open
16/6/2008 19:00:02 Laundry Washing Machine
On
B e h a v i o u r R e c o g n i t i o n
12S19th Jan 2009 18:03:16
4S19th Jan 2009 18:07:56
78S19th Jan 2009 18:20:27
101M19th Jan 2009 18:33:44
11S19th Jan 2009 18:50:12
23S19th Jan 2009 19:01:08
5S
19th Jan 2009 19:37:2117M
19th Jan 2009 19:41:26
.
.
.
.
.
.
…
KitchenDining/Living
Room
Bedroom
Laundry
Office/Study
Figure extracted from: The Aware Home, 2002
B e h a v i o u r R e c o g n i t i o n
12S19th Jan 2009 18:03:16
4S19th Jan 2009 18:07:56
78S19th Jan 2009 18:20:27
101M19th Jan 2009 18:33:44
11S19th Jan 2009 18:50:12
23S19th Jan 2009 19:01:08
5S
19th Jan 2009 19:37:2117M
19th Jan 2009 19:41:26
.
.
.
.
.
.
…
KitchenDining/Living
Room
Bedroom
Laundry
Office/Study
Figure extracted from: The Aware Home, 2002
B e h a v i o u r R e c o g n i t i o n
12S19th Jan 2009 18:03:16
4S19th Jan 2009 18:07:56
78S19th Jan 2009 18:20:27
101M19th Jan 2009 18:33:44
11S19th Jan 2009 18:50:12
23S19th Jan 2009 19:01:08
19th Jan 2009 19:37:2117M
19th Jan 2009 19:41:26
.
.
.
.
.
.
…
KitchenDining/Living
Room
Bedroom
Laundry
Office/Study
Q: How do we recognise behaviours?
Figure extracted from: The Aware Home, 2002
B e h a v i o u r R e c o g n i t i o n
B e h a v i o u r R e c o g n i t i o n
KitchenDining/Living
Room
Bedroom
Laundry
Office/Study
12S19th Jan 2009 18:03:16
4S19th Jan 2009 18:07:56
78S19th Jan 2009 18:20:27
101M19th Jan 2009 18:33:44
11S19th Jan 2009 18:50:12
23S19th Jan 2009 19:01:08
19th Jan 2009 19:37:21 17M...
.
.
.
Figure extracted from: The Aware Home, 2002
Challenges:
(a) Exact activities are not directly observed, only the
sensor observations
. . .
B e h a v i o u r R e c o g n i t i o n
KitchenDining/Living
Room
Bedroom
Laundry
Office/Study
12S19th Jan 2009 18:03:16
4S19th Jan 2009 18:07:56
78S19th Jan 2009 18:20:27
101M19th Jan 2009 18:33:44
11S19th Jan 2009 18:50:12
23S19th Jan 2009 19:01:08
19th Jan 2009 19:37:21 17M...
.
.
.
. . .
Figure extracted from: The Aware Home, 2002
Challenges:
(a) Exact activities are not directly observed, only the
sensor observations
B e h a v i o u r R e c o g n i t i o n
KitchenDining/Living
Room
Bedroom
Laundry
Office/Study
12S19th Jan 2009 18:03:16
4S19th Jan 2009 18:07:56
78S19th Jan 2009 18:20:27
101M19th Jan 2009 18:33:44
11S19th Jan 2009 18:50:12
23S19th Jan 2009 19:01:08
19th Jan 2009 19:37:21 17M...
.
.
.
Figure extracted from: The Aware Home, 2002
Challenges:
(a) Exact activities are not directly observed, only the
sensor observations
?
. . .
B e h a v i o u r R e c o g n i t i o n
Challenges:
(b) Same sensor activations will be involved in
multiple behaviours
B e h a v i o u r R e c o g n i t i o n
Challenges:
(b) Same sensor activations will be involved in
multiple behaviours
Figure extracted from: www.rebecca-waring.com, www.cyh.com, www.chow.com
B e h a v i o u r R e c o g n i t i o n
Challenges:
(c) No. of observations can vary between activities
Making breakfast Making dinner
Fridge
Toaster
Cupboard Cupboard
Stove
Microwave Oven
Tap
Drawer
B e h a v i o u r R e c o g n i t i o n
Challenges:(d) Behaviours are rarely identical on each use
E.g. Making a cup of tea
With / without Milk / water first? How long?
components can be present/absent the order of individual components happen can change length of time each piece takes can change
B e h a v i o u r R e c o g n i t i o n
Challenges:(d) Behaviours are rarely identical on each use
E.g. Making a cup of tea
With / without Milk / water first? How long?
components can be present/absent the order of individual components happen can change length of time each piece takes can change
Stochastic Approach
B e h a v i o u r R e c o g n i t i o n
The Hidden Markov Model (HMM)
probabilistic graphical model
Source: Rabiner, L. (1989)
uses probability distributions to determine the
states for a sequence of observations over time
B e h a v i o u r R e c o g n i t i o n
The Hidden Markov Model (HMM)
probabilistic graphical model
Source: Rabiner, L. (1989)
1tO tO 1tOObservations We know this..
uses probability distributions to determine the
states for a sequence of observations over time
B e h a v i o u r R e c o g n i t i o n
The Hidden Markov Model (HMM)
probabilistic graphical model
Source: Rabiner, L. (1989)
1tO tO 1tOObservations We know this..
1tS tS 1tS ……States But, not this
uses probability distributions to determine the
states for a sequence of observations over time
1tS tS 1tS
1tO tO 1tO
…
…States
Observations
Markov property:
The probability of transition to a state (St+1) depends only
on the current state (St) [represented by solid line]
The observation at Ot depends only on the state St
at that time slice [represented by dashed line]
B e h a v i o u r R e c o g n i t i o n
…
The Hidden Markov Model (HMM)
Source: Rabiner, L. (1989)
1tS tS 1tS
1tO tO 1tO
…
…States
Observations
Markov property:
The probability of transition to a state (St+1) depends only
on the current state (St) [represented by solid line]
The observation at Ot depends only on the state St
at that time slice [represented by dashed line]
B e h a v i o u r R e c o g n i t i o n
…
The Hidden Markov Model (HMM)
Source: Rabiner, L. (1989)
1tS tS 1tS
1tO tO 1tO
…
…States
Observations
Markov property:
The probability of transition to a state (St+1) depends only
on the current state (St) [represented by solid line]
The observation at Ot depends only on the state St
at that time slice [represented by dashed line]
B e h a v i o u r R e c o g n i t i o n
…
The Hidden Markov Model (HMM)
Source: Rabiner, L. (1989)
B e h a v i o u r R e c o g n i t i o n
KitchenDining/Living
Room
Bedroom
Laundry
Office/Study12S19th Jan 2009 18:03:16
4S19th Jan 2009 18:07:56
78S19th Jan 2009 18:20:27
101M19th Jan 2009 18:33:44
11S19th Jan 2009 18:50:12
23S19th Jan 2009 19:01:08...
.
.
.
B e h a v i o u r R e c o g n i t i o n
KitchenDining/Living
Room
Bedroom
Laundry
Office/Study12S19th Jan 2009 18:03:16
4S19th Jan 2009 18:07:56
78S19th Jan 2009 18:20:27
101M19th Jan 2009 18:33:44
11S19th Jan 2009 18:50:12
23S19th Jan 2009 19:01:08...
.
.
.
4S12S 78S101MObservations
B e h a v i o u r R e c o g n i t i o n
KitchenDining/Living
Room
Bedroom
Laundry
Office/Study12S19th Jan 2009 18:03:16
4S19th Jan 2009 18:07:56
78S19th Jan 2009 18:20:27
101M19th Jan 2009 18:33:44
11S19th Jan 2009 18:50:12
23S19th Jan 2009 19:01:08...
.
.
.
?
4S12S 78S101MObservations
Cupboard CoffeeMachine
FridgeStates
To use HMM to recognise behaviours:
(1) Segmentation break the token sequence into
appropriate pieces that represent individual
behaviours
1 32 54 6 T. . .7 Observations
start startend end
B e h a v i o u r R e c o g n i t i o n
1 32 54 6 T. . .7
(2) Classification identify the behaviours using
the HMM
Observations
To use HMM to recognise behaviours:
“Behaviour A” “Behaviour B”
B e h a v i o u r R e c o g n i t i o n
Behaviour Recognition using HMM
Our approach:
Use a set of HMMs that each recognise different
behaviours
“Making coffee” “Showering”
These HMMs will compete to explain the current
observations Model selection is based on maximum likelihood
“Making lunch”
. . .
B e h a v i o u r R e c o g n i t i o n
Source: Chua, Marsland and Guesgen (2009)
Experiment: Competition between HMMs
Datasets
MIT PlaceLab Designed a set of simply installed state-change
sensors that were placed in two different apartments
with real people living in them
Source: Tapia (2004)
B e h a v i o u r R e c o g n i t i o n
Experiment: Competition between HMMs
Datasets
The subjects kept a record of their activities that form a
set of annotations for the data
“Ground-truth” segmentation of the dataset
We used the dataset from the first subject
77 sensors collected for 16 consecutive days
B e h a v i o u r R e c o g n i t i o n
Datasets
Activities take place in one room (kitchen) Location of the sensors is known a priori Behaviours:
Prepare breakfast (toaster) Prepare breakfast (cereal) Prepare beverage Prepare lunch Do the laundry
Experiment: Competition between HMMs
B e h a v i o u r R e c o g n i t i o n
Based on 727 observations (using 11 days testing and 5 days training set)
B e h a v i o u r R e c o g n i t i o n
Based on 727 observations (using 11 days testing and 5 days training set)
B e h a v i o u r R e c o g n i t i o n
Based on 727 observations (using 11 days testing and 5 days training set)
B e h a v i o u r R e c o g n i t i o n
Experimental Results
Method works effectively
performs segmentation and detects changes of activities
B e h a v i o u r R e c o g n i t i o n
Microwave Fridge Coffee Machine
DrawerDrawer
1 2 3 4 5 6 7 . . .observation
T
Fridge Cupboard
Experimental Results
Method works effectively
performs segmentation and detects changes of activities
B
Preparing lunch Preparing a beverage
Microwave Fridge Coffee Machine
DrawerDrawer
1 2 3 4 5 6 7 . . .observation
T
Fridge Cupboard
e h a v i o u r R e c o g n i t i o n
Discussion
Lack of spatio-temporal information
Misclassification:
The end of one behaviour contains observations that
should be the start of the next
Microwave Cupboard Fridge Fridge Coffee Machine DrawerDrawer1 2 3 4 5 6 7 …
observation
Preparing lunch Preparing a beverage
T
B e h a v i o u r R e c o g n i t i o n
Discussion
Lack of spatio-temporal information
Misclassification:
The end of one behaviour contains observations that
should be the start of the next
Microwave Cupboard Fridge Fridge Coffee Machine DrawerDrawer1 2 3 4 5 6 7 …
observation
Preparing lunch Preparing a beverage
T
B e h a v i o u r R e c o g n i t i o n
Discussion
Lack of spatio-temporal information
Misclassification:
The end of one behaviour contains observations that
should be the start of the next
Microwave Cupboard Fridge Fridge Coffee Machine DrawerDrawer1 2 3 4 5 6 7 …
observation
Preparing lunch Preparing a beverage
T
B e h a v i o u r R e c o g n i t i o n
Microwave Cupboard Fridge Fridge Coffee Machine DrawerDrawer1 2 3 4 5 6 7 . . .
observation
Preparing lunch Preparing a beverage
T
Preparing lunch Preparing a beverage
Discussion
Lack of spatio-temporal information
Misclassification:
The end of one behaviour contains observations that
should be the start of the next
B e h a v i o u r R e c o g n i t i o n
A: Augment current algorithm to include spatio-temporal information
Q: How to reduce misclassification?
Discussion
Lack of spatio-temporal information
B e h a v i o u r R e c o g n i t i o n
NOT directly interested in the exact coordinates
Spatial information (Where?)
S
So, what are we interested in?
Room location
e.g.
Figures extracted from: www.istockphoto.com, www.clubjam.jammag.com, www.nancilea.blogspot.com
p a t i o - t e m p o r a l
Spatial information (Where?)
S p a t i o - t e m p o r a l
Current study used very basic spatial information
(just the kitchen!)
In the future, . . .
B e d r o o m K i t c h e n
B a t h r o o mD i n i n g R o o m
L i v i n g R o o m
Preparing a beverage
Cooking
Showering
Grooming
Computing
Sleeping
Washing dishes
Eating
Reading
Watching TV
Exercising
Sitting around fireplace
Resting
S p a t i o - t e m p o r a l
B e d r o o m K i t c h e n
B a t h r o o mD i n i n g R o o m
L i v i n g R o o m
Preparing a beverage
Cooking
Showering
Grooming
Computing
Sleeping
Washing dishes
Eating
Reading
Watching TV
Exercising
Sitting around fireplace
Resting
S p a t i o - t e m p o r a l
S p a t i o - t e m p o r a l
. . .
observation
TMicrowave Cupboard FridgeDrawer
1 2 3 4
Preparing lunch
Fridge Coffee Machine
Drawer7 8 9
Preparing a beverage
Fan5
Shower6
Showering
Kitchen Bathroom Kitchen
Spatial information (Where?)
S p a t i o - t e m p o r a l
Spatial information (Where?)
. . . is this sufficient for reasoning?
WITHOUT temporal, the system cannot differentiate:
Bathroom
3 am
8 am
Vs.
Figure extracted from: http://hazard.com/graphics
S p a t i o - t e m p o r a l
Temporal information
When does a behaviour occur?
Source: Guesgen and Marsland (2009)
How long does behaviour take?
How often does behaviour occur?
Mapping to time scale
Duration
Frequency
e.g. Mary vacuums every Sunday
e.g. Microwave used for a dangerously long time
e.g. Peter showers 3 times a day
S p a t i o - t e m p o r a l
Temporal information (When)
3.03 pm
weekends vs. weekdays
winter vs. summer
½ hour after shower
having breakfast 2 hours before meeting
am vs. pm
Absolute time Relative time
.
.
.
.
S p a t i o - t e m p o r a l
Time scales
Yearly (e.g. Christmas, New Year, Easter, etc.)
Weekly (e.g. vacuuming, visit from health worker, etc.)
Daily (e.g. showering, eating, etc.)
S p a t i o - t e m p o r a l
Temporal information
(a) segment the behaviours
(b) generate a sequence of behavioural patterns
tells us when, for how long and how frequent behaviour occurs
Source: Guesgen and Marsland (2009)
S p a t i o - t e m p o r a l
a m p m
E v e n i n g
N i g h t
Preparing a beverage
Cooking
Showering
Grooming Computing
Washing dishes
Eating
Reading
Watching TV
Exercising
Resting
Sleeping
Watching TVCooking
Cooking
Washing dishes
Washing dishes
Computing
Computing
Preparing a beverage
Sitting around fireplace
Exercising
Reading
Reading
Eating
Eating
Preparing a beverage
Showering
S p a t i o - t e m p o r a l
a m p m
E v e n i n g
N i g h t
Preparing a beverage
Cooking
Showering
Grooming Computing
Washing dishes
Eating
Reading
Watching TV
Exercising
Resting
Sleeping
Watching TVCooking
Cooking
Washing dishes
Washing dishes
Computing
Computing
Preparing a beverage
Sitting around fireplace
Exercising
Reading
Reading
Eating
Eating
Preparing a beverage
Showering
S p a t i o - t e m p o r a l
a m p m
E v e n i n g
N i g h t
Preparing a beverage
Cooking
Showering
Grooming Computing
Washing dishes
Eating
Reading
Watching TV
Exercising
Resting
Sleeping
Watching TVCooking
Cooking
Washing dishes
Washing dishes
Computing
Computing
Preparing a beverage
Sitting around fireplace
Exercising
Reading
Reading
Eating
Eating
Preparing a beverage
Showering
S p a t i o - t e m p o r a l
Temporal information
(a) segment the behaviours
(b) generate a sequence of behavioural patterns
tells us when, for how long and how frequent behaviour occurs
S p a t i o - t e m p o r a l
Time
Cooking
Reading
Resting
Preparing a beverage
Washing dishes
Eating
Watching TV
Computing
Exercising
Cooking
Watching TV
Behaviour
Time
Cooking
Reading
Resting
Preparing a beverage
Washing dishes
Eating
Watching TV
Computing
Exercising
Cooking
Watching TV
.
.
.
Behaviour
S p a t i o - t e m p o r a l
Dining Room
Eating
Reading
Kitchen
Preparing a beverage
Cooking
Washing dishes
Living Room
Watching TV
Exercising
Sitting around fireplace
Resting
Space
Competition among HMMs
Bedroom
Grooming
Computing
Sleeping
Time
Cooking
Reading
Resting
Preparing a beverage
Washing dishes
Eating
Watching TV
Computing
Exercising
Cooking
Watching TV
Behaviour
S p a t i o - t e m p o r a l
Space
Competition among HMMs
.
.
.
Dining Room
Eating
Reading
Kitchen
Preparing a beverage
Cooking
Washing dishes
Living Room
Watching TV
Exercising
Sitting around fireplace
Resting
Bedroom
Grooming
Computing
Sleeping
Time
Cooking
Reading
Resting
Preparing a beverage
Washing dishes
Eating
Watching TV
Computing
Exercising
Cooking
Watching TV
Behaviour
S p a t i o - t e m p o r a l
Space
Competition among HMMs
.
.
.
Dining Room
Eating
Reading
Kitchen
Preparing a beverage
Cooking
Washing dishes
Living Room
Watching TV
Exercising
Sitting around fireplace
Resting
Bedroom
Grooming
Computing
Sleeping
Time
Cooking
Reading
Resting
Preparing a beverage
Washing dishes
Eating
Watching TV
Computing
Exercising
Resting
Watching TV
Behaviour
S p a t i o - t e m p o r a l
Space
Competition among HMMs
.
.
.
Dining Room
Eating
Reading
Kitchen
Preparing a beverage
Cooking
Washing dishes
Living Room
Watching TV
Exercising
Sitting around fireplace
Resting
Bedroom
Grooming
Computing
Sleeping
What happens if the person is late one day
and makes lunch at 3 pm?
The system may make mistakes, particularly
with time!
Fuzzy logic system
S p a t i o - t e m p o r a l
C
How was the current situation is reached?
Contextual information
What else is happening?
What is the state of the environment?
. . . needs to be considered!
o n t e x t u a l R e a s o n i n g
C
. . . is this normal?
“ John is boiling water in the middle of the night ”
o n t e x t u a l R e a s o n i n g
C
“ John is boiling water in the middle of the night ”
Spatial: Kitchen
Temporal: Middle of the night
Is the information sufficient for reasoning?
o n t e x t u a l R e a s o n i n g
C
“ John is boiling water in the middle of the night
after watching late night movie ”
Contextual information
Spatial: Living room Kitchen
Temporal: Middle of the night and is Saturday
. . . he stays up longer !!!
o n t e x t u a l R e a s o n i n g
C
Competition between HMMs a possible mechanism for behaviour recognition and segmentation
Spatio-temporal and context awareness play an important role in interpreting behaviour
o n c l u s i o n
A
Stephen Marsland, Hans Guesgen
Massey University Smart Environment
(MUSE) members
School of Engineering and Advanced
Technology (SEAT)
Massey University
c k n o w l e d g e m e n t s
Thank you!(Merçi!)
F i n a l l y . . .