Recognizing Activities of Daily Living

from Sensor Data

Henry KautzDepartment of Computer Science

University of Rochester

Activity Recognition

Much recent interest in recognizing human activity from heterogeneous sensor data Motion sensors GPS RFID Video

Compelling applications Military / security operations (e.g. ASSIST) Smart homes & offices

Gathering data on indoor activities

Interpreting RFID Data (using Switching HMM)

Gathering Multi-view Video

Interpreting Video

Computing scene statisticsAi = activityOi = objectSi = scene statisticDi = object statisticsRi = RFID label (for training)

Computing object statistics

Gathering data on outdoor activities

Raw GPS

Discovering significant places

Conditional Random Field

Predicting transportation goals

Dynamic Bayesian Network

Issue

Previous work on activity recognition has used a wide variety of probabilistic models for different tasks and kinds of data HMMs, DBNs, CRFs, …

Background knowledge is implicitly encoded in the structure of the models E.g.: Relation between transportation goals,

plans, actions Increasingly difficult to scale & integrate

Markov Logic

Markov logic will provide common modeling language & inference tools, enabling Easier integration of multiple sensors Easier generalization

From one activity at a time to multiple ongoing activities

From one individual to multiple individuals Easier modification of background knowledge

Add / modify library of plans and goals

Example Scenario

John goes into his kitchen (video) He takes out a jug from the refrigerator, and a

bowl from the cabinet (RFID) He leaves his apartment, and walks to a

convenience store (GPS) He returns carrying a box (video) He pours the box into the bowl (accelerometer)

and the contents of the jug (accelerometer & RFID) Why did John leave the apartment? What did he do?

UR Contributions to MURI: Scenario Development & Data Collection

Develop set of activity recognition scenarios of increasing complexity Activities in the home Outdoor activities

Enact and gather sensor data Heterogeneous: GPS, RFID, video, motion, … Intermittent and noisy

Make dataset available to team Including feature sequences extracted from video and

acceleration data Ground truth 1st data set mid-Year One, then ongoing

UR Contributions to MURI: Unified ML Model of Daily Activities

Recast our previous work on recognition using HMMs, DBNs, CRFs in Markov Logic

Integrate and generalize earlier results Year One:

HMM ML Generalize to multiple ongoing activities Handle novel observations using similarity

Representing actions, intentions, and goals Extend ML to include “modal operators” Distinguish beliefs of observer from beliefs of subject Ability to model imperfect agents, whose plans are flawed

From HMMs to ML

Hidden Markov models describe the world as probabilistic state machine

ML encoding of HMM can be relaxed to allow subject to be in multiple states (multiple activities) by making “unique state” constraint soft

: , . ( , ) ( , )w a a i Activity a i Activity a i

From HMMs to ML

Novel observations can be handled by applying background knowledge about similarity: , , . ( , ) ( , )

( , )

w a obj obj Uses a obj Similar obj obj

Uses a obj

Modal Operators Most previous work on probabilistic activity

recognition does not distinguish What system infers is true about the world What the subject believes is true about the world What the system predicts will happen What the subject intends to happen

Modal operators relate agents to attitudes Bel( John, contains(jug, gasoline) )

But system may know jug is empty Goal( John, ignite(jug) )

Knowledge of subject’s goal can drive cooperative system to help subject, or antagonistic system to block user

Semantic Inference Modal operators do not work like ordinary predicates

or logical connectives Modal proof theory is hard to automate

However: Modal operators have well-understood “possible world”

semantics Agent believes P in possible world W iff P is true in all worlds

W’ such that reachable(W,W’) ML’s inference engine works at the semantic level (direct

search over possible worlds) Promising approach: semantic inference for modal

constructs in ML Explicitly model reachability relationships for each attitude

and agent

Idea Alchemy searches over models (truth

assignments) Modal formulas are evaluated over structures

Structure = set of models and accessibility relationships over the models Structures are too big to explicitly search

Modify Alchemy to search over samples drawn from structures

( ( , ), )

. ( , , ) ( , )

Holds Bel agent formula w

w R agent w w Holds formula w