recognizing c omplex human a ctivities – from top down to bottom up and back
DESCRIPTION
Recognizing C omplex Human A ctivities – From Top Down to Bottom Up and Back. Dr.- Ing . Ulf Blanke Wearable Computing Lab | ETH Zürich Samsung Jul 25, 2014. Vita. 2001-2006 Dipl. (M.Sc.) Informatik , TU Darmstadt 2007-2011 PhD, Multimodal Interactive Systems Group, TU Darmstadt - PowerPoint PPT PresentationTRANSCRIPT
Recognizing Complex Human Activities –From Top Down to Bottom Up and Back
Dr.-Ing. Ulf Blanke Wearable Computing Lab | ETH ZürichSamsung Jul 25, 2014
Vita
2001-2006 Dipl. (M.Sc.) Informatik, TU Darmstadt
2007-2011 PhD, Multimodal Interactive Systems Group, TU Darmstadt- 3y scholarship, German Research Foundation- Prof. Dr. Bernt Schiele
Post-Doc, Max Planck Institut for Informatics, Saarbrücken- Computer Vision and Multimodal Computing
2011-2012 Senior Researcher at AGT International (R&D Division)- Integrated safety and security solutions- Headquarter: Switzerland, R&D: Darmstadt
2012 ... Senior Scientist and Pioneer Fellow at ETH-Z - Wearable Computing Lab, Prof. Gerhard Tröster
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 2
Overview
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 3
Composite Activities- Challenges - Discovering and combining relevant events (LoCA09)- Transfer and recombine relevant events (ISWC10)
Overview of current projects
……
t t
1 2 3 1 2 3
Composite Activities
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 4
time
Recognizing composite activitiesby decomposition into isolated activity events Excellent work addressing isolated activity recognition
Only little work on composite activities
Data from wearable sensors
ChallengesAtomic activity events
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 5
t
1. Inner-class variability and intra-class similarity2. Large corpus of irrelevant and ambiguous data
Screwing
Drilling
Variabilitywithin activity
Similarity acrossdifferent activities
Screwingtime
ChallengesComposite Activities
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 6
1. Variation (duration, irrelevant data, e.g. by interruptions)2. Changing order of underlying activity events
timeOther challenges laterTowards less supervision
Recognizing Composite ActivitiesOne way of doing it
Layer 2 …
…
Layer 3…
…Composite
t
Layer L1
Data
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 7
Compositeactivities
Sensor data
Which low level events are important for composite activities?
Learning(automatic Selection)
walking eating eating
dinnerlunch
Recognizing Composite ActivitiesLearning relevant events
picking up food
Prepfood
Doing dishes
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 8
eating eating
Compositeactivities
Recognizing composite activities by activity spotting feasible?
Activity Spotting
dinnerlunch
Recognizing Composite ActivitiesSpotting and combining relevant events
Sensor data
walkingpicking up food
Prepfood
Doing dishes
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 9
Can we learn distinctive parts of composite activities?
Can we gain computational efficiency by reducing todata important for recognition?
Research QuestionsActivity spotting for composite activities
1
2
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 10
Approach
compositeactivities
Histogram calculationK-means clustering
Joint boosting
Feature-CalculationSensor data
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 11
Low Level Activity Selection (Joint)Boosting
(1)Combinationof low level activities to infer high-level activities
(2) Automatic Selection of most discriminative low level activitiesBoosting (Friedman2000)
(3) Sharing features(i.e. low level activities) across high level activities
JointBoosting (Torralba2004)
+
others
lunch
lunch
dinner
lunch dinner
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 12
Experimental SetupDataset
7 days of a life from a single person (Huynh08) Two layers of annotation
4 high level routines, more than 20 low level activities
Wrist
working working dinner
lunch commutingcommuting
2 acceleration sensors
walkingstandingin line
having a coffee
Lunch
walkingeating
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 13
Experimental SetupFixed Parameters
High-level activities
Low level activities
Sensor data
Doing dishes
Feature-Calculation
K-means clusters
Joint boosting
Mean and Variance - over 0.4s window- on (x,y,z)-acceleration- of pocket and wrist
Histograms- over 30min
window
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 14
Results
80 10 4 (Huynh08)Precision in % 88 83 83 77Recall in % 90 88 84 66Data used in % 74 18 13 100
Cluster Soft Assignments
80 10 486 77 7390 82 8345 5 2
Cluster Hard Assignments
Amount of data used
Recall
Precision
Number of classifiers204060801000
1020
90
80
7060
504030
100
30
01020
9080
7060
5040
100
204080100 60
in %
in %
Number of classifiers
Observed data reduced dramatically at superior performance
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 15
DinnerCommuteLunch
sitting/desk activities (47.24%)
driving car (32.90%),
driving car (21.71%)
Time
walking (99.23%)
sitting / desk activities (97.86%)
walking (96.09%)
driving bike (47.86%) walking (22.51%) picking up food (16.81%)
queuing in line (43.86%) picking up food (14.59%)
driving bike (16.76%)
sitting/desk activities (31.20%)
3664248
291353
Lunch WorkCommuteDinner
Time
Time
Distribution of low level labels for clustering
ResultsWhich low level activities are used?
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 16
Overview
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 19
Composite Activities- Challenges - Discovering and combining relevant events (LoCA09)- Transfer and recombine relevant events (ISWC10)
Overview of current projects
……
t t
1 2 3 1 2 3
Composite activitiesKnowledge transfer
Work on knowledge transfer (Zheng09), (Kasteren10), (Banos12), … different aspects of transferring knowledge
Here: “Partonomy” (Miller&Johnson-Laird76, Tversky90…) Borrowed from object perception relationship between sub-parts
Composite C2Composite C1
New Composite?
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 20
Research Questions
Does a partonomy-approach improve state of the art?
Can we transfer knowledge of activity events to learn and recognize new activities with minimal training?
Can we use composition knowledge to improve recognition of underlying activity events?
For composite activity recognition…
1
2
3
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 21
L2-Composite L2-CompositeLayer 2 …L2-Composite L2-Composite
…
L3-CompositeLayer 3 L3-Composite ……
Layer nLn-Composite
t
Bottom-up “construction” to hierarchy of multiple layers
Layer L1
DataFrom step 1: scores x
Partonomy-Based Activity RecognitionLn-Composite Activity Modeling
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 22
Raw data stream + Segm.
(2) Classifier
(1) Feature Calculation(e.g. mean, var, FFT)*
Sensors
Spotting atomic activitiesPipeline
Central time t of segment Normalized confidences of event classes U per segment s
Groundtruth
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 23
Ln-compositeactivity
x1x0
z0 z1
y
Unary potentials: scores of individual events
Pairwise potentials:Co-occurrence of relevant events (temporal dist. and class of event)
Probability for composite model
The right events, combined at the right time
y y
z0 z0z1 z2 z2z1 z3
x2x1 x2x0 x0 x1 x3
t
Step 2: Ln-Composite Activity ModelingConditional Random Field
L1-activityevents
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 24
Output of Step 2
Before Non-Maximum surpression
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 25
ExperimentsBookshelf Dataset
5 Xsens IMU’s at upper body
10 subjects 6 L2-composite activities
Make back partJoin 2 parts Assemble box …
…L1
L2
… …
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 26
Partonomy vs. single-layer for L2-composites Joint boosting, sliding window Leave-one-subject out cross-validation
Reduction: Single-layer:-
10% Partonomy: -2%
Training samples
Avg.
EER
for a
ll 6
class
es
83%
62%
85%72%
ExperimentsResults on Bookshelf-Dataset
vs.
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 27
Research Questions
Does a partonomy-approach improve state of the art?
Can we transfer knowledge of activity events to learn and recognize new activities with minimal training?
Can we use composition knowledge to improve recognition of underlying activity events?
For composite activity recognition…
1
2
3
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 28
ExperimentsBookshelf Dataset
5 Xsens IMU’s at upper body
10 subjects 6 L2-composite activities
Make back partJoin 2 parts Assemble box …
…L1
L2
… …
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 29
ExperimentsTarget-Dataset Mirror
5 Xsens IMU’s at upper body 6 subjects 10 L1-events, 6 L2-composites and
4 L3-composites
……
prepare backside 2Join 2 parts ……
Finish backpart
Prepare frames 14x
L2
L3
…
…
L1
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 30
Hanging up on wall
Mark Drill Screw Hang up26% 50% 99% 50%
100%
Fix side frame
Mark Drill hammer26% 50% 65%
90 %1 - E
qual
err
or ra
teExperimentsL2-composite activities
With transferred activity event detectorscomposite activity recognition possible with minimal training
71%
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 31
Contribution and Conclusion
Discovering and combining events (LoCA09) automatic discovery of relevant parts using Joint boosting Efficient method, outperforms approach using all data
Transferring and recombining events (ISWC10) Outperforms direct approach Knowledge transfer possible Improves lower level recognition
Ulf Blanke | Recognizing Composite Human Activity | 36
Today…
User Independent, Multi-Modal Spotting of Subtle Arm Actions with Minimal Training Data. G. Bauer, U. Blanke, P. Lukowicz, and B. Schiele. 10th Percom Workshop (ComoRea 2013). IEEE
South by South-East or sitting at the desk. Can orientation be a place?U. Blanke, R. Rehner and B. Schiele, (ISWC 2011). IEEE
Remember and Transfer what you have Learned - Recognizing Composite Activities based on Activity Spotting.U. Blanke and B. Schiele, (ISWC 2010), IEEE.
Towards Human Motion Capturing using Gyroscopeless Orientation Estimation.U. Blanke and B. Schiele, (ISWC 2010), IEEE.
Visualizing Sleeping Trends from Postures.M. Borazio, U. Blanke and K. Van Laerhoven, (ISWC 2010), IEEE.
All for one or one for all? – Combining Heterogeneous Features for Activity Spotting.U. Blanke, B. Schiele, M. Kreil, P. Lukowicz, B. Sick and T. Gruber (CoMoRea in conj. with Percom 2010), IEEE.
An Analysis of Sensor-Oriented vs. Model-Based Activity Recognition.A. Zinnen, U. Blanke and B. Schiele, (ISWC 2009). IEEE.
Daily Routine Recognition through Activity Spotting.U. Blanke and B. Schiele, (LoCA 2009), Springer.
Sensing Location in the Pocket.U. Blanke and B. Schiele, (Ubicomp 2008, adjunct proceedings)
Scalable Recognition of Daily Activities with Wearable Sensors.Tâm. Huynh, U. Blanke and B. Schiele, (LoCA 2007), Springer.
Selected Publications
Overview
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 38
Composite Activities- Challenges - Discovering and combining relevant events (LoCA09)- Transfer and recombine relevant events (ISWC10)
Overview of current projects
……
t t
1 2 3 1 2 3
Collective Crowd behavior
1M visitors
GPS
Supervised projects
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 41
Parkinson’s Disease Activities, travel purposes, places towards less supervision
Past projects
Place detection
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 42
Past projects
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 43
Sleep studies
Co-Supervision of PhD Students
Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 44
Sinziana Mazilu Long-Van Nguyen-Dinh Zack Zhu
Development team (Project Züri Fäscht)
Sascha Negele
Tobias Franke
David Bannach
Robin Guldener
Torben Schnuchel
Enes Poyarez
Dominik Riehm
William Ross
KellyStreich
PhD Students
Thank you for your kind attention.