action recognition (thesis presentation)
TRANSCRIPT
Human action recognition using spatio-temporal features
Nikhil Sawant(2007MCS2899)
Guide : Dr. K.K. Biswas
Human activity recognitionH
ighe
r res
oluti
on
Longer Time Scale
Courtesy : Y. Ke, Fathi and Mori, Bobick and Davis, Schuldt et al, Leibe et al, Vaswani et al.
Pose Estimation
Action Recognition
Action Classification
Tracking
Activity Recognition
Use Action recognition?
• Video surveillance• Interactive environment• Video classification & indexing• Movie search• Assisted Care• Sports annotation
Goals….
• Action recognition against the stable background• Action classification • Event detection• Scale invariant action recognition• Resistant to change in view upto certain degrees
Goals….
• Action recognition against the stable background• Action classification • Event detection• Scale invariant action recognition• Resistant to change in view upto certain degrees
• Action recognition in cluttered background• Action detection invariant of speed
Existing Approaches
• Tracking interest points
• Flow based Approaches
• Shape based Approaches
Tracking interest points
Images Courtesy : P. Correra
Tracking 5 crucial points i.e. Head, 2 hands, 2 feet. Mostly present at the local maxima on the plot of geodesic distance
• Use of Moving light displays (MLDs) by Johansson in 1973– Not feasible for as additional constraints are added
• Use of silhouette and geodesic distance by P. Correra
Tracking interest points
• Use of Moving light displays (MLDs) by Johansson in 1973– Not feasible for as additional constraints are added
• Use of silhouette and geodesic distance by P. Correra– It is difficult to track all the Crucial points all the time– Occlusion creates problem in tracking– Complex actions involving occlusion of body parts are
difficult to track– Results depend on the quality of the silhourtte
Flow based approaches
• Action recognition is done by making use of flow generated by motion– Use of optical flows– Spatio-temporal features– Spatio-temporal regularity based features
Shape based Approaches
• Blank et. al. shown Action can be describe as space time shape– Use of possion equation for features– Local space time saliency– Action dynamics– Shape structure and orientation
Images Courtesy : M. Blank
Our Approach• flow based features + shaped based features• spatio-temporal features• Viola-Jones type rectangular features• Adaboost
• STEPS:-– Target Localization – Background subtraction– Local oriented histogram– Formation of descriptor– Use of Adaboost for learning
Optical flow and motion features
Target Localization
• Possible search space is xyt cube• Action needs to be localized in space and time• Target localization helps reducing search space• Background subtraction• ROI marked
Original Video Silhouette Original Video with ROI marked
Motion estimation
• Make use of optical flows for motion estimation• Optical flow is the pattern of relative motion
between the object/object feature points and the viewer/camera
• Several methods : motion compensation encoding, object segmentation, etc
• We make use of Lucas – Kanade, two frame differential method
• Opencv implementation used
Noise removal• Presence of noisy optical flows• Noise removal by averaging• Optical flows with magnitude > C * Omean are
ignored, where C – constant [1.5 - 2], Omean - mean of optical flow within ROI
Noisy Optical flows After noise removal
Organizing optical flow
• Local oriented Histogram
• Weighted averaging
Organizing optical flow(Local oriented Histogram)
• We fix XDIV x YDIV grid around ROI
• On(u, v) is considered in bij if xi < u < xi+1
yj < v < yi+1
• Obij = Σ On(u, v) / Σ 1Such that, xi < u < xi+1
yj < v < yi+1
for all i < XDIV & j < YDIV
• Membership of the optical flows should be inversely proportional to their distance from the centre
Organizing optical flow(Local oriented Histogram)
C(0,0)d2
d1
O1
O2
Oe
Oe
Organizing optical flow(Weighted Averaging)
• Oj = (O1, O2,…..Om)
such that for all i Є {1,....,N}
Organizing optical flows
Formation of motion descriptor
• Optical flow is represented in xy component form
• Effective optical flow from each box is written in a single row as[Oex00, Oey00, Oex10, Oey10,….. ] vector
• Vectors for each action are stored for every training subject
• Adaboost is used to learn the patterns
Learning with Adaboost
Strong classifier
Weak classifierWeight
Featuresvector
Classification Example taken from Antonio Torralba @MIT
Weak learners from the family of lines
h => p(error) = 0.5 it is at chance
Each data point has
a class label:
wt =1and a weight:
+ 1 ( )
-1 ( )yt =
This one seems to be the best
This is a ‘weak classifier’: It performs slightly better than chance.
Classification Example
Each data point has
a class label:
wt =1and a weight:
+ 1 ( )
-1 ( )yt =
We set a new problem for which the previous weak classifier performs at chance again
Each data point has
a class label:
wt wt exp{-yt Ht}
We update the weights:
+ 1 ( )
- 1 ( )yt =
Classification Example
We set a new problem for which the previous weak classifier performs at chance again
Classification Example
Each data point has
a class label:
wt wt exp{-yt Ht}
We update the weights:
+ 1 ( )
- 1 ( )yt =
We set a new problem for which the previous weak classifier performs at chance again
Classification Example
Each data point has
a class label:
wt wt exp{-yt Ht}
We update the weights:
+ 1 ( )
- 1 ( )yt =
We set a new problem for which the previous weak classifier performs at chance again
Classification Example
Each data point has
a class label:
wt wt exp{-yt Ht}
We update the weights:
+ 1 ( )
- 1 ( )yt =
The strong (non- linear) classifier is built as the combination of all the weak (linear) classifiers.
f1 f2
f3
f4
Classification Example
Our Dataset
• Video resolution 320 x 240 • Stable background
ACTION SUBJECTS VIDEOS
Walking 8 34
Running 8 20
Flying 5 25
Waving 5 25
Pick up 6 24
Stand up 6 48
Sitting down 6 24
Our Dataset (Tennis actions)
• Small tennis dataset
ACTION SUBJECTS VIDEOS
Forehand 3 11
Backhand 3 10
Service 2 9
Training and Testing Dataset• Training and testing data is mutually exclusive• Training and testing subjects are mutually
exclusive• Frames used for training and testing
ACTION TRAINING TESTING
Walking 1184 1710
Running 183 335
Flying 182 373
Waving 198 317
Pick up 111 160
Stand up 128 187
Sitting down 230 282
Classification result (framewise)
• Overall Error : 12.21 %
Walking Running Flying Waving Pick up Sit down Stand up Error
Walking 1644 46 0 17 1 2 3.86%
Running 35 295 3 2 11.94%
Flying 1 2 349 11 9 1 6.43%
Waving 11 8 269 29 15.14%
Pick up 8 7 1 120 23 1 25%
Sit down 1 1 26 179 14.97%
Stand up 23 282 8.15%
Classification results (clipwise)
• Overall error : 6.94%
Walking Running Waving1 waving2 bending Sit-down Stand-up Error
Walking 10 0.0%
Running 10 0.0%
Waving1 9 1 10.0%
waving2 10 0.0%
bending 9 1 10.0%
Sit-down 10 0.0%
Stand-up 1 9 10.0%
Action classification
Classification results(Tennis events)
• Overall Error : 19.17% (per frame)
Forehand Backhand Service Error
Forehand 54 7 11 21.95%
Backhand 11 53 10.75%
Service 8 49 14.04%
Event Detection
• Confusion at the junction two actions
• Use of prediction logic
Current frame ‘f’Next n framesPrevious n frames
f f+1 f+2 f+3 f+4… …f-1f-2f-3f-4……f-n f+n
Event Detection
Without using prediction logic With prediction logic
Weizmann DatasetACTION SUBJECTS VIDEOS
Bend 9 9
Jack 9 9
Jump 9 9
Pjump 9 9
Run 9 10
Side 9 9
Skip 9 10
Walk 9 10
Wave1 9 9
Wave2 9 9
Standard Dataset(Weizmann Dataset)
Walk Side Skip
Wave1 Wave2
Bend
Run
Jack Jump Pjump
confusion matrix (framewise)Bend Jack Jump Pjump Run Side Skip Walk Wave1 Wave2
Bend 271 1 1 20 3 30 11
Jack 18 368 8 48 3 2 3 9 16
Jump 9 3 157 8 2 26 19 7
Pjump 36 26 237 22 6
Run 4 2 5 158 3 50 6 1 2
Side 11 9 77 1 1 84 3 58 2 1
Skip 3 9 76 43 5 109 24 1 7
Walk 2 5 16 2 13 5 395
Wave1 47 2 12 238 27
Wave2 30 6 1 4 1 55 269
• Overall Error : 29.17% (per frame)
Weizmann dataset
• Smaller resolution (180 x 144), Previously (320 x 240)
• Weaker motion vectors compare to previous experiments– Weizmann : mag 0 – 1.75 px – Earlier experiment : mag 0 – 5.5 px
• Lack of background frames available, used already given poor quality silhouette
Use of MV + Shape Info(SI)• Only MV are not enough• Shape of the person also gives information
about the actions• No. of foreground pixels in each box
• Error : 23.45%
Use of MV + Differential SI • We calculate Differential Shape Info• Make use of Viola-Jones rectangular features• Rectangular features are used at grid level
rather than pixel level
• Error : 19.69%
confusion matrix (framewise)
Bend Jack Jump Pjump Run Side Skip Walk Wave1 Wave2
Bend 326 7 2 2
Jack 6 418 39 1 3 8
Jump 18 1 189 1 5 4 13
Pjump 11 55 243 6 1 11
Run 2 2 173 2 45 7
Side 8 30 11 1 152 12 33
Skip 1 20 32 83 4 121 13 1 2
Walk 1 1 2 1 1 432
Wave1 43 1 10 10 232 30
Wave2 13 25 328
Spatio-temporal features
TSPAN
TLEN
Spatio-temporal descriptor
• Volume Descriptor in row form– [Frame1 | Frame2 | Frame3 |
Frame4 | Frame5 | ……]
• Motion and Differential shape information for the volume
• Error : 8.472% (per frame)
Event classification (clipwise)bend Jack Jump Pjump Run Side Skip Walk Wave1 Wave2 Error
bend 9 0.0%
Jack 9 0.0%
Jump 9 0.0%
Pjump 9 0.0%
Run 9 1 10.0%
Side 9 0.0%
Skip 10 0.0%
Walk 10 0.0%
Wave1 8 1 11.1%
Wave2 9 0.0%
• Error : 2.15% Better than 12.7% error rate reported by T. Goodhart et.al., Action recognition usign spatio-temporal regularity based feature, 2008
Action recognition in cluttered background
Cluttered environment
• background is not stable• The actor might be occluded• Slight change in camera location (panning)• Scale variation• Speed variation
Training
• Training is done without background subtraction• Manually mark Start and end of action in training
videos• Also the bounding box around the actor is marked• No shape information is added in the training data• Training is done with noisy background• Currently bending and drinking actions supported
Training data drinking
Training data bending
Template length
• bending – – Average no. of frames for action – 55– Variation – 40 – 110– TLEN 45 frames
• Drinking –– Average no. of frames for action – 50– Variation – 35 – 70– TLEN 40 frames
Single template formation• Length of the template kept constant• Some of the frames eliminated
• One action - one template• Adds robustness in the training• Speed variation during training is tackled
1 2 3 4 5 6 7 8 9 10 11 12 13 14 151 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 3 4 5 6 8 9 10 11 13 14 151 2 3 4 5 6 7 8 9 10 11 12
Optical flow and Adaboost
• We have constant length sequences• Optical flows are calculated• No shape information as background is cluttered,
background subtraction not possible• Formation of Spatio-temporal template with
TSPAN = 1 and TLEN = length of sequence = const• Templates are learned with Adaboost.
Testing
• An action Cuboid is formed with specific height, width, length
• Cuboid is moved over each and every valid starting location in the video
Height
Width Length
tx
y
Testing
• An action Cuboid is formed with specific height, width, length
• Cuboid is moved over each and every valid starting location in the video
• A spatio-temporal template is formed for each Cuboid location and tested with Adaboost
• Appropriate entry is made in confidence matrix• Height, width and length updated for scale and
speed invariance
Height
Width Length
Confidence matrix
• Confidence matrix is a 3D matrix• Confidence matrix has an entry for each and
every valid location of cube in the video• Confidence matrix contains the confidence value
given by the Adaboost over various iteration• We expect that true positives will be surrounded
with a dense fog of large confidence values• averaging is done to reduce the effect of the
false positives.
Confidence matrix
Results
Results
Results
Results
Results
Results
Key References• Y. Ke, R. Sukthankar, M. Hebert, “Spatio-temporal Shape and Flow Correlation for Action Recognition”, In Proc.
Visual Surveillance Workshop, 2007.• P. Viola and M. Jones. “Robust real-time face detection”. In ICCV, volume 20(11), pages 1254-1259, 2001.• M. Lucena, J.M. Fuertes and N. P. la Blanca, “Using Optical Flow for Tracking”, Volume 2905/2003, Progress in
Pattern Recognition, Speech and Image Analysis.• Y. Ke, R. Sukthankar, and M. Hebert. “Event detection in crowded videos”. In ICCV, 2007.• F. Niu and M. Abdel-Mottaleb, “View –Invariant Human Activity Recognition Based on Shape and Motion Features,”
in Proc. of the IEEE Sixth International Symposium on Multimedia Software Engineering, pp. 546-556, 2004.• D.M. Gavrila. “The visual analysis of human movement: A survey”. Computer Vision and Image Understanding,
73:82–98, 1999.• D. M. Gavrila. “A bayesian, exemplar-based approach to hierarchical shape matching”. IEEE Trans. Pattern Anal.
Mach. Intell., 29(8):1408–1421, 2007.• K. Gaitanis, P. Correa, and B. Macq, “Human Action Recognition using silhouette based feature extraction and
Dynamic Bayesian Networks”. • M. Ahmad, S. Lee, “Human action recognition using shape and CLG-motion flowfrom multi-viewimage sequences”,
7th IEEE International Conference on Automatic Face and Gesture Recognition, April 2006.• 10. Haritaoglu, D. Harwood, and L. Davis, “W4: real-time surveillance of people and their activities,” IEEE
Transactions on Pattern Analysis and Machine Intelligence 22, pp. 809–830, Aug 2000.• Ismail Haritaoglu, David Harwood, and Larry S. Davis, “W4: Who? When? Where? What? a Real-time System for
Detecting and Tracking People," Proc. the third IEEE International Conference on Automatic Face and Gesture Recognition Nara, Japan , IEEE Computer Society Press, Los Alamitos, Calif., 1998, pp.222-227.
• P. Correa1, J. Czyz1, T. Umeda1, F. Marqu, X. Marichal3, B. Macq, “Silhouette-based probabilistic 2D human motion estimation for real time application”, in ICIP 2005.
• Y. Ke, R. Sukthankar, and M. Hebert. “Efficient visual event detection using volumetric features, In ICCV’05.