histogram of oriented displacements...
TRANSCRIPT
![Page 1: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/1.jpg)
Histogram of Oriented Displacements (HOD): Describing Trajectories of Human
Joints for Action Recognition
Mohammad A. Gowayyed1, Marwan Torki1, Mohamed E. Hussein1, Motaz El-
Sabban2
Advanced Technology Labs Cairo
1 2
![Page 2: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/2.jpg)
Agenda
• Introduction
• Related Work
• Approach
• Experiments
• Conclusion
![Page 3: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/3.jpg)
Human Action Recognition
• Given: video of one or more humans performing an “action”
• Output: action label(what are they doing?)
• Examples of actions:
– Walking
– Running
– Throwing a ball
– Waving
![Page 4: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/4.jpg)
Human Action Recognition
![Page 5: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/5.jpg)
Pose Estimation with Kinect
• [Shotton et al.]* introduced a real-time pose estimation framework using Kinect from a single depth image.
• Perform extensive training on synthetic data
• Provide joint positions at each frame
• We use these joints positions in our recognition approaches
*[Shotton et al.] Real-time human pose recognition in parts from single depth images. In CVPR, 2011.
![Page 6: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/6.jpg)
Pose Estimation with Kinect
![Page 7: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/7.jpg)
Problem Formulation
• Represent a sequence of skeletal joint motions over time using compact, efficient and discriminative descriptor.
• Input – Joints Positions
• XnJoints * nFrames
• YnJoints * nFrames
• ZnJoints * nFrames
• Output – Descriptor to use as an input to a classifier
![Page 8: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/8.jpg)
Agenda
• Introduction
• Related Work
• Approach
• Experiments
• Conclusion
![Page 9: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/9.jpg)
Related Work
Similarity measure Dynamic Temporal Warping
Deal with each frame as a state Recurrent Neural Network
Hidden Markov Model
State-of-the-art:-CVPR 2012
Actionlets Ensemble*
*[Wang et al.] Mining actionlet ensemble for action recognition with depth cameras, In CVPR, 2012.
![Page 10: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/10.jpg)
Agenda
• Introduction
• Related Work
• Approach
• Experiments
• Conclusion
![Page 11: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/11.jpg)
Approach
![Page 12: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/12.jpg)
Histogram of Oriented Displacements (HOD)
• Describe a 2D trajectory using a histogram that records how long the object moved in which range of directions.
• This loses the temporal information.
• We use a temporal pyramid to capture the temporal evolution.
• What about 3D?
– described using the HOD of their 3 2D projections: xy, xz, and yz.
![Page 13: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/13.jpg)
P1 P2
P3
P4
P5 P6 P7
Approach
![Page 14: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/14.jpg)
P1 P2
P3
P4
P5 P6 P7
Approach
![Page 15: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/15.jpg)
P1 P2
P3
P4
P5 P6 P7
Approach
![Page 16: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/16.jpg)
P1 P2
P3
P4
P5 P6 P7
1
8
4
5
2
6
3
7
ϴ
Approach
![Page 17: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/17.jpg)
0
0.5
1
1.5
2
2.5
3
3.5
1 2 3 4 5 6 7 8
Histogram of Oriented Displacements
P1 P2
P3
P4
P5 P6 P7
1
8
4
5
2
6
3
7
ϴ
Magnitude
Angle Quantization Histogram of Oriented Displacements
![Page 18: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/18.jpg)
HOD is speed-invariant*
P1 P2
P3
P4
P5 P6 P7
P1 P3
P5
P7
P8 P9 P10
P2 P4
P6 ≡
*Given that movement is not far from linearity between positions in the lower resolution.
High Speed Low Speed
![Page 19: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/19.jpg)
HOD is scale-invariant*
≡
*Given that the histogram is L2 normalized at the end.
P1 P2
P3
P4
P5 P6 P7
Large Scale Small Scale
P1 P2
P3
P4
P5 P6 P7
![Page 20: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/20.jpg)
Temporal Information
• If we used HOD to just describe the entire trajectory we will lose the temporal information.
• We solve this by applying a temporal pyramid:
– describing it all, halves, and quarters (for 3-level pyramid).
• The final HOD is the concatenation of the all descriptors (7 in case of a 3-level HOD).
![Page 21: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/21.jpg)
Temporal Information
P1 P2
P3
P4
P5 P6 P7
P1 P2
P3
P4 P4
P5 P6 P7
• For a 2-level HOD, the final descriptor is the concatenation of the next three trajectories:
The entire trajectory Second half First half
+ +
![Page 22: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/22.jpg)
Temporal Pyramid
• 3-level HOD
![Page 23: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/23.jpg)
Using HOD for 3D Trajectories
• Our approach is to describe the 3D trajectories by the HOD of their 3 2D projections (xy, yz, and xz).
3D Trajectory
xy projection
xz projection yz
projection
![Page 24: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/24.jpg)
Agenda
• Introduction
• Related Work
• Approach
• Experiments
• Conclusion
![Page 25: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/25.jpg)
Datasets
• MSR-Action3D
– 20 Joints locations are available using a kinect sensor.
– 567 videos.
– Same setup as in *
Action Set 1 Action Set 2 Action Set 3
Horizontal Wave High Wave High Throw
Hammer Hand Catch Forward Kick
Forward Punch Draw X Side Kick
High Throw Draw Tick Jogging
Hand Clap Draw Circle Tennis Swing
Bend Hand Wave Tennis Serve
Tennis Serve Forward Kick Golf Swing
Pickup and Throw
Side Boxing Pickup and Throw
*[Wang et al.] Mining actionlet ensemble for action recognition with depth cameras, In CVPR, 2012.
![Page 26: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/26.jpg)
Datasets
• HDM05
– 30 Joints locations are available using a Motion Capture system.
– Actions:
• deposit floor, elbow to knee, grab high, hop both legs, jog, kick forward, lie down floor, rotate both arms backward, sneak, squat, and throw basketball
– Same setup as in *
*[Ofli et al.] Sequence of the most informative joints (smij): A new representation for human skeletal action recognition. In CVPRW, 2012
![Page 27: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/27.jpg)
Results
• MSR-Action3D
*[Wang et al.] Mining actionlet ensemble for action recognition with depth cameras, In CVPR, 2012.
Method Accuracy (%)
Actionlets Ensemble* 88.2
2-level 16-bin HOD (20 joints) 91.26
2-level 16-bin HOD (right hand joint only) 74.07
1-level 4-bin HOD (weakest configuration) 84.47
![Page 28: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/28.jpg)
Results
• MSR-Action3D
![Page 29: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/29.jpg)
Results
• HDM05 – clean data
*[Ofli et al.] Sequence of the most informative joints (smij): A new representation for human skeletal action recognition. In CVPRW, 2012
Method Accuracy (%)
Sequence of Most Informative Joints* 84.4
3-level 4-bin HOD (20 joints) 97.27
3-level 8-bin HOD (right elbow joint only) 82.72
1-level 4-bin HOD (weakest configuration) 80.0
![Page 30: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/30.jpg)
Comparison with the Actionlets Ensemble*
Their approach:
Use Fourier coefficients of relative positions of the whole set of joints as their main descriptor.
Introduced a mining algorithm to extract a set of actionlets for each action (each actionlet is a set of joints).
Multiple Kernel Learning to combine the actionlets.
Has a lot of parameters that are not easy to tune: ambiguity and confidence.
*[Wang et al.] Mining actionlet ensemble for action recognition with depth cameras, In CVPR, 2012.
![Page 31: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/31.jpg)
Comparison with the Actionlets Ensemble*
Ours:
Simpler framework!
No ensemble, the descriptor is used directly.
We have only two parameters (number of pyramid levels and number of histogram bins), easier to tune.
Our weakest configuration still performs very well.
*[Wang et al.] Mining actionlet ensemble for action recognition with depth cameras, In CVPR, 2012.
![Page 32: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/32.jpg)
Agenda
• Introduction
• Related Work
• Approach
• Experiments
• Conclusion
![Page 33: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/33.jpg)
Conclusion
• Introduced HOD: a novel 2D trajectory descriptor.
• Used it to efficiently describe the 3D trajectories of human body joints for action recognition.
• HOD is scale-invariant and speed-invariant.
• Outperformed the state-of-the-art on two popular datasets: MSR-Action3D and HDM05 using Linear SVM.
![Page 34: Histogram of Oriented Displacements (HOD)eng.staff.alexu.edu.eg/~mtorki/Publications/HODpresentation.pdf · Histogram of Oriented Displacements (HOD): Describing Trajectories of Human](https://reader036.vdocuments.mx/reader036/viewer/2022062605/5fd731c52fc7ac60ba6309f7/html5/thumbnails/34.jpg)
Thanks, Questions?