![Page 1: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/1.jpg)
Gesture Recognition in a Class Room Environment
Michael Wallick
CS766
![Page 2: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/2.jpg)
Virtual Videography
Place cameras in an environment
Automatically edit video off-line
Output should look like a professional editor
![Page 3: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/3.jpg)
Our Implementation
Looking at the classroom domain
Recorded one semester of CS559 (Computer Graphics)
![Page 4: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/4.jpg)
Computer Vision in Virtual Videography Understand what is happening on the
chalkboard Writing on the board
Understand what the professor is doing Location Actions
![Page 5: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/5.jpg)
Chalkboard…
Partition the board into regions
Regions are semantically related groups of writing
Regions can be approximated using computer vision Let’s treat this as a black box … it just
happens
![Page 6: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/6.jpg)
Gesture Recognition
Understand gestures or actions by a performer
Generally used as an input to a computer
Understand what the professor is doing Pointing Writing Reaching
![Page 7: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/7.jpg)
Writing can be confused with Pointing and Reaching
![Page 8: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/8.jpg)
Template Matching for G. R.
Generate templates of known gestures
Match an unknown frame with a template matching algorithm Sum of Squared Difference Cross Correlation Image Difference …
![Page 9: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/9.jpg)
Implement of Gesture Recognition
The user selects several template images Pointing
Reaching
![Page 10: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/10.jpg)
Format the templates
Separate the lecturer
Crop the image
Resize the images 256x256
![Page 11: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/11.jpg)
Build the Recognition Mask
Load each template into the mask
For each “on” pixel, increment the mask at that location
![Page 12: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/12.jpg)
Recognizing Gestures
Separate the lecturer from foreground Crop and resize For every “on” pixel, increment the “Score” by
that value in the mask Compute Confidence as
(float) (Score/Mask_Total) Compute Confidence for all gestures
![Page 13: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/13.jpg)
A Gesture Matches if Confidence is:
Under 50% but much larger than other gestures
Over 50% and not too close to other gestures
![Page 14: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/14.jpg)
Example: Ground State
![Page 15: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/15.jpg)
Example: Pointing
![Page 16: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/16.jpg)
Example: Reaching
![Page 17: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/17.jpg)
Mistakes
Overall the results are good
Sometimes individual frames are not correct
![Page 18: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/18.jpg)
Solution
For each frame, look at surrounding frames
Label frame with gesture of the majority
![Page 19: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/19.jpg)
Where to go from here…
Use the regions to Validate the gestures Determine what is being pointed at
Incorporate the writing information with the gestures
Write paper and webpage!
![Page 20: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/20.jpg)
Conclusions
We want to use gesture recognition for Virtual Videography
Gestures can be used to drive camera model
Find gestures by template matching
For each frame, take the “average” around a region of frames to correct errors
![Page 21: Gesture Recognition in a Class Room Environment Michael Wallick CS766](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649f4d5503460f94c6e1ff/html5/thumbnails/21.jpg)
Thank You!
Questions/ Comments?