mobile motion tracking using onboard camera
DESCRIPTION
Mobile Motion Tracking using Onboard Camera. Supervisor Prof. LYU, Rung Tsong Michael Presented by Lam Man Kit, Wong Yuk Man. Outline. 1. Motivation & Objective. 2. Motion Input. 3. Previous Work. 4. Improvement & New Components. 5. Applications. 6. Experimental Result. 7. Demo. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/1.jpg)
Company
LOGO
Mobile Motion Tracking using Onboard Camera
SupervisorProf. LYU, Rung Tsong Michael
Presented byLam Man Kit, Wong Yuk Man
![Page 2: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/2.jpg)
Outline
1. Motivation & Objective1. Motivation & Objective
2. Motion Input 2. Motion Input
6. Experimental Result 6. Experimental Result
5. Applications 5. Applications
3. Previous Work3. Previous Work
4. Improvement & New Components4. Improvement & New Components
7. Demo7. Demo
![Page 3: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/3.jpg)
Motivation
Rapid increase in the use of camera-phone.
Camera-phones’ processing power is increasing
Symbian OS makes programming on mobile phone possible
![Page 4: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/4.jpg)
Objective
Develop motion input method on Symbian phone without requiring additional hardware
This acts as an innovative input method for applications such as: Camera mouse to control the cursor New input method for interactive games Gesture input
![Page 5: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/5.jpg)
What is Motion Input?
![Page 6: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/6.jpg)
Why Motion Input?
Motion input is quick and convenient Common way to play game on mobile
phone is to use joystick or keypad Limited freedom of movement Lack of maneuverability Difficult to press the small-sized keypad
Motion input doesn’t have these problems!
![Page 7: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/7.jpg)
Motion Tracking
Motion tracking is a process to find the motion vector of the current frame from the reference frame
We use block matching algorithm to find the motion vector
![Page 8: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/8.jpg)
Block Matching Algorithm
Comparison of blocks
(dx,dy)
The motion vector which corresponds to
the best match
New Position
![Page 9: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/9.jpg)
Block Matching Algorithm
Evaluate the "goodness" of a match by Sum of Absolute Difference
Select the candidate block with the lowest error
N
i
N
j
jiYjiXyxSAD1 1
|),(),(|),(
![Page 10: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/10.jpg)
Motion Tracking
Block matching algorithm Traditional block matching algorithm is slow Our solution: Hybrid-type fast block matching
algorithm That is “Adaptive Spiral SEA PPNM SSD” algorithm
Adaptive Search Window Spiral Scan Method Successive Elimination Algorithm Progressive Partial Norm Matching Sum of Squared Difference
![Page 11: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/11.jpg)
Previous Work
In the previous semester, Developed a testing platform in both Window and
Symbian OS Developed our own block matching algorithm and
feature selection algorithm Built a real-time translational motion tracking engine Developed some applications using the engine
![Page 12: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/12.jpg)
Improvement on Feature Selection
A good feature block: High variance -> complex block Not in a repeated pattern
If a good feature block is found, the performance of the motion tracking will be improved
We have made improvements to our existing algorithm
![Page 13: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/13.jpg)
squares
Old Feature Selection
Divide the current frame into squares Apply Feature Selection Algorithm to each square However, the best feature block may appear between
two squares
Best feature block
![Page 14: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/14.jpg)
New Feature Selection
More blocks are sampled Sample 26x26 squares with size 15x15 in a
54x54 window Old feature selection only samples 12x12
squares
Search in spiral way and stop searching when selection criteria are matched Prefer to find a feature block in the center Prevent out of bound problem (block appear
out of screen)
![Page 15: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/15.jpg)
New Feature Selection
If a feature block is found on the edge, the following will happen:
The tracking algorithm will fail to find the exact match
Solution: Apply additional
constraint
![Page 16: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/16.jpg)
New Feature selection
In order to find a feature block that is not on the edge, we apply a checking algorithm
Important difference in all directions -> interest point
![Page 17: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/17.jpg)
New Feature Selection
Now our newest feature selection algorithm becomes: Find a block with a large variance which
indicates the complexity of the block Check if the block is on the edge or not by
calculating the SAD between the candidate block and its 4 neighbors
If either one of the SAD is small -> the block is on edge -> reject Else the block is not on edge
![Page 18: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/18.jpg)
Rotation Tracking Engine
Observation and Motivation As we rotate the phone,
the object can still be tracked correctly (center of object always roughly equals center of green box)
![Page 19: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/19.jpg)
First approach Track one block by
translational motion tracking engine, rotate the current block and then find which rotation of the current block best match with the previous block
Fail if tracking object is the same for different angle
One-block approach
Rotation Tracking Engine
![Page 20: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/20.jpg)
Our approach If two blocks are
tracked simultaneously, angle of line connecting the two blocks indicates the rotation angle of the phone
Use only translational motion tracking
Two-block approach
Rotation Tracking Engine
![Page 21: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/21.jpg)
Rotation Tracking Engine
Modification of adaptive search window method to suit rotation tracking
Phone’s motion can be both linear motion and circular motion
Linear Adaptive method used in translational motion tracking is not used in this engine
Why need adaptive search window method? To increase performance based on the fact that
during block matching, reaching the best candidate block earlier increases the elimination effect of SEA, PPNM, PDE
![Page 22: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/22.jpg)
Rotation Tracking Engine
What to predict? Predict the coordinates of the blocks in the next frame Given the coordinates of the tracking blocks in the current frame
and the previous frames When to predict?
After each run of rotation tracking algorithm
θθ
Line L2
Line L1
Line L3
![Page 23: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/23.jpg)
Rotation Tracking Engine
Solution Angle between L2 and L3 = θ
θ = Tan-1(slope(L2)) – Tan-1(slope(L3))
Coordinates of the next tracking block (xL1, yL1) are calculated by multiplying the column matrix of coordinates of the previous block with a rotation matrix
2
2
1
1
cossin
sincos
L
L
L
L
y
x
y
x
![Page 24: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/24.jpg)
Rotation Tracking Engine
Solution The prediction of the positions of the next tracking blocks
should also take the translational movement into account Horizontal displacement = Tx
Tx = ( xL21 + xL22
– xL31 -
xL32
)/2
Vertical displacement = Ty Ty = ( yL21
+ yL22– yL31
- yL32
)/2
Coordinates of the next tracking block (xL1, yL1) is calculated by
1111
cossin
sincos
12
2
1
1
L
L
L
L
y
x
Ty
Tx
y
x
![Page 25: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/25.jpg)
Rotation Tracking Engine
Reducing error by using level Increase/decrease one level only
when change is large To give less sensitive but more
desirable output for game e.g. skiing game: skier face only
to 7 directions Reduce difficulty of the game,
increase reliability of the engine
40°
80°
120°
- 40° 0°Level 1
Level 2
Level 3
Level 0
Level -1
![Page 26: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/26.jpg)
Rotation Tracking Engine
Some screenshot showing how rotation tracking is carried out and how the result is displayed
![Page 27: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/27.jpg)
Rotation Tracking Engine
A short video demo
![Page 28: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/28.jpg)
Tailor-made background for both engines
Objective Max. performance measurement Increase usability
Condition Feature selection can always find a
good feature point Within certain distance
No repeat pattern Very distinct pattern
Pattern is nearly the same when rotated. E.g. Circle is good for rotation detection
![Page 29: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/29.jpg)
Virtual Mouse
An application that make full use of the translational motion tracking engine
Remote control the mouse of PC by mobile phone using motion as input
Advantages: It allows input in many directions and provides high
levels of control Many buttons left for other purposes
E.g. Joystick can be used as course adjustment of mouse cursor
E.g. Some buttons can be used as shortcuts of applications
![Page 30: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/30.jpg)
Virtual Mouse
Blu
etoo
th C
omm
unic
atio
n C
hann
el
Cable
![Page 31: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/31.jpg)
Virtual Mouse
Server Configure Bluetooth device to provide
RFComm Service and regards the Bluetooth transmission port as Comm. Port
Server receives message from that Comm. Port
Call function in MouseAction.h to make the mouse move and trigger mouse click event
![Page 32: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/32.jpg)
Virtual Mouse
Client Search for Bluetooth device nearby Connect to the selected Bluetooth device Every time motion tracking algorithm
finishes, results are sent to server (14 times/sec)
Joystick and keypad can also be used to control mouse in PC (Multi-button mouse)
![Page 33: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/33.jpg)
Car Racing Game
A “Car Racing Game” is developed using the motion tracking engine.
Game lags. It is because: The engine takes time to find the motion vector CPU speed of the Symbian phones is low
Solutions: Double Buffering Direct Screen Access
![Page 34: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/34.jpg)
Skiing Game
We have developed a skiing game that makes use of the rotation tracking engine Rotate the phone to control the angle of the
skier
After using the rotation tracking engine: The game becomes more interactive The game has more degree of freedom for
rotating the skier
![Page 35: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/35.jpg)
Skiing Game
![Page 36: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/36.jpg)
Skiing Game
The process of using the rotation tracking engine is very simple Create an instance of the tracking engine Call the function of the engine to find the
motion vector Use the motion vector for the game logic
![Page 37: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/37.jpg)
Skiing Game
![Page 38: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/38.jpg)
Experimental Result on Symbian phone
Testing Environment Platform
Symbian Phone Nokia 6600 Algorithm
Our final (hybrid-type) algorithm (Block matching algorithm featured with Adaptive Window, Spiral Scan, SEA, PPNM and PDE method)
Algorithm parameter Block size = 17 x 17 (pixels) Search window size = 16 x 16 (pixels)
Number of block to track in each run of algorithm = 1 block ONLY
![Page 39: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/39.jpg)
Result
Run time of translational motion tracking algorithms
0
10
20
30
40
50
60
background
Tim
e (
ms)
Full Search (ms)
Partial Sprial (ms)
Adaptive SEA (ms)
![Page 40: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/40.jpg)
Experimental Result on Symbian phone
Testing Result (avg time to run) Final algorithm
7ms Partial Spiral algorithm
22ms Full Exhaustive Search algorithm (the most
simplest one) 55ms
![Page 41: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/41.jpg)
Result
Accuracy of the rotational engine
505560657075808590
bg 1 bg 2 bg 3 bg 4 bg 5 bg 6 bg 7 bg 8
measured angle
![Page 42: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/42.jpg)
Conclusion
We have developed: The real-time motion tracking algorithm and
feature selection method The translational and rotational motion
tracking engine on Symbian A series of applications/games that make full
use of our engines
![Page 43: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/43.jpg)
Demo
Virtual Mouse
![Page 44: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/44.jpg)
Demo
Act as an innovative input device for games
![Page 45: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/45.jpg)
Q & A
![Page 46: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/46.jpg)
Thank You !!
![Page 47: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/47.jpg)
Experimental Result1 - Backgrounds
![Page 48: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/48.jpg)
Experimental Result2 - Backgrounds
![Page 49: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/49.jpg)
Stereo
If we want to track the movement in 3D, we have to: Use two cameras Know the focal length of the cameras Know the distance between two cameras
Depth can be recovered with two images Depth can be recovered with two images and triangulationand triangulation
![Page 50: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/50.jpg)
Stereo
Therefore, stereo has two steps Finding matching points in the images Then using them to compute depth.
OO22
P’P’22
PP
OO11
P’P’11
![Page 51: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/51.jpg)
Stereo
Suppose we have found the matching point P. Now the step to compute the depth is as follow:
f: focal length of the cameras
b: distance between two cameras
Note: y-axis is perpendicular to the slide
![Page 52: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/52.jpg)
Stereo
If b is the distance between the two cameras, and f is the focal length of the cameras, we have
Use similar triangular, we can solve to get
z
bx
f
xl 2/'
z
bx
f
xr 2/'
z
y
z
y
f
y rl ''
''
'' 2/)(
rl
rl
xx
xxbx
''
'' 2/)(
rl
rl
yy
yyby
''
rl xx
fbz
![Page 53: Mobile Motion Tracking using Onboard Camera](https://reader033.vdocuments.mx/reader033/viewer/2022042822/56814374550346895daff31b/html5/thumbnails/53.jpg)
Mono
In our project, only one camera is available in the Symbian phone
If we want to find the z-axis movement, we can find it by comparing the size of the capture image Getting closer -> larger image Getting farther -> smaller image
However, we have tried that using this algorithm is not accurate enough