dodecapen: accurate 6dof tracking of a passive stylus
TRANSCRIPT
DodecaPen: Accurate 6DoF Tracking of a Passive Stylus
Po-Chen Wu*โ , Robert Wangโ , Kenrick Kinโ , Christopher Twiggโ , Shangchen Hanโ
Ming-Hsuan Yangโก, Shao-Yi Chien*
*National Taiwan University, โ Oculus Research, โกUC Merced
Outline
Introduction
Proposed System
Experimental Results
Conclusion
@Students in AI WorkshopDodecaPen 2
Outline
Introduction
Proposed System
Experimental Results
Conclusion
@Students in AI WorkshopDodecaPen 3
Introduction
Our system can track the 6DoF pose of a calibrated pen from a single camera with submillimeter accuracy.
@Students in AI WorkshopDodecaPen 4
@Students in AI WorkshopDodecaPen 5
@Students in AI WorkshopDodecaPen 6
Why Dodecahedron?
To prevent pose jumping (due to coplanar points)
@Students in AI WorkshopDodecaPen 7
Explanation of Pose Jumping (1/2)
@Students in AI WorkshopDodecaPen 8
Explanation of Pose Jumping (2/2)
Multiple local minima of a cost function (due to coplanar points)
@Students in AI WorkshopDodecaPen 9
Co
st
Pose
OriginalImage
NoisyImage
Outline
Introduction
Proposed System
Experimental Results
Conclusion
@Students in AI WorkshopDodecaPen 10
Approximate Pose Estimation (APE)
โข Marker Detection
โข Minimize reprojection error ๐ธ๐(๐ฉ) with P๐P algorithm to get the initial pose ๐ฉโฒ
๐ธ๐ ๐ฉ =1
๐
๐=1
๐
เท๐ฎ๐ โ ๐ฎ๐ ๐ฑ๐; ๐ฉ2
11
เท๐ฎ: detected point in the camera image๐ฑ: point on the dodecahedron๐ฎ: transformed ๐ฑ point in the camera image๐ฉ: pose (including rotation matrix ๐น and translation vector ๐ญ)
Inter-frame Corner Tracking (ICT)
โข Pyramidal Lucas-Kanade marker corner tracking
โข P๐P algorithm to get the initial pose ๐ฉโฒ
12
If APE does not succeedโฆ
Marker Intensity Normalization
โข We normalize the intensity to ensure intensity invariance before minimizing the residual between the model and image.
13
Dense Pose Refinement (DPR)
โข Minimize appearance distance ๐ธ๐(๐ฉ) with Gauss Newton and backtracking line search (BLS) to get the final pose ๐ฉโ
๐ธ๐ ๐ฉ =1
๐
๐=1
๐
๐ผ๐ ๐ฎ๐ ๐ฑ๐; ๐ฉ โ ๐๐ก ๐ฑ๐2
14
๐ผ๐: camera image๐๐ก: target object๐ฑ: point on the dodecahedron๐ฎ: transformed ๐ฑ point in the camera frame๐ฉ: pose (including rotation matrix ๐น and translation vector ๐ญ)
Marker MipmapsMipmap Masks
Masked Mipmaps
Vectorization
Gauss-Newton Iteration
ฮ๐ฉโ = argminฮ๐ฉ
1
๐
๐=1
๐
๐ผ๐ ๐ฎ๐ ๐ฉโฒ + ฮ๐ฉ โ ๐๐ก ๐ฑ๐2
โ argminฮ๐ฉ
1
๐
๐=1
๐
๐ผ๐ ๐ฎ๐ ๐ฉโฒ + แค๐๐ผ๐๐๐ฉ
๐ฉ=๐ฉโฒ
ฮ๐ฉ โ ๐๐ก ๐ฑ๐
2
15
๐ธ๐ ๐ฉ =1
๐
๐=1
๐
๐ผ๐ ๐ฎ๐ ๐ฉ โ ๐๐ก ๐ฑ๐2
๐๐ฮ๐ฉ = ๐๐ก โ ๐๐แค๐๐๐๐๐ฉ
๐ฉ=๐ฉโฒ
โก ๐๐ ๐ธ๐โฒ ๐ฉ = 0
ฮ๐ฉ = ๐๐T๐๐
โ1๐๐T(๐๐ก โ ๐๐)
Backtracking Line Search (BLS)
Gauss-Newton iteration does not always converge with a fixed step size since our least squares problem is nonlinear.
We shrink ฮ๐ฉ by ฮ๐ฉ โ ฮฑฮ๐ฉ until it meets the Armijo-Goldstein condition below:
๐ธ๐ ๐ฉโฒ + ฮ๐ฉ โค ๐ธ๐ ๐ฉโฒ + ๐๐ป๐ธ๐ ๐ฉโฒ Tฮ๐ฉ
๐ป๐ธ๐ ๐ฉโฒ is the local function gradient
ฮฑ = 0.5 , ๐ = 10โ4
@Students in AI WorkshopDodecaPen 16
Dodecahedron Calibration (DC)
Determine the precise pose of each marker with respect to the dodecahedron
One-time offline bundle adjustment
๐ธ๐ ๐ฉ๐ , ๐ฉ๐ =
๐
๐
๐
๐ผ๐ ๐ฎ๐ ๐ฑ๐; ๐ฉ๐; ๐ฉ๐ โ ๐๐ก ๐ฑ๐
2
17
๐ผ๐: camera image๐๐ก: target object๐ฑ: point on the dodecahedron๐ฎ: transformed ๐ฑ point in the camera frame๐ฉ: dodecahedron pose๐ฉ: marker pose
Outline
Introduction
Proposed System
Experimental Results
Conclusion
@Students in AI WorkshopDodecaPen 18
Evaluation with Synthetic Data
We generate synthetic image sequences with 24 motion patterns of the virtual DodecaPen Each sequence consists of 301 frames with the same
resolution (1280*1024) and intrinsics as our real camera.
We further evaluate the proposed approaches under varying shot noise, spatial blur, camera resolutions, and mask kernel widths.
@Students in AI WorkshopDodecaPen 19
@Students in AI WorkshopDodecaPen 20
Pen-tip Trajectories
@Students in AI WorkshopDodecaPen 21
@Students in AI WorkshopDodecaPen 22
Evaluation with Real Data
Four real drawings
Compare with a motion capture system Constructed with 16 OptiTrack Prime 17W (1.7 megapixels, 70 degrees field-
of-view) cameras
@Students in AI WorkshopDodecaPen 23
24
DodecaPen VS. Mocap
@Students in AI WorkshopDodecaPen 25
The accuracy of the proposed method is comparable to a motion capture system with 10 active cameras.
TW$ 10,000 VS. TW$ 1,000,000
Outline
Introduction
Proposed System
Experimental Results
Conclusion
@Students in AI WorkshopDodecaPen 26
Conclusion
6DoF tracking using a set of readily available and easy-to-assemble components
Sub-millimeter accurate
Single camera pose estimation can be fast enough and robust enough for drawing in 2D, 3D and in VR
@Students in AI WorkshopDodecaPen 27
Potential AI Research Topics
General Object Pose Tracking
@Students in AI WorkshopDodecaPen 28