event&basedfeature+tracking+ and+visualinertial...
TRANSCRIPT
![Page 1: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/1.jpg)
Event-‐based Feature Tracking and Visual Inertial Odometry
Kostas Daniilidiswith Alex Zhu and Nikolay Atanasov
University of PennsylvaniaPapers at ICRA 2017 and CVPR 2017
www.youtube.com/watch?v=m93XCqAS6Fcwww.youtube.com/watch?v=X3QIFj5Qc4A
![Page 2: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/2.jpg)
Night Scene, Very Low Lighting0.1x Real Time
Corresponding grayscale image, 50ms exposure
![Page 3: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/3.jpg)
Truck Passing 3m from the Camera at 60 miles/hr, 0.06x Realtime
Optical flow is on the order of 5000 pixels/s.Sequence is 600ms in realtime.
240FPS iPhone 6 image with KLT. Event-based tracking on DAVIS 240C.
![Page 4: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/4.jpg)
Frame-based Cameras
![Page 5: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/5.jpg)
Event-based Cameras
Event-based cameras output asynchronous events (x, y, t, p) at microsecondresolution when | log 𝐼 𝑥, 𝑡) − log (𝐼(𝑥, 𝑡)-.)| ≥ 𝜃
![Page 6: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/6.jpg)
What is a feature in classic vision?
Features are defined through motion: good flow means good features!
But they defined with a spatial neighborhood!
![Page 7: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/7.jpg)
Speed is dealt with multiple scales
![Page 8: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/8.jpg)
A feature is a set of 2D events induced by the same point in 3D.
![Page 9: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/9.jpg)
A feature is a set of 2D noisy events induced by the same point in 3D.
![Page 10: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/10.jpg)
A feature is a set of events with same flow
![Page 11: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/11.jpg)
But we do not know the association so we will take the expectation
![Page 12: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/12.jpg)
Optical Flow Estimation
min5,6
77 7𝑟)9𝑟:9
;
9<.
|| 𝑥) − 𝑡)𝑣 − 𝑥:− 𝑡:𝑣 ||>?
:<.
?
)<.
Data association probability Propagated events through time
![Page 13: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/13.jpg)
E-‐Step
M-‐Step (linear least squares)
![Page 14: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/14.jpg)
How long temporal window?
0.5ms window 2ms window 5ms window
![Page 15: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/15.jpg)
𝜏 =𝑘
||𝑣||>
How do we choose the right temporal window?
0.05s
0.02s
![Page 16: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/16.jpg)
Over longer time:Monitor quality of feature with an affine motion model!
![Page 17: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/17.jpg)
Drift Correction - Stabilization
minC,D,5
77𝑟)9||𝐴 𝑥) − 𝑡)𝑣 + 𝑏 − 𝑝9||>;
9<.
?
)<.
Data associationWarped propagated events
Template points
![Page 18: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/18.jpg)
Scope of each feature
![Page 19: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/19.jpg)
Results: KLT Comparison
20 windows initialized, no reacquisition if tracks are discarded. Frame based images from DAVIS.
![Page 20: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/20.jpg)
Sparsify and deal with aperture problem:FAST corner selection
in the aggregation of warped images
![Page 21: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/21.jpg)
Visual Inertial Odometry
•Given the event-‐based feature tracks and a set of IMU observations, how do we obtain an accurate estimate of the camera pose?•MSCKF (Roumeliotis’ group)• Enforce 3D rotation in 2D tracking
![Page 22: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/22.jpg)
Instead of affine warp…
We use the current estimate of rotation and we estimate only scale and local translation
![Page 23: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/23.jpg)
Outlier Rejection
• The EKF uses the L2 loss, and so is very susceptible to outliers in the measurements. To remove these outliers, we apply two RANSAC steps during the tracking.• RANSAC 1: Pure Translation
• After each temporal window, two point RANSAC is applied given the rotation estimated from the IMU to reject failed trackers.
• RANSAC 2: Triangulation over frames• As each feature track is residualized, a second RANSAC step is applied to find the largest inlier set that agrees on a 3D pose of the feature, given the observations and their corresponding camera poses.
![Page 24: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/24.jpg)
EVIO Summary
Zhu A., Atanasov N., and Daniilidis D. “Event-based Visual Inertial Odometry”, Submitted to Computer Vision and Pattern Recognition (CVPR) 2017.
![Page 25: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/25.jpg)
Results: General Scene
![Page 26: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/26.jpg)
Results: HDR Scene
![Page 27: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/27.jpg)
Results: Motion Independent of Camera
![Page 28: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/28.jpg)
![Page 29: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/29.jpg)
![Page 30: Event&basedFeature+Tracking+ and+VisualInertial Odometryrpg.ifi.uzh.ch/docs/ICRA17workshop/Daniilidis.pdfTruck+Passing+3m+from+the+Camera+at+ 60miles/ hr,0.06x+Realtime Optical flow](https://reader033.vdocuments.mx/reader033/viewer/2022060900/609dc3820432a92d8d06ad58/html5/thumbnails/30.jpg)
The future of robot vision is event-‐based!