bi-objective bundle adjustment
TRANSCRIPT
![Page 1: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/1.jpg)
Bi-objective Bundle Adjustmentwith Application to Multi-Sensor SLAM
Contact: [email protected]
Julien MichotAdrien Bartoli
François Gaspard
![Page 2: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/2.jpg)
Julien Michot - 3DPVT - 05/18/2010
2
Context
• Monocular SLAM Localization & reconstruction with only
one camera Localization up to a similarity
Main problems: drifts! on the trajectory
(position/orientation) on the scale factor
• Solutions? External data!
Inertial (gyroscope, IMU,...), INS, GPS, Odometers, Laser range finder, GIS, CAD models,
…
• Objective Real-time 3D localization in unknown environment (SLAM).
- Purple: Ground truth- Green: Monocular SLAM
• Approach Incorporate complementary data into an Incremental Structure-from-Motion SLAM
...directly in the optimization process (Bundle Adjustment)
![Page 3: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/3.jpg)
Julien Michot - 3DPVT - 05/18/2010
3
Presentation Outline
• Background
• Bi-objective Bundle Adjustment Weighted Bundle Adjustment Automatic Weight Selection
• Application to Multi-sensor SLAM
• Experimental Results on Multi-sensor SLAM Hand-held camera application Vehicle application
• Conclusion
![Page 4: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/4.jpg)
Julien Michot - 3DPVT - 05/18/2010
4
Background
![Page 5: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/5.jpg)
Julien Michot - 3DPVT - 05/18/2010
5
Background
• Incremental Structure-from-Motion SLAM[Mouragnon06, Engles06, Klein07] Aim: real-time reconstruction & localization
Tracking Features detection (salient corners) Features description (surf,...) Features matching Camera localization (linear/nonlinear)
Keyframe selection
Mapping 3D reconstruction of points
Bundle Adjustment
![Page 6: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/6.jpg)
Julien Michot - 3DPVT - 05/18/2010
6
Background
• Bundle Adjustment[Triggs99] Optimize camera poses and scene reconstruction. Based on a MAP estimator + Gaussian probability functions NLLS.→
![Page 7: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/7.jpg)
Julien Michot - 3DPVT - 05/18/2010
7
Background
• Bundle Adjustment[Triggs99] Optimize camera poses and scene reconstruction. Based on a MAP estimator + Gaussian probability functions NLLS.→
Reprojection error:
with
Global Bundle Adjustment
...with Levenberg-Marquardt.
![Page 8: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/8.jpg)
Julien Michot - 3DPVT - 05/18/2010
8
Bi-objective Bundle Adjustment
![Page 9: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/9.jpg)
Julien Michot - 3DPVT - 05/18/2010
9
Bi-objective Bundle Adjustment
• Objective Integrates external data directly in the BA.
• How? Constraints each pose
Camera position constraints
Camera orientation constraints
Trajectory scale factor constraint
... New BA cost function:
Select a “good” weighting factor... When variances are unknown/untrusted...
![Page 10: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/10.jpg)
Julien Michot - 3DPVT - 05/18/2010
10
Bi-objective Bundle Adjustment
• Selecting an efficient weighting factor?Incremental approach: one by one.Aim: Choose a weight that minimizes both objectives bi-objective optimization.→
![Page 11: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/11.jpg)
Julien Michot - 3DPVT - 05/18/2010
11
Bi-objective Bundle Adjustment
• Selecting an efficient weighting factor?Incremental approach: one by one.Aim: Choose a weight that minimizes both objectives bi-objective optimization.→
![Page 12: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/12.jpg)
Julien Michot - 3DPVT - 05/18/2010
12
Bi-objective Bundle Adjustment
• Selecting an efficient weighting factor?Incremental approach: one by one.Aim: Choose a weight that minimizes both objectives bi-objective optimization.→
We propose two methods: Machine learning Criterion
![Page 13: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/13.jpg)
Julien Michot - 3DPVT - 05/18/2010
13
Methods for selecting the weighting factor
![Page 14: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/14.jpg)
Julien Michot - 3DPVT - 05/18/2010
14
Bi-objective Bundle Adjustment
• Selecting an efficient weighting factor?
Learning method: Cross Validation (leave-one-out)A good λ should predict well missing observations.
Observationsset
Learning set Test set
Find:
![Page 15: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/15.jpg)
Julien Michot - 3DPVT - 05/18/2010
15
Bi-objective Bundle Adjustment
• Selecting an efficient weighting factor? Criteria-based selection: L-Curve [Hansen01]
Initially for regularization problems
Principle: solve L NLLS with different value of λ l=1..L
,
Log-log plot of the trade-off of each objective (Pareto
frontier):
(reprojections) (sensor constraint)
Select the weighting factor with the higher curvature: near the corner.
with ' and '' first & second derivatives wrt λ.
![Page 16: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/16.jpg)
Julien Michot - 3DPVT - 05/18/2010
16
Application to a Real-time Multi-sensor SLAM
![Page 17: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/17.jpg)
Julien Michot - 3DPVT - 05/18/2010
17
Application to Multi-sensor SLAM
• Overview Sensors
Calibrated pinhole camera One additional sensor (ex: gyroscope)
Incremental Approach Each new keyframe is constrained with an independent weighting factor
• Building a constraint Predict the system pose (at the time of the keyframe)
Previous measurements (buffer) Motion model (asynchronous measurements)
Choose a constraint Absolute or relative On the position, orientation, ...
• When should we compute the lambda? In the LBA?
Too long (multiple LBA: >seconds)
In a simplest problem During the pose refinement (tracking)
![Page 18: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/18.jpg)
Julien Michot - 3DPVT - 05/18/2010
18
Application to Multi-sensor SLAM
• Selecting the weighting factor Solve L NLLS problems! need to be real-time!→ Estimate the weighting factor in a reduced BA: the camera refinement.
Pose refinement Optimize only 6 parameters (position&orientation of the camera)
Bi-objective pose refinement 1. Add the constraint to the cost function
(reprojections) (unknown weight) (constraint)
2. Estimate an efficient weighting factor (Cross Validation, L-Curve, …) for this sub-problem 3. Finally, optimize the camera pose with the selected weighting factor.
→ More stable: the scene is fixed (less local minima)
![Page 19: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/19.jpg)
Julien Michot - 3DPVT - 05/18/2010
19
Application to Multi-sensor SLAM
• Local Bundle Adjustment Optimize only recent camera poses
and parts of the scene reconstruction.
Local Bundle Adjustment[Mouragnon06] Cost function (reprojections)
Constrained Local Bundle Adjustment Add the constraint on every optimized camera poses with the selected weighting factor.
(reprojections) (weighting factor) (constraints)
![Page 20: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/20.jpg)
Julien Michot - 3DPVT - 05/18/2010
20
Application to Multi-sensor SLAM
• Constrained Structure-from-Motion SLAM
Tracking Features detection (salient corners) Features description (surf,...) Features matching Camera localization (linear/nonlinear)
Keyframe selection Build the constraint Learn the weighting factor
Mapping 3D reconstruction of points
Constrained Bundle Adjustment
![Page 21: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/21.jpg)
Julien Michot - 3DPVT - 05/18/2010
21
Experimental Results
![Page 22: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/22.jpg)
Julien Michot - 3DPVT - 05/18/2010
22
Experimental Results on Multi-sensor SLAM
• Hand-held camera localization Visual SLAM with an IMU
- Purple: Ground truth- Red: Visual + IMU SLAM and L-Curve
Trajectory: indoor, 8mGround truth: 3D laser tracker systemConstraint: relative rotations
Mean 3D RMS with GT- Visual SLAM only: 3.15cm- Visual+IMU SLAM with EKF: 3.17cm - Visual+IMU SLAM with L-Curve: 2.99cm- Visual+IMU SLAM with Cross Validation: 3.11cm- Visual+IMU SLAM with LTN: 3.01cm
![Page 23: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/23.jpg)
Julien Michot - 3DPVT - 05/18/2010
23
Experimental Results on Multi-sensor SLAM
• Vehicle localization Visual SLAM with an odometer
Trajectory: outdoor, 400mGround truth: INS+GPS RTK (cm)Constraint: scale factor
Mean 3D RMS with GT- Visual SLAM only: 3.67m - Visual+odometer SLAM with L-Curve:1.79m- Visual+odometer SLAM with Cross Validation: 2.39m- Visual+odometer SLAM with LTN: 3.27m
- Purple: Ground truth- Yellow: Visual SLAM only- Red: Visual with an odometer SLAM and L-Curve
![Page 24: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/24.jpg)
Julien Michot - 3DPVT - 05/18/2010
24
Experimental Results + Future Work
• Vehicle localization on long distances Visual SLAM with an odometer
Trajectory: outdoor, ~4kmGround truth: INS+GPS RTK (cm)Constraint: scale factor
The scale factor drift is reduced but:- the orientation drift is always here We also need a constraint on the orientation.
- Purple: Ground truth- Green: Visual SLAM only- Red: Visual with an odometer SLAM and L-Curve
- a small drift on the scale factor remains Absolute data are needed for long distance visual based localization.
![Page 25: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/25.jpg)
Julien Michot - 3DPVT - 05/18/2010
25
Conclusion
• We've presented A bi-objective bundle adjustment
Bi-objective: no priors on the variance of the measurements/parameters A way to add constraints in BA with automatic weighting factor selection
Two methods that estimates the weighting factor a learning method (Leave-one-out Cross Validation) a criterion (L-Curve, variants)
Application: two multi-sensor real-time SLAM Monocular + IMU SLAM Monocular + odometer SLAM
• Future work Integrate absolute data such as GPS Extend the method to multiple constraints (multiple weights)
![Page 26: Bi-objective Bundle Adjustment](https://reader031.vdocuments.mx/reader031/viewer/2022022000/589ee3771a28ab3d4d8b7559/html5/thumbnails/26.jpg)
Julien Michot - 3DPVT - 05/18/2010
26
Thank you Thank you