SALMA

Stereo Visual Odometry

April 26 2019

Victor MARTIN-LAC

2

Features

Hardware

● Build a stereo rig

Software

● user interface allowing to

– record multiple view video stream,

– calibrate cameras and stereo rigs,

– compute visual odometry trajectory,

– display trajectory and odometry data.

● stereo visual odometry algorithm.

● algorithm to track calibration target.

● multiple view video stream acquisition.

3

Numbers

Begun in august 2018

About 20 000 lines of C++

About 1100 € spent

http://github.com/vmartinlac/salma/

Hardware

5

USB

Ethernet * 2I/O * 2

Rig

Arduino PC

Architecture

6

Architecture

Take Received Image

Trigger

Process Image

Trigger

7

Cameras and Lenses

2* Allied Vision TechnologiesManta G125C

Gigabit Ethernet1292 * 964 @ 30 fps

2 * Fujinon TF8DA-8B8mm F/2.2 3-CCD Lens

FoV 33° × 25°

2 * KOWA LM4NCL3.5mm F1.4 – 16FoV 76.7° x 57.7°

8

The Platform

● Designed in FreeCAD● 3d-printed at http://www.fabric-insa.fr/

9

Power and Trigger

10

PC

My PC 500 Go SSD drive Gigabit ethernetexpresscard

11

Other Stuff

CAT7 RJ45 cable * 2 I/O cable with Hirose12-pin connector * 2

12V 3A DCpower supply

12

Calibration Targets

https://www.pixartprinting.fr/

13

Putting It All Together

PC CameraCalibrationTarget

Stereo Rig

Arduino

RigCalibration

Target

Hardware Legacy

15

The Platform (legacy)

23 cm baseline → hard to produce dense disparity maps in room-scale environment

16

Ethernet * 2I/O * 2

Rig

Signal Gen. PC

Architecture (legacy)

Hard to associate frames received from several cameras

17

Power and Trigger (legacy)

24 Hz square wave generator

18

Calibration Targets (legacy)

Check that background of SVG file is white, not transparent!

Software

20

Dependencies and ToolsDependencies

● Aravis

● Eigen

● g2o

● libjpeg-turbo

● Nanoflann

● OpenCV

● Openscenegraph

● Qt

● Sophus

Tools:

● C++

● Cmake

Portability: Linux only!

● Aravis only on Linux

● Use of open() stat() write() close()

21

Software Architecture

slam

persistence

ui

targetmvpnpbuildinfo videocalibration stereomatcher

uicommon

uirecording uicalibration uireconstruction

Software

Video Acquisition & Recording

23

Video Acquisition

● Rely on Aravis library as implementation of GenICam norm (Vimba SDK from Allied Vision Technologies can be used from only one user thread)

● Some multithreading involved.

● Minimize CPU usage.

24

Video Recording

● Use libjpeg-turbo for encoding

● Synchronous write on file system

– avoid sudden increase of write time when file system cache gets full.

Software

Detecting the Calibration Target

26

Detecting the calibration targetAlgorithmic steps:

1) Good Features to Track

2) Thresholding

3) Interest point filtering

4) Computation of topology and edge filtering

5) Propagation of a coordinate system

6) Dots detection and correction of orientation

7) Numbering the points

27

Detecting the calibration target

(1) (2) (3)

28

Detecting the calibration target

(4) (5) (6)

29

Detecting the calibration target

(7) (8) (9)

30

Detecting the calibration target

Implemented and potential applications of this algorithm:

● Camera calibration (extrinsic and/or intrinsic parameters)

● Rig calibration (relative pose of cameras)

● Pose estimation

Software

Stereo Visual Odometry Method

32

Why Stereo?Pro stereo:

● Easier initialization

● Scale does not drift

● Handle pure rotation

Pro mono:

● Half cost

● Half amount of data

33

Algorithm

Triangulation

Stereo Matching

Local Bundle Adjustment

Keyframe Selection

Alignment

Temporal Matching

Feature Extraction

● Sparse indirect method

● Same family as PTAM and ORB-SLAM.

34

Feature Extraction● Purpose: Detect and describe key points.

● Use ORB features.

35

Temporal Matching● Purpose: establish 2d-3d correspondences between keypoints and local map points.

● Difference with ORB-SLAM: no projection.

● Match left (right) view with k previous left (right) views.

36

Alignment● Purpose: compute pose of stereo rig given 2d-3d correspondances.

● Multiple-view perspective-N-points (MVPnP).

● Use Levenberg-Marquardt + RANSAC.

37

Keyframe Selection● Purpose: decide whether this frame is to be regarded as keyframe or not.

● Check whether translation or rotation angle since last keyframe are above some threshold.

38

Local Bundle Adjustment● Purpose: adjust poses of last n keyframes and positions of visible mappoints to

minimize reprojection error.

● Multiple view.

39

Stereo Matching● Purpose: match keypoints from left and right views.

● Criteria: epipolar constraints, Lowe criterion, distance between descriptors.

40

Triangulation● Purpose: from each stereo match, compute corresponding map point.

● Correction of key point positions according to epipolar constraints (Lindstrom method).

● Middle of common perpendicular.

41

Results

42

Results

43

Runtime● Minimizing runtime is not the focus of the project.

● Pipeline can be CUDA-accelerated if compiled with some CMake option.

● On my Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz

– FEATURES DETECTION: 472 ms

– TEMPORAL MATCHER: 50 ms

– ALIGNMENT: 24 ms

– KEYFRAME SELECTION: 0 ms

– LOCAL BUNDLE ADJUSTMENT: 418 ms

– STEREO MATCHING: 52 ms

– TRIANGULATION: 0 ms

Software

User Friendlinesses

45

Data Persistence

For each project:

● Images are stored individually on the file system as JPEG.

● Everything else is stored in one main SQLITE database:

● Recording metadata,

● Calibration data,

● Reconstruction data.

46

User Interface and 3d Visualization● Qt5 for user interface.

● Openscenegraph for 3d visualization.

Conclusion

48

● Evaluate odometry precision and robustness

● Implement dense reconstruction (nice to have)

– Experimented with libelas and OpenCV

– Read some papers

What Next?

49

Thank you for your attention

Questions & Answers