francisco’s steepest streets · fixed width stixels, variable number of stixels per column...
TRANSCRIPT
![Page 1: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/1.jpg)
Slanted Stixels: Representing San Francisco’s Steepest Streets
Daniel Hernández-Juárez*, Lukas Schneider*, Antonio Espinosa, David Vázquez,
Antonio M. López, Uwe Franke, Marc Pollefeys and Juan C. Moure* Contributed equally
![Page 2: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/2.jpg)
Stixel World: Compact representation of the world
Stereo + Horizon Line + Road Slope Stixels
Obj.
Sky
Obj.
Obj.
Grnd
slopehorizon
Stereo Images
Fixed width stixels, variable number of stixels per column
Widespread in stereo vision for automotive industry
Stixel = Stick + Pixel
![Page 3: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/3.jpg)
Motivation & Objectives
3
Left Image Original Stixels
Previous work: • Stixel World [Pfeiffer 2011]• Semantic Stixels [Schneider 2016]
Problem:• Slanted roads not well modeled
Contribution:• New model to represent slanted
roads• Proposal to reduce computational
work
![Page 4: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/4.jpg)
Semantic Stixels: Unified approach
Stereo + Horizon Line + Road Slope
Semantic Stixels
Buil.
Sky
Ped.
Side
Road
slopehorizon
Stereo Images
Semantic segmentation
![Page 5: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/5.jpg)
Stixels: Original Model
5
Sky: 0 disparity Assumption: Far away
Object: Disparity mean Assumption: Constant distance
Ground: Precomputed model Assumption: Constant height
Unified probabilistic approach:Disparity term considers the accordance of measurements to modelSemantic energy favors classes that fit to the observed input
Enforces constraints: no sky below horizon, no neighbors objects at same disparity …
Computed independently for each column
Stixel models:
![Page 6: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/6.jpg)
New model: Slanted Stixels
6
• New model to represent all classes:
• With priors according to the class: 𝐸𝑝𝑙𝑎𝑛𝑒 𝑠𝑖 = (𝑎−𝜇𝑐𝑖
𝑎
𝜎𝑐𝑖𝑎 )2 + (
𝑏−𝜇𝑐𝑖𝑏
𝜎𝑐𝑖𝑏 )2 − log 𝑍
• Each class has different parameters: slanted ground and objects and 0 disparity for sky
• Optimized jointly with Semantic Stixel probabilistic framework
𝜇 𝑠𝑖 , 𝑣 = 𝑏𝑖 ∗ 𝑣 + 𝑎𝑖
Original Stixels
Slanted Stixels
![Page 7: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/7.jpg)
New dataset: SYNTHIA-San Francisco
7
• Generated with SYNTHIA toolkit to evaluate our algorithm, features slanted roads
• Photorealistic virtual sequence (2224 images), pixel-level depth and semantic ground truth
• Expensive to generate equivalent real-data sequence
![Page 8: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/8.jpg)
Results: Disparity Error
8
• Much better accuracy for slanted road sequence, similar for others
Metric Dataset Original Ours
Disp Err (%)Ladicky 17.3 16.9
KITTI 15 10.9 11.0
SYNTHIA-SF 30.9 12.9Lower is better
![Page 9: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/9.jpg)
Results: Semantic segmentation IoU
9
• Better accuracy for slanted road sequence, similar for others
Metric Dataset Original Ours
Disp Err (%)Ladicky 17.3 16.9
KITTI 15 10.9 11.0
SYNTHIA-SF 30.9 12.9
IoU (%)Ladicky 63.5 63.4
Cityscapes 65.7 65.8
SYNTHIA-SF 46.0 48.5 Higher is better
![Page 10: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/10.jpg)
Results: Frame-rate
10
• Our version is slightly slower because of the increased complexity
Metric Dataset Original Ours
Disp Err (%)Ladicky 17.3 16.9
KITTI 15 10.9 11.0
SYNTHIA-SF 30.9 12.9
IoU (%)Ladicky 63.5 63.4
Cityscapes 65.7 65.8
SYNTHIA-SF 46.0 48.5
Frame-rate(Hz)
KITTI 15 113 61
Cityscapes 20.9 6.6
SYNTHIA-SF 19.4 4.7 Higher is better
Stixels time measured on
6-core Intel i7
![Page 11: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/11.jpg)
Stixel Computation Complexity: Dynamic Programming
11
First-order Markov property: Only pair-wise Stixel relations are taken into account
Combinatorial explosion (of possible configurations): Dynamic programming to evaluate efficiently
Dynamic Programming
( h2 ), h = image height
Semantic Segmentation
Disparity Image
Ground Object Sky
Pixel Size
![Page 12: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/12.jpg)
Stixel Computation Complexity: Presegmentation
12
Dynamic Programming
( h’ x h’ ), h’ << h
Semantic Segmentation
Disparity Image
Ground Object Sky
• Infer possible Stixel cuts (presegmentation) from image• Avoid checking all possible Stixel combinations• If given the correct Stixel cuts, same accuracy (or better!)
Presegmentation
( h )
![Page 13: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/13.jpg)
Presegmentation Results: Disparity Error
13
• Depth error is higher because we do not check all combinations
Presegmentation
Metric Dataset Original Ours Original Ours
Disp Err (%)Ladicky 17.3 16.9 18.5 17.8
KITTI 15 10.9 11.0 11.8 11.7
SYNTHIA-SF 30.9 12.9 33.9 15.4
Lower is better
![Page 14: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/14.jpg)
Presegmentation Results: Semantic segmentation IoU
14
• The same semantic segmentation accuracy is maintained
Presegmentation
Metric Dataset Original Ours Original Ours
Disp Err (%)Ladicky 17.3 16.9 18.5 17.8
KITTI 15 10.9 11.0 11.8 11.7
SYNTHIA-SF 30.9 12.9 33.9 15.4
IoU (%)Ladicky 63.5 63.4 63.9 63.7
Cityscapes 65.7 65.8 65.7 65.8
SYNTHIA-SF 46.0 48.5 46.9 48.5
Higher is better
![Page 15: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/15.jpg)
Presegmentation Results: Frame-rate
15
• Presegmentation speeds up both original and Slanted Stixels
Presegmentation
Metric Dataset Original Ours Original Ours
Disp Err (%)Ladicky 17.3 16.9 18.5 17.8
KITTI 15 10.9 11.0 11.8 11.7
SYNTHIA-SF 30.9 12.9 33.9 15.4
IoU (%)Ladicky 63.5 63.4 63.9 63.7
Cityscapes 65.7 65.8 65.7 65.8
SYNTHIA-SF 46.0 48.5 46.9 48.5
Frame-rate(Hz)
KITTI 15 113 61 120 116
Cityscapes 20.9 6.6 36.6 27.5
SYNTHIA-SF 19.4 4.7 38.9 33.1
Higher is better
![Page 16: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/16.jpg)
Presegmentation Results: Frame-rate
16
Presegmentation
Metric Dataset Original Ours Original Ours
Disp Err (%)Ladicky 17.3 16.9 18.5 17.8
KITTI 15 10.9 11.0 11.8 11.7
SYNTHIA-SF 30.9 12.9 33.9 15.4
IoU (%)Ladicky 63.5 63.4 63.9 63.7
Cityscapes 65.7 65.8 65.7 65.8
SYNTHIA-SF 46.0 48.5 46.9 48.5
Frame-rate(Hz)
KITTI 15 113 61 120 116
Cityscapes 20.9 6.6 36.6 27.5
SYNTHIA-SF 19.4 4.7 38.9 33.1
Higher is better
Up to 2x
![Page 17: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/17.jpg)
Presegmentation Results: Frame-rate
17
Presegmentation
Metric Dataset Original Ours Original Ours
Disp Err (%)Ladicky 17.3 16.9 18.5 17.8
KITTI 15 10.9 11.0 11.8 11.7
SYNTHIA-SF 30.9 12.9 33.9 15.4
IoU (%)Ladicky 63.5 63.4 63.9 63.7
Cityscapes 65.7 65.8 65.7 65.8
SYNTHIA-SF 46.0 48.5 46.9 48.5
Frame-rate(Hz)
KITTI 15 113 61 120 116
Cityscapes 20.9 6.6 36.6 27.5
SYNTHIA-SF 19.4 4.7 38.9 33.1
Higher is better
Up to 7x
![Page 18: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/18.jpg)
Visual examples
18
Left Image Original Stixels Slanted Stixels
![Page 19: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/19.jpg)
Visual example: Avoid emergency break
19
Left Image Original Stixels Slanted Stixels
![Page 20: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/20.jpg)
Summary: Real sequence video
20
![Page 21: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/21.jpg)
Future work
21
• Improve presegmentation for better accuracy using a CNN
• Embedded GPU version on NVIDIA Drive PX 2 for autonomous driving
![Page 22: Francisco’s Steepest Streets · Fixed width stixels, variable number of stixels per column Widespread in stereo vision for automotive industry Stixel = Stick + Pixel. Motivation](https://reader036.vdocuments.mx/reader036/viewer/2022070915/5fb5fd3977d06008e135807b/html5/thumbnails/22.jpg)
Thank you
Daniel Hernández-Juárez*, Lukas Schneider*, Antonio Espinosa, David Vázquez,
Antonio M. López, Uwe Franke, Marc Pollefeys and Juan C. Moure
Autonomous University of Barcelona
More information:www.cvc.uab.es/people/dhernandez