![Page 1: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/1.jpg)
1
Andrea Censi 1
Low-Latency Event-Based Visual OdometryDavide Scaramuzza 2
Acknowledgments: Jonas Strubel, Christian Brandli, Tobi Delbruck, Matia Pizzoli, Christian Forster.
1 Laboratory for Information and Decision Systems Massachusetts Institute of Technology
2 Department of Informatics University of Zurich
slides available at censi.mit.edu/slides
![Page 2: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/2.jpg)
2
Low-Latency Event-Based Visual Odometry
![Page 3: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/3.jpg)
3
Low-Latency Event-Based Visual Odometry
recovering camera motion
![Page 4: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/4.jpg)
4
Low-Latency Event-Based Visual Odometry
brightness increased
brightness decreased
using a “neuromorphic” sensor
![Page 5: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/5.jpg)
4
Low-Latency Event-Based Visual Odometry
brightness increased
brightness decreased
using a “neuromorphic” sensor
![Page 6: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/6.jpg)
5
‣ What is a “neuromorphic” sensor?!
‣ Why would you want to use in robotics?
Come to the “Non-traditional Cameras” workshop on Thursday for technical details!
![Page 7: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/7.jpg)
6
![Page 8: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/8.jpg)
7
Slide by Christian Brandli
![Page 9: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/9.jpg)
‣ Traditional camera: frames at fixed intervals.
8
timeframe
![Page 10: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/10.jpg)
‣ Traditional camera: frames at fixed intervals.
8
timeframe next frame
![Page 11: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/11.jpg)
9
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
brightness
time
![Page 12: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/12.jpg)
9
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
brightness
time
![Page 13: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/13.jpg)
9
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
brightness
time
![Page 14: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/14.jpg)
9
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
brightness
time
![Page 15: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/15.jpg)
9
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
brightness
time
![Page 16: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/16.jpg)
9
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
brightness
time
![Page 17: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/17.jpg)
9
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
brightness
time
![Page 18: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/18.jpg)
9
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
brightness
time
![Page 19: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/19.jpg)
9
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
brightness
time
![Page 20: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/20.jpg)
9
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
brightness
time
![Page 21: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/21.jpg)
9
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
brightness
time
![Page 22: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/22.jpg)
9
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
brightness
time
![Page 23: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/23.jpg)
9
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
brightness
time
![Page 24: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/24.jpg)
10
time
events stream
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
![Page 25: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/25.jpg)
10
time
events stream
k-th event:
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
![Page 26: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/26.jpg)
10
time
events stream
timestamp (1 µs resolution)
k-th event:
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
![Page 27: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/27.jpg)
10
time
events stream
timestamp (1 µs resolution)
pixel coordinates
k-th event:
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
![Page 28: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/28.jpg)
10
time
events stream
timestamp (1 µs resolution)
pixel coordinates
sign (+1 or -1)
k-th event:
‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.
![Page 29: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/29.jpg)
‣ “Neuromorphic” sensors:!- Event-based !- Asynchronous and distributed
per-pixel processing!- Analog computation
11
Liu, Delbruck. Neuromorphic Sensory Systems. 2010 DOI:10.1016/j.conb.2010.03.00
![Page 30: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/30.jpg)
‣ “Neuromorphic” sensors:!- Event-based !- Asynchronous and distributed
per-pixel processing!- Analog computation
11
Liu, Delbruck. Neuromorphic Sensory Systems. 2010 DOI:10.1016/j.conb.2010.03.00
![Page 31: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/31.jpg)
‣ “Dynamic Vision Sensor” (DVS): a neuromorphic sensor developed by Delbruck’s group (ETH).!- Commercially available ($2.5k).
12
Lichtsteiner, Posch, Delbruck. A 128x128 120 dB 15µs Latency Asynchronous Temporal Contrast Vision Sensor. 2008 DOI:10.1109/JSSC.2007.914337
![Page 32: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/32.jpg)
13
‣ Earliest “silicon retina” developed by Carver Mead at Caltech.
Mead. Neuromorphic electronic systems. IEEE Proceedings, 1990!
![Page 33: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/33.jpg)
‣ The DVS is fast!
14
![Page 34: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/34.jpg)
‣ The DVS is fast!
14
Mueggler et al., IROS 2014
![Page 35: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/35.jpg)
‣ The DVS is fast!
14
CMOS
Mueggler et al., IROS 2014
![Page 36: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/36.jpg)
‣ The DVS is fast!
14
CMOS DVS
Mueggler et al., IROS 2014
![Page 37: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/37.jpg)
‣ The DVS is fast!
14
CMOS DVS
‣ DVS visualization: events histogram in a time slice.
Mueggler et al., IROS 2014
![Page 38: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/38.jpg)
15
‣ The DVS is really fast!
![Page 39: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/39.jpg)
15
‣ The DVS is really fast!
![Page 40: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/40.jpg)
16
1 video frame = 33 ms
1 video frame = 1 ms
1 video frame = 0.05 ms
‣ The DVS is really fast!
![Page 41: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/41.jpg)
16
1 video frame = 33 ms
1 video frame = 1 ms
1 video frame = 0.05 ms
‣ The DVS is really fast!
![Page 42: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/42.jpg)
17
“fast” sensorslow-latency control loop
commands
‣ Goal: low-latency control architecture.
![Page 43: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/43.jpg)
17
“fast” sensorslow-latency control loop
commands
“slow” sensors
‣ Goal: low-latency control architecture.
![Page 44: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/44.jpg)
17
“fast” sensorslow-latency control loop
commands
“slow”, “cognitive” level
“context”
“slow” sensors
‣ Goal: low-latency control architecture.
![Page 45: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/45.jpg)
18
DVS
CMOS camera
![Page 46: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/46.jpg)
19
“slow”, “cognitive” level
low-latency control loop
“context”
DVS
CMOS camera
![Page 47: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/47.jpg)
19
“slow”, “cognitive” level
low-latency control loop
“context”
DVS
CMOS camera
![Page 48: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/48.jpg)
20
Event-Based Visual Odometry
‣ Goal: estimate the camera motion from the events.
![Page 49: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/49.jpg)
‣ Large solution space!- use additional sensors? !
this work: yes, use additional CMOS sensor!- use “features”? !
this work: no, use raw events
20
Event-Based Visual Odometry
‣ Goal: estimate the camera motion from the events.
![Page 50: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/50.jpg)
21
(calibration details in the Thursday workshop)
DVS events timeCMOS frames
![Page 51: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/51.jpg)
21
(calibration details in the Thursday workshop)
DVS events timeCMOS frames
![Page 52: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/52.jpg)
21
(calibration details in the Thursday workshop)
DVS events timeCMOS frames
![Page 53: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/53.jpg)
22
DVS events timeCMOS frames
![Page 54: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/54.jpg)
22
- one event doesn’t tell much!- low resolution (128 × 128)!
DVS events timeCMOS frames
![Page 55: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/55.jpg)
22
- one event doesn’t tell much!- low resolution (128 × 128)!
DVS events timeCMOS frames
- you see incremental motion!- you can decouple the motion along the axes
![Page 56: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/56.jpg)
23
DVS events timeCMOS frames
![Page 57: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/57.jpg)
23
‣ Idea: reduce the problem to “localization” with respect to the previous CMOS frame.
DVS events timeCMOS frames
![Page 58: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/58.jpg)
24
The 1D case — pure rotation
time
pixel
![Page 59: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/59.jpg)
24
The 1D case — pure rotation
time
pixel
![Page 60: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/60.jpg)
24
The 1D case — pure rotation
time
pixel
![Page 61: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/61.jpg)
24
The 1D case — pure rotation
time
pixel
![Page 62: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/62.jpg)
24
The 1D case — pure rotation
time
pixel
estimated velocity
![Page 63: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/63.jpg)
24
The 1D case — pure rotation
time
pixel
estimated velocity
![Page 64: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/64.jpg)
24
The 1D case — pure rotation
time
pixel
estimated velocity
![Page 65: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/65.jpg)
24
The 1D case — pure rotation
time
pixel
estimated velocity
![Page 66: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/66.jpg)
24
The 1D case — pure rotation
time
pixel
estimated velocity
![Page 67: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/67.jpg)
24
The 1D case — pure rotation
time
pixel
estimated velocity
![Page 68: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/68.jpg)
24
The 1D case — pure rotation
time
pixel
estimated velocity
![Page 69: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/69.jpg)
24
The 1D case — pure rotation
time
pixel
estimated velocity
![Page 70: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/70.jpg)
24
The 1D case — pure rotation
time
pixel
estimated velocity
![Page 71: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/71.jpg)
25
estimated velocity
![Page 72: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/72.jpg)
25
estimated velocity
![Page 73: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/73.jpg)
‣ Very good accuracy for rotation !drift comparable to visual odometry
26
![Page 74: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/74.jpg)
‣ Very good accuracy for rotation !drift comparable to visual odometry
26
1.0°-8.9°
2.1°
2.1°
90°
odometry estimate
truth0°
![Page 75: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/75.jpg)
‣ Very good accuracy for rotation !drift comparable to visual odometry
26
‣ Poor tracking of translation!very low SNR due to small apparent motion!(also: needs to know the image’s depth map)
1.0°-8.9°
2.1°
2.1°
90°
odometry estimate
truth0°
![Page 76: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/76.jpg)
‣ DVS: Interesting new sensor for agile robots!
‣ Goal: low-latency control architectures!
‣ So far: worked on inference problems!- blinking LEDs tracking (IROS 2013)!- visual odometry w/CMOS fusion (this work)!- feature-based visual odometry (Mueggler et al., IROS 2014)
27
Summary
Come to the “Non-traditional Cameras” workshop on Thursday for technical details!
![Page 77: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking](https://reader035.vdocuments.mx/reader035/viewer/2022063008/5fbdfc217ae78c67c469765d/html5/thumbnails/77.jpg)
‣ DVS: Interesting new sensor for agile robots!
‣ Goal: low-latency control architectures!
‣ So far: worked on inference problems!- blinking LEDs tracking (IROS 2013)!- visual odometry w/CMOS fusion (this work)!- feature-based visual odometry (Mueggler et al., IROS 2014)
27
Summary
Come to the “Non-traditional Cameras” workshop on Thursday for technical details!