imecs 2017: in situ real-time vision-based lane detection
TRANSCRIPT
![Page 1: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/1.jpg)
In Situ Real-Time Vision-Based Lane Detection on Raspberry Pi using 1D Haar Wavelet Spikes
Vladimir KulyukinDepartment of Computer Science
Utah State University
IMECS 2017
![Page 2: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/2.jpg)
Outline
IntroductionLane Detection with 1D Haar Wavelet Spikes EvaluationConclusions
![Page 3: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/3.jpg)
Introduction
![Page 4: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/4.jpg)
Self-Driving Cars
Pros– Enhanced mobility for the elderly and disabled– Reduction in traffic accidents
Cons– Major job losses in driving and transportation industries– Loss of privacy and increased risk of hacking attacks
For now, human drivers are still indispensable
![Page 5: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/5.jpg)
Related Work
S. Mallat and W. L. Hwang. “Singularity Detection and Processing with Wavelets.” IEEE Trans on Information Theory, vol. 38, no. 2, pp. 617 – 643, March 1992
Y. Wang, E. Teoha, D. Shen. “Lane detection and tracking using B-snake.” Image and Vision Computing, vol. 22, pp. 269-280, 2008
Z. Kim. “Robust lane detection and tracking in challenging scenarios.” IEEE Trans. on Intelligent Transportation Systems, vol. 9, no. 1, pp. 16 – 26, Mar. 2008
P. Hsiao, C. Yeh, S. Huang, L. Fu. “A portable vision-based real-time lane departure warning system: day and night.” IEEE Trans. on Vehicular Technology, vol. 58, no. 4, pp. 2089 – 2094, May 2009
P. Mandlik, A. Deshmukh. “Raspberry-pi based real time lane departure warning system using image processing.” International Journal of Engineering Research and Technology, vol. 5, issue 06, June-2016, pp. 755 – 762
![Page 6: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/6.jpg)
Lane Detection with
1D Haar Wavelet Transform Spikes
![Page 7: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/7.jpg)
Up-Down Spikes
Up-down spikes
![Page 8: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/8.jpg)
Down-Up Spikes
Down-up spikes
![Page 9: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/9.jpg)
Hardware
The hardware consists of an RPi 3 Model B ARM v8 1GB RAM computer, an RPi Camera Board v2, an RPi Night Vision Camera, and a 7 inch RPi touchscreen
The hardware is placed inside a Jeep Wrangler with a camera attached to the windshield from the inside and is powered through a regular 12V-to-5V car charger
As the car moves along a road, the detected lanes are drawn in real time in a small bottom right window on the touchscreen
![Page 10: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/10.jpg)
HaarSpiker: Spike Detection Algorithm
HaarSpiker is an algorithm that uses 1D Haar Spikes to detect lanesThe algorithm is implemented in Python 2.7.9 with OpenCV 3.0.0The algorithm consists of three phases: pre-processing, spike detection, line fitting
![Page 11: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/11.jpg)
Preprocessing: Cropping a Region of Interest (ROI)
A 360 x 240 PNG image is taken and a 56 x 200 ROI is cropped in the bottom center portion of the image where road is likely to be
![Page 12: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/12.jpg)
Preprocessing: Grayscale, Blur, ThresholdGrayscale cropped ROI
![Page 13: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/13.jpg)
Preprocessing: Grayscale, Blur, ThresholdGrayscale cropped ROI
Blur with Gaussian 7x7 kernel
![Page 14: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/14.jpg)
Preprocessing: Grayscale, Blur, ThresholdGrayscale cropped ROI
Blur with Gaussian 7x7 kernel
Threshold with Otsu
![Page 15: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/15.jpg)
Spike Detection
- Take image row segments on both sides of the image- Each segment is 64 pixels long (but this is a parameter)- Apply 2 iterations of Haar Wavelet Transform to each row segment- Detect up-down spikes in each row and compute the means of the flat segments for each detected spike- Adjust the start and end row segments on the basis of where the mean is detected in the previous row
![Page 16: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/16.jpg)
Spike Detection
- Do you go top to bottom or bottom to top? Bottom to top, because you want to detect what is closest to the car - Do you process every row? No, go up in increments of 5 rows. This is a parameter that can be adjusted
![Page 17: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/17.jpg)
Line Fitting
1D polynomial line fitting is applied to the (x, y) spike positions The lines are filtered by inclination angles:
- Left line threshold values are -60° to -30°- Right line threshold values are 30° to 60°
The final left and right lines indicate left and right lanes
![Page 18: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/18.jpg)
Ideal Case of Lane Detection
![Page 19: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/19.jpg)
Evaluation
![Page 20: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/20.jpg)
Data
All the images taken from a Jeep Wrangler going at a speed of 60 mph on State Route 30 in Northern Utah
Sample 1: October 12th, 2016, a sunny day, consists of 1000 PNG images
Sample 2: November 12th, 2016, a cloudy day, consists of 1000 PNG images
Sample 3: January 6th, 2017, a snowy night, consists of 1000 PNG images
Sample 4: January 7th, 2017, a snowy day, consists of 775 PNG images
Image Size: 360 X 240 pix
![Page 21: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/21.jpg)
Results
Sample Num. of Images Both Lanes Detected (%) At Least 1 Lane Detected (%)
1 1000 61.90 91.20
2 1000 34.10 77.40
3 1000 16.90 64.10
4 775 15.74 57.03
![Page 22: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/22.jpg)
Results
All true negatives in sample 1 are caused by shadows cast by other vehicles or trees and changes in road surface texture
The algorithm’s accuracy on sample 2 which is 74.1% is worse due to faded road lanes on a cloudy day
![Page 23: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/23.jpg)
Results
The algorithm’s accuracy drops to 64.10% on sample 3 on a snowy night
The algorithm’s accuracy deteriorates to 57.03% on sample 4 on a snowy day
![Page 24: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/24.jpg)
ConclusionsThe algorithm processes 22 images/second in situ at a speed of 60 mph
The system’s hardware can be placed inside a car, next to the windshield, and can be powered through a regular 12V-to-5V car charger, i.e. the power requirement is 10W
The hardware and software components of the presented algorithm can be replicated with off-the-shelf hardware components and open source software
The algorithm performs well in fair weather; the performance declines at night and on snowy roads
![Page 25: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/25.jpg)
Data and Source Code Availability
Images used in this study are available at https://www.dropbox.com/sh/yqpq0adt42n54dt/AAA6m5OF4s_C2KYYDLI4kc2ra?dl=0
Py code is available at https://github.com/VKEDCO/PYPL/tree/master/haar_spiker
![Page 26: IMECS 2017: In Situ Real-Time Vision-Based Lane Detection](https://reader033.vdocuments.mx/reader033/viewer/2022042907/58d1b5161a28ab98278b5301/html5/thumbnails/26.jpg)
Thank You!