Autonomous Vehicle with Obstacle Detection and Avoidance using Image Processing

Project Report

Team:- Praveen Yadav, Ravi Raj, Nandkishor Guptaunder the guidance of Associate Prof. Debashis GhoshDate:- 8th May 2014

CANDIDATES DECLARATION

We hereby declare that the work presented in this report entitled, Autonomous Vehicle with Obstacle Detection and Avoidance using Image Processing submitted by us in partial fulfillment for the award of the degree of Bachelor of Technology in Electronics & Communication Engineering, submitted in the Department of Electronics & Communication Engineering, Indian Institute of Technology Roorkee (India), is an authentic record of our own work carried out under the supervision and guidance of Dr. Debashis Ghosh, Associate Professor, Department of Electronics & Communication Engineering, Indian Institute of Technology Roorkee. The matter embodied in this report has not been submitted by us anywhere else for the award of any other degree or diploma.

Praveen Yadav (10116037) Ravi Raj (10116040)Nandkishor Gupta (10116034)

Date: 8th May 2014Place: Roorkee

SUPERVISORS CERTIFICATE

This is to certify that the above statement made by the candidates is correct to the best of my knowledge and belief.

Dr. Debashis GhoshAssociate ProfessorE&CE DepartmentIndian Institute of Technology Roorkee

Roorkee-247667Date: 8th of May,2014 Place: RoorkeeAbstract

This report describes a cost-effective design of an autonomous vehicle that can detect and avoid obstacles in real time. Obstacles include still and moving obstacles provided the speed of the moving obstacles is within certain limits. This is based on processing the image obtained from a camera mounted on the vehicle itself and planning the path of the vehicle accordingly. The design has the ability to sense the environment continuously as limited by the processing speed to ensure that moving obstacles can also be detected and avoided. We have used the MatLab for image processing and control signals have been send to the vehicle through microcontroller. This vehicle successfully avoided different kinds of obstacles such as bottles, wooden blocks and walls.

Acknowledgements

It gives us great pleasure to take this opportunity to thank and express our deep sense of gratitude to our guide Dr. Debashis Ghosh, Associate Professor, Department of Electronics & Communication Engineering, Indian Institute of Technology Roorkee, for providing his valuable guidance, constant motivation, patience and enthusiasm and most important of all with an opportunity to do our project work on the topic Autonomous Vehicle with Obstacle Detection and Avoidance using Image Processing. Also, we are thankful to God and our parents who have been a constant source of inspiration for us.

Table of Contents

Abstract4Acknowledgements5Table of Contents6List of Figures7Introduction8Literature Review10Methodology19Findings and Results29Conclusion31Discussion32Appendices33References 44

7

6

13

List of Figures

1. Edges and derivatives of their intensityFig. 2.1.2. Different Edge Detection Masks Fig. 2.2.3. Fitting of structuring elements S1,S2 Fig.2.3.4. Arduino Pin Diagram Fig.2.4.5. Pin Diagram of L298N Fig.2.5.6. Original Image Fig.3.1.7. Edge Detected Image Fig.3.2.8. Side Filled from below Fig.3.39. Erosion Fig.3.4.10. Farthest accessible point Fig.3.511. Vehicle Control Fig.3.6.12. Different process for the floor containing more than one pixel colorFig 3.713. Floor Finder TechniqueFig.3.8.14. Floor Detection Integration Fig.4.1.

Chapter 1: INTRODUCTION

An autonomous vehicle is a self-piloted vehicle that does not require an operator to navigate and accomplish its tasks. Autonomous vehicles are a recently developed subset of robotics and can come in three general forms; air, ground and submarine. Many companies and organizations are spending a lot of time and money developing autonomous vehicles for numerous applications. Possible applications include precision formation flying, investigating areas too dangerous for humans (i.e. chemical fires or radioactive areas), checking line integrity of underwater pipelines, exploring the surface of other planets, defense vehicles used during wars.One possible task of an autonomous vehicle is to navigate a pre-programmed route while avoiding any obstacles the vehicle may encounter. The vehicle can accomplish this task by using sensors to see where it is and what is around it. These sensors vary from close range infrared sensors to longer ranged high frequency radar and global positioning satellites. Using cameras or computer vision instead of infrared or ultrasonic sensors is also an alternative that is explored in this project. One of the major challenges of this project was to keep it low-cost and to make it adaptable to any kind of surface not just a plain road that has no variations.The design is divided into three parts for better implementation and understanding i.e. processing, control and feedback each of which has its own functions and limitations. The processing part basically deals with capturing frames at regular intervals from camera and processing image for path planning whereas control part has operations related to the actual movement of the vehicle. Its core lies in sending appropriate control signals to vehicle through MatLab-Arduino Interface. The last part feedback keeps a check on the control part of the vehicle. It helps in dynamically determining the accurate amount of turn that will be required in order to cross an upcoming obstacle without colliding with it..

Processing part can also be referred to as obstacle detection and control and feedback part combinedly represent the obstacle avoidance module.This design has another module that makes the vehicle adaptable to any kind of surface on which it is moving. The surfaces can range from a plain white sheet of paper to a more complex, random concrete floor with multi-colored marbles embedded in it. It is capable of moving on any of them. We have named this module as floor finder details of which will be presented in further sections.Most of the hardware in the project are used considering the low cost model in mind. Obviously it resulted in some limitations but overall its a good model keeping in mind the resources that we had at our disposal.We have used Intex IT-306 PC Webcam for image capturing that has very narrow viewing angle. All the algorithms have been implemented in MatLab and the processor used for image processing is of the laptop itself (Intel Core i5).Microcontroller used for sending control signal to motors is Arduino ATMega-32 and motor driver is L298N.

INTRODUCTION

LITERATURE REVIEW

Chapter 2: LITERATURE REVIEW

Computer vision is a subfield of artificial intelligence in computer science. It is often described as essentially the emulation of human vision by a machine. This is not considered as entirely accurate by many computer scientists since despite both forms of vision being reliant on light radiated from the environment, the principles behind the implementation of human and computer vision are far from similar. In order to perceive the world through vision, the machine must be able to recognize certain characteristics of its immediate environment. This is done through the analysis of images taken by a camera. The ability to recognize and differentiate textures, shapes, colors etc. is vital to the replication of human vision which is the ultimate goal of all computer vision research. Perhaps the greatest challenge to researchers in the field is the interpretation of 2D images into 3D scenes since a significant proportion of a scenes characteristics are absent in its image. Moreover, a prior knowledge of the environment (e.g. lighting, texture, background etc.) is required since they affect the appearance of objects which the computer is trying to describe.

Computer vision problems can be subdivided into low-level ones and high level ones. The former refers to the initial stages of image processing with very little other knowledge of the environment and includes operations such as edge detection, boundary drawing etc. whereas the latter refers to highly task-specific operations performed on presumably perfect data from low-level processes with some or no additional data and includes operations such as obstacle detection, face recognition etc.

Some of the techniques that are generally used in computer vision and also have been used in this project are explained below

2.1 Edge Detection:-

Edges are characterized by abrupt changes on an image indicating boundaries or other features. The detection of edges is vital in computer vision in accomplishing several operations such as segmentation, object modeling, registration etc. Most of the edge detection algorithms are based on the gradient of intensity on the two sides of an edge. That gradient is used to detect edges. shows parts of two images with varying intensity along the horizontal. The intensity function of each pixel and its derivatives are vital in detecting edges on any object as is obvious by observing the distinct characteristics of these derivatives at edges.

Edge detection contains three steps namely Filtering, Enhancement and Detection.

The overview of the steps in edge detection are as follows:

Filtering:- Images are often corrupted by random variations in intensity values, called noise. Some common types of noise are salt and pepper noise, impulse noise and Gaussian noise. Salt and pepper noise contains random occurrences of both black and white intensity values. However, there is a trade-off between edge strength and noise reduction. More filtering to reduce noise results in a loss of edge strength.

Enhancement:- In order to facilitate the detection of edges, it is essential to determine changes in intensity in the neighborhood of a point. Enhancement emphasizes pixels where there is a significant change in local intensity values and is usually performed by computing the gradient magnitude.

Detection:- Many points in an image have a nonzero value for the gradient, and not all of these points are edges for a particular application. Therefore, some method should be used to

determine which points are edge points. Frequently, thresholding provides the criterion used for detection.

Fig. 2.1. Edges and derivatives of their intensity

Several edge detection methods are used to detect edges. The Sobel, Prewitt and Roberts methods find edges using their approximations to the derivative returning points where the gradient is maximum at edges. Some of edge detection techniques are shown in fig. 2.2:

Fig. 2.2. Different Edge Detection Masks

2.2 Canny Edge Detection :-

Canny Edge Detection is used in this project so it has been explained here. Canny Edge detection comprises of following processes:2.2.1 Noise Reduction: (Filter any noise using Gaussian Filter)Filters based on Gaussian functions are of particular importance because their shapes are easily specied and both the forward and inverse Fourier transforms of a Gaussian function are real Gaussian functions. Further if the frequency domain lter is narrower, the spatial domain lter will be wider which attenuates the low frequencies resulting in increased smoothing/blurring. These Gaussian lters are typical linear lters that have been widely used for image denoising. In

Gaussian lters, weight of the pixels decays by distance from the center of the lter as given by equation 2.1 (2.1)Gaussian lters assume that images have smooth spatial variations and pixels in a neighborhood have close values, by averaging the pixel values over a local neighborhood suppresses noise while preserving image features. However, this assumption fails at edges where the spatial variations are not smooth and the application of Gaussian lter blurs the edges2.2.2 Finding Intensity Gradient of the imageAn edge in an image may point in a variety of directions, so the Canny algorithm uses four filters to detect horizontal, vertical and diagonal edges in the blurred image. Theedge detection operator(Roberts,Prewitt,Sobelfor example) returns a value for the first derivative in the horizontal direction (Gx) and the vertical direction (Gy). From this the edge gradient and direction can be determined by equation 2.2: (2.2)

2.2.3 Non-maximumsuppression is applied. This removes pixels that are not considered to be part of an edge. Hence, only thin lines (candidate edges) will remain. Non-maximum suppression is anedge thinningtechnique. Given estimates of the image gradients, a search is carried out to determine if the gradient magnitude assumes a local maximum in the gradient direction. At every pixel, it suppresses the edge strength of the center pixel (by setting its value to 0) if its magnitude is not greater than the magnitude of the two neighbors in the gradient direction.

2.2.4 Hysteresis:

The final step. Canny does use two thresholds (upper and lower)If a pixel gradient is higher than theupperthreshold, the pixel is accepted as an edge

If a pixel gradient value is below thelowerthreshold, then it is rejected. If the pixel gradient is between the two thresholds, then it will be accepted only if it is connected to a pixel that is above theupper threshold.

2.3 Side Fill :This works by starting at the bottom of the image and proceeding vertically pixel by pixel filling each empty black pixel until a non-black pixel is seen. The filling then stops that vertical column and proceeds with the next.

2.4 Erosion :The erosion of binary image f by a structuring element s (denoted by f - s) produces a new binary image g with ones in all locations (x,y) of a structuring elements origin at which that structuring element s fits the input image f, i.e. g(x,y) = 1 is s fits f and 0 otherwise, repeating for all pixel coordinates (x,y). When a structuring element is placed in a binary image, each of its pixels is associated with the corresponding pixel of the neighborhood under the structuring element. The structuring element is said tofitthe image if, for each of its pixels set to 1, the corresponding image pixel is also 1.Fitting of binary image with structuring elements are shown in fig. 2.3.

Fig.2.3. Fitting of structuring elements S1,S2Structuring element S1 fits only with A, and structuring element S2 fits with A,B.

2.5 Arduino Atmega 328Arduinois a popularopen-sourcesingle-board microcontroller, descendant of the open-sourceWiring platform,designed to make the process of using electronics in multidisciplinary projects more accessible. The hardware consists of a simple open hardware design for the Arduino board with anAtmel AVRprocessor and on-boardinput/outputsupport. The software consists of a standard programming language compiler and theboot loaderthat runs on the board.Arduino hardware is programmed using a Wiring-based language (syntax and libraries), similar toC++with some slight simplifications and modifications, and aProcessing-basedintegrated development environment.Pin diagram of Arduino ATmega328 is shown in fig.2.4. Pins named D0-D4, D5-D13 are digital input/otput pins (out of which 6 can be used as PWM) and pins A0-A5 are analog input/output pins.

Fig.2.4. Arduino Pin Diagram2.6 Motor Driver (L298N):LITERATURE REVIEW

LITERATURE REVIEWThe L298 is an integrated monolithic circuit in a 15-lead Multi-watt and PowerSO20 packages. It is a high voltage, high current dual full-bridge driver designed to accept standard TTL logic levels and drive inductive loads such as relays, solenoids, DC and stepping motors. Two enable inputs are provided to enable or disable the device independently of the input signals. The emitters of the lower transistors of each bridge are connected together and the corresponding external terminal can be used for the

connection of an external sensing resistor. An additional supply is provided so that the logic works at a lower voltage.The pin diagram of L98N is shown in fig.2.5

Fig.2.5. Pin Diagram of L298NLITERATURE REVIEW

20

Chapter 3: METHODOLOGY

Our obstacle detection system is purely based on the appearance of individual pixels. Any pixel that differs in appearance from ground classified as an obstacle. The method is based on three assumptions that are reasonable for a variety of indoor and outdoor environments:1. Obstacles differ in appearance from ground.2. The ground is relatively flat.3. There are no overhanging obstacle.

The whole design is basically divided into three modules1. Processing2. Control3. Feedback

3.1 Processing Module:-The first technique that exploits the differences between obstacle and ground uses a canny edge detector to produce edge only version of previous image fig.3.1. Using this we get an image look like fig.3.2.

Fig.3.1. Original Image

Fig.3.2. Edge Detected Image

We can see that the obstacles are outlined by the edge detection routine. This helps to identify the objects but still does not give us a correct bearing on what direction to go in order to avoid the obstacles.The next step is to understand which obstacles would be hit first if the vehicle moved forward. To start this process we use SideFill to fill in the empty space at the bottom of the image as long as an edge is not encountered. This works by starting at the bottom of the image and proceeding vertically pixel by pixel filling each empty block pixel until a non-black pixel is encountered. The filling then stops that vertical column and proceeds with the next. The image we get is fig. 3.3.

Fig.3.3 Side Filled from belowSingle width vertical lines that appear in the image are quickly noticeable. These are caused by holes where the edge detection routine failed. As they specify potential paths that are too thin, we want to remove them so that these are not considered as possible candidates for available vehicle paths.

We can do this by just eroding or shrinking the current image horizontally by an amount large enough such that the resulting white areas would be large enough for the vehicle to pass without hitting any obstacle. After Erosion, we get the image which looks like fig.3.4.

Now that we have all possible paths we will pick our goal direction, which will be from starting point to the farthest accessible point (X) as indicated in fig. 3.5.

Fig.3.4. Erosion Fig.3.5 Farthest accessible point

The location of highest accessible point is known, vehicle will move according to different criteria set to steer vehicle right, left or straight, or it will stop. Movement of vehicle is explained in fig.3.6, where s is stop criterion, tl and tr are turn criterion, midx is midpoint of the image, and (X,Y) is the pixel of farthest accessible point.This algorithm works reasonably well as long as the floor is single color. We can notice the failure of this technique in the fig.3.7 which if fully processed would only see non-obstacle space in the lower 10 pixels of the image. This technique is not applicable for the floor which contains more than one pixel color.

3.2 Vehicle Control and Feedback:-Vehicle control module controls the direction in which the vehicle has to move or it has to stop and the feedback part controls the amount of movement in either direction.

All these decisions will be taken according to some dynamically defined parameters which we have referred to here as criterions. It is the effectiveness in computing these criterions which will determine the performance of the vehicle.Farthest Accessible Point=(X, Y)Mid Point of Image(midx)Turn Criterion (tl, tr)Stop Criterion(s)

Details of the vehicle control and feedback system are presented in the form of a flowchart here for the ease of understanding

Fig.3.6. Vehicle Control

Resulting Issues if the floor is not uniform:-As presented above this method works only when the floor on which vehicle is moving is uniform or we can say have pixels of almost similar colors. The idea behind this is that the edge detection technique

that is used here should not detect any edges due to the variations in floor. Fig. 3.7 depicts the possible that could arise if this method without any improvement is used on a floor with variations. Fig 3.7 Different process for the floor containing more than one pixel color

It can be seen clearly that edges are detected for every different colored pixel on the floor. It results in erroneous results from further processing of the edge detected image and makes the navigation practically impossible.To improve on our design we need to understand that the carpet or floor plane contains more than one pixel color. While we could detect both colors and perhaps merge them in some way an easier approach is to make an assumption that the immediate foreground of the vehicle is obstacle free. For this we have used Floor Finder module in which a rectangular area in front of the mobile vehicle is used for reference. Floor Finder module assumes that a small rectangular area in immediate front of the vehicle is floor or we can call it obstacle free area. Based on this assumption, it will find out the areas which can correspond to floor on an average. First step is to crop a rectangular strip in front of vehicle and break it into three R, G and B channels. Next computes and average of pixel value in all of these channels and saves it for reference.Now, it scans the whole image with small shaped cells, breaks each into three RGB channels and compute means of their pixel value. If the absolute difference between means of all the three channels of a small squared cell and that rectangular strip lies within certain predefined limits for any cell, it will regard that cell as a part of floor and makes it white. If the difference is beyond limits then that cell doesnt corresponds to floor area and it leaves it as it is.The dimensions of squared shaped cell has to be chosen intelligently keeping in mind the amount of variations in the floor. If the dimensions are too small then it may happen that sometimes it covers only different colored pixels(variations) in the floor and cannot recognize it as floor. If the dimensions are too

large then the resulting image will give large white blocks that can be seen as a loss of information from the image.Instead of square other shapes likes triangle and hexagonal can be used for specific cases but those will be computationally costly.Different steps of floor finder is shown in fig.3.8.

Fig.3.8. Floor Finder Technique

METHODOLOGYMETHODOLOGY

Chapter 4: RESULTS AND FINDINGS

On successful execution of the code for image processing and vehicle control we were able to obtain the desired results for edge detection and other techniques used in the process. After properly assembling all the hardware components of the vehicle and interfacing the microcontroller with MatLab for controlling vehicle, a well-functioning vehicle is designed which can detect and avoid different obstacles like bottles, walls, wooden blocks and everything else in real time.As an improvement the module for floor detection has also been added and obtained its successful integration with the working model which is shown in fig.4.1.

Limitations:-

While working on this project we came across a lot of difficulties some of which were resolved and others were just impossible to resolve given the amount of resources. So, in the end it is necessary to deal with some of limitations of this design due to its low cost implementation. First limitation in this arises due to the limited viewing angle of the camera used in the project. Its viewing angle is as narrow as 65 degrees and design can be improved substantially if a wide lens camera is used.Another major limitation of this design is processing speed. The vehicle has to stop for a while it processor processes the image and send control signals to the arduino. It cannot move non-stop as could have been the case if a dedicated image processor is used for the purpose. Also it would have removed the cables that are used to interface the vehicle with the computer.Due to limited processing speed, it is difficult for the vehicle to avoid moving obstacles if their speed is more than a certain limit and it can collide with them.

29

Fig.4.1. Floor Detection Integration

RESULTS AND FINDINGS

30Chapter 5: CONCLUSION

The purpose of our project was to create an inexpensive and efficient obstacle detection and avoidance robot. In conclusion of this report, the robot was able to display its turning strategy according to its distance from obstacle. The specific work that has been done is as follows: Camera with satisfactory performance is used at a relatively low cost, fixed them and adjusted them for good computer vision system. After attaching the camera, we successfully developed the software to detect obstacle and send out signals to control the motors with MATLAB. The serial communication established between laptop and Arduino through USB port helped in sending out a signal for the motor condition to Arduino board. After receiving the signal the motors will steer accordingly.

Though this design has some limitations as described earlier, but it is reasonably good model to demonstrate the obstacle detection and avoidance system at this level and investement.

Chapter 6: DISCUSSION

In order to optimize the movement of the robot, we have many considerations for improvement. However, most of these ideas will cost more money and time as well. For the choice of cameras, it is better to get CCD or industrial use ones to get clear and fast pictures. The speed of the processor is a problem because low frame rates will result in slow picture output. For choice of board, a faster and more efficient board will be more challenging. We can use AVR, ARM and dedicated DSPs for optimal system.

Practical Applications of this project:-With the motivation with which this project was undertaken and its possible contribution to the real world if developed by industry professionals, some of the practical applications of the project are listed here:- Can build driver-free war vehicle and have other important applications in Defense. Future on-road vehicles and autonomous transport system. Surveillance vehicles for some of the hostile areas where it is difficult for humans to work. Mars Rovers.

APPENDIX ASource Code: Arduino-MatLab interface:a=arduino('COM3');a.pinMode(3,'output')a.pinMode(4,'output')a.pinMode(5,'output')a.pinMode(6,'output')a.pinMode(9,'output')a.pinMode(10,'output')

Image Acquisition

vidobj=videoinput('winvideo',2,'YUY2_160x120');vidObj.Timeout = 15;set(vidobj, 'ReturnedColorspace', 'RGB');%vidobj.FrameGrabInterval=6;vidobj.FramesPerTrigger=Inf;framesneeded=Inf; flushdata(vidobj);start(vidobj);pause oncount=1;im=getsnapshot(vidobj);IC=imcrop(im,[0,100,160,20]);imshow(IC)redChannel = im(:, :, 1);greenChannel = im(:, :, 2);blueChannel = im(:, :, 3);dR=mean2(redChannel)dG=mean2(greenChannel)dB=mean2(blueChannel)irc=im;pause(1)i=1;while i