“pickabo” · 2014. 12. 9. · “pickabo” chaudhari anup s december 9, 2014 university of...

“Pickabo” Chaudhari Anup S

December 9, 2014

University of Florida

EEL 5666 – Intelligent Machines Design Lab (IMDL)

Instructors: Dr. A. Antonio Arroyo, Dr. Eric M. Schwartz

Teaching Assistants: Andy Gray, Nick Cox

Table of Contents Abstract

…………………………………………………………………………………………………………1

Executive Summary

…………………………………………………………………………………………………………1

Introduction

…………………………………………………………………………………………………………1

Integrated System

…………………………………………………………………………………………………………2

Mobile Platform

…………………………………………………………………………………………………………3

Actuation

…………………………………………………………………………………………………………3

Sensors

…………………………………………………………………………………………………………3

Detailed Description of the Special Sensor System

…………………………………………………………………………………………………………4

Behaviours

…………………………………………………………………………………………………………5

Experimental Layout and Results

…………………………………………………………………………………………………………6

Conclusion

…………………………………………………………………………………………………………6

Appendix

…………………………………………………………………………………………………………7

Abstract: The project aims towards building an autonomous robot that will be able to search for red

coloured balls, grab them and take them towards the base stations.

Executive Summary: To achieve the expected objective the robot must be able to perform following tasks:

1. Navigate in a room in search of ball.

2. Recognize the ball and its colour.

3. Grab the ball using a mechanical system.

4. Search for the base station and go towards it to drop the ball.

Introduction: Pickabo is an autonomous robot intended to detect and grab red coloured spherical objects.

Pickabo is primarily controlled by an Arduino Mega but all of the vision processing is

handled by an on-board HP ENVY J001TX laptop running a version of Ubuntu. Locomotion is

provided by two independent motors and everything is powered by a 5000 mAh lithium

polymer battery. Pickabo uses a webcam to detect the red spheres and ultrasonic sensors to

avoid obstacles. Pickabo identifies the targets using algorithms native to OpenCV.

This report details the hardware used on Pickabo and provides a high-level description of

Sentinel’s software and behavior architecture. Descriptions of the tests performed as well as

ideas for future work are provided at the end. Finally, full copies of the software can be

found in the appendices.

Integrated System:

Integrated System

Pickabo has two main processors on board.

1. The main processor is an HP ENVY J001TX laptop which is responsible for vision

processing and behavioural algorithms.

HP ENVY J001TX

2. The other processor is an Arduino Mega 2560. It has a serial communication with the

laptop and is used to send commands to the drive motors and the grabber servo.

The Arduino also reads inputs from the ultrasonic sensors and relay that information

to the laptop.

Arduino Mega 2560

Mobile Platform: The platform of the robot consists of four wheels. Two of these wheels are driver wheels

(hooked to DC motors) that allow it to use differential drive to move in its surroundings. These

wheels are situated towards the front of the platform. The other two wheels are caster wheels

situated towards the back of the platform. This allows for simple control of the robot while

still maintaining stability.

Actuators: 1. 2 DC motors for differential drive.

Arduino motor shield which lets you drive two DC motors with Arduino board,

controlling the speed and direction of each one independently.

2. Servo motor for grabbing mechanism.

Sensors: 1. Pickabo uses a web camera to capture a real time video.

PlayStation 3 Eye

PlayStation 3 Eye

2. Pickabo has three Ultrasonic sensors to determine the distance of obstacles from the

bot.

PING))) Ultrasonic Sensor

PING))) Ultrasonic Sensor

Detailed Description of the Special Sensor System: Machine vision is one of the most effective and low-cost methods for detecting any object.

Pickabo’s special sensor is the PlayStation 3 Eye web camera that allows it to perform the

task of locating coloured balls. While the other sensors allow it to drive around and avoid

obstacles, it is the webcam that allows Pickabo to collect critical data about its environment.

The webcam, ultrasonic sensors and actuators work in unison in order for Pickabo to make

intelligent decisions and complete the objective.

The vision processing is handled by the laptop running Ubuntu 14.04 LTS and an algorithm

developed using OpenCV 2.4.10. The algorithm first captures a temporary image i.e. a frame

from the video and converts it from BGR to HSV. Then it eliminates all other colours from the

frame except red by image thresholding. Hough circle transform is used for detecting circles

in the frame. It also determines the radius as well as the coordinates of the centre of circle.

The laptop then sends commands to the Arduino via serial communication according the

coordinates of the centre.

Image Processing

Behaviours: 1. Search the ball:

At first, the robot starts revolving about itself in search of a ball at the location where

it started. The web cam will check if it sees any ball irrespective of its colour, every

time it pauses revolving for a short time. The bot starts to align with the ball after

detecting it.

2. Approach towards ball and grab it: Once the robot has found a ball, it needs to approach towards it. It does this by first

centring the ball in the view. Once it centres the ball, it will drives straight towards it.

Every now and then, it checks again to make sure the ball is still at the centre of the

view and adjust if it isn’t. Also, it constantly checks the distance of the ball in terms of

its radius in the frame. Once it reaches a distance where the ball is close enough, it

grabs the ball by the front claw.

3. Search for the base station: Once the bot grabs the ball, it will starts revolving at that place in search of a base

station. Once the base station is found, the bot starts approaching towards it.

4. Approach towards base station and drop ball: Once the base station has been found, the bot keeps approaching towards it by

keeping it at the centre of the view. During the approach, it constantly checks the

distance of the base station in terms of distance between the centroids of two

different strips on it. Once the distance increases to a certain specified value the bot

stops and drops the ball.

Experimental Layout and Results: Originally, Pickabo was going to use an Odroid-U3 as a processor. But due to my lack of

experience in this field I ended up using my laptop as a processor. The chassis of the robot

had to be redesigned to accommodate the bigger and bulkier processor on-board.

Conclusion: Pickabo was my first full-fledged robotics project. So building it was quite an enlightening

experience. There is however a large amount of room for improvement, particularly in the

area of enabling Pickabo to detect and differentiate between spheres according to their

colours as originally intended. The obstacle avoidance behaviour couldn’t be integrated with

the main code. So, it is something that can be worked upon.

Appendix:

Image processing code #include "opencv2/imgproc/imgproc.hpp"

#include "opencv2/highgui/highgui.hpp"

#include "opencv2/imgproc/imgproc_c.h"

#include <stdlib.h>

#include <stdio.h>

#include <iostream>

#include <string.h>

#include <unistd.h>

#include <stdint.h>

#include <fcntl.h>

#include <termios.h>

#include <errno.h>

#include <sys/ioctl.h>

#pragma comment (lib, "opencv_core2410d.lib")

#pragma comment (lib, "opencv_highgui2410d.lib")

#pragma comment (lib, "opencv_imgproc2410d.lib")

#pragma comment (lib, "opencv_video2410d.lib")

#pragma comment (lib, "opencv_features2d2410d.lib")

using namespace cv;

using namespace std;

int main(int argc, char** argv)

{

char mode;

char buffer[2]; //Character buffer for data string sent from the ODROID to the Arduino

int fd; //The device label for the Arduino

struct termios toptions; //Store the serial port settings

//Open the serial port

fd = open("/dev/ttyACM0", O_RDWR | O_NOCTTY);

printf("fd opened as %i\n", fd);

usleep(3500000);

// Wait for the Arduino to reboot

tcgetattr(fd, &toptions);

//Get current serial port settings

//Set 9600 baud both ways

cfsetispeed(&toptions, B9600);

cfsetospeed(&toptions, B9600);

// Serial settings: 8 bits, no parity, no stop bits

toptions.c_cflag &= ~PARENB;

toptions.c_cflag &= ~CSTOPB;

toptions.c_cflag &= ~CSIZE;

toptions.c_cflag |= CS8;

// Canonical mode

toptions.c_lflag |= ICANON;

//Commit the serial port settings

tcsetattr(fd, TCSANOW, &toptions);

VideoCapture cap(1); //capture the video from webcam

if (!cap.isOpened()) // if not success, exit program

{

cout << "Cannot open the web cam" << endl;

return -1;

}

namedWindow("Control-1", CV_WINDOW_AUTOSIZE); //create a window called

"Control"

namedWindow("Control-2", CV_WINDOW_AUTOSIZE); //create a window called

"Control"

int iLowH1 = 0;

int iHighH1 = 15;

int iLowH2 = 160;

int iHighH2 = 179;

int iLowS1 = 149;

int iHighS1 = 255;

int iLowS2 = 149;

int iHighS2 = 255;

int iLowV1 = 59;

int iHighV1 = 255;

int iLowV2 = 59;

int iHighV2 = 255;

//Create trackbars in "Control" window

createTrackbar("LowH1", "Control-1", &iLowH1, 179); //Hue (0 - 179)

createTrackbar("HighH1", "Control-1", &iHighH1, 179);

createTrackbar("LowH2", "Control-2", &iLowH2, 179); //Hue (0 - 179)

createTrackbar("HighH2", "Control-2", &iHighH2, 179);

createTrackbar("LowS1", "Control-1", &iLowS1, 255); //Saturation (0 - 255)

createTrackbar("HighS1", "Control-1", &iHighS1, 255);

createTrackbar("LowS2", "Control-2", &iLowS2, 255); //Saturation (0 - 255)

createTrackbar("HighS2", "Control-2", &iHighS2, 255);

createTrackbar("LowV1", "Control-1", &iLowV1, 255);//Value (0 - 255)

createTrackbar("HighV1", "Control-1", &iHighV1, 255);

createTrackbar("LowV2", "Control-2", &iLowV2, 255);//Value (0 - 255)

createTrackbar("HighV2", "Control-2", &iHighV2, 255);

//Capture a temporary image from the camera

Mat imgTmp;

cap.read(imgTmp);

//Create a black image with the size as the camera output

Mat imgLines = Mat::zeros(imgTmp.size(), CV_8UC3);;

while (true)

{

Mat imgOriginal;

bool bSuccess = cap.read(imgOriginal); // read a new frame from video

if (!bSuccess) //if not success, break loop

{

cout << "Cannot read a frame from video stream" << endl;

break;

}

Mat imgHSV;

Mat imgThresholded;

Mat imgThresholded1;

Mat imgThresholded2;

cvtColor(imgOriginal, imgHSV, COLOR_BGR2HSV); //Convert the captured

frame from BGR to HSV

inRange(imgHSV, Scalar(iLowH1, iLowS1, iLowV1), Scalar(iHighH1, iHighS1,

iHighV1), imgThresholded1);

imshow("Thresholded Image 1", imgThresholded1);

inRange(imgHSV, Scalar(iLowH2, iLowS2, iLowV2), Scalar(iHighH2, iHighS2,

iHighV2), imgThresholded2);

imshow("Thresholded Image 2", imgThresholded2);

bitwise_or(imgThresholded1, imgThresholded2, imgThresholded); //Overlap

two thresholded images

//morphological opening (removes small objects from the foreground)

erode(imgThresholded, imgThresholded,

getStructuringElement(MORPH_ELLIPSE, Size(5, 5)));

dilate(imgThresholded, imgThresholded,


//morphological closing (removes small holes from the foreground)

dilate(imgThresholded, imgThresholded,


erode(imgThresholded, imgThresholded,


double threshold = 100;

vector<vector<Point> > contours;

vector<Vec4i> hierarchy;

vector<int> small_blobs;

double contour_area;

Mat temp_image;

// find all contours in the binary image

imgThresholded.copyTo(temp_image);

findContours(temp_image, contours, hierarchy, CV_RETR_CCOMP,

CV_CHAIN_APPROX_SIMPLE);

// Find indices of contours whose area is less than `threshold`

if (!contours.empty()) {

for (size_t i = 0; i<contours.size(); ++i) {

contour_area = contourArea(contours[i]);

if (contour_area < threshold)

small_blobs.push_back(i);

}

}

// fill-in all small contours with zeros

for (size_t i = 0; i < small_blobs.size(); ++i) {

drawContours(imgThresholded, contours, small_blobs[i], cv::Scalar(0),

CV_FILLED, 8);

}

//hough transform works well with blurred image

GaussianBlur(imgThresholded, imgThresholded, Size(9, 9), 2, 2);

vector<Vec3f> circles;

HoughCircles(imgThresholded, circles, CV_HOUGH_GRADIENT, 1,

imgThresholded.rows / 4, 100, 40, 10, 70);

for (size_t i = 0; i < circles.size(); i++)

{

Point center(cvRound(circles[i][0]), cvRound(circles[i][1]));

int radius = cvRound(circles[i][2]);

bool ball = false;

//Calculate the moments of the thresholded image

Moments oMoments = moments(imgThresholded);

double dM01 = oMoments.m01;

double dM10 = oMoments.m10;

double dArea = oMoments.m00;

//calculate the centroid of the white region

int posX = dM10 / dArea;

int posY = dM01 / dArea;

Point p1(posX, posY);

//cout << "\ncentroid :"<<Mat(p1)<< endl; //Print the centroid

//compare the centroid found by calculating moments with the center

found by hough transform

if ((fabs(posX - circles[i][0])<10) && (fabs(posY - circles[i][1])<10)) ball

= true;

else ball = false;

if (ball)

{

// circle center

circle(imgOriginal, center, 3, Scalar(0, 255, 0), -1, 8, 0);

// circle outline

circle(imgOriginal, center, radius, Scalar(0, 0, 255), 3, 8, 0);

cout << "\nI found a ball..." << "\ncenter : " << center <<

"\nradius : " << radius << endl;

if (radius <= 64)

{

if ((circles[i][0] - 320) > 15)

{

mode = 'r';

cout << "turning right to align with ball" <<

endl;

}

else if ((circles[i][0] - 320) < -15)

{

mode = 'l';

cout << "turning left to align with ball" << endl;

}

else

{

mode = 'f';

cout << "approaching towards the ball" << endl;

}

}

else if (radius >64 && radius<=70)

{

mode = 'g';

cout << "engaging the gripper to grab the ball" << endl;

}

}

else

{

mode = 'n';

cout << "\nI'll keep looking..." << endl;

}

}

imshow("Thresholded Image", imgThresholded); //show the thresholded

image

namedWindow("Hough Circle Transform Demo", CV_WINDOW_AUTOSIZE);

imshow("Hough Circle Transform Demo", imgOriginal);

if (waitKey(30) == 27) //wait for 'esc' key press for 30ms. If 'esc' key is

pressed, break loop

{

cout << "esc key is pressed by user" << endl;

break;

}

sprintf(buffer, "%c", mode); //Construct data string

cout << buffer << endl; //Display the data string for debug

purposes

write(fd, buffer, 2); //Sent the data string to the Arduino

usleep(100000); //Slow data transmission rate to

accommodate the Arduino

}

return 0;

}

Arduino Code #include <Servo.h> //include the servo library to control the RobotGeek Servos

#define MICRO_SERVOPIN 10 //pin that the micro servo will be attached to

Servo microServo; //create an servo object for the 9g FT-FS90MG micro servo

const int speed = 255;

const int

PWM_A = 3,

DIR_A = 12,

BRAKE_A = 9,

SNS_A = A0;

//Motor A is the RIght Side motor

const int

PWM_B = 11,

DIR_B = 13,

BRAKE_B = 8,

SNS_B = B0;

//Motor B is the Left Side motor

void setup()

{

// Configure the A output

pinMode(BRAKE_A, OUTPUT); // Brake pin on channel A

pinMode(DIR_A, OUTPUT); // Direction pin on channel A

// Configure the B output

pinMode(BRAKE_B, OUTPUT); // Brake pin on channel B

pinMode(DIR_B, OUTPUT); // Direction pin on channel B

microServo.attach(MICRO_SERVOPIN);

microServo.write(150); // sets the servo position to 150 degress, positioning the servo for the

gripper open

Serial.begin(9600);

}

char mode;

void loop()

{

mode=Serial.read();

if(mode=='l') left(); // THE BOT MOVES IN LEFT DIRECTION

else if(mode=='r') right(); // THE BOT MOVES IN RIGHT DIRECTION

else if(mode=='f') front();// THE BOT MOVES IN FORWARD DIRECTION

else if(mode=='g') gripper(); // THE BOT ENGAGES THE GRIPPER

else if(mode=='n') look(); // THE BOT LOOKING FOR HOUGH CIRCLE

Serial.flush();

}

void left()

{

Serial.println("turning left to align with ball");

digitalWrite(BRAKE_A, LOW); // setting brake LOW disable motor brake

digitalWrite(BRAKE_B, LOW); // setting brake LOW disable motor brake


digitalWrite(DIR_A, LOW); // setting direction to LOW the motor will spin forward

analogWrite(PWM_A, speed); // Set the speed of the motor, 255 is the maximum value


digitalWrite(DIR_B, LOW); // setting direction to HIGH the motor will spin backward

analogWrite(PWM_B, speed); // Set the speed of the motor, 255 is the maximum value

delay(20);

digitalWrite(BRAKE_A, HIGH); // setting brake LOW disable motor brake

digitalWrite(BRAKE_B, HIGH); // setting brake LOW disable motor brake

delay(100);

}

void right()

{

Serial.println("turning right to align with ball");




digitalWrite(DIR_A, HIGH); // setting direction to LOW the motor will spin backward



digitalWrite(DIR_B, HIGH); // setting direction to HIGH the motor will spin forward


delay(20);



delay(100);

}

void front()

{

Serial.println("approaching towards the ball");




digitalWrite(DIR_A, LOW); // setting direction to HIGH the motor will spin forward



digitalWrite(DIR_B, HIGH); // setting direction to LOW the motor will spin forward


delay(80);



delay(100);

}

void gripper()

{

microServo.write(70); //set gripper to 150 degrees = fully closed

//delay(200); //wait 3 seconds



delay(50000);

}

void look()

{int dir = random(0,2);

if (dir ==0)

{




digitalWrite(DIR_A, LOW); // setting direction to LOW the motor will spin forward



digitalWrite(DIR_B, LOW); // setting direction to HIGH the motor will spin backward


delay(20);



delay(100);

}

if (dir == 1)

{




digitalWrite(DIR_A, HIGH); // setting direction to LOW the motor will spin backward



digitalWrite(DIR_B, HIGH); // setting direction to HIGH the motor will spin forward


delay(20);



delay(100);

}

}

“pickabo” · 2014. 12. 9. · “pickabo” chaudhari anup s december 9, 2014 university of...

Documents