Computer Science Department Ben-Gurion University of the Negev

TCV152 – Mini-project in Computational Vision

Instructor : Prof. Ohad Ben-Shahar

TA : Ehud Barnea

2 TCV152 introduction

In general

• Exposure to computational vision and robotic

• Hands on and practical experience in these two topics

And specifically

• Get to know ROS – the Robot Operating System

• Get to know Komodo – a real mobile robotic platform equipped with manipulators and sensors

• Get to know OpenCV – and free computer vision library

• Apply your knowledge to program a real robot to do some computer vision task

Goal

3 TCV152 introduction

• This is a mini-project, i.e., you will mostly be on your own

• You will learn much technical material, all from online sources and by self experimentation.

• You will work in groups. Group size depends on number of students enrolled in the course

• Cooperation between groups is encouraged

• You will work first without and then with a real robot (more about this in a moment)

• ROS programming is done in Linux and C++ (or Python)

• Programming the robot is difficult, frustrating, but eventually rewarding

• We have a TA (Ehud Barnea) in a very part time position, i.e., he will help in emergencies only.

What to expect

4 TCV152 introduction

• Eventually you will work with a real robot

• BUT…do not work with the real robot unless you have to. During large parts of the project you will be able to do without it.

• When you do end up needing it, the robot and its ROS computer are located in the AI and robotics lab, room 37/211.

• When you end up working in the robot, make sure that most of the time it is placed on a platform that keeps the wheels in the air. This is an important safety requirement. The projects in this mini-project course will not need any movement of the mobile platform.

• The lab is the home of graduate students from the AI group. Respect them and their work space. Do not touch anything except the robot and the ROS computer. If you come to work solo, you are welcome any time during the day. If you come as a team (2 students or more), you will be able to access the lab from 17:00 only.

• Since the robot is a critical resource and only one team can work at a time, we will use a scheduling system where you reserve your preferred time slot ahead of time. More information on this will follow.

• Before you can use the robot, you will need to sign a form in which you verify that you understand the risks and that you will follow safety procedures

The robot and the lab

5 TCV152 introduction

• Instructor: Prof. Ohad Ben-Shahar

• Office: 37/114

• Phone: (08-64)77868

• Email: ben-shahar@cs.bgu.ac.il

• Office hours: Tue 10:00-12:00; best to schedule via email.

• TA: Mr. Ehud Barnea <barneaeh@cs.bgu.ac.il>

• Meetings: Only 3-4 during the semester (see course plan)

• Group size: 2-4 depending on enrollment and robot availability.

• Grading: 30% Assignments; 70% Project

Course info

6 TCV152 introduction

Week 1: No class

Week 2: Class meeting

• Introduction to course

• ROS overview lecture

Weeks 2-4: Independent work

• Read the ROS Getting Started section, including the Introduction, Concepts, and Higher-Level Concepts

• Read the ROS Installation guide. Installing ROS on your laptop is optional but desired.

• Read and practice the Beginner level ROS Tutorials (tutorials #1.1.1-1.1.20). Some tutorials are similar up to the language used (C++ or Python) and you are free to use your language.

• Submit lab assignment #1

Detailed course plan (no need to copy, you will have on the web + email)

7 TCV152 introduction

Weeks 5-6: Independent work

• Read the Komodo user manual

• Read the ric_robot package documentation with focus on using the sensors (cameras' depth sensors, laser scanner, etc..)

• Read and practice the ric tutorials, and in particular tutorials 1,3,6,8,9

• Submit lab assignment #2

Week 7: Class meeting

• Computer vision overview lecture

• Introduction to OpenCV

Weeks 7-8: Independent work

• Read the fundamental aspects of the OpenCV documentation Try to cover the basic parts of the user guide, and review the essentials from the introductory, core functionality, image processing, 2D features tutorials.

• Submit lab assignment #3

Detailed course plan

8 TCV152 introduction

Week 9: Independent work

• Review the project topics.

• Select a project

• Submit a 1-page proposal.

Weeks 10-14 and until project submission date (TBD):

• Study any additional OpenCV and ROS resources necessary for your project

• Implement your project

• Submit report + code + documentation

• Demonstrate your project

Detailed course plan

9 TCV152 introduction

A 3 node ROS system that does the following

• "First" node - Generates messages at random times but no more than 1 second apart. Each message contains a random number in the range [0,100]. Messages are published on the topic "FirstNumber" and logged in a "FirstLog.txt" file with their generation time.

• "Second" node - Generates messages at random times but no more than 0.5 second apart. Each message contains a random number in the range [-100,0]. Messages are published on the topic "SecondNumber" and logged in a "SecondLog.txt" file with their generation time.

• "Adder" node - Using callback functions and threads, this node should subscribe to the two topics above, and add and display the line

"<first>+<second>=<sum> at times <time1> and <time2>"

only if the two numbers were published 0.1 second apart or less.

• The Adder node should also log all the addition lines in "AdderLog.txt".

Lab assignment #1 (Due April 3)

First Second

Adder

10 TCV152 introduction

Implement and demonstrate a 3 node ROS system that does the following

• A "sensor" node that asynchronously publishes the readings of the laser range sensor (this node is already supplied by RoboTiCan as part of ric_robot)

• A "brain" node that subscribes to the sensor topic, finds the closest obstacle from the robot, and publishes its distance to the "ObstacleDistance" topic.

• A "speed" node that subscribes to the ObstacleDistance topic, computes a speed that is inversely proportional to the present distance, and continuously publishes this speed to the proper topic that actually controls the speed of the wheels. Make sure to rectify the computed and published speed to some reasonable maximal value.

If your assignment is successful, you will be able to continuously control the speed of the wheels by simply moving your hand closer and farther away from the robot.

IMPORTANT: never remove the robot from the safety platform so wheels run in the air without any robot motion.

Lab assignment #2 (Due April 26)

11 TCV152 introduction

Implement and demonstrate the following system that uses the color and depth images that the Asus RGB-D camera publishes on several topics

• A "Face detection" service - Create a ROS service that gets a color image as input, detects faces in the image, and returns the coordinates of boundary box of the detected faces.

• A “Control” node that listens to the color and depth images from the Asus camera, detects faces in the color image (using the face detection service above), and moves the wheels forward when a face is detected. This node should also save the original image with colored rectangles over the detected faces, where the brightness of each rectangle depends on the distance between the robot and the (central pixel of the) face.

If your assignment is successful, you will be able to control the wheels by simply hiding/exposing your face to the robot.

IMPORTANT: never remove the robot from the safety platform so wheels run in the air without any robot motion.

Lab assignment #3 (Due May 10)

12 TCV152 introduction

ROS wiki: Portal http://wiki.ros.org/

Tutorials http://wiki.ros.org/ROS/Tutorials

Komodo roc_robot http://wiki.ros.org/ric_robot

roc tutorials http://wiki.ros.org/ric/Tutorials

OpenCV Documentation portal http://opencv.org/documentation.html

Tutorials http://docs.opencv.org/master/doc/tutorials/tutorials.html

Optional resources Gazebo http://wiki.ros.org/gazebo

PCL http://pointclouds.org/

ActionLib http://wiki.ros.org/actionlib

MoveIt http://moveit.ros.org/

Resource (no need to copy, you will have on the web + email)

13 TCV152 introduction

Object detection: Detect a cup; Reach

Object detection: Detect an elevator button; Reach

Object detection: Detect an elevator; Approach

Object detection: Detect a closed door; Approach

Object detection: Detect an open door; Approach

Object detection: Detect a door handle; Reach

Face detection: Detect a known face; Orient toward it

Text in the wild: Detect room numbers; OCR it

View point change: From where should I look next? Reach

Do you have a related idea of your own? Let us know and get our approval

Project examples

Computer Science Department Ben-Gurion University of the Negev

TCV152

Introduction to ROS

Ehud Barnea

15 TCV152 introduction

• What is ROS?

• “The Robot Operating System (ROS) is a set of software libraries and tools that help you build robot applications.”

• More precisely:

• ROS provides a flexible architecture for robot programs.

• A simple plug-in system.

• ROS is bundled with helpful libraries (e.g., OpenCV, PCL)

What is ROS

16 TCV152 introduction

Robot architectures:

• Many interacting components

• Asynchronous communication

• ROS provides the architecture,

we provide the components

A common robot architecture

Decision making

Right eye

Object detection

Obstacle detection

Ears

Nose

Left eye Actions

…

Sensors

Processing

17 TCV152 introduction

Nodes and Topics

• Each program is a node

• Nodes interact using topics

Services

• Provide a function-like mechanism (blocking)

ROS basic concepts

Camera Object detector Motion Planner

… Images Objects …

Camera Motion Planner

Object detector

…

18 TCV152 introduction

ROS provides useful tools to make your life easier

• Command line tools – e.g., list all topics, working nodes, etc.

• RQT console – follow topics

• ROS bags – record/play everything in the system

ROS features

19 TCV152 introduction

talker.cpp

void main(int argc, char **argv) {

ros::init(argc, argv, "talker");

ros::NodeHandle n;

ros::Publisher chatter_pub = n.advertise<std_msgs::String>("chatter", 1000);

ros::Rate loop_rate(10);

while (ros::ok()) {

std::stringstream ss;

ss << "hello world " << count;

std_msgs::String msg;

msg.data = ss.str();

chatter_pub.publish(msg);

ros::spinOnce();

loop_rate.sleep();

}

}

ROS code example - Publisher

20 TCV152 introduction

listener.cpp

void chatterCallback(const std_msgs::String::ConstPtr& msg) {

ROS_INFO("I heard: [%s]", msg->data.c_str());

}

int main(int argc, char **argv) {

ros::init(argc, argv, "listener");

ros::NodeHandle n;

ros::Subscriber sub = n.subscribe("chatter", 1000, chatterCallback);

ros::spin();

return 0;

}

ROS code example - Subscriber

21 TCV152 introduction

cmake_minimum_required(VERSION 2.8.3) project(beginner_tutorials) ## Find catkin and any catkin packages find_package(catkin REQUIRED COMPONENTS roscpp rospy std_msgs genmsg) ## Generate added messages and services generate_messages(DEPENDENCIES std_msgs) ## Declare a catkin package catkin_package() ## Build talker and listener include_directories(include ${catkin_INCLUDE_DIRS}) add_executable(talker src/talker.cpp) target_link_libraries(talker ${catkin_LIBRARIES}) add_dependencies(talker beginner_tutorials_generate_messages_cpp) add_executable(listener src/listener.cpp) target_link_libraries(listener ${catkin_LIBRARIES}) add_dependencies(listener beginner_tutorials_generate_messages_cpp)

ROS code example - CMakeLists

22 TCV152 introduction

Good luck