Robot Vision System

Robotics in GeneralIndustrial RoboticsMedical RoboticsRobot VisionWhat can Computer Vision do for Robotics?Vision SensorsIssues / ProblemsVisual ServoingApplication ExamplesSummary

CONTENTS

What is a robot?"A reprogrammable, multifunctional manipulator

designed to move material, parts, tools, or specialized devices through various programmed motions for the performance of a variety of tasks"

Robot Institute of America, 1979 Industrial

Mostly automatic manipulation of rigid parts with well-known shape in a specially prepared environment.

MedicalMostly semi-automatic manipulation of

deformable objects in a naturally created, space limited environment.

Field RoboticsAutonomous control and navigation of a mobile

vehicle in an arbitrary environment.

Robot vs Human

Robot Advantages:StrengthAccuracySpeedDoes not tireDoes repetitive tasksCan Measure

• Human advantages:• Intelligence• Flexibility• Adaptability• Skill• Can Learn• Can Estimate

Requirements:

AccuracyTool QualityRobustnessStrengthSpeed Price Production CostMaintenance

Industrial Robot

Production Quality

Medical (Surgical) Robot

Requirements

SafetyAccuracyReliabilityTool QualityPriceMaintenanceMan-Machine Interface

Vision for robots requires the ability to identify and accurately determine the positions of all relevant three dimensional objects within the robot work place.

Robot Vision

Robot vision may be defined as the process of extracting, characterizing, and interpreting information from images of a three dimensional world

2000 Jaskaran Singh ROBOT VISION

Purpose of A Machine Vision System

Analyzes images and produces descriptions of what is being imaged.

Input to the system- ImageOutput from the system- satisfy two

criteria.

2000 Jaskaran Singh ROBOT VISION

Robot Vision-Fundamental Tasks

-Image transformation

-Image analysis

-Image understanding

ROBOT VISION2000 Jaskaran Singh

General Purpose Robot Vision

Important thing- System should capture the relevant data and with the

motion of the object it should be able to update the information.

Four steps to General Purpose Robot Vision Object verification and tracking Fast extraction of stable image features Object model acquisition Efficient indexing of the model database

What can Computer Vision do for Robotics?Accurate Robot-Object PositioningKeeping Relative Position under MovementVisualization / Teaching / TeleroboticsPerforming measurementsObject RecognitionRegistration

Vision Sensors

Single Perspective CameraMultiple Perspective Cameras (e.g. Stereo

Camera Pair)Laser ScannerOmnidirectional CameraStructured Light Sensor

Vision Sensors

Single Perspective Camera

XPx x43

Multiple Perspective Cameras (e.g. Stereo Camera Pair)

Vision Sensors

Multiple Perspective Cameras (e.g. Stereo Camera Pair)

0Fxx'T Fxl'

Vision Sensors

Laser ScannerVision Sensors

Laser ScannerVision Sensors

Omnidirectional CameraVision Sensors

Omnidirectional CameraVision Sensors

Structured Light Sensor

                                                                    

         

                                                      

       

                                                  

       

Figures from PRIP, TU Vienna

Vision Sensors

Issues/Problems of Vision Guided RoboticsMeasurement Frequency

Measurement Uncertainty

Occlusion, Camera Positioning

Sensor dimensions

Visual Servoing

Vision System operates in a closed control loop. Better Accuracy than „Look and Move“ systems

Figures from S.Hutchinson: A Tutorial on Visual Servo Control

Example: Maintaining relative Object Position

Figures from P. Wunsch and G. Hirzinger. Real-Time Visual Tracking of 3-D Objects with Dynamic Handling of Occlusion

Visual Servoing

Visual Servoing Camera Configurations:

End-Effector Mounted Fixed

Figures from S.Hutchinson: A Tutorial on Visual Servo Control

Servoing Architectures

Figures from S.Hutchinson: A Tutorial on Visual Servo Control

Visual Servoing

Position-based and Image Based control

Position based: Alignment in target coordinate system The 3D structure of the target is rconstructed The end-effector is tracked Sensitive to calibration errors Sensitive to reconstruction errors

Image based: Alignment in image coordinates No explicit reconstruction necessary Insensitive to calibration errors Only special problems solvable Depends on initial pose Depends on selected features

target

End-effector

Image of target

Image of end effector

Visual Servoing

EOL and ECL control EOL: endpoint open-loop; only the target

is observed by the camera

ECL: endpoint closed-loop; target as well as end-effector are observed by the camera

EOL ECL

Visual Servoing

Position Based Algorithm:1. Estimation of relative pose2. Computation of error between current

pose and target pose3. Movement of robot

Example: point alignment

p1

p2

Visual Servoing

Position based point alignment

Goal: bring e to 0 by moving p1

e = |p2m – p1m|u = k*(p2m – p1m)

pxm is subject to the following measurement errors: sensor position, sensor calibration, sensor measurement error

pxm is independent of the following errors: end effector position, target position

p1m p2m

d

Visual Servoing

Image based point alignment

Goal: bring e to 0 by moving p1e = |u1m – v1m| + |u2m – v2m|

uxm, vxm is subject only to sensor measurement error

uxm, vxm is independent of the following measurement errors: sensor position, end effector position, sensor calibration, target position

p1 p2

c1 c2

u1u2

v1 v2d1 d2

Visual Servoing

Example Laparoscopy

Figures from A.Krupa: Autonomous 3-D Positioning of Surgical Instruments in Robotized Laparoscopic Surgery Using Visual Servoing

Visual Servoing

Example Laparoscopy

Figures from A.Krupa: Autonomous 3-D Positioning of Surgical Instruments in Robotized Laparoscopic Surgery Using Visual Servoing

Visual Servoing

RegistrationRegistration of CAD models to scene features:

Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching

Registration of CAD models to scene features:

Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching

Registration

Tracking

Instrument tracking in laparoscopy

Figures from Wei: A Real-time Visual Servoing System for Laparoscopic Surgery

SummaryComputer Vision provides accurate and versatile

measurements for robotic manipulators

With current general purpose hardware, depth and pose measurements can be performed in real time

In industrial robotics, vision systems are deployed in a fully automated way.

In medicine, computer vision can make more intelligent „surgical assistants“ possible.