a subsumptive, hierarchical, and distributed vision-based...
TRANSCRIPT
![Page 1: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/1.jpg)
A Subsumptive , Hierarchical, and DistributedVision -Based Architecture for Smart Robotics
Vision-Guided and IntelligentRobotics Lab - ViGIR
Guilherme DeSouza
![Page 2: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/2.jpg)
Introduction
� Vision and Automation� Automation vs. robotics constraints
� Powerful computers vs. more demanding tasks
This image cannot currently be displayed.
![Page 3: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/3.jpg)
Introduction (cont.)
� Vision-Based Architectures� Horizontal Layers (Brooks-86)
� Top-Down vs. Bottom-up Hierarchical vs. "incremental" Subsumption
� Interaction between behaviors (Vectors, Fuzzy,...)
� TCA (Simmons-94), DAMN (Rosenblatt-95)
� Advantages and Disadvantages� "Intractable" (Tsotsos-95)
� Fault tolerance vs. All behaviors must be active
![Page 4: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/4.jpg)
Introduction (cont.)� Traditional Control of Industrial
Processes
SCADA Systems
Distribution => Redundancy => Fault Tolerance
� Vision-Based Control
Failures represent large financial costs
Fault Tolerance?
![Page 5: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/5.jpg)
Introduction (cont.)
� Proposed Vision-based Control Architecture
� Subsumption
� Hierarchical
� Distributed� CPU-Efficient
� Redundant
� Fault-tolerant
![Page 6: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/6.jpg)
Introduction (cont.)
� Line Tracking� Visual servoing for Assembly of
automotive parts on-the-fly
� Subsumptive tasks
� Mobile Robotics� Visual servoing of the robot w.r.t.
Multiple cues
� Map-based with localization by tracking natural landmarks
![Page 7: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/7.jpg)
Agenda
� Introduction
� Visual Servoing
� The New Software Architecture
� Implemented Visual Servos� Camera, Hand-Eye, and Head-Eye
Calibrations
� Applications of the Software Architecture� Line Tracking
� Mobile Robot Navigation
� Results and Conclusions
![Page 8: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/8.jpg)
Agenda
� Introduction
� Visual Servoing
� The New Software Architecture
� Implemented Visual Servos� Camera, Hand-Eye, and Head-Eye
Calibrations
� Applications of the Software Architecture� Line Tracking
� Mobile Robot Navigation
� Results and Conclusions
![Page 9: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/9.jpg)
Visual Servoing
� Closed-loop Control Systems that uses cameras as sensing devices. Its task is to position a robot with respect to a target object
� Task Space� 2DOF (bin-picking, pick-and-place, etc.)
� 3DOF (gaze control, mobile robots)
� 6DOF (assembly of parts)
![Page 10: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/10.jpg)
Classification� Position of the Cameras
� Eye-in-hand (Calibration)� Fixed-mounted� Combination
� Extracted Feature Points� Endpoint Open-Loop (EOL) - target object� Endpoint Closed-Loop (ECL) - target and end-
effector
� Hierarchy of the Control� Dynamic look-and-move� Direct servo
� Space of the Error Function � Position-based� Image-based
![Page 11: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/11.jpg)
Position -BasedDirect Servo
CartesianControl Law
PowerAmplifiers
PoseEstimation
ImageFeatureExtraction
+_
X*
YX
![Page 12: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/12.jpg)
Position -BasedLook -and-Move
X*Cartesian
Control Law
Joint Control
PowerAmplifiers
PoseEstimation
ImageFeature
Extraction
+_
YX
![Page 13: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/13.jpg)
Image-BasedDirect Servo
Feature spaceControl Law
PowerAmplifiers
ImageFeature
Extraction
+_
Y*
Y
X
![Page 14: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/14.jpg)
Image-BasedLook -and-Move
Feature spaceControl Law
JointControl
PowerAmplifiers
ImageFeatureExtraction
+_
Y*
Y
X
![Page 15: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/15.jpg)
Discussion� Position-Based
� Accuracy is highly dependent on Calibration� Hand-Eye and Head-Eye calibrations
� Feature Extraction and 3D Reconstruction� Stereo, Model-based, Structure from Motion
(Huang94)
� Image-Based� Non-linear Mapping from Task Space to
Feature Space (Large dimension feature space)
� Easy to find for Fixed-mounted cameras (Sanderson87)
� Approximate using Jacobian� Difficult inverse, not stable, convergence.
![Page 16: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/16.jpg)
Observations drawn from prior work
� Position-based is usually preferred� Specially for moving targets (same
space)
� Visual-servoing systems are closed-loop, discrete time, dynamic systems
� Sampling rate impacts the performance, convergence, and effectiveness
� Time delays add even more latency
![Page 17: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/17.jpg)
Observations drawn from prior work (cont.)
� Requisites for Visual-servoing� Fast sampling rates
� Robust
� Redundancy (multiple cues)
� Accurate (Calibration)
� CPU-efficient (minimize delays)
� Fault-tolerant
� Modular, Flexible, etc...
![Page 18: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/18.jpg)
Agenda
� Introduction
� Visual Servoing
� The New Software Architecture
� Implemented Visual Servos� Camera, Hand-Eye, and Head-Eye
Calibrations
� Applications of the Software Architecture� Line Tracking
� Mobile Robot Navigation
� Results and Conclusions
![Page 19: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/19.jpg)
Control Software Architecture
� Client/Server Infrastructure� Resources
� Processes and Threads
� Client/Server
� Wrappers
� Pipes/Sockets
� Shared Memory
� Modularity/Encapsulation
![Page 20: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/20.jpg)
Wrappers in a Client-Server Infrastructure
![Page 21: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/21.jpg)
Advantages of the Software Architecture
� Fault-tolerance� Redundant modules, subsumption
� CPU-Efficiency� Distribution
� Elasticity� Dynamic creation, deletion
� Flexibility� No effect of redesign of modules
![Page 22: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/22.jpg)
Primary Characteristics of the Architecture
� Multi-loop Control Architecture� Independent parallel loops
� Arbitration based on 'de-coupled' information from loops (confidence levels)
� Modules can be subsumed or replicated
� Slow modules do not affect others
� Communication vs. Application threads
![Page 23: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/23.jpg)
CartesianControl Law
Joint Control
PowerAmplifiers
PoseEstimation
ImageFeature
Extraction
+_
X*
YX
ImageFeatureExtraction
Y
Feature spaceControl Law
+_
Y*
?
?
+_
Arbitrator
![Page 24: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/24.jpg)
Notation� Quaternions
q = (s,v) where s = cos(θ/2)v = sin(θ/2) * e
� HTM� Denavit-Hartemberg Notation:
� Camera model� Pin-hole with Radial Distortion
� Cal = Int * Ext (Ext is a HTM)
� 3D Reconstruction� Stacking two calibration matrices (Least
Square)
� Object pose from cross-product of 3 feature points
� XYZOAT Representation
H bw =[R3x3
�T t
�0t 1 ]
![Page 25: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/25.jpg)
Position -BasedVisual Servoing
� Position task:� Kinematics Error Function = maps the
task space T (all poses of the end-effector) into an error vector (e.g. XYZOAT)
� Usually, one controller for each degree of freedom
� Use HTMs� Error Function:
E :T ��m
E�T e�= H oe − H o
*e
E�T e�= H ce
� H oc − H o
*e
![Page 26: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/26.jpg)
Position -BasedVisual Servoing
� Difference (?!):
� Proportional Controller (Quaternions)
u = k p � H e*e
E�T e�= H oe
� H e*o
![Page 27: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/27.jpg)
Image-BasedVisual Servoing
� Position Task: � Non-linear mapping from task space T
into the k-dimensional feature vector Y
� Linearization
E : f �T ���k
Y = f �T�
∂Y = J �∂T
![Page 28: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/28.jpg)
Image-BasedVisual Servoing
� Error Function
� Proportional Controller
� Update Jacobian
E�T e�= J− p�∇Y
u = k p� J − p�∇Y
J k�1 = J k ��∇Y − J k�∇T��∇T t
∇T t�∇T
![Page 29: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/29.jpg)
Agenda
� Introduction
� Visual Servoing
� The New Software Architecture
� Implemented Visual Servos� Camera, Hand-Eye, and Head-Eye
Calibrations
� Applications of the Software Architecture � Line Tracking
� Mobile Robot Navigation
� Results and Conclusions
![Page 30: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/30.jpg)
Agenda
� Introduction
� Visual Servoing
� The New Software Architecture
� Implemented Visual Servos� Camera, Hand-Eye, and Head-Eye
Calibrations
� Applications of the Software Architecture� Line Tracking
� Mobile Robot Navigation
� Results and Conclusions
![Page 31: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/31.jpg)
Recall
� Position-Based� Accuracy is highly dependent on Calibration
� Hand-Eye and Head-Eye calibrations
� Feature Extraction and 3D Reconstruction� Stereo, Model-based, Structure from Motion
(Huang94)
![Page 32: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/32.jpg)
Camera Calibration
� Intrinsic and Extrinsic parameters� Intrinsic parameters relate the standard
coordinate system and the image coordinate system (pixels)
� Extrinsic parameters relate a world coordinate system and the standard coordinate system
UVS
=� fk u 0 u0 0
0 � fk v v 0 0
0 0 1 0
Intrinsic
�R
˜T
˜0 1
Extrinsic
�
xyz1
![Page 33: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/33.jpg)
Other CalibrationsHand-eye and Head-eye Calibrations
Necessary for eye-in-hand configurations
No “world” is available:AX=XB Problem (Shiu & Ahmad)
X is the desired calibration matrix
and A, B are known matrices: extrinsic parameters
of the camera and kinematics of the end-effector)
(DeSouza et al, ICRA2001 & ICRA2002)
![Page 34: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/34.jpg)
Hand-Eye Calibration (ICRA2001)
![Page 35: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/35.jpg)
Hand-Eye Calibration
Calibrate Hand-Eye and World-Base
(AX = YB)
Iterative Algorithm:Find n equations such as: Ai* eiHci
= bHw* Bi
Solve for: eiHci
"Average" the n solutions to find: eHc
Use to solve: Ai * eHc= biHwi* Bi
"Average" the n solutions to find: bHw
Repeat until it converges
![Page 36: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/36.jpg)
Hand-eye Calibration"Average" using quaternions for rotation part,
and normal vector addition for the translation part.
Proof of convergenceAt each step, the averaged "error quaternion"
has a larger s-component
As the s-component grows towards 1 (unit vectors), the error quaternion becomes the "identity quaternion"
![Page 37: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/37.jpg)
Hand-eye CalibrationTest without Noise
![Page 38: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/38.jpg)
Hand-eye CalibrationTest with Noise
![Page 39: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/39.jpg)
Hand-eye Calibration
Results:
Less than 0.01 radians (0.3 degrees)
Less than 0.001 meters
![Page 40: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/40.jpg)
Head-Eye Calibration (ICRA2002)
Goal: to calibrate mobile robot's stereo head without worrying about its pose
Again, to calibrate means: camera calibration (two pairs of Intrinsic and Extrinsic
matrices)
head-eye/neck-eye calibrations (four HTM matrices)
![Page 41: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/41.jpg)
Stereo Head
3 DOF: pan, tilt, and vergence
![Page 42: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/42.jpg)
Stereo Head - HTMs
One extra HTM: cHe and EHb
Independently of the world coordinate system
![Page 43: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/43.jpg)
Calibration Procedure
Fix pan and tilt
Choose values for vergence vi
Obtain corresponding ciHw
Equate and solve
Aij* cHe= cHe* Bij
![Page 44: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/44.jpg)
Test A
![Page 45: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/45.jpg)
Results for Test A
Error in XYZ-Distance for objects @1,200mm
Various pan/tilt combinations (-10o, 10o)
DistanceAverage 0.4530 1.4530 2.4530 3.4530
Std Deviation 1.8849 3.9256 1.6090 1.5473
dX dY dZ
![Page 46: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/46.jpg)
Results for Test A (cont.)
Error in XYZ-Distance for objects @2,100m
Various pan/tilt combinations (-15o,15o)
DistanceAverage 9.3548 -9.3320 0.1588 15.4897
Std Deviation 4.5064 11.2934 3.9168 6.9080
dX dY dZ
![Page 47: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/47.jpg)
Test B
![Page 48: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/48.jpg)
Results for Test B
Error in XYZ-Distance
Pixel AccuracyDepth Left Right Top Bottom
1601.5 14.74 14.47 1.12 1.037483.6 337 309.3 48.7 21.6
15029.8 1409.3 1187.2 243.7 26.4
Actual Depth Calculated Depth DX DY DZ Distance1602 1604 1.47 11.42 -2.12 12.757484 7467 -6.4 28.23 1.88 147.68
15030 15467 -22.33 -518.9 10.84 520.79
![Page 49: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/49.jpg)
Agenda
� Introduction
� Visual Servoing
� The New Software Architecture
� Implemented Visual Servos� Camera, Hand-Eye, and Head-Eye
Calibrations
� Applications of the Software Architecture� Line Tracking
� Mobile Robot Navigation
� Results and Conclusions
![Page 50: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/50.jpg)
Feature -based Tracking
Model composed of straight line features
Extended Kalman Filter for pose etimation
![Page 51: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/51.jpg)
Kalman -Filtering
Kalman Filter
State statisticsupdate
Landmarkprediction
Landmarkmeasure
Uncertainty region
Update uncertainty usingassembly line dynamics
Current pose estimate State prediction
Iterate for all features
![Page 52: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/52.jpg)
Feature -based tracking (cont.)
Predicted Pose (red) Extracted Pose (yellow)
![Page 53: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/53.jpg)
Feature -based Tracking (ICRA2003)
Model composed of Elliptical shapes
Binocular stereo tracking
Object pose is calculated with simple object reference frame definition
![Page 54: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/54.jpg)
Example of Stereo -vision Tracking
![Page 55: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/55.jpg)
Object pose calculation
ModelFeature 1
ModelFeature 2
ModelFeature 3
X
Z
YO
![Page 56: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/56.jpg)
American Sign Language
Facial Expressions (Eyebrow)
![Page 57: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/57.jpg)
Active Appearance Models -AAM
Timothy Cootes (ECCV 1998 – ASM CVIU 1995)
Shape Vector: X = (x1 x2 …. xn y1 y2 ….yn)T
Texture Vector: g = (g1 g2 …. gm)T
PCA Analysis:
Or
ss bXX φ+=gg bgg φ+=
cPgg
cPXX
g
s
+=+=
![Page 58: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/58.jpg)
AAM Intuitively…
Synthesize any appearance by changing a set of parameters:
p = (t|c)
where c = appearance parameters
and t = 2D pose parameters (tx, ty, s, Θ)
Ex: 2-class & 2-sample case
![Page 59: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/59.jpg)
Facial Expressions
![Page 60: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/60.jpg)
AAM and Pose Estimation (ICRA2004)
Tracking the control points (shape) frame by frame
3D Reconstruction of the shape
Pose Estimation
![Page 61: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/61.jpg)
AAM – Tracking
cQxx s+=
cQgg g+=
)(xTX t=
• Sampling the pixels inside the current shape
• Image warping into the mean shape ( )
-
Parameter Updating
c
ct
c=c+�c
t=t+ �tr(p)
)()(* prpRp −=∆
p = [cT | tT]
x
![Page 62: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/62.jpg)
AAM – Tracking (Iterations)
cQxx s+=
cQgg g+=
)(xTX t=
• Sampling the pixels inside the current shape
• Image warping into the mean shape ( )
-
Parameter Updating
c
ct
c=c+�c
t=t+ �tr(p)
)()(* prpRp −=∆
p = [cT | tT]
x
![Page 63: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/63.jpg)
First Method: Stereo AAM
AAM matchingLeft appearance model
3-D reconstruction
Pose estimation
Estimated Pose {Rx, Ry, Rz, Tx, Ty,Tz}
X
Y
ZX
Y
Z
3D model points
Right appearance model
3D scene points
AAM matching
![Page 64: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/64.jpg)
Second Method: 3D Parameterization
• Sampling the pixels inside the current shape
• Image warping into the mean shape
-
Parameter Updatingp=p+�p r(p))()(* prpRp −=∆
XZ
y
XZ
pi Perspective
projectionR, t
bP gggg +=
p=[Rx Ry Rz tx ty tz bg]T
[Rx Ry Rz tx ty tz]T
bg
o c
![Page 65: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/65.jpg)
Final Result
![Page 66: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/66.jpg)
Comparative Results
(a)
(b)
(c)
X (meters) 0.0070 0.0097 0.0105 Y 0.0061 0.0076 0.0078 Z 0.0034 0.0025 0.0035
Yaw (degree) 0.3530 0.5880 0.4875 Pitch 0.5268 0.5239 0.7470
Mean of absolute error
Roll 0.2541 0.4149 0.2467 X (meters) 0.0056 0.0030 0.0082
Y 0.0071 0.0040 0.0068 Z 0.0026 0.0028 0.0032
Yaw (degree) 0.4389 0.2556 0.4142 Pitch 0.3921 0.0726 0.5082
Std of absolute error
Roll 0.2311 0.0957 0.2031 X (meters) 0.0250 0.0183 0.0516
Y 0.0284 0.0178 0.0379 Z 0.0109 0.0099 0.0281
Yaw (degree) 1.7634 1.2527 2.5138 Pitch 2.1204 0.6439 3.0327
Max of absolute error
Roll 0.9914 0.6759 1.1188 Note (a) The result of the first method (Stereo AAM) (b) The result of the second method (3D Parameterization) (c) The result of our previous approach (Feature-Based ICRA2003).
![Page 67: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/67.jpg)
0 50 100 150 200 250 300-0.25
-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
met
er
Frame number
X
black=ground-truthred=1st methodblue=2nd methodgreen=circular feature approach
![Page 68: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/68.jpg)
0 50 100 150 200 250 300-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
met
er
Frame number
Y
black=ground-truthred=1st methodblue=2nd methodgreen=circular feature approach
![Page 69: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/69.jpg)
0 50 100 150 200 250 300-0.1
-0.08
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0.08
0.1
met
er
Frame number
Z
black=ground-truthred=1st methodblue=2nd methodgreen=circular feature approach
![Page 70: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/70.jpg)
0 50 100 150 200 250 300-15
-10
-5
0
5
10
15
degr
ee
Frame number
Yaw
black=ground-truthred=1st methodblue=2nd methodgreen=circular feature approach
![Page 71: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/71.jpg)
0 50 100 150 200 250 300-20
-15
-10
-5
0
5
10
15
20
degr
ee
Frame number
Pitch
black=ground-truthred=1st methodblue=2nd methodgreen=circular feature approach
![Page 72: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/72.jpg)
0 50 100 150 200 250 300-6
-4
-2
0
2
4
6
degr
ee
Frame number
Roll
black=ground-truthred=1st methodblue=2nd methodgreen=circular feature approach
![Page 73: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/73.jpg)
Real-Time Multi-AAM(IEEE Motion 2005 - IEEE Trans. PAMI)
![Page 74: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/74.jpg)
CartesianControl Law
Joint Control
PowerAmplifiers
PoseEstimation
ImageFeature
Extraction
+_
X*
YX
ImageFeatureExtraction
Y
Feature spaceControl Law
+_
Y*
?
?+_ Arbitrator
![Page 75: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/75.jpg)
Assembly On -the-fly
![Page 76: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/76.jpg)
Assembly On -the-fly -Accuracy
![Page 77: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/77.jpg)
Agenda
� Introduction
� Visual Servoing
� The New Software Architecture
� Implemented Visual Servos� Camera, Hand-Eye, and Head-Eye
Calibrations
� Applications of the Software Architecture� Line Tracking
� Mobile Robot Navigation
� Results and Conclusions
![Page 78: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/78.jpg)
Mobile Robot Navigation
Multiple cues
No maps, natural landmark tracking
“Object-driven”
![Page 79: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/79.jpg)
Object-driven Navigation
“Follow directions”:
Select multiple objects to be tracked
Identify object attributes ("recognize")
Track and determine pose of object
Servo the robot w.r.t. objects
![Page 80: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/80.jpg)
Control Software Architecture
Multiple Control Loops:
Differences:
Dynamic priorities
Dynamic creation/deletion of loops
![Page 81: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/81.jpg)
CartesianControl Law
PoseEstimation
ImageFeatureExtraction
+_
X*
YX
+_
+_ Arbitrator
PoseEstimation
ImageFeatureExtraction
YX
PoseEstimation
ImageFeatureExtraction
YX
CartesianControl Law
CartesianControl Law
X*
![Page 82: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/82.jpg)
Object-Oriented Implementation
![Page 83: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/83.jpg)
Classes, Objects, and Methods
![Page 84: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/84.jpg)
PathPlanner
InstantiateappLight, appDoor, appObstAvoid, etc…
Assign priorities and pose (HTM)
Re-instantiateappDoor and appLight
Assign new priorities and new poses
![Page 85: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/85.jpg)
Hallway Follower
Uses appLight
Processing StepsBuild X-histogram and Y-histogram
Segment images based on brightness
Select largest blobs and apply heuristics
Apply Sobel to extract lines
Apply linear regression to find line equations
Find corner points and attributes of light fixture
Apply heuristics and send coordinate to Arbitrator
![Page 86: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/86.jpg)
Image Processing
![Page 87: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/87.jpg)
Door Tracker
Same as before except:
Histograms and segmentation are based on color
Different heuristics
Only 2 corners are extracted
![Page 88: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/88.jpg)
Image Processing
![Page 89: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/89.jpg)
Obstacle Avoidance
Check Sonar (and Lidar)
If there is an obstacle, circumnavigate.
![Page 90: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/90.jpg)
Servo Control Module
� Different for Position and Image-based
� Calculates XYZOAT (or delta XYZOAT)
� Send commands to Arbitrator with Confidence level:
� image processing
� reconstruction
![Page 91: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/91.jpg)
CartesianControl Law
PoseEstimation
ImageFeatureExtraction
+_
X*
YX
+_
+_ Arbitrator
PoseEstimation
ImageFeatureExtraction
YX
PoseEstimation
ImageFeatureExtraction
YX
CartesianControl Law
CartesianControl Law
X*
![Page 92: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/92.jpg)
Agenda
� Introduction
� Visual Servoing
� The New Software Architecture
� Implemented Visual Servos� Camera, Hand-Eye, and Head-Eye
Calibrations
� Applications of the Software Architecture� Line Tracking
� Mobile Robot Navigation
� Results and Conclusions
![Page 93: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/93.jpg)
Results
� Line Tracking running in five computers
� Simulating failures in zero, one, or more modules
� Results
� Arbitrator and Fine Control: ~45fps
![Page 94: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/94.jpg)
Contributions� Multi-loop control architecture
� Distributed, fault-tolerant, modular, elastic, CPU-efficient, and flexible software architecture
� Efficient Client/Server infrastructure that satisfies the constraints of vision-based systems
� Simple control law using quaternions
� Accurate hand-eye and head-eye calibration algorithms
� General architecture: can be applied to visual servoing as well as to mobile robotics (etc.)
![Page 95: A Subsumptive, Hierarchical, and Distributed Vision-Based ...vigir.ee.missouri.edu/~gdesouza/ece4330/Lecture... · Accuracy is highly dependent on Calibration Hand-Eye and Head-Eye](https://reader033.vdocuments.mx/reader033/viewer/2022042201/5ea1c8804f412f1aa43e8635/html5/thumbnails/95.jpg)
Conclusion
� Architecture reduces the delays by distributing modules over the network
� A modular design allows for redundancy, fault-tolerance, and subsumption of tasks
� Future Work� Better controllers:
� PID using quaternion representation of the error function
� Prediction using Kalman Filtering
� Geometric models of target (reduce error in posing)