human-robot outline collection - mcgill university

Human-Robot Collaboration for Data

Collectionin The Aqua Project

Gregory DudekDirector, School of Computer Science

James McGill ProfessorMember, Center for Intelligent Machines

McGill University

Gregory Dudek, http://www.cim.mcgill.ca/~dudek

Outline• Introduction: robotics & coral reef studies

• Hardware: our robots

• Recognition of places and surfaces

• Navigation and control

• Terrain learning, gait selection

• Human-Robot Interaction

• Conclusion


McGill, MontrealMcGillMontreal

Boston


McGillMontreal

BarbadosSan

Diego

McGillArcticstation

What is robotics?

TakingAction

Planning"Thinking"

In the Robot

OutsideWorld

Text

MeasuringFiguring outwhat we see

Spot the Robot

http://www.cim.mcgill.ca/~dudek

http://www.aquarobot.net

http://www.marstoday.com/news/viewsr.html?pid=27647

Microwave oven?

Measures something (how cooked)

Makes a decision (another 10 minutes).

Does something to the outside world

(cooks stuff).

About Robotics (in general).

• Robotics (broadly):

» The science and engineering of replicating the attributes of living beings, and humans in particular, in machines.

• Encompasses artificial intelligence, computational vision, machine learning, psychology, mechatronics and “traditional robotics”.

• Major resurgence of deep science, technological breakthroughs in the last 10 years.

• Really using robots depends on interacting with people.

• Significant economic impact already.

Applications Already Exist

Applications Already Exist


What are traditional robots best suited for?

• Environments that are... • dangerous, • inaccessible, • expensive to access, • tiring, • inhospitable.

• e.g. Exploration, radiation cleanup, military surveillance, hazardous waste assessment, factory production line, non-stop deliveries, ....


What are traditional robots best suited for?

• Environments that are... • dangerous, • inaccessible, • expensive to access, • tiring, • inhospitable.

• e.g. Exploration, radiation cleanup, military surveillance, hazardous waste assessment, factory production line, non-stop deliveries, ....

Undersea: inaccessible, dangerous, costly, demanding.

Most of the world is undersea, yet it’s the environment on earth we understand the least well!

Aqua Project: objectives

• Aqua is about developing a robot that can walk and swim, and which exhibits the ability to use vision to know where it is and what is near it.– Underlying science: model the world using artificial cognition.

– Current application is to assist a human biologist.

• Survey and monitor the conditions on a coral reef. – By being able to land on the bottom and move around, the robot

can make regular observations without disturbing the natural organisms.

• The ability to walk, swim and use vision underwater is unique to AQUA.

11

Aqua Project: objectives

• Aqua is about developing a robot that can walk and swim, and which exhibits the ability to use vision to know where it is and what is near it.– Underlying science: model the world using artificial cognition.

– Current application is to assist a human biologist.

• Survey and monitor the conditions on a coral reef. – By being able to land on the bottom and move around, the robot

can make regular observations without disturbing the natural organisms.

• The ability to walk, swim and use vision underwater is unique to AQUA.

11

G. Dudek, MRI presentation for CSA

Issues• How to control the robot.

– Just follow a diver, take instructions, work autonomously.

• How to model (describe) the bottom.

– By building 2-dimensional and 3-dimensional models.

• Remembering where we have been before.

– Using image-based localization methods.

– Remembering the appearance of the reef.

• Select “good” images.

• Image correction [e.g. with Luz-Abril Torres-Mendez].

Applications

! Marine life inspection.! Ship inspection, underwater cables/pipeline,

security.! Diver’s ”buddy:” Take instructions, follow diver,

help out.! Long-term surveillance, e.g. on coral reefs.


Coral Reefs

• Oceans: 70% of earth’s surface.• Reefs: Greatest diversity / area of any marine

ecosystem• 4-5% of all species (91 000) found on coral reefs

• Significant to the health of the planet: • 1/2 of the calcium that enters the world’s

oceans /year is taken up and bound intoCoral Reefs as Calcium Bicarbonate.


Coral Reefs• 20% of the world’s reefs have been destroyed.• 24% of reefs are under imminent threat of collapse

due to human pressure, 26% under longer term threat of collapse!• Dec. 2005 there was a terrible coral bleaching (and

destruction) in the Caribbean.• 95% of Jamaica’s reefs are dead or dying.

• We need to be able to measure the changes, both to understand and ameliorate.

• This is currently taxing, error-prone, tiring and dangerous.


Reef Studies• Detailed mosaics of reef 2D and 3D structure.• Transects of reefs

• Effects of rugosity (texture), topology and structure of intra-species interaction and mobility.

With Katerine Turgeon and Don Kramer, Biology.








• Conclusion








• Conclusion


Aqua Vehicle OverviewHigh-mobility amphibious capability

Walking Swimming

IEEE Computer Magazine, January 2007.

19

Aqua robots: Overview

! “Control Stack” uses standard QNX operating system

! The “Sensor Stack” runs a custom-built OS based on Linux

! Visual Sensing runs embedded, approximately 35 KLOC in C++ implementation

! 6-DOF flipper motion! Power-autonomous! 3 cameras! Inertial and depth

measurement

Aqua: Generations

Aqua Hardware

Three basic categories

! Locomotion: direct-drive legs! Computation: multiple computers! Sensing...

Hardware: Sensing

! 3 Firewire (IIDC1394) cameras! Inertial measurement unit! Internal sensors

(thermal, current, leg position via Hall Effect, etc.)! Depth sensing! Microphone

Underwater vehicles

UT-1 Ultra Trencher 7.8 x 7.8 x 5.6 meters

Argo

Autonomous Benthic Explorer (ABE)

1200 pounds and a little over 2 meters long.


Ease of deployment

Robustness.


Ease of deployment

Robustness.








• Conclusion








• Conclusion

Much of our work has been inspired by observing how people understand their world using vision.

Arbitrary Environments: what landmarks ?

G. Dudek

Key Principle: Feature Detection

! Detection of “unusual” points in the image (or the world itself).

! Features detected a saliency operator (SIFT, SURF, Harris edges). 29

Results: Place Recognition

Results: Place RecognitionBy remembering “interesting” points, we

recognize where we (probably) are.


Test Images

By remembering “interesting” points, we recognize where we (probably) are.


Test Images Training Images

Best match



Test Images Training Images

Best match

Best match


Performance Versus R

eco

gnitio

n R

ate

100%

Number of features

Note: 10 windows of size 15x15 meansusing only 0.7% of the total image

content.

Optional: skip?








• Conclusion








• Conclusion

•Thrust is generated by oscillating foils with amplitude Aflipper around angle !offset.

•Phase offset " between the flippers to reduce parasitic oscillations of robot.

Swimming Gait Description

Giguere & Dudek

Flexible Paddle Model•90% of the paddle is rigid and do not deflect•The paddle is separated into 100 elements of equal length•Assumption: paddle hinge is fixed relative to the surrounding fluid

Drag force:

Forces Acting on the Paddle3 forces are acting on the paddle:

1. Torque:

2. Hydrodynamic force: Hi

3. Resistive Force: F

The resistive force is given by:

Bending Moment1. Bending moment in continuous form:

2. Bending moment in Discrete form:

3. Elastic bending theory:

•Controlling moments are generated by modifying:

• !offset for pitch, roll

•Amplitude Aflipper for yaw

Swimming Gait (con’t)

Pitch Roll Yaw

Giguere & Dudek Gregory Dudek, http://www.cim.mcgill.ca/~dudek







• Conclusion








• Conclusion

Recall: it walks (hard real time)

Canadian Space Agency, simulated Mars terrain.

Gait and terrainHow we walk depends on what we are

walking on.

52

Various terrains, various gaits

G. Dudek, M. Jenkin, C. Prahacs, A. Hogue, J. Sattar, P. Giguere, A. German, H. Liu, S. Saunderson, A. Ripsman, S. Simhon, L. A. Torres-Mendez, E. Milios, P. Zhang, I. Rekleitis. A Visually Guided Swimming Robot, Proceedings of the 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1749-1754, 2005.

Various terrains, various gaits

G. Dudek, M. Jenkin, C. Prahacs, A. Hogue, J. Sattar, P. Giguere, A. German, H. Liu, S. Saunderson, A. Ripsman, S. Simhon, L. A. Torres-Mendez, E. Milios, P. Zhang, I. Rekleitis. A Visually Guided Swimming Robot, Proceedings of the 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1749-1754, 2005.

Unsupervised Terrain Identification

Giguere & Dudek, Robotics Science and Systems (RSS), 2008.Giguere & Dudek, Robotics, Autonomous Robots, 2009.

Unsupervised Terrain Identification

Giguere & Dudek, Robotics Science and Systems (RSS), 2008.Giguere & Dudek, Robotics, Autonomous Robots, 2009.

AQUA Contact Sensors

Unsupervised CaseLearning of Terrains and Places

Unsupervised CaseLearning of Terrains and Places

Learn the combination of features that distinguishes the classes.

Class 1

Class 2

Classifier Training

Class 1?

Classifier Training

Giguere & Dudek, 2008

cost function

• For unsupervised case, our cost function capture the notion of what a good classification would look like.

Features +Dimensionality Reduction



Giguere & Dudek, 2008

2 Terrains (Linear Separator)

Giguere & Dudek

More complex case

• Can we learn to distinguish many different terrain types?

More complex case

• Can we learn to distinguish many different terrain types?

5 Terrains Data

Giguere & Dudek

5 Terrains Data

Giguere & Dudek

5 Terrains (Gaussian Mix. Model)

Classification Rate: 91%

Giguere & Dudek

5 Terrains (Gaussian Mix. Model)

Giguere & DudekGregory Dudek, http://www.cim.mcgill.ca/~dudek







• Conclusion








• Conclusion

Vision: soft real time

Vision: soft real time


Guidance• Full sensor-based autonomy is just too difficult

– And the costs of failures is too high,• Select trajectories by (initially) following a diver.• Diver specifies specific actions as desired.• Diver specifies where and how data is collected.


Traditional (former) Procedure• Operator on surface controls robot.• Submerged diver can communicate with operator, for

example with hand signals.• Tether user for communication• Deficits:

– Tether dynamics & bouyancy– Entanglement risk (danger to robot & diver)– Inattention by operator– Cognitive load on diver


Diver-Operator Gesture Language


Diverse Command Set


Solution: servo-based control• Instruct the robot directly using robust (engineered)

markers.• Navigation by servo-control with respect to a diver.

• No “top-side” navigator required.– Multiple tracker subsystems

• color blobs, • probabilistic model of target motion (mean-shift tracker), • color combinations with adaptive learning,

• Spatio-chromatic filters

• human motion (Fourier tracker)

• Tracker provides feedback to gait selector and controller.

Xu, Sattar, Dudek, IEEE Int. Conf. Robotics and Automation (ICRA) 2008. Giguere, Sattar, Dudek, IJRR 2009.Skip tracker details

Tracking: Learning Spatio-chromaticity for tracking

! Filters are tuned to hue variations over space! Use Boosting to learn the color distribution of

target objects over space! Hence a Spatio-Chromatic tracker

Sattar & Dudek,IEEE Int. Conf. Robotics and Automation (ICRA) 2009

Spatio-Chromatic trackers! Uses color-based tracking

! Many possible simple color filters

! Distributed and oriented in space to capture color spread of the target object

! Four types based on distribution

Basic filter/tracker types

Much like the receptive fields of color cells in the primary human visual cortex.

Boosted Ensemble

Boosted Ensemble Boosted Ensemble

Weak Spatio-chromatic trackers

Boosted Ensemble

Weak Spatio-chromatic trackers

Trackers importance weights

Boosted Ensemble

Boosted Ensemble Tracker family size

! A very large number of weak trackers can be generated for training

! e.g. For type 4 trackers:

! The value of Colors lie in the real number space (for N-RGB)

Approach Example Training data

Tracker spread Some results

Servo-based Control Behavior Control• Since visual methods are used for servo-control, use them for navigation,

behavior & action control as well.• Want very robust methods and occasional landmarks.• Also make tether almost completely unnecessary.

• Achromatic visual symbolic markers • Redundant geometrically unique robust error-correcting encoding.• Variable-resolution content, partial information if viewing

conditions are bad.• Implemented using Fourier- Tags [Dudek, Sattar, et al.].


Tag-based Control (v1)

• Visual tags provide language for symbolic content.

• Tags specify:• Atomic actions• Parameters• Complex behaviors.

• Diver carries an dictionaryof tags from which hecan select.

Fiducial-based Human Interface

Solutions:

Cube

Flipbook

Deltohedron

•! Problem: large expression vocabulary =a lot of markers needed for explicit encoding


Operator initiated start & left turn








• Conclusion








• Conclusion

Additional contributors to this work include:Philippe Giguere, Junaed Sattar, Chris Prahacs,

Yogesh Girdhar, John-Paul Lobos, Mike Jenkin


Conclusion• Navigation: internal model, vision based

sensing.

• Real-work behavior involves many levels of control and interaction.

• Need a person “in the loop” for many real tasks. Use both implicit interaction (follow the leader) and explicit cues (“do this”).

• Vision-based Human-Robot interaction is rich and effective.


Future Work• When should the robot ignore it’s

programming or otherwise act on it’s own?

• When should the robot ask for help?

• What constitutes a useful data item?

• Challenges of optimizing a language for human-robot interaction.

• Software systems that span multiple levels of control, but are easy to use. (You can ask about these...)

Gregory Dudek, http://www.cim.mcgill.ca/~dudekGregory Dudek, http://www.cim.mcgill.ca/~dudek

AQUA underwater robot


AQUA underwater robot Sub-optimal legs (1)

• Good for walking, bearable but poor for swimming.

Sub-optimal legs (2)• Good for swimming, bearable but poor for walking.

Amphibious legs

• Fast

• Too fast

Amphibious legs

• Fast

• Too fast

human-robot outline collection - mcgill university

Documents