robotic vision - vision for robotics #ieee #qld #cis #colloquium
TRANSCRIPT
Robotic Vision
Jürgen ‘Juxi’ Leitner arc centre of excellence for robotic vision
queensland university of technology
a vision for robotics
[email protected] - http://Juxi.net
http://roboticvision.org/2
2013
http://roboticvision.org/3
recognising objects & stuff
recognising places
detec4ng mo4on move to see
see to move
context for seeing
seeing for context
seeing creates memories
memory helps seeing
paying a;en4on
recognizing humans, their ac4vi4es and intent
Seeing
http://roboticvision.org/4
10 96%eyedesigns
animalshave eyes
Nature Reviews | Neuroscience
Protostomes
Bilateria
Ecdysozoa
Lophotrochozoa
~580 Mya
~550 Mya
~530 Mya
~500 Mya
~430 Mya
Deuterostomes
Chordates*
Craniates*
Vertebrates*
Gnathostomes*
Arthropods
Annelids
Molluscs
Hemichordates
Echinoderms
Cephalochordates*
Tunicates*
Myxiniformes*
Petromyzoniformes*
Last fossil jawless fish
Stages of interest in vertebrate eye evolution Cambrian
Mya 600 550 500 450 400 0
2 1 3 4 5 6
Lampreys
Jawed vertebrates
Hagfish
Lancelets
Sea squirts
Ocellus
Eye patch
ProtostomeAn animal belonging to the protostome super-phylum, which is characterized by its members’ embryonic development, in which the first opening (the blastopore) becomes the mouth (protostome is Greek for ‘first mouth’). All protostomes are invertebrates.
DeuterostomeAn animal belonging to the deuterostome super-phylum of the animal kingdom, which is characterized by its members’ embryonic development, in which the first opening (the blastopore) becomes the anus (deuterostome is Greek for ‘second mouth’). In addition to the chordate phylum (which includeds vertebrates), the other two main phyla are the echinoderm phylum and the hemichordate phylum.
ChordateAn animal belonging to the chordate phylum, which comprises vertebrates, tunicates and cephalochordates. These animals are characterized by the presence of a notochord, a dorsal-nerve cord and pharyngeal slits or pouches.
AgnathanA jawless fish within the chordate phylum (agnatha is Greek for ‘no jaw’). The two extant groups are hagfish and lampreys.
GnathostomeThe jawed vertebrates (gnathostome is Greek for ‘jaw mouth’), comprising fish and tetrapods (including birds and mammals).
craniates, hagfish have the most basal body-plan. They possess neither jaws nor vertebrae and are therefore usually regarded not as vertebrates but rather as a sister group. The vertebrates comprise an early jaw-less (agnathan) division, of which the only living examples are lampreys, and a later jawed division, the gnathostomes, which includes fish and tetrapods.
Controversy has long surrounded the interrelation-ship between hagfish, lampreys and jawed vertebrates. BOX 1 summarizes current views, and in FIG. 1 we show hagfish diverging either before the divergence of lam-preys or else after lampreys separated from the line that would become the jawed vertebrates.
Not only has extensive gene duplication occurred throughout the evolution of animals22, but in addition it is widely accepted that two rounds of whole-genome duplication occurred early in vertebrate evolu-tion23–29; most likely, one duplication occurred before the agnathans split from the vertebrate line and one occurred after (FIG. 1; for reviews, see REFS 30–32). It is also clear that the vertebrate organizer, which deter-mines the body plan of developing embryos, arose in early chordates33–35. These genetic developments are likely to have been of crucial importance in early vertebrate evolution, but they are beyond the scope of this Review.
Figure 1 | The origin of vertebrates. The evolution of jawed vertebrates is illustrated against an approximate time-scale of millions of years ago (Mya). The taxa considered in this Review are indicated with an asterisk and are accompanied by schematics and diagrams of the ‘eye’ region. The earliest chordates, represented by extant cephalochordates and tunicates, are thought to have appeared around 550 Mya. Jawless craniates (agnathans) were present in the early Cambrian, by 525 Mya, and a time of 530 Mya has been indicated for their presumed first appearance. As elaborated on in BOX 1, there is considerable controversy as to whether myxiniformes (solely represented by extant hagfish) diverged before or after the separation of lampreys from jawed vertebrates (shown as dashed black and grey lines). Numerous lines of jawless fish evolved between 500 and 430 Mya ago, although none have survived to the present day. The first jawed vertebrate arose around 430 Mya, and this line is represented today by cartilagenous fish, bony fish and tetrapods. Six ‘stages of interest’ in vertebrate eye evolution correspond to the time intervals between the divergence of important surviving taxa. This diagram does not include the evolutionary changes that have occurred in the last 400 million years. The presented timeline is based primarily on evidence from the fossil record; see REFS 2,13,15,17,18,144,160–163. The schematics are modified, with permission, from REF. 11 (1996) Oxford University Press (lancelet, sea squirt, hagfish and lamprey) and REF. 164 (2004) Academic Press (jawed vertebrate). The eye images are reproduced, with permission, from the following references: lancelet, REF. 165 BIODIDAC (1996) University of California Museum of Paleontology; sea squirt, REF. 63 (2006) Blackwell Publishing; hagfish, REF. 166 (2006) Australian Museum. Lamprey and jawed vertebrate eye images are courtesy of G. Westhoff and S. P. Collin).
R E V I E W S
NATURE REVIEWS | NEUROSCIENCE VOLUME 8 | DECEMBER 2007 | 961
http://roboticvision.org/
http://roboticvision.org/6
tinyurl.com/QUTRobotics
roboticvision.org
http://roboticvision.org/7
http://roboticvision.org/
http://roboticvision.org/9
Juxi Leitner
h"p://Juxi.net
http://roboticvision.org/
Dalle Molle Institute for AI (IDSIA)
10
WorkJuxi
Leitner
PhD Informatics / Intelligent Systems
MSc Space Robotics & Automation
BSc Information & Software Engineering
Intelligent (Space) Robots European Space Agency (ESA)
Erasmus Intelligent Systems
Work (Humanoid) Robot VisionInstituto Superior Técnico (IST)
Mobility Intelligent Space Systems Laboratory
About Me
Work Robotic Vision and Actions
h"p://Juxi.net
Queensland University of Technology (QUT)
position, velocity, orientation
thruster
[Leitner et al, iSAIRAS 2010]
http://roboticvision.org/14
projectIM-CLeVeR
http://robotics.idsia.ch/im-clever/
perceptionvisual
thanks to G. Metta and IIT for this picture
objectsdetecting
objectsdetecting
Harding, Leitner, Schmidhuber, 2013Leitner et al., ICDL 2012, IJARS 2012, BICA 2012, CEC 2013
embedding domain knowledge
+ min dilate avg INP INP INP OpenCV func5ons
full images
+ min dilate avg INP INP INP diff
using building blocksOpenCV
+ min dilate avg INP INP INP thresh+ min dilate avg INP INP INP blur+ min dilate avg INP INP INP normalize+ min dilate avg INP INP INP input
icVision
icImage* BlueCupFilter::runFilter() { icImage* node43 = InputImages[4]; icImage* node49 = node43->LocalAvg(15);
icImage* out = node49->threshold(81.532f); return out; }
framework
cartesian genetic programming
+ min dilate avg INP INP INP
Harding, Leitner, Schmidhuber, 2013Leitner et al., ICDL 2012, IJARS 2012, BICA 2012, CEC 2013
learningapproach
detection
icImage GreenTeaBoxDetector::runFilter() { icImage node0 = InputImages[6]; icImage node1 = InputImages[1]; icImage node2 = node0.absdiff(node1); icImage node5 = node2.SmoothBilateral(11); icImage node12 = InputImages[0]; icImage node16 = node12.Sqrt(); icImage node33 = node16.erode(6); icImage node34 = node33.log(); icImage node36 = node34.min(node5); icImage node49 = node36.Normalize();
//cleanup ... icImage out = node49.threshold(230.7218f); return out; }
detect
detect
detection
approachcgp
[Leitner et al, iSAIRAS 2012]
approachsupervised learning
BUT
clusteringfeature
saliencymap
Autonomous Approach
[Leitner et al, ICDL/EpiRob 2012]
presegmentation
resultscomparing
[Leitner et al, ICDL/EpiRob 2012]
features
CGP-‐IP
learningspatial perception
trainingset
9DOF
iCubpositio
n in the frame
2/6 per eye
Carte
sian
Coor
dinate
s
setuplearning
trainingset
9DOF
iCubpositio
n in the frame
2/6 per eye
Carte
sian
Coor
dinate
s
.
.
.~1000
spatial perception neural network
...
9DO
FiC
ubpo
sitio
n in
the
imag
e2/
6 pe
r eye
Cart
esian
Coor
dina
tes
fully
con
nect
ed
fully
con
nect
ed
...
results ANN
-700
-600
-500
-400
-300
-200
-100
0
100
200
300
400
0 200 400 600 800 1000
Y (
mm
)
Sample Index
PredictedExpected
100
150
200
250
300
350
400
450
500
550
600
0 200 400 600 800 1000
X (
mm
)
Sample Index
-1300
-1200
-1100
-1000
-900
-800
-700
-600
-500
-400
-300
-200
0 200 400 600 800 1000
Z (
mm
)
Sample Index
-700
-600
-500
-400
-300
-200
-100
0
100
200
300
400
0 200 400 600 800 1000
Y (
mm
)
Sample Index
PredictedExpected
100
150
200
250
300
350
400
450
500
550
600
0 200 400 600 800 1000
X (
mm
)
Sample Index
-1300
-1200
-1100
-1000
-900
-800
-700
-600
-500
-400
-300
-200
0 200 400 600 800 1000
Z (
mm
)
Sample Index
[Leitner et al, IJCNN 2013]
MoBeEv2[Frank, Leitner et al. 2012, 2013]
hand/armop-space forcing
CSWorld
CSHand
CSR/CSL
[Leitner et al, in prep]
teleoperation
generationmotion
Shak
ey 2
013
Win
ner
coordinationhand-eye
model
manipulation for improved perception
http://roboticvision.org/57
case #1 learning
from supervised to robotic-assisted unsupervised learning
Autonomous Learning Of Robust Visual Object Detection And Identification On A Humanoid. J. Leitner, P. Chandrashekhariah, S. Harding, M. Frank, G. Spina, A. Förster, J. Triesch, J. Schmidhuber. ICDL/EpiRob 2012.
http://roboticvision.org/58
case : poking
segmentation before & after action
http://roboticvision.org/59
deep learning visual control[Zhang et al, ACRA 2015/ICRA2016]
http://roboticvision.org/60
deep learning visual control[Zhang et al, ACRA 2015/ICRA2016]
http://roboticvision.org/61
conclusions
a novel way of object segmentationlearning and teaching perceptionintegration action-perception side
reactive reaching/graspingimproving perception with (inter-)actions
learning neuro-controllers
http://roboticvision.org/
Australian Moonshot
62
tinyurl.com/QUTLunaRoo
http://roboticvision.org/
Australian Deep Learning WS
63
http://Juxi.net/workshop/deeplearning-applications-vision-robotics-2015/
for listeningthank [email protected] http://Juxi.net/projects