alex sorokin: opencv/mechanical turk presentation

Summer 09 is over.

When did that happen?

What have I done?

• Tools for annotating data• Tools to for data access• Executive to capture data autonomously• Annotation lifting• PR2 challenge

Data annotation

New modes of annotation

• Several new ways toannotate data– Multiple fast

bounding boxes– Zoom-in interface for

detailed polygonalannotation

– Grouping images totrack people

– Free text labels

http://vm7.willowgarage.com/mt/good_results/bottles-p-2/p1/

New modes of annotation

• Several new ways toannotate data– Multiple fast

bounding boxes– Zoom-in interface for

detailed polygons– Grouping images to

track people– Free text labels

http://vm7.willowgarage.com/mt/good_results/bottles-p-2/p1/

New modes of annotation

• Several new ways toannotate data– Multiple fast

bounding boxes– Zoom-in interface for

detailed polygonalannotation

– Grouping images totrack people

– Free text labels

New modes of annotation

• Several new ways toannotate data– Multiple fast

bounding boxes– Zoom-in interface for

detailed polygonalannotation

– Grouping images totrack people

– Free text labels

Grading!

• Just filter out poorresults

Much easier thanannotating yourself

• New task to run gradingon MechTurk

• Random sample ofthumbnails

• Grading by worker• Track grading conflicts

http://vm7.willowgarage.com/mt/grading/thumbnails_random/prf-jul-08-L2-p/

Grading!

• Just filter out poorresults

Much easier thanannotating yourself

• New task to rungrading on turk

• Random sample ofthumbnails

• Grading by worker

http://vm7.willowgarage.com/mt/grading/thumbnails_random/prf-jul-08-L2-p/

Data annotation server

• Django server to manage dataannotation process

• Stable server• Consistent user interface• Scalable task architecture• First 5 users• ROS integration across 2+ different masters

– Image in (over HTTP post)– cv_mech_turk/ExternalAnnotation out

http://vm7.willowgarage.com/

Data access

• Bag server– Random access into a bag file– Replay of select messages

• Bag image view– Browsing images off a bag– Example client to the bagserver

Data lifting

• 2D to 3D correspondence• Annotation of laser clouds with object

labels, image coordinates an image IDs• Image storage server• Static map of all annotations• Whiteboard detector from static object

map

Camera “calibration”

• Create a virtual cameraand move it to align withthe laser cloud or robotmodel

• Fix major calibrationerrors

http://pr.willowgarage.com/wiki/camera_offsetter

before

after

Data collection executive

• High level executive based on M2 pythonexecutive– Move to waypoints– Position head– Record data– Record action state

• Select images when the “record” action wasactive and annotate them.

PR2 challenge

• Integrated ROS components into a completebehavior.

• Preemptable high level state machine:– Allows to switch between autonomy and teleop– Preempts child actions– External GUI to watch and control the executive

• Great prototype for hardware in the loop test

PR2 challenge• Integrated ROS

components into acomplete behavior.

• Preemptable high levelstate machine:– Allows to switch between

autonomy and teleop– Preempts child actions– External GUI to watch

and control the executive• Great prototype for

hardware in the looptest and learning

Thank you