cs 378: autonomous intelligent roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · surf...

90
CS 378: Autonomous Intelligent Robotics Instructor: Jivko Sinapov http://www.cs.utexas.edu/~jsinapov/teaching/cs378/

Upload: others

Post on 09-Mar-2021

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

CS 378: Autonomous Intelligent Robotics

Instructor: Jivko Sinapovhttp://www.cs.utexas.edu/~jsinapov/teaching/cs378/

Page 2: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Multimodal Perception

Page 3: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Announcements

Final Projects Presentation Date:

Thursday, May 12, 9:00-12:00 noon

Page 4: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Project Deliverables

• Final Report (6+ pages in PDF)

• Code and Documentation (posted on github)

• Presentation including video and/or demo

Page 5: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Multi-modal Perception

Page 6: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

The “5” Senses

Page 7: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

The “5” Senses

[http://edublog.cmich.edu/meado1bl/files/2013/03/Five-Senses2.jpg]

Page 8: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

The “5” Senses

[http://edublog.cmich.edu/meado1bl/files/2013/03/Five-Senses2.jpg]

Page 9: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 10: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

[http://neurolearning.com/sensoryslides.pdf]

Page 11: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 12: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

How are sensory signals from different modalities integrated?

Page 13: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 14: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

[Battaglia et. al. 2003]

Page 15: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Locating the Stimulus Using a Single Modality

Standard Trial

Comparison Trial

Is the stimulus in Trial 2 located to the left or to the right of the stimulus in Trial 1?

Page 16: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Locating the Stimulus Using a Single Modality

Standard Trial

Comparison Trial

Is the stimulus in Trial 2 located to the left or to the right of the stimulus in Trial 1?

Page 17: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 18: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 19: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Multimodal Condition

Standard Trial

Comparison Trial

Page 20: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 21: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 22: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

[Ernst, 2006]

Page 23: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 24: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Take-home Message

During integration, sensory modalities are weighted based on their individual

reliability

Page 25: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Further Reading

Ernst, Marc O., and Heinrich H. Bülthoff. "Merging the senses into a robust percept." Trends in cognitive sciences 8.4 (2004): 162-169.

Battaglia, Peter W., Robert A. Jacobs, and Richard N. Aslin. "Bayesian integration of visual and auditory signals for spatial localization." JOSA A 20.7 (2003): 1391-1397.

Page 26: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Sensory Integration During Speech Perception

Page 27: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

McGurk Effect

Page 28: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

McGurk Effect

https://www.youtube.com/watch?v=G-lN8vWm3m0

https://vimeo.com/64888757

Page 29: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 30: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Object Recognition Using Auditory and Proprioceptive Feedback

Sinapov et al. “Interactive Object Recognition using Proprioceptive and Auditory Feedback”International Journal of Robotics Research, Vol. 30, No. 10, September 2011

Page 31: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

What is Proprioception?

“It is the sense that indicates whether the body is moving with required effort, as well as where the various parts of the body are located in relation to each other.”

- Wikipedia

Page 32: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Why Proprioception?

Page 33: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Why Proprioception?

FullEmpty

Page 34: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Why Proprioception?

Hard Soft

Page 35: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Exploratory BehaviorsLift:

Drop:

Push:

Shake:

Crush:

Page 36: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Objects

Page 37: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Sensorimotor Contexts

lift

shake

drop

press

push

audio proprioception B

ehav

iors

Sensory Modalities

Page 38: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Feature ExtractionJ 1

J 7.

. .

Time

Page 39: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Feature Extraction

Training a self-organizing map (SOM) using sampled joint torques:

Training an SOM using sampled frequency distributions:

Page 40: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Discretization of joint-torque records using a trained SOM

is the sequence of activated SOM nodes over the duration of the interaction

Discretization of the DFT of a sound using a trained SOM

is the sequence of activated SOM nodes over the duration of the sound

Feature Extraction

Page 41: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Proprioceptive Recognition Model

Auditory Recognition Model

Weighted Combination

Proprioception sequence Audio sequence

Page 42: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 43: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Accuracy vs. Number of Objects

Page 44: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Accuracy vs. Number of Behaviors

Page 45: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 46: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Results with a Second Dataset• Tactile Surface Recognition:

– 5 scratching behaviors

– 2 modalities: vibrotactile and proprioceptive

Artificial Finger Tip

Sinapov et al. “Vibrotactile Recognition and Categorization of Surfaces by a Humanoid Robot”IEEE Transactions on Robotics, Vol. 27, No. 3, pp. 488-497, June 2011

Page 47: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Surface Recognition Results

Chance accuracy = 1/20 = 5 %

Page 48: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Scaling up: more sensory modalities, objects and behaviors

Microphones in the head

Torque sensors in the joints

ZCam (RGB+D)

Logitech Webcam

3-axis accelerometer

Page 49: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

100 objects

Page 50: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Exploratory Behaviors

grasp lift hold shake drop

tap poke push press

Page 51: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Object Exploration Video

Page 52: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Object Exploration Video #2

Page 53: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Coupling Action and Perception

… … …

… … …

Time

Action: poke

Perception: optical flow

Page 54: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

look

grasp

lift

hold

shake

audio (DFT)

proprioception (joint torques)

Color

drop

tap

poke

push

press

Optical flow

SURFproprioception (finger pos.)

Sensorimotor Contexts

Page 55: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Sensorimotor Contexts

look

grasp

lift

hold

shake

audio (DFT)

proprioception (joint torques)

Color

drop

tap

poke

push

press

Optical flow

SURFproprioception (finger pos.)

Page 56: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Feature Extraction: Proprioception

Joint-Torque values for all 7 Joints

Joint-Torque Features

Page 57: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Feature Extraction: Audio

audio spectrogram

Spectro-temporal Features

Page 58: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Feature Extraction: Color

Color Histogram (4 x 4 x 4 = 64 bins)

Object Segmentation

Page 59: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Feature Extraction: Optical Flow

… … …

Angular bins

Co

un

t

Page 60: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Feature Extraction: Optical Flow

… … …

Page 61: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Feature Extraction: SURF

Page 62: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Feature Extraction: SURF

Each interest point is described by a 128-dimensional vector

Page 63: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Feature Extraction: SURF

Visual “words”

Co

un

t

Page 64: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Dimensionality of Data

audio (DFT)

proprioception (joint torques)

ColorOptical

flowSURF

proprioception (finger pos.)

100 70 64 10 2006

Page 65: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Data From a Single Exploratory Trial

look

grasp

lift

hold

shake

audio (DFT)

proprioception (joint torques)

Color

drop

tap

poke

push

press

Optical flow

SURFproprioception (finger pos.)

Page 66: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Data From a Single Exploratory Trial

look

grasp

lift

hold

shake

audio (DFT)

proprioception (joint torques)

Color

drop

tap

poke

push

press

Optical flow

SURFproprioception (finger pos.)

x 5 per object

Page 67: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Overview

Category Recognition Model

Sensorimotor Feature Extraction

Interaction with Object Category Estimates

Page 68: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Context-specific Category Recognition

Observation from poke-audio context

Mpoke-audio

Recognition model for poke-audio context

Distribution over category labels

Page 69: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

• The models were implemented by two machine learning algorithms:K-Nearest Neighbors (k = 3)Support Vector Machine

Context-specific Category Recognition

Page 70: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Support Vector Machine• Support Vector Machine: a discriminative learning algorithm

1. Finds maximum margin hyperplane that separates two classes

2. Uses Kernel function to map data points into a feature space in which such a hyperplane exists

[http://www.imtech.res.in/raghava/rbpred/svm.jpg]

Page 71: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Combining Model Outputs

. . . .

Mlook-color Mtap-audio Mlift-SURF Mpress-prop.. . . .

Weighted Combination

Page 72: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Model Evaluation: 5 fold Cross-Validation

Train Set Test Set

Page 73: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Recognition Rates (%) with SVMAudio Proprioception Color Optical Flow SURF All

look 58.8 58.9 67.7

grasp 45.7 38.7 12.2 57.1 65.2

lift 48.1 63.7 5.0 65.9 79.0

hold 30.2 43.9 5.0 58.1 67.0

shake 49.3 57.7 32.8 75.6 76.8

drop 47.9 34.9 17.2 57.9 71.0

tap 63.3 50.7 26.0 77.3 82.4

push 72.8 69.6 26.4 76.8 88.8

poke 65.9 63.9 17.8 74.7 85.4

press 62.7 69.7 32.4 69.7 77.4

Page 74: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 75: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 76: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 77: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 78: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold
Page 79: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Distribution of rates over categories

Page 80: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Can behaviors be selected actively to minimize exploration time?

Page 81: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Active Behavior Selection• For each behavior , estimate such that

• Let be the vector encoding the robot’s current estimates over the category labels and let be the remaining set of behaviors available to the robot

Page 82: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Example with 3 Categories and 2 Behaviors

A B C

A

B

C

A B C

A

B

C

A B C

Remaining Behaviors and Associated Confusion:

B1

Current Estimate:

B2

Page 83: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Active Behavior Selection: Example

A B C

A

B

C

A B C

A

B

C

A B C

Current Estimate: Remaining Behaviors and Associated Confusion:

B1 B2

Page 84: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Active Behavior Selection

Page 85: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Active vs. Random Behavior Selection

Page 86: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Active vs. Random Behavior Selection

Page 87: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

Discussion

What are some of the limitations of the experiment?

What are some ways to address them?

What other possible senses can you think of that would be useful to a robot?

Page 88: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

References

Sinapov, J., Bergquist, T., Schenck, C., Ohiri, U., Griffith, S., and Stoytchev, A. (2011) Interactive Object Recognition Using Proprioceptive and Auditory Feedback. International Journal of Robotics Research, Vol. 30, No. 10, pp. 1250-1262

Sinapov, J., Schenck, C., Staley, K., Sukhoy, V., and Stoytchev, A. (2014)Grounding Semantic Categories in Behavioral Interactions: Experiments with 100 Objects. Robotics and Autonomous Systems, Vol. 62, No. 5, pp. 632-645

Page 89: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold

THE END

Page 90: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/... · 2016. 4. 21. · SURF proprioception (finger pos.) Data From a Single Exploratory Trial look grasp lift hold