exploiting ontologies for automatic image annotation

25
Exploiting Ontologies for Automatic Image Annotation M. Srikanth, J. Varner, M. Bowden, D. Moldovan Language Computer Corporation www.languagecomputer.com Richardson, Texas

Upload: forest

Post on 21-Jan-2016

52 views

Category:

Documents


0 download

DESCRIPTION

M. Srikanth, J. Varner, M. Bowden, D. Moldovan Language Computer Corporation www.languagecomputer.com Richardson, Texas. Exploiting Ontologies for Automatic Image Annotation. Contents. Motivation Automatic Image Annotation Problem Ontologies for Defining Visual Vocabularies - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Exploiting Ontologies for Automatic Image Annotation

Exploiting Ontologies for Automatic Image Annotation

M. Srikanth, J. Varner, M. Bowden, D. Moldovan

Language Computer Corporationwww.languagecomputer.com

Richardson, Texas

Page 2: Exploiting Ontologies for Automatic Image Annotation

Motivation Automatic Image Annotation Problem Ontologies for

Defining Visual Vocabularies Hierarchical Models for image annotation

Related Work Experiments & Results Conclusion and Future Work

ContentsContents

Page 3: Exploiting Ontologies for Automatic Image Annotation

Majority of efforts in Q/A focus on textual corpora and processing

Large amounts of information held within multimedia sources – images/audio/video

Extend the Power of Q/A into the realm of multimedia

Exploit commonality and union of text and multimedia information

Motivation: Multimedia Question Motivation: Multimedia Question AnsweringAnswering

Page 4: Exploiting Ontologies for Automatic Image Annotation

Some ways in which multimedia can be used in Q/A Multimedia (video clip/image) as Answer Multimedia and Lexical combination providing enhanced

understanding to Answer questions

Caption: Ronaldo seals Brazil's place in the last eight with a shot through Geert de Vlieger's legs late on to eliminate Belgium

Question: What color jersey did Brazil wear in the World Cup?

Multimedia Question AnsweringMultimedia Question Answering

Page 5: Exploiting Ontologies for Automatic Image Annotation

Feature extraction High- and Low-level features

Object recognition Auto Annotation of images

Object semantics extraction Locative/temporal/etc

Build Knowledge Representation from Image/Video

Merge with audio/text Knowledge Representation Lexical information from ASR and VOCR

Provide Multimedia Q/A based using Multimedia Ontologies

ApproachApproach

Feature extraction High- and Low-level features

Object recognition Auto Annotation of images

Object semantics extraction Locative/temporal/etc

Build Knowledge Representation from Image/Video

Merge with audio/text Knowledge Representation Lexical information from ASR and VOCR

Provide Multimedia Q/A based using Multimedia Ontologies

Page 6: Exploiting Ontologies for Automatic Image Annotation

Automatic Image AnnotationAutomatic Image Annotation

Task of automatically assigning words to an image that describe the contents of the image

Most models exploit the correlation between images and words

Exploit the correlation between the annotation words themselves to1. Define visual vocabularies

2. Develop hierarchical models for automatic image annotation

Use ontological information about annotation words to improve image annotation

Page 7: Exploiting Ontologies for Automatic Image Annotation

Models for translating visual representation of concept to textual representation (Duygulu et al., 2002)

Based on Brown model for Machine Translation (Brown et al., 1993)

Image Features translate to Annotation Words K-Means used to cluster image features to generate

blobs

Dependencies between blobs and words is not explicitly captured

Use ontology to drive the definition of blobs

Prior Work: Translation ModelsPrior Work: Translation Models

Page 8: Exploiting Ontologies for Automatic Image Annotation

Hierarchical Aspect Cluster Model (T. Hofmann, 1998)

Induces an hierarchical structure from co-occurrence of image features

Topology is externally defined Depth of the induced hierarchy is user selected Levels define the generality of the concept

expressed in regions and words

The hierarchies defined in ontologies have well-defined semanticsImage feature hierarchy induced from a text ontology

Prior Work: HACM ModelPrior Work: HACM Model

Page 9: Exploiting Ontologies for Automatic Image Annotation

Estimate P(w|I) to classify an Image I (represented by image features) into one of the classes (annotation word w)

Generative Models Flat classification: Learn one classifier per annotation word SVM Classifier (Cusano et al., 2004)

Discriminative Models Jeon and Manmatha (2004) showed improvements over

translation using Maximum Entropy Models Unigram (blob, word) and Bigram: (horizontal blob pairs,

word) feature

Explore hierarchical classification using ontology

Prior Work: Classification ApproachesPrior Work: Classification Approaches

Page 10: Exploiting Ontologies for Automatic Image Annotation

Image Representation usingImage Representation usingVisual VocabularyVisual Vocabulary

Image Segmentatio

n

Feature Extraction

Image Representation

Image

Image Segmentation1. Image regions corresponding to objects in the image2. Grid-based image segmentation

Feature Extraction Extract image features from image regions

Color, Shape, Texture

Image Representation1. real-valued feature vectors2. Visual vocabulary derived based on clustering feature

vectors Cluster centers (Blobs) define the vocabulary

Page 11: Exploiting Ontologies for Automatic Image Annotation

Visual vocabulary from OntologiesVisual vocabulary from Ontologies

Image regions from images are organized in the hierarchy based on the image annotation

Image attributes of children nodes are related parent node’s image attributes

Page 12: Exploiting Ontologies for Automatic Image Annotation

Using Ontologies in Translation Models Using Ontologies in Translation Models for Automatic Image Annotationfor Automatic Image Annotation

1. Ontology-induced visual vocabulary– Annotation word hierarchy used in selecting the initial set of

blobs for K-means clustering

2. Ontology-weighed K-means clustering– Weight the cluster membership of image regions in the

estimation of cluster centers (blobs)

c

r

Rr

Ww

rfcrwtc

Wcn

cwncrwt

)(),(

||)(

1),(),(

*

n(w,c) – number of image regions in cluster c associated with word wn(c) – number of image regions in cluster cf(r) – feature vector for region r

Page 13: Exploiting Ontologies for Automatic Image Annotation

Image Annotation by Hierarchical Image Annotation by Hierarchical ClassificationClassification

• Based on hierarchical approach to text classification (McCallum et al., 1998)– Statistical, back-off model induced by the hierarchy derived from

annotation word ontology

– Given an image I with blob sequence , the probability of word w is given by

– Assuming a Bernoulli model for annotations, the blob likelihood given a word is estimated as

},...,,{ 21 mbbb

||

1 1

1

),|()|(

),|()|(

)|(

),|()|(),|(

W

r

m

l rlr

m

l l

wbPwP

wbPwP

IP

wIPwPIwP

||

1

||

1

||

1

)|(),(

)|(),(1),|( V

s

T

i iis

T

i ii

JwPJbNV

JwPJbNwbP

V – Visual vocabularyT – Training set of annotated imagesW – Set of annotation words

Page 14: Exploiting Ontologies for Automatic Image Annotation

Image Annotation using Hierarchical Image Annotation using Hierarchical Classification (contd.)Classification (contd.)

The IS-A hierarchy among annotation words is used to estimate blob-likelihood probability

))(),'|'(),'|'(

),'|'(),'|'(()'|'(

bPanimalbPfelinebP

catbPtigerbPftigerbP

mlemle

mlemle

wv

mle vbPwvwbP )|(),(),|(

tiger

cat

feline

animal

ROOT

cougar leopard lion lynx

• Feature weights learned using EM algorithm

Page 15: Exploiting Ontologies for Automatic Image Annotation

Corel Data Set Annotated images using pre-processed data from

(Duygulu, et al., 2002) 4500 images annotated using 374 words 4000 for training; 500 for testing

Image Representation Image Segmentation using N-cuts (Duygulu et al.,

2002) 36 different image features represent each image

region

Ontology: WordNet Hierarchy with 714 unique concepts was induced from

374 annotation words

ExperimentsExperiments

Page 16: Exploiting Ontologies for Automatic Image Annotation

Annotation systems predict P(w|I) A cut-off or threshold required to assign annotations Unnormalized: take top 5 words Normalized: take top m words, where m is #of

annotations for I

Metrics Number of words of positive recall Mean per-word Precision-Recall

All words in the dictionary Selected set of words

Retrieved: words retrieved using the method Common: words predicted by all annotation systems Union: all words predicted by at least one annotation system

Image Annotation EvaluationImage Annotation Evaluation

Page 17: Exploiting Ontologies for Automatic Image Annotation

Features Description Precision Recall Predicted Positive Recall

KM-500 Baseline K-means clustering 0.2204 0.2412 28 27

WKM-500 Weighted K-means clustering 0.2042 0.2524 27 26

ONT-714Using 714 clusters with one cluster per word in the induced ontology

0.2634 0.2724 36 35

ONT-500Reducing ONT-714 to 500 clusters by combining “close clusters”

0.2482 0.2499 33 32

Results: Translation Models and Results: Translation Models and OntologiesOntologies

Precision/Recall numbers are average over “pooled” set of 42 words Observations

Using ontologies increase the number of words predicted with postive recall

Hierarchy based initial clusters attaches better semantics to clusters

Results for ontology-induced clusters is based on ‘One blob per concept’

Page 18: Exploiting Ontologies for Automatic Image Annotation

Results: Classification Approaches and Results: Classification Approaches and OntologiesOntologies

Comparing Flat classification versus Hierarchical classification for image annotations

Features Precision Recall # Ret. #Pos. Recall

Flat + KMeans-500 0.1627 0.2766 152 86

Hier + KMeans-500 0.1805 0.3174 146 93

Precision/Recall numbers correspond to using the KM-500 visual vocabulary

Observations Improved Precision (10%) and Recall (14%) values Increase in number of annotations with positive recall Hierarchy derived from annotation ontology results in improved

performance

Page 19: Exploiting Ontologies for Automatic Image Annotation

Results: Hierarchical Classification with Results: Hierarchical Classification with Ontology-induced Visual VocabulariesOntology-induced Visual Vocabularies

Hierarchical approach improves precision/recall values on different visual vocabularies

ONT-714 has improved positive recall numbers Ontologies defined on text annotations provide a good

framework for developing hierarchical models for image features

Measures KM-500 WKM-500 ONT-714 ONT-500

Baseline – Flat Classification Method

Precision 0.1627 0.1867 0.1647 0.1643

Recall 0.2766 0.2831 0.2724 0.2697

Predicted 152 153 150 141

Positive Recall 86 90 84 80

Hierarchical Classification Method

Precision 0.1805 0.1882 0.1723 0.1754

Recall 0.3174 0.3135 0.2926 0.2903

Predicted 146 140 150 137

Positive Recall 93 91 91 81

Page 20: Exploiting Ontologies for Automatic Image Annotation

Results: Comparing Translation and Results: Comparing Translation and Classification ApproachesClassification Approaches

Measures KM-500 WKM-500 ONT-714 ONT-500

# Common Words 27 26 35 32

Translation Method

Precision 0.3270 0.3134 0.3040 0.3124

Recall 0.3720 0.4043 0.3244 0.3253

Flat Classification Method

Precision 0.3243 0.3157 0.2924 0.3000

Recall 0.5666 0.5649 0.5591 0.5632

Hierarchical Classification Method

Precision 0.3223 0.3104 0.3018 0.3068

Recall 0.5652 0.5362 0.5453 0.5605

Comparison based on common annotation words predicted by different models

Significant improvement in recall using classification approaches

Page 21: Exploiting Ontologies for Automatic Image Annotation

Experimental Results: Ontology in translation model

19.5% increase in average precision 13% increase in average recall

Ontology in classification 10% increase in average precision 14% increase in average recall

Using word hierarchies improve annotation results when used as a source for selecting initial blobs, and as framework for hierarchical classification

Ontologies in Automatic Image Ontologies in Automatic Image AnnotationAnnotation

Page 22: Exploiting Ontologies for Automatic Image Annotation

Proposed methods for using ontologies in automatic image annotation

Translation Models: Defining Visual vocabulary Hierarchical Classification Models: Provide the

hierarchy for models defined image features Explore the use of ontologies in other approaches to

automatic image annotation Discriminative models

Exploit the dependence between annotation words in automatic image annotation

Correlation between annotation words of an image can be exploited

Summary and Future WorkSummary and Future Work

Page 23: Exploiting Ontologies for Automatic Image Annotation

Utilize hierarchical organization of concepts and language models on image blobs to develop multi-modal ontologies

Use multi-modal ontologies in Q/A

Summary and Future Work (Contd.)Summary and Future Work (Contd.)

Page 24: Exploiting Ontologies for Automatic Image Annotation

Transportation WordNet hierarchy with Multimedia data

Multimedia Ontology: Example NodeMultimedia Ontology: Example Node

Page 25: Exploiting Ontologies for Automatic Image Annotation

Thank You.