Semi-Automatic Semi-Automatic Image AnnotationImage Annotation

Liu Wenyin, Susan Dumais, Yanfeng Sun, HongJiang Zhang, Mary Czerwinski

and Brent FieldMicrosoft Research

Outline

Introduction: What, Why, and How

Our Approach: Semi-Automatic Processes and Algorithms

Automated Performance Evaluation

Usability Studies

Concluding Remarks

What it is and Why

Image Annotation is a process of labeling images with keywords to describe semantic content

For image indexing and retrieval in image databases

Annotated images can be found more easily using keyword-based search

Image Annotation Approaches

Totally Manual Labeling (Gong et al., 1994) Enter keywords when image is loaded/registered/browsed Accurate but labor-intensive, tedious, and subjective

Direct Manipulation Annotation (Shneiderman and Kang 2000) Drag and drop keywords (from a predefined list ) onto image Still manual, also limited to predefined keywords (can’t be many)

Automatic Approaches: Efficient but less reliable and not always applicable compared to human annotation---how to grab this when no text context? By Image Understanding/Recognition (Ono et al. 1996) By Associating with environmental text (Shen et al. 2000; Srihari et al.

2000; Lieberman 2000)

Our Proposed Approach

Semi-Automatic Approach User provides initial query and relevance feed back. Feedback used to “semi-automatically” annotate ima

ges Trade-off between manual and automatic Achieve both accuracy and efficiency Increase productivity

Employ Content-Based Image Retrieval (CBIR), text matching, and Relevance Feedback (RF)

CBIR and RF Process and Framework

Image Retrieval and Relevance Feedback System (IRRFS)

Image Browser Query

Interface

Image Retrieval and Relevance

Feedback Algorithm Module

Feedback

Interface

UI

User

Algorithms for Matching

Visual Similarity Measurement Features: color histogram/moments/coherence, Tam

ura coarseness, pyramid wavelet texture, etc Distance model: Euclidean distance

Semantic (Keywords) Similarity Measurement Features: keyword vectors, TF*IDF Metrics: dot product and cosine normalization

Overall similarity: weighted average of the above two

Algorithms to Refine Search

Image Relevance Feedback Algorithms There are many algorithms can be used Cox et al. (1996) Rui and Huang (2000) Vasconcelos and Lippman (1999)

Lu et al. 2000 is employed in MiAlbum for text and images Modified Rocchio’s Formula Uses both semantics (keywords) and image-based

features during relevance feedback

Semi-Automatic Annotation During Relevance Feedback

In each keyword-query search cycleWhen positive and negative examples provided, Increase the weight of the keyword for all positive

examples Decrease the weight of the keyword for all negative

examples Relevance feedback algorithm refines and puts more

relevant images in top ranks for further selection as positive examples

Repeat the feedback process

Possible Future Automatic Annotation

When a new image is added…

Find top N similar images using image metrics

Most frequent keywords among annotations of these top N similar images are potential annotations, and could be automatically added with low weight or presented to user as potential annotations

TBD--Need to be confirmed in further RF process

Automated Performance Evaluation

Test Ground Truth Database 12,200 images in 122 categories from Corel DB Category name is ground truth annotation

Automatic Experimental Process Use category name as query feature for image retrieval Among first 100 retrieved images, those belonging to this

category are used as positive feedback examples others as negative

Performance Metrics Retrieval accuracy and annotation coverage

Image retrieval accuracy and annotation coverage

0

10

20

30

40

50

60

70

80

90

1001 3 5 7 9 11 13

15

17

19

# of Feedback Iterations

Imag

e R

etri

eval

Acc

ura

cy/

An

no

tati

on

Co

ver

age

(%)

10% initialannotation

0% initialannotation

Usability Studies

Objectives 2 studies examined overall usability of MiAlbum The usability of the semi-automatic annotation strategy

Tasks Import pictures, annotate pictures, find pictures, and use

relevance feedback

Questionnaires including but not limited to Overall ease of entering annotations for images Impact of annotation on ease of searching for images Satisfaction of search refinement & relevance feedback

Questionnaire Results

Overall ease of entering annotations: 5.6/7.0Ease to search annotated photos: 6.3/7.0Intuitiveness of refining search: 4.1/7.0Other Comments Positive on “semi-automatic”: (1) When using the up

and down hands the software automatically annotated the photos chosen. (2) The ability to rate pictures on like/dislike and have the software go from there.

Negative: difficulties in understanding the feedback process and how the matching algorithm operated.

Concluding Remarks

A Semi-automatic Annotation Strategy Employing Available image retrieval algorithms and Relevance feedback

Automatic Performance Evaluation Efficient compared to manual annotation? More accurate than automatic annotation

Usability Studies Preliminary usability results are promising Need to improve the discoverability of the feedback process

and the underlying matching algorithm