Improved image retrieval using automatic image sorting and semi-automatic generation of image semantics

Kai Uwe Barthel FHTW Berlin, Treskowallee 8, 10313 Berlin, Germany, barthel@fhtw-berlin.de

Problem of Image Search SystemsBoth keyword-based and content-based image retrieval systems are not capable of searching images according to the human high-level semantics of images.

Keyword-Based Image Search

Most Internet images have not been annotated manually. Keywords are taken from the file names or words from the context of the web page containing the image.

Conventional Internet Image Search System (Query: „Eiffel Tower“)

As the intention of the searching user is not known to the search system, there will be problems due to homonyms, family names, labels, etc.:

Content-Based Image Retrieval (CBIR)CBIR systems rely on the assumption that images similar to a query image do also share similar features.There is an important semantic gap between the low level features and human high level semantic concepts. Even sophisticated CBIR systems cannot determine similarities between semantically similar images.

Relevance Feedback (RF)is used improve the retrieval quality. Problems of RF: Users do not like to give feedback, feedback will be incomplete. The search system does not know why a feedback was given, which feature was the reason for the feedback.

Combining keyword search and CBIRRelated approaches to combine high-level semantic keyword-based metadata and low-level statistical metadata to improve image retrieval: Image retrieval systems allowing the user to

choose from either a semantic or a visual queryWei Wang et al. (2003): SemView: A Semantic-sensitive Distributed Image Retrieval System

Automatic image annotationImages with similar low-level metadata were assigned to the same keywords. Jia-Yu Pan et al., 2004: Graph-based Automatic Image Captioning

Problematic with images from different sources!

Image Search Techniques combining semantic keywords and low-level features.By using relevance feedback techniques links are assigned between the images and the keywords.Y. L. Lu et al., 2000: A unified framework for semantics and feature based relevance feedback in image retrieval systems X. Zhou, T. Huang, 2002: Unifying Keywords and Visual Contents in Image Retrieval

Linking images to keywords does help to filter out not-suiting images, but cannot help to distinguish between homonyms or different types of images.

Proposed New SchemeNew image search scheme using both keywords and low-level visual metadata to semi-automatically generate semantic inter-image relationships.CBIR techniques to visually sort retrieved images The visually sorted arrangement allows to inspect more result images. Within this larger set the user can quickly identify those images, which are good candidates for his desired search result.

New Idea: Images are not linked to keywords but images are linked to other semantically similar images.

Semantic relationships are learned exclusively from the human users’ interaction with the image search system – from the . The proposed system can be used to search huge image sets more efficiently.

Overview of the Proposed System

Visual Sorting

Images are sorted using a self-organizing map (SOM).

Best sorting results using optimized features derived from the MPEG-7 Color-Layout descriptor: Only strong variance DCT coefficients Higher weight for chrominance coefficients No quantization Square torus-shaped Batch SOM

Very fast sorting due to incremental filters(200 images in less than 50 ms)

Candidate images will be used to visually filter (refine) the search result and to learn the semantic inter-image relationships.

Visual Filtering

Best visual filtering results were achieved using the multimodal neighborhood signature.

How to Collect Semantic InformationUsers do not like to give feedback. However, they will mark candidate images in order to improve the quality of the retrieved image search result.

By selecting a set of candidate images a user expresses the fact that according to his desired search result these images do share some common semantic meaning.

Even though the particular semantic relationship is not known to the system, this selection of candidate sets - when collected over many searches - can be used to semi-automatically model the semantic inter-image relationships.

Inter-image relationships are described with weighted links, that describe how often two particular images have been selected together as part of a candidate set.

Every time a user selects of two or more candidate images, the links of these images will be updated.

Semantic Filtering

Semantic filtering can be achieved by retrieving those images having the highest link weight.

=

€

≠

Internetimage search

Small searchresult set

Display of unsorted images

Keywordquery

Internetimage search

1: Keyword query

Display of images sorted by visual

similarity

Larger searchresult set

Huge search result set

Refined result set

Calculation of visual filter parameters

Database

Calculation of semantic filter parameters

Internetimage search

2a: Selection of result candidates

2b: Query with the same keyword

Visual sorting

Visual filter

Semantic filter

Unsorted result images (in the order as delivered by Google)

Visually sorted result images(candidate images marked in red)

Refined result images (visually filtered from 840 images)

Query Image

...

Result Images