Descriptive Semantic Image Retrieval

David NortonDerral Heath

Motivation Retrieve an image based on a descriptive

query: “Find me an image that is red, dark, scary, and

beautiful”

Content-Based Image Retrieval Retrieve an image strictly from image features

color texture shape

General semantic based image retrieval is hard “Find me a picture of a piranha”

Emotional Semantic Image Retrieval Query images matching emotional words or

word-pairs “Find me a happy picture”

Usually adjectives

Usually a small orthogonal subset of terms

Query via single words (or pairs)

Descriptive Semantic Image Retrieval Open to all descriptive words

Query via any number of words “Find me an image that is red, dark, scary, and

beautiful”

Three Components Extraction of image features

Semantic representation of image

Mapping between visuals and language

Extraction of Image Features Color (12)

Average RGB values Color count

Texture (50) Entropy

Shape (10) Eccentricity

Extraction of Image Features

Average R Average G Average B Hue Count Hue Percent Entropy Eccentricity

114.0887 99.0219 118.648 20 0.1332 8.924632 0.109940296

113.5899 98.5232 118.145 18 0.1411 8.200488 0.101902564

113.6192 98.5358 118.153 20 0.1511 8.015762 0.082569348

Extraction of Image Features

Average R Average G Average B Hue Count Hue Percent Entropy Eccentricity

146.4833 75.8135 1.547 5 0.5259 7.805696 0.064152

145.9641 75.2931 1 4 0.5317 7.654775 0.073264

146.0377 75.3628 1.054 4 0.5496 7.502806 0.054083

Semantic Representation of Image How do we obtain a description?

What is a descriptive word?

What are the features?

User Input Interface

Bright has 11 Senses 1. (17) bright -- (emitting or reflecting light readily or in large amounts; "the sun was bright and hot"; "a

bright sunlit room") 2. (6) bright, brilliant, vivid -- (having striking color; "bright dress"; "brilliant tapestries"; "a bird with vivid

plumage") 3. (5) bright, smart -- (characterized by quickness and ease in learning; "some children are brighter in one

subject than another"; "smart children talk earlier than the average") 4. (3) bright -- (having lots of light either natural or artificial; "the room was bright and airy"; "a stage bright

with spotlights") 5. (1) bright, burnished, lustrous, shining, shiny -- (made smooth and bright by or as if by rubbing;

reflecting a sheen or glow; "bright silver candlesticks"; "a burnished brass knocker"; "she brushed her hair until it fell in lustrous auburn waves"; "rows of shining glasses"; "shiny black patents")

6. (1) bright -- (splendid; "the bright stars of stage and screen"; "a bright moment in history"; "the bright pageantry of court")

7. undimmed, bright -- (not made dim or less bright; "undimmed headlights"; "surprisingly the curtain started to rise while the houselights were still undimmed")

8. bright, brilliant -- (clear and sharp and ringing; "the bright sound of the trumpet section"; "the brilliant sound of the trumpets")

9. bright -- (characterized by happiness or gladness; "bright faces"; "all the world seems bright and gay") 10. bright, shining, shiny, sunshiny, sunny -- (abounding with sunlight; "a bright sunny day"; "one shining

morning"- John Muir; "when it is warm and shiny") 11. bright, promising -- (full or promise; "had a bright future in publishing"; "the scandal threatened an

abrupt end to a promising political career")

Narrowing Down the Feature Space Interface:

Adjectives from WordNet

Restrict characters

Reduce available senses

Narrowing Down the Feature Space Post Processing:

Use Synsets

Frequent synsets

Fit ORM ontology lexicons

Image ORM Ontology

Mapping between visuals and language Series of Neural Networks

Bayes Net

Fuzzy Logic

Evaluation Let machine label images

Let humans label images

Let different humans evaluate machine and human labels

Compare evaluations

Related Work Aesthetic Visual Quality Assessment of

Paintings (2009) Congcong Li and Tsuhan Chen

Labeled impressionistic style landscape paintings as ‘high’ or ‘low’ quality using machine learning.

Algorithmic Inferencing of Aesthetics and Emotion in Natural Images: An Exposition (October 2008) Ritendra Datta, Jia Li, and James Z. Wang

Overview of research involving predicting the quality class, score, and emotional label of photographs.

Related Work A Survey on Emotional Semantic Image

Retrieval (2008) Weining Wang and Qianhua He

Surveys ongoing Emotional Semantic Image Retrieval research.

Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction (October 2006) Wang Wei-ning, Yu Ying-lin, and Jiang Sheng-ming

Labeled paintings with 12 emotional word pairs. Psychological research involved in choosing word pairs.

Further Motivation Augment the study of human perception and

cognition.

Establish a linguistic-visual foundation for an artificially creative artist.

Questions?