large scale tag recommendation using different image representations

Web Science & Technologies

University of Koblenz ▪ Landau, Germany

Large Scale Tag Recommendation

Using Different Image Representations

Rabeeh Abbasi

Marcin Grzegorzek

Steffen Staab

Web Science & Technologies Rabeeh [email protected]

SAMT'09, Graz, Austria2 of 20

Tag Recommendation

http://www.flickr.com/photos/64255998@N00/283890351/

ClocktowerUhrturmGrazAustriaEuropeHerbstAutumnNightNacht

47° 4′ 32″ N 15° 26′ 12″ E

ImageImage TagsTags



Tag Recommendation

http://www.flickr.com/photos/64255998@N00/283890351/

Tag Recommender


InputInput OutputOutput

47° 4′ 32″ N 15° 26′ 12″ E



Tag Recommendation

TagsGeographical Coordinates

Low-level Features

Tag Recommender



47° 4′ 32″ N 15° 26′ 12″ E



Tag Recommendation

TagsGeographical Coordinates

Low-level Features

Which image features to use?

Tag Recommender



47° 4′ 32″ N 15° 26′ 12″ E



Features Available in Large Datasets

CoPhIR Dataset contains106 million images from Flickr with

Title, location, tag, comment, etcFive standard MPEG-7 Features



Tag Recommender – System Overview

TrainingData

Location 1

Location N

Location 1Clusters

Location NClusters

Model

Features

Features



Tag Recommender

Tag Recommender – System Overview

TrainingData

Location 1

Location N

Location 1Clusters

Location NClusters

Model

Features

Features

Image Features

Tags

ClocktowerGrazAustriaEuropeAutumnNightUhrturmNachtHerbstHDRPhotomatix


47° 4′ 32″ N 15° 26′ 12″ E



Tag Recommender – Clustering

Used Standard K-Means clustering



Tag Recommender – Representative Tags

Given a cluster containing homogeneous imagesFind most representative tags

Tag Rank ~ Frequency of Users

Clocktower (4)Graz (4)Austria (3)Europe (3)Autumn (2)Night (2)Uhrturm (1)Nacht (1)Herbst (1)HDR (1)Photomatix (1)

Cluster Tags

User 2 User 4

User 1 User 3



Tag Recommender – Classification

Map a new image to one of the existing clustersFind the nearest cluster centroid

Assign representative tags of the mapped cluster to the image

RepresentativeTags

NearestCentroid



Evaluation

Evaluated on tags of ~400,000 Flickr images75% training data25% test data / gold standard

Data is randomly split based on users (not resources)Ignored

10 most frequent tags for each location (city) and Tags used by less than three users

Evaluation based on the correctly identified tags of test dataPrecisionRecallF-Measure



Micro Precision

1 2 3 4 5 10 15 20

0.0000

0.0200

0.0400

0.0600

0.0800

0.1000

0.1200

0.1400

0.1600

0.1385

0.1008

0.0867

0.0783

0.0686

0.0478

0.03820.0331

0.05020.0441

0.0412 0.0388 0.03640.0302

0.02640.0236

0.0451

0.03820.0326

0.0295 0.0274

0.02030.0170 0.0150

0.03380.0302 0.0284 0.0282 0.0275

0.0237 0.0211 0.0193

Geo EHD Tags Random

Number of tags recommended

Pre

cisi

on

Geo CoordinatesLow-level FeaturesTagsBaseline (Random)



Micro Recall

1 2 3 4 5 10 15 20

0.0000

0.0500

0.1000

0.1500

0.2000

0.2500

0.0515

0.0749

0.0966

0.1163

0.1273

0.1749

0.2071

0.2357

0.0187

0.0328

0.0460

0.0577

0.0677

0.1120

0.1469

0.1748

0.0168

0.02790.0351

0.04170.0478

0.0668

0.07950.0882

0.01260.0225

0.03170.0419

0.0511

0.0880

0.1174

0.1432

Geo EHD Tags Random


Re

call



Micro F-Measure

1 2 3 4 5 10 15 20

0

0.02

0.04

0.06

0.08

0.1

0.0751

0.0859

0.09140.0936

0.0892

0.0750

0.0645

0.0581

0.0272

0.0376

0.04350.0464 0.0474 0.0475

0.04480.0417

0.0244

0.0323 0.0338 0.0346 0.03490.0311

0.02810.0256

0.0183

0.02580.0300

0.03370.0357 0.0373 0.0358 0.0341

Geo EHD Tags Random


F-M

eas

ure



Coverage

Number of different tags correctly suggested

800

1000

1200

1400

1600

1800

2000

Geo EHD CL Tag Random

Nu

mb

er

of ta

gs c

orr

ectly

su

gg

este

d



Edge Histogram vs. Color Layout

1 2 3 4 5 10 15 20

0.03

0.03

0.04

0.04

0.05

0.05

0.0272

0.0376

0.0435

0.0464

0.0474 0.0475

0.0448

0.0417

0.0280

0.0390

0.0434

0.04540.0459

0.0466

0.0435

0.0408

EHD CL


F-M

eas

ure



Cosine vs. Euclidean

1 2 3 4 5 10 15 20

0.02

0.03

0.03

0.04

0.04

0.05

0.05

0.0451

0.0416

0.0382

0.03680.0360

0.0311

0.0281

0.0256

0.0311 0.03080.0302

0.0293 0.0291

0.0277

0.0261

0.0249

Tags (Cos) Tags (Eucl)


F-M

eas

ure



Euclidean vs. Manhattan

1 2 3 4 5 10 15 20

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

Geo (Eucl) EHD (Eucl) Random (Eucl)Geo (Manh) EHD (Manh) Random (Manh)


F-M

eas

ure



Conclusions

New images can be automatically taggedGeographical coordinates give best results

Require less computational power (just 2 dimensions)No low-level image feature analysis requiredLimited to location based tag recommendations only!

Future WorkOptimize individual modules of the system

Clustering, representative tags, classification etc.Combination of multiple image features

Thank You

☺

large scale tag recommendation using different image representations

Entertainment & Humor

tag recommendations

tag recommender classification

e tag recommender clustering

cluster tags user

frequent tags

e features available

location nlocation

location city