RECOD @ Placing Task of MediaEval 2015

L. T. Li1, J. A. V. Muñoz1, J. Almeida1,3,R. T. Calumby1,4, O. A. B. Penatti1,2, I. C. Dourado1,

K. Nogueira6, P. R. Mendes Júnior1,D. C. G. Pedronette1,5, J. A. dos Santos6,

M. A. Gonçalves6, and R. S. Torres1

Daniel MoreiraOn behalf of the authors.

1. UNICAMP – 2. UNIFESP – 3. SAMSUNG – 4. UEFS – 5. UNESP – 6. UFMG– BRAZIL –

2015 Participation

● We focused on the Localization subtask;● Innovations concerning the last year (2014)

– Rank agreggation based on Genetic Programming;

– Geocoding improvement with Ranked List Density Analysis (RLDA).

Submitted Runs

● Run 1Textual Features with Genetic Programming– Image and Video

Descriptors

BM25*TF-IDF*IBS*LMD*

Ranks

Ranked list 1Ranked list 2Ranked list 3Ranked list 4

Rank Aggregation

Single ranked list(GP-Agg - based)

* All from Lucene package (http://lucene.apache.org/core/)

GP-Agg Framework

Parameter Value

Number of generations

30

Genetics operators

Reproduction, Mutation, Crossover

Fitness functions

FFP1, WAS, MAP, NDCG

Rank Agg. methods

CombMAX, CombMIN, CombSUM, CombMED, CombANZ, CombMNZ,

RLSim, BordaCount, RRF, MRA

Submitted Runs

● Run 1Textual Features with Genetic Programming– Image and Video

Descriptors Ranks

Ranked list 1Ranked list 2Ranked list 3Ranked list 4

Rank Aggregation

Single ranked list(GP-Agg - based)

BM25*TF-IDF*IBS*LMD*

* All from Lucene package (http://lucene.apache.org/core/)

Submitted Runs

● Run 2Visual Features with RLDA– Image

Descriptor

BIC

Rank

BICRanked list

GeocodingImprovement

Improved geocode(top 100 RLDA-based)

Ranked List Density Analysis

Ranked list 1~N

ID1 LAT1 LONG1ID2 LAT2 LONG2… … ...IDX LATX LONGX

Top X items' lat/long defined as points of a OPF cluster

Node v: (lat1,long1)

edge (v, u): connect k-nn d(v, u)

d(v, u): geo-distance between v and uk=3

u: (lat2,long2)

Submitted Runs

● Run 2Visual Features with RLDA– Image

Descriptor

BIC

Rank

BICRanked list

Improved geocode(top 100 RLDA-based)

GeocodingImprovement

Submitted Runs

● Run 2Visual Features with RLDA– Video

Descriptors

LIRE 1...LIRE nHMP

Ranks

Ranked list 1...Ranked list nRanked list n+1

Rank Aggregation

Single ranked list(GP-Agg - based)

Improved geocode(top 100 RLDA-based)

GeocodingImprovement

Submitted Runs

● Run 3Multimodal Solution with RLDA– Image

Descriptors

BM25TF-IDFBSLMDBIC...SCD

Ranks

Ranked list 1Ranked list 2 Ranked list 3Ranked list 4Ranked list 5…Ranked list 8

Rank Aggregation

Single ranked list(GP-Agg - based)

Submitted Runs

● Run 3Multimodal Solution with RLDA– Video

Descriptors

BM25TF-IDFBSLMDHMP...MFCC

Ranks

Ranked list 1Ranked list 2 Ranked list 3Ranked list 4Ranked list 5…Ranked list n

Rank Aggregation

Single ranked list(GP-Agg - based)

Submitted Runs

● Run 4Textual with RLDA– Image and Video

Descriptors

BM25TF-IDFBSLMD

Ranks

Ranked list 1Ranked list 2 Ranked list 3Ranked list 4

Rank Aggregationand Improvement

Single ranked list(top 5 RLDA-based)

Results – 2015 Global

Run 1

Run 2

Run 3

Run 4

0 5 10 15 20 25 30 35 40 45 50

0,15

0

0,14

0,12

0,54

0,01

0,53

0,62

5,49

0,09

5,35

6,44

19,75

0,44

19,11

21,74

36,6

1,99

35,31

38,38

44,89

3,57

43,26

46,91

58,97

20,38

57,67

63,22

1m10m100m1km10km100km1,000km

GP-Agg Combined Textual

GP-Agg Combined Non-textual

GP-Agg Multimodal

Textual RLDL

Results – Median Distance

With Metadata Runs

Run4

Run3

Run1

50 150 250 350 450 550 650 750

309.86

394.89

196.01

Conclusions

● GP-Agg automatically combines lists and aggregation functions.

● Top-N RLDA improves even GP-Agg results.● RLDA was better (see Run 4) than using GP-

Agg alone.● There is room for improving the GP-Agg

approach.

Future Work

● Develop other fitness fuctions in the GP-Agg approach.

● Use more visual descriptors.● Evaluate different clustering strategies.

Acknowledgments

● MediaEval 2015● FAPESP● CNPq ● CAPES● Samsung

Thank You!

{lintzyli,pedro.mendes,luis.pereira,rtorres}@ic.unicamp.br,jurandy.almeida@unifesp.br,rtcalumby@ecomp.uefs.br,

o.penatti@samsung.com,jalvarm.acm@gmail.com,icaro.dourado@students.ic.unicamp.br,daniel@rc.unesp.br,

{keiller.nogueira,jefersson, mgoncalv}@dcc.ufmg.br

L. T. Li, J. A. V. Muñoz, J. Almeida,R. T. Calumby, O. A. B. Penatti, I. C. Dourado,

K. Nogueira, P. R. Mendes Júnior,D. C. G. Pedronette, J. A. dos Santos,

M. A. Gonçalves, and R. S. Torres

Support Slides

Run1 Run3 Run4 Min. : 0.000 Min. : 0.000 Min. : 0.000 1st Qu.: 1.897 1st Qu.: 2.124 1st Qu.: 1.482 Median : 309.865 Median : 394.889 Median : 196.008 Mean : 2913.598 Mean : 2976.632 Mean : 2483.614 3rd Qu.: 5573.930 3rd Qu.: 5766.894 3rd Qu.: 3798.622 Max. :19959.808 Max. :19959.808 Max. :19954.130

Basic analysis of distances (km) in Test set:from Predicted to Expected lat/long

Run2 Min. : 0 1st Qu.: 1240 Median : 5883 Mean : 5597 3rd Qu.: 8637 Max. :19972

Videos-only Test Results (%)

Run 1: GP-Agg TextualRun 2: GP-Agg only visual (HMP+ all Lire) + RLDA (top100) Run 3: GP-Agg multimodal (text, visual, audio)Run 4: BM25_RLDA (top 5)

Run 1 Run 2 Run 3 Run 4

1m 0.08 0 0.08 0.06

10m 0.4 0 0.37 0.41

100m 5.46 0.01 5.13 5.79

1km 17.62 0.02 16.74 17.89

10km 32.44 0.11 32.1 32.24

100km 40.27 3.8 39.69 39.92

1,000km 54.13 20.39 53.67 55.68

10,000km 90.57 91.97 91.5 93.16

Video-only Summary

Run1 Run2 Run3 Min. : 0.000 Min. : 0.05 Min. : 0.000 1st Qu.: 2.914 1st Qu.: 1304.13 1st Qu.: 3.138 Median : 619.747 Median : 6351.71 Median : 660.864 Mean : 3191.163 Mean : 5709.17 Mean : 3158.724 3rd Qu.: 6035.939 3rd Qu.: 8463.49 3rd Qu.: 6154.147 Max. :19656.624 Max. :19596.38 Max. :19524.628

Run4 Min. : 0.000 1st Qu.: 2.992 Median : 547.648 Mean : 2879.899 3rd Qu.: 5548.780 Max. :19656.624

GP-Agg Individual example – Run 1

Input: bm25 (description, fusion, tags, title), tf-idf(description, fusion, tags, title), lmd_fusion, ibs_fusion

Individual:

CombSUM( MRA( CombMNZ( RRF(ibs_fusion,lmd_fusion),CombMNZ(td-idf_fusion, bm25_fusion), RLSim(lmd_fusion, bm25_tags, tf-idf_fusion)), CombMNZ(CombSUM(tf-idf_tags, tf-idf_fusion, tf-idf_tags), CombMIN(tf-idf_description, bm25_description, tf-idf_tags), RLSim(bm25_title, tf-idf_title, tf-idf_title)), RRF(CombMAX(bm25_fusion, bm25_tags), RRF(tf-idf_fusion, ibs_fusion, tf-idf_fusion), BordaCount(tf-idf_fusion, bm25_fusion, tf-idf_tags))), RLSim(CombSUM(CombSUM(tf-idf_tags, tf-idf_tags, lmd_fusion), bm25_fusion), bm25_fusion))

Visual Features (HMP): Extracting

● Histograms of Motion Patterns● Keyframes: Not used● Applying an algorithm to compare video

sequence(1)partial decoding; (2)feature extraction; (3)signature generation.

“Comparison of video sequences with histograms of motion patterns”, J. Almeida et al. ICIP, 2011.

Visual Features (HMP): overview

[Almeida et al., Comparison of video sequences with histograms of motion patterns. ICIP 2011]

HMP: Comparing Video

OPF Density formula

● d(s,t) distânce from s to t (used haversine dist.)● A(s): list of adjacency of s. ● directed graph