04 sem cert hparticipation_final

1Information Technologies InstituteCentre for Research and Technology Hellas

CERTH at MediaEval 2014 Synchronization of Multi-User

Event Media Task

Konstantinos Apostolidis, Christina Papagiannopoulou,

Vasileios Mezaris

Information Technologies Institute / Centre for Research and Technology Hellas


• Media from different users related to an event are exchanged/shared in a number of ways.

– time information attached to some of the captured media may be wrong

– geolocation information may be missing

Objective: The alignment and presentation of media of different users in a consistent way, preserving the temporal evolution of the event.

Introduction


Proposed method


Time Synchronization (1/4)

• Near Duplicate image detection– SIFT descriptors extraction.– Clustering to form a vocabulary.– Encoding of each image using VLAD encoding.– Keypoints geometry consistency checking using Geometric Coding to

refine the returned near duplicate images.• Modified Near Duplicate image detection

– Use color information (HSV histograms). Near duplicate candidates that are very similar in color are considered even if the Geometric Coding score is relatively low.



• Apply the modified NDD on the union of all galleries to come up with pairs of near duplicate images belonging to different images.

• Filter results, rejecting pairs of which:– the location distance of the two photos is greater than a distance threshold.– the time difference between the photos is above a time threshold.

• The remaining of near duplicate photos belonging to different galleries are considered as links between those galleries.



• Graph Construction– Galleries → Nodes– Links between galleries → Edges – Number of links between galleries → Edge weights


Time Synchronization (4/4)• Graph traversal:

1. Start from the node corresponding to the reference gallery.2. Select the edge with the highest weight.3. Compute the time offset of the node on the other end.

Time offset is equal to the median of time differences of near duplicate image pairs from the corresponding galleries.

4. Repeat steps 2 and 3 considering as possible starting point any visited node.

• Once the galleries time offsets are calculated, align the image timestamps.


Media Clustering to Events

• Cluster events using the corrected timestamps.

• Two approaches are followed:1. Consider all photo galleries as a single photo collection

and perform clustering in this collection.2. Make pre-clustering within each gallery separately and

then merge the clusters across the galleries.


1st Clustering Approach (1/3)• Sort all the photos according to the corrected timestamps• Construct the similarity matrix using the similarity measure:

where ti, tj the timestamps of the images i,j and K is a parameter for the sensitivity of the measure.

• The larger the value of K the coarser the clusters created.

K = 30 min K = 12 h


• Compute the novelty scores for each photo i, vK, to detect the cluster boundaries– Correlate a checkerboard kernel g along the main diagonal of each SK. Kernel g

– , where l = 1

• Detect the peaks that have difference greater than a threshold.

K = 30 min K = 12 h

Repeat the above procedure for K = 15, 30, 60, 120, 720, 1440 min

1st Clustering Approach (2/3)


1st Clustering Approach (3/3)• Select the boundary list, BK = {b1, … , bnK}, which forms the larger within-

cluster similarity and the lower between-cluster similarity, C(BK), using :

• The time alignment may not be perfect.• Split each of the clusters using the geolocation information.

– Photos with close geolocation are assigned to the same cluster– Photos which have great geolocation difference are assigned to

different clusters• The photos that do not have geolocation information are assigned to the

geo-cluster which is more similar according to the color information (e.g. HSV histogram).


2nd Clustering Approach (1/2)• In each gallery, find the minimum time difference of dissimilar

photos which is greater than the maximum time difference of the near-duplicate photos (based on the similarity matrix of GC).


2nd Clustering Approach (2/2)• The clusters that are formed by these time gaps, are merged

according to time and geolocation similarity.

• For the clusters that do not have geolocation information, the merging is continued by considering:– the time and low-level feature similarity – the time and the concept detector confidence similarity scores


Submitted runs• Run #1: Compute gallery time offsets using our modified NDD. Concept

Detection scores are used to merge clusters using the 2nd approach in clustering.

• Run #2: Compute gallery time offsets using our modified NDD. HSV histogram similarity is used to merge clusters using 2nd clustering approach.

• Run #3: Compute gallery time offsets using our modified NDD. Clustering is performed using the 1st clustering approach.

• Run #4: Compute gallery time offsets using our modified NDD and traverse of the graph by considering the weights of multiple edges. Concept Detection scores are used to merge clusters using the 2nd clustering approach.

• Run #5: Compute gallery time offsets using NDD without HSV color information. Concept Detection scores are used to merge certain events using the 2nd clustering approach.


Evaluation (1/2)

• Time synchronization:– Modified NDD approach gives the best results in time

alignment for the Vancouver testset.– Standard NDD yields a slightly better time synchronization

than modified NDD, for the London testset.

Run #1 Run #2 Run #3 Run #4 Run #5

Vancouvertestset

Time Sync. Precision 0.9118 0.9118 0.9118 0.5294 0.9118

Time Sync. Accuracy 0.7375 0.7375 0.7375 0.7014 0.7279


Londontestset

Time Sync. Precision 0.6111 0.6111 0.6111 0.2222 0.6389

Time Sync. Accuracy 0.7127 0.7127 0.7127 0.6996 0.7299


Evaluation (2/2)• Sub-event Clustering:

– Exploitation of consistent timestamps in a gallery and the use of Concept Detection scores gives a good performance for the Vancouver testset.

– HSV histogram similarity clearly gives the best clustering results for the London testset.


Vancouvertestset

Clustering Rand Index 0.9770 0.9734 0.9526 0.9601 0.9656

Clustering Jaccard Index 0.2581 0.2315 0.2856 0.1782 0.2861

Clustering F-Measure 0.2052 0.1880 0.2222 0.1512 0.2225


Londontestset

Clustering Rand Index 0.9885 0.9910 0.9838 0.9829 0.9863

Clustering Jaccard Index 0.5051 0.5614 0.3232 0.2739 0.4849

Clustering F-Measure 0.3356 0.3596 0.2443 0.2150 0.3266


Conclusions• Using color information in modified NDD, we can discover

more pairs of images between different galleries.• However, due to multiple events being carried out at the

same location with characteristic color background in London testset, modified NDD returned some wrong pairs, yielding to worse results than the standard method.

• In sub-event clustering, the use of HSV histograms similarity seems to give a better performance than the Concept Detector confidence scores, in a dataset with increased color diversity, such as London testset.


Questions?

More information:http://www.iti.gr/~bmezaris [email protected]

04 sem cert hparticipation_final

Software