robust video fingerprinting system daniel [email protected] luis...

Robust video fingerprinting systemRobust video fingerprinting system

Daniel Pereira [email protected] Loyola [email protected]

Confidential 22

Robust video fingerprinting system

• Summary

• Purpose of the system

• What is video fingerprinting

• Practical problems to solve

• Proposed solution

• Results analysis

Confidential 33


• Purpose of the system

• SkillUpJapan distributes digital contents

• FujiTV, TV Tokyo, SkyPerfecTV, Warner Brothers Japan, …

• Our platform, Uliza, is an extensible digital content

management system

• Piracy and DRM are of importance to digital contents

rights holders

Confidential 44


• Video fingerprinting

• A way to effectively tie a video, or a segment of it, to a

unique hash value

• Information needs to be stored and searched efficiently

• Avoid to store original contents provided by clients

• Contents should not be recreated from said fingerprint

Confidential 55


• Key technical aspects about video

• Measured characteristics

• Luma and chroma (brightness and color components)

• Edge detection, gradient orientation

• Time variance

• A movie is, after all, a sequence of images that change over time at

a defined rate

• Amount of data per frame

Confidential 66


• Efficiency metrics

• Uniqueness

• Accurately find videos we search; not return videos that

are not what we search

• Database

• Efficiently index the results in a database

• Solution must be fast

• Find the clip among many other videos in fastest time

Confidential 77


• Some practical problems to solve

• Current solutions have relatively accurate algorithms,

however the process is computationally expensive

• Partitioning of frames, complex algorithm

• Database storage is not taken seriously

• It is unaffordable to store information for every frame, or large

arrays of information for each fingerprint

• Slow search times when database grows

Confidential 88


• Proposed solution

• Lowers the needed resources and process time, while improving upon

results (Luma and time based indexes)

• Addresses algorithmic complexity by using simple methods (Euclidean

distance of vectors and Tanimoto correlation)

• Stores information in an efficient way, allowing for quick retrievals,

with use of Look-up Tables

Confidential 99


• Proposed solution (video properties):• Average the Luma value of each frame

• Luma values show prolonged, relatively constant, values that can be indexed to an interval of time

Lum

a

Time

ThresholdLum

a

Time

Confidential 1010


• Proposed solution (video properties):

• Average Luma calculated according to

Lum

a

Duration [start-time, end-

time[ (secs)

Average Luma calculated in T

seconds

[2, 8[ 102

[8,13[ 107

[13,14[ 198

[14,19[ 85

[590,600] 48

……

Time

€

ST =1

w × h × NL[x,y, f ]

y=0

h

∑x=0

w

∑f =0

N −1

∑

Confidential 1111


• Proposed solution (database):

• Using those indexes to store only segments we can save lots of space

• Each segment of several seconds has a value of 2 bytes

• Luma values range from 0 to 255

• Look-up table for segments

2.0

Luma

120

10.3 140

16.0

19.4

85

110

21.4 120

Time

Luma

time2.0 10.3 16.019.4

120

85

140

21.4

Confidential 1212


• Proposed solution (database):

4

Luma

10

15 160

19

32

125

13

… …

Time

2

Luma

120

5 140

9

12

85

110

… …

Time

2

Luma

120

5 140

9

12

85

110

… …

TimeComparisons

Fingerprints on database Fingerprint A

…

Fingerprint A

Fingerprint 1

Fingerprint 3

Time

Confidential 1313


• Proposed solution (algorithm):

• Tanimoto

• Tanimoto makes a correlation between C and the remaining elements outside C

• Euclidean vector distance

A BC

€

D = (Ai − Bi )2

i=0

n

∑

€

T(A,B) =A⋅ B

A2

+ B2

− A⋅ B

Confidential 1414


• Proposed solution (algorithm):

• Hierarchical approach

1. Look-up Table of segments

2. Compares the time indexes

3. & 4. Tanimoto Correlation and Vector Distance of Luma

• Look-up Tables discard perceptually different movies efficiently

• Comparison of time indexes also behaves efficiently

• The number of movies that are ultimately analyzed with Tanimoto

Correlation and Euclidean Vector Distance is very low

Confidential 1515


• Evaluation of algorithm:

• 220 movies were analyzed with each other

• Quality varies from FullHD to SD

• Duration ranges from 15 second commercials to full length movies

• Frame-rate of movies varies from 15fps to 30fps

• Comparison against C.G.O. (Centroids of Gradient Orientation) [1]

• Tests were conducted by searching scenes of 10 seconds

• Evaluation compares algorithm, database size and robustness of solutions

[1] Sunil Lee and Chang D. Yoo, “Robust Video Fingerprinting for Content-Based Video Identification”,

IEEE Trans. Circuits and Systems for Video Technology, vol. 18, no. 7, pp983-988, July, 2008

Confidential 1616


• Obtained results (database size):

SUJ CGO

DB Threshold Size (Kbytes) Size (Kbytes)

1 4,944

107,900

2 3,390

3 2,611

5 1,783

10 1,020

15 678

20 512

25 412

30 348

Confidential 1717


• Obtained results (run-time):

SUJ CGO

DB Threshold Time (Seconds) Time (Seconds)

1 74.16

92.41

2 31.56

3 21.13

5 10.77

10 8.84

15 8.66

20 10.1

25 12.42

30 15.11

Confidential 1818


• Obtained results (robustness):

SUJ CGO

DB threshold

False Positives

False Negatives

False Positives

False Negatives

1 8 0

211 1

2 12 0

3 9 0

5 11 0

10 11 3

15 8 8

20 9 6

25 11 14

30 9 18

Confidential 2020


• Summary

• State of the art solutions need to better address

practical issues

• The proposed algorithm can improve upon state of

the art algorithms on storage and speed of analysis

• Evaluation shows that the proposed solution also

provides higher robustness

Confidential 2121


Questions?


Confidential 22

Robust video fingerprinting systemRobust video fingerprinting system


robust video fingerprinting system daniel [email protected] luis...

Documents

fastest time slide

system skillupjapan

results luma

frame slide

process time

fingerprint slide

characteristics luma

luma duration