alan f. smeaton dublin city university & paul over nist the trec-2002 video track: overview

Alan F. SmeatonDublin City University

&Paul Over

NIST

The TREC-2002 Video Track: Overview

2

1. Introduction and Context

– Last year’s talk…– gave an intro to video coding & compression;

– highlighted predominant access mechanism as manual tagging via metadata

– noted emerging automatic approaches are based on shot boundary detection, feature extraction and keyframe identification, followed by feature searching with keyframe browsing

– noted there is no test collection of video

– provided an overview of what 12 groups did on 11 hours of video in shot boundary detection and searching tasks

– Last year was TV101, this year is TV201

3

New this year (1)

– More participants and data: – 17 participating teams (up from 12), – 73 hours (up from 11)

– Shot boundary determination (SBD)– new measures– 3-week test window

– New semantic feature extraction task– features defined jointly by the participants– task is to identify shots with those features

– Several groups donated extracted features– identified features from test videos early– shared their output (in MPEG-7 defined by IBM) in time for

others to use as part of their search systems

4

New this year (2)

– 25 topics for the search task, – developed by NIST – 4 weeks between release and submission– text, video, image and/or audio

– Average precision added as measure – new emphasis on ranking

– A common set of shot definitions– donated by CLIPS-IMAG, formatted by DCU– common units of retrieval for feature and search tasks– allowed pooling for assessment

5

New this year (3)

– Searching was:– Interactive: full human access and iterations, or – Manual: a human with no knowledge of the test data gets

one shot to formulate the topic as a search query– No fully automatic topic-to-query translation

– Elapsed search time was added as a measure of effort for interactive search, groups gathered data on searcher characteristics

Shots Features

SBD Feature Extr. Searching

6

Carnegie Mellon U. (US)

CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)

Dublin City University (Irl)

Fudan Univ. (China) X

IBM Research (US) X

Imperial College London (UK) X

Indiana University (US)

Institut Eurecom (Fr)

Mediamill/U Amsterdam (NL)

Microsoft Research Asia (China) X

National Univ. Singapore (Sing.) X

Prous Science (Esp)

RMIT University (Aus) X

Univ. Bremen (D) X

U. Maryland/INSA/U. Oulu (US)

Univ. Oulu/VTT (Fin)

Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.

X X X X X X X X X X X

X X X X X

X

X X X X


X X X X X X X X X X X X

X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

The 17 groups and the tasks they completed

7

Video Data

– Difficult to get video data for use in TREC because ©

– Used mainly Internet Archive– advertising, educational, industrial, amateur films 1930-

1970 – produced by corporations, non-profit organisations,

trade groups, etc.– Noisy, strange color, but real archive data– 73.3 hours partitioned as follows:

4.85

5.07

23.26

40.12Search test

Feature development(training and validation)

Feature test

Shot boundary test

8

2. Shot Boundary Detection task

– Not a new problem, but a challenge because of gradual transitions and false positives caused by photo flashes, rapid camera or object movement

– 4 hours, 51 minutes of documentary and educational material

– Manually created ground truth of 2,090 transitions (thanks Jonathan) with 70% hard cuts, 25% dissolves, rest are fades to black and back, etc.

– Up to 10 submissions per group, measured using precision and recall, with a bit of flexibility for matching gradual transitions

9

2001: Recall and precision for cuts

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cis

ion

CLI PSFudanI BMI CMKMMSRADCUJ HUAPLMB_ FrequencyMediaMillUMDLAMP

10

2002: Recall and precision for cuts

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cis

ion

CLI PS (10)Fudan (10)I BM (5)I CMKM (6)MSRA (10)TZI -Ubremen (1)NUS (2)RMI T (10)The data must have

gotten a little harder?

11

2001: Gradual Transitions

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cis

ion


12

2002: Gradual Transitions

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cis

ion

CLI PS (10)Fudan (10)I BM (5)I CMKM (6)MSRA (10)TZI -Ubremen (1)NUS (2)RMI T (10)

Still room for improvement.Precision/recall knobs working for some systems

13

2001: Frame-recall & -precision for GTs

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cis

ion


14

2002: Frame-recall & -precision for GTs

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cis

ion

CLI PSFudanI BMI CMKMMSRATZI -UbremenNUSRMI T

So, who did what ?

The approaches….

16


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

CLIPS-IMAG (Fr):

Refined 2001 SBD system which is based on frame comparisons, filters photo “flashes”, several runs with parameters varied

Shot Boundary Detection:

17


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

Fudan University (China):

An update on their 2001 SBD based on frame-frame comparisons, also filters photo flashes and has fade-in/-out detection


18


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


IBM Research (US):

Presentation to follow

19


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


Imperial College London (UK):

Frame comparison based on colour histograms, extended for gradual transitions, and also addressing photo flashes

20


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


Microsoft Research Asia (China):

Based on 2001 SBD but refined to address gradual transitions … also based on frame differences

21


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


National Univ. of Singapore:

Used wavelet coefficients to detect potential transitions and filtered for flashbulbs, object and camera motion, with an adaptive threshold for different video types

22


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


RMIT University (Aus.):

Used each frame as a query to a window of frames, and based on rank positions of other frames in this window, did SBD

23


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


University of Bremen (D):

Another based on comparing frames using histograms, with adaptive thresholding

24

3. Feature Extraction

– FE is – interesting itself but when it serves to help video

navigation and search then its importance increases

– Objective was to – begin work on benchmarking FE – allow exchange of feature detection output among

participants

– Task is as follows: – given small standard dataset (5.02 hours, 1,848 shots)

with common shot bounds, – locate up to 1,000 shots for each of 10 binary features– Feature frequency varied from “rare” to “everywhere”

25

The Features

1. Outdoors2. Indoors3. Face - 1+ human face with nose, mouth, 2 eyes4. People - 2+ humans, each at least partially visible5. Cityscape - city/urban/suburban setting6. Landscape - natural inland setting with no human

development such as ploughing or crops7. Text Overlay - large enough to be read8. Speech - human voice uttering words9. Instrumental Sound - 1+ musical instruments10. Monologue - 1 person, partially visible, speaking for a

long time without interruption

26

05

1015202530

True shots contributed uniquely by each run

– Small values imply lots of overlap between runs– Likely due to relative size of result set (1,000 shots) and total

test set (1,848 shots)

27

AvgP by feature (runs at median or above)

0

0.2

0.4

0.6

0.8

Feature

Avera

ge p

recis

ion

CMU_ r1

A_ CMU_ r2

CLIPS-LIT_ GEOD

CLIPS-LIT-LIMSU

DCUFE2002

Eurecom1

Fudan_ FE_ Sys1

Fudan_ FE_ Sys2

IBM-1

IBM-2

MediaMill1

MediaMill2

MSRA

UnivO_ MT1

UnivO_ MT2

Avg Prec Cap

Outd

oors

Indo

ors

Face

Peop

le

City

scap

e Land

sca

pe Text

ov

erla

ySp

eec

h

Inst

rum

enta

l

so

und

Mon

olo

gRandom baseline

28

Groups and Features


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

Outd

oors

In

doors

F

ace

People

Cit

ysca

pe

L

andsc

ape

T

ext

ove

rlay

Speech

M

usi

c

M

onolo

gue

29


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

Groups and Features

Carnegie Mellon University (US):

Hand-label feature training data, extract low-level image or audio features and combine in a SVM training process

Outd

oors

In

doors

F

ace

People

Cit

ysca

pe

La

ndsc

ape

T

ext

Ove

rlay

Speech

M

usi

c

M

onolo

gue

30


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

Groups and Features

CLIPS-IMAG (Fr):

4 features, face/people using CMU’s publicly available tool, speech/monologue using LIMSI’s ASR transcript

Outd

oors

In

doors

F

ace

People

Cit

ysca

pe

La

ndsc

ape

T

ext

Ove

rlay

Speech

M

usi

c

M

onolo

gue

31


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

Groups and Features

Dublin City University (Irl):

Speech/music detection based on energy peaks in encoded bitstream, face detection based on skin masks

Outd

oors

In

doors

F

ace

People

Cit

ysca

pe

La

ndsc

ape

T

ext

Ove

rlay

Speech

M

usi

c

M

onolo

gue

32


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

Groups and Features


Colour histograms and edge direction histograms, trained on sample frames for I/O/C/L; face/people based on skin colour, motion and shape, text block detection and audio detection based on audio features

Outd

oors

In

doors

F

ace

People

Cit

ysca

pe

La

ndsc

ape

T

ext

Ove

rlay

Speech

M

usi

c

M

onolo

gue

33


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

Groups and Features

IBM Research (US):

Presentation to followO

utd

oors

In

doors

F

ace

People

Cit

ysca

pe

La

ndsc

ape

T

ext

Ove

rlay

Speech

M

usi

c

M

onolo

gue

34


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

Groups and Features

Institut Eurecom (Fr):

Visual feature detection based on classifying 16x16 pixel macroblocks, directly from MPEG encoding

Outd

oors

In

doors

F

ace

People

Cit

ysca

pe

La

ndsc

ape

T

ext

Ove

rlay

Speech

M

usi

c

M

onolo

gue

35


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

Groups and Features

Mediamill/TNO (NL):

Feature detection on all features using an active learning technique

Outd

oors

In

doors

F

ace

People

Cit

ysca

pe

La

ndsc

ape

T

ext

Ove

rlay

Speech

M

usi

c

M

onolo

gue

36


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

Groups and Features


Operated on multiple keyframes per shot, visual feature identification used colour moments and edge direction histograms; audio feature detection based on SVM with inputs from low-level audio analysis

Outd

oors

In

doors

F

ace

People

Cit

ysca

pe

La

ndsc

ape

T

ext

Ove

rlay

Speech

M

usi

c

M

onolo

gue

37


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

Groups and Features

University of Bremen (D):

Indoor/outdoor classifier based on colour distributions, input into a neural network

Outd

oors

In

doors

F

ace

People

Cit

ysca

pe

La

ndsc

ape

T

ext

Ove

rlay

Speech

M

usi

c

M

onolo

gue

38


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

Groups and Features

Univ. Md./INSA/Univ. Oulu (US):

Used text overlay detector from INSA de Lyon

Outd

oors

In

doors

F

ace

People

Cit

ysca

pe

La

ndsc

ape

T

ext

Ove

rlay

Speech

M

usi

c

M

onolo

gue

39


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

Groups and Features

Univ. Oulu/VTT (Fin):

Edge detection gradients for city/landscape, skin detector for faces, speech/music from low-level audio analysis

Outd

oors

In

doors

F

ace

People

Cit

ysca

pe

La

ndsc

ape

T

ext

Ove

rlay

Speech

M

usi

c

M

onolo

gue

40

4. The Search Task

– Task is similar to text analogue … – topics are formatted descriptions of an information need – task is to return up to 100 shots that meet the need

– Test data: 40.12 hours (14,524 common shots)– Features and/or ASR donated by CLIPS, DCU, IBM,

Mediamill and MSRA– NIST assessors

– judged top 50 shots from each submitted result set– subsequent full judgements showed only minor variations

in performance

– Used trec_eval to calculate measures

41

Search Topics

– Topics (25) multimedia, created by NIST– 22 had video examples (avg 2.7 each), 8 had

image (avg 1.9 each)– Requested shots with specific/generic:

– People: George Washington; football players– Things: Golden Gate Bridge; sailboats– Locations: ---; overhead views of cities– Activities : ---; rocket taking off– Combinations of the above:

• People spending leisure time at the beach• Locomotive approaching the viewer• Microscopic views of living cells

42

Search Types: Interactive and Manual

43

Manual runs: Top 10 (of 27)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cisi

on

Prous Science

IBM-2

CMU_ MANUAL1

IBM-3

LL10_ T

CLIPS+ASR

Fudan_ Search_ Sys4

CLIPS+ASR+X

ICMKM-2

UMDMqtrec

44

Interactive runs top 10 (of 13)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cisi

on

CMUInfInt1

DCUTrec11B.1

DCUTrec11C.2

CMU_ INTERACTIVE_ 2

UnivO_ MT5

IBM-4

DCUTrec11B.3

DCUTrec11C.4

UMDIqtrec

MSRA.Q-Video.2a

Prous Science

IBM-2

CMU_MANUAL1

45

Mean AvgP vs mean elapsed time

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0

5

10

15

20

25

30

IuVf1CMUInfIn

t1CMU_IN

TERACTIVE

DCUTrec11

B.

1DCUTrec

11B.

3DCUTrec

11C.

2DCUTrec

11C.4

IBM-4

ICMKM-1

MSRA.Q-

Video.1

MSRA.Q-V

ideo.2.

A

UMDIqtrec

UNivO_M

T5

Mean average precision

Mean elapsed time (mins.)

Wide variation in elapsed time.Not the dominant factor in effectiveness

46

Search: Unique relevant shots from each run

010203040506070

Interactive runs contributed most as expected.

47

Distribution of relevant shots

Top vs bottom of halves of result sets

1

51

101

151

201

251

301

351

Topic

Re

lev

an

t sh

ots

Bottomhalf

Tophalf

Not many additional relevant found in bottom half of result sets except for topics with a lot already.

48

Max/median AvgP by topic - interactive

0

0.2

0.4

0.6

0.8

1

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

Topic

Maximum

Median

0

50

100

150

200

250

300

350

Relevantshots

0

10

20

30

40

50

60

70

Relevantvideos

49

0

0.2

0.4

0.6

0.8

1

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

Topic

Maximum

Median

Why better median performance on topics 76, 84, 90, 97 ?

No single, simple explanation… e.g., topics with more relevant shots or videos, or those with example(s) from the search test collection aren’t necessarily “easier”

0

50

100

150

200

250

300

350

Relevantshots

0

10

20

30

40

50

60

70

Relevantvideos

Max/median AvgP by topic - manual

50

Relevant shots by file id (topics 75-87)

16

111621263136414651566166717681869196

101106111116121126131136141146151156161166171

1 6

11

16

21

26

31

36

41

46

51

56

61

66

71

76

81

86

91

96

10

1

10

6

11

1

11

6

12

1

12

6

13

1

13

6

14

1

14

6

15

1

15

6

16

1

16

6

17

1

Shot number in file

Vid

eo

file

id

75767778798081828384858687

Relevant shots are adjacent in many but not all cases … Local browsing among video shots can be important

51

Relevant shots by file id (topics 88-99)

16

111621263136414651566166717681869196

101106111116121126131136141146151156161166171176

1 6

11

16

21

26

31

36

41

46

51

56

61

66

71

76

81

86

91

96

10

1

10

6

11

1

11

6

12

1

12

6

13

1

13

6

14

1

14

6

15

1

15

6

16

1

16

6

17

1

Shot number in file

Vid

eo

file

id

888990919293949596979899

52

The Groups and Searching


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

53


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


Carnegie Mellon University (US):

Modified Informedia to include TREC features and multiple image search engines with an experienced user for interactive search; manual search combined ASR, OCR and image matching with pseudo relevance feedback and query expansion

54


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


CLIPS-IMAG (Fr):

3 manual runs using (a) donated features, (b) LIMSI transcript (c) both

55


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


CWI Amsterdam (NL):


56


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


Dublin City University (Irl):

Incorporated a donated ASR transcript, 7 donated features + 3 of their own features into the Fischlar video retrieval system, and ran interactive search with 12 users.

57


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound



3 manual runs combining similarities from face recognition (for face topics), text recognition, image similarity and ASR transcript; used their own + donated features, LIMSI transcript, and some topics only.

58


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


IBM Research (US):


59


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


Imperial College (UK):

Query matched against 1 keyframe per shot using colour features and thumbnail placement relevance feedback for interactive search

60


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


Indiana University (US):

Interactive search using ViewFinder video IR system

61


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound



used M/S Q-Video system including the 10 semantic features and 9 other low-level visual features, plus ASR; automatic shot matching and ranking with user varying the weights for different features.

62


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


Proust Research (Spain):

Presentation to follow - one manual run which realised best or near-best performance for 18 of 25 topics !

63


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


U.Md./INSA/U. Oulu (US):

Used donated ASR and OCR text, 8 donated features and colour correlogram matching of image vs. keyframe for manual runs with multi- dimensional browsing tool for interactive runs

64


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X







Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound


Univ. Oulu/VTT (Fin):

Also used donated ASR and OCR text, 8 donated features and their VIRE system with multi-modal indexing based on SOMs, and 8 users using an interactive video navigation tool

65

Groups doing the “Full Monty”


CLIPS-IMAG (Fr) X

CWI Amsterdam (NL)



IBM Research (US) X





Microsoft Research Asia (China)X


Prous Science (Esp)


Univ. Bremen (D) X



Feature Search

1 2 3 4 5 6 7 8 9 10 Int. Man.


X X X X X

X

X X X X



X X

X

X X X X X X X

X X X X X X X X X X


X

X X

X X X

X X X X X X X

Shot Bound

66

5. Conclusions

– This track has grown significantly… data, groups, tasks, measures, complexity

– Donated features enabled many sites to take part and greatly enriched the progress .. this cannot be overstated … very collegiate and beneficial all-round

– Common shot definition – implications for measurement need closer look – seems it was successful

– The search task is becoming increasingly interactive, and we could do with guidance here

– Evaluation framework has settled down – should be repeated on new data with only minor adjustments

– Need more data (especially for feature extraction), more topics – looking at 120 hours of news video from 1998

– Need to encourage progress on manual/automatic processing – how? focus evaluation more?

– Probably ready to become one-day pre-TREC workshop with report-out/poster at TREC

alan f. smeaton dublin city university & paul over nist the trec-2002 video track: overview

Documents

bremen d x

hours of video

test data

search tasks

overview slide

video data difficult

shot boundary detection

search query