TR

EC

-5

TR

EC

-6

TR

EC

-7

TR

EC

-8

TR

EC

-5

TR

EC

-6

TR

EC

-7

TR

EC

-8

0

20

40

60

80

100

Per

cent

of

Tot

al

Percentage of Pool Percentage of Relevant

Others

Mixture

Manual

Automatic

. . . . . . . . . . . . . . . . . . . . . . . . . . . . .0

100

200

300

400

Tot

al u

niqu

e ov

er 5

0 to

pics

OthersMixtureManualAutomatic

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .0

100

200

300

400

Tot

al u

niqu

e ov

er 5

0 to

pics

OthersMixtureManualAutomatic

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .0

100

200

300

400

Tot

al u

niqu

e ov

er 5

0 to

pics

OthersMixtureManualAutomatic

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .0

100

200

300

400

Tot

al u

niqu

e ov

er 5

0 to

pics

mixturemanualauto

umd9

9a1

ok8a

lxSa

b8A

1uw

mt8

a1ac

sys8

alo

ic99

dafb

ibm

s99b

ibm

s99a

ok8a

mxc

Sab8

A2

Flab

8as

mds

08a4

mds

08a3

INQ

604

INQ

603

tno8

d3tn

o8d4 1

Uni

NE

T8S

tuw

mt8

a2pi

r9A

atd

isa2

5U

T81

0D

m8T

Fbn

Flab

8atd

nM

ITSL

Stdn

MIT

SLSt

dfu

b99t

fM

er8A

dtd1

Mer

8Adt

d2at

t99a

tde

att9

9atd

cun

c8al

32pi

r9A

ttdnt

td8a

lxri

c8dp

xis

a50

plt8

ah2

fub9

9aU

niN

ET

8Lg

unc8

al42

acsy

s8am

nnt

td8a

lesu

rfah

i2G

E8A

TD

N1

iit99

au1

apl8

c221

Scai

8Adh

ocap

l8n

surf

ahi1

ibm

g99b

UT

813

ric8

tpx

Ant

Hoc

1pl

t8ah

1w

eave

r2kd

d8ps

16ib

mg9

9akd

d8qe

01ku

adho

cw

eave

r1D

m8N

bn

Run

0

20

40

60

80N

umbe

r un

ique

by

grou

p

0.000

0.001

0.002

Absolute difference in M

ean Average P

recision scores

0.0 0.2 0.4 0.6 0.8 1.0

Recall

0.0

0.2

0.4

0.6

0.8

1.0

Prec

isio

n

pir9Attdok8amxc att99atde fub99td ibms99a MITSLStd Flab8atd2 tno8d3

BEST AUTOMATIC SHORT ADHOC

0.0 0.2 0.4 0.6 0.8 1.0

Recall

0.0

0.2

0.4

0.6

0.8

1.0

Pre

cisi

on

READWARE2 orcl99man iit99ma1 CL99XTopt 8manexT3D1N0

BEST MANUAL ADHOC

’92 System ’93 System ’94 System ’95 System ’96 System ’97 System ’98 System ’99 System

Cornell/Sabir SMART Systems

.1

.2

.3

.4

Mea

n A

vera

ge P

reci

sion

TREC-1 taskTREC-2 taskTREC-3 taskTREC-4 taskTREC-5 task (short)TREC-6 task (description)TREC-7 task (description)TREC-8 task (title & description)

51 55 60 65 70 75 80 85 90 95 100

Topic

0.0

0.2

0.4

0.6

0.8

1.0

Aver

age P

recis

ion