EXYSTENCE Topical WorkshopTrust-Based Networks and Robustness in Organizations

ETH Zurich, March 15, 2006

Collective Evaluation: Practice and Theory

Yi-Cheng Zhang (Fribourg University, Switzerland)

Collective Evaluation: Practice and Theory

Main Themes●New generation of Web Sites● From— People—For—People

Wikipedia(user generated encyclopedia)de l.icio.us (social bookmarks)

Ohmynews.com society digg.com reddit.com technocrati.com

flickr.comblogdex.com

SWITZERLAND

Collective Evaluation: Practice and Theory

GoogleAmazon.comEpinions.comRatemyteachers.com

...professor.comEconomy ...doctor.com Ebay.com

ZAGAT Guides(restaurant) Dianping.com

Douban.com

.

SWITZERLAND

Collective Evaluation: Practice and Theory

Successful Case impacting all of us:Google, Amazon.com, Ebay.com,

Wikipedia.org, Slashdot.org, World66.com....

(new “animals ” are added fast by the day,explicitly&implicitly using “New” info-Theory)

Google: (Page Rank) by S. Brin & L. Page and by Kleinberg( 96~98)

Hubs vs Authorities

SWITZERLAND

Collective Evaluation: Practice and Theory

Old Model:newspapers, government, agency.

New Model:“Matchmaker site”

SWITZERLAND

Collective Evaluation: Practice and Theory

Objects can be:

books, products, news, stories, tourist places, ideas, favorite actress/actors, music, scientists, teachers(!)

SWITZERLAND

Collective Evaluation: Practice and Theory Advantages Disadvntages Old expertise corruption

easy control manipulation

New people are not noisy corrupted!!! contradictory distortion everyone is good at pseudo-experts something, sometimes laziness even better than experts Not reliable!!!

SWITZERLAND

Collective Evaluation: Practice and Theory

Science may help?Objective: to get the best approximation possible to assure quality information.

1) pure democracy one-man-one-vote2) authority; people NO influence

our challenge: find the best balance of the two- distribution of weights

SWITZERLAND

Collective Evaluation: Practice and Theory

Simple Model (XY):

SWITZERLAND

Q l l=1 ,2 , .... , M Intrinsic quality ( “God-Given)”

ql Approximate evaluation

X il i=1 ,2 , ... , Nl=1 ,2 , ... , M

“i”th expert's evaluationon object α

Y i “God -Given”judging capacity

Naive democratic method:

“Tyranny of Majority”!

More refined approach:

Collective Evaluation: Practice and Theory

SWITZERLAND

ql=1N∑i=1

N

xil

ql=∑i=1

N

w i x il w i : Weight of “i”th expert

What weight to assign?

Collective Evaluation: Practice and Theory

ql=∑i=1

N

w i x il

i2≈V i≡

1M∑

l=1

M

xil−ql2

SWITZERLAND

Collective Evaluation: Practice and Theory

∑i=1

N 1V i

x il−ql =0 l=1 ,2 , ...M ;

∑l=1

M x il−q l2

V i=M i=1 ,2 , ...N

SWITZERLAND

Rewriting as:

Collective Evaluation: Practice and Theory

SWITZERLAND

= 1M∑

l=1

M

ql−Q l 2

I p≡ 1[ pM ] ∑R=1

[ pM ]

nR/R

[ I p]0=[ pM ]1

2M p /2 M≫1

Collective Evaluation: Practice and Theory

“New” Information Theory for the Internet Age

Classical Information Theory (Claude Shannon 1948)

Original message

SWITZERLAND

0 0 0 1 0 1 0 0 0 1 1 0 0 1

noise transmission

1 0 0 1 1 1 0 0 0 1 1 0 0

Error

Need of redundancy(large number repeats)

Collective Evaluation: Practice and Theory

The goal:

SWITZERLAND

Use the least number of repeats to recover original message

Give a finite number of repeats, get the best approximation

Collective Evaluation: Practice and Theory

Challenge of Noise

Classical Information Theory: natural noise

“New” Information Theory:natural noise

intentional noise herding noise

SWITZERLAND

Collective Evaluation: Practice and Theory

When objects are too complex, need intermediate layers XY→XYZ

Amazon.com example

book 1 book 2.... ...... ....... ......book M

reviewson books

Readers

Reviews are easier than books to evaluateSWITZERLAND

Collective Evaluation: Practice and Theory

Collaborative Filtering & Predictions

Challenge: “Movielens Project”

(Michigan University 1997-2000)

3 Millions data 2000 movies rated by 17'000 viewers

SWITZERLAND

Collective Evaluation: Practice and Theory

example

Movies

123456

8

109

1 2 3 4 5 6 .. .. .. .. .. ... .. 20

√

√

√

√

√√

√

√

√

√√

√7

Viewers

√√

√

√

√

√

√

√

SWITZERLAND

Collective Evaluation: Practice and Theory

SWITZERLAND

v i Viewer i votes on movie α

I, j = 1,2,...Nα, β =1,2,...M

v i=1 ,2 ,3 , 4 ,5 Stars

Collective Evaluation: Practice and Theory

SWITZERLAND

ij=∑=1

M

vi v j

∑=1

M

∣vi∣∣v j ∣, ij∈−1 ,1

Person-to-person relationship:

d ij=1−ij/2Distance

Collective Evaluation: Practice and Theory

=∑i=1

N

vi vi

∑i=1

N

∣vi∣∣vi ∣, ∈−1 ,1

d =1−/2

Movie-to-movie relationship:

Distance

SWITZERLAND

Collective Evaluation: Practice and Theory

SWITZERLAND

2 23...

'=

1−

More Refined Approach

Collective Evaluation: Practice and Theory

SWITZERLAND

=∑ij∑S i

∣V test∣

≃0.83 0.9

Predictions

Collective Evaluation: Practice and Theory

Theory Group for“Interdisciplinary Physics”

Post.doc. Paolo Laureti (Roma) Ph.D. Marcel Blattner (Zürich)

Ph.D. Lionel Moret (EPFL)Post.doc. Hasan Masum (Ottava, Canada)

SWITZERLAND

Fribourg Members:

Collective Evaluation: Practice and Theory

SWITZERLAND

External Members:

Yi-kuo Yu (NIH, Washington)Damien Challet (Oxford)Matteo Marsili (ICTP; Italy)Sergei Maslov (BNL, New York)

Collective Evaluation: Practice and Theory

Thank You very much for your attention!

SWITZERLAND

Workshop: Internet @nd Information Filtering Mechanism

SWITZERLAND

Workshop: Internet @nd Information Filtering Mecanisme

SWITZERLAND

Workshop: Internet @nd Information Filtering Mechanism

much harder... but our task is modest:

best approximation,can not dream about decoding “God-Given”

SWITZERLAND