towards vandalism detection in knowledge bases: corpus construction and analysis (sigir '15...

Motivation

Related workConcentrates on unstructured knowledge bases

Solution IdeaContext Problems

Corpus Construction

SIGIR '15 - Santiago de Chile heindorf@uni-paderborn.de

Towards Vandalism Detection in Knowledge Bases:

Corpus Construction and AnalysisStefan Heindorf Martin Potthast Benno Stein Gregor Engels

Corpus Analysis

Conclusion & Outlook• Vandalism can reduce the quality of knowledge bases• First standardized corpus to study vandalism in structured knowledge bases• Next step: detect vandalism automatically

Download our corpus!

Manual Validation 1,000 rollbacked revision

Manually reviewed 1,000 undone/reverted revisions 1,000 inconspicuous revisions

Option 1: Rollback (by administrators) 86 ± 3 %* revisions indeed vandalism

Option 2: Undo/Restore (by all users) 62 ± 3 %* revisions indeed vandalism

* 95 % confidence level

Overview of Corpus• 24 million revisions• 103 thousand vandalism revisions

Corpus Revisions over Time

What is vandalized? Who vandalizes?

For our corpus:Rollbacked revisions are considered vandalism

Information systems structured knowledge bases

Patrollers are busy

Vandalism is notdetected in time

Some people vandalize those knowledge bases MLMachine learning

to detect vandalismVandalismcorpus

Vandalism?

Revision 1: yesRevision 2: no

⋮

use

40% of vandalism in structured data

New features beneficial86 % of vandalism

by unregistered users

32%57%

3%8%

32%

Top vandalized items Top items by category

Unreg-istered

Registered

Cases Item title47 Cristiano Ronaldo43 Lionel Messi43 One Direction41 Portal:Featured content34 Justin Bieber33 Barack Obama29 English Wikipedia29 Selena Gomez

Contributions• Wikidata Vandalism Corpus WDVC-2015• Corpus analysis

Automatic Revision Labeling• Wikidata revision history• Only non-bot revisions considered• Goal: Automatic labeling as vandalism/non-vandalism

Option 1: Rollback (by administrators) 103,205 rollbacked revisions

Option 2: Undo/Restore (by all users) 64,820 undone/reverted revisions

1

2

3

4

2

5

revert

Culture

People

Society

Nature

Meta items

Technology

Places

Other

20%

20%

16%

14%

13%

9%

8%

1%

12%

21%

9%

15%

8%

4%

31%

1%

Top 1,000 of all itemsTop 1,000 vandalized items

http://www.uni-weimar.de/en/media/chairs/webis/corpora/corpus-wdvc-15/