Text Analysis ConferenceKnowledge Base Population

2013

Hoa Trang DangNational Institute of Standards and Technology

Sponsored by:

TAC KBP Goals• Goal: Populate a knowledge base (KB) with information about

entities as found in a collection of source documents, following a specified schema for the KB

• KBP 2009-2011: Focus on augmenting an existing KB. Decompose KBP into two tasks▫ Entity-Linking: link each given named entity mention to a node in

reference KB (or create new node)▫ Slot-Filling: Learn attributes about target entities from the source

documents and add new information about the entity to the reference KB• KBP 2012: Combine entity-linking and slot-filling to build a KB from

scratch -> Cold Start• KBP 2013:

▫ Conversational, informal data (discussion fora)▫ Temporal constraints for Slot Filling (2011 pilot)▫ Sentiment analysis for Slot Filling

TAC KBP 2013 Track Participants• Track coordinators

▫ Hoa Dang (Slot Filler Validation)▫ Jim Mayfield (Entity Linking, Cold Start KBP)▫ Margaret Mitchell (Sentiment Slot Filling)▫ Mihai Surdeanu (English Slot Filling and Temporal Slot

Filling)• LDC linguistic resource providers: Joe Ellis, Jeremy Getman,

Justin Mott, Xuansong Li, Kira Griffitt, Stephanie M. Strassel, Jonathan Wright

• Coordinators emeritus: Ralph Grishman, Heng Ji• Advisor: Boyan Onyshkevych• 45 Teams

▫ 14 countries (21 USA, 9 China, 3 Spain, 2 Germany,….)

6 (8) TAC KBP 2013 Tracks• Entity-Linking

▫ English▫ Chinese▫ Spanish

• Slot-Filling (English)▫ Regular▫ Sentiment▫ Temporal▫ Slot Filler Validation Task

• Cold Start (English)

Entity Linking and Slot Filling Tracks• Goal: Augment a reference knowledge base (KB) with info

about query entities (PER, ORG, GPE) as found in a diverse collection of documents

• Reference KB: Oct 2008 Wikipedia snapshot. Each KB node corresponds to a Wikipedia page and contains:▫ Infobox▫ Wiki_text (free text not in infobox)

• English source documents:▫ 1M News docs▫ 1M Web docs▫ 99K Discussion Forum docs (threads)

• Chinese source documents: 2M news, 800K Web• Spanish source documents: 900K news

Entity-Linking Evaluation Results• English

▫ Participants: 26 teams▫ Highest F1: 0.721 (0.730 in 2012)▫ Median F1: 0.583 (0.536 in 2012)

• Chinese▫ Participants: 4 teams▫ Highest F1: 0.622 (0.740 in 2012)▫ Median F1: 0.619 (0.617 in 2012)

• Spanish▫ Participants 3 teams▫ Highest F1: 0.709 (0.641 in 2012)▫ Median F1: 0.651 (0.612 in 2012)

Regular Slot Filling Evaluation Results

•Participants: 18 teams•Human F1: 0.685 (0.814 in 2012)•Highest System F1: 0.373 (0.517 in 2012)•2nd Highest System F1: 0.339 (0.296 in 2012)•Median System F1: 0.150 (0.099 in 2012)

Sentiment Slot Filling Track

• Sentiment analysis for KBP:▫Holder (PER, ORG, GPE)▫Target (PER, ORG, GPE)▫Polarity (positive, negative)

• Implemented as regular slot filling, with different set of slots▫{per,org,gpe}:positive-towards▫{per,org,gpe}:negative-towards▫{per,org,gpe}:positive-from▫{per,org,gpe}:negative-from

• Participants: 3 teams• Evaluation results:

▫Human F1: 0.727▫Highest System F1: 0.132▫Median System F1: 0.014

Temporal Slot Filling Track• Find tightest temporal constraints [T1 T2 T3 T4] on a

given relation▫ Relation is true for a period beginning between T1 and

T2▫ Relation is true for a period ending between T3 and T4

• Participants: 5 teams• Evaluation results:

▫ Human Accuracy: 0.688▫ Highest System Accuracy: 0.331▫ Median System Accuracy: 0.148

Slot Filler Validation Track (SFV)

• Task: Determine whether or not a candidate slot filler is correct

• Objective: improve precision without excessive reduction of recall

• Participants: 5 teams• Some SFV runs had overwhelmingly positive impact

on individual SF runs!

Cold Start KBP Track • Goal: Build a KB from scratch, containing all targeted info about all

entities as found in a relatively closed domain corpus of documents• KB schema: same entity types and slots as regular slot-filling task• Source document collection:

▫ 50K Web pages from small-town publications (from TREC KBA document stream)

• Required capabilities:▫ Entity-linking: Grounding all named entity mentions in docs to KB

nodes▫ Slot-filling: Learning attributes about all named entities

• Post-submission evaluation queries traverse KB starting from a single entity node (entity mention):▫ 0-hop: Find all children of Michael Jordan▫ 1-hop: Find date of birth of each of the children of Michael Jordan

Cold Start Evaluation Results (Preliminary)• Participants: 3 teams• 0-hop queries:

▫ Highest F1 0.384 (0.497 in 2012)• 1-hop queries:

▫ Highest F1 0.145 (0.255 in 2012)• Combined 0-hop and 1-hop F1

▫ Highest F1: 0.278 (~0.352 in 2012)

TAC KBP Discussion/Planning Sessions• Monday, November 18 (2:15-3:10pm):

▫ English Slot Filling▫ Slot Filler Validation▫ Temporal Slot Filling?▫ +Spanish Slot Filling?▫ +Event identification and argument extraction?

• Tuesday, November 19 (3:00-4:00pm):▫ Cold Start▫ English Entity Linking (as queries in Cold Start

framework?)▫ Cross-Lingual Spanish and Chinese Entity Linking

+ Discussion forum