moving beyond entity extraction to entity resolution - human language technology conference

Basis Technology – Human Language Technology Conference 2012 1

Things, not Strings: From Entity Extraction to Entity Resolution David Murgatroyd

VP, Engineering

Basis Technology


Your job is to analyze reciprocal antagonism between Christian and Islamic extremists across the globe. You want to find information on the Internet on Christian extremist reaction to the killing of the U.S. Ambassador to Libya.

Motivation

✗

✗

✓

✗

✗


That was a lot of work. Can text analytics help?

Help?


✓

✗

✗

Filter out pages with the wrong guy?

Filter?

Filter Example


✓

✗

✗

Add some filters (a/k/a facets)…

Filter?


✓

✗

✗


Filter?


✓

✗

✗


Filter?

Filter results by…

People <choice 1> <choice 2> <choice 3> …


✓

✗

✗

But what can we use as choices?

Filter?




Find names of person, places, organizations in document.

Entity Extraction (Name Tagging)


Group names referring to the same person, within a document.

In-document Coreference Resolution


✓

✗

✗

But what can we use as choices?

Filter choices?




✓

✗

✗

Choices: first way that each person was mentioned in each document?

Filter choices?


Persons named Kris Stephens Chris Stephens Dan Cathy George LiBle …


✓

✗

Choices: first name string for each person in each document?

Filter?

Add filters…

Persons named Dan Cathy George LiBle …

Filtered by…

Persons named Chris Stephens ✗


✓

✗

Choices: first name string for each person in each document?

Filter?

Add filters…


Filtered by…

Persons named Chris Stephens


✓

✗

Problem: Ambiguity – one name, many entities

Filter?

Add filters…


Filtered by…



✓

✗

Problem: Variety – one person, many names

Filter?

Add filters…

Filtered by…

Add filters…


Filtered by…



✓

✗

Problem: Variety – one person, many names

Filter?

Add filters…

Persons named Dan Cathy George LiBle … Chris Stevens J. Christopher Stevens …

Filtered by…



Where does your favorite data set fall?

Ambiguity

Variety

Thousands

1

# of documents

Millions

Billions


✓

✗

✗

Magically group names by person across documents.

Deal with ambiguity and variety?




✓

✗

✗

But there’s still the problem of choices…

Labels for choices?




✓

✗

✗

Use person’s name from highest ranked doc? Still some ambiguity.

Labels for choices?


People Kris Stephens Chris Stephens 1 Chris Stephens 2 …


✓

✗

✗

Entity Resolution: group and also link to a database of known entities (e.g., Wikipedia).

Labels for choices?


People Kris Stephens Chris Stephens 1 Chris Stephens 2 …

Kris Stephens J. Christopher Stevens Chris Stephens …


✓

✗

✗

Labels for choices?


People

For items not in the database, infer a unique label (e.g., for hypothetical Wikipedia page).

Kris Stephens J. Christopher Stevens Chris Stephens …


✓

✗

✗

For items not in the database, infer a unique label (e.g., for hypothetical Wikipedia page).

Filter?


People Kris Stephens (pastor) J. Christopher Stevens Chris Stephens (pastor)


✓

✗

✗

Let’s give it a try…

Filter.


People Kris Stephens (pastor) J. Christopher Stevens Chris Stephens (pastor) Dan Cathy George LiBle …


✓

✗


Filter.

Add filters…

People Kris Stephens (pastor) Chris Stephens (pastor) Dan Cathy George LiBle …

Filtered by…

People J. Christopher Stevens

✗


✓


Filter.

Add filters…


Filtered by…



✓


Filter.

Add filters…


Filtered by…


✓

✓


Does it work?

How do you measure?


Imagine this was the result of applying the filter with the name from wikipedia.

How do you measure?

Add filters…


Filtered by…



Precision: for each document, how much of the stuff grouped with it is correct?

How do you measure?

Add filters…


Filtered by…


✓ ✓

✗ 2 / 3 = 67%

2 / 3 = 67%

1 / 3 = 33%


Recall: for each document, how much of the correct stuff is grouped with?

How do you measure?

Add filters…


Filtered by…


✓ ✓

2 / 5 = 40%

2 / 5 = 40%

✗ ✗ ✗


Does it work?

We often combine Precision and Recall measurements into a single measurement, called “F”.


Where does your favorite data set fall?

Ambiguity

Variety

Thousands

1

# of documents

Millions

Billions


ACE 2005 WEPS-‐2 TAC pre-‐2012 TAC eng 2012 TAC zho 2012 TAC spa 2012 Basis Balanced Basis Ambig Basis Variance 1 Basis Variance 2

Where does your favorite data lie?

Ambiguity

Variety

1

F>=70

F>=?

Thousands

# of documents

Millions

Billions F>=85

corpus


Let’s pretend you’re researching the pastors instead.

Trading off Errors


People Kris Stephens (pastor) J. Christopher Stevens Chris Stephens (pastor) Dan Cathy George LiBle …


What if you think there are too many (or too few)? Add a slider for making filter more fine (or coarse).

Trading off Errors

Add filters…

People J. Christopher Stevens Chris Stephens (pastor) Dan Cathy George LiBle …

Filtered by…

People Kris Stephens (pastor)


Make the filter more fine.

Trading off Errors

Add filters…

People J. Christopher Stevens Chris Stephens (pastor) Dan Cathy George LiBle …

Filtered by…

People Kris Stephens (pastor)


Questions

•  Suggested questions: – Doesn’t Google already do this? – Speed? Scale? – Multi-lingual? – What other uses are there for entity resolution

beyond faceted search?


For more information: Visit www.basistech.com

Write to [email protected]

Call 617-386-2090

Thank you!


Doesn’t Google already do this? Some, when searching for famous entities.


Speed/Scale

•  Support from BRAVE for scale in CY13! •  Research version:

–  tested up to 1m docs – Sub-second per document –  Incremental updates (i.e., you see documents

published minutes ago)


Doesn’t Google already do this?


Other uses for entity resolution ?

•  Supporting relationship resolution by resolving participating entities in the them.

•  Knowledge base population •  Integrating disparate data sets •  Alerting •  Improving relevance of search results •  Predictive Analytics