representational and inferential foundations for possible large-scale information extraction and...

Representational and inferential foundations for possible large-scale information extraction and

question-answering from the web

Stuart Russell

Computer Science Division

UC Berkeley

Goal

• A system that knows everything on the Web*– Answer all questions– Discover patterns– Make predictions

• Raw data → useful knowledge base (→ $$$)

• Requires: NLP, vision, speech, learning, DBs, knowledge representation and reasoning

• Berkeley: Klein, Malik, Morgan, Darrell, Jordan, Bartlett, Hellerstein, Franklin, Hearst++

Past projects: PowerSet

• “Building a natural language search engine that reads and understands every sentence on the Web.”

• Parsing/extraction technology + crowdsourcing to generate collections of x R y triples

• Example:– Manchester United beat Chelsea– Chelsea beat Manchester United

• Bought by Microsoft in 2008, merged into Bing

Current projects: UW Machine Reading

• Initially based on bootstrapping text patterns– Birthplace(Elvis,Tupelo)

=> “Elvis was born in Tupelo”

=> “Obama was born in Hawaii”

=> “Obama’s birthplace was Hawaii” => ….• [Google: Best guess for Elvis Presley Born is January 8, 1935]

– Inaccurate, runs out of gas, learned content shallow, 99% of text ignored

• Moving to incorporate probabilistic knowledge, inference using Markov logic

Current Projects: NELL (CMU)

• Bootstrapping approach to learning facts from the web using text patterns (642,797 so far)

• Initial ontology of basic categories and typed relations• Examples:

– the_chicken is a type of meat 100.0%– coventry_evening_telegraph is a blog 99.0%– state_university is a sports team also known as

syracuse_university 93.8% – orac_values_for_mushrooms is a fungus 100.0%– Hank Paulson is the CEO of Goldman 100.0%

Problems

• Language (incl. speech act pragmatics)– … Jerry Brown, who has been called the first American in space

• Uncertainty– Reference uncertainty is ubiquitous– Bootstrapping can converge or diverge; exacerbated by

“accepting” uncertain facts, naïve probability models

• Universal ontological framework (O(1) work)– Taxonomy, events, compositional structure, time…– Compositional structure of objects and events– Knowledge, belief, other agents– Semantic content below lexical level (must be learned)

• E.g., buy = sell-1, ownership, transfer, etc.

Technical approach• Web is just evidence; compute

P(World | web) α P(web | World) P(World)

• What is the domain of the World variable? – Complex sets of interrelated objects and events

• How does it cause the Web variable?– Pragmatics/semantics/syntax (and copying!)

• Uncertainty about– What objects exist– How they’re related– What phrases/images refer to what real objects

• => Open-universe, first-order probabilistic language

9

Brief history of expressiveness

atomic propositional first-order/relational

logic

probability

5th C B.C. 19th C

17th C 20th C 21st C

10

Brief history of expressiveness

atomic propositional first-order/relational

logic

probability

5th C B.C. 19th C

17th C 20th C 21st C

(be patient!)

12

Herbrand vs full first-order

GivenFather(Bill,William) and Father(Bill,Junior)How many children does Bill have?

13



Herbrand semantics:2

14



Herbrand semantics:2First-order logical semantics:

Between 1 and ∞

15

Possible worlds• Propositional

16


• First-order + unique names, domain closureA B

C D

A B

C D

A B

C D

A B

C D

A B

C D

17


• First-order + unique names, domain closure

• First-order open-universe

A B

C D

A B

C D

A B

C D

A B

C D

A B

C D

A B C D A B C D A B C D A B C D A B C D A B C D

18

Open-universe models in BLOG

• Construct worlds using two kinds of steps, proceeding in topological order:– Dependency statements: Set the value of a

function or relation on a tuple of (quantified) arguments, conditioned on parent values

19

Open-universe models in BLOG

• Construct worlds using two kinds of steps, proceeding in topological order:– Dependency statements: Set the value of a

function or relation on a tuple of (quantified) arguments, conditioned on parent values

– Number statements: Add some objects to the world, conditioned on what objects and relations exist so far

20

Technical basics

Theorem: Every well-formed* BLOG model specifies a unique proper probability distribution over open-universe possible worlds; equivalent to an infinite contingent Bayes net

Theorem: BLOG inference algorithms (rejection sampling, importance sampling, MCMC) converge to correct posteriors for any well-formed* model, for any first-order query

21

Example: Citation Matching[Lashkari et al 94] Collaborative Interface Agents,

Yezdi Lashkari, Max Metral, and Pattie Maes, Proceedings of the Twelfth National Conference on Articial Intelligence, MIT Press, Cambridge, MA, 1994.

Metral M. Lashkari, Y. and P. Maes. Collaborative interface agents. In Conference of the American Association for Artificial Intelligence, Seattle, WA, August 1994.

Are these descriptions of the same object?

Core task in CiteSeer, Google Scholar, over 300 companies in the record linkage industry

22

(Simplified) BLOG model#Researcher ~ NumResearchersPrior();

Name(r) ~ NamePrior();

#Paper(FirstAuthor = r) ~ NumPapersPrior(Position(r));

Title(p) ~ TitlePrior();

PubCited(c) ~ Uniform({Paper p});

Text(c) ~ NoisyCitationGrammar (Name(FirstAuthor(PubCited(c))), Title(PubCited(c)));

23







24







25







26







27







28







Evidence: lots of citation stringsQuery: who wrote what? Which paper is being cited in this string? Are these two people the same?

29

Citation Matching Results

Four data sets of ~300-500 citations, referring to ~150-300 papers

0

0.05

0.1

0.15

0.2

0.25

Reinforce Face Reason Constraint

Error

(Fraction of Clusters Not Recovered Correctly)

Phrase Matching[Lawrence et al. 1999]

Generative Model + MCMC[Pasula et al. 2002]

Conditional Random Field[Wellner et al. 2004]

30

Cross-Citation Disambiguation

Wauchope, K. Eucalyptus: Integrating Natural Language Input with a Graphical User Interface. NRL Report NRL/FR/5510-94-9711 (1994).

Is "Eucalyptus" part of the title, or is the author named K. Eucalyptus Wauchope?

Kenneth Wauchope (1994). Eucalyptus: Integrating natural language input with a graphical user interface. NRL Report NRL/FR/5510-94-9711, Naval Research Laboratory, Washington, DC, 39pp.

Second citation makes it clear how to parse the first one

Example: multitarget tracking#Aircraft(EntryTime = t) ~ NumAircraftPrior();

Exits(a, t) if InFlight(a, t) then ~ Bernoulli(0.1);

InFlight(a, t)if t < EntryTime(a) then = falseelseif t = EntryTime(a) then = trueelse = (InFlight(a, t-1) & !Exits(a, t-1));

State(a, t)if t = EntryTime(a) then ~ InitState() elseif InFlight(a, t) then ~ StateTransition(State(a, t-1));

#Blip(Source = a, Time = t) if InFlight(a, t) then

~ NumDetectionsCPD(State(a, t));

#Blip(Time = t) ~ NumFalseAlarmsPrior();

ApparentPos(r)if (Source(r) = null) then ~ FalseAlarmDistrib()else ~ ObsCPD(State(Source(r), Time(r)));

32

#Person ~ LogNormal[6.9, 2.3]();Honest(x) ~ Boolean[0.9]();#Login(Owner = x) ~ if Honest(x) then 1 else LogNormal[4.6,2.3]();Transaction(x,y) ~ if Owner(x) = Owner(y) then SibylPrior() else TransactionPrior(Honest(Owner(x)), Honest(Owner(y)));Recommends(x,y) ~ if Transaction(x,y) then if Owner(x) = Owner(y) then Boolean[0.99]() else RecPrior(Honest(Owner(x)), Honest(Owner(y)));

Evidence: lots of transactions and recommendationsQuery: Honest(x)

Example: cybersecurity sibyl defence

33

Example: Global seismic monitoring

• CTBT bans testing of nuclear weapons on earth– Allows for outside inspection of 1000km2

• Need 9 more ratifications for “entry into force” including US, China

• US Senate refused to ratify in 1998– “too hard to monitor”

34

254 monitoring stations

36

Vertically Integrated Seismic Analysis

• The problem is hard:– ~10000 “detections” per day, 90% false– CTBT system (SEL3) finds 69% of significant events plus

about twice as many spurious (nonexistent) events– 16 human analysts find more events, correct existing ones,

throw out spurious events, generate LEB (“ground truth”)– Unreliable below magnitude 4 (1kT)

47

# SeismicEvents ~ Poisson[time_duration * event_rate];IsEarthQuake(e) ~ Bernoulli(.999);EventLocation(e) ~ If IsEarthQuake(e) then EarthQuakeDistribution()

Else UniformEarthDistribution();Magnitude(e) ~ Exponential(log(10)) + min_magnitude;Distance(e,s) = GeographicalDistance(EventLocation(e), SiteLocation(s));IsDetected(e,p,s) ~ Logistic[site-coefficients(s,p)](Magnitude(e), Distance(e,s);#Arrivals(site = s) ~ Poisson[time_duration * false_rate(s)];#Arrivals(event=e, site) = If IsDetected(e,s) then 1 else 0;Time(a) ~ If (event(a) = null) then Uniform(0,time_duration)

else IASPEI(EventLocation(event(a)),SiteLocation(site(a)),Phase(a)) + TimeRes(a);TimeRes(a) ~ Laplace(time_location(site(a)), time_scale(site(a)));Azimuth(a) ~ If (event(a) = null) then Uniform(0, 360)

else GeoAzimuth(EventLocation(event(a)),SiteLocation(site(a)) + AzRes(a);AzRes(a) ~ Laplace(0, azimuth_scale(site(a)));Slow(a) ~ If (event(a) = null) then Uniform(0,20)

else IASPEI-slow(EventLocation(event(a)),SiteLocation(site(a)) + SlowRes(site(a));

48

Fraction of LEB events missed

49

Fraction of LEB events missed

50

Event distribution: LEB vs SEL3

51

Event distribution: LEB vs NET-VISA

Open questions

• Efficient inference

• Model construction: creating useful new categories and relations

• HCI: Presenting infinite heterogeneous posterior distributions: Who wrote what when “who” and “what” vary across worlds?

• Making use of partially extracted or unextracted information – “data spaces” (Franklin, Halevy)

• Adversarial data: game-theoretic analysis?

Summary

• Basic components (accurate parsing, first-order and modal probabilistic logics, universal ontology) are mostly in place; NLP is moving back towards combined syntax/semantics

• Vertically integrated probabilistic models can be much more effective that bottom-up pipelines

• The Web is Very Big– Does not imply we must use trivial methods– Does not imply that trivial methods will suffice– Won’t happen for free

55

Example of using extra detections

56

NEIC event (3.0) missed by LEB

57


58


TREC 9 Results (2000)

representational and inferential foundations for possible large-scale information extraction and...

Documents

herbrand vs

herbrand semantics

web variable

orderrelationallogicproba

probabilistic knowledge

real objects

pworld web pweb world

learning facts