Advanced AI - Part II

Luc De Raedt

University of Freiburg

WS 2004/2005

Many slides taken from Helmut Schmid

Topic

Statistical Natural Language Processing Applies

Machine Learning / Statistics to Learning : the ability to improve one’s behaviour at a

specific task over time - involves the analysis of data (statistics)

Natural Language Processing

Rationalism versus Empiricism

Rationalist Noam Chomsky - innate language structures AI : hand coding NLP Dominant view 1960-1985

Empiricist Ability to learn is innate AI : language is learned from corpora Dominant 1920-1960 and becoming increasingly

important

Rationalism versus Empiricism

Noam Chomsky: But it must be recognized that the notion of

“probability of a sentence” is an entirely useless one, under any known interpretation of this term

Fred Jelinek (IBM 1988) Every time a linguist leaves the room the

recognition rate goes up. (Alternative: Every time I fire a linguist the

recognizer improves)

This course

Empiricist approach Focus will be on probabilistic models for learning

of natural language No time to treat natural language in depth !

(though this would be quite useful and interesting)

Deserves a full course by itself

Ambiguity

Statistical Disambiguation

• Define a probability model for the data

• Compute the probability of each alternative

• Choose the most likely alternative

NLP and Statistics

Statistical Methods deal with uncertainty.They predict the future behaviour of a systembased on the behaviour observed in the past.

Statistical Methods require training data.

The data in Statistical NLP are the Corpora

NLP and Statistics

Corpus: text collection for linguistic purposes

TokensHow many words are contained in Tom Sawyer? 71.370

TypesHow many different words are contained in T.S.? 8.018

Hapax Legomenawords appearing only once

Corpora

The most frequent words are function words

word freq word freq

the 3332 in 906

and 2972 that 877

a 1775 he 877

to 1725 I 783

of 1440 his 772

was 1161 you 686

it 1027 Tom 679

Word Counts

f nf

1 39932 12923 6644 4105 2436 1997 1728 1319 8210 9111-50 54051-100 99> 100 102

How many words appear f times?

Word Counts

Word Counts

Word Counts

word f r f*r word f r f*rthe 3332 1 3332 turned 51 200 10200and 2972 2 5944 you‘ll 30 300 9000a 1775 3 5235 name 21 400 8400he 877 10 8770 comes 16 500 8000but 410 20 8400 group 13 600 7800be 294 30 8820 lead 11 700 7700there 222 40 8880 friends 10 800 8000one 172 50 8600 begin 9 900 8100about 158 60 9480 family 8 1000 8000more 138 70 9660 brushed 4 2000 8000never 124 80 9920 sins 2 3000 6000Oh 116 90 10440 Could 2 4000 8000two 104 100 10400 Applausive 1 8000 8000

Zipf‘s Law: f~1/r (f*r = const)

Zipf‘s Law

Some probabilistic models

N-grams Predicting the next word

Artificial intelligence and machine …. Statistical natural language ….

Probabilistic Regular (Markov Models) Context-free grammars

Illustration

Wall Street Journal Corpus 3 000 000 words Correct parse tree for sentences known

Constructed by hand Can be used to derive stochastic context free

grammars SCFG assign probability to parse trees

Compute the most probable parse tree

Conclusions

Overview of some probabilistic and machine learning methods for NLP

Also very relevant to bioinformatics ! Analogy between parsing

A sentence A biological string (DNA, protein, mRNA, …)