Corpora and Statistical Methods Lecture 12

Language modelling using N-GramsCorpora and Statistical Methods

Example taskThe word-prediction task (Shannon game)

Given:a sequence of words (the history)a choice of next word

Predict:the most likely next word

Generalises easily to other problems, such as predicting the POS of unknowns based on history.Applications of the Shannon gameAutomatic speech recognition (cf. tutorial 1):given a sequence of possible words, estimate its probability

Context-sensitive spelling correction:Many spelling errors are real wordsHe walked for miles in the dessert. (resp. desert)

Identifying such errors requires a global estimate of the probability of a sentence.

Applications of N-gram models generallyPOS Tagging:predict the POS of an unknown word by looking at its history

Statistical parsing:e.g. predict the group of words that together form a phrase

Statistical NL Generation:given a semantic form to be realised as text, and several possible realisations, select the most probable one.A real-world example: Googles did you meanGoogle uses an n-gram model (based on sequences of characters, not words).

In this case, the sequence apple desserts is much more probable than apple deserts

How it worksDocuments provided by the search engine are added to:An index (for fast retrieval)A language model (based on probability of a sequence of characters)

A submitted query (apple deserts) can be modified (using character insertions, deletions, substitutions and transpositions) to yield a query that fits the language model better (apple desserts).

Outcome is a context-sensitive spelling correction:apple deserts apple dessertsfrod baggins frodo bagginsfrod ford

The noisy channel modelAfter Jurafsky and Martin (2009), Speech and Language Processing (2nd Ed). Prentice Hall p. 198

The assumptions behind n-gram modelsThe Markov AssumptionMarkov models:probabilistic models which predict the likelihood of a future unit based on limited history

in language modelling, this pans out as the local history assumption:the probability of wn depends on a limited number of prior words

utility of the assumption:we can rely on a small n for our n-gram models (bigram, trigram)long n-grams become exceedingly sparseProbabilities become very small with long sequencesThe structure of an n-gram modelThe task can be re-stated in conditional probabilistic terms:

Limiting n under the Markov Assumption means:greater chance of finding more than one occurrence of the sequence w1wn-1more robust statistical estimations

N-grams are essentially equivalence classes or binsevery unique n-gram is a type or bin

Structure of n-gram models (II)If we construct a model where all histories with the same n-1 words are considered one class or bin, we have an (n-1)th order Markov Model

Note terminology:n-gram model = (n-1)th order Markov ModelMethodological considerationsWe are often concerned with:building an n-gram modelevaluating it

We therefore make a distinction between training and test dataYou never test on your training dataIf you do, youre bound to get good results.N-gram models tend to be overtrained, i.e.: if you train on a corpus C, your model will be biased towards expecting the kinds of events in C.Another term for this: overfittingDividing the dataGiven: a corpus of n units (words, sentences, depending on the task)A large proportion of the corpus is reserved for training.A smaller proportion for testing/evaluation (normally 5-10%)Held-out (validation) dataHeld-out estimation:during training, we sometimes estimate parameters for our model empirically

commonly used in smoothing (how much probability space do we want to set aside for unseen data)?

therefore, the training set is often split further into training data and validation datanormally, held-out data is 10% of the size of the training data

Development dataA common approach:train an algorithm on training dataa. (estimate further parameters on held-out data if required)evaluate itre-tune it go back to Step 1 until no further finetuning necessaryCarry out final evaluation

For this purpose, its useful to have:training data for step 1development set for steps 2-4final test set for step 5

Significance testingOften, we compare the performance of our algorithm against some baseline.

A single, raw performance score wont tell us much. We need to test for significance (e.g. using t-test).

Typical method:Split test set into several small test sets, e.g. 20 samplesevaluation carried out separately on eachmean and variance estimated based on 20 different samplestest for significant difference between algorithm and a predefined baselineSize of n-gram modelsIn a corpus of vocabulary size N, the assumption is that any combination of n words is a potential n-gram. For a bigram model: N2 possible n-grams in principleFor a trigram model: N3 possible n-grams.

Size (continued)Each n-gram in our model is a parameter used to estimate probability of the next possible word.too many parameters make the model unwieldytoo many parameters lead to data sparseness: most of them will have f = 0 or 1

Most models stick to unigrams, bigrams or trigrams.estimation can also combine different order modelsFurther considerationsWhen building a model, we tend to take into account the start-of-sentence symbol:the girl swallowed a large green caterpillar thethe girl

Also typical to map all tokens w such that count(w) < k to :usually, tokens with frequency 1 or 2 are just considered unknown or unseenthis reduces the parameter spaceBuilding models using Maximum Likelihood EstimationMaximum Likelihood Estimation ApproachBasic equation:

In a unigram model, this reduces to simple probability.

MLE models estimate probability using relative frequency.

Limitations of MLEMLE builds the model that maximises the probability of the training data.

Unseen events in the training data are assigned zero probability.Since n-gram models tend to be sparse, this is a real problem.

Consequences:seen events are given more probability mass than they haveunseen events are given zero massSeen/unseenAAProbability mass of events in training dataProbability massof events not in training dataThe problem with MLE is that it distributes A among members of A.The solutionSolution is to correct MLE estimation using a smoothing technique.

More on this in the next part

But cf. Tutorial 1, which introduced the simplest method of smoothing known.

Adequacy of different order modelsManning/Schutze `99 report results for n-gram models of a corpus of the novels of Austen.

Task: use n-gram model to predict the probability of a sentence in the test data.

Models:unigram: essentially zero-context markov model, uses only the probability of individual wordsbigramtrigram4-gramExample test caseTraining Corpus: five Jane Austen novels Corpus size = 617,091 words Vocabulary size = 14,585 unique types

Task: predict the next word of the trigram inferior to ________from test data, Persuasion: [In person, she was] inferior to both [sisters.]Selecting an n

Vocabulary (V) = 20,000 wordsnNumber of bins(i.e. no. of possible unique n-grams) 2 (bigrams)400,000,0003 (trigrams)8,000,000,000,0004 (4-grams)1.6 x 1017Adequacy of unigramsProblems with unigram models:not entirely hopeless because most sentences contain a majority of highly common wordsignores syntax completely:P(In person she was inferior) = P(inferior was she person in)Adequacy of bigramsBigrams:improve situation dramaticallysome unexpected results:p(she|person) decreases compared to the unigram model. Though she is very common, it is uncommon after person

Adequacy of trigramsTrigram models will do brilliantly when theyre useful.They capture a surprising amount of contextual variation in text.Biggest limitation:most new trigrams in test data will not have been seen in training data.

Problem carries over to 4-grams, and is much worse!Reliability vs. Discriminationlarger n: more information about the context of the specific instance (greater discrimination)

smaller n: more instances in training data, better statistical estimates (more reliability)Backing offPossible way of striking a balance between reliability and discrimination:backoff model:where possible, use a trigramif trigram is unseen, try and back off to a bigram modelif bigrams are unseen, try and back off to a unigramEvaluating language modelsPerplexityRecall: Entropy is a measure of uncertainty: high entropy = high uncertainty

perplexity:if Ive trained on a sample, how surprised am I when exposed to a new sample?a measure of uncertainty of a model on new data

Entropy as expected valueOne way to think of the summation part is as a weighted average of the information content.

We can view this average value as an expectation: the expected surprise/uncertainty of our model.

Comparing distributionsWe have a language model built from a sample. The sample is a probability distribution q over n-grams. q(x) = the probability of some n-gram x in our model.

The sample is generated from a true population (the language) with probability distribution p.p(x) = the probability of x in the true distributionEvaluating a language modelWed like an estimate of how good our model is as a model of the languagei.e. wed like to compare q to p

We dont have access to p. (Hence, cant use KL-Divergence)

Instead, we use our test data as an estimate of p.Cross-entropy: basic intuitionMeasure the number of bits needed to identify an event coming from p, if we code it according to q:We draw sequences according to p;but we sum the log of their probability according to q.

This estimate is called cross-entropy H(p,q)Cross-entropy: p vs. qCross-entropy is an upper bound on the entropy of the true distribution p:H(p) H(p,q)if our model distribution (q) is good, H(p,q) H(p)

We estimate cross-entropy based on our test data.Gives an estimate of the distance of our language model from the distribution in the test sample.Estimating cross-entropy

Probability accordingto p (test set)Entropy accordingto q (language model)PerplexityThe perplexity of a language model with probability distribution q, relative to a test set with probability distribution p is:

A perplexity value of k (obtained on a test set) tells us:our model is as surprised on average as it would be if it had to make k guesses for every sequence (n-gram) in the test data.

The lower the perplexity, the better the language model (the lower the surprise on our test data).

Perplexity example (Jurafsky & Martin, 2000, p. 228)Trained unigram, bigram and trigram models from a corpus of news text (Wall Street Journal)applied smoothing38 million wordsVocab of 19,979 (low-frequency words mapped to UNK).Computed perplexity on a test set of 1.5 million words.J&Ms resultsTrigrams do best of all.Value suggests the extent to which the model can fit the data in the test set.Note: with unigrams, the model has to make lots of guesses!N-Gram modelPerplexityUnigram962Bigram170Trigram109SummaryMain point about Markov-based language models:data sparseness is always a problemsmoothing techniques are required to estimate probability of unseen events

Next part discusses more refined smoothing techniques than those seen so far.Smoothing (aka discounting) techniquesPart 2OverviewSmoothing methods:Simple smoothingWitten-Bell & Good-Turing estimationHeld-out estimation and cross-validation

Combining several n-gram models:back-off models

Rationale behind smoothingSample frequencies seen events with probability P unseen events (including grammatical zeroes) with probability 0

Real population frequencies seen events (including the unseen events in our sample)+ smoothingto approximateLower probabilities for seen events (discounting). Left over probability mass distributed over unseens (smoothing).results inLaplaces law, Lidstones law and the Jeffreys-Perks lawInstances in the Training Corpus:inferior to ________

F(w)Maximum Likelihood Estimate

F(w)Unknowns are assigned 0% probability massActual Probability Distribution

F(w)These are non-zero probabilities in the real distributionLaPlaces Law (Add-one smoothing)

F(w)LaPlaces Law (Add-one smoothing)

F(w)LaPlaces Law

F(w)NB. This method ends up assigning most prob. mass to unseensGeneralisation: Lidstones Law

P = probability of specific n-gramC(x) = count of n-gram x in training dataN = total n-grams in training dataV = number of bins (possible n-grams) = small positive numberM.L.E: = 0LaPlaces Law: = 1 (add-one smoothing)Jeffreys-Perks Law: = Jeffreys-Perks Law

F(w)Objections to Lidstones LawNeed an a priori way to determine .

Predicts all unseen events to be equally likely

Gives probability estimates linear in the M.L.E. frequencyWitten-Bell discountingMain intuitionA zero-frequency event can be thought of as an event which hasnt happened (yet).The probability of it happening can be estimated from the probability of sth happening for the first time (i.e. the hapaxes in our corpus).

The count of things which are seen only once can be used to estimate the count of things that are never seen.Witten-Bell methodT = no. of times we saw an event for the first time. = no of different n-gram types (bins)NB: T is no. of types actually attested (unlike V, the no of possible in our previous estimations)

Estimate total probability mass of unseen n-grams:

Basically, MLE of the probability of a new type event occurring (being seen for the first time)This is the total probability mass to be distributed among all zero events (unseens)

no of actual n-grams (N) + no of actual types (T)Witten-Bell methodDivide the total probability mass among all the zero n-grams. Can distribute it equally.

Remove this probability mass from the non-zero n-grams (discounting):

Witten-Bell vs. Add-oneIf we work with unigrams, Witten-Bell and Add-one smoothing give very similar results.

The difference is with n-grams for n>1.

Main idea: estimate probability of an unseen bigram from the probability of seeing a bigram starting with w1 for the first time. Witten-Bell with bigramsGeneralised total probability mass estimate:

No. bigram types beginning with w1No. bigram tokens beginning with w1Estimated total probability of bigrams starting with w1Witten-Bell with bigramsNon-zero bigrams get discounted as before, but again conditioning on history:

Note: Witten-Bell wont assign the same probability mass to all unseen n-grams. The amount assigned will depend on the first word in the bigram (first n-1 words in the n-gram).

Good-Turing Frequency EstimationGood-Turing methodIntroduced by Good (1953), but partly attributed to Alan Turingwork carried out at Bletchley Park during WWII

Simple Good-Turing method (Gale and Sampson 1995)

Main idea:re-estimate amount of probability mass assigned to low-frequency or zero n-grams based on the number of n-grams (types) with higher frequencies

RationaleGiven:sample frequency of a type (n-gram, aka bin, aka equivalence class)

GT provides:an estimate of the true population frequency of a typean estimate of the total probability of unseen types in the population.Ingredientsthe sample frequency C(x) of an n-gram x in a corpus of size N with vocabulary size V

the no. of n-gram types with frequency C, Tc

C*(x): the estimated true population frequency of an n-gram x with sample frequency C(x)N.B. in a perfect sample, C(x) = C*(x)in real life, C*(x) < C(x) (i.e. sample overestimates the true frequency)Some backgroundSuppose:we had access to the true population probability of our n-gramswe treat each occurrence of an n-gram as a Bernoulli trial: either the n-gram is x or noti.e. a binomial assumption

Then, we could calculate the expected no of types with frequency C, Tc= the expected frequency of frequency

where:TC = no. of n-gram types with frequency CN = total no. of n-grams

Background continuedGiven an estimate of E(TC), we could then calculate C*Fundamental underlying theorem:

Note: this makes the estimated (true) frequency C* a function of the expected number of types with frequency C+1. Like Witten-bell, it makes the adjusted count of zero-frequency events dependent on events of frequency 1.

Background continuedWe can use the above to calculate adjusted frequencies directly.Often, though, we want to calculate the total missing probability mass for zero-count n-grams (the unseens):

Where: T1 is the number of types with frequency 1N is the total number of items seen in training

Example of readjusted countsFrom: Jurafsky & Martin 2009Examples are bigram counts from two corpora.

A little problemThe GT theorem assumes that we know the expected population count of types!Weve assumed that we get this from a corpus, but this, of course, is not the case.

Secondly, TC+1 will often be zero! For example, its quite possible to find several n-grams with frequency 100, and no n-grams with frequency 101!Note that this is more typical for high frequencies, than low ones.

Low frequencies and gapsLow C: linear trend.Higher C: angular discontinuity.Frequencies in corpus display jumps and so do frequencies of frequencies.This implies the presence of gaps at higher frequencies.

C: log10 frequency(after Gale and Sampson 1995)TC: log10 frequency of frequencyPossible solutionUse Good-Turing for n-grams with corpus frequency less than some constant k (typically, k = 5). Low-frequency types are numerous, so GT is reliable.High-frequency types are assumed to be near the truth.

To avoid gaps (where Tc+1 = 0), empirically estimate a function S(C) that acts as a proxy for E(TC)

Proxy function for gapsFor any sample C, let:

where: C is the next highest non-zero frequencyC is the previous non-zero frequency

log10 frequency(after Gale and Sampson 1995)log10 SC

Gale and Sampsons combined proposalFor low frequencies (< k), use standard equation, assuming E(TC) = TC

If we have gaps (i.e. TC =0), we use our proxy function for TC. Obtained through linear regression to fit the log-log curve

And for high frequencies, we can assume that C* = C

Finally, estimate probability of n-gram:

GT Estimation: Final stepGT gives approximations to probabilities. Re-estimated probabilities of n-grams wont sum to 1necessary to re-normaliseGale/Sampson 1995:

A final word on GT smoothingIn practice, GT is very seldom used on its own.

Most frequently, we use GT with backoff, about which, more later...Held-out estimation & cross-validationHeld-out estimation: General ideahold back some training data

create our language model

compare, for each n-gram (w1wn):Ct: estimated frequency of the n-gram based on training dataCh: frequency of the n-gram in the held-out dataHeld-out estimationDefine TotC as:total no. of times that n-grams with frequency C in the training corpus actually occurred in the held-out data

Re-estimate the probability:

Cross-validationProblem with held-out estimation:our training set is smaller

Way around this:divide training data into training + validation data (roughly equal sizes)use each half first as training then as validation (i.e. train twice)take a meanCross-Validation(a.k.a. deleted estimation)Use training and validation dataABtrainvalidatevalidatetrainModel 1Model 2Model 1Model 2+Final ModelSplit training data:train on A, validate on Btrain on B, validate on Acombine model 1 & 2Cross-ValidationCombined estimate (arithmetic mean):

Combining estimators: backoff and interpolationThe rationaleWe would like to balance between reliability and discrimination:use trigram where usefulotherwise back off to bigram, unigram

How can you develop a model to utilize different length n-grams as appropriate?Interpolation vs. BackoffInterpolation: compute probability of an n-gram as a function of:The n-gram itselfAll lower-order n-gramsProbabilities are linearly interpolated.Lower-order n-grams are always used.

Backoff: If n-gram exists in model, use thatElse fall back to lower order n-grams

Simple interpolation: trigram exampleCombine all estimates, weighted by a factor.

All parameters should sum to 1:

NB: we have different interpolation parameters for the various n-gram sizes.

More sophisticated versionSuppose we have the trigrams: (the dog barked)(the puppy barked)

Suppose (the dog) occurs several times in our corpus, but not (the puppy)

In our interpolation, we might want to weight trigrams of the form (the dog _) more than (the puppy _) (because the former is composed of a more reliable bigram)

Rather than using the same parameter for all trigrams, we could condition on the initial bigram.Sophisticated interpolation: trigram exampleCombine all estimates, weighted by factors that depend on the context.

Where do parameters come from?Typically:We estimate counts from training data.We estimate parameters from held-out data.The lambdas are chosen so that they maximise the likelihood on the held-out data.

Often, the expectation maximisation (EM) algorithm is used to discover the right values to plug into the equations. (more on this later)BackoffRecall that backoff models only use lower order n-grams when the higher order one is unavailable.

Best known model by Katz (1987).Uses backoff with smoothed probabilitiesSmoothed probabilities obtained using Good-Turing estimation.Backoff: trigram exampleBackoff estimate:

That is:If the trigram has count > 0, we use the smoothed (P*) estimateIf not, we recursively back off to lower orders, interpolating with a parameter (alpha)

Backoff vs. Simple smoothingWith Good-Turing smoothing, we typically end up with the leftover probability mass that is distributed equally among the unseens.So GF tells us how much leftover probability there is.

Backoff gives us a better way of distributing this mass among unseen trigrams, by relying on the counts of their component bigrams and unigrams.So backoff tells us how to divide that leftover probability.Why we need those alphasIf we rely on true probabilities, then for a given word and a given n-gram window, the probability of the word sums to 1:

But if we back off to lower-order model when the trigram probability is 0, were adding extra probability mass, and the sum will now exceed 1.We therefore need:P* to discount the original MLE estimate (P)Alpha parameters to ensure that the probability from the lower-order n-grams sums up to exactly the amount we discounted in P*.

Computing the alphas -- IRecall: we have C(w1w2w3) = 0Let (w1w2) represent the amount of probability left over when we discount (seen) trigrams containing w3

The sum of probabilities P for seen trigrams involving w3 (preceded by any two tokens) is 1. The smoothed probabilities P* sum to less than 1. Were taking the remainder.Computing the alphas -- IIWe now compute alpha:

The denominator sums over all unseen trigrams involving our bigram. We distribute the remaining mass (w1w2) overall all those trigrams.What about unseen bigrams?So what happens if even (w1w2) in (w1w2w3) has count zero?

I.e. we fall to an even lower order. Moreover:

And:

Problems with Backing-OffSuppose (w2 w3) is common but trigram (w1 w2 w3) is unseen

This may be a meaningful gap, rather than a gap due to chance and scarce datai.e., a grammatical null

May not want to back-off to lower-order probabilityin this case, p = 0 is accurate!ReferencesGale, W.A., and Sampson, G. (1995). Good-Turing frequency estimation without tears. Journal of Quantitative Linguistics, 2: 217-237