pedersen acl disco-2011 workshop

Identifying Collocations
to Measure Compositionality : Shared Task System Description

Ted PedersenDepartment of Computer ScienceUniversity of Minnesota, Duluthhttp://www.d.umn.edu/~tpederse

My original intent...

Cluster contexts that contain the candidate pair

Identify the number of clusters automatically http://senseclusters.sourceforge.net

If a small number of clusters found...A single underlying meaning exists

Non-compositional!

If a larger number of clusters found No systematic underlying meaning

compositional

Revised Idea

Methods that identify collocations often claim that what they find are non-compositional, or interesting in some way

The Duluth systems seek to evaluate that claim by using various measures of association commonly employed to identify collocationsText::NSP the Ngram Statistics Package

http://ngram.sourceforge.net

Hypothesis #1

An ngram that has a high score according to a measure of association (for identifying collocations) will be less compositional (and less literal) than those that have lower scoresNote that the hypothesis is stated in relative terms, not absolute

Well suited for shared task

Measures of Association

Log-likelihood ratio (ll)

Mutual Information (tmi)

Pearson's chi-squared test (x2)

Pointwise Mutual Information (pmi)

Poisson-Stiring (ps)

Fisher's Exact Test (leftFisher)

Jaccard Coefficient (jaccard)

Odds Ratio (odds)

Dice Coefficient (dice)

T-score (tscore)

Measures of Association

In general these compare the frequency of a word or pair of words with an expected value based on the assumption of independencep(w1,w2) = p(w1)*p(w2) ??

If the frequency of a word or pair of words is about what would be expected if they were independent, then these get a low score and aren't considered interestingLess likely to be non-compositional

Comparing Observed with Expected

p(w1,w2) = n_11 / n_++

p(w1) = n_1+ / n_++

p(w2) = n_+1 / n_++

m_11 = (n_1+ * n_+1) / n_++Generalizes to m_ij

W2 NOT W2W1n_11n_12n_1+

NOT W1n_21n_22n_2+

n_+1n_+2n_++

ACL 2011 abstract corpus
http://www.d.umn.edu/~tpederse/acl2011.abstracts.txt

translation NOT translationmachinen_11 = 65m_11 = 1.58n_12 = 14m_12 = 77.4179

NOT machinen_21 = 48m_21 = 111.42n_22 = 5,512m_22 = 5448.585,560

1135,5265,639

Do n_ij and m_ij diverge enough to reject the model of independence?Different measures answer this question different ways

Counting with windows

Text : a b c d e f g h i j k

Window 2 (w1 w2)a b, b c, c d, d e, e f, f g,

Window 4 (w1 * * w2)a b, a c, a d, b c, b d, b e, c d, c e, c f, ...

Window 10 (w1 * * * * * * * * w2)a b, a c, a d, a e, a f, a g, a h, a i, a j, b c, b d ...

Development of Duluth Systems

Duluth-1 (aka The Flagship) : based on the measure that had the highest correlation with the fine grained gold standard data

Duluth-2 (aka Coward's Comfort) : use the measure most distinct from Duluth-1

Duluth-3 (aka Why not?) : three submissions allowed, so why not...

Rank correlation with
fine grained gold standard

2410

tscore0.14810.21140.2674

tmi0.13350.19080.2361

ll0.13360.19130.2358

frequency0.18650.21000.2126

ps0.09920.15540.1874

x20.11570.11720.1654

phi0.12530.11670.1646

jaccard0.12530.12550.1602

dice0.12530.12550.1602

odds0.02160.00600.0257

pmi-0.0241-0.01450.0143

rightFisher-0.1768-0.08170.0740

leftFisher0.13160.0686-0.0870

twotailed-0.1445-0.0651-0.1064

So many tests??
I suggested the t-score in 1991...

The Flagship : Duluth-1

t-score with a window size of 10Rank correlation of 0.2674

t = ( n_11 m_11 ) / sqrt (n_11)

BUT...

t-score with window size of 2 has huge rank correlation with frequency (0.9857)Somewhat less with window size of 10 (0.8477) but still high...

Can a measure that correlates so well with frequency really be effective?

Hypothesis #2

Very frequent word pairs are more likely to be compositional (i.e., highly literal) than are less frequent word pairsHighly frequent word pairs tend to be very literal and non-compositional (e.g., for the) and it would in general be a surprise to expect a compositional pair to attain as high a frequency

Coward's Comfort?

PMI with window size of 2Rank correlation of -0.0241 with gold standard

Low correlation of 0.2487 with frequency

PMI = log (n_11/m_11)

RightFisher and two tailed Fisher had lower correlation, but aren't really suitable for collocation discovery

PMI has long history of use in collocation discovery

I'm no coward, and I like PMI!

Why not? .. Duluth-3

PMI with a window size of 2

PMI very biased towards word pairs that only occur togetherHighest score always for pairs that occur just 1 time and only with each other

Wide window in Duluth-2 means that pairs with high PMI scores generally occur only together

Narrow window in Duluth-3 might tend to miss other occurrences of words (outside window)

Scoring

Shared task scoring on a scale of 0 100 where 100 means highly literal.

In measures of association higher scores mean less literal (at least according to hypothesis 1)

Association scores converted to 0 100 scale by normalizing and subtracting from 100100 * (1 m(w1,w)/max(m(W1,W2)))

Binned 0-33 low, 34-66 medium, 67-100 high

Results

Duluth-1 top ranked for coarse evaluation Duluth-3 top ranked for coarse EN_V_SUBJ (by a large margin...??)

Duluth-1 middle of pack for numerical scoring

Duluth-2 and Duluth-3 generally less effective for both coarse and numerical scoring

No correlation with numerical scoring?I found correlations with training data...?

Conclusions

Standard techniques for ranking collocations are effective at identifying compositionality

Scoring of compositionality is less successful

The t-score is successful because it optimizes two potentially competing hypotheses : word pairs with high association scores are more likely to be non-compositional, and

more frequent word pairs are likely to be compositional

Thank You!

All experiments conducted with version 1.23 of the Ngram Statistics Package

http://ngram.sourceforge.net

Click to edit the title text format


Klicken Sie, um das Format des Titeltextes zu bearbeiten

Klicken Sie, um die Formate des Gliederungstextes zu bearbeitenZweite GliederungsebeneDritte GliederungsebeneVierte GliederungsebeneFnfte GliederungsebeneSechste GliederungsebeneSiebente GliederungsebeneAchte GliederungsebeneNeunte Gliederungsebene


Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level

Made with OpenOffice.org


Pulse para editar el formato del texto de ttulo

Pulse para editar los formatos del texto del esquemaSegundo nivel del esquemaTercer nivel del esquemaCuarto nivel del esquemaQuinto nivel del esquemaSexto nivel del esquemaSptimo nivel del esquemaOctavo nivel del esquemaNoveno nivel del esquema































23456789





Klicken Sie, um die Formate des Gliederungstextes zu bearbeitenZweite GliederungsebeneDritte GliederungsebeneVierte RegierungsebeneFnfte GliederungsebeneSechste GliederungsebeneSiebente GliederungsebeneAchte GliederungsebeneNeunte Gliederungsebene




,

OpenOffice.org












pedersen acl disco-2011 workshop

Education