A Study on the Distribution of Cooccurrence Weight Patterns ofClassical Japanese PoetryHilofumi Yamamoto

Tokyo Institute of TechnologyBor HodoscekOsaka University

← more generic words more specific words →

Tall HeadTall HeadWord distributionby frequency-rank

Long Tail DinosaurLong Tail Dinosaur

x

σ0−σ

y

1

Function Tail Content Tail

C.F.Gauss(1777–1855)

ProblemProblemHairball Effect!!!

SolusionSolusionCooccurrence Weight⇒ Bell curve!

ring!ring!

It allows us to divide words into3 kinds:⇒ high, middle, and low.

Cooccurrence weight word patternslike two person’s weights

See what happens!See what happens!Content TailContent TailOver 1σ ⇒

cherry blossom (28/131,4.28): CT ctf.>0.00;non-dist=off; idf=off; pruned under U:1:

here way home1village.11

lodging1

scatter.1

1

forget1

confuse

1

do.1

1

(adv.neg)

1

(must)

1

while

2

�

2

�

1

11

1

1

1

1

1

1

2 21

1

1

1

1

1

1

1

22 1

1

1

1

1

1

1

2 21

1

21

104

2

8

15

8

perplexed

1

(adv.will)4

forcibly

1

�9

�6

even

3

see.310�11

��

2

get used

1

know

1say

2

(p.wish)

6

�7

bloom

1

ç�¡ã�˙5

(adv.would)

1

ã�˛

4break off

1

�

1

�

2

ã�`ã��

1

increase

1

love.1

1

will not

1

kurabu.pl

1

�2

cloud.vi

1

such as.1

1

��2

wait2

�8�3

borrow1

regrettable1

here we go1

like.2

1

blow

2

�佯�

1

regret

1

bear1

think

1

fast/early1

ã�˙ 1

(ku)

3

lovely

1

sorrowful.1

11

1

1

1

22 1

1

1

1

2

3

1

1

1

1

1

1

1

1 1

1

6

3 2

2

1

1

2

(aux.past)

1(aux.perf)1

2

1

8

4

118

41

2 13

1

1

2

113

3

6 1

1

33

(suffix.forbid)

11

painful.1

1high.1

2

(emph)

1

wish

1

1

(interj.)

2

Yoshino.PN1seemed

1

3

2

1

151

quiet

2

although

1

21

1

2

22 1

2 1

1

15

71

7

7

4

��

1

seem

1

1

5

2

1

1

11

5

14

2

ã�ˆã�˝

2

2

11

6

2

1

11

1

7

7

2

2

3

2

2

2

1 2

2

1

1

2

233

1

1

42

2

1

2

1

41

1

either way

1 till

1

person

1

road1

stop.vi.21

go

1

1

1

1

1

2

2

1

1

11

1

1

2

1

1

2 111

1

1

11 lodge.n

1

be.3

1

take1

1

1

1

1

4

2

727 8

2

1

1

1

1

6

1

3

34

73

mountain village

1

1

3

1

2

mind2

wind.n1

mountain2

desolate

1

come1

2

be lonely

1

prosper

1

1

1

1

1

stone

1

waterfall

1

hand

11

run

1

1

stand.vi

11

cross

1

go over

1

send.v2 word

1

hear

1 1

1

present

1

tell

1

2

add 1

get tired.1

12

1

name

1

1vain

1

rare.11

1

1

1

fall

1

1spring rain

1

tear

1

1

1

1

1

1

2

1

1

1

1

1

2

1

2

2

1

1

exchange

1

11

1

1

1

1

1

1

1

2

1112

1

1

2

1

2

2

11

1

see.1

1

1

1

1

1

12

65

4

1

3

2

4

1

1

1

3

1

2

1

1

1

1

1

2

4 8

1

45

2

3

2

1

2

1

1

1

2

1

12

1

1

13

2

3

1

12

1

what1

1

2

thing.2

1

between

1

1spring haze1

hide.vt1

1

1

1

1

1

2

1

11

1

11

1

21

1

24

11

2

11

1

12

1

2

1

myself1

2

1

number

1

only

1

1

1 1

1

1

11

1

1

1

2

1

2 1

1

11

2

1

12

3

11

11

2

1

2

1 1

1

1

1

1

11 1

1

11

111 1

1

snow

1

11

1

such as.2

1

11

1

1

1

11

1

12

1

1

1

2

1

23

1

2

1

1

1

1

trail2

1

change.1

1

transfer.1

11

1

2

1

1

1

1

71

1

7

56

62

2

1

10

1

3

5

23

11

3

1

trust.2

1

1

1

1

2

1

1 11

3

1

1

1

4

2

1

2

51

1

11

1

1 11

11

1

1

1

2

1

1

2

12

1 1

1

thing.1

11

1

spring

1

this year1

��

1

begin1

11

11

1

1

1

21

11

1

1

1

1

1

111

1

1

21

1

year

1

1

11

1

1

1

1

1

111

11

1

11

1

2

1

111

1

1

1

1

1

1

1

white cloud

1

foot

1

pull

1

1

1

1

1

11

1

1

11

1

1

11

1

1

1

1

1

1

11 1

2

1

1

11

11

1

2

2

1

22

1

1

2

1

4

1

2

1

1

1

1

1

3

21

1

2

1

out

1

12

1

111

after

11

11

11112

11

1

1

1

11

1

1

11

2

111

11

1

1

2 225

1

1

11

2

12

1

1

1 1

1

1

1

1

11

1

1

5 25

9

3

2

1

1

34

14

1

1

1

1

1

1

1

1

3

1

1

1

1

1

211

1

31

2

1

1

1

1

mi

221

1

1 1

121

1

1

1

day

111

2

1

cloud.1

1

3

1

1

1

1

1

1

1

1

1

3

1

11

1

1

11

11

1

1

1

1

1

1

1

1

1

13122

1 11

211

1 12

1

11

1

1 12

1

1

11

1

1

1

112

1 11

1

1

11

1

1

1

11

1

1

1

1

413

3

1

1

1 132

1

2

2

1

2

1

1

1

1

21

1 1

1

1

1

11

11

1

1

1

3

2

233

3

11

2

1

2

12

1

1

1

1

1

1 1

1

wave

1

sky

1

11

1

11

1

2

1

1

1

1

11

11

1

1

1

1

1

1

1

1 415

9

1

2

2

2

3

43

121

1

1

1

1

2

1

1

1

1

1

32

1 11

1

1

11

1

1

1

2

1

1

1 1

1

1

1

1

1

2

1

1

1

11

1

1

1

1

1

1

1

1

6

1

32

33

22

1

5

3

12

2

1

1

1

1

1

11

2

1

1

11

1

1

1

2

1

1

24

2

4

421

2

1

1

1

11

1

1

1

1

1

11

1

111

1

1

1

1

1

11

13

1 1

1

1

11

1

1

1

11

11

1

111

1

1

1

1

11

1

1

1

1

11

1

1

21

1

11

1

1

1

1 11

1

22

1

12

1 1

1

1

1

1

1

1

1

for

11

1

1 11

1

1

1

1

1

11 1

1

1

11

11

1

1

1

1

11 1

1

1

1 1

111

1

1

1

1 1

1

1

131 1

1

2

11 11

1

1

11

1

1

1

1

1

1

11

11

1

1

1

1

1

1 1

11

remnant

112

1

1

1

1

1

water1

1

1

11

1 121

1

11

1

1

1

11

1121

1

1

1

1

11 11

2 1

1

11

1

1

11

41

1

11

1

1

2

1

11

1

1

1

1

112

1

11

1

11

1

1

1

1

1

1

1

1

5

4

1

22

3

11

1

1

11

1

1

1

1

1

111

1

1

1

1

11

1

1

1

2

mistake

1

21

11

1

112

1

11

11

1

112

1

1

11

111 2

1

11

1

1 2

1

11

1

111

1

12

1

11

1

111

112

1

1

1 1

1

gorge12

1

1

1

1

1

1

1

1

1

12

1

1

1

1

1

1

1

12

1

1

1

1

1

121

1

1

1

every 1

3

1

12

11

1

1

1 1

1

1

3

1

21

1

1

1

1

31

2

1

1

1

11

old age

1

1

1

1

11

1

11

11

1

1

1

1

11

11

1

11

11

1

1

1

1

131

21

1

211

11

1

1

12

1

1

11

1

1

1

1

1 1

11

1

11

1

1

2

2

2

31

1

1

1

1

1

2

1

2

1

2

1

1

11

1

111

1

2

1

2

2

1

1

1

1

11

11

1

1

21 1

1

1

1

1

2

1

1

1

1

11

1

1

today2

1

1

21

1

11

1

1

1

a glance

1

you.3

1 12

1

1

2

1

1

1

1

1

1

1

11

21

1 1

1

1

1

33

1

1

11

11

11

1

1

1

1

1

34

1

1

2

2

2

1

1

mountain area

11

1

1

1

1

1

1

1

1

11

1

112

3

1

111

1

11

1

1

2

111

1

1 1

1

11

1

11

11

1

1

1 1

1

1

1

1

1

111

1

11

11

1

11

11

1

332

3

2

12

13

3

1

2

1

2

1

1

1

1

2

1

32

1

1

11

1

1

1

1

1

1

1

1

remain

1

be over

111

1

1

1

11

1

1

1

11

1

1

1

1

1

1111

11

1

1

1

1

111

1

11

1

1

1

11 1

11

1

1

1

1

1

1

1

2

1

1

11

1

1

1

1

1

1

1

1

2 12

2

2

1

1

1 121

1

1

1

2 12

31

2

11

1

1

1

1

2

1

1

1

112

1

1

1

12

1

11

11

1

2

111

1

1

1

1 11

1

11

1

1 11

1

1

1 1

1

1

1

1

111

1

1 11

1

21

1

1

2

1

1

1

1

21

1

1

2

11

1

1

1

2

1

person from old village32

1

11

1

1

1

1

1

1

1

world.2

1

11

1

1

1

1

11

11

111

1

1

1

1

11

11

1

1

1

1

11

111

1

1

111 1

colour1

1

1

1

1

1

1

1

1

1

1

11

1

1

1

1

1

1

1

1

11

Function TailFunction TailUnder −1σ ⇒

omou (think.1) B1

VERB a3 LK a1 a2 DN gamma b1 b2 END

omou(think)

suru(do.2)

1

haseru(run)

1

(ra)reru(PASS/POT)5

te/de(LK)

24

ni(LK)

1

ta(PERF)

5

no/koto(DN)38

node(because.2)

1

hodo(as much as..)

2

yo(FP)

7

wa(FP)

13

.

37

u(CJR)

1

torawareru(be captured)

1

1

1

2

11

iru(be-AS)23 kuru

(toward-AS)1 koi(love.2)

1

fuan(anxiety)

1

1

15

ka(Q)

1

2

5

5

1

5

da(ASSER)

34

4 sa(FP)

125

kara(because.1) 1

taisetsu(precious)

1

1 1

6

1

14

20

cherry blossom (28/131,4.28): CT ctf.>0.00;

non-dist=off; idf=off; pruned under U:1:

here

way home

1

lodging1

confuse

1

1

1

village.1

1

scatter.1

1

forget

1

(must)

1

1cloud.vi

1

such as.1

1

either way

road

1

stop.vi.2

1

perplexed1

forcibly1

1

1

1

1

2

exchange

11

1

1

1

what

hide.vt

1

out after1

mountain village

1 1

mi

cloud.1

1

mountain

3

desolate1

prosper

1

painful.1

1

high.13

seem

1

1

break off

1

1

3

trail

2

1

1

1

(suffix.forbid)

1

be lonely

1

(emph)

1

1

11

1

1

1

1

1

for

stone

1waterfall 1

hand

1

run 1

wish

1

1

1

1

1

1

1

1

1

1

1

1

present

1

tell

1

1 1

come

1

1

1

remnant

wave1

1

1

2

1kurabu.pl

1

1(p.wish)

6

see.310even3

(ku)3

lovely

1

fast/early

1

between

1

2

spring haze

1

send.v2 word

1

1

1

2

transfer.11

2

get tired.1

1Yoshino.PN

1

gorge

white cloud

1

foot 1

pull

1

bloom

1

1

seemed1

1

1

11

1

1

1

1

such as.2

1

quiet

1

1

1

1

1

every1

11

1

1

(adv.will)

2

old age

1

11

1

say

11

1

1

1

1 1

1

11

spring

add

1

1

(aux.past)

1

(aux.perf)

1

year

1

rare.1

1

day

1

1

this year

begin 1

get used

1

11

1

1

1

1

today

a glance

1

2

1

you.3

1

2

wait

1

mountain area

mistake

1

1

1

1

1

number

1

till

1

borrow

1

like.2

1

1

regrettable

1

here we go

1

1

1

1

1

only

1

1

myself

1

1

1

1

1

1 1

1

1

person from old village3

1

11

lodge.n

1

world.2

1

sorrowful.1

1

name

1

vain

1

1

wind.n

3

1

1

1

spring rain

regret1

snow1

6

remain

1

be over

1

take

1 1

1

11

1

1

1

1

1

1

1

1

change.11

cross

1

go over1

1

increase

1

bear

1

think

1

1

love.11

blow

1

trust.2

1

1

will not

1