mô hình tách từ, gán nhãn từ loại và hướng tiếp cận tích hợp cho tiếng...

7/28/2019 M hnh tch t, gn nhn t loi v hng tip cn tch hp cho ting Vit

1/56

M hnh tch t, gn nhn tloi v hng tip cn

tch hp cho ting Vit

TRN TH OANH

Master Thesis

Ging vin hng dn: TS. L Anh Cng

PGS.TS. H Quang Thy

2008


2/56

1


3/56

i


4/56

LI CAM OAN

Ti xin cam oan y l cng trnh nghin cu ca bn thn. Cc s liu, kt qu trnh

by trong lun vn l trung thc v cha tng c ai cng b trong bt k cng trnhno trc y.

ii


5/56

LI CM N

Trc tin, ti xin gi li cm n chn thnh v sbit n su sc ti PGS.TS H Quang

Thu v TS L Anh Cng (Trng i hc Cng ngh, i hc Quc gia H Ni) tn tnh hng dn ti trong sut qu trnh thc hin kho lun ny.

Ti xin chn thnh cm n cc thy c gio ging dy ti trong sut cc nm ti theo

hc ti trng i hc cng ngh cho ti nhng kin thc qu bu ti c th vng

bc trn con ng i ca mnh.

Ti xin gi li cm n cc anh ch em trong nhm seminar v khai ph d liu phng

th nghim Cc h tch hp thng minh (SISLAB) trng i hc Cng ngh nhit

tnh ch bo trong qu trnh ti tham gia nghin cu khoa hc v thc hin lun vn ny.V li cui cng, ti xin by t lng chn thnh v bit n v hn ti cha m, v cc anh

ch ti, nhng ngi lun bn cnh ti nhng lc ti kh khn nht, gip ti vt qua

kh khn trong hc tp cng nhtrong cuc sng.

H Ni, ngy 30 thng 12 n m 2008

Sinh vin

Trn Th Oanh

iii


6/56

M UPhn on t(Word segmentation) v gn nhn tloi( Part-of-speech tagging POS

tagging) l hai bi ton c bit quan trng trong x l ngn ngtnhin. Tuy nhin,

cc nghin cu ti Vit Nam v hai vn ny vn cn giai on ban u. Do , nhu

cu l rt ln c v c s khoa hc v xy dng cng c thc hin sn dng. Thc t cho

thy hai qu trnh ny c lin quan vi nhau v nh hng n cht lng ca mt h

chung. V th, trong lun vn ny chng ti nghin cu M hnh tch t, gn nhn

tloi v hng tip cn tch hp cho ting Vit. ng gp ca lun vn l vic

tm hiu, nghin cu v xut m hnh thc hin tch t, gn nhn t loi POS ting

Vit; xy dng cng c thc hin lin quan; bn cnh hun luyn v kim thm

hnh chng ti cng tin hnh xy dng mt corpus ting Vit v tch tv gn nhn t

loi xp x 8000 cu. Lun vn cng tin hnh thc nghim mt hng tip cn tch hp

cho hai bi ton ny. Cc kt qu ny c s c bit hu ch cho cc nghin cu mc

cao hn nhdch my, tm tt vn bn, phn tch c php ...

Cu trc ca lun vn c trnh by nhsau:

Chng 1: Khi qut v tch t v gn nhn t loi ting Vit : Trong

chng ny, lun vn gii thiu hai bi ton c bn trong xl ngn ngtnhin

l bi ton tch tv bi ton gn nhn tloi ting Vit - cc hng tip cn cho

mi bi ton v tnh hnh nghin cu chung Vit Nam cng nh trn th gii.

Chng cng trnh by cc hng tip cn tch hp hai bi ton ny nng cao

hiu qu ca c hai m hnh c p dng thnh cng cho ting Trung.

Chng 2: M hnh tch t ting Vit : Chng ny nghin cu v xut

mt m hnh cho bi ton tch tting Vit tn dng thng tin ttri thc tnhiu

ngun khc nhau nhm lm tng chnh xc ca b tch t.

Chng 3: M hnh gn nhn t loi ting Vit: Chng ny nghin cu

v xut m hnh gn nhn tloi ting Vit, cc cng vic m lun vn tin

hnh nhm xy dng mt m hnh gn nhn POS hiu qu nhthit k corpus gn

iv


7/56

nhn t loi, xut m hnh sdng Maximum Entropy Markov Model (MEM)

v thit k cc tp c trng khc nhau tm ra cc c trng hu ch cho ting

Vit.

Chng 4: M hnh tch hp tch tv gn nhn t loi ting Vit: Tcc nghin cu trnh by trong chng 2 v chng 3 v c im ting Vit,

chng ny trnh by mt m hnh tch hp p dng cho ting Vit.

Kt lun: Phn ny tm tt li ni dung ca lun vn v nhng ng gp chnh

m lun vn thc hin.

Ph lc A: Mt s thut ngAnh - Vit : Mt s thut ngting Anh hay

dng v chvit tt.

Ph lc B: Ch gii tp t loi vnPOS : M t c th gii thch thm v

cc nhn tloi m lun vn xut xy dng corpus gn nhn tloi cho ting

Vit.

v


8/56

Mc lc

M u v

1 Khi qut v tch tv gn nhn tloi ting Vit 1

1.1 Khi qut v tch tting Vit . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Khi qut v gn nhn tloi - POS tagging . . . . . . . . . . . . . . . . . 2

1.2.1 Gii thiu v bi ton gn nhn tloi . . . . . . . . . . . . . . . . 2

1.2.2 Cc hng tip cn bi ton POS tagging . . . . . . . . . . . . . . 4

1.2.3 Cc nghin cu gn nhn tloi cho ting Vit . . . . . . . . . . . 6

1.3 Vn tch hp tch tv gn nhn tloi . . . . . . . . . . . . . . . . . 7

2 M hnh tch tting Vit 9

2.1 Cc m hnh lin quan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.1.1 M hnh da vo t in . . . . . . . . . . . . . . . . . . . . . . . . 9

2.1.2 M hnh nhn dng tn thc th - Named Entity Recognition . . . 10

2.1.3 M hnh N-gram . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.2 Phn tch cc m hnh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.3 Thit k tp c trng . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3.1 FS1: c trng trch tm hnh tch tda vo t in . . . . . . 13

2.3.2 FS2: c trng da vo m hnh nhn dng tn thc th . . . . . . 13

2.3.3 FS3: c trng da vo m hnh Ngram . . . . . . . . . . . . . . . 14

2.4 Kt qu thc nghim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.4.1 nh gi cc c trng FS1 v FS2 so vi cc m hnh trc . . 15

2.4.2 nh gi tm quan trng ca tng tp thuc tnh . . . . . . . . . . 16

vi


9/56

2.5 nh gi kt qu tch t . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3 M hnh gn nhn t loi ting Vit 19

3.1 Xy dng corpus gn nhn tloi cho ting Vit . . . . . . . . . . . . . . 19

3.1.1 Thit k tp th VnPOSTag . . . . . . . . . . . . . . . . . . . . . . 20

3.1.2 M t b dliu lm vnPOS corpus . . . . . . . . . . . . . . . . . 21

3.1.3 Xy dng vnPOS corpus . . . . . . . . . . . . . . . . . . . . . . . . 22

3.2 Gn nhn tloi bng phng php Maximum Entropy Markov Model . . 24

3.2.1 M hnh xc sut . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.2.2 Cc c trng ca POS tagging . . . . . . . . . . . . . . . . . . . . 25

3.3 xut m hnh gn nhn tloi cho ting Vit . . . . . . . . . . . . . . 263.3.1 Gn nhn tloi da vo thng tin t . . . . . . . . . . . . . . . . 27

3.3.2 Gn nhn tloi da vo m tit . . . . . . . . . . . . . . . . . . . 29

3.4 nh gi kt qu gn nhn tloi . . . . . . . . . . . . . . . . . . . . . . 30

4 M hnh tch hp tch tv gn nhn t loi ting Vit 32

4.1 La chn m hnh tch hp cho ting Vit. . . . . . . . . . . . . . . . . . . 32

4.2 Xy dng m hnh v tin hnh thc nghim . . . . . . . . . . . . . . . . . 334.2.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.2.2 Gii m . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.2.3 Kt qu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.3 Tho lun . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

A Mt s thut ngting Anh tng ng 40

B Ch gii tp tloi vnPOS 41

vii


10/56

Danh sch hnh v

2.1 Word segmentation using N-gram model. . . . . . . . . . . . . . . . . . . . 11

2.2 Cc m hnh lin quan cn trch cc c trng. . . . . . . . . . . . . . . 12

2.3 Biu o F1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.1 Giao din cng c tr gip gn nhn vnPOS. . . . . . . . . . . . . . . . . . 23

3.2 Kin trc gn nhn POS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.1 Kin trc tch hp tch tv gn nhn tloi ting Vit. . . . . . . . . . 32

4.2 Giao din cng c tch tting Vit . . . . . . . . . . . . . . . . . . . . . 35

4.3 Giao din cng c tch tting Vit . . . . . . . . . . . . . . . . . . . . . 36

viii


11/56

Danh sch bng

2.1 Table of feature sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.2 Example of feature set 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.3 V d mt cu c tch t . . . . . . . . . . . . . . . . . . . . . . . . . . 152.4 Kt qu nh gi hiu qu ca c trng da vo t in . . . . . . . . . . 16

2.5 Result to estimate the importance of NER-based features . . . . . . . . . . 16

2.6 Kt qu thc nghim khi b i ln lt tng tp c trng. . . . . . . . . 17

2.7 Kt qu thc nghim sdng tng loi c trng ring. . . . . . . . . . . 17

3.1 Tp th vnPOSTag ca tloi ting vit. . . . . . . . . . . . . . . . . . . . 22

3.2 Kt qu gn nhn POS da vo thng tin mc t . . . . . . . . . . . . . . 29

3.3 Kt qu gn nhn POS da vo thng tin m tit . . . . . . . . . . . . . . 31

4.1 Mt v d ouput ca m hnh tch hp. . . . . . . . . . . . . . . . . . . . . 33

4.2 Kt qu thc nghim tch hp WS v POS tagging. . . . . . . . . . . . . . 34

A.1 Bng thut ngAnh - Vit . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

ix


12/56

Chng 1

Khi qut v tch tv gn nhn t

loi ting Vit

1.1 Khi qut v tch tting Vit

Ting Vit l mt ngn ng n lp, khng bin hnh, cc k t c da trn h chci

Latin. Cng ging nhvi cc thting khc nhting Trung, ting Nht, ting Hn th

ttrong ting Vit khng c xc nh bi khong trng. Mt tting Vit c th c

to bi mt hoc nhiu hnh v v mi hnh v phn tch nhau bi cc khong trng. T

l n v c bn phn tch cu trc ca ngn ng, do vy tin ti nhng ng dng

xa hn v xl ngn ngting Vit nhgn nhn chc nng c php, phn tch c php

th vic u tin ta phi gii quyt bi ton phn on t.

Cc nh nghin cu xut mt s hng tip cn gii bi ton phn on t

[3, 5, 9, 10, 12, 13, 11] . Nhn chung, cc hng tip cn c chia thnh 2 loi: tip cn

da trn t in v tip cn da trn thng k. Hai phng php tiu biu ca hng tip

cn da vo t in l Longest Matching v Maximal Matching. Hu ht nhng nghin

cu khi thy v phn on t u da trn cch tip cn ny nhtrong [13][18]. Hng

tip cn ny c c im l n gin, d hiu tuy nhin hiu qu mang li khng cao.

L do l bi n khng x l c rt nhiu trng hp nhp nhng cng nhkhng c

kh nng pht hin tmi trong vn bn. Chnh v vy m cc h thng phn on t

c cht lng cao hin nay thng sdng hng tip cn da trn thng k. V d nh

i v

i ti

ng Trung th c cc nghin c

u lin quan nh

[9, 12], ti

ng Thi [13] c

ng nh

1


13/56

ting Vit [3, 8]. C th, i vi ting Vit th tc gi L An H[10] xy dng corpus

10M v sdng thng tin N-gram ti u tng cc xc sut phn on cho mi phn

cm (chunk). Kt qu thc nghim tuy khng cao nhng cng cho thy N-gram s

tr nn hu ch nu ta bit cch sdng thng tin ny khi lin kt vi cc ngun thngtin khc. Hin nay, rt nhiu h thng phn on t ph bin s dng hng tip cn

lai. V d, nhm tc gi Cm T [3] nghin cu ng dng cc m hnh CRF v SVM

phn on tting Vit. Hoc nhnhm tc gi inh in [8] sdng m hnh

MEM sdng gii thut ti u GIS hun luyn b phn on trn corpus gn nhn.

Trong nghin cu , tc gi phn tch hai qu trnh nhn dng t mi (unknown

word recognition) v phn on t bit (known word segmentation) nhhai tin trnh

c lp nhau. Tuy nhin, chng ti nhn thy rng hai tin trnh ny nn c tin hnhng thi nng cao chnh xc. Mt v d in hnh ca hng tip cn nhvy cho

ting Trung c J.Gao cp trong [9].

Trong cc phng php lai, cc tc gi tn dng thng tin t in v mt s thng tin

khc nhm pht hin tn thc th. Tuy nhin, trong cc nghin cu u cha quan

tm thch ng ti vic nh gi nh hng ca tng ngun tri thc v c bit l cha

c nhng nghin cu pht hin tmi (khng ch l tn thc th v cc dng factoid).

Nghin cu cc phng php pht hin tmi ta thy thng tin Ngram ng vai tr huch gip ta pht hin tmi khi m corpus thng k ln v xc nh c o ph

hp. Cu hi t ra l lm cch no tn dng c ttt c cc ngun tri thc .

y cng chnh l ng lc cho lun vn ny.

1.2 Khi qut v gn nhn tloi - POS tagging

1.2.1 Gii thiu v bi ton gn nhn tloi

Gn nhn t loi l mt cng vic quan trng v bt buc phi c i vi mi h x l

ngn ngtnhin. Cng vic gn nhn t loi cho mt vn bn l xc nh t loi ca

mi ttrong phm vi vn bn , tc l phn loi cc tthnh cc lp t loi da trn

thc tin hot ng ngn ngtrong :

Input: Mt chui cc tv tp nhn t loi (V d i vi ting Anh: Book that

flight., v tp th Penn Treebank)

2


14/56

Output: Mt nhn tt nht cho tng t trong cu (V d: Book/VB that/DT

flight/NN ./.)

Qu trnh gn nhn tloi c th chia lm 3 bc nhsau:

1. Giai on tin x l: Phn tch xu k t thnh chui cc t. Giai on ny

c th n gin hay phc tp tu theo ngn ngv quan nim v n v tvng.

Chng hn i vi ting Anh hay ting Php, vic phn tch tphn ln l da vo

cc k hiu trng. Tuy nhin vn c nhng t ghp hay nhng cm t gy tranh

ci v cch xl . Trong khi vi ting Vit th du trng cng khng phi l du

hiu xc nh ranh gii cc n v tvng do tn s xut hin tghp rt cao.

2. Khi to gn nhn: Tc l tm cho mi ttp tt c cc nhn t loi m n cth c. Tp nhn ny c th thu c t c s d liu t in hoc kho ng liu

gn nhn bng tay. i vi mt tmi cha xut hin trong c s ng liu th

c th dng mt nhn ngm nh hoc gn cho n tp tt c cc nhn. Trong cc

ngn ngbin i hnh thi ngi ta cng da vo hnh thi t on nhn lp

tloi tng ng ca t ang xt.

3. Quy

t

nh k

t qu

gn nhn:

l giai

o

n lo

i b

nh

p nh

ng, t

c l l

achn cho mi tmt nhn ph hp nht vi ngcnh trong tp nhn khi to ni

trn. C nhiu phng php thc hin vic ny, trong ngi ta phn bit ch

yu cc phng php da vo quy tc ngphp m i din ni bt l phng php

Brill v cc phng php xc sut. Ngoi ra cn c cc h thng s dng mng

n-ron, cc h thng lai sdng kt hp tnh ton xc sut v rng buc ngphp,

gn nhn nhiu tng, ...

Vic gn nhn t loi c quan tm t rt sm, cng vi n l s xut hin ca

rt nhiu phng php gii quyt. Ti nay, cc phng php mi vn ang tip tc c

nghin cu nhm hon thin hn na cc kt qu t c.

Hin nay, bi ton gn nhn t loi cho ting Anh c gii quyt kh tt, t kt

qu rt kh quan. Bn cnh vic hon thin hn na cc b gn nhn c, ngy cng

nhiu b gn nhn mi ra i, em li kt qu gn nh ti u. Tuy nhin, i vi cc

ngn ngkhc, c bit l cc ngn ngtng hnh (nhting Trung Quc, Nht, Hn

Quc ..), cc ngn ngca n , Thi Lan, A Rp, Nga cng nh i vi ting Vit

3


15/56

th bi ton gn nhn t loi vn cn l mt thch thc ln. Cc phng php v cng

c c xy dng gn nhhon thin cho Ting Anh khi em p dng cho cc ngn

ngkhc loi trn thng a li kt qu thp. Nhvy, yu cu t ra vi tng ngn

ngl phi k tha, tn dng c cc phng php sn c, tin hnh hiu chnh hoc l xut ra cc hng tip cn mi sao cho ph hp vi cc c im ring bit ca ngn

ngmnh.

1.2.2 Cc hng tip cn bi ton POS tagging

Theo [4], hu ht cc thut ton gn nhn t loi ri vo mt trong hai lp: gn nhn

da trn lut (rule-based) hoc b gn nhn xc sut (stochastic taggers).

Cc b gn nhn da trn lut thng lin quan ti mt c s dliu ln cc lut c

vit bng tay. V d mt tnhp nhng ang xt c xu hng l mt danh thn l mt

ng tnu n i sau mt tch nh. Phn tip sau s m t mt b gn nhn da trn

lut mu, ENGTWOL, da trn kin trc c php rng buc ca Karlson nm 1995.

B gn nhn xc sut thng gii quyt nhp nhng bng cch sdng mt corpus hun

luyn tnh ton xc sut ca mt t cho sn s c gn mt th no trong ng

cnh cho trc. Phn sau s m t mt b gn nhn HMM (HMM Tagger), hay cn c

gi l Maximum Likelihood Tagger, hoc mt b gn nhn Markov Model, cng da trn

m hnh Markov n.

Ngoi ra cn c cc hng tip cn khc gm b gn nhn da trn bin i transformation-

based tagger hoc b gn nhn Brill (Brill tagger). B gn nhn Brill s sdng cc c

tnh ca c 2 kin trc gn nhn trn. Ging nhb gn nhn da trn lut, n da vo

lut xc nh khi mt tnhp nhng th n c kh nng l mt th no nht. Ging

nhb gn nhn xc sut, n c mt thnh phn hc my to ra cc lut mt cch t

ng tmt corpus hun luyn c gn nhn trc. Tuy nhin, trong phm vi lun

vn ny chng ti khng trnh by c th vic nghin cu 2 phng php ny (xem thm

trong [4]).

Gn nhn chc nng c php da trn lut

Cc thut ton khi thy gn nhn t ng tloi thng gm hai giai on. Giai on

mt n sdng mt t in gn cho mi tmt danh sch cc tloi c th c. Giai

4


16/56

on 2 n sdng mt danh sch gm tp cc lut khng c nhp nhng thng c

son bng tay gn cho mi t ch mt t loi ph hp nht. M b gn nhn in

hnh p dng cho ting Anh l b gn nhn ENGTWOL[4].

Gn nhn t loi xc sut

Phn ny trnh by mt b gn nhn xc sut in hnh sdng m hnh Markov n.

Thut ton ny la chn chui nhn tt nht cho ton b cu. V thng thng ngi ta

hay sdng thut ton Viterbi tm chui th tt nht . Gi svi cu u vo l

W ta cn tm mt chui th T=t1,...,tn tha mn cng thc 1.1:

bT = argmaxT2P(T|W) (1.1)Sdng lut Bayes, P(T|W) c vit theo cng thc 1.2

P(T|W) =P(T)P(W|T)

P(W)(1.2)

Ta ang quan tm ti tm chui th ph hp nht lm cc i cng th3.1 nn mu

s trong tt c cc trng hp l ging nhau. Do vy, bi ton tr thnh tm chui th

tha mn cng thc 1.3bT = argmaxT2P(T)P(W|T) (1.3)

p dng lut chui xc sut ta c cng thc 1.4:

P(T)P(W|T) =nYi=1

P(wi|w1t1...wi1ti1ti)P(ti|w1t1...wi1ti1) (1.4)

Vn khng c phng php hiu qu tnh ton xc xut ca chui ny mt cch chnhxc, n yu cu qu nhiu d liu. Tuy nhin, xc sut c th c xp x bi mt xc

sut n gin hn bng cch p dng cc gi thit c lp iu kin. Mc d cc gi thit

ny l khng thc t nhng trong thc hnh th vic nh gi l vn hp l . y,

ta sdng gi thit N-gram m hnh ha xc sut chui t. C th ta dng m hnh

ph bin nht l m hnh tri-gram. u tin, ta lm n gin ha rng xc sut ca mt

tth ch ph thuc vo th ca n (xem cng thc 1.5):

5


17/56

P(wi|w1t1...wi1ti1ti) = P(wi|ti) (1.5)

Tip n, ta gi thit rng cc th pha trc c th c xp x bi 2 th gn n nht

(xem cng thc 1.6):

P(ti|w1t1...wi1ti1) = P(ti|ti2ti1) (1.6)

V vy cui cng ta la chn chui th lm cc i cng thc 1.7:

P(t1)P(t2|t1)nYi=3

P(ti|ti2ti1)[nYi=1

P(wi|ti)] (1.7)

Cc thnh phn tha s trong cng thc 1.7 c th c tnh ton tcorpus hun luyn

ca m hnh. Ch rng c th trnh xc sut bng 0 ta cn sdng cc k thut

lm trn.

1.2.3 Cc nghin cu gn nhn tloi cho ting Vit

i vi ting Anh th bi ton ny gn nh c gii quyt xong t chnh xc rt

cao ln ti >96% [1]. Tuy nhin, i vi cc vn bn Vit ng, vic gn nhn t loi cnhiu kh khn, c bit l bn thn vic phn loi tting Vit cho n nay vn l mt

vn cn nhiu tranh ci, cha c mt chun mc thng nht.

Hin nay c mt s nghin cu gn nhn t loi cho ting Vit v t c mt

s thnh tu nht nh. in hnh l b gn nhn t loi xc sut vnQTAG ca nhm

tc gi Nguyn Th Minh Huyn [16]. fi tng ca phng php l xc nh phn b xc

sut trong khng gian kt hp gia dy cc tSw v dy cc nhn tloi St. Sau khi c phn b xc sut ny, bi ton loi b nhp nhng t loi cho mt dy cc t c

a v bi ton la chn mt dy t loi sao cho xc sut iu kin P(St|Sw) kt hp

dy tloi vi dy t cho t gi tr ln nht. Nhm tc gi Nguyn Quang Chu

[15] trnh by mt hng tip cn cho bi ton gn nhn tloi trong vn bn ting Vit

trn c s vn dng cc m hnh thng k da vo kho ngliu, t in, c php v ng

cnh. Ngoi ra cn mt hng tip cn khc s dng kho ng liu song ng Anh-Vit

[6]. Cc hng tip cn ny c u im l tn dng c cc cng c pht trin gn

6


18/56

nhhon thin dng cho ting Anh tuy nhin nhc im ca n l: Do skhc nhau v

hnh thi gia ting Anh v ting Vit nn php chiu trc tip khng n gin l php

chiu 1-1 m thng l php chiu phc tp m-n. Skhc nhau v c bn gia hai ngn

ngny l nguyn nhn ca khng t nhp nhng cn phi gii quyt, n cn c th timtng nhiu trng hp m nhm tc gi cng nhcc c gi cha tnh ti.

1.3 Vn tch hp tch tv gn nhn tloi

Vit Nam cha c mt cng trnh nghin cu no v tch hp hai bi ton rt quan

trng trong xl ngn ngtnhin l bi ton tch tv bi ton gn nhn tloi ting

Vit. Cc nghin cu ny ch yu mi c nghin cu cho ting Trung nh[22][14][20]. fitng ca phng php tch hp l c th kt hp hai tin trnh li vi nhau nhm nng

cao hiu qu ca chng. Cc hng tch hp c th chia lm 2 loi: Mt l loi tch hp

gi pseudo-integration v mt loi l tch hp thc strue-integration.

Hng tch hp gi: [19] m t mt phng php gm 3 bc chnh:

1. To ra N chui tch ttt nht(N-best word sequences) i vi mt cu cho sn.

2. Thc hin gn nhn POS cho mi chui t , sau chn ra N chui th POS ttnht tng ng.

3. Sdng nh gi c trng s ca (1) v (2) chn gii php tch tv gn nhn

POS tt nht cho cu u vo .

Trong h thng ny, vic gii m cho tch t v gn POS vn c thc hin ring r,

v s suy lun chnh xc cho c hai l iu c th. Tuy nhin, s tng tc gia POS

v segmentation b hn ch bi reranking: thng tin POS c sdng ci tin chtlng phn on i vi ch N segmentor output.

Hng tch hp thc stiu biu trong hai cng trnh [19, 14] Trong [19] cc tc gi

xut mt phng php da trn CRFs hai tng sdng gii m ng thi tch tv

gn POS. Trong phng php ny, tc gi m hnh bi ton tch t v gn nhn bng

mt CRFs hai tng. Lc gii m, u tin thc hin gii m ring mi tng. Sau ,

mt khung xc sut c xy dng tm ra gii m kt hp tt nht cho c hai bi ton.

7


19/56

Cn khi hun luyn, tc gi hun luyn mt ln cc CRF ring cho hai bi ton, i

vi phm vi ng dng ny th hun luyn ng thi s tn cng hn. Kt qu nh gi

tch tv POS tag thu c kt qu state-of-the-art trn c tp PCT v First SIGHAN

Bakeoff datasets. Trong c hai bi ton, phng php xut ci tin so vi phng phpbaseline khng thc hin gii m ng thi.

Trong [14] trnh by mt nghin cu tch hp kh cng phu. xy dng mt b gn

nhn POS, c hai cu hi c t ra:

1. Thc hin gn nhn sau khi tch ttheo hai pha ring bit (one-at-a-time), hoc

thc hin lin kt gn nhn tloi v tch tthnh mt bc n ng thi nhau

(all-at-one approach).

2. Gn th POS da trn nn tng t(ging English), tn dng cc c trng mc t

ca ngcnh (word-based), hoc da trn nn tng k tvi cc c trng ca k

t(character-based)?

Bi bo trnh by mt nghin cu t m v kin trc x l v biu din c trng cho

gn POS ting Trung vi khung Maximum Entropy. H phn tch hiu qu ca tng tip

cn nhm tm ra hng tip cn ph hp nht. Kt qu thc nghim cho thy tip cn

character-based tt hn so vi tip cn da trn word-based i vi bi ton POS tag l

khng c g ng ngc nhin. Khc vi English m mi English letter khng c ngha, th

nhiu character ting Trung li mang ngha. Hn na, t l OOV i vi Chinese words

th cao hn so vi Chinese characters, i vi unknown words, vic sdng cc character

thnh phn trong tgip d on chnh xc nhn POS l mt heuristic tt. Tip cn

all-at-once xem xt tt c cc kha cnh ca thng tin sn c theo mt khung tch hp

ng nht cho kt qu tt hn nhng cng yu cu chi ph tnh ton cao hn. Tuy nhin,

im bt li ca phng php ny l skh khn khi tch hp ton b thng tin v t

vo vic gn POS. V d, c trng chun word + POS tag s khng th ng dng r

rng c.

8


20/56

Chng 2

M hnh tch tting Vit

Trong lun vn ny, chng ti chn m hnh maximum entropy lm phng php hcmy trong cc c trng ca m hnh c la chn da trn nhng nghin cu v tri

thc ca cc m hnh khc v cc c im ca ngn ngting Vit. C th, chng ti s

dng thng tin c c tba ngun l m hnh phn on tda vo t in, m hnh

N-gram v m hnh nhn dng thc th. Chng ti cng lm nhng thc nghim nh

gi tnh hiu qu ca h thng da trn tp dliu gn nhn. Bn cnh , chng ti

nh gi nh hng ca tng ngun tri thc i vi m hnh cui cng. Trong thc

nghim, chng ti ly phng php Longest Matching lm m hnh c s (baseline) so

snh.

2.1 Cc m hnh lin quan

2.1.1 M hnh da vo t in

Hai phng php kinh in ca hng tip cn da trn t in l: Longest Matching(LM) v Maximal Matching (MM).

Phng php LM duyt cu u vo tun t t tri qua phi v chn t di nht nu

t c trong t in. R rng l phng php ny rt n gin nhng b phn lp sai

trong nhiu trng hp nhp nhng. V d cu l cch truyn thng tin, nu p

dng phng php LM th cu ny s b phn tch sai thnh l cchtruyn_thng

tin.

Phng php MM s to ra tt c cc phn on c th cho mt cu bt k , sau cu

9


21/56

c phn on ng c chn l cu cha t t nht. Ging nh phng php trn

phng php ny cng c yu im l khng th a ra phn on ng trong trng

hp nhiu kt qu phn on li cha cng mt s lng tt nht. V d cu Hc sinh

hc sinh hc c hai ng c l Hc_sinh hc sinh_hc v Hc sinh_hc sinh_hc.Trong nhng trng hp ny, ta cn p dng cc phng php hc my trn mt c s

dliu ln xc nh c phn on ng.

2.1.2 M hnh nhn dng tn thc th - Named Entity Recog-

nition

Bi ton nhn dng tn thc th l bi ton gn nhn mi t trong vn bn vo mt

trong cc lp c nh ngha trc nhtn ngi, tn a danh, tn t chc, ngy thng,

s, tin t, ... Mt v d l:

[PERSON ng Nguyn H u Minh] c c ch c tng gim c ca [ORG Cng ty

i ] nhim k [DTIME2002-2006]."

Nhiu phng php hc my c p dng thnh cng cho bi ton nhn dng ny,

trong cc phng php ch yu da vo cc c trng ngn ngv thng tin ngcnh

c

a t

xc

nh l

p cho m

i t

. V d

, Tri Tran Q. [21]

nghin c

u s

d

ng SVM gii bi ton ny v kt qu t c l kh quan. Hoc nhJ.Gao cng ng tc gi

[9] xut mt khung ton hc thc hnh va thc hin phn on cc t bit

cng nhpht hin tmi. Nhng nghin cu nhvy ch ra rng bi ton nhn dng

thc th c mt mi lin h gn gi vi bi ton phn on t.

2.1.3 M hnh N-gram

M hnh ngn ngN-gram th hin kh tt mi quan h ngcnh ca t. Trong m hnh

, mi t c coi nhph thuc xc sut vo n-1 ttrc n.

P(W) = P(w1w2...wn) =nYi=1

P(wi|win+1...wi1). (2.1)

M hnh N-gram c ng dng phn on t trong vi mi cu th phn on

tt nht theo m hnh ny l phn on c xc sut P(W) c tnh theo cng thc 2.1

l ln nht. Trong , cc xut sut v sph thuc ca mt tvo n ttrc c

10


22/56

thng k da trn mt corpus ln. Ty vo gi thit v tnh ph thuc m ta c cc

m hnh 2-gram hoc 3-gram tng ng. Phng php ny l mt trong nhng phng

php thng k chnh gii bi ton phn on tkhi khng c thng tin t in v d

liu gn nhn. M hnh phn on tsdng N-gram c biu din nhhnh bn di(hnh 2.1). Khi p dng phng php ny i hi chng ta phi xc nh mt o tt

Tagged Corpus

Statistical Process(n-gram)

Optimization ProcessInput sentence Segmented sentence

Hnh 2.1: Word segmentation using N-gram model.

ph hp vi bi ton nh gi kh nng mi cm hnh v c l mt t hay khng?

C rt nhiu o c th sdng nh: n gin ch sdng thng tin v tn sut xut

hin ca t, hoc c th sdng thng tin mutual information hoc t-score, ... V d,

Maosong v cc ng tc gi [12] s dng o mutual information v t-scores v

mt s k thut khc xc nh t cho ting Trung v thu c kt qu kh cao(>90%). i vi ting Vit tc gi L An H[10] n gin sdng tn sut N-gram

ti u xc sut ca mi chunk. Kt qu thc nghim tuy khng cao nhng chng t

rng N-gram l mt phng php ph hp c th ng dng cho bi ton phn on t

ting Vit ni ring.

2.2 Phn tch cc m hnh

Hiu qu ca cc phng php da trn t in nh Longest Matching v Maximal

Matching ph thuc phn ln vo bao ph ca t in. Tuy nhin, trn thc t khng

tn ti mt t in hon thin no c kh nng bao ph ht cc mc t ca mt ngn

ngbt k bi v nhng tmi lun lun xut hin. Theo thng k, corpus SIGHANs

PK c xp x 30% OOVs [9]. Corpus ting Vit m chng ti chn nh gi m hnh

cng cha 11.6% OOVs[3]. y l nhng t l kh cao.

11


23/56

OOVs thng c hai loi: Mt l cc dng tn thc th hoc dng factoid; Hai l

nhng t mi khng thuc loi 1. nng cao cht lng phn on t th cc gii

php cn kt hp thng tin t in v cc k thut pht hin tmi. Xem xt dng

1, chng ti thy factoid c th c nhn din d dng nh dng biu thc chnh qui.Tuy nhin, cc NE khng d nhn din nh vy m cn nghin cu cc phng php

pht hin thc th v c im ring ca cc NE ting Vit. Kt qu ca nhng nghin

cu ny s c trnh by r hn trong phn la chn c trng pht hin NE s

dng m hnh MEM. Cn cc t mi thuc loi 2 thng l nhng thut ng chuyn

ngnh, t nc ngoi c Vit ha, ... Vi nhng t ny th khng c qui tc ring

no pht hin m cch thng c sdng nht l thng k tn sut t. Nu t

c dng trn mt ngng no th ta c th coi l mt t. Do vy, chng ti ssdng thng tin N-gram nh gi kh nng mt cm hnh v c phi l thay khng?.

T nhng thng tin lin quan , chng ti trch c trng cho m hnh Maximum

Entropy Markov Model hun luyn b phn lp. C th cc m hnh gm: m hnh

da trn t in, m hnh nhn din thc th, m hnh N-gram v mt s ngun dliu

khc (xem biu din hnh di y).

Documents

Preprocessing

Extract features

for each chunks

Additionalresources

N-graminformation

NERinformation

Dictionary

Hnh 2.2: Cc m hnh lin quan cn trch cc c trng.

2.3 Thit k tp c trng

Da trn cc phn tch trn, chng ti a ra thit k chi tit cc c trng chia ra lm

3 tp nhsau:

12


24/56

Bng 2.1: Table of feature setsNo Model Type of Features Detailed Features

FS1 Tch da vo t in Slin kt m tit SC Mi SC c phi l entry ca t in?

FS2NER model

External Resource Mi SC c phi l valid Name?

- Dictionary In Location List?- Name List Is-Regular-Expression(0,0)- Location List Is-Initial-Capitalization(0,0)

Is_All_Capitalization(0,0)Is_First_Observation(0,0)Is_Marks(0,0))

Factoid Is_RegexFS3 N-gram Model N-gram information The log of probability (2-gram, 3-gram)

Bng 2.2: Example of feature set 1Syllable Features set 1

... ... ...

thoi

SC(-3,0) In_dictionary: 0SC(-2,0) In_dictionary: 0SC(-1,0) In_dictionary: 1SC(0,0) In_dictionary: 0

... ... ...

2.3.1 FS1: c trng trch tm hnh tch tda vo t in

Khc vi cc tip cn trc [3][5][8], thay v sdng thng tin ca cc m tit trc v

sau m tit hin ti, chng ti ch s dng thng tin ca cc m tit ng trc. y

cng l tng to t ng ctrong phng php Longest Matching. Xt v d cu Th

tr ng in thoi di ng ang rt nng , gi sta trch c trng cho m tit hin ti

thoi th cc c trng thuc tp FS1 gm c cc c trng c m t trong bng 2.2.

2.3.2 FS2: c trng da vo m hnh nhn dng tn thc th

Nh tho lun phn trn, cc dng factoid s c nhn bit nh sdng biu thc

chnh qui. Do , trong tp c trng ny s c mt c trng isRegex nhn bit cc

dng nhngy thng, thi gian, tin t, s, email, s in thoi, fax v a ch web.

nhn dng tn ngi ta s da vo mt danh sch tn ting Vit gm khong 21.000 tn.

Tdanh sch v c im tn ting Vit ta nhn thy tn ngi hp l thng tun theo

qui tc:

Tn ng i hp l = H + Tn m + Tn

13


25/56

Do vy, da vo danh sch ta lit k 3 tp danh sch tng ng gm: danh sch cha

cc h, danh sch cha cc loi tn m v danh sch cc tn ring. V nhn bit tn

ring th tng ng vi mi cm lin kt hnh v trong phn FS1, ta s c thm mt c

trng tng ng kim tra xem cm c phi l mt tn hp l trong ting Vit haykhng da vo qui tc trn. c trng ny cng nhn gi tr:

1 nu SC tun theo lut

0 nu ngc li

Mt dng tn thc th na c xt y l tn a danh hoc tn ca cc cng ty.

pht hin cc thc th thuc loi ny ta s da vo mt danh sch a danh gm khong

800 tn. Tng ng vi mi lin kt hnh v ta s c mt c trng nhn gi tr:

1 nu SC c trong danh sch a danh

0 nu ngc li

Mt im cn lu l: Cc tn thc th c xt thng c k t u tin ca mi hnh

v c vit hoa. Do , cc hnh v u mi cu rt d b nhm vi tn thc th.

trnh nhm ln ny ta cn thm mt c trng na l Is_First_Observation(0,0) nhngi tr 1 nu hnh v ny ng u cu v 0 nu ngc li.

2.3.3 FS3: c trng da vo m hnh Ngram

Cc m hnh phn on tda vo N-gram sdng xc sut ca tng n-gram nhmt

n v thng tin c s. Cc xc sut ny c tnh da vo thng k corpus ln c

bao ph hnh v v bao ph t tin cy. Khi sdng N-gram phn on ttc

gi xy dng corpus 10M hnh v, cn trong nghin cu ny chng ti thu thp 14M

corpus twww.wikipedia.com . Chng ti thng k xc sut mc 2-gram v 3-gram. V

do corpus cha ln lm th nn mt s cm hnh v c tn sut xut hin nh. Th nn,

khi sdng thng tin xc sut cc n-gram ny chng ti khng sdng trc tip nhng

xc sut m s nh x chuyn chng v on [0,1] theo cc cng thc 2.2 v2.3.

mi = Log(P(N gram)) = Log(f) Log(14000000). (2.2)

14


26/56

Bng 2.3: V d mt cu c tch tTh trng chng khon ang i xungB_W I_W B_W I_W B_W B_W B_W

The market stock being go down

Info(N gram) = (1 |mi + |max_N gram||

|min_N gram|). (2.3)

Theo thng k tcorpus th( 14M-syllable Wiki), ta c:

P(2-gram) : min_2-gram 41, max_2-gram 8.00

P(3-gram) : min_3-gram 41, max_3-gram 10.00

2.4 Kt qu thc nghim

M hnh c sdng l m hnh maximum entropy [1] vi gii thut ti u BLMVM [2]

c h tr gi tr l s thc. Khi sdng m hnh ny, bi ton phn on tting Vit

c chuyn v bi ton phn lp trong mi m tit s c phn v mt trong hai lp

l B_W (Begin of word) hoc I_W (inner of word). Mt v d cu phn on c cho

trong bng 2.3: Cng c MEM c dng trong cc thc nghim c ly thttp://www-

tsujii.is.s.u-tokyo.ac.jp/ tsuruoka/maxent/. V corpus, chng ti thc nghim trn corpus

c cng b trong bi bo [3] ti a ch http://www.jaist.ac.jp/ hieuxuan/vnwordseg/data.

Corpus dng thng k thng tin N-gram c ly ttrang wikipedia.

2.4.1 nh gi cc c trng FS1 v FS2 so vi cc m hnh

trc

Cc nghin cu trc cng thit k cc c trng da trn t in v m hnh NER, tuy

nhin cc c trng c thit k y khc so vi cc xut trong [8][9]. Kt qu thc

nghim di y s so snh v nh gi tnh ph hp ca cch chn c trng ny. i

vi cc xt thng tin da vo t in, chng ti tin hnh thc nghim v kt qu

cho thy cch tip cn ca m hnh ny cho kt qu cao hn cch tip cn trc (xem

bng 2.4):

Vi cc c trng da vo NER, thay v kim tra tng m tit c trong danh sch tn

m, tn h, tn hay khng (nhtip cn trnh by trong [3]), chng ti s kim tra tng

15


27/56

Bng 2.4: Kt qu nh gi hiu qu ca c trng da vo t inNo Features da trn Precision Recall F1 measure

1 Left and right syllables 94.03 93.64 93.842 Only left syllables 94.95 94.2 94.58

Bng 2.5: Result to estimate the importance of NER-based featuresNo Features based on Precision Recall F1 measure

1 Old 94.92 94.22 94.572 Our approach 95.15 94.43 94.79

lin kt m tit trong FS1 c phi l mt tn hp l? Kt qu thc nghim trong bng

2.5 chng minh c trng ny hiu qu hn hn. L do c gii thch l: do ting Vit

c c im l tn h, tn m v tn ring c th trng nhau nn khi sdng kim traring r nh[3] th s gy nhm ln v dn ti d on sai.

2.4.2 nh gi tm quan trng ca tng tp thuc tnh

Lun vn cng trnh by kt qu thc nghim nh gi nh hng ca tng tp c trng

ti hiu qu phn on cng nhchng minh tnh hiu qu ca m hnh cui cng.

nh gi chng ti sdng 3 o l: chnh xc, hi tng v o F1 trn 5-fold

test. Cc kt qu c ch ra trong bng 2.6 v3.1. Khi so snh chng ti ly phng

php Longest Matching lm c s. nh gi tc ng ca tng thuc tnh ti m hnh

cui cng chng ti thit k hai loi thc nghim s dng cc lin kt c trng khc

nhau cho m hnh MEM. Trong loi u tin, ta s ln lt b i tng c trng mt

vi kt qu thc nghim cho trong bng 6. Loi thc nghim th2 ta s thc nghim vi

tng tp c trng mt v kt qu c cho trong bng 3.1.

Nhn vo kt qu thc nghim ta d dng nhn thy rng tp c trng da trn t

in c nh hng ln nht: Nu ch sdng t in th o F1 l 94.58%, cn nu t

in i th kt qu l t nht so vi thc nghim b i mi c trng khc ( o F1 l

87.5%). iu ny d gii thch bi t in cha mt lng tkh n nh v thng tin

tl chun xc. c trng c tm quan trng th2 l cc c trng gip pht hin Nes

(ta c F1 l 93.55% nu b c trng ny i v F1 l 91.32% nu ch sdng NEs). c

trng c nh hng t nht l c trng ca m hnh N-gram.

Khi cc c trng c kt hp vi nhau th kt qu thu c l cao nht: 95.30%

16


28/56

Bng 2.6: Kt qu thc nghim khi b i ln lt tng tp c trng.No Not use (Reject) Method Precision Recall F1 measure

1 - Longest Matching 81.07 87.97 84.522 Dict.-based feature set MEM 96.99 77.1 87.05

3 NER-based feature set MEM 97.21 89.88 93.554 N-gram-based feature set MEM 95.15 94.43 94.795 MEM 96.71 93.89 95.30

Bng 2.7: Kt qu thc nghim sdng tng loi c trng ring.No Only use Method Precision Recall F1 measure

1 - Longest Matching 81.07 87.97 84.522 Dict.-based feature set MEM 94.95 94.2 94.583 NER-based feature set MEM 90.89 91.74 91.324 N-gram-based feature set MEM 97.98 60.5 79.24

5 All MEM 96.71 93.89 95.30

o F1. iu chng t rng cc tri thc v ngn ngv ngcnh ca t c cung

cp cng nhiu th cht lng phn on ca gii php xut cng chnh xc. Mt biu

din trc quan ca o F1 cho hai loi thc nghim c trnh by trong hnh 2.3.

2.5 nh gi kt qu tch t

Lun vn trnh by mt hng tip cn mi cho bi ton phn on tting Vit trong

m hnh c chn l m hnh Maximum Entropy Markov Model vi gii thut ti u

BLMVM c h tr gi tr thc. Lun vn kt hp rt nhiu c trng hu ch tcc

m hnh khc gm: m hnh phn on tda vo t in, m hnh nhn dng tn thc

th v m hnh N-gram. Khc vi cc tip cn trc [3, 8], lun vn nghin cu cch trch

chn c trng hu ch hn tcc m hnh da vo t in v m hnh nhn dng tn

thc th. Mt im na khc vi cc nghin cu trc l chng ti dng thm thngtin N-gram nhm pht hin thm cc tmi.

Kt qu thc nghim ch ra rng m hnh sdng c 3 loi tp c trng ni trn lm

tng ng k cht lng phn on (95.30% o F1). Thc nghim cng nh gi tm

quan trng ca tng loi thuc tnh i vi m hnh phn on v kt qu cho thy c

trng c nh hng ln nht l ca m hnh da vo t in, tip l m hnh NE v

m hnh N-gram c nh hng t nht.

nh gi kh nng tch tca m hnh so vi cc m hnh tt nht hin nay, chng ti

17


29/56

Hnh 2.3: Biu o F1

tin hnh so snh trn cng corpus, kt qu tch t t chnh xc state of the art vi

95.30% F1 (cao hn tip cn trong [3]). Cng vi m hnh , chng ti cng lm cc thc

nghim kim thtrn corpus ca Trung tm t in hc Vit Nam www.vietlex.com.vn

v o o F1 t 94.76% (>94.44% nhbo co trong [8]).

18


30/56

Chng 3

M hnh gn nhn tloi ting Vit

Trc khi xy dng v kim thm hnh gn nhn t loi, chng ti tin hnh xydng tp th t loi sau gn nhn corpus t loi ting Vit 8000 cu. Xut pht t

thnh cng ca m hnh Maximum Entropy Markov Model (MEM) c p dng cho

ting Anh, ting Trung, ... lun vn cng xut xy dng m hnh gn nhn t loi

ting Vit da trn m hnh . Vi m hnh la chn ny, lun vn tin hnh nghin cu

v thnghim cc c trng khc nhau nhm tm ra tp c trng hu ch i vi ting

Vit.

3.1 Xy dng corpus gn nhn tloi cho ting Vit

Xy dng corpus l mt cng vic rt tn thi gian v cng sc. Trong lun vn ny,

chng ti c gng xy dng mt corpus ting Vit dng cho cc ng dng v sau.

Corpus ny c xy dng da vo corpus tch t ca nhm tc gi Cm T c

cng b trong [3]. xy dng corpus, chng ti thc hin cc cng vic sau:

Thit k b VnPOS tag cho ting Vit gm 14 nhn t v >10 nhn k hiu

(symbols).

Xy dng tool tr gip gn nhn POS cho vn bn sau khi tch t theo ng

nh dng v ti liu i km.

Gn nhn POS cho 8000 cu thuc nhiu lnh vc khc nhau.

19


31/56

3.1.1 Thit k tp th VnPOSTag

Ch xt ring i vi ting Anh tn ti rt nhiu tp th POS khc nhau in hnh

(theo [4] ) l:

Brown corpus: 87 nhn

Penn Treebank: 45 nhn

Lancaster UCREL C5: 61 nhn

Chn tp nhn ln s lm tng kh nhng tp nhn nh hn c th khng p ng

cho mt mc ch nht nh no . Vic chn tp nhn no s ty thuc vo tng ng

dng c th, ni cch khc l ty thuc vo s lng thng tin m ng dng i hi.Do , cn phi c scn i gia:

C c lng thng tin r rng hn (Tc l phm vi phn lp t loi nh hn,

chia thnh nhiu tloi hn da trn nhiu yu t th hin skhc bit)

C kh nng tin hnh thc hin vic gn nhn (Tc l s lng cc tloi cng t

cng d tin hnh)

Tc l cn phi c mt stho hip t c mt b ch thch tloi khng qu

ln v c cht lng. Vi ting Vit th vic thit k tp th POS cng kh khn hn bi

ngay trong ting Vit th tloi vn cn l vn gy nhiu tranh ci. Da theo mt s

ti liu tham kho v c php ting Vit th cc mc ttrong ting Vit nhn chung c

phn chia thnh cc nhm, mi nhm li c phn chia su hn ty loi [7]. Theo Dip

Quang Ban [7], vic tp hp v quy loi cc tthng c ba tiu chun phn loi sau:

1. Tiu chun 1: ngha khi qut. Cc t loi l nh ng nhm t rt to ln v khi

l ng m mi nhm c mtc tr ng phn loi: tnh vt th, phm cht, hnhng

hoc trng thi,v.v. . . V d , nh ng t nh : nh, bn, hc sinh, con, quyn, s v.v...

c phn vo lp danh t , vngha t v ng ca chngc khi qut ha v tr u

t ng ha thnh ngha th c th - ngha phm tr ng php ca danh t .

2. Tiu chun 2: khn ng kt hp. Vi ngha khi qut, cc t c thc khn ng

tham gia vo mt kt hp c ngha. mi vtr ca kt hp c thxut hin nh ng

20


32/56

t c khn ng ln l t thay thnhau, trong khi , cc v tr khc trong kt hp,

cc t cn li to ra bi cnh cho s xut hin kh n ng thay thca nh ng t ni

trn. Nh ng t cng xut hin trong cng mt bi cnh, c khn ng thay th nhau

cng mt vtr, c tnh cht th ng xuyn, c tp hp vo mt lp t . Vn d ngvo ting Vit, nh ng t : nh, bn, ct, v.v... c thxut hin v thay thnhau

trong kt hp kiu: nh ny, bn ny, ct ny, ny, v.v... v c xp vo lp

danh t . Chng khng th xut hin v thay th cho nhau trong kt hp kiu : hy

n, hy mua, n xong, mua xong v.v ..., vn l kiu kt hp ca lp ng t .

3. Tiu chun 3: ch c n ng c php. Tham gia vo cu to cu, cc t c th ng

mt hay mt s v tr nht nh trong cu, hoc c th thay th nhau v tr ,

v cng biu th mt mi quan h v ch c n ng c php vi cc thnh phn khc

trong cu to cu, c th phn vo mt t loi. V d , cc t : nh, bn, ct, ...

c th ng nhiu v tr trong cu. Chng c th thay thnhau nh ng v tr,

v c quan hvch c n ng ging nhau vi cc thnh phn khc trong cu mi v

tr, nh ng th ngvtr chng trong quan hvi vng (l hai ch c n ng cbn

trong cu to cu). Ch c n ng ch ng l ch c n ng c php chyu phn loi

cc t ni trn vo lp danh t . Cn ch c n ng vng li l ch c n ng c php ch

yu ca ccng t (v tnh t ), v.v ...

Trong , tiu chun (2) v (3) lm trng tm trong sphn nh cc tp th. Vic

xc nh tp th ty thuc vo tng loi ng dng xem cn thng tin c php tvng

ti mc no m c nhiu cch phn chia th, mn khc nhau. thun tin cho vic lm

corpus v phc v mt s ng dng nht nh trong nghin cu ca nhm nh(Question

Answering System, Text Summarization, . . . ), chng ti xc nh tp th vnTagSet mc

th gm cc th c lit k nh bng 3.1 vi ngha mi loi xin xem thm phnph lc B.

3.1.2 M t b dliu lm vnPOS corpus

B d liu dng xy dng corpus t loi ting Vit chnh l b d liu c sdng

trong phn tch t trnh by trn. Kch c ca corpus c 8000 cu c thu thp t

cc bo in tca ting Vit thuc nhiu ch khc nhau nhcng ngh thng tin,

kinh t, chnh tr, x hi, php lut, i sng, ...

21


33/56

Bng 3.1: Tp th vnPOSTag ca tloi ting vit.

STT Tn th fi ngha ca th1 NN Danh tthng2 NC Danh tch loi3 NP Danh tring4 VB ng t5 JJ Tnh t6 PP i t7 D nh tv s t8 AD Ph t9 IN Gii t

10 CC Lin t11 UH Thn t

12 RB Tr

t

13 TN Thnh ng14 X Cc tkhng th phn loi c

15++ Symbols Cc k hiu c bit khc (, #, $, ...)

3.1.3 Xy dng vnPOS corpus

Xy dng tool tr gip gn nhn vnPOS

gip cho vic gn nhn chc nng c php POS c thun li v nhanh chng, chng

ti xy dng mt cng c tr gip gn nhn c giao din nhhnh 3.1.

Khi file c ti vo RichTextBox phn trung tm th tool c a ra gi v th

PoS ca tng tnhsau:

Cc ttrong t in s c gn nhn dng:

Mu xanh BLUE (ngnh//NC) tc l trong t in c sn n ch gimt chc

nng .

Mu RED (c//VB) tc l trong t in n ginhiu hn mt chc nng

c php.

Cc tcn li th trng nhn

Ngoi ra, tool cn c chc nng tm kim th ca tng t c gn nhn trc

a gi thm trong qu trnh lm dliu.

22


34/56

Hnh 3.1: Giao din cng c tr gip gn nhn vnPOS.

23


35/56

Thc hin gn nhn vnPOS

Kt qu ca qu trnh ny l mt corpus gn nhn POS ting Vit gm xp x 8000

cu ly tcc bo in tthuc nhiu ch khc nhau gm khoa hc cng ngh, kinh

t, chnh tr, x hi, m t xe my, i sng, php lut. y l mt v d cu c gnnhn trong corpus.

Vi//IN khong//D 8//D triu//D thu_bao//NN GSM//NP th//IN th_tr ng//NN

trong//IN n c//NC l//RB rt//AD ln//JJ vi//IN nh ng//D nh//NC khai_thc//VB

,//, cung_cp//VB dch_v //NN GTGT//NN trn//INin_thoi_di_ng//NN .//.

H//PPu//AD hoan_nghnh//VB s //NCi_mi//VB c//PP hai//D ph ng_din//NN

kinh_t//NN v//CC chnh_tr//NN .//.

3.2 Gn nhn t loi bng phng php Maximum

Entropy Markov Model

3.2.1 M hnh xc sut

Theo [1] m hnh xc sut c nh ngha trn khng gian HxT, trong H l tp t

c th v ngcnh t loi, hoc cn gi l "lch s", v T l tp cc th c th c. Xc

sut m hnh cuar lch sh cng vi th t c nh ngha theo cng thc 3.1:

p(h, t) = kY

j=1

fj(h,t)j (3.1)

trong l hng s chun ha, {, 1, ..., k} l cc tham s mang gi tr dng ca

m hnh v {f1, ...,

fk} chnh l cc

c tr

ng "features", th

a

fj(

h, t)

2{0

,1}. Ch

rng mi tham s j tng ng vi mt c trng fj.

Cho trc mt tp cc t{w1, ..., wn } v mt chui th {t1, ..., tn } c xem l dliu

hun luyn, ta nh ngha hi l lch s khi d on th ti. Cc tham s {, 1, ..., k}

c chn sao cho lm cc i likelihood d liu hun luyn s dng p theo cng thc

3.2:

L(p) =nY

i=1

p(hi, ti) =nYi=1

kYj=1

fj(hi,ti)j (3.2)

24


36/56

M hnh ny c xem xt di dng Maximum Entropy, trong mc tiu l cc i

entropy ca mt phn phi di nhng rng buc nht nh. y, entropy ca phn

phi p c nh ngha theo cng thc 3.3

H(p) = X

h2H,t2

p(h, t)logp(h, t) (3.3)

v cc rng buc c cho bi cng thc 3.4

Efj = eEfj , 1 j k (3.4)

trong k vng c trng ca m hnh l 3.5

Efj =X

h2H,t2

p(h, t)fj(h, t) (3.5)

v k vng c trng quan st l 3.6

eEfj =nXi=1

ep(hi, ti)fj(hi, ti) (3.6)

trong ep(hi, ti) l xc sut ca (hi,ti) trong d liu hun luyn. V th, cc rng bucny s p buc m hnh phi ph hp (match) cc k vng c trng vi k vng c

trng quan st trong dliu hun luyn.

3.2.2 Cc c trng ca POS tagging

Xc sut ng thi ca lch s h l th t c xc nh bng cc tham s m cc c

trng tng ng ca n l hu ch, v d, j tha mn fj(h, t) = 1. Khi cho trc (h,t),mt c trng phi tn ti trn bt cword hoc tag trong lch sh, v phi cha thng

tin gip d on th t, v d nhthng tin chnh t ca thin ti, hoc thng tin v

hai th trc t hin ti. Ngcnh word v tag xc nh i vi mt feature c cho

bng nh ngha ca lch shi nhcng thc 3.7:

hi = {wi, wi+1, wi+2, wi1, wi2, ti1, ti2} (3.7)

25


37/56

V d,

fj(hi, ti) =

8