kyoungryol kim meeting information extraction from meeting announcement in korean

Kyoungryol Kim

Meeting Information Extraction from Meeting Announcement in Korean

2

Table of Contents

1. Introduction Motivation Goal Problem Definition Contribution

2. The Proposed Method Finding

3. Discussion

3

Introduction

4

Motivation (1/3) : Necessity

Everyday we receive a lot of Meeting Announcement Conference, Seminar, Workshop, Meeting, Appointment… Meeting announcement accounts for 17%

(30,201 out of 183,022) of emails in Enron Email Dataset.

Smartphone era Many people manage schedule using online-calendar via

smartphonee.g. Google Calendar

But, typing by touch screen keyboard make many errors and even it’s difficult.

* Enron Email Dataset, August 21, 2009 version, http://www.cs.cmu.edu/~enron/

5

Goal

Extracting schedule information from meeting announcement,and update them to the calendar, automatically.

무더운 날씨가 본격적으로 시작되는 즈음하여 유니브캐스트의 상반기 평가와 하반기 운영을 위한 정기팀장회의를 개최합니다 .날짜 : 7 월 19 일 ( 토 ) 오후 2 시장소 : 민들레영토민들레영토 오는길지도와 같이 명동역 8 번 출구로 나오셔서 쭉 상가 끼고 걸어가시면 저기 YMCA 빌딩 1 층에 있습니다 .

startTime 2011-07-19T14:00

isHeldAt서울시 중구 명동 1-3 민들레영토Latitude : 126.9797848, Longitude : 37.5687868

locationLandmark

서울지하철 4 호선 명동역 8 번출구Latitude : 126.9864660, Longitude : 37.5609660

Meeting Announcement

Extract Update

6

Problem DefinitionTo find Meeting Location, the problem divided into 3 parts :

1. Finding locations for each type of complexity.

2. Named entity disambiguation on found locations.

무더운 날씨가 본격적으로 시작되는 즈음하여 유니브캐스트의 상반기 평가와 하반기 운영을 위한 정기팀장회의를 개최합니다 .

날짜 : 7 월 19 일 ( 토 ) 오후 2

시장소 : 민들레영토기본 안건- 제작지원비 지급 지연에 대한 설명- 기금 조정 운영안- 가을 워크샵 준비위 구성- 기타 ( 기타 안건으로 상정할 것이 있으면 각 팀장들은 제안해 주시기 바랍니다 )

민들레영토 오는길지도와 같이 명동역 8 번 츨구로 나오셔서 쭉 상가 끼고 걸어가시면 저기 YMCA 빌딩 1

층에 있습니다 .

참고하세요

1. Finding Locations(Location-type NER)


날짜 : 7 월 19 일 ( 토 ) 오후 2

시장소 : 민들레영토기본 안건- 제작지원비 지급 지연에 대한 설명- 기금 조정 운영안- 가을 워크샵 준비위 구성- 기타 ( 기타 안건으로 상정할 것이 있으면 각 팀장들은 제안해 주시기 바랍니다 )

민들레영토 오는길지도와 같이 명동역 8 번 츨구로 나오셔서 쭉 상가 끼고 걸어가시면 저기 YMCA 빌딩 1

층에 있습니다 .

참고하세요

Start/End Time Extraction

2. NE Disambiguation

7

isHeldAt

민들레영토민들레영토YMCA 빌딩 1 층

locationLandmark 명동역 8 번출구

3. Normalization &Co-reference

startTime 2011-07-19T14:00

isHeldAt서울시 중구 명동 1-3 민들레영토Latitude : 126.9797848, Longitude : 37.5687868

locationLandmark

서울지하철 4 호선 명동역 8 번출구Latitude : 126.9864660, Longitude : 37.5609660

8

Definition Definition 1. Location Named Entity

A particular point or place in physical space (Wiktionary).

[Cyber Space] Exceptionally, If the cyber space is used as a place gathering people, then the cyber space can be a location.

e.g. MSN에서 9시에 모입니다 .

[Road, Street, Transportation] cannot be a location, except if it points particular place or it is necessary to de-scribe the location.

e.g. 진천 I/C, 왼쪽에 석촌지하차도가 보임 [Bridge] can be a location.

e.g. 납안교 , 한강대교 [Train/Subway Station, Bus-stop] can be a location.

e.g. 도곡역 1번출구 , 뱅뱅사거리 [Address] Full/partial address can be a location.

e.g. 전북 무주군 설천면 심곡리 43-15

[Organization, Company, Heritage, Building] can be a location if it is used to represent the location. [Parenthesis] If the location is ambiguous when the string in the parenthesis is removed and separated by the

parenthesis, then the string including parenthesis are the part of the location.e.g. COEX 컨퍼런스센터 4층 (402호 ), 건국대학교 (서울 ) 의생명연구동 강당 , 경인교육대학교 (경기캠퍼스 ),부산벡스코 (BEXCO) 컨벤션홀 201호 , 생명과학관 (녹지 ) 139호

[Enumeration] The different representations for same location are recognized separately.e.g. 장소 ? 가야 레스토랑 . 전화 /215-654-8900, 주소 /1002 Skippack Pike, Blue Bell, PA 19422전주 화산체육관 (전북 전주시 완산구 중화산동 1가 45번지 ), 2. 장소 : 늘푸름 (오산시 은계동 91-8)

Definition 2. Meeting LocationMeeting Location is the Location where the meeting will be held.

Definition 3. Location LandmarkLocation Landmark is the Location where can be used as a landmark to go to the meeting location.

9

Complexity of the problems

d

10

The Proposed Method1) Location Named Entity Recognition2) Relation Type Classification3) Co-reference4) Normalization

11


날짜 : 7 월 19 일 ( 토 ) 오후 2 시장소 : 민들레영토기본 안건- 제작지원비 지급 지연에 대한 설명- 기금 조정 운영안- 가을 워크샵 준비위 구성- 기타 ( 기타 안건으로 상정할 것이 있으면 각 팀장들은 제안해 주시기 바랍니다 )

민들레영토 오는길지도와 같이 명동역 8 번 츨구로 나오셔서 쭉 상가 끼고 걸어가시면 저기 YMCA 빌딩 1 층에 있습니다 .

참고하세요



명동 민들레영토 오는길지도와 같이 명동역 8 번 츨구로 나오셔서 쭉 상가 끼고 걸어가시면 저기 YMCA 빌딩 1 층에 있습니다 .

참고하세요

Overall Architecture

InputDocument

OUTPUTRelation TypeClassification

Named EntityRecognition(Location)



민들레영토 오는길지도와 같이 명동역 8 번 츨구로 나오셔서 쭉 상가 끼고 걸어가시면 저기 YMCA 빌딩 1 층에 있습니다 .

참고하세요

Co-reference Normalization

isHeldAt

민들레영토민들레영토YMCA 빌딩 1 층


서울시 중구 명동 YMCA 빌딩 1 층

민들레영토

12

1) Location Named Entity Recognition

13

Architecture of Location NERTraining the system

(supervised learning)Testing the system

(actual use and evaluation)

Training Corpus Web

Gazetteer

Gazetteer Extraction

FeatureExtraction

CRFsModel

Input: Morpheme-level tok-enized sentence list

Output: NE Annotated Email Document

FeatureExtraction

BoundaryTagging (IOB2)by CRFs Model

Tokenization

TF-IDFCalculation Boundary Mark-

ing (IOB2)

CRFsLearning

TF-IDFScore Data

BoundaryMerging

14

NER - Boundary Detection Boundary Tagset : IOB2 Features

Linguistic {-2,-1,0,1,2} POS-level word, {-2,-1,0,1,2} POS-tag,

POS-tag + length of the word

Orthographic : 18 types of the word isKorean, isAlpha, isAlnum, 2DigitNum, ...

Gazetteer : Person/Location Pronoun dictionary (ETRI 99) from Training corpus :

Heading words, Surrounding words, NE words External resources :

Person : Chosun/Joins.com Person DB (64,042) Location :

Nate Local DB 35,335, Sigaji.com 8,193, Ofood 43,390BusStop 19,431, Address,B/D 23,365, Subway 1,288,Hotel (Auction accomodation, hotelnjoy) 884,Country/Place name 11,946, School(Elementary~University) 21,957

Syntactic : Position of the POS-level word in the chunk (relative:S/C/E, absolute) Position of the chunk in the sentence (relative:S/SC/CE/E, absolute) Position of the sentence in the document (relative:S/SC/CE/E, absolute) TF-IDF

15

Features : Gazetteer data

Location : Shop Name (80,436)

Nate Local DB (3~10 chars.) (http://localinfo.nate.com)

Sigaji.com Shop DB (3~10 chars.) (http://sigaji.com/location/)

oFood (http://ofood.co.kr)

Hotel Name (884) Auction Accomodation

(http://accommodations.auction.co.kr) Hotelnjoy

(http://www.hotelnjoy.com) Public Transportation (20,719)

Subway stations Bus-Stop names

Address (from Zipcode DB) (23,365) Si/do, Gu/gun, Dong/myun/ri, B/D names

16

Evaluation Result (1/2) Baseline Boundary Detection

Target : 13,076 sentences in 1,011 documents. CRFs Model, 10-fold cross validation, 3-order, Exact Matching Baseline is the case applying Word and POS-tag feature only

baseline B-Location

Precision 49.99% 47.93%

Recall 16.97% 64.84%

F-measure 24.34% 55.11%

B-Location I-Location

baseline I-Location

Precision 24.94% 77.99%

Recall 39.82% 69.58%

F-measure 32.99% 73.54%

Precision Recall F-measure 0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Precision Recall F-measure 0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

17

2) Relation Type Classification

18

Architecture of Relation Type Classifier

Training the system(supervised learning)

Testing the system(actual use and evaluation)

Training Corpus Web

Gazetteer

Gazetteer Ex-traction

FeatureExtraction

SVMsModel

Input: Location NE-taggedDocument

Output: Extracted NE withMeeting-NE Relation Type

FeatureExtraction

Relation TypeClassification

By SVMs Model

Tokenization

SVMsLearning

TemplateGeneration

19

Statistics of Relation Types

Document-Location Relation Type Classification Target : 1,844 Location-type Terms

848 isHeldAt (45.99%) 161 locationLandmark (8.78%) 835 generalLocation (45.28%)

835

848

161

General Location

isHeldAt

locationLandmark

20

Features

Linguistic1. Gazetteer

A. Named Entity Dictionary Nate Local DB 35,335, Sigaji.com 8,193, Ofood 43,390

BusStop 19,431, Address,B/D 23,365, Subway 1,288,Country/Place name 11,946,

B. from Training Corpus : Heading words in the current sentence Heading words in the previous sentence NE consisting words

2. Lexical PatternA. POS-tag feature before and next to the NE

B. Is this NE the first location NE next to colon?

C. Is this term in the parenthesis?

D. Is parenthesis opened and closed next to the NE ?

E. Is direction word just next to the NE?

Syntactic3. Syntactic Features

A. Is the NE the first or the last Location-type of NE in the sentence?

B. Ratio of location NE in the current sentence to the document

C. Relative position of the NEs in the sentences

D. Is the NE the longest location NE in the sentence?

21

Experiment : Features (1/3)

1. Gazetteer

A.Named Entity Dictionary Collected from the web Check if each morpheme, eojeol or term

matches the word in the dictionary.

I. Nate Local DB, Sigaji.com, Ofood

II. Address, Building name

III. Bus-stop, Subway station

IV. Country name

V. Location-related Vocabulary

B.from Training Corpus :I. Heading words in the current sentence.

II. Heading words in the previous sentence.Heading word is the word before the colon in the sentence

e.g. 장소 : 피오레웨딩컨벤션 ( 봉계동 여수 세무서 옆 )

III. Eojeol-level NE consisting words

　 isHeldAtP (%) R (%) F (%)

locationLandmarkP (%) R (%) F (%)

Acc.(%)

I 59.32 / 98.94 / 74.17 57.14 / 02.45 / 04.71 59.09

+II 60.45 / 93.99 / 73.58 47.62 / 06.13 / 10.87 58.39

+III 63.12 / 89.52 / 74.04 62.96 / 31.29 / 41.80 60.76

+IV 64.55 / 88.57 / 74.68 62.96 / 31.29 / 41.80 62.30

+V 70.64 / 86.45 / 77.75 70.91 / 47.85 / 57.14 67.20

Feature 1A



Acc.(%)

+I 81.45 / 85.87 / 83.60 71.09 / 55.83 / 62.54 75.17

+II 80.56 / 84.92 / 82.68 69.40 / 57.06 / 62.63 75.03

+III 84.86 / 87.16 / 86.00 77.44 / 63.19 / 69.59 79.93

Feature 1 (A+B)

22

Experiment : Features (2/3)2. Lexical Patterns

A. POS-tag feature just before and next to the NEe.g. 장소 : 피오레웨딩컨벤션 ( 봉계동 여수 세무서 옆 )

B. Is this NE the first location NE next to colon?e.g. 장소 : 피오레웨딩컨벤션 ( 봉계동 여수 세무서 옆 )

C. Is this NE in the parenthesis?e.g. 장소 : 피오레웨딩컨벤션 ( 봉계동 여수 세무서 옆 )

D. Is parenthesis opened and closed next to the NE ?e.g. 장소 : 피오레웨딩컨벤션 ( 봉계동 여수 세무서 옆 )

E. Is direction word just next to the NE? 34 direction words : 위 , 아래 , 밑 , 옆 , 앞 , 내 , 외 , …

e.g. 장소 : 피오레웨딩컨벤션 ( 봉계동 여수 세무서 옆 )

F. Is the unit of length appeared in the next 3 eojeolsof the NE? [0-9]+(m|km|ft|yd|mile| 미터 | 킬로미터 | 피트 | 야드 | 마일 | 리 | 초 | 분 | 시간 )

G. Is transportation words contained in the left eojeol?



Acc.(%)

+A 86.52 / 86.93 / 86.72 77.10 / 61.96 / 68.71 80.77

+B 86.94 / 86.22 / 86.58 73.79 / 65.64 / 69.48 80.63

+C 87.98 / 86.22 / 87.09 76.19 / 68.71 / 72.26 81.05

+D 88.38 / 86.93 / 87.65 75.33 / 69.33 / 72.20 80.63

+E 88.52 / 87.16 / 87.83 77.18 / 70.55 / 73.22 81.82

+F 88.76 / 87.40 / 88.07 77.70 / 70.55 / 73.95 82.10

+G 88.33 / 87.40 / 87.86 80.41 / 73.01 / 76.53 82.24

Feature 1+2 (A~G)

23

Experiment : Features (3/3)3. Syntactic Features

A. Is the NE the first or the last Location-type of NE in the sentence?e.g. (1 호선에서 갈아탈 경우 동묘역에서 6 호선을 갈아타고 봉화산방향으로 타고 오시면 2 번째 정거장이 보문역입니다 .

B. Ratio of NEs in the current sentence to the document (<25%,<50%,<75%,<100%,=100%)C. Relative position of the sentence to the document. (S / SC / CE / E)D. Relative position of the eojeol to the sentence. (S / SC / CE / E)E. Relative position of the NEs in the sentence (S / SC / CE / E)

e.g. (1 호선에서 갈아탈 경우 동묘역에서 6 호선을 갈아타고 봉화산방향으로 타고 오시면 2 번째 정거장이 보문역입니다 . S CE E

F. Is the NE the longest location NE in the sentence? G. Is this only location NE in the sentence?H. Is the NE on the previous/next to the NE in the sentence?I. Is same type of NE in the prev/next sentence ?J. Is phone number on the left or right side of the NE?K. Surrounding word (on the right side of the NE, n=1, pos=etc,p,j,m,x ) ?L. Is Colon included in the curr/prev/next sentence?M. Is the sentence starts/ends with the NE?N. # of chunks of the NE (max:99)O. Length of the NE (max:300)P. Is location related word included in heading ?Q. Is location related word included in heading of prev. sentence?R. is the NE which is appeared more than 2, is next to this NE?S. Is the NE appeared more than 2?T. Transport Dic Feature (left / right)U. Order of the ne in the sentence.V. does sentence starts with special char?



Acc.(%)

+GH 89.42 / 89.10 / 89.26 79.22 / 75.78 / 77.46 82.23

Feature 1+2+3 (G,H)

24

Experiment: Relation Type Classification

Meeting-Location Relation Type Classification Target : 1,844 Location-type NEs SVMs 3 classes (multi-class) classifier Total Accuracy : 82.23

　 isHeldAt locationLandmark

Precision 89.42% 79.22%

Recall 89.10% 75.78%

F-measure 89.26% 77.46%

isHeldAt locationLandmark0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Precision Recall F-measure

25

3) Normalization & Co-reference

26

Architecture of Normalizer

Country | State/City | City/Gu/Gun | Dong/Eup/Myeon | Ri | House no. | Org. | B/D | Floor | Shop Name | Room no.

1 2 3 4 5 6 7 8 9 10 11

isHeldAt

민들레영토 민들레영토 YMCA 빌딩 1 층


InputDocument OUTPUT

VerifyingElements

민들레영토민들레영토

YMCA 빌딩 (A8) / 1 층 (A9)명동역 (S3) / 8 번출구 (S4)

Pattern

Addr.

Address Format

Subway Format

City | Line no. | Station Name | Gate no.

1 2 3 4

Expansion

민들레영토민들레영토

YMCA 빌딩 (A8) / 1 층 (A9)서울시 (S1) 4 호선 (S2) 명동역 (S3) / 8 번출구 (S4)

Combine

A1 : 대한민국A2 : 서울시A3 : 중구A4 : 명동A5 : -A6 : -A7 : -A8 : YMCA 빌딩A9 : 1 층A10 : 민들레영토A11 : -

Subway Open Map Services(Google Maps, Yahoo! Maps,

Daum Map, Naver Map)

27

Discussion1) Limitations2) Applications

28

Limitations

1. Performance Both system should be refined more in detail, with sophisticated experiment.

2. Scaling Up For our corpus consist of 1,011 emails, the method to cover more data in the

real-world should be mentioned.

3. Feature Selection Since we use +165,000 word-gazetteer and many of these features always zero

in the training data. In order to save memory and to maximize the perfor-mance, these unsupported features need to be removed.

29

Applications

1. Smartphone application Extracting start/end time, location from email and update them to

Google Calendar.

2. Contribution to OpenStreetMap community Update found locations automatically to openstreetmap.com

kyoungryol kim meeting information extraction from meeting announcement in korean

Documents