Source: G. Neumann

1999

Information ExtractionInformation Extraction

Technologies & Applications

Günter Neumann

LT-lab, DFKI

Source: G. Neumann

1999

The increased availability of electronic text data requires new technologies for extracting relevant information

INFORMATION EXTRACTION (IE)

The goal of IE research is to build systems that find and link relevant information from NL text while ignoring extraneous and irrelevant information

The core functionality of an IE system is quite simple:

Input: 1. Specification of the relevant information in form of templates (feature structures), e.g., company information, product information, management succession, meetings of important peoples2. A set of real-world text documents

Output: A set of instantiated templates filled with relevant text fragments (eventually normalized to some canonical form)

Source: G. Neumann

1999

Example Information Extraction 1

Lübeck (dpa) - Die Lübecker Possehl-Gruppe, ein im Produktions-, Handel- und Dienstleistungsbereich tätiger Mischkonzern, hat 1994 den Umsatz kräftig um 17 Prozent auf rund 2,8 Milliarden DMgesteigert. In das neue Geschäftsjahr sei man ebenfalls „mit Schwung“gestartet. Im 1. Halbjahr 1995 hätten sich die Umsätze des Konzernsim Vergleich zur Vorjahresperiode um fast 23 Prozent auf rund 1,3Milliarden erhöht.

type = turnover c-name = Possehl1 year = 1994 amount = 2.8e+9DM tendency= + diff = +17%

type = turnover c-name = Possehl1 year = 1995/1 amount = 1.3e+9DM tendency= + diff = +23%

Source: G. Neumann

1999

Example Information Extraction 2

Parts from RWE‘s Anual Report (1998):Eine Schwerpunktregion im Rahmen der Internationalisierung im Energiebereich istOsteuropa. Hier haben wir unser Engagement im abgelaufenen Geschäftsjahr weiterausbauen können.Nach dem Kauf weiterer Anteile halten wir inzwischen jeweils knapp über 50% an den ungarischen Energieversorgungsunternehmen ELMÜ, ÉMÁSZ und MÁTRA. Im Falle von MÁTRA hat RWE Energie im April 1998 Anteile an Rheinbraun abgegeben. Die Präsenz in Polen wurde durch Kooperationsvereinbarungen mit den Regionalversorgern Zaklad Energetyczny Krakow S.A. (ZEK) und Stoleczny Zaklad Energetyczny S.A. (STOEN) ....im Frühjahr 1998 weiter ausgebaut.

Group/Subs. YEAR KIND FROM TO POT AMOUNTRWE 1998 + ELMÜ >50%RWE 1998 + ÉMÁSZ >50%RWE 1998 + MÁTRA >50%RWE Energie 1998 - MÁTRA Rheinbraun 4.1998

Source: G. Neumann

1999

From the viewpoint of natural language processing (NLP), IE is attractive for many reasons, including

• Extraction tasks are well defined

• IE uses real-world texts

• IE poses difficult and interesting NLP problems

• IE needs systematic interface specification between NL and domain knowledge

• IE performance can be compared to human performance on the same task

IE systems are a key factor in encouraging NLP researchers to move from small-scale systems and artificial data to large-scale systems operating on human language (Cowie & Lehnert, 1996)

Source: G. Neumann

1999

IE has a high application impact

• IE and information retrieval: construction of sensitive indices which are more closely linked to the actual meaning of a particular text

• IE and text classification: getting fine-grained decision rules

• IE and text mining: improve quality of extracted structured information

• IE and data-base systems: improve semi-structured DB approaches

• IE and knowledge-base systems: combine extracted information with KB

Quelle: GN

1999

The advanced IE technologies improve intelligent indexing and retrieval

improved indexing:

marked text &templatesIE core system

text files

indexing construction(phrasal, concept indices)

improved retrieval:

search engine

query

IE core system

Source: G. Neumann

1999

Shallow text processing as a common pre-processing tool for TM & IE

STP:TokenizationLex. ProcessingChunk Parsing

Data MiningInformationExtraction

Text documents

Str

uctu

re

Langu

ages

Domain

LinguisticEntities

LinguisticEntities

Instantiatedtemplates

Complexrelationsare known

Complexrelationsto be discovered

Source: G. Neumann

1999

IE as core component in incremental knowledge engineering systems

< >

IE core system

Statistical eval. &visualization

Construction of ontology

OntologyDomainlexicon

The core idea:

Hand-craft construction of domainontology on basis of linguistic informationextracted from texts.

Simulatenously construct domain lexiconwhich is used in next acquisition cycle.

Source: G. Neumann

1999

From a system development point of view there exists two approaches for IE

• Language technology approach (dem Ingeniör is nix zu schwör) linguistic knowledge specified manually by experts mapping between NL and domain knowledge hand-coded manual inspection of corpus to find out how specific domain knowledge is

expressed via NL still best approach to build reasonable complex systems development of tools for supporting application building

• Learning approach

– apply statistical methods where possible

– learn template filler rules from annotated corpora using Machine Learning shows promising results for IE subtasks (proper name recognition, flat slot

filler rules) mapping between NL and domain knowledge automatically induced still needs high amount of annotated corpora

Source: G. Neumann

1999

Common two both approaches is the use of shallow re-usable NL core components (of course, differing in granularity)

• tokenization, text scanning/information wrapping (e.g., analysis of tables, head lines)

– in most cases simple, but very important

• Morphological & lexical processing

– high-coverage and fast morphological analysis

– processing of unknowns & compounds

– part-of-speech tagging

• Recognition of named entities (proper names, expressions for dates, values, measurement); important issue here: new name creation, multilingual terms; Demo)

• shallow parsing: very long sentences (> 30 words), relclauses, coordination

– integration of specific subgrammars

– partial parsing

– very robust and efficient strategies needed (weighted finite state transducers)

• discourse analysis (NP-analysis, co-reference, relational links)

Source: G. Neumann

1999

IE poses difficult and interesting NLP problems, which has partially lead to the renaissance of „old“ methods and their improvement

• High amount of robusntess and efficiency requested:

– finite state technology (weighted finite state transducers)

– text-skipping as a realistic approach for handling real-world text

• systematic specification of NL and domain knowledge

• short system application cycle because of „daily news“

• multilingual methods are required

• evaluation of the predicting power of IE systems

– recall R = #correct found answers / #total possible correct

– precision P = #correct found answers / #found answers

– f-measure F = (2 + 1) R P / 2 R+P 0.6 barrier

Source: G. Neumann

1999

At DFKI´s LT-lab we have developed powerful domain-independent shallow text processing components in order to support a fast IE system development cycle

Text Tokenization

Lexical processor• Morphology• Compounds• Tagging

Chunk Parser• sentence topology• phrase recognition• sentence structure• grammatical fct.

> 120.000 main stems;> 12.000 verb frames;special name lexica;tagging rules;

general (NPs, PPs, VG);special (lexicon-poor,Time/Date/Names);general sentence patterns;

Lexical DB

Grammars (FST)

Text

Set of

Underspecified

Fct. Descr

Shallow Text Processor

Source: G. Neumann

1999

We have identified the needs for better chunk parsing strategies in order to improve robustness and coverage on the sentence level

Phrase recognition

Text (morph. analysed)

Clause recognition

Stream of phrases

Stream of sentences

Current chunk parser

bottom-up:first phrases and then sentence structure

main problem: even recognition of simple sentence structure depends on performance of phrase recognition

example: - complex NP (nominalization style)- relative pronouns

[Die vom Bundesgerichtshof und den Wettbewerbern als Verstoss gegen das Kartellverbot gegeisselte zentrale TV-Vermarktung] ist gängige Praxis.([central television marketing censured by the German Federal High Court and the guards against unfair competition as an act of contempt against the cartel ban] is common practice)

grammatical fct. recognition

Source: G. Neumann

1999

A new chunk parser has been developed that increases robustness and coverage on the sentence level

Sentencetopology

Text (morph. analysed)

phrase recognition

Stream of sentence structure

Stream of sentences

New chunk parser

top-down/bottom-up:first compute topological structure of sentencesecond apply phrase recognition to the fields

[coord [core Diese Angaben konnte der Bundesgrenzschutz aber nicht bestätigen], [core Kinkel sprach von Horrorzahlen, [relcl denen er keinen Glauben schenke]].(This information couldn‘t be verified by the Border Police, Kinkel spoke of horrible figures that he didn‘t believe.)

Advantages- simple (topology recognition based on keywords and verb groups)- resolution of critical ambiguities (coordination, pronouns/determiners)- no restriction on # of subclauses- good coverage (first tests on 670 sentence: 85-90%, Urli)

grammatical fct. recognition

Source: G. Neumann

1999

The Shallow Text Processor has several Important Characteristics

Modularity: each subcomponent can be used in isolation;

Declarativity: lexicon and grammar specification tools;

High coverage: more than 93 % lexical coverage of unseen text;

high degree of subgrammars (70% general NPs, >90% sentence topology);

Efficiency: finite state technology in all components;

specialized constrained solvers

(e.g. agreement checks & grammatical functions);

Run-time example:1-3 second per text page real time

Source: G. Neumann

1999

We´re using typed feature structures for modeling the domain knowledge and its relationship to NL expressions

Templ[action,date]

Move-T[from, to, unit]

Loc-T[action]

Fight-T[victim,attacker,attacked]

Meeting[visitor,visitee]

Phrase

Np

LocNpLocPp

DatePP

PpFdesc[process,mods]

trans[subj,obj]

intrans[subj]

Fight-L[subj=1, obj=2,temp=[attacker=1, attacked=2]

[process=1,subj=2,obj=3,dateMod = < ... =4 ...>,locMod = < ... =5...>,templ = [ action kill=1, attacker soldier=2, attacked general=3, date 3/8/93=4, Loc Mostar=5]]

UFD

DomainLex:kill=Fight-L

FillTempl

TypeMerge

STP

Source: G. Neumann

1999

The model as several advantages

• New applications are defined basically through

– template hierarchy (independently from linguistic knowledge)

– integration of domain and linguistic knowledge only through

linked types between abstract linguistic categories

• Support of faster adaptation to new applications

• The focus of the next research periods:

– systematic integration of lexical semantics (in particular nominal entities)

– template merging

– methods for automatic knowledge acquisition

– learning of domain lexicon

– learning of linked types

Source: G. Neumann

1999

Recently, the problem of using machine learning methods to induce IE routines has received more attention

• The majority of current approaches are variants of inductive supervised learning

• Goal: given a set of annotated text documents induce automatically template filler rules by successively generalizing the initialized instantiated rules computed from the tagged examples

• Usually, the documents are preprocessed by NL components– tokenization (Freitag, 98)– POS tagging (Califf&Mooney,98)– phrase recognition (Hufman,96)– shallow sentence parsing (Riloff,96a;Soderland,97)

• most approaches learn slot-filler rules, some newer learn relational structures (Califf&Mooney,98)

• current trend is towards minimally supervised strategies (Riloff,96b)

Source: G. Neumann

1999

The majority of current approaches are variants of inductive supervised learning

• Example

<PNG> Sue Smith </PNG>, 39, of Menlo Park, was appointed <TNG> president </TNG> of <CNG> Foo Inc. </CNG>

n_was_named_t_by_c:noun-group(PNG, head(isa(person-name))),

noun-group(TNG, head(isa(title))),

noun-group(CNG,head(isa(company-name))),

verb-group(VG, type(passive), head(named or elected or appointed)),

prep(PREP, head(of or at or by)),

subject(PNG,VG), object(VG,TNG), post_nominal_prep(TNG,PREP), prep_obj(PREP,CNG)

management_appointment(M,person(PNG), title(TNG), company(CNG)) supervised and unsupervised methods

• Current approaches show already impressive results (for flat sentence-based templates):

– Huffman,96: management changes task => 85.2% F (89.4%)– Califf&Mooney,98 : computer-related job postings => 87.1% P & 58.8% R

Source: G. Neumann

1999

In case of multilingual name recognition learning approaches are very promising

• Nymble, a high performance learning name-finder (Bikel et al. 97, BBN)

– hidden Markov model

– word-features (e.g., allCaps, twoDigitNum)

– Results for F-measure

• English: 93% (best MUC-6: 96%)

• Spanish: 90% (93%)

• Gallippi, 96:

– data-driven knowledge acquisition strategy based on decision trees (ID3)

– features: POS, Abbrev., list of known names, word-features

– Results for F-measure (av. across companies, persons, locations, date)

• English: 94%

• Spanish: 89.2%

• Japanese: 83.1%

Source: G. Neumann

1999

In summary: IE is a very attractive research area for building application systems

• IE is an interdisciplinary approach

– language technology

– statistical methods

– machine learning

– knowledge representation

– software engeenering

– expert knowledge

• What mix of methods is best depends on complexity of target information

– hand-crafted vs. automatic or mixed strategies

• Next system generation will

– be more intelligent in adapting itself to new domains

– learn from experience being minimally supervised

– support shorter development cycle (dynamic environment)

– understand several languages