by nuno cardoso faculty of sciences, university of lisbon ... · faculty of sciences, university of...
TRANSCRIPT
Geographic IR Challenges
by Nuno Cardoso
Faculty of Sciences, University of Lisbon, LASIGE
Presentation held at SINTEF ICT, Oslo, Norway, 4th December, 2007
LaSIGE Laboratory
● Large-Scale Informatics System Laboratory
● 4 research groups: XLDB, Navigators, DIALNP, HCIM
● XLDB research lines: Information retrieval, text mining, natural language processing, web archiving and search, information visualization, bioinformatics
● XLDB's staff: Ana Paula Afonso, Ana Vaz, André Falcão, Catarina Rodrigues, Cátia Pesquita, Daniel Faria, Daniel Gomes, David Cruz, Francisco Couto, Hugo Bastos, Leonardo Andrade, Liliana Moreira, Luís Russo, Marcirio Chaves, Mário Silva, Nuno Cardoso, Paulo Carreira, Paulo Pombinho, Sérgio Freitas, Tiago Grego
(In bold: Linguateca 's XLDB Node)
• A distributed resource center for Portuguese language technology
• IRE model:
– Information
– Resources
– Evaluation
www.linguateca.pt
Linguateca, a project for Portuguese
Odense
Coimbra 3
São Carlos Lisboa
LabEL
Braga
Oslo 2.5
Lisboa 2.8COMPARA 4 (2.8)
Lisboa 3XLDB
Porto 2
XLDB / LaSIGE
Human capital of Linguateca(in bold: intersection with XLDB)
Senior researchers: Diana Santos, José João Almeida, Eckhard Bick, Belinda Maia, Ana Frankenberg Garcia, Mário J. Silva, Paulo Gomes, Luís Costa
Researchers: Luís Miguel Cabral, Susana Inácio, Rosário Silva, Paulo Rocha, Cláudia Freitas (PhD), Sérgio Matos (PhD), Hugo Oliveira, Pedro Martins, David Cruz
PhD students: Marcirio Chaves, Alberto Simões, Nuno Seco, Anabela Barreiro, Nuno Cardoso
Former: Rachel Aires (PhD, 2005), Signe Oksefjell, Susana Afonso, Raquel Marchi, Renato Haber, Alex Soares, Pedro Moura, Ana Sofia Pinto, Débora Oliveira, Isabel Marcelino, Cristina Mota, Luís Sarmento, António Silva, Rui Vilela
In all, 36 (22) persons with some link to Linguateca (not counting trainees or undergraduate students)
5
Presentation overview
1) What is GIR?
– GIR terminology
– Motivation for GIR
2) GIR approaches
– The case of GeoTumba
3) GIR evaluation
– In-house evaluation of GIR components
– International evaluation contests: GeoCLEF
4) My PhD research plan
7
GIR terminology
Why is it important?● We are dealing with words written in natural
language that correspond to concepts. ● Mapping words in context to concepts is not
straightforward...
Language World (representation of)
WORDS CONCEPTS
8
GIR terminology
Words with geographic meaning: ● Placenames – words used to name places.
For example: Oslo, Trøndelag, Norway, Scandinavia, “Syden”, Middle East.
● Common nouns – words denoting classes of places. For example: island, mountain, lake, city, country, continent.
● “Relators” – words used to establish a relation between entities (verbs, prepo-sitions, adverbs, etc). For example: “near”, “located in”, “on the shores of”, “up”.
Language
9
GIR Terminology (cont.)
Geographical concepts: ● Features (ISO-19109) – an unambiguous
location. It can be described by one or more placenames. For example: Paris.
● Feature Types (ISO-19109) – classes of features. For example, island, mountain, lake (physical), city, continent, NUT-3 (administrative). A feature has only one feature type.
● Relations – Links joining features OR feature types: part of, adjacent, capital of, etc. Examples: [Oslo] part-of [Norway], [city] part-of [country].
World (represen-tation of)
10
Geographic Information Retrieval
● Geographic Information Retrieval (GIR) = Information Retrieval (IR) + geographic reasoning!
● Goal: research methods to retrieve relevant information in a document collection.
● Relevance now comes twofold:– Thematical (ex: “Find me
documents about curling!”)– Geographical (ex: “in the
Scandinavian countries”)
13
Web search engines go local...
● Local Yahoo! (local.yahoo.com)
● Local Google (= Google Maps)
How well are they handling our geographic restrictions?
17
Motivation for GIR
● Users have information needs that have a given scope of interest.
● Present search engines do not understand placenames in queries as a different kind of relevance criteria.
● Documents should be retrieved according to topic relevance and geographic relevance.
● IR systems must adapt to the users, and not the users to the IR system!
19
How does a typical web IR system work?
1) Crawling: downloads “all” documents from the web;
2) Storage: pre-processes and stores documents locally;
3) Indexing and Ranking: generates term indexes and weighs documents;
4) Interface: processes queries and presents results to the user.
20
How does a web GIR system work?● ... can't tell! GIR is a recent area:
– First workshop: Analysis of Geographical References, hosted in NAACL-HLT in 2003.
– Annual GIR workshops started in 2004.– Evaluation contests
started in 2005 (GeoCLEF).
● Many systems with different approaches. - GeoCLEF 2007: 13 participants, 8 approaches!
Mandl et al, ”Overview of GeoCLEF 2007”
21
The case of
● GeoTumba is a web search engine with geographic capabilities developed by the University of Lisbon.
● Research Team: Mário J. Silva (head), Ana Paula Afonso, Bruno Martins, David Cruz, Marcirio Chaves, Nuno Cardoso and Paulo Pombinho, Leonardo Andrade and Sérgio Freitas
● GREASE project since January 2004.– Diana Santos and Chris Jones (advisors)– Collaboration with SINTEF/Linguateca.
23
's main characteristics:
● Geographic knowledge: provided by a geographic ontology, GeoNET PT 01.
● Text mining for placenames, match them into features of the ontology.
● Assign a list of features to each document, the document geographic signature.
● The non-geographic part (curling) is ranked by text weighting scheme BM25; geographic part (Norway) is ranked by a set of heuristics.
● Doc. ranking score: 0.5 non-geo + 0.5 geo.
24
what
spatial relat.
where
QueOndeQuery parser
QuerColQuery
expansion
Geographic Ontology
Queries
FaíscaText
Mining Sidra5Indexing
Resultsdocu
ments
TermIndex
Documentsignatures Sidra5
Ranking
GeoIndex
Query string:
Query signature
QE terms+
GIR system (2007)
25
TermIndex
what
spatial relat.
where
QueOndeQuery parser
QuerColQuery
expansion
Geographic Ontology
Queries
FaíscaText
Mining Sidra5Indexing
Resultsdocu
ments Documentsignatures Sidra5
Ranking
GeoIndex
Query string:
Query signature
QE terms+
GIR system (2007)
Example: Sea traffic in Portuguese islands =
Sea traffic in Portugal islands
● Splits into <what, spat.rel., where>● Recognizes features & feature types
1. Query parsing
26
TermIndex
what
spatial relat.
where
QueOndeQuery parser
QuerColQuery
expansion
Geographic Ontology
Queries
FaíscaText
Mining Sidra5Indexing
Resultsdocu
ments Documentsignatures Sidra5
Ranking
GeoIndex
Query string:
Query signature
QE terms+
GIR system (2007)
2. Query expansion
Madeira, Porto Santo,São Miguel, Faial, Pico, S. Jorge, Terceira, etc...
1. Term expansion:
Blind Relevance Feedback
2. Geographic expansion:
based on query type, driven by spatial rel., features & feat. types
Example:
Portugal islands
27
what
spatial relat.
where
QueOndeQuery parser
QuerColQuery
expansion
Geographic Ontology
Queries
FaíscaText
Mining Sidra5Indexing
Resultsdocu
ments
TermIndex
Documentsignatures Sidra5
Ranking
GeoIndex
Query string:
Query signature
QE terms+
GIR system (2007)
3. Geographic knowledge
● Query parsing patterns
● Text mining patterns
● Geographic query expansion
28
Our geographic ontology● Encodes geographic knowledge and
provides it to GIR components.● Contains features, feature types and
relationships among them.
29
Mapping placenames into features and feature types
1) sea traffic in Portugal 2) sea traffic in islands 3) sea traffic in Portuguese islands
Europe
Norway
Svinør isl.
Magerøya isl.
Portugal
Oslo Lisbon
1)
2) 3)Madeira Isl.
S. Miguel Isl.
ON
TO
LOG
Y
30
GIR open questions...
● How to represent geographic knowledge?● How to use it properly on GIR components?● How to handle ambiguity of placenames in
documents and queries?● How to classify documents geographically? ● How to measure the geographic affinity
between queries and documents?● How to present the results to the user?
31
3) GIR Evaluation– In-house evaluation of GIR components
– International evaluation contests: GeoCLEF
32
Evaluation of GIR systems
1) In-house evaluation– test each GIR component in a
“glass-box” style evaluation.
– analyse points of failure,measure performance ofeach component.
2) Evaluation contests– “black-box” style evaluation.
– compare different systems,several approaches.
– common evaluation environment.
33
GeoCLEF evaluation setup
● Text collections: newspapers in 3 languages– Portuguese collection provided by Linguateca.
● Every year, 25 topics released in several languages.
● Relevance judgement made by humans
● Measure: Prec., Rec., MAP, ...
<top><num>10.2452/51-GC</num><title>Oil and gas extraction found between the UK and the European Continent</title><desc>To be relevant documents describing oil or gas production between the UK and the European continent will be relevant</desc><narr>Oil and gas fields in the North Sea will be relevant.</narr></top>
34
Our evaluation results so far
● XLDB participated in GeoCLEF 2005, 2006 and 2007 (since its beginning).
● Main challenge: outperform IR.● So far, GIR failed to outperform IR
(same results for other groups).● What are we doing wrong?
approach? query processing? text mining? geographic ranking? scope assignment?
Only detailed evaluation will tell...
35
XLDB in GeoCLEF 2007
● IR outperformed by a IR/GIR mixed strategy.● Our GIR only approach still worse than IR! ● In progress:
– Detailed analysis for each of the 25 topics.– Thorough evaluation on each GIR
component: query parsing, query expansion, text mining, index & ranking.
37
PhD Grant
“Automatic Query Reformulation for Web Geographic Search Engines”
● PhD advisors: – Diana Santos (Linguateca/SINTEF ICT) – Mário J. Silva (University of Lisbon)
● Started 1st March 2007● 3 year scholarship
38
Milestones 2007/2008● 1st step: GeoCLEF experiments. ☑ done.● 2nd step: Evaluation.
– a) Detailed error analysis of each topic.– b) development of a good evaluation environment.
● 3rd step: Acquiring information about non-standard placenames. ☒planned.
– Example: SINTEF, Eiffel Tower, Ground Zero– implies new NER system with relation extraction.– Evaluate it in Second HAREM
● 4th Step: GeoCLEF 2008
ongoing.
39
2a) Kinds of possible errors
Off-line processing:● Document processing: failure to capture
geographic evidence (placenames and relations), failure to reason into ontology features.
● Document Geographic Signatures: erroneous ranking of irrelevant geographic evidence.
40
2a) Kinds of possible errors
On-line:● Query parsing: failure to detect relevant terms,
spatial relationship, feature and feature types.● Term Expansion: adding unrelated terms,
skipping related terms. ● Ontology Processing: failure to translate query
scope into a list of features ● Document Ranking: failure to bring relevant
documents to the top of the result list.
Analyse one topic and identify points of failure.
41
2b) In-house evaluation environment
● Input: topics/queries and human relevance judgements.
● Output: evaluation reports.● Desiderata:
– Measure each component separately.
– Logging and debugging capabilities for each point of failure.
– Invoke components under controlled scenarios.
42
Evaluation for different scenarios:● Topic by topic – debug a particular
component or a query.● By topic type – measure system's ability to
deal with a particular type of input.– See Santos & Chaves (2005) type classification
● Overall – measure global performance and contribution of each component to the overall results.
2b) In-house evaluation environment
43
3) non-standard placenames
● Ex: SINTEF, Eiffel Tower, Ground Zero● Develop a new NER system, REMBRANDT
– Detection of placenames in context● Why? Try “Norwegian Wood”
– Exploit Wikipedia's link structure, category and annotation metadata, and entity relation detection.
● Participation in Second HAREM (March 2008)
– Independent evaluation contest, with already manually annotated corpora
44
4) GeoCLEF 2008
● Improve GIR system with REMBRANDT● Use evaluation environment to measure
impact on performance● Participation in GeoCLEF 2008 (May/June 2008)
– Special sub-task for Wikipedia
45
For GeoCLEF 2008:
● Pilot subtask: use Wikipedia snapshot as document collection:– Richer resource for documents
with geographic content;– allows different types of topics.
● Documents about “Riots in Los Angeles” is more likely to be found in newspapers...
● ... but documents about “Vineyard areas near European rivers” more likely to be reported in Wikipedia.
46
Epilogue
● GIR systems are useful and challenging● Still on its early steps, still not clear what
are the most promising approaches● More message understanding & geographic
reasoning must be made carefully● How to measure progress: evaluation,
evaluation, evaluation.
Geographic IR Challenges
by Nuno Cardoso
Faculty of Sciences, University of Lisbon, LASIGE
Presentation held at SINTEF ICT, Oslo, Norway, 4th December, 2007
The End.
48
Entity Relation Extraction for placenames
● Inclusion: (ex: “Adelaide, Australia”): Australia suggests that Adelaide is a placename, not a proper name
– partOf(Adelaide, Australia)● Grounding: (ex: “SINTEF ICT is in the outskirts of
Oslo”) - now, I know that documents mentioning SINTEF ICT may have Oslo as a potential scope of interest (Ex: “research units near Oslo”).
– locatedIn(Sintef ICT, Oslo)● Alternative: (ex: “California as CA”)
– Alternative(CA, California)