javelin project briefing 1 aquaint year i mid-year review language technologies institute carnegie...

1AQUAINT Year I Mid-Year Review

JAVELIN Project Briefing

Language Technologies InstituteCarnegie Mellon University

Status Update forMid-Year Program Review

May 15, 2002



Eric Nyberg

Jamie Callan Jaime Carbonell Bob Frederking

Alon Lavie

Teruko Mitamura Dave Svoboda

Jeongwoo Ko

Michael DugganKrzysztof Czuba

Vasco Pedro

Yifen HuangLaurie Hiyakumoto Lucian Lita

Current Team



Research Objectives• QA as Planning

– Create a general QA planning system– How should a QA system represent its chain

of reasoning?

• QA and Auditability– How can we improve a QA system’s ability

to justify its steps?– How can we make QA systems open to

machine learning?



Research Objectives [2]

• Utility-Based Information Fusion– Perceived utility is a function of many different

factors– Create and tune utility metrics, e.g.:

U = Argmax k [F (Rel(I,Q,T), Nov(I,T,A), Ver(S,Sup(I,S)), Div(S), Cmp(I,A)), Cst(I,A)]

I: Info item, Q: Question, S: Source, T: Task context, A: Analyst

- relevance- novelty- veracity, support- diversity- comprehensibility- cost



Project Status Summary

• Started in November 2001• Fully staffed in December 2001• Basic end-to-end architecture

operational w/limited coverage• Working on question types for TREC

evaluation• Working to integrate advance JAVELIN

capabilities (e.g., Planner)



Research Plan & Progress• Develop end-to-end system

– Architecture, Planner, Repository– Individual QA Modules

First version being tested now• Evaluation:

– English queries– English, Chinese and

Japanese documents

Starting with English only



QA Evaluation in the Large• Information-Based Evaluation

current focus, for TREC QA track• Utility-Based Evaluation

when planner is integrated• Architectural Evaluation

future task: specify and analyze the properties of the design

JAVELIN team participatedin LREC ’02 workshop onQA systems evaluation



JAVELIN Basic Architecture

All objects created orretrieved are storedcentrally for reuse

Details of moduleimplementation arehidden

Planner is independentfrom the particular QAmodules being used

FirstEnd-to-EndSystemCompleted

Next Step:Integrate Planner



JAVELIN Data Flow

Merged



Revised Architecture

DataRepository

JAVELIN GUI

QuestionAnalyst

AnswerGenerator

RetrievalStrategist

ExecutionManager

...search engines &document collections

process historyand results

operator (action) models

RequestFiller

PlannerDomainModel



Module Integration

• Via XML DTDs for each object type• Modules use simple XML object-passing

protocol built on TCP/IP• Execution Manager takes care of

checking objects in/out of Repository



End-to-End System

• User Interface• Repository• Execution Manager• Question Analyzer• Retrieval Strategist• Request Filler• Answer Generator• Planner

Java clientMicrosoft SQL ServerJava server/IIMModule server/KANTOOModule server/InqueryModule serverModule serverModule server



Evaluation: Current

• Execution Manager can run in “lights out” batch mode

• Nightly tests on different test suites (starting with TREC QA)

• Results include scores and logs for debugging

• Working up to full TREC QA evaluation for TREC ‘02



Sample TestResults Page



Sample Results



Sample LogFile Excerpt



JAVELIN User Interface



Repository ERD

(Entity Relationship Diagram)



Short-Term Goals

• Prepare for TREC QA evaluation!• Integrate Planner with end-to-end

system• Acquire Japanese and Chinese resources



Post-TREC Tasks

• Execution Manager & UI:– Support for interactive dialog– Extended evaluation capability– Ability to re-run prior question with

modifications

• Repository:– Support for answer justification– Support for end-user repository search



Post-TREC Tasks [2]

• Planner:– Ablation studies– Advanced question types– Planning parameter variations

• Question Analyzer:– Broaden coverage of question parsing– Support for additional question types– Produce interlingua for request object



Post-TREC Tasks [3]

• Retrieval Strategist:– Add Japanese and Chinese document

collections, relational DB support– Add Google as a document collection– Support for answer verification– Switch to Lemur Toolkit

• Request Filler:– Reference resolution– Deeper NL analysis



Post-TREC Tasks [4]

• Answer Generator:– Combining more evidence types (predicate

argument structure, event boundaries)– Extended answer types (hypotheticals)



Gathering Evidence in Answer Generation

• Q: Name all the bills that were passed during the Bush administration.

• Not likely to find passages mentioning `bill’, `pass’, `Bush administration’.

• When was Bush administration??

• `Symbolic’ QA: look for explicit answer in collection, might not be present.

• `Statistical’ QA: look at distribution of documents mentioning Bush administration.

• Combining evidence of different sorts!



Gathering Evidence [2]

• Can we figure out if Bush administration was around when document was written?

• Look at tense/aspect/wording.

• Forward time references – Bush administration will do something

• Backward time references – Bush administration has done something

• Hypothesis: – Backward time references provide information about onset of

event;– Forward time references provide information about end of event.



Clustering Evidence

• Bush administration forward references

AdministrationchangeEvent end

Time stamps

#docsmentioningBush adm.on givenday



Clustering Evidence [2]

• Bush administration backward references

#docsmentioningBush adm.on given day

Time stamps

AdministrationchangeEvent onset

javelin project briefing 1 aquaint year i mid-year review language technologies institute carnegie...

Documents