northeast regional research center (nrrc) summer 2002 workshops june 11, 2002 mitre mark maybury...

Northeast Regional Research Center (NRRC) Summer 2002 Workshops

June 11, 2002MITRE

Mark [email protected]

Copyright © 2002 The MITRE Corporation. All rights reserved.

Human Science Question Answering How long it takes the Earth to orbit the sun?

One year. 54% correctly answered Do antibiotics kill viruses as well as bacteria?

False. 51% Human beings developed from earlier species of animals.

True, according to the theory of evolution, which is accepted by the majority of scientists, but not by many religious leaders. 53%.

The earliest humans lived at the same time as the dinosaurs.

False. Dinosaurs died off millions of years before humans appeared. 48%. Do lasers work by focusing sound waves?

False. Lasers focus light. 45% The universe began with a huge explosion.

True, according to the "Big Bang" theory widely accepted by scientists, but dismissed by some religious leaders. 33%

Answer % Correct


Mission and Goals

ARDA - Revolutionary not evolutionary- Well-defined goals, measurable results, scientific methodology

NRRC Goals- Scientific results with + impact on intelligence problems- Engage regional experts (.com, .edu, .org)- Reinforce ARDA thrusts- Infuse technology into government workforce- Transfer technology to and from industry

The NRRC is sponsored by ARDA, a US Government entity which sponsors and promotes research of import to

the IC which includes but is not limited to the CIA, DIA, NSA, NIMA and NRO.


The NRRC Management Team

Executive Director – Dr. Mark Maybury

Program Administrator – Bev Nunan

Administrative Assistant – Paula MacDonald

And other specialists providing infrastructure, subcontracting, security support and technical

expertise

Penny Lehtola (PM), Rick Steinheiser (COTR)

Deputy Program Manager – David Day

Program Manager – Penny Chase

Speaker Series Administrator – Pam Davis


Executive Committee (EC) Members

INDUSTRY Joe Marks, MERL Salim Roukos, IBM Ralph Weischedel, BBN

ACADEMIA Liz Liddy, Syracuse Stephen Kosslyn, Harvard George Cybenko, Dartmouth Gene Charniak, Brown Bruce Croft, UMass George Miller, Princeton Kathy McKeown, Columbia

SENIOR GOVERNMENT EXPERTS Kelcey Allwein, DIA John Donelan, USGC Steve Dennis, NSA Curt Boylls, NSA

GOVERNMENT (ex officio) John Prange, AQUAINT PM Rick Steinheiser, COTR Penny Lehtola, ARDA REP PM

MITRE (ex officio) Mark Maybury, NRRC Exec Dir Penny Chase, NRRC PM


Summer 2002 Workshop

Focus: Advanced QUestion and Answering for INTelligence (AQUAINT) program

Unsolved Q&A Challenges:- Temporal Analysis - Generating a sequence of events

along an evolving timeline; resolving temporal references across a series of sources.

- Multiple Perspectives - Handling multiple answers to questions from multiple sources with different (e.g., political, operational) perspectives.

- Re-use of accumulated knowledge - Maintaining prior knowledge (questions and answers) to help answer the same question later.

- Habitability - How can a question & answer system tell a user what it can do and fail gracefully.


Program Committee (PC) Members Team Leads

- James Pusteyovsky (Brandeis) - Temporal- Jan Wiebe (Univ of Pittsburgh) - Multiperspectives- Marc Light (MITRE) and Abraham Ittycheriah (IBM) - Reuse

Technical Experts/Advisors- Ed Hovy (ISI)- Kathy McKeown (Columbia)- Bruce Croft (UMass)- Ralph Weischedel (BBN)

Senior Government Experts- John Donelan, USGC; Steve Dennis, NSA- John Prange, AQUAINT PM

NRRC ExDir, NRRC PL Ex Officio: ARDA REP PM, COTR


Questions from Web logs

Reuse- could i learn more about the middle ages which is from 500-1500?

Multiple Perspectives - Where Can I find a Dr.'s opinion of Body Piercing?

- where can i find opinions about Christmas?- Are my opinions more democratic or republican?

Time - Is there life after death?- Will Jess Powley get her laundry done before the new millenium?


FY02 Current Accomplishments

1 Thrust Area and 4 Focus topics selected 3 Workshops underway (2 Large/1 Small)

- participation from industry, academia, and government- 4 graduate students, 1 high school student

Major startup issues addressed- proposal process and selection criteria- public release- web site- intellectual property- subcontracting- facilities customized/constructed (3K and 1M)- infrastructure and connectivity

Speaker series launched


Participants

20 from Academia -- Brandeis (3), University of Sheffield, University of Tübingen, University of Michigan (2), Georgetown University, Syracuse University, University of Rochester (2), University of Montreal , University of Pittsburgh (2), SUNY Buffalo, Cornell University (2), Ohio State University, Boston University (2)

11 from Industry -- IBM Research (2), LingoMotors (2), MetaCarta (2), Sun Microsystems, SRA, XEROX PARC, SabIR Research, BBN Technologies

Government -- SPAWAR, plus ARDA & analyst participation FFRDCs -- MITRE (3)


NRRC Products LREC QA Workshop Paper, QA Roadmap workshop Reuse Workshop

- Reuse Corpus: 221 single Qs (29 topics), 2130 multiple Qs (13 topics)- 100 examples of reuse annotated- Draft Final Report including QA reuse classification

Temporal - TERQAS- TimeML 0.2, annotation guideline 0.1, standards influence- TIMEBANK; algorithms- Report

Multiple Perspectives - MPQA- Perspective conceptualization- Opinion annotation scheme (phrases, sources)- Opinion corpus- Opinion recognition, clustering, evaluation- Report

completedin-progress

Research Roadmap of Question Answering 2003-2006Shared data, problems, test suites, evaluation at milestones

2003

2006

Resources(Development, Evaluation)

Methodolgies for QA Analysis Resource &Evaluation

Methods & AlgorithmsSystems

(Performance & Eval)

TIMEBANK

Task Modeling

Evaluation

MeasurableProgress

ComposableToolkit for QA

Productive, Quality QA

Resource Selection

TIMEML

Semi-structured Data

QA as Planning

Multimodal QA

Related Fields: HPKB, TDT, DB, Virtual Ref Desk, User Modeling

Constrained QA(Resource/Solution)

FeasibilityTesting

RequirementsDetermination

Empirical Studies

USC/ISIS Question Typology

TREC QAtrec.nist.gov/data/qa.html

Collect QA Logs

START, FaqFinder,

Ionaut, QANDA

Web Services(e.g., Google API)

Public Taxonomies (e.g., OpenDirectory in RDF)

Create QA Sets

Multisessional QA(including change

detection)

Temporal QAQA Reuse

Collaborative QA

Stereotypical and Indivdualized QA

Multilingual QA

Interactive Dialog

Personalized QA

Task Model

Reuse acrosssessions

User Typology

“Perspective BANK“

Question/AnswerTypologies

Factoid Questions

How Questions

Why Questions

What If Questions

InteroperabilityQuality Assurance

Script/Template Answers

Fact Answers

Multimodal Answers

Answer Toplogy

User Expectations

Wizard of Oz QA Sets

Answer Fusion

Answer Justification

Crosslingual QA

Speed Limit

Inference Speed Limit

Robust NLP

Reusable Test Collection

Copyright © 2002 Mark Maybury


Officer asks “Can I see your license?”Top Ten Worst Answers:

10. Sorry, Officer, I didn't realize my radar detector wasn't plugged in.

9. Hey, you must've been doin' about 125 mph to keep up with me. Good job!

8. Do you know why you pulled me over? Okay, just so one of us does.

7. I thought you had to be in relatively good physical condition to be a police officer.

6. Gee, Officer! That's terrific. The last officer only gave me a warning, too!

5. I was trying to keep up with traffic. Yes, I know there are no other cars around. That's how far ahead of me they are.

4. Are You Andy or Barney?

3. Aren't you the guy from the Village People?

2. You're not gonna check the trunk, are you?

1. I can't reach my license unless you hold my beer. (OK in Texas)


Upcoming Events June 6 Midterm June 7 Analytic Disciplines Lecture Day (see nrrc.mitre.org) July 22 Final Report Out AAAI Spring Symp Future Directions in Question Answering

Monthly Speaker Series at MITRE and via VTC 25 July National Intelligence Analysis and Production

John Gannon, former ADCI A&P 9 Aug In-Q-Tel

Mr. Gilman G. Louie, President and CEO 17 Sept Technology for Intelligence

Dr. John Phillips, Chief Scientist, CIA 22 Oct The Intelligence Analysis Process - Jim Simon, ADCI 17 Dec Multi-INT Tasking, Processing, Exploitation, and Dissemination

Lt Gen Jim Clapper, USAF (Ret), Dir/NIMA 19 Nov Measurement and Signatures Intelligence Analysis (MASINT)

John Morris, former CMO director - SECRET 23 Jan ‘03 The Future of Open Source Analysis

Gina Genton, Director, CIA/FBIS 23 Feb ‘03 Intelligence Analysis

Wesley Wark, Fellow, Munk Centre for Int Studies, U. of Toronto


nrrc.mitre.org

northeast regional research center (nrrc) summer 2002 workshops june 11, 2002 mitre mark maybury...

Documents