aquaint testbed john aberdeen, john burger, conrad chang, scott mardis the mitre corporation ©...
TRANSCRIPT
![Page 1: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/1.jpg)
AQUAINT Testbed
John Aberdeen, John Burger,Conrad Chang, Scott Mardis
The MITRE Corporation
© 2002, The MITRE Corporation
![Page 2: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/2.jpg)
Testbed Activities• Testbed
– Provide access to Q&A systems on classified data.– Solicit user feedback and conduct user studies.
• Testweb– Provide public access to a broad variety of Q&A capabilities.– Evaluate systems and architectures for inclusion in testbed.
• User Studies– Obtain feedback from users on utility and usability
• Architecture support
![Page 3: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/3.jpg)
Testbed
• Classified network (ICTESTNET)– access to users, data, scenarios will be restricted
• Evaluate systems prior to installation– Testweb is a good first step – MITRE installations are more than rehearsal
![Page 4: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/4.jpg)
Testbed Progress
• MITRE installations– QUIRK (CYCorp/IBM) – Javelin (CMU)– LCC
• Assessing– Portability to the IC environment– Maintainability– Features– Resources– Configuration
![Page 5: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/5.jpg)
Deployment Issues
• Restrictive version requirements• Fixed pathname/hostname issues• Configure scripts
• Remote debugging (installation, configuration) is inefficient
• Differences in machine configuration, environment• Unknown or unrealized dependencies• Schedule conflicts
• Firewalls and other security issues
![Page 6: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/6.jpg)
Developing a Testbed Vision
• USNR Analyst Task• MITRE gains first-hand knowledge about:
– analyst’s working environment• tools• methods of analysis
– nature of analyst’s tasking– long-term analysis goals
• MITRE obtains expert feedback about:– suitability of QA tools for analysis tasks– representativeness of CNS scenarios– methodology for further tool assessment– methodology for future user studies
![Page 7: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/7.jpg)
Plan for USNR Visit• Discussions of analyst's history with analysis tasks• Hands-on presentation of QA and Search tools• MITRE and USNR analyst jointly work through 1 to 2 CNS
scenarios using QA tools, search tools, telephone, etc.• Analyst formally works through 1 to 3 more CNS
scenarios on their own– MITRE monitors the activity through logging and survey
instruments– The analyst provides feedback so adjustments can be provided as
necessary
• The analyst and MITRE will jointly analyze the results• MITRE and the analyst develop a draft test plan
![Page 8: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/8.jpg)
Testweb
Q/Asystem
CNS
TREC2002
Javelin, LCC,Qanda, TextMap
Q/APortal/Demo
Q/A repository
Google API
Othercollections
IRservice
Clarityservice
UsersQ/APortal/Demo
•Loose federation of systems
•Simple services over TCP/HTTP/SOAP
![Page 9: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/9.jpg)
Q&A Demo
![Page 10: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/10.jpg)
CNS Data
http://aquaint.mitre.org/cns
![Page 11: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/11.jpg)
UMASS Clarity
• Many thanks to Steve Cronen-Townsend & Bruce Croft• Clarity measures …• Clarity service
– SOAP interface– Access via Google Java client – Data sets: AQUAINT CNS, TREC-11 (soon)
• User interface– For demonstration, simple output of clarity score– To come, low clarity warning
![Page 12: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/12.jpg)
CNS Search w/ UMASS Clarity
http://aquaint.mitre.org/clarity
![Page 13: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/13.jpg)
IR Service
• Utilizing Google SOAP API• Tied SOAP to Lucene search• Data sets: AQUAINT CNS, TREC-11• Access via Google Java client
![Page 14: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/14.jpg)
Q/A Architecture
Meeting: April 22, 2003 @ IBM in Hawthorne, NY
Eric Nyberg (CMU), David Ferrucci & Chris Welty (IBM), Scott Mardis & John Burger (MITRE)
Topics:– Javelin– PIQUANT– Catalyst– UIMA– AQUAINT Testbed
![Page 15: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/15.jpg)
Some Directions
• Architecture is important– Enable collaboration– May be task dependent
• Common Components/Subsystems– Question analysis– Query formulators– IR– Taggers – Knowledge sources
![Page 16: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/16.jpg)
MITRE’s Mission on Architecture
• Encourage collaborations– Promote regularization of interfaces
• Facilitate broader use of best components
• Advocate for users (IC analysts) and user tasks
• Coordinate with Ex. Comm.
![Page 17: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/17.jpg)
MITRE’s Roles on Architecture
Services
CommunitySupport
APIs, Standards
InfrastructureResources
Data Resources
![Page 18: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/18.jpg)
MITRE’s Roles on Architecture
• User logs, including Q&A, simple search, documents viewed, etc.
• Extended WN & XWN APIs
• Query formulation API• Corpora URIs
Services,Libraries
APIs,Standards
InfrastructureResources
Data Resources
APIs, Standards:CommunitySupport
![Page 19: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/19.jpg)
MITRE’s Roles on Architecture
• IR• Clarity• Unit conversions ?• Time normalization ?
CommunitySupport
InfrastructureResources
Data Resources
Services, Libraries:
APIs,Standards
Services,Libraries
![Page 20: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/20.jpg)
MITRE’s Roles on Architecture
• Modified CNS corpus• Ontology library ?
CommunitySupport
InfrastructureResources
Data Resources:
APIs,Standards
Services,Libraries
DataResources
![Page 21: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/21.jpg)
MITRE’s Roles on Architecture
• Middleware tools ?
Infrastructure Resources:
APIs,Standards
Services,Libraries
DataResources
CommunitySupport
InfrastructureResources
![Page 22: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/22.jpg)
MITRE’s Roles on Architecture
• Tool website (index)• Wiki for architecture ?• Mailing lists
InfrastructureResources
Community Support:
APIs,Standards
Services,Libraries
CommunitySupport
DataResources
![Page 23: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/23.jpg)
Component Offerings
• LCC– Ontology tools (development & browsing)– Extended WordNet (XWN)
• ISI– Question analysis (QTarget)– Question paraphraser– Query formulator– Web-based IR– Answer identifier (extractor)
![Page 24: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/24.jpg)
Component Offerings (cont’d)
• IBM– NE tagger (Resporator – 80 types)
• CyCorp– Answer sanity checking
– Taxonomy (well-developed for CNS/WMD)– Unit conversion– Cyc concept paraphrases
• UMASS– Clarity
![Page 25: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/25.jpg)
Component Needs ?
• IR subsystems• User context & history representation• More Ontologies & Tools• Answer combination• Entity normalization (time, location, organization)• Structured data services (DBs)• Other media
![Page 26: AQUAINT Testbed John Aberdeen, John Burger, Conrad Chang, Scott Mardis The MITRE Corporation © 2002, The MITRE Corporation](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56649f285503460f94c411a0/html5/thumbnails/26.jpg)
Many Faces of AQUAINT• Leverage all of AQUAINT’s successes• Expose a greater range of interfaces to analysts
– They may find something they can use now– Make the Q/A system more transparent
• Exploiting the middle ground– Queries that are more complex than boolean search, but more
structured than free natural language questions