will question answering become main theme of ir research? · natural language dialogue 1970 1990...
TRANSCRIPT
![Page 1: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/1.jpg)
Will Question Answering Become Main Theme of IR Research?
Hang Li
Huawei Noah’s Ark Lab
AIRS 2016 Beijing, China Dec 1, 2016
![Page 2: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/2.jpg)
Outline
• Question Answering Will Become Main Paradigm of Information Access
• Well-Studied Problems in Question Answering
• Human Information Retrieval vs Computer Information Retrieval
• New Problems in Question Answering
• Research on Question Answering at Noah’s Ark Lab
![Page 3: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/3.jpg)
New Paradigm in Information Retrieval
Library Search
Web Search
Natural Language Dialogue
1970
1990
2010
![Page 4: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/4.jpg)
Information Access through Natural Language Dialogue
• Multi-turn dialogue • Goal: task completion, mostly information access • Evaluation: completion / cost • Including traditional search and question answering as special cases
……
![Page 5: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/5.jpg)
Example One: Hotel Booking on Smartphone
P: How may I help you? U: I'd like to book a hotel room for tomorrow. P: For how many people? U: Just me. What is the total cost? P: That would be $120 per night. U: No problem. Book the room for one night, please.
![Page 6: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/6.jpg)
Example Two: Auto Call Center
• U: hello • H: hello, how can I help you? • U: can you tell me how to find
ABC software? • H: please go to this URL to
download • U: how to activate the software? • H: please see this document
![Page 7: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/7.jpg)
Outline
• Question Answering Will Become Main Paradigm of Information Access
• Well-Studied Problems in Question Answering
• Human Information Retrieval vs Computer Information Retrieval
• New Problems in Question Answering
• Research on Question Answering at Noah’s Ark Lab
![Page 8: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/8.jpg)
Well-Studied Problems in Question Answering
• Factoid Question Answering
• Community Question Answering
• Retrieval based Approach • Language Analysis based Approach • Hybrid Approach
![Page 9: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/9.jpg)
Factoid Question Answering
Documents Q: Who invented mobile phone?
A: Martin Cooper. Q: Where is Huawei Technologies based?
A: Huawei’s headquarter is in Shenzhen China
Question Answering System
Question Answer
![Page 10: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/10.jpg)
Factoid Question Answering System
Index of Documents
Matching
Answer Aggregating & Ranking
Question
Passage Retrieval
Retrieved Passages
Ranked Answers
Matching Models
Aggregating & Ranking
Model
Online
Offline
Best Answer
Matched Answers
Classification and Analysis
Models
Question Classification
&Analysis
![Page 11: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/11.jpg)
Community Question Answering
Frequently Asked Questions
Q: Why is sky blue?
A: A clear cloudless day-time sky is blue because molecules in the air scatter blue light from the sun more than they scatter red light. When we look towards the sun at sunset, we see red and orange colors because the blue light has been scattered out and away from the line of sight.
Question Answering System
Question Answer
![Page 12: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/12.jpg)
Community Question Answering System
Index of Question and Answer Pairs
Matching
Answer Ranking
Question
Question & Answer Retrieval
Retrieved Questions and Answers
Ranked Answers
Matching Models
Ranking Model
Online
Offline
Best Answer
Matched Questions and Answers
![Page 13: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/13.jpg)
Outline
• Question Answering Will Become Main Paradigm of Information Access
• Well-Studied Problems in Question Answering
• Human Information Retrieval vs Computer Information Retrieval
• New Problems in Question Answering
• Research on Question Answering at Noah’s Ark Lab
![Page 14: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/14.jpg)
Goal of Information Retrieval = Making Computer Extension of Brain
for Information and Knowledge Storage
![Page 15: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/15.jpg)
Human Brain
![Page 16: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/16.jpg)
Encoding, Storage, and Retrieval of Information in Human Brain
Modified from Frank Longo 2010
Sensory Register • vision 1 sec • auditory 5 sec
Short-term Memory • 18-30 sec • 7±2 • displacement
Long-term Memory
• synapse mod • limitless cap • declarative and non-declarative
Attention Consolidation
Retrieval Central Executive Unit • planning • conscious thought
visual auditory kinaesthetic olfactory gustatory
![Page 17: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/17.jpg)
Human Information Retrieval
• Hippocampus (short term memory) Cerebral Cortex (long term memory)
• Information and knowledge is stored in long term memory
• Hebb’s hypothesis: fire together wire together
• Consolidation: create connections between neurons (patterns) in long term memory
• Retrieval: activate related neurons through connections in long term memory
![Page 18: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/18.jpg)
Information Retrieval in Human Brain v.s. Information Retrieval on Computer
Brain Computer
Computing paradigm
Parallel processing Sequential processing
Capability Mathematically Ill-posed problems
Mathematically well-formed problems
Representation of information
Represented in neurons and synapses
Represented by digitized symbols, numbers, data
structures
Language to encode information
Mentalese (hypothetical language of thought, cf. Pinker)
Mainly in natural language
Means of retrieval
Association of neurons IR models
![Page 19: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/19.jpg)
Strategy in Information Retrieval
• Simplify the process
• Avoid the great challenge of language understanding
• Computer can “pretend” to understand language
![Page 20: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/20.jpg)
Simplified Problem Definition - Question Answering
Generation
Decision
Retrieval
Inference
Understanding
Analysis
Generation
Retrieval
Analysis
Question answering, including search, can be practically performed, because it is simplified
![Page 21: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/21.jpg)
This Strategy Works Well, But Sufficiently Well
How Can We Gradually Make Progress?
![Page 22: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/22.jpg)
Outline
• Question Answering Will Become Main Paradigm of Information Access
• Well-Studied Problems in Question Answering
• Human Information Retrieval vs Computer Information Retrieval
• New Problems in Question Answering
• Research on Question Answering at Noah’s Ark Lab
![Page 23: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/23.jpg)
New and Open Problems in Question Answering
• Question Answering from Knowledge Base
• Generative Question Answering
• Robust Question Answering
• Interactive Question Answering
• Question Answering from Multiple Sources
• Inference in Question Answering
• … …
![Page 24: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/24.jpg)
Question Answering from Knowledge Base
![Page 25: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/25.jpg)
Question Answering from Knowledge Base
• Answers exist in relational database, knowledge graph, and are in form of structured data
• Related to semantic parsing
Q: How tall is Yao Ming? QA System
Name Height Weight
Yao Ming 2.29m 134kg
Liu Xiang 1.89m 85kg
2.29m
![Page 26: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/26.jpg)
Semantic Parsing
Q: What is the largest prime less than 10? A: 7
Liang 2016
• Executor: execute command based on logic form and context
• Grammar: set of rules for creating derivations based on input and context
• Model: model for ranking derivations with parameters
• Parser: find most likely derivation under learned model
• Learner: learn parameters of model from data
czy
),( cxD
),|( cxdP
*d
n
iiii ycx1
),,(
![Page 27: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/27.jpg)
Challenges in Question Answering from Knowledge Base
• Synonymy and polysemy of terms in question and knowledge base items
• Complicated structure of question and knowledge base
• Complicated matching between question and knowledge base items
![Page 28: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/28.jpg)
Generative Question Answering
![Page 29: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/29.jpg)
Generative Question Answering
• Generation of natural answer
• Might be similar to human information retrieval
QA System
Knowledge Base
Question Answer
Retrieval Module
Language Module
![Page 30: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/30.jpg)
Challenges in Generative Question Answering
• Generating answer in more appropriate way according to question
– Q: How high is Mount Everest?
– A: Mount Everest is 8,848 meter high.
v.s.
– Q: What is the height of Mount Everest in feet?
– A: It is 29,029 feet.
![Page 31: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/31.jpg)
Robust Question Answering
![Page 32: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/32.jpg)
Robust AI
• AI systems must produce accurate confidence values – Should “abstain” when they are uncertain
• AI systems should explain their reasoning – Help software engineers and end users
develop appropriate trust
• AI systems should be robust to incorrect design assumption – “Unknown unknowns”
• We need verification and validation methodologies for AI systems – Automated “adversarial” test?
Thomas Dietterich
![Page 33: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/33.jpg)
Robust AI (cont’)
Known Knowns:
“I know that I know”; can fully control
Known Unknows:
“I know that I do not know”; can abstain
Unknown Knowns:
“I do not know that I know”; does not matter
Unknown Unknowns:
“I do not know that I do not know”; should avoid!
Robust AI can cope with
![Page 34: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/34.jpg)
IBM Watson • Beat human champion at quiz jeopardy
• Designed to answer 70 percent of questions, with greater than 80 percent precision, in 3 seconds or less.
![Page 35: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/35.jpg)
Architecture of DeepQA System
Evaluating confidence of answers
![Page 36: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/36.jpg)
Robust Question Answering
• Need verify the correctness of answer
• Can abstain from answering question, if not confident
QA System
Knowledge Base
Question Answer
![Page 37: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/37.jpg)
Challenges in Robust Question Answering
• Identifying incorrect candidate answers, due to failure of processing, errors in knowledge base,
– e.g., “Yao Ming is .229cm tall”
• Giving balanced summary of answers, if there are contradictory results.
– e.g., “Is it safe to use talcum powder? “
![Page 38: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/38.jpg)
Interactive Question Answering
![Page 39: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/39.jpg)
Interactive Question Answering
QA System Knowledge Base
Question
Answer
• QA System can
1. Confirm intent of question, help formulate question
2. Give summary of answers (e.g., no answer, many answers)
3. Allow users to ask additional questions, if user needs more information.
![Page 40: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/40.jpg)
Challenges in Interactive Question Answering
• Understanding intent of user
• Evaluation of retrieval result
• Mangement of dialogue, e.g., “Where is it?”
Interactive Question Answering = Task-oriented Multi-turn Dialogue
![Page 41: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/41.jpg)
Outline
• Question Answering Will Become Main Paradigm of Information Access
• Well-Studied Problems in Question Answering
• Human Information Retrieval vs Computer Information Retrieval
• New Problems in Question Answering
• Research on Question Answering at Noah’s Ark Lab
![Page 42: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/42.jpg)
Question Answering from Relational Database
Yin et al. 2016
![Page 43: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/43.jpg)
Question Answering from Relational Database
Relational Database
Q: How many people participated in the mmgame in Beijing?
A: 4,200 SQL: select #_participants, where city=beijing
Q: When was the latest game hosted? A: 2012 SQL: argmax(city, year)
Question Answering System
Q: Which city hosted the mmlongest Olympic game mmbefore the game in Beijing? A: Athens
Learning System
year city #_days #_medals
2000 Sydney 20 2,000
2004 Athens 35 1,500
2008 Beijing 30 2,500
2012 London 40 2,300
![Page 44: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/44.jpg)
Neural Enquirer
• Query Encoder: encoding query • Table Encoder: encoding entries in table • Five Executors: executing query against table
Conducting matching between question and database entries multiple times
![Page 45: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/45.jpg)
Query Encoder and Table Encoder
• Creating query embedding using RNN • Creating table embedding for each entry using DNN
RNN
…
Query Representation
Query
DNN
Entry Representation
Field Value
Table Representation
Query Encoder Table Encoder
![Page 46: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/46.jpg)
Executors
• Five layers, except last layer, each layer has reader, mannotator, and memory • Reader fetches important representation for each row,
me.g., city=beijing • Annotator encodes result representation for each row, me.g., row where city=beijing
Select #_participants where city = beijing
![Page 47: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/47.jpg)
Experimental Results
• Experiment
– Olympic database
– Trained with 25K and 100K synthetic data
– Accuracy: 84% on 25K data, 91% on 100K data
– Significantly better than SemPre (semantic parser)
– Criticism: data is synthetic
25K Data 100K Data
Semantic Parser
End-to-End Step-by-Step Semantic Parser
End-to-End Step-by-Step
65.2% 84.0% 96.4% NA 90.6% 99.9%
![Page 48: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/48.jpg)
Question Answering from Knowledge Graph
Yin et al. 2016
![Page 49: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/49.jpg)
Question Answering from Knowledge Graph
(Yao-Ming, spouse, Ye-Li) (Yao-Ming, born, Shanghai) (Yao-Ming, height, 2.29m) … … (Ludwig van Beethoven, place of birth, Germany) … …
Knowledge Graph
Q: How tall is Yao Ming? A: He is 2.29m tall and is visible from space. (Yao Ming, height, 2.29m)
Q: Which country was Beethoven from? A: He was born in what is now Germany. (Ludwig van Beethoven, place of birth, Germany)
Question Answering System
Q: How tall is Liu Xiang? A: He is 1.89m tall
Learning System
Answer is generated
![Page 50: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/50.jpg)
GenQA
• Interpreter: creates representation of question using RNN
• Enquirer: retrieves top k triples with highest matching scores using CNN model
• Generator: generates answer based on question and retrieved triples using attention-based RNN
• Attention model: controls generation of answer
Short Term Memory
Long Term Memory
(Knowledge Base)
How tall is Yao Ming?
Interpreter
Enquirer
Generator
He is 2.29m tall
Attention Model
Key idea: • Generation of answer based on question and retrieved kresult • Combination of neural processing and symbolic rprocessing
![Page 51: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/51.jpg)
Enquirer: Retrieval and Matching
• Retaining both symbolic representations and vector representations • Using question words to retrieve top k triples • Calculating matching scores between question and triples using CCNN model • Finding best matched triples
(how, tall, is, liu, xiang)
< liu xiang, height, 1.90m> < yao ming, height, 2.26m> … … <liu xiang, birth place, shanghai>
Retrieved Top k Triples and Embeddings
Question and Embedding Matching
![Page 52: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/52.jpg)
Generator: Answer Generation
• Generating answer using attention mechanism • At each position, a variable decides whether to ggenerate a word or use the object of top triple
2s 3s
3c
He is
03 z 13 z
2.29m tall
< yao ming, height, 2.29m>
3z
How tall is Yao Ming ?
3y2y
… o
…
…
…
3y
![Page 53: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/53.jpg)
Experimental Results
• Experiment
– Trained with 720K question-answer pairs (Chinese) associated with 1.1M triples in knowledge-base, data is noisy
– Accuracy = 52%
– Data is still noisy
Question Answer Who wrote the Romance of the Three Kingdoms?
Luo Guanzhong in Ming dynasty
correct
How old is Stefanie Sun this year?
Thirty-two, he was born on July 23, 1978
wrong
When will Shrek Forever After be released?
Release date: Dreamworks Pictures
wrong
![Page 54: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/54.jpg)
Take-away Messages
• Question Answering Will Become Main Paradigm of Information Access
• Many New and Challenging Problems in Question Answering, Including – Question Answering from Knowledge Base
– Generative Question Answering
– Robust Question Answering
– Interactive Question Answering
• Deep Learning Is Powerful Tool for Question Answering
![Page 55: Will Question Answering Become Main Theme of IR Research? · Natural Language Dialogue 1970 1990 2010 . Information Access through ... Factoid Question Answering System Index of Documents](https://reader033.vdocuments.mx/reader033/viewer/2022053015/5f16570e39e25118cf6e3df4/html5/thumbnails/55.jpg)
References 1. Frank Longo, Learning and Memory: How It Works and When It Fails. Stanford
Lecture. 2010.
2. Steven Pinker. The Language Instinct: How the Mind Creates Language. 1994.
3. Percy Liang. Learning Executable Semantic Parsers for Natural Language Understanding. Communications of the ACM, Vol. 59 No. 9, Pages 68-76, 2016.
4. Thomas Dietterich, Steps toward Robust Artificial Intelligence, AAAI 2016.
5. Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A.A., Lally, A., Murdock, J.W., Nyberg, E., Prager, J. and Schlaefer, N., 2010. Building Watson: An overview of the DeepQA project. AI magazine, 31(3), pp.59-79.
6. Konstantinova, N. and Orasan, C., 2012. Interactive question answering. Emerging Applications of Natural Language Processing: Concepts and New Research, pp.149-169.
7. Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, Xiaoming Li. Neural Generative Question Answering. Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI’16), 2972-2978, 2016.
8. Pengcheng Yin, Zhengdong Lu, Hang Li, Ben Kao. Neural Enquirer: Learning to Query Tables with Natural Language. Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI’16), 2308-2314, 2016.