question answering over freebase with multi-column convolutional neural networks · 2017. 9....

Question Answering over Freebase with Multi-Column Convolutional Neural Networks

Li Dong1, Furu Wei2, Ming Zhou2, Ke Xu11SKLSDE, Beihang University, Beijing, China

2Microsoft Research, Beijing, China

Freebase Large-scale knowledge base A rich resource to answer open-domain questions

Challenge natural language questions ~ structured semantics of Freebase How to bridge the gap?

Question Answering over Freebase

when did Avatar release in UK 2009-12-17Question: Answer:

Semantic parsing (Berant et al., 2013; Bao et al., 2014; etc.) Question Formal Meaning Representation Structured Queries Answer Example

Utterance: Which college did Obama go to Logical form: (and (Type University) (Education BarackObama)) Denotation: Occidental College, Columbia University

Challenges Huge search space Lexical triggers

Mainstream Methods (1/2)

Example is borrowed from the website of SEMPRE

Information extraction over knowledge base 1. Retrieve candidate answers from Freebase 2. Extract features 3. Classification / Ranking

Mainstream Methods (2/2)

Question

Candidate Answers

Features

Classifier

Correct Answer

(Yao and Van Durme, 2014)

Question

Candidate Answers

Ranking score

Correct Answer

(Bordes et al., 2014a; 2014b)

Sum of Word Embeddings

Candidate Embedding

Question answering -> Constraint matching Answer type, answer path (relation), answer context

Question understanding with convolutional neural networks

Proposed Method

Question

Candidate Answers

Type

Multi-columnConvolutional Neural Networks

Relation Context Type Relation Context

Matching Score

Ranker Answer

when did Avatar release in UK

Convolutional Layer

Max-Pooling Layer

Shared Word Representations

Avatarm.0bth54

James Cameronm.03_gd

film.film.directed_by

type.object.type

people.person

film.producer

type.object.typem.09w09jk

film.film.release_date_s

type.object.typefilm.film_regional_release_date

United Statesof Americam.09c7w0

film.film_regional_release_date.film_release_region

film.film_regional_release_date.release_date

2009-12-18

datetime

value_type

m.0gdp17zfilm.film.release_date_s

type.object.type

film.film_regional_release_date

United Kingdomm.07ssc



2009-12-17

datetime

value_type

Score Layer

Score

+ +Dot Product

Answer Path

Answer Context

Answer Type

Model Overview


Convolutional Layer

Max-Pooling Layer


Avatarm.0bth54



type.object.type

people.person

film.producer







2009-12-18

datetime

value_type


type.object.type





2009-12-17

datetime

value_type

Score Layer

Score

+ +Dot Product

Answer Path

Answer Context

Answer Type

Model Overview

Avatar.0bth54

cted_byfilm.film.release

_date_s


type.object.type





2009-12-17

datetime

value_type

Answer Context

Answer Type

Answer Path

Candidate

Learn vector representations for candidate answers (Bordes et al., 2014a; Bordes et al., 2014b)

Answer path relations between the candidate node and the entity asked in

question 𝑎𝑎𝑎𝑎𝑎𝑎 𝒓𝒓𝟏𝟏, 𝒓𝒓𝟐𝟐, … ,𝒓𝒓𝒏𝒏 : average of relation embeddings

Answer contextAnswer type

Embedding Candidate Answers

Avatarm.0bth54


film.film.directed_byfilm.film.release

_date_s

film.film region


type.object.type





2009-12-17

datetime

value_type

Answer Path

Answer Context

Answer Type

Candidate

Asked entity


Answer context 1-hop entities and relations connected to the answer path 𝑎𝑎𝑎𝑎𝑎𝑎 𝒄𝒄𝟏𝟏, 𝒄𝒄𝟐𝟐, … , 𝒄𝒄𝒏𝒏 : average of context entity and relation embeddings

Answer pathAnswer type


Avatarm.0bth54



_date_s

film.film region


type.object.type





2009-12-17

datetime

value_type

Answer Path

Answer Context

Answer Type


Answer type common.topic.notable_types, value type (e.g., float, string, datetime) 𝑎𝑎𝑎𝑎𝑎𝑎 𝒕𝒕𝟏𝟏, 𝒕𝒕𝟐𝟐, … , 𝒕𝒕𝒏𝒏 : average of type embeddings

Answer pathAnswer context


Avatarm.0bth54



_date_s

film.film region


type.object.type





2009-12-17

datetime

value_type

Answer Path

Answer Context

Answer Type


Convolutional Layer

Max-Pooling Layer


Avatarm.0bth54



type.object.type

people.person

film.producer







2009-12-18

datetime

value_type


type.object.type





2009-12-17

datetime

value_type

Score Layer

Score

+ +Dot Product

Answer Path

Answer Context

Answer Type

Model Overview


Convolutional Layer

Max-Pooling Layer


t

people.pe


Convolutional Layer

Max-Pooling Layer


Avatarm.0bth54



type.object.type

people.person

film.producer







2009-12-18

datetime

value_type


type.object.type





2009-12-17

datetime

value_type

Score Layer

Score

+ +Dot Product

Answer Path

Answer Context

Answer Type

Model Overview

film film

dat

Score Layer

Score

+ +Dot Product

Answer Path

Answer Context

Answer Type

Negative instance 𝑎𝑎𝑎 is randomly sampled from the set of candidate answersHinge loss for (𝑞𝑞, 𝑎𝑎) and (𝑞𝑞, 𝑎𝑎𝑎)

Objective function 𝐴𝐴𝑞𝑞: set of correct answers 𝑅𝑅𝑞𝑞 ⊆ 𝐶𝐶𝑞𝑞\A𝑞𝑞: set of wrong answers

Back-propagation, AdaGrad, max-norm regularization

Model Training

, where

Inference (During Test)

when did Avatar release in UKCandidate Answers

(2-hop entities/attributes)

Type



Matching Score

Ranker Answer

Avatar 1. Link to entity in Freebase

2. Retrieve candidates

3. Compute vector representations

4. Compute scores

If there are more than one correct answers Use the margin 𝑚𝑚 in objective function as threshold Candidates whose scores are not far from the best answer are

regarded as predicted results

Inference (During Test)

Question understanding results of paraphrases should be same who is the father of A who is A’s father

So, the vectors of paraphrases computed by neural networks should be similar Hinge loss Negative instance is randomly sampled

Question Paraphrases for Multi-Task Learning

WebQuestions (Berant et al., 2013) wh- questions collected by querying Google Suggest API Annotated in Amazon Mechanical Turk Train: 3023, Dev: 755, Test: 2032

Example Question: what is the name of justin bieber brother? Url: http://www.freebase.com/view/en/justin_bieber Answers: {Jazmyn Bieber, Jaxon Bieber}

Paraphrases (Fader et al., 2013) Collected from the WikiAnswers website ~2.4M questions, grouped into ~355k paraphrase clusters

Experiments

http://www.freebase.com/view/en/justin_bieber

Better or comparable results than baseline methods

Experimental Results

Semantic Parsing

Information Extraction

Ablation experiments w/o path/type/context

without the specific column w/o multi-column

tying parameters of multiple columns w/o paraphrase

without question paraphrases 1-hop

1-hop paths to generate candidates

Model Analysis

Salience score How much a word affects question understanding Replace a word with stop words, how much the vectors are affected

Salient Question Words Detection

is the microsoft located

where is the microsoft located Multi-ColumnNeural Networks Type Relation Context

Type Relation Context

Multi-ColumnNeural Networks

vector distance is salience score

Observations wh- words nouns dependent of the wh- words

type/country/leader verbs

speak/located

Salient Question Words Detection

Question answering over unstructured text

Future Work

… Titanic is a 1997 American epic romantic disaster film directed, written, co-produced, and co-edited by James Cameron. …

Who is the director of Titanic?

Type: Person

Type



Matching Score

Sentence Modeling

https://en.wikipedia.org/wiki/James_Cameron

THANKS!

Question Answering over Freebase with Multi-Column Convolutional Neural NetworksQuestion Answering over FreebaseMainstream Methods (1/2)Mainstream Methods (2/2)Proposed MethodModel OverviewModel OverviewEmbedding Candidate AnswersEmbedding Candidate AnswersEmbedding Candidate AnswersModel OverviewModel OverviewModel TrainingInference (During Test)Inference (During Test)Question Paraphrases for Multi-Task LearningExperimentsExperimental ResultsModel AnalysisSalient Question Words DetectionSalient Question Words DetectionFuture WorkTHANKS!幻灯片编号 24

question answering over freebase with multi-column convolutional neural networks · 2017. 9....

Documents