a causal framework for explaining the predictions of black-box … · 2020-01-26 · bengio,...
TRANSCRIPT
A causal framework for explaining the predictions ofblack-box sequence-to-sequence models
David Alvarez-Melis and Tommi S. JaakkolaPresenter: Ji Gao
https://qdata.github.io/deep2Read
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 1 / 20
Outline
1 Introduction
2 MethodPerturbation ModelCausal ModelExplanation Selection
3 Experiments
4 Reference
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 2 / 20
A causal framework for explaining the predictions ofblack-box sequence-to-sequence modelsEMNLP’17
Black-box intrepretation on NLP sequence generation tasks.
Explanation: A sets of input and output tokens that have causaldependencies under the model.
Adopt a VAE to generate semantically related sentence variations
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 3 / 20
Related Work
Local Interpretable Model-agnostic Explanations (LIME)[RSG16]
On classification task. Other works include [LBJ16]
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 4 / 20
Related Work
Local Interpretable Model-agnostic Explanations (LIME)[RSG16]
On classification task. Other works include [LBJ16]
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 4 / 20
Definition
Model
A black-box model is defined as F : X → Y.Input x ∈ X = {x1, x2, .., xn}, output y ∈ Y = {y1, y2, .., yn}
Assumption
The behaviour of the model can be represented as a bipartite graphG = (Vx ∪ Vy ,E ).Vx and Vy are elements in x and y, respectively.An edge Eij is weighted with the occurrence of token xi and yj .
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 5 / 20
Definition
Assumption
The behaviour of the model can be represented as a bipartite graphG = (Vx ∪ Vy ,E ).Vx and Vy are elements in x and y, respectively.An edge Eij is weighted with the occurrence of token xi and yj .
Explanation
An explanation is a collection of sub-graphs in G .Suppose a component G k = (V k
x ∪ V ky ,E
k), then an explanation
Ex→y = {G 1, ...,G k}
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 6 / 20
Pipeline
3 steps:
1 Generate perturbed versions of inputs
2 Use the perturbed inputs to estimate a causal graph model
3 Generate explanations(Subgraphs)
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 7 / 20
Step 1: Perturbation Model
Generate perturbation on arbitrary structured data is difficult.
Use a VAE to generate perturbation: sample small perturbation onlatent space, and use the decoder to generate perturbed samples.
Discrete VAE model from [BVV+15]
Scale the variance to get different levels of perturbation
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 8 / 20
VAE on text sequence[BVV+15]
Inspired from Variational Recurrent Autoencoder(VRAE)
Key equation:
L(θ; x) = −KL(qθ(z |x)||p(z)) + Eqθ(z|x)[log pθ(x |z)]
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 9 / 20
Samples generated by VAE model
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 10 / 20
Causal model
Use logistic regression to estimate the model
P(yj ∈ y |x) = σ(θTj φx(x)) (1)
φx ∈ {0, 1}|x | is the binary embedding vector of sample x.
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 11 / 20
Explanation selection
Model the graph partitioning problem into a MIP programmingproblem[FZP12]
min(xuik ,x
vjk ,yij )∈Y
n∑i=1
m∑j=1
θijyij + maxS :S⊂J,|S |≤Γ
(it ,jt)∈J/S
∑(i ,j)∈S
θijyij + (Γ− |Γ|)θit ,jtyit ,jt
(2)
After solving the problem, sort the importance defined asimportance(E k) = −
∑(i ,j)∈Xk
θij .
Return the top ranked slices.
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 12 / 20
Algorithm
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 13 / 20
Model Details in experiments
For VAE to handle sequential input, use stacked RNN on both sides.and a stacked variational layer.
Use optimization library gurobi to solve the partition models at a MIPproblem
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 14 / 20
Experiment 1: Recovering simple mapping
Dataset: CMU dictionary of word pronunciation, mapping words tophonemes. Including 130K words.
vowels→ V AW 1 AH0 L Z
Result:
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 15 / 20
Machine Translation
Use multiple models: Azure, NMT, Human
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 16 / 20
”mediocre” Dialogue System
Use a Seq2seq model on OpenSubtitle corpus
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 17 / 20
Bias detection
Simulate a biased corpus: In a English to French dataset, prepend theword ’However’ when the translation includes every informalregistry(e.g. tu)
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 18 / 20
Bias detection
Use Azure model, translate several simple French sentences that lacksgender specification in English, but require gender-declined words inthe output.
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 19 / 20
Reference I
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and SamyBengio, Generating sentences from a continuous space, arXiv preprint arXiv:1511.06349(2015).
Neng Fan, Qipeng P Zheng, and Panos M Pardalos, Robust optimization of graphpartitioning involving interval uncertainty, Theoretical Computer Science 447 (2012),53–61.
Tao Lei, Regina Barzilay, and Tommi Jaakkola, Rationalizing neural predictions, arXivpreprint arXiv:1606.04155 (2016).
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin, Why should i trust you?:Explaining the predictions of any classifier, Proceedings of the 22nd ACM SIGKDDinternational conference on knowledge discovery and data mining, ACM, 2016,pp. 1135–1144.
David Alvarez-Melis and Tommi S. Jaakkola Presenter: Ji GaoA causal framework for explaining the predictions of black-box sequence-to-sequence modelshttps://qdata.github.io/deep2Read 20 / 20