posterior predictive checking and generalized graphical modelsgelman/presentations/ggr.pdf ·...

Posterior Predictive Checking and GeneralizedGraphical Models

Andrew Gelman

Dept of Statistics and Dept of Political Science, Columbia University, New York(Visiting Sciences Po, Paris, for 2009–2010)

24 novembre 2009

Andrew Gelman Predictive Checking and Graphical Models

Generalized graphical models

I Expand graphical modeling to include:

I Predictive model checkingI Fake-data simulationI Scaffolding

I Common features:

I Small changes to an existing fitted modelI Comparisons of nodes between models

I Expand graphical modeling to include:

I Predictive model checkingI Fake-data simulationI Scaffolding

I Common features:

I Expand graphical modeling to include:I Predictive model checking

I Fake-data simulationI Scaffolding

I Common features:

I Expand graphical modeling to include:I Predictive model checkingI Fake-data simulation

I ScaffoldingI Common features:

I Expand graphical modeling to include:I Predictive model checkingI Fake-data simulationI Scaffolding

I Common features:

I Common features:I Small changes to an existing fitted model

I Comparisons of nodes between models

I Common features:I Small changes to an existing fitted modelI Comparisons of nodes between models

I (applied) Building confidence in our computations and ourmodels

I (methodological) Being able to do this routinelyI (theoretical): A unified framework for model building, model

fitting, and model checkingI (computational): Implementing in a Bugs-like language

I (methodological) Being able to do this routinely

I (theoretical): A unified framework for model building, modelfitting, and model checking

I (computational): Implementing in a Bugs-like language

fitting, and model checking

I (computational): Implementing in a Bugs-like language

6 challenges in statistical modeling

I Setting up a realistic (i.e., complicated) modelI Regularization or partial poolingI Fitting the modelI Checking the fit to dataI Confidence buildingI Understanding the fitted model

I Setting up a realistic (i.e., complicated) model

I Regularization or partial poolingI Fitting the modelI Checking the fit to dataI Confidence buildingI Understanding the fitted model

I Setting up a realistic (i.e., complicated) modelI Regularization or partial pooling

I Fitting the modelI Checking the fit to dataI Confidence buildingI Understanding the fitted model

I Setting up a realistic (i.e., complicated) modelI Regularization or partial poolingI Fitting the model

I Checking the fit to dataI Confidence buildingI Understanding the fitted model

I Setting up a realistic (i.e., complicated) modelI Regularization or partial poolingI Fitting the modelI Checking the fit to data

I Confidence buildingI Understanding the fitted model

I Setting up a realistic (i.e., complicated) modelI Regularization or partial poolingI Fitting the modelI Checking the fit to dataI Confidence building

I Understanding the fitted model

I Setting up a realistic (i.e., complicated) modelI Regularization or partial poolingI Fitting the modelI Checking the fit to dataI Confidence buildingI Understanding the fitted model

The models I’m fitting

I Hierarchical generalized linear models

I yi = α+ βxi + εiI yi = αj[i ] + βj[i ]xi + εi (separate regression in each group)

(αjβj

)∼ N

((µα

(σ2α ρσασβ

ρσασβ σ2β

)), for j = 1, . . . , J

I Also can have group-level predictors and nonnested groupingfactors

I Hierarchical generalized linear modelsI yi = α+ βxi + εi

I yi = αj[i ] + βj[i ]xi + εi (separate regression in each group)

(αjβj

)∼ N

((µα

(σ2α ρσασβ

ρσασβ σ2β

)), for j = 1, . . . , J

I Hierarchical generalized linear modelsI yi = α+ βxi + εiI yi = αj[i ] + βj[i ]xi + εi (separate regression in each group)

(αjβj

)∼ N

((µα

(σ2α ρσασβ

ρσασβ σ2β

)), for j = 1, . . . , J

(αjβj

)∼ N

((µα

(σ2α ρσασβ

ρσασβ σ2β

)), for j = 1, . . . , J

(αjβj

)∼ N

((µα

(σ2α ρσασβ

ρσασβ σ2β

)), for j = 1, . . . , J

Application: public opinion in population subgroups

Models for deep interactions

I Main effects, 2-way, 3-way, etc.I Example: predicting public opinion given 4 age categories, 5

income categories, 50 statesI 4 + 5 + 50 + 4× 5 + 4× 50 + 5× 50 + 4× 5× 50 parameters

(“effects”)I Also, group-level predictors (linear trends for age and income,

previous voting patterns for states)I Need a richer modeling language

I No easy way to write this in Bugs or to program it oneself!

I Main effects, 2-way, 3-way, etc.

I Example: predicting public opinion given 4 age categories, 5income categories, 50 states

I 4 + 5 + 50 + 4× 5 + 4× 50 + 5× 50 + 4× 5× 50 parameters(“effects”)

I Also, group-level predictors (linear trends for age and income,previous voting patterns for states)

I Need a richer modeling language

income categories, 50 states

I 4 + 5 + 50 + 4× 5 + 4× 50 + 5× 50 + 4× 5× 50 parameters(“effects”)

(“effects”)

previous voting patterns for states)

Posterior predictive checking: 3 examples

Example 1: a normal distribution is fit to the following data:

Example 1 of 3: checking a fit to a univariate dataset

20 replicated datasets under the model:

Example 2: checking a model fit to data with time ordering

> plot (y, type="l")> lines (y.rep)

Example 3: checking a model with three-way structure

Data and 7 replications:

Theoretical framework for predictive checking

I Data model: θ → yI Data and replicated data: θ → y , y rep

I Posterior predictive distribution,p(y rep|y) =

∫p(y rep|θ, y)p(θ|y)dθ

I Computation:

I Simulate θ from the posterior distribution, p(θ|y)I Simulate y rep from the predictive distribution, p(y rep|θ, y)I Compare y to the replicated datasets y rep

I The generalized graphical model:M --> theta --> y

\\y.rep

I Data model: θ → y

I Data and replicated data: θ → y , y rep

I Computation:

\\y.rep

I Computation:

\\y.rep

I Computation:

\\y.rep

I Computation:

\\y.rep

I Computation:I Simulate θ from the posterior distribution, p(θ|y)

I Simulate y rep from the predictive distribution, p(y rep|θ, y)I Compare y to the replicated datasets y rep

\\y.rep

I Computation:I Simulate θ from the posterior distribution, p(θ|y)I Simulate y rep from the predictive distribution, p(y rep|θ, y)

I Compare y to the replicated datasets y rep

\\y.rep

I Computation:I Simulate θ from the posterior distribution, p(θ|y)I Simulate y rep from the predictive distribution, p(y rep|θ, y)I Compare y to the replicated datasets y rep

\\y.rep

I Computation:I Simulate θ from the posterior distribution, p(θ|y)I Simulate y rep from the predictive distribution, p(y rep|θ, y)I Compare y to the replicated datasets y rep

\\y.rep

Quelques pensées sur la vérification posterior predictive

I Tous les modèles sont fauxI Nous voudrons trouver les aspects des modèles que ne sont

pas en forme des donnéesI L’objectif, c’est apprendre les lacunes de notre histoire

substantiveI Par example, problèmes du modèle des erreurs, ou des

interactions importantes que nous n’avons encore inclusesI Voilà la connection d’analyse exploratoire des données (EDA)

et la presentation visuelleI Les “p-values” sont les moins importants choses dans la

vérification posterior predictive!

I Tous les modèles sont faux

I Nous voudrons trouver les aspects des modèles que ne sontpas en forme des données

I L’objectif, c’est apprendre les lacunes de notre histoiresubstantive

I Par example, problèmes du modèle des erreurs, ou desinteractions importantes que nous n’avons encore incluses

I Voilà la connection d’analyse exploratoire des données (EDA)et la presentation visuelle

I Les “p-values” sont les moins importants choses dans lavérification posterior predictive!

pas en forme des données

I L’objectif, c’est apprendre les lacunes de notre histoiresubstantive

substantive

interactions importantes que nous n’avons encore incluses

et la presentation visuelle

et la presentation visuelleI Les “p-values” sont les moins importants choses dans la

vérification posterior predictive!

Checking graphical models through predictive replications

I Quick summary of posterior predictive checking

I Data y , inference from p(θ|y)I Predictive replications from p(y rep|θ)I Compare y to y rep using (visual) test variablesI Graphical structure: y ← θ → y rep

I More general formulation

I Data y , inference from p(θ|X , y)I Predictive replications from p(y rep|X , θ)I Connection to graphical models!

I Quick summary of posterior predictive checking

I Data y , inference from p(θ|y)I Predictive replications from p(y rep|θ)I Compare y to y rep using (visual) test variablesI Graphical structure: y ← θ → y rep

I Quick summary of posterior predictive checkingI Data y , inference from p(θ|y)

I Predictive replications from p(y rep|θ)I Compare y to y rep using (visual) test variablesI Graphical structure: y ← θ → y rep

I Quick summary of posterior predictive checkingI Data y , inference from p(θ|y)I Predictive replications from p(y rep|θ)

I Compare y to y rep using (visual) test variablesI Graphical structure: y ← θ → y rep

I Quick summary of posterior predictive checkingI Data y , inference from p(θ|y)I Predictive replications from p(y rep|θ)I Compare y to y rep using (visual) test variables

I Graphical structure: y ← θ → y rep

I Quick summary of posterior predictive checkingI Data y , inference from p(θ|y)I Predictive replications from p(y rep|θ)I Compare y to y rep using (visual) test variablesI Graphical structure: y ← θ → y rep

I More general formulationI Data y , inference from p(θ|X , y)

I Predictive replications from p(y rep|X , θ)I Connection to graphical models!

I More general formulationI Data y , inference from p(θ|X , y)I Predictive replications from p(y rep|X , θ)

I Connection to graphical models!

I More general formulationI Data y , inference from p(θ|X , y)I Predictive replications from p(y rep|X , θ)I Connection to graphical models!

Predictive checking and graphical models

I A posterior predictive check requires:

I Set of conditioning variables θI Set of fixed design variables X (e.g., sample size)I Test variable T (y) (more generally, T (X , y , θ)

I Simulating posterior predictive replications is a fundamentaloperation in graphical models

I Requires a new node, y rep, whose distribution is implied by theexisting model

I A posterior predictive check requires:

I Set of conditioning variables θI Set of fixed design variables X (e.g., sample size)I Test variable T (y) (more generally, T (X , y , θ)

I A posterior predictive check requires:I Set of conditioning variables θ

I Set of fixed design variables X (e.g., sample size)I Test variable T (y) (more generally, T (X , y , θ)

I A posterior predictive check requires:I Set of conditioning variables θI Set of fixed design variables X (e.g., sample size)

I Test variable T (y) (more generally, T (X , y , θ)

I A posterior predictive check requires:I Set of conditioning variables θI Set of fixed design variables X (e.g., sample size)I Test variable T (y) (more generally, T (X , y , θ)

Fake-data debugging

I Models can be debugged by simulating fake data:

I Sample θtrue from the prior distribution p(θ)I Sample y from the model p(y |θ)I Perform Bayesian inference, simulations from p(θ|y)I Check calibration of posterior means, predictive intervals, etc.

compared to θtrue

I General procedure in Cook, Gelman, and Rubin (2007)

I Fake-data simulation is a fundamental operation in graphicalmodels

I θtrue is a new nodeM --> theta.true --> y

\ /\ /

\ /theta

Fake-data debugging

I Models can be debugged by simulating fake data:

I Sample θtrue from the prior distribution p(θ)I Sample y from the model p(y |θ)I Perform Bayesian inference, simulations from p(θ|y)I Check calibration of posterior means, predictive intervals, etc.

compared to θtrue

\ /\ /

\ /theta

Fake-data debugging

I Models can be debugged by simulating fake data:I Sample θtrue from the prior distribution p(θ)

I Sample y from the model p(y |θ)I Perform Bayesian inference, simulations from p(θ|y)I Check calibration of posterior means, predictive intervals, etc.

compared to θtrue

\ /\ /

\ /theta

Fake-data debugging

I Models can be debugged by simulating fake data:I Sample θtrue from the prior distribution p(θ)I Sample y from the model p(y |θ)

I Perform Bayesian inference, simulations from p(θ|y)I Check calibration of posterior means, predictive intervals, etc.

compared to θtrue

\ /\ /

\ /theta

Fake-data debugging

I Models can be debugged by simulating fake data:I Sample θtrue from the prior distribution p(θ)I Sample y from the model p(y |θ)I Perform Bayesian inference, simulations from p(θ|y)

I Check calibration of posterior means, predictive intervals, etc.compared to θtrue

\ /\ /

\ /theta

Fake-data debugging

I Models can be debugged by simulating fake data:I Sample θtrue from the prior distribution p(θ)I Sample y from the model p(y |θ)I Perform Bayesian inference, simulations from p(θ|y)I Check calibration of posterior means, predictive intervals, etc.

compared to θtrue

\ /\ /

\ /theta

Fake-data debugging

compared to θtrue

\ /\ /

\ /theta

Fake-data debugging

compared to θtrue

\ /\ /

\ /theta

Fake-data debugging

compared to θtrue

\ /\ /

\ /theta

I Step 0 (already done): Expressing a statistical model as agraph; Bayesian computation on the graph

I Step 1: Graph of models

I Each model is a node of this super-graphI Two models are connected if they differ by only one feature

(adding/removing a variable, allowing a parameter to vary bygroup, adding/removing a grouping factor, changing aprobability distribution or link function, . . . )

I Step 2: Integrated graph

I Nodes within models are linked within a larger graphI All models coexistI Analogy to computational method of parallel tempering

I Step 1: Graph of modelsI Each model is a node of this super-graph

I Two models are connected if they differ by only one feature(adding/removing a variable, allowing a parameter to vary bygroup, adding/removing a grouping factor, changing aprobability distribution or link function, . . . )

I Step 1: Graph of modelsI Each model is a node of this super-graphI Two models are connected if they differ by only one feature

I Step 2: Integrated graphI Nodes within models are linked within a larger graph

I All models coexistI Analogy to computational method of parallel tempering

I Step 2: Integrated graphI Nodes within models are linked within a larger graphI All models coexist

I Analogy to computational method of parallel tempering

I Step 2: Integrated graphI Nodes within models are linked within a larger graphI All models coexistI Analogy to computational method of parallel tempering

Imagine a cleaned-up Bugs language

I Example:for (i in 1:n){

y[i] ~ dnorm (y.hat[i], tau.y)y.hat[i] <- a[state[i]] + b[state[i]]*x[i]e.y[i] <- y[i] - y.hat[i]

}tau.y <- pow(sigma.y, -2)sigma.y ~ dunif (0, 100)

I We would prefer:y ~ norm (a[state] + b[state]*x, sigma.y)

I Also, instead of y.hat, sigma.y, e.y, we want a more general“operator” notation, for example E(y), sd(y), error(y)

Automatic posterior predictive checking

I Example in Bugs:for (i in 1:n){

y[i] ~ dnorm (y.hat[i], tau.y)y.rep[i] <- dnorm (y.hat[i], tau.y). . .

I But y rep should be included automaticallyI Implicit graphical structure for model checking: y ← θ → y rep

I But y rep should be included automatically

I Implicit graphical structure for model checking: y ← θ → y rep

Predictive checking and fake-data debugging

I Model checking or debugging in ideal graphical model software(“DreamBUGS”):

I Set an on/off switch for each node: is it conditioned on oraveraged over?

I Specify a test summary (numerical or graphical) of data andparameters

I Various off-the-shelf test summaries will be available

I Design of data collection is integrated with graphical modeling

Résumé et les directions vers l’avenir

I On peut généraliser les modéles graphiques:M --> theta --> y M --> theta --> y

\ \\ ou \y.rep theta.rep --> y.rep

I Tous les quantitées—θ, y , y rep—existent ensembleI La vérification, c’est possible être plus formal dans la théorie et

dans la computationI Tout est complètement Bayesien—il n’y a jamais le utilisation

double des données!I On peut faire un réseau des modèles pour augmenter notre

comprensionI C’est la unification thèoreticale (de Bayes et de l’enquête)I Et la unification de l’inference et la utilization des modéles

dans les applications

I Tous les quantitées—θ, y , y rep—existent ensemble

I La vérification, c’est possible être plus formal dans la théorie etdans la computation

I Tout est complètement Bayesien—il n’y a jamais le utilisationdouble des données!

I On peut faire un réseau des modèles pour augmenter notrecomprension

I C’est la unification thèoreticale (de Bayes et de l’enquête)I Et la unification de l’inference et la utilization des modéles

dans la computation

I Tout est complètement Bayesien—il n’y a jamais le utilisationdouble des données!

double des données!

comprension

comprensionI C’est la unification thèoreticale (de Bayes et de l’enquête)

I Et la unification de l’inference et la utilization des modélesdans les applications

posterior predictive checking and generalized graphical modelsgelman/presentations/ggr.pdf ·...

Documents

generalized hopfield networks and nonlinear...

computing generalized method of moments and generalized

generalized taylor and generalized calvo price and wage

interactions in multilevel...

uses of generalized convexity and generalized monotonicity...

generalized system

generalized coordinates

generalized pca

generalized conversational

generalized function

prior distributions for variance parameters in hierarchical...

generalized concavity

fitting and understanding multilevel (hierarchical)...

some questions (and a few answers) about multilevel...

merged -...

general linear model general linear model generalized linear...

generalized mosaics

construction of some generalized inverses of operators ......

generalized .pdf

generalized inverse