graphical inference

A graphical grammar + graphical inference =

a grammar of graphical inference?

Hadley Wickham, Rice University

1. Motivation

2. Resampling methods (graphical inference)

3. Graphical display (grammar of graphics)

4. Future work

−6

−4

−2

0

2

4

6

8

5

●

●

●

●●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

−6 −4 −2 0 2 4 6

species● Concinna● Heikert.● Heptapot.

−6−4−2

02468

−6−4−2

02468

−6−4−2

02468

1

● ●●

●

● ●●●●●●●●

●●

●● ●

●●●● ●

●●

●

●

●●

●●

●

●●●●

●

●

●●

●

●●

●

●●

●

●●●

●

●

●

●

●●

●●●●●●

●●●●●●●●●●

●

●

5

●●●

● ●

●●

● ●●

●

●

●

●

●

●●

●

●● ●

●

●●●

●●

●

●

●●●

●

●

●

●●

●●

●

●

●●

●●

●

●●● ●

●

●

●

●●

●●●●

●

●●

●

●●

● ●

●

●

●

●●

●

●

9

● ●●

●● ●●●

●●

●

●●●

●

●

●

●●

●

●

●●●●

●

● ●● ●

●●

●

●

●

●

●● ●

●

●●

●

●● ●●

●●

●

●

●

●●

●●●●

●

●●● ●●●●●

● ●●

●

●●●

−6−4−2 0 2 4 6

2

●

●

●●

●

●●●●●

●

●

●

●●

●

●

●

●●

●●●

●

●

●

●

●

●●

●●●●●●●●●●

●●

●●●

●

●

●

●

●●●●

●●●●

●

●

●●

●

●● ●●

●●

●●

●● ●●

6

●●●●

●●●●

● ●●

●

●●

●●

●●●

●

●

●●

●

●

●● ●●●●

●●

●●

●● ●●

●●●

●

●●

●

●●

●

●

●●●●

●

●●

●

●

●

●●

●● ●●

●

●●

●●

●●

●

10

●●●

●●

●

●●●

●

● ●

●

●●●●

●●●

●

●●

●

●

●

●

●

●

●●●●●

●

●●●● ●●●

●●

●

●

●● ●●●

●●

●

●

●●●

● ●

●

●

●●

●●

●● ●

●

● ●●

●

−6−4−2 0 2 4 6

3

●

●●

● ●●●●

●●●

●●

●

●●

●

●

●●

●

●●●●●

●●●

●

●●

●●●●

●●●

● ●●●●

●● ●●●

●● ●●

●●●

●●●●●●

●●●

●

●

●●

●●

●●

●

7

●●

●

●

●●● ●● ●

●

●

●●●●●●

●●●

●●●

●● ●●●●●

●

●●●

●●

●

●●

●

●

●● ●●

●●●●●●

●●

●●●

●

●

●

●●●

●●●

●●

●●

●●

●●

11

●●

●

●●

●

●●●●

●● ●

●

●●

●

●

●

●●

●

●●●●●●

●●

●

●

●●

●●●

●

●●●●●

●●●

●

●

●

●●

●

●

●

●● ●●

●●

●

● ●

●●

●

●

●

●

●●

●●

●

−6−4−2 0 2 4 6

4

● ●●●

●●●

●●●

●●

●

●

●●

●●

●

●●●●

●●●●

●

●●

●

●

●●

●●

●●●

●●●●

●●

●●

●

●●●

●

●●

●

●●●

●

●

●●●

●

●●●●

●●

● ●

● ●

8

●● ●●

●●

● ●●●

● ●●

● ●●●

●●●●

●

●

●●●●●

●●

●●

●●●

●

●

●●●

●●

●●

●

●

● ●●●●

●●●

●

●●

● ●

●

●

●

●

●

●

●●

●●●

● ●

●

●

12

●●

●●

●

●

●

●

●●

●

●● ●●●

●●

●●●

●

●●

●

●●●

●●

●●●

● ●●●●●

●●

● ●

●●

●

●●

● ●●

●●

●

●

●●

●

●●

●

●

●●●●

●

●

●

●●

●

●●

−6−4−2 0 2 4 6


−4

−2

0

2

4

−4

−2

0

2

4

−4

−2

0

2

4

1

●

●

●● ●

●

●

●●

●

●

●

●

●

●●●

●●

●

● ●

●●●

●

●●

●

●

●

●

●

●

●

●●

●● ●●

●●

●●

●

●

●

●●

●

●

●

●●

●●

●

●

●●

●

●

●

●

●

●

● ●●

●

●

●

●

5

●

●

●

●

●

●●●●

●●

●

●●

●●●

●

●

●

●

● ●

●●

●

●

●

●

●

●●

●

●

●●

●

●●

● ●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●●●

●●

●

●

● ●

● ●●

●

● ●

●●

9

●●

●●

●

●

●

●●●●

●●

●●

●

●

●●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●● ●

●

● ●

●

●● ●

●

●

●●

●●

●●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●●

● ●

●

●

−4 −2 0 2 4

2

●●

●

●●●

●

●

● ●

●

●

●

●●

●●

●●●

●

● ●●

●●●

●

●

●

●●

● ●●●

●

● ●●

●●●

●●●

●

● ●● ●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

● ●●

●

●

●●

●

6

●●

●

●

●●

●

●

● ●

●●

●

●

●●

●●

●

● ● ●

●

●

●●

●

● ●

●

●

●

●

●

●

●●●

●●

●●

●

●

●

● ●●●

●

●●

●

●

●

●

●

●

●●●

●●

●

●

●●●

●

●

●

●

●●

10

● ●

●●

●●

●

●

●

●

●

●

●

●

●●●

●●● ●

●

●

●

●

●

●

●

●●●

●●

●

● ●●●●

●●

●

●

●

●●

●

●● ●

●

●

●● ●

●

●●

●●●

●

●

●

●

●

●●

●

●

●●●

●

−4 −2 0 2 4

3

●

●

●

●

●

●

●●●

●

●●

●

●

●●

●

● ●

●●

●

●

●

●

●

●

●

●

●

●●

● ●●

●

●●●●

●

●●

●

● ●●

●

●●

●

●●

●

●● ●

●●●●

●

●

●

●

●

●●

●

●

●●●

●

7

●●

●

●

●●●

●

●

●

●

●

●●

●

●●●●

● ●

● ●●

●

●●●

●

●●

●●

●

●

●

●●● ●

●

●●

●

●

●

●

●

●●●

● ●

●

●

●●

●

●

●

●●

●●

●●

●●

●●

● ●●

●

11

●

●

● ●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●●

●●

●●●

●

●●●

●● ●

●

●

●

●●

● ●●

●

● ● ●●

●

●●

●●

●

●

●●

●

●●●●

●

●

● ●●

−4 −2 0 2 4

4

●

●

●●

●●

●

●●

●

●●●

●

●●

●

●

●●

●

●

●●●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●●

●●

●●

●●

●● ● ● ●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

8

●

●

● ●

●

●

●

●

●

●

●

●

●

●●

● ●●●

●

●

●●

●

●●

●

●●

●

●●

●

●

●●●

●●

●

●

●●

●

●●

●

●

●

● ●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●●●

●

12

●

●

●

●

●

●●

● ●

●●

●●

●

●● ●

●

●

●

●●●

●

●

●●●

●

●●

●

●

●●

●

●

●

●

●

●

● ●

●●

●●

●●● ●

●●●

●● ●

●

●

●

● ●

●

●

●

●

●●

●

●

●

●

●●

−4 −2 0 2 4


ProblemWant to make it very easy to make these types of display - should be able to create as easy as doing as simple numerical test.

With the right tools, it turns out to be really easy!

Two basic problems: how to do resampling under the null, and how to define the graphic.

Resampling

Many resampling methods have particularly elegant expression in R code. For example, if null hypothesis is that of independence:

df <- transform(df, resp = sample(resp))

However, better to generalise the pattern.

Generalisationpermute_var <- function(var) { function(df) { df[[var]] <- sample(df[[var]]) df }}

f <- permute_var("mpg")f(mtcars)permute_var("mpg")(mtcars)

Advantages

Separate description from implementation.

Call describes action.

Can later replace implementation if better approach discovered.

Unimportant details concealed.

n resamples

Each method gives us a single resample, but we need n.

Trivial to repeat n times with rdply() from the plyr package.

(rdply() is a generalisation of replicate() that returns a data frame with a column that labels each replicate.)

Overallresamp <- function(true, method, n = 19, pos = sample(n + 1, 1)) { samples <- rdply(n, method(true)) if (missing(pos)) { message("True data in position ", pos) } add_true(samples, true, pos)}

add_true <- function(samples, true, n) { samples$.n <- with(samples, ifelse(.n >= n, .n + 1, .n)) true$.n <- n all <- rbind(samples, true) all[order(all$.n), ]}

Will see application shortly

Other nulls

Need functions for other common null hypotheses.

Experimental null_model(), which computes rotation residuals given a specified linear model. (Demo a little later)

Display

How is the plot of the simulated data with true data different from a single plot? Just need to repeat the same display n times and label appropriately.

This is easy if we have a description of the plot, independent of the data.

Grammar of GraphicsThis is one principle of the grammar of graphics: should describe the graphic we want, not how to create it.

Implemented with the ggplot2 package in R.

Easy to modify a graphic after it has been created. For graphical inference, we need to change the data and facetting.

Example

displ

cty

10

15

20

25

30

35

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●●

●

●●●

●●●

●

●

●

●

●

●●

●●

●

●●

●●

●

●

●

●

●

●●

●●

●●

●

●

● ● ●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

● ●●

●

●

●●

●●

●

●

●●

●●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●

●

● ●

●● ●●

●

●

●

● ●●

●

●●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●●●

●

●●

●●

●●

●

●

●

●

●

● ● ●

● ●

●●

●●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

● ●●

●●

●

●

●

2 3 4 5 6 7

factor(year)● 1999● 2008

displ

1/ct

y * 1

00

4

6

8

10

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●●

●

●

●●

●

● ●●● ●● ●

●

●

●

●

●●

●●

●

●●

●●

●

●●

●

●

●●●●

●●

●

●

● ● ●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

● ●●

●

●

●●

●●

●●

●●

●●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●●●

●

● ●

●

● ●●

●

●

●

● ●

●

●

●●

●

●

●

●

● ●● ●

●

●

●●

●

●●●

● ●●

●

●●

●

●

● ●

●

●

●

●

●

● ● ●

● ●

●●

●●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

● ●●

●●

●

●

●

2 3 4 5 6 7


displ

1/ct

y * 1

00

468

10

468

10

468

10

468

10

1

●●●

●●

●●

●

●●

●●

●●

●

●

● ●●●

●

●●●●●● ●

●

●●

●

●

●

●●●

●

●

●● ●●

●●

●

●●

●

●●●

●●● ●●●●●

●●

●

●

● ●●

●●

●●

●

●● ●

● ●

●●●●●●●●●

● ●●

●●● ●

●●

●●●●

●

●●●

●●

●●● ●

●

●●●●

●● ●

●●●

●

●●

●●●●●

●●

●●

●●

● ●● ●●●

●●●

●

● ●

●

●●

●

●

●●●● ●●●

●● ●

●●

●●●●

●●●●●●

●●●

●●

●●●

●

●●●

● ●

●●

●

●●

●

●

● ●●●

●●●●●●

●●

●

●●●

●●●

●●

●

●

● ●

●●●

●

●

●● ●

●

●●

●●

●●

6

●●●●

●●●

●

●●

●

●

●

●

●●

●

●●●●●

●

●●●● ●

●

● ●

●●●

●

●

●●●

●●

●●●

●●●●

●●●●

●●●●●●

●● ●●

●●

●●

●●● ●

●●

●

●● ●

●

●

●●●●●●

●●●●●

●●

●

●● ●

●●●

●●

●●●

●

●●● ●●

●●

●●

●

●● ●●●●

●●●●

●

●

●●●

●

●

●

●●

●●●●

●●

●● ●

●●

●

●

●

●

●

●●

● ●● ●●

●● ●●

●

●●

●●

●●●●●●●●

●●●●

●

●

●●

●

●● ●

●●

●

●●

●●

● ●●●

●●●●

●●

●

●●●●

●●

●●

●

●●●

●● ●●

●●

●

●

●●●● ●●●

●●●

11

●●

●●●

●●

●●●●●

●

●

●●

●

●

●●●●●●

●

●●

●●

● ●●

●●●●

●

●

●●●●● ●●●

●

●●●●

●●●●

●●●●●● ●●●

●●●●●

●

●●

●

●● ●

●●

●●

●

●●

●●●●

●●●●●●

● ●●●

●

●●●●●

●●●

●●

●●

●

● ●

●

● ●●●●●

●●

●●●●

●● ●

●

●●

●●●●●●●●●● ●●

●●

●

●

●

●● ●● ●●

●●●● ●

●●●●

●

●●●●

●●●

●

●●

●●●

●●●

●

●● ●● ●

●●

●

●●●

●●●●

●●●

●●●●

●●●

●

●

● ●●●

●●●●

●

●

●●● ●

●●

●

● ●●●●

●● ●

16

●●●●●

●

●

●●

●●

●

●

●●

●

●

●

●●● ●●●●

●

●●

●●

●

●

●

●

●

●

●

●●

●●

●●●

●●●●

●●

●●●●● ●●

●●●

●●● ●

● ●

●

●●●

●●

●

●●●●

●●●●●

●●

●●●●● ●

●●●

●●●

●

●●

● ●●

●●

●

●●● ●●

●

●● ●

●●

●●●

●

●●●

●

●

●

●

●● ●

●

●●

●●

●●

●●● ●

●

●●

●

●●●●●

●

●●

●

● ●●

●● ●●

●

●

●●●●

●

●●●●●●

●●

●●●

●●●●

●●

●

●●

●●

●●

●●

●

● ●

●

●●●●●

●●●●●

●●

●●

●●●●

●

●●

●

●●

●●

●

●

●●

●

●●● ● ●

2 3 4 5 6 7

2

●●●

●●●

●●

●●

●

●

●●

●

●●●●●●●●

●●

●

● ●

●

●

●

●

●● ●●

●●●●

●●

●●

●

●●●●

●●●●

●●

●●

● ● ●●●

● ●●

●●

●

●●

●

●

●●●●●

●●●●●●

●●●

●

●●●

●●●

●

●●●●

●

●

● ●

●●

●●

●

●●

● ●●●●●

● ●

●●●

●

●●●●

●●

●

●● ●

●

●

●

●●

●●●●

●

●●

●●●

●●

●

●

●

●●●●

●●

●

●●

●●●

●

●

●●

●●●●

●●●●

●●● ●●●●●

●●●

●

●●

●●●

●

●● ●●● ●

●●

●●●●●●

●●

●●●●

●●●

●

●●

● ●●

●●

●

●●● ●

●●

● ●

●

7

●

●

●

●●●

●

●

●●

●

●

●●

●● ●●

●

●●●●

●● ●

●

●

●

●

●

●

●●

●

●

●

● ●

●●● ●●

●

●●●

●●●●

●●

● ●●●

●●●

●●

●●

●●●

●●●

●●●

● ●

●●●●●●

●●●●

●

●●●●●

●

●●●

●

●●

●●

●●

●

●● ●●

●●

●

● ●●●● ●●●●

●●●●

●

●●●

●

●●●

●●

●●

●

●●●●●

●● ●

●

●

●

●

●●●

● ●●

●

●●

●●●

●

●

●

●●●●●●●

●●●●●●●

●●

●

●

●●●

●

●● ●●

● ●●

●

●

● ●●●

●●●●●

●●

●●●

●●● ●

●●●●

●●

●

●●●

●●

●

●●

●

●●●

●●

12

●

●

●

●●

●

●

●●● ●

●

●

●

●

● ●● ●● ●

●● ●● ●● ●

●

●●

●

● ●●

●●●

●

●●●●

●●●●●●

●●●●

●●

●

●●

●●●

●● ●

●●

●●

● ●

●●

●

●●

●●●●●●●●●

●●●

●●●●●

●●

●●

●

●

●● ●●

●●

●

●● ●●

●

● ●●●●

● ●●●●

●●●●

●

●●●●●

●

●

●

●

●

● ●●●●

●

●●●

●● ●

●

●●

●

● ●●●

●

●

●

●● ●●

●●●

●●

●●●

●

●

●●●

●●

●●●●●●● ●●●

●

●

●

●●●

●

●●●

●

●●●●

●●●●

●● ●● ●

●●●

●●●●

●

●●●

●●

●

●●

●●

●

● ●●

●

17

●●

●●●

●

●●●●

●

●

● ● ●●

●●

●

●●●●

●●●

●● ●●

●●

●

●

●●

●

●●

●●

●●

● ●●●

●

●●●●●

●●●●

●●●

●●●

●

●

●

● ●

● ●

● ●●●

●

●●

●●●●●●●●

● ●

●● ●

●●●●

●●●●

●●

●●●

●

●

●●● ●●

●

●

● ●●

●●●

●●

●

●●●

●

●

●●

● ●

●

● ● ●● ●●

●

●●

●

●●

● ●●●●

●

● ●● ●

●●●●● ●

●●

●●

●●●●

●●●●● ●●●●

●●●

●

●●●

●●●

●●

●●●

● ●

●●

●

●●●●●

●●●●

●●

●

●

●●●●

●●●●

●

●

●●● ●

●

●●

●

●

● ●●●●

●

2 3 4 5 6 7

3

●●

●

●●●●●

●

●●●

● ●

●

●

●

●●●●●

●

●●●● ●●

●

●

●

●

●

●●

●●

●● ●●●

●●●● ●●●

●●●

●

●●●●

●●●

●●

●

●

●●

●●●

●●

●● ●

●

●●●●●

●●●●●

●●●

●

●●

●●●

●●

●

●●●

●●

●●●●

●

●

●● ●

●

●●

●

●●●●

●●●

●

●

●●

●● ● ●

●

●

●●●●●●●

●●

●

●

●

●

●●●

●

●

●●●●●

●

●●

●●●

●●●●●

●●● ●●●

●●●

●●

●

●●

●●●

●● ●●

●

●

●●

●●

●

●●

●●●●●●●

●●

●●●● ●

●●

●●●●

●

●

●●

●● ●

●

●● ●●●●●

● ●

8

●●●●●

●●

●●

●

●

●●●

●●

●

●

●

●●

●●●● ●●●●

●● ●

●

●

●

●

●

● ●

● ●●

●●

●●●

●●

●●●●●●●

●●●●●●●

●●

●

● ●●

●● ●

●

●●

●

●●

●●●

●●●●●

●●

●●●

●●●● ●

●●●●

●

●

● ●●

●●

●

●● ●

●

●●

●●

●

●

●●●●●●●●

●●●

●

●●●

●

●●

●●

●

●

●

● ●●

●●●●

●

●● ●

●●●● ●●●

●●

●

●

●●●●

●

●●● ●●

●●●●

●

●

●

●●●● ●

●

●●

●●

●●●●

●

● ●

●●●●●●●

●

●●

●●

●

●●●●

●

●●

● ●

●●

●

●●

●

●

●● ●●●● ● ●

13

●●●●●●

●●●

●

●

●●●

● ●●

● ●● ●●●

●●●

●●

● ●

●●●●

●

●

●●

●●●●

●

●

●

●●

●●●●

●●●●● ●●●●●

●●●

●●●

● ●●

●● ●

●● ●● ●

●●

● ●●

●●●●●●

● ●

●●●●●●

●

●●●

●●

●●

●

● ●●

●

●●

●●●

● ●●●●●

●●●●●

● ●●

●●

●

●

●●●●

●

●●●●

●●●

●●

● ●

●

●

●

●

● ●●

●●●

●●●

●●●

●●●●

●●●●

●●●●●

●●

●

●●

●

●

●●

●●

●

●

●●●

●●

●●●●●

● ●●●

●●●● ●

●●●●●●●

●

●

●●

●●

●

●●●

●

●● ●●●

●

18

●●●

●●

●

●●●

●

●●●

●

●●

● ●●●● ●

● ●●

●

●

●

●

●

●

●●●

●

●

●●

●●● ●●●

●●●

●●●

●●●●

●●●●

●●●●●

●●●

●

●●

●●

●

●

●

●●

●

●●●

●●●

●●

●●● ●

●●

●●●

●●●

●

●●

● ●●

●●●

●●

●

●●

● ●●●●

●

●●●●●

●●●

●

●●●●

●●

●

●

●●●●●

●

●●●●●●

●

●

●

●

●●

●

●●●

●●

●

●

●

●

●●●●

●●●●

●●●

●●

●●

●

●

●●

●●

●● ●

●●

●

●●●●

●

●●

●●●●●

●●●●●●

●●●

●

●●●●●●●

●

●●

●

●

●●● ●

●●

● ●●● ●

2 3 4 5 6 7

4

●●●●

●●●●

●

●

●

●

●●

● ●●

●

●●●

●●

●●

●

●

●

● ●

●

●

●●● ●

●●●● ●●●

●

●

●

●●●

●●

●●●●●

●●

●

●

●● ● ●

●

●●

●●

●●●

●

●

●●

●

●●●●●●●

● ●

●●●●●● ●

●●●●

●●●

●

●●

●

●●●

●

● ●

●●

●

●●

●●●

●

●●●

●

●

●●●

●

●

●

●

●●

●●

●

●

●● ●●●●

●●●

●

●

●

● ●●●●

●

●

●●

●●●

●●●●

●●

●●●●●●

●●●● ●●●

●●

●●●●

●●●●

● ●●

●●

●●

● ●●●●●●●●

●●

●

●

●

●●

●●

●

●●●

●

●

●

●●● ●

●

●● ●●

●●

●

9

●●●●

●

●●●●

●●●●

● ●

●

●●●●

●

●●●●● ●

●

●

●

●●●

●

●

●

●● ●●

●●

●●●

● ●

●●

●●●

●●●●●●

●●●●

●●

●●●

●●●

●●●

●●

● ●

●

●

●●

●●●●●●

●●

●

●●●●

●●●

●●

●

●

●● ●●

●●

●●

●

●

●●

● ●● ●●

●

●●

●

●

●●●

●

●●

●

●

●

●

●●

● ●●●●

●●●

●

●

●

●

●

●

●

●

●●

●

●● ●●●

●

●●

●●

●●

●●● ●●●

●

●

●●

●● ● ●●

●●

●●●

●●● ●

●●

● ●●

●●●●●●●●

● ●●

●● ●

●●●

●

●●●

●●●

●●

●

●●● ●

●

● ●●●●●

●

14

●●

●●●●

●

●●●●

●

●●

●●

●

●

●

● ●●●●●

●

●●

●

●

●

●

● ●

●●●

● ●

●●●●

●

●●●●●●

●●●● ●●●

●●

●●●

●

●●

●

●

●● ●

●●

●●●●

●●●●●●●●

●●

●

●●

●●

●●●●

●●

●●

●●●

●

●

●●● ●●

●

●

●● ●●●

●●●

●

●

●●

●

●●

●● ●

●

●

●●● ●

●

●●●

●● ●●

●

●

● ●

●

●

●

●

●●

●

●●

●

● ●●

●● ●●●

●●

●

●●●●

●●

●●●●●●●

●●

●

●

●●

●● ●●

● ●

●●

●●

●

●

●●●●●●●

● ●●

●●●● ●

●●●

●●

●

●

●

●●●

●●

●

●

●●●

●●

19

●●

●●●

●●●●

●

●●

●

●

● ●● ●●●

●●

●●

●●

●

● ●●● ●● ●

●●●

●●●

●●

●

●●

●●

●●●●●

●●●●●●

●●

● ● ●●

●●

●● ●●

●●

●●●●●

●●●●

●●

●●

●●●●

●●●●

●●●

●

●●

●●

●●

●

●

●●

●●

●●

●●

●●

●●

●●●

●●●●●

●

●

●

●

●

●● ●

●

●

●

●●●●

●

●●●● ●●

●

●

●

● ●●

●

●●●

●

●●

● ●● ●

●●

●●●

●●●●●●

●●●

●●●●

●

●

●●

●●● ●

● ●

●●

●●●

●

●

●●

●●●●●●●●

●

●

●●●

●●

●

●●

●

●●●

●●●● ●●●●●●

●

2 3 4 5 6 7

5

●●●

●

●

●●●●

●

●

●

●●

●● ●

●

●

●

●●●●

●●●

●●

●

●

●●

● ●

●

●●●

●

● ●●

●●

●●●● ●

●

●

●●●●●●

●●●●● ●●●●

●●●

●

●●

●

●

●●

●●●

●●

●

●●●

●●●●

●●

●●● ●

●●●

●

●

●

●●

●●

●

●

●● ●

●●

●●●

●

●●●●●

●

●●

●

●

●●

●

● ●●●●●

●

●●● ●

● ●

●

●

●●

●

●

● ●

●●● ●●

●

●●●

●●●●

●●

●●

●

●●●

●

●

●

●●

● ●●

●● ●

●

●

●●

●

●●

●●●●

●●●●●●

●●●

●

●●●

●●●

●

●●●●

● ●● ●●

●●●● ●

●●

● ●●●● ● ●

10

●

●●●●●●

●●

●

●●

●

●●

●●

●●

●

●●●

●● ●●

●●

●

●

●

●●

●

●

●●

●●●

● ●

●●●●●

●

●●●●

●●●●●●

●●●●

●

●

●

●●

●●

●●● ●

●●

●

●●●●

●●●●●●●

●● ●

●●●●●●

●

●● ●

●

●●

●

●

●● ●

●●

●

● ●●●●●●●●●

●●

●●●

●

●

●

●● ●●

●●● ●●●●

●●●

●

● ●●

●

●●●

●●●●

●●●●●

●●

●

●●

●

●●●●

●●●●

●●

●●

●●

●●●

● ●●●

●●

●

●

●

●

●●●●●●●●

●●●

●●●●● ●

●●●●

●●●

●

●●

●

●●●● ●

●

● ●●

●● ●

15

●●

●●●●●

●●●

●

●●

●

●

●●

●●●● ●●●

●●

●●

●

●●

●●

● ●

●

● ●●●

●● ●●● ●

●●●

●●●●

●●●●

●●●●●●●

●●●

●

●●● ●

●

● ●●

●

●

●●

●●●●●●●●

●●

●

●●●●●

●

●

●

●●●

●●●●●● ●

●●

● ●●●●●●●

●

●

●●●

●

●

●●

●●

●●

●

●

●●

●

●●

●● ●

●

●

●

●●

●

●●●

●

●

●

●●

●●●●

●

●

●

●●●●

●● ●●●

●●●

●

●●

●● ●

●● ●●

●●●

●● ●

●●

●●●

●●●●●●

● ●●

●●● ●

●●●●

●●●●

●

●●

●●

●

●

●

● ●

●●

●●

●● ●

20

●●●

●

●●●●●

●●

●●

● ●

●●

● ●●●

●

●●●● ●●

●

●● ●

●

●●

●●●●

●

● ●●●

●● ●

●●●

●●●●●

●●●●●●●

●●●●

●

●

●

●●●

●●●

●●●

●

●

●

● ●●

●●●●●●●

●●

●●

●●●●

●●

● ● ●●

●●●

●

●●● ●

●

●● ●●●●●●

●

●●

●

●●●

●●

●

●●

●● ●●●

●●

●●

●●

●

●

●●

●

●

●

●

●●

●●●●

●

●●

●●●●

●●●

●●●

●●●●

●

●●●

●●

●

● ●●● ●

●●

●●●●●

●●●●

●●

●●

●●●●●

●●●●●●

●

●

●●

●

●

●

●●

●

●●

●● ●●

●●

●●

2 3 4 5 6 7


qplot(displ, cty, data = mpg, colour = factor(year))qplot(displ, 1 / cty * 100, data = mpg, colour = factor(year))

mpg_perm <- resamp(mpg,permute_var("year"))last_plot() %+% mpg_perm + facet_wrap(~ .n)

Is a linear model with displacement as single

predictor adequate?

displ

gp10

0m

2468

10

2468

10

2468

10

2468

10

1

●

● ●●●

●●

●

●

●●●

●●

●

●● ●●●

●●●

●

●●

●

● ●

●●

●● ●

●●

●

●

●

●●

●

●

●●

●●

●●●●

●●●●●

●●

● ●● ●●●

●

●

●●

●●

●●● ●

●

●●

●●●●

●

●

●

●

●●●●

●

●

●

●

●●●

●

●

●

●●●

●

●

●

●●●

●●●●●

●●

●

●●●

●●● ●

●●

●●●● ●●

●●

●●

●

●●●

●●●

●

●

●

●●

●●●●

●

●●

●

●

●

●

●●

●

●

●

●●

●

●●●

●

●●

●

●●

●

●●

●●

●

●

● ●●

●●

●●●

●●

●●●●●

●●

●

●

●

●

●

●●

●

●

● ● ●

●●

●

●

● ●

●

●

●

●

●●

●●●

●

●

●●●

●

●

●

6

●

● ●

●●●

●

●

●

●●●●

●

● ●●

●●●

●●

●

●

●●●

●●●

●●

●

●

●

●

●

●

●

●●

● ●●

●

●

●

●

●

●

●● ●

●●●

●●●

●●

●●

●

● ●●

●

●●●

●

●●●

●●●●

●●●●

●●

●●

●●●● ●

●

●

●

●

●●●

●

●

●●

●

●

●● ●

●●

●

●●

●●

●

●●

●

●

●

● ●

●

● ●●●

● ●●● ●

●

●

●

●

●

●

●● ●●●

●●●●●

●

●

●

●

●

●●●●

●

●

●

●●

●●●

●● ●

●●●●●

●●●

●

●

●●●

●

●●

●

●

●●

●

●

●

●

● ●●

●●

●

●

●

●●●●●

●●●

●●

●●

●●

● ●●●

●

●●

●

●●●●

●●●

●

●●

●

11

●● ●

●

●●

●

●●

●

●

●●

●

●●

●●●●

●

●

●●

●●

●

●

●

●●

●

●●

●●●

●

●

●●●

●

●

● ●

●

●

●●●●

●

●●●●

●

●

●●

●●

●

●

●

●●

●●●●●

●●

●●●●

●●●●

●

●

●●

●

●

●

● ●●●

●

●

●

●●

●

●

●●●

●●●

●

●

●

●●

●

●

●

●●●●

●●●

●

●

●

● ●●

●

●

●● ●

●●

● ●●

●●●

●

●

●● ●

●

●●●

●●

●●●●●●

●●

●●●

●

●

● ●●

●

●●●●

●

● ●●

●

●●

●●

●

●● ●

●●

●

●

● ●

●●

●●●

●●

●

●

●●●●●

●

●

●●

●

●●

●●●

●

●●

●

●

●

●

●

●●●

●●●●●●

●

16

●●

●

●

●

●

●●

●●●●● ●

● ●

●

●●●

●

●●

●

●●●

●●

●●

●●

●

●●●

●

●

●

●● ●

●

● ●●

●

●

●

●

●

●●●●

●●

●

●● ●

●●

●●

●

●

●●●●●

●●●

●●●●●

●●●●

●●

●

●

●

● ●●

●

●

●

●●

●

●

●

●●

●

●●●

●●

●

●●●

●●

●●●

●

●●● ●

●

●

●

●

●

● ●●

● ●●

●

●●

●

●

●

●●●●●

●

●

●●

●

●●●

●

●●●● ●

●

●●●

●

●

●●●

●

●●●

●●●

●●●

●●●

●

●

●●

●●

●●

●

●●

●●

●

●

●●

●● ●

●●

●●

●●●●

●

●

●●

●●●

●●

●

●

● ●

●

●

●●●

●

●

●●

●●

●

2 3 4 5 6 7

2

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●● ●

●●●

●●●

●

●

●

● ●

●

● ●

●●

●

●

●

●●

●●●

●

●

●●●

●

●●●

●●

●●●

●

●

●

●●

●

●●

● ●●

●

●

●

●●

● ●

●

●●●

●●●●●

●●

●●

●

●

●

●

●●●

●

●

●

●●

●

●●

●●

●

●●

●●

●

●●●●

●

●

●●

●

●

●

●

●

●

● ●●●●

●

●● ●●

●

●

●●

●

●●

●●●

●●

●●●●●

●●

●●●

●● ●●

●●● ●

●●

●

●

●

●●●●

●● ●●●

●●

●●

●

●

●

●

●●●

●

●●

●

●●●●

●●

●

●

●●●

●

●

●

●●

●

●●

●

●

●

●●●

●

●

● ●●

●●●● ●

●

●

●

●

● ●

7

●●

●●●●

●●

●

●

●

●●

●

● ●

●

●

●●

●●●

● ●

●

●

●●●●●●

●●●●

●

●

●

●●

●

●

●●●

●

●

●

●● ●

●

●

●●●

●

●● ●

●●● ●

●●

●

●●

●

●

●●●

●●●

●●●●

●●

●●

●●

●

●

●

●

●●

●

●●

● ●

●

●●

●●●

●

●●●

●●●

●

●●

●●● ●●

●

●●●

●●

●

●

●●●

●

●

●

●●●

●●●

●●

●

●

●●●

●

●●

●

●

●●●●

●

●●

●

●●●

●●●

●

●

●●

●●

●

● ●●

●

●●

●

● ●●●

●●●● ●

●

●

●●●●●

●

●

● ●

●●

●

●●●●● ● ●

●●●

●●

●●

●

●

●

●

●●●

●●

●

●●

●

●●

●

12

●

●

●

●

●●

●

●●

●●

●

●●

●●●●

●

●●●

●●

●●

●

●●

●

●

●

●

●

●●

●

●

●

●●●

●

●

● ●●

●

●

●●

●

●●●●●

● ●

●

●

● ●●●●

●●●●●●

●

●●

●

●●

●●●●●

●●●

●●

●●● ●●

●

●

●

●

●●

●●

●●

●

●●●

●

●

●●

●

●

●●

●●●

●

●

●

●

●●●

●

●●●

●

●

●

●●●

●

●

●

●

●

●

●

●●●

●●

●●●

●

●

●●●

●●●

●

●●

●● ●

●●

●●●

●●●●

●●●

●●●●●●

● ●●●

●

●●

●●●

●

●

●●● ●

●● ●

●

●

●●

●●

●

●

● ●

●

●

●

●

●

●

●●

●

●

●●

●

●●

●

●●●

●●

●

●

17

●●

●●

●

●●

●●

●

●

●

●

●

●

●● ●●

●

●

●

●

●

●

●●

● ●

●

● ●● ●●

●●

●

●●

●

●

●

●

●

●●

●●●

●●

●

●●●

●●

●

●

●●

●

●● ●

●

● ●●●●●

●●

●

●

●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●●●

●

●●●●

●●

●●●

●

●

●●

●

●

●

●

●●●●

●

●

●●

●

●

●●

●●●

●

●●

●

●

●

●●●●●●●●

●●

●●●●●

●

●

●●●●●● ●

●

●

●●

●

●

●

●●●

●●●

●●

●

●●

●●

●●●

●

●

●●

●

●●

●

●●

●

●●

●●●

●●

●

●

●●●●

●●●

●

●

●

●

●●

●

●●●

●

●

●

●●

●

●

●

●

●●●

●

●

●

2 3 4 5 6 7

3

●

● ●●●

●

●●●

●

●

●●●

●

●● ●●●

●●●

●●●

●●

●●●

●

●

●

●

●

●●

●●

●●

●

●●

●●

●

●●●●●

●

●

●

●●

● ●

●● ●

●

● ●●

●

●

●●

●

●

●●

●●●●

●

●●●

●

●

●●

●

●●

●

●

●

●●●

●

●●

●●

●●

●●●

●

●

●

●

●●

●●● ●●

●

●

●

●●●

●

●

●●●●

●

●●

●

●●●

●●

●●

●

●●●●

●

●●

●●●

●●●

●●●

● ●

●● ●

● ●●●●● ●

●●●

●

●

● ●

●

● ●●

●●

●●

● ●

●

●●

●●

●

●●

●●● ●

●

●

●

●

●●●●●●●

●

●

●●

●

●●

●

●●

●

●

●●

●

●●

●●●●

●●●

●

8

●

●

●

●●●

●●●

●

●

●●●

●

●

●

●

●

●●●

●

●

●●●

● ●

●●

●

●

●●

●●

●●

●

●●

●

●●

●

●

●

●

●●● ●●●

●

●●

● ●●●

●

●●

●

●●

●●

●●●

●●

●●●

●

●●●

●●

●●●

●

●

●●

●●●●

●

●

●

●

●●●●

●●●●

●●●●●●●●

●●

●

●

●

●●●

●

●

● ●

●

●

●

●

●

●

●●

● ●

●

●

●

● ●

●●

● ●●

●

●

●● ●●

●●●●●

●

●● ●

●

●

●●

●

●●

●●●

●●●●

●●

●●●●

●

●●

●

●●

●

●

● ●●● ●

●●●

●●●

●

●●●

●●●●

●

●

●●

●

●●●

●

●

●

●

●●

●●

●● ●

●

●●●

●

●

13

●

●●

●

●

●

●●

●

●●●●

●

●

●

●●●

● ●●

●●

●

●

●●

●

●● ●●

●

●

●

●

●●

●

●●

●

●●

●●

●

●●●

●

●●●●

●●

●

●●

●

●

●

●

●●●●●●●

●

●●●

●

●

●

●

●●●

●

●

●●

●

●

●●●●●●

●

●●

●

●●

●

●

●

●

●

●●

●

●●

●●●●

●

●●●

●

●

●

●

●●

●

●●●●●●

●●●

●

●●

●

●

●●

●

●

●

●

●

●●●

● ●●

●

●●

●

●

●

●●●

●

●●● ●

●

●

●●●●●

●

●

●

●

●●●●

●

●

●

●●

●●

●

●

●

●

● ●

●●●

●

●

●

●

●

●●

●●

●

● ●

●

●●

●●

●

●

●●

●

●●●

●●

●

● ●

●●●

●●

●

18

●

●

●

●

●●

●

●●

●●●● ●

●

●●

●

●●

●●●

●●●●●

●●

●

●

●

●●

●●

●●●

●

●

●

●●●

●

●●

●●●

●●●●

●●

●

●●

● ●

●

●

●

●●●●●●

●●●

●

●

●●●

●

●

● ●●

●●●

●●

●

●●●●●

●

●●

●

●

●●

●●●

●

●

●●

●●●●●

●

●

●● ●●●

●

●●

●

●●

● ●●● ●●●

●

●●

●

●●

●

●●●

●●●

●

●

● ●

●

●

●

●

●

●

●

●●

●● ●

●

●●

●

●

●●●

●●

●

●

●●

●●●●

●●●

●

●●●

●●●

●

●

●●●

●

●●

●

●●

●

●●●●

●

●

●●

●

●●

●

●

●●

●

●●●●

●●

●●

●

●

●

●● ●

2 3 4 5 6 7

4

●●

●

●

●●●

●●

●

●

●●

●

●

●●

●

●● ●

●●●●●●●

●

●

● ●●●

●●●

●

●●

●

●

●

●●

●

●

●●●●●

●●●

●

●●

● ●

●●

●●

● ●

●●

●●●●

● ●●●

●●●●

●

●

●●

●●●

●●

●●

●●

●●●

●

●● ●●

●●

●

●●

●

●●

●●●

●●●

●●●●●●

●

●●●

● ●

●● ●

●

●●●

●

●

●

●●●

●

●

●●

●●●●●●

●●

●●

●●

●●

●

●

●

●

●

●●●

●

●●

●

●●●●

●●

●●

●●

●

●

● ●●●

●

●

● ●

●●

●●●

●●● ●●

●

●●●●●

●●

●

●

●

●●●

●

●

●●●

●

●

●●

●

●

●●

●

●

●●

●●●

9

●●

●●●

● ●●●

●

●●

●

●

● ●● ●●●●●●●

●●

●

● ●●● ●● ●

●●●

●●●

●●

●

●●

●●

●●●●●

●●●●●●

●●

● ● ●●

●●

●● ●●

●●

●●●●●

●●●●

●● ●●

●●●●

●●●●

●●●

●

●● ●●

●●

●

●

●●

●●

●●

●●

●●

●●

●●●

●●●●● ●

●

●

●

●

●● ●●

●

●●●●●

●

●●●● ●●

●

●

●

● ●●

●

●●●

●

●●

● ●● ●

●●●●

●

●●●●●●

●●●●●●●

●

●

●●

●●● ●

● ●

●●●●●

●

●

●●

●●●●●●●●

●●

●●●

●●●

●●

●

●●●

●●●● ●●●●●●

●

14

●●●●

●●

●●●

●●●●

●

●●

●●

●●

●

●

●

●

●

●

●

●●●

●

●●

●

●●

●●

●●

●●

●●

●

●●

●●●●●

●

●

●

●

●

●

●

●● ●

●

●

●●

●● ●●●

●

● ●●●●●

●●●●●

●●●

●

●●

●

●●

●●●

●

●

●● ●

●

●

●

●●●

●

●●

●●●●

●

●

●●

●●

●

●

●

●●● ●

●●●

●

●● ●

●

●

●

●●●

●●●

●

●

●●

●●●

●

● ●

●

●●

●

●●

●

●●●●

●

●

● ●

●●

●●●●

●

●

●

●

●●●

●

●

●

●●

●●

●●

●

●

●

●

●

●●

●

●●

●

●

●●

●

●●●●

●

●

●

●

● ●

●

●

●●

●

●

●

●●

●●●

●

●

●●●●●

●

19

●●●

●

●

● ●●●

●●●

● ●

●

●

●●

●

●●

●●

●

●

●

●

●

●●●

●

● ●●●●

●

●

●

●●

●

●●

●

●

●●●

●●

●

●

●

●

●

●

● ●

● ●

●●●

●

●

●

●●●●●

●

●

●●●●●

●●

●●●

●●

●

●

●

● ●

●●●

●

●●● ●

●

●●

●

●

●● ●

●●

●

●●

●

●●●

●●

●

●●

●

●

●● ●

●●

●

●●

●

●

●

● ●

●

●

●

●

●

●●●

●●●

●●

●

●

●

●

●

●●●

●

●● ●

●●

●

●

●●

●

●

●

●●●●

●●●

●●●

● ●

●

●

●●

●

●

●

●●

●●

●●●

●

●●

●

●

●

●●●●●●

●

●

●●

●

●●

●

●●

●

●●●

●

●

●● ●

●●●

●

●

●

2 3 4 5 6 7

5

●

●

●●

●●

●●●

●●

●● ●

●●

●

●●●

●●

●

●

●●● ●

●●●

●●

●●●●

●

●

●

●● ●

●● ●

●

●

●●

●

● ●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●●●

●

●●●

●●●

●●

●

●

●

●

●●●

●●

●

●● ●●

●

●

●●

●

●●

●

●●

●

●

●

●

●●●

●

●●

●

●●●

● ●●●

●

●●

●

●

●

●

●

●

●●●

●●●●

●

●

●

●

●

●

●●

●●●●●

●●

●

●●

●

●

●

●

● ●

●

●●

●

●

● ●

●

●

●●●●

●

●

●

●●

●●●

●

●

●

● ●

●

●

●

●

●

●

●●

●

●

●●

●●

●

●

●

●

●

●●

●

●●

●

●

●●

●●

●

●

●●

●●

●●

●

10

●

●●●

●

●

●

●●

●

●

●●

●

●●● ●●●

●●

●

● ●●

●

●●

●● ●

●

●●●●

●●●●

●

●●●

●

●

●●●●●

●

●

●

●●

●

●

●●

●

●●●

●

●

●

●●

●●●

●●

●●●●

●●

●

●

●●

●●●●

●●

●●●●●

●

●●●●

●●

●●●●

●●●●

●

●

●

●●●●● ●

●●

●

●

●

●●●

●

●

●●

●

●●

●

●●

●●

●

●●●●

●●●●●● ●●

●●●●● ●●

●

●

● ●

●

●

●●

●

●

●●

●●●

●

●

●●●●●

●●

● ●●

●●

●

●

●

●●

●●●

●

●

●

●

●

●

●●

●

●●

●●

●

●●

●●

● ●●●

●

●

●●

●●

●

●●●●●

●●

●

15

●

●

●●

●●

●

●

●

●●●● ●

●●

● ●

●●

●

●

●

●

●

●

●

● ●●●

●

●

●●

●

●

●●●

●●

●

●●

●

●

●

●●●●

●

●●

●

●●

● ●

● ●

●●

●

●

●

●

●

●●●●

●●

●

●

●●●

●

●●

●

●

●●

●●●● ●

●●●

●

●

●

●●

●

●● ●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●

●

●●●

●●● ●

●●

●

●

●●●

●

●

●●

●●

●●●

●●

●●

●

●

●

●

●

●●

●

●●

●●●

●●

●

●

●●

●

●

●

●●●

●●

●

●

●●●

●

●

●●

●

●

●

●

●

●●●

● ●●

●

●●

●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●

●

●●

●●●●

●

●●●

●

●●

20

●●

●

●

●●

●

●●

●●●●

●

●

●● ●

●●

●

●●●

●●

●●

●

●●●

●

●

●●●

●

●●●●

●●●

●●

●●

●

●

● ●●●●

●●

●

●

●

●

●●

● ●

●

●

●

●●●●

●●●

●●

●

●

●

●

● ●

●●● ●

●

●

●●●

●●●

●●●

●

●

●●

●

●

●

● ●●

●

●

●●

●● ●

●●

●●

●●●●●

●

●

●●

●

●

● ●

●

●

●

●●

●

●●●●

●

●

●●●●●●

●●

●

●

●

●●

●●

●

●●●●● ●

●

●

●●

●●

●

●

●

●●

●●●●

●●

●●

●

●

●

●

●

●

●

●

●

●●

●●

● ●

●

●

●

●

●

●

●●

●

●

●

●●●

●

●

●●

●

●●

●

●

●●●

●

●

●

●

●

● ●

2 3 4 5 6 7


mpg$gp100m <- 100 / mpg$ctyqplot(displ,gp100m, data = mpg, colour = factor(year))

mpg_rotate <- resamp(mpg, null_model(gp100m ~ displ))last_plot() %+% mpg_rotate + facet_wrap(~ .n)

Maybe there are fewer bigger cars?

displ

count

5

10

15

20

25

30

35

2 3 4 5 6 7

factor(year)19992008

displ

coun

t

10203040

10203040

10203040

10203040

1

6

11

16

2 3 4 5 6 7

2

7

12

17

2 3 4 5 6 7

3

8

13

18

2 3 4 5 6 7

4

9

14

19

2 3 4 5 6 7

5

10

15

20

2 3 4 5 6 7

factor(year)19992008

qplot(displ, data = mpg, colour = factor(year) geom = "freqpoly", binwidth = 1)last_plot() %+% mpg_perm + facet_wrap(~ .n)

Future work

Methods for more null hypotheses.

Is it possible to guess plausible null hypotheses from the plot specification?

graphical inference

Education