lecture 16: nonparametric regression ii · i...

Lecture 16: Nonparametric regression II

Pratheepa Jeganathan

05/08/2019

Recall

I One sample sign test, Wilcoxon signed rank test, large-sampleapproximation, median, Hodges-Lehman estimator,distribution-free confidence interval.

I Jackknife for bias and standard error of an estimator.I Bootstrap samples, bootstrap replicates.I Bootstrap standard error of an estimator.I Bootstrap percentile confidence interval.I Hypothesis testing with the bootstrap (one-sample problem.)I Assessing the error in bootstrap estimates.I Example: inference on ratio of heart attack rates in the

aspirin-intake group to the placebo group.I The exhaustive bootstrap distribution.

I Discrete data problems (one-sample, two-sample proportiontests, test of homogeneity, test of independence).

I Two-sample problems (location problem - equal variance,unequal variance, exact test or Monte Carlo, large-sampleapproximation, H-L estimator, dispersion problem, generaldistribution).

I Permutation tests (permutation test for continuous data,different test statistic, accuracy of permutation tests).

I Permutation tests (discrete data problems, exchangeability.)I Rank-based correlation analysis (Kendall and Spearman

correlation coefficients.)I Rank-based regression (straight line, multiple linear regression,

statistical inference about the unknown parameters,nonparametric procedures - does not depend on thedistribution of error term.)

I Smoothing (density estimation, bias-variance trade-off, curse ofdimensionality)

I Nonparamteric regression (Local averaging, local regression,kernel smoothing, local polynomial, penalized regression)

Nonparametric regression II

Introduction

I Cross-ValidationI Variance EstimationI Confidence BandsI Bootstrap Confidence Bands

Choosing smoothing parameter


I Risk depends on unknown function r (x).

R (h) = E(1

n (rn (xi )− r (xi ))2).

1) Training errorI

1n∑n

i=1 (Yi − rn (xi ))2.

I Using data twice.I to estimate r .I to estimate the risk R.

I Function estimate is chosen to make 1n∑n

i=1 (Yi − rn (xi ))2 smallso risk is underestimated.


2) Leave-one-out cross-validation score

CV = R (h) = 1n

n∑i=1

(Yi − r(−i) (xi )

)2

I r(−i) is the estimator obtained by omitting i-th pair (xi ,Yi ).I r(−i) (x) =

∑nj=1 Yj lj,(−i) (x) , where

lj,(−i) (x) =

0 if j = i

lj (x)∑k 6=i lk (x) if j 6= i . (1)

I Set weight on xi to 0 and renormalize the other weights to sumto one.

I Do this for different h.


2) Leave-one-out cross-validation

I Intuition: E(Yi − r(−i) (xi )

)2≈ σ2 + E (r (xi )− rn (xi ))2 =

predictive error. R score is nearly unbiased estimate of the risk.I Shortcut formula to compute R

R (h) = 1n

n∑i=1

(Yi − rn (xi )1− Lii

)2,

where Lii = li (xi ) is the i-th diagonal element if the smoothingmatrix L.


3) Generalized cross-validation

GCV (h) = 1n

n∑i=1

(Yi − rn (xi )1− ν/n

)2,

where ν = tr (L) is the effective degrees of freedom.I a formula similar to Colin Mallows Cp statistic.

Variance estimation

Variance estimation

I We assume V (εi ) = σ2.I constant variance

1) For linear smoother r = LY , an unbiased estimate of σ2 is

σ2 =∑n

i=1 (Yi − r (xi ))2

n − 2ν + ν,

where ν = tr (L) and ν = tr(LT L

)=∑n

i=1 ||l (xi )||2 .

I If r is sufficiently smooth, then σ2 is a consistent estimator ofσ2.

Variance estimation

2) Alternative formula (Rice 1984).

I Suppose xis are ordered.

σ2 = 12(n − 1)

n−1∑i=1

(Yi+1 − Yi )2 .

I Intuition: an average of the residuals that results from fitting aline to the first and third point of each consecutive triple ofdesign points.

Variance estimation (Spatially inhomogeneous functions)

I Inhomegenity of variance.I rn (x) is relatively insensitive to heteroscedastic.I We need to account for the unconstant variance when making

confidence bands.

Example

I Doppler functionlibrary(ggplot2)r = function(x){

sqrt(x*(1-x))*sin(2.1*pi/(x+.05))}ep = rnorm(1000)y = r(seq(1, 1000, by = 1)/1000) + .1 * epdf = data.frame(x = seq(1, 1000, by = 1)/1000, y = y)ggplot(df) +

geom_point(aes(x = x, y = y))

Example

−0.4

0.0

0.4

0.00 0.25 0.50 0.75 1.00

x

y

Example

I Doppler function is spatially inhomogeneous (smoothness variesover x).

I Estimate by local linear regressionlibrary(np)doppler.npreg <- npreg(bws=.005,

txdat=df$x,tydat=df$y,

ckertype="epanechnikov")

doppler.npreg.fit = data.frame(x = df$x,y = df$y,kernel.fit = fitted(doppler.npreg))

p = ggplot(doppler.npreg.fit) +geom_point(aes(x = x, y = y)) +geom_line(aes(x = x, y= kernel.fit), color = "red")

Example

−0.4

0.0

0.4

0.00 0.25 0.50 0.75 1.00

x

y

Example

I Doppler function fit using local linear regression.I Effective degrees of freedom 166.I Fitted function is very wiggly.I If we smooth more, right-hand side of the fit would look better

at the cost of missing structure near x = 0.I Wavelets

Variance estimation

I Estimate r (x) with any nonparamteric method to get rn (x).I Compute the squared residuals Zi = (Yi − rn (xi ))2.I Regress Zi on xi to get an estimate q (x) .I σ (x) = q (x).

Confidence Bands

Confidence Bands

I Can we get confidence bands for r (x)?I Let mean and standard deviation of rn (x) is rn (x) and sn (x),

respectively.I Bias Problem:

rn (x)− r (x)sn (x) = rn (x)− rn (x)

sn (x) + rn (x)− r (x)sn (x)

=Zn (x) + bias (rn (x))√variance (rn (x))

.

(2)

I Typically Zn (x) = rn (x)− rn (x)sn (x) follows a standard normal

and used to derive confidence bandsI In nonparametric regression, the second term in (2) does not

vanish.I Optimal smoothing balance between bias and the standard

deviation.

Confidence Bands

I Confidence bands for rn (x) is

rn (x)± c × se (x) ,

where c > 0 some constant.I rn (x) = E (rn (x)).

I We don’t get a confidence band for r (x).I c is computed from the distribution of the maximum of a

Gaussian process. Choose c by solving

2 (1− Φ (c)) + κ0πe−c2/2 = α,

where κ0 =∫ b

a

∣∣∣∣∣∣T ′ (x)∣∣∣∣∣∣ and Ti (x) = li (x)

||li (x)|| .

Confidence Bands

I To get simultaneous confidence band, compute c such that

2 (1− φ (c)) + κ0πec2/2 = α.

I The variance of rn (x) is

V (rn (x)) =n∑

i=1σ2 (xi ) l2

i (xi ) .

I The approximate confidence band is

I (x) = rn (x)± c

√√√√ n∑i=1

σ2 (xi ) l2i (xi ).

Bootstrap Confidence Bands

I Reference: [linkhere](https://www.stat.cmu.edu/~cshalizi/402/lectures/08-bootstrap/lecture-08.pdf#page20).

1) Resample rows:I Resample (x , y) pair.

2) Resample residuals:I Hold the x fixed, but make T equal to r (x) plus a randomly

re-sampled εi .I Errors need to be iid.

https://www.stat.cmu.edu/~cshalizi/402/lectures/08-bootstrap/lecture-08.pdf#page20


Bootstrap Confidence Bands (Example)

I Resample rowslibrary(NSM3)library(dplyr)data("ethanol")ethanol.df = select(ethanol,

c(E, NOx))

resample.data = function(df) {sample.rows = sample(1:nrow(df),

replace = TRUE)return(df[sample.rows,])}


# use kernel smoothinglibrary(np)npr.nox.on.E = function(df.star) {

bw = npregbw(NOx ~ E,data = df.star)

fit = npreg(bw)return(fit)

}

Bootstrap Confidence Bands (Example)# Use uniform grid points to predict the values.evaluation.points = seq((min(ethanol.df$E) -.1),

(max(ethanol.df$E)+.1), by =.01)

eval.npr = function(npr) {return(predict(npr,

exdat = evaluation.points))}

ethanol.npr = npr.nox.on.E(ethanol.df)

##Multistart 1 of 1 |Multistart 1 of 1 |Multistart 1 of 1 |Multistart 1 of 1 /Multistart 1 of 1 |Multistart 1 of 1 |

obs.curve = eval.npr(ethanol.npr)


npr.cis = function(B,alpha, df, obs.curve) {tboot= replicate(B,

eval.npr(npr.nox.on.E(resample.data(df))))low.quantiles = apply(tboot, 1,

quantile,probs = alpha/2)

high.quantiles = apply(tboot, 1,quantile,probs = (1-alpha/2))

low.cis = 2*obs.curve - high.quantileshigh.cis = 2*obs.curve - low.quantilescis <- rbind(low.cis, high.cis)return(list(cis=cis, tboot= t(tboot)))

}


df.plot.ci = data.frame(x = evaluation.points,obs.curve = obs.curve,low.cis = ethanol.npr.cis$cis[1,],upper.cis = ethanol.npr.cis$cis[2,])

p = ggplot() +geom_point(data = ethanol.df,

aes(x = E, y = NOx)) +

geom_line(data = df.plot.ci,aes(x = evaluation.points, y = low.cis),color = "red", linetype = "dashed",size = 1)

## Bootstrap Confidence Bands (Example)

p = p +geom_line(data = df.plot.ci,

aes(x = evaluation.points, y = upper.cis),color = "red", linetype = "dashed",size = 1) +

geom_line(data = df.plot.ci,aes(x = evaluation.points, y = obs.curve),color = "blue",size = 1)

0

1

2

3

4

0.4 0.6 0.8 1.0 1.2

E

NO

x

I NotesI Confidence bands get wider where there is less data.I If variance is not constant, use resampling residuals with

heteroskedasticity method describe in the following [link4.4](https://www.stat.cmu.edu/~cshalizi/402/lectures/08-bootstrap/lecture-08.pdf#page20).



References for this lecture

W Chapter 5

Reference for bootstrap confidence bands: [linkhere](https://www.stat.cmu.edu/~cshalizi/402/lectures/08-bootstrap/lecture-08.pdf#page20).



lecture 16: nonparametric regression ii · i...

Documents