![Page 1: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/1.jpg)
Empirical Processes with Applications in Statistics4. Tools to work with empirical processes
Johan Segers
UCLouvain (Belgium)
Conférence Universitaire de Suisse Occidentale
Programme Doctoral en Statistique et Probabilités Appliquées
Les Diablerets, February 4–5, 2020
1 / 23
![Page 2: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/2.jpg)
So far so good, but. . .
So far: general framework for proving weak convergence
Gn =√
n (Pn − P) G, n → ∞
in `∞(F ) for suitable families of functions F ⊂ L2(P)
How to find the asymptotic distribution of estimators of “parameters” arisingas statistical functionals φ?
ϑn = φ(Pn),
ϑ = φ(P)
=⇒ Functional delta method
Often, the limit distribution is unknown and complicated: in practice?=⇒ Bootstrap
Main sources for this lecture: van der Vaart and Wellner (1996), Kosorok (2008)
2 / 23
![Page 3: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/3.jpg)
Tools to work with empirical processes
Functional delta method
Bootstrap
3 / 23
![Page 4: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/4.jpg)
Ordinary delta methodRandom vectors Xn in Rd . Suppose θ ∈ Rd and 0 < rn → ∞ and
Zn =: rn(Xn − θ) d−→ Z , n → ∞
Delta method. Let φ : Rd → Rp be differentiable at θ. Then
rnφ(Xn) − φ(θ)
d−→ φ′(θ)Z
where φ′(θ) ∈ Rp×d is the Jacobian of φ at θ.
Proof: Write
rnφ(Xn) − φ(θ)
= rn
φ(θ + r−1
n Zn) − φ(θ)
= gn(Zn)
By differentiability of φ, for all sequences zn → z ∈ Rd ,
limn→∞
gn(zn) = g(z) = φ′(θ)z
Apply the extended continuous mapping theorem.
4 / 23
![Page 5: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/5.jpg)
Delta method in normed spaces?
Empirical process Gnf =√
n(Pnf − Pf) indexed by P-Donsker F ∈ L2(P):
Gn G, in `∞(F )
“Smooth” transformationφ : `∞(F )→ E
where E is some other normed real vector spaceOften E is Rp or `∞(T) for some set T
Functional delta method? Asymptotic distribution of statistical functionals√
nφ(Pn) − φ(P)
φ′P(G), in E
with “derivative” φ′P : `∞(F )→ E, linear and bounded
5 / 23
![Page 6: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/6.jpg)
Differentiable mappings between normed spacesMap φ : D→ E between normed real vector spaces
Definition: Hadamard differentiability.φ : D → E is Hadamard differentiable at θ if there exists boundedlinear φ′θ : D→ E such that, in E,
limt↓0
1tφ(θ + tht ) − φ(θ)
= φ′θ(h)
whenever limt↓0 ht = h in D.
I Stronger than Gateaux differentiability (fixed ht ≡ h)I Weaker than Fréchet differentiability (bounded ht )I In Rd , all three definitions are the same as the usual oneI Similar definition if φ has domain Dφ ⊂ D
6 / 23
![Page 7: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/7.jpg)
Slight generalization in view of applications
Definition: H-dif tangentially to a set.φ : Dφ → E is Hadamard differentiable at θ ∈ Dφ tangentially to D0 ⊂
D if there exists bounded linear φ′θ : D0 → E such that, wheneverht t>0 ⊂ D satisfiesI θ + tht ∈ Dφ for all sufficiently small t > 0I limt↓0 ht = h ∈ D0
thenlimt↓0
1tφ(θ + tht ) − φ(θ)
= φ′θ(h)
Chain rule for differentiation applies as usual: facilitates proving H-dif
7 / 23
![Page 8: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/8.jpg)
Functional delta method
I Normed linear spaces D and EI Dφ ⊂ D
I φ : Dφ → E
I Tn : Ω→ Dφ
I 0 < rn → ∞ and θ ∈ Dφ
Theorem: Functional delta method. If
1. If rn(Tn − θ) Z in D with Z taking values only in D0
2. φ is Hadamard differentiable tangentially to D0
then, in E,rn
φ(Tn) − φ(θ)
φ′θ(Z), n → ∞
8 / 23
![Page 9: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/9.jpg)
Asymptotic normality of statistical functionals
Suppose F ⊂ L2(P) is P-Donsker.
Setting D = `∞(F ) and Tn = Pn and θ = P, we find, in E,√
nφ(Pn) − φ(P)
φ′P(G), n → ∞
providedI Pn takes values in Dφ ⊂ `
∞(F )
I G takes values in D0 ⊂ `∞(F )
I φ : Dφ → E is Hadamard-differentiable at P tangentially to D0
The E-valued weak limit φ′P(G) is Gaussian as wellFor every continuous linear functional ρ : E→ R, the random variable ρ φ′P(G) is Gaussian
9 / 23
![Page 10: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/10.jpg)
Example: sample quantiles
Iid random variables X1,X2, . . . ∼ F on R, empirical distribution function Fn.
Population and sample quantiles: for 0 < p < 1,
Q(p) = infx ∈ R | F(x) > p
= φ(F)
Qn(p) = infx ∈ R | Fn(x) > p
= φ(Fn)
Asymptotic normality of√
nQn(p) − Q(p)
by functional delta method:I D = `∞(R) and E = R
I Dφ = all distribution functions on RI Tn = Fn and θ = F and rn =
√n
I Limit process G(x) = B(F(x)), x ∈ R, with B a Brownian bridge
10 / 23
![Page 11: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/11.jpg)
“Vervaat’s lemma”If F has derivative f at Q(p), then φ is H-dif at F tangentially to
D0 =h ∈ `∞(R) | h is continuous at Q(p)
with derivative map
φ′F (h) = −h(Q(p))
f(Q(p)), h ∈ D0
By the functional delta method,
√nQn(p) − Q(p)
d−→ −
B(p)
f(Q(p))∼ N
(0,
p(1 − p)
(f(Q(p)))2
)
Extension: empirical quantile processQn(p)
p∈[a,b]
11 / 23
![Page 12: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/12.jpg)
Ds
Il 0A
V
12 / 23
![Page 13: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/13.jpg)
Example: copulas
The copula C of a d-variate random vector X = (X1, . . . ,Xd) with joint cdf Fand continuous marginals F1, . . . ,Fd is the joint cdf of
(U1, . . . ,Ud) = (F1(X1), . . . ,Fd(Xd))
A copula is a cdf on [0, 1]d with uniform [0, 1] marginals. Explicit formula:
C(u1, . . . , ud) = F(F−1 (u1), . . . ,F−d (ud)
)with F−j the quantile function, defined carefully as
F−j (uj) =
infx ∈ R | Fj(x) > uj
, 0 < uj 6 1
supx ∈ R | Fj(x) = 0
, uj = 0
Context: Modelling dependence in a margin-free way
13 / 23
![Page 14: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/14.jpg)
Empirical copula processiid sample Xi = (Xi1, . . . ,Xid) from F . Empirical copula:
Cn(u1, . . . , ud) = Fn
(F−n,1(u1), . . . ,F−n,d(ud)
)with empirical cdfs
Fn(x) =1n
n∑i=1
1
Xi1 6 x1, . . . ,Xid 6 xd
Fn,j(xj) =
1n
n∑i=1
1
Xij 6 xj
and generalized inverses as on previous slide. Rank-based estimator
Weak convergence in `∞([0, 1]d) of the empirical copula process?√
n (Cn − C) =√
nφ(Fn) − φ(F)
14 / 23
![Page 15: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/15.jpg)
Differentiability of the copula mappingI D = E = `([0, 1]d)I Dφ =
cdf F on [0, 1]d with margins Fj(0) = 0
I φ : Dφ → E : F 7→ F(F−1 , . . . ,F
−d )
Bücher and Volgushev (2013). Assume the d partial derivatives ∂jCexist and are continuous on u ∈ [0, 1]d | 0 < uj < 1. Then φ isHadamard differentiable at C tangentially to
D0 =α ∈ C([0, 1]d)
∣∣∣ α(1, . . . , 1) = 0,
α(x1, . . . , xd) = 0 if minj
xj = 0
with derivative φ′C : D0 → `∞([0, 1]d) given by
(φ′C(α)
)(u) = α(u) −
p∑j=1
∂jC(u)α(1, . . . , 1, uj , 1, . . . , 1)
15 / 23
![Page 16: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/16.jpg)
Weak convergence of the empirical copula process
For C ∈ Dφ satisfying the assumption, we obtain weak convergence√
n (Cn − C) β, n → ∞
in `∞([0, 1]d) with weak limit
β(u) = GC(u) −d∑
j=1
∂Cj(u)GC(1, . . . , 1, uj , 1, . . . , 1), u ∈ [0, 1]d
with GC a C-Brownian bridge indexed by functions 1[0,u]
Weak convergence of empirical copula process under the same or similarconditions already before obtained by other authors
16 / 23
![Page 17: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/17.jpg)
Tools to work with empirical processes
Functional delta method
Bootstrap
17 / 23
![Page 18: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/18.jpg)
Using asymptotic distributions
Estimator ϑn = φ(Pn) of ϑ = φ(P). Asymptotic distribution√
n(ϑn − ϑ
) φ′P(G), n → ∞
Usage:I asymptotic confidence intervalsI critical values for hypothesis tests
Problem: limit distribution depends on unknown P via φ′P and G
Possible solutions:I Replace P in limit distribution by Pn
I Consistent procedure under fairly general conditionsI Estimated limit distribution may be difficult to handle
Imagine a Kolmogorov–Smirnov test for the empirical copula process
I Bootstrap
18 / 23
![Page 19: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/19.jpg)
Resampling with replacement
Empirical measure when resampling from the data with replacement:
Pn =1n
n∑i=1
Mni δXi
I Mni : number of times Xi was selected out of n trialsI (Mn1, . . . ,Mnn) multinomial (n; 1/n, . . . , 1/n) independent of X1, . . . ,Xn
I δx is degenerate probability measure at x
Bootstrap: estimate the distribution of√
nφ(Pn) − φ(P)
by the one of
√nφ(Pn) − φ(Pn)
To be shown: with large probability, the conditional distribution of the abovequantity given the data X1, . . . ,Xn is close to the limit distribution on slide 18
19 / 23
![Page 20: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/20.jpg)
Showing consistency of the bootstrapTwo steps:
1. Show consistency of the bootstrapped empirical process:with large probability, the conditional distribution of
Gn =√
n(Pn − Pn
)=
1√
n
n∑i=1
(Mni − 1)(δXi − Pn)
given the data X1, . . . ,Xn is close to the one of G
2. Apply (a suitable extension of) the delta method
Method extends to more general weights Wni than Mni − 1I multiplier bootstrapI wild bootstrapI . . .
20 / 23
![Page 21: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/21.jpg)
Multiplier central limit theoremConsistency in probability of the bootstrapIf F ⊂ L2(P) has finite envelope, then F is P-Donsker if and only ifGn is asymptotically measurable and
suph∈BL1
∣∣∣EM[h(Gn)] − E[h(G)]∣∣∣ P→ 0, n → ∞
I EM means conditional expectation over (Mn1, . . . ,Mnn) given X1, . . . ,Xn
I BL1 is the set of h : `∞(F )→ [−1, 1] such that∣∣∣h(z) − h(y)∣∣∣ 6 ‖z − y‖∞, y, z ∈ `∞(F )
“Bounded Lipschitz distance” suph∈BL1| . . . | metrizes weak convergence
I Proof based on
I Poissonization of (Mn1, . . . ,Mnn)I (conditional) multiplier central limit theorem
21 / 23
![Page 22: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/22.jpg)
SummaryThis course:I Weak convergence of empirical processes indexed by function classesI Tools to leverage the power of the asymptotic theoryI Examples
What’s next:I Convergence rates different from
√n (van de Geer, 2000)
I Concentration inequalities (Giné and Guillou, 2001)I Empirical processes indexed by estimated functions (van der Vaart and
Wellner, 2007)I Application to M and Z-estimatorsI Almost sure versions, strong approximationsI Time series dataI . . .
Thank you!
22 / 23
![Page 23: Empirical Processes with Applications in Statistics - 4. Tools ......Empirical Processes with Applications in Statistics 4. Tools to work with empirical processes Johan Segers UCLouvain](https://reader036.vdocuments.mx/reader036/viewer/2022081601/60f69b5bf105f96d583eb259/html5/thumbnails/23.jpg)
References
Bücher, A. and S. Volgushev (2013). Empirical and sequential empirical copula processesunder serial dependence. Journal of Multivariate Analysis 119, 61–70.
Giné, E. and A. Guillou (2001). On consistency of kernel density estimators for randomlycensored data: rates holding uniformly over adaptive intervals. Ann. Inst. HenriPoincaré 37, 503–522.
Kosorok, M. R. (2008). Introduction to Empirical Processes and Semiparametric Inference.New York: Springer Sciences+Business Media.
van de Geer, S. A. (2000). Empirical Processes in M-Estimation. Cambridge: CambridgeUniversity Press.
van der Vaart, A. W. and J. A. Wellner (1996). Weak Convergence and Empirical Processes.With Applications to Statistics. New York: Springer Sciences+Business Media.
van der Vaart, A. W. and J. A. Wellner (2007). Empirical processes indexed by estimatedfunctions. In Asymptotics: particles, processes and inverse problems, Volume 55 of IMSLecture Notes Monogr. Ser., pp. 234–252. Beachwood, OH: Inst. Math. Statist.
23 / 23