ON NONPARAMETRIC INTERVAL ESTIMATION

OF A REGRESSION

FUNCTION BASED ON THE RESAMPLING

ALEXANDER ANDRONOVRiga Technical University

Riga, Latvia

OUTLINE 1. Introduction 2. Averaging Method 3. Median Smoothing 4. Case Study References

1. INTRODUCTIONWe consider nonparametric regression

xmY , (1.1)

where Y is a dependent variable, m is an unknown regression function, x is a d -dimensional vector of independent variables (regressors), is a random term. It is supposed that the random term has zero expectation ( 0E ) and variance )(2 xVar where 2 is an unknown constant and )(x is a known weighted function. Furthermore we have a sequence of independent observations ii xY , ,

ni ..,,2,1 . On that base we need to construct an upper confidence bound xm~ for xm at the point x corresponding to probability :

xmxmP ~ . (1.2)

Usual way [DiCicco and Efron, 1996] consists of using a consistent and asymptotic normal distributed estimate xm of xm . A final expression contains derivatives xm , xm and variance 2 that are replaced by corresponding estimators.

The resampling approach [Wu, 1986, Andronov and Afanasyeva, 2004] gives an alternative way that can be described as follows. For fixed point x we take k nearest neighbors

kxxx ...,,, 21 of x among nxxx ...,,, 21 (in some sense, for example using any kernel function iH xxK , Mahalanobis or other distance):

)(:,...,, 21 xIixxxx cik ,

where

....,,, among of neighborsnearest theof one is : 21 nic xxxxkxixI

Now we have sample kk YxYxYx ,...,,,,, 2211 instead of

nn YxYxYx ,...,,,,, 2211 .

Then we derive sample without replacement { riii ...,,, 21 } of size r ( kr ) from set {1, 2, …, k}, form resample YxYxYx r ,...,,,,, 2211 ,

where jij xx and

jij YY , and calculate estimate )(xm of our function

of interest xm .

Then we return all selected elements into initial samples and repeat this procedure R times. As a result the sequence of estimators xmxmxm R

...,,, 21 takes place. After ordering we have sequence xmxmxm R)()2()1( ...,,, where xmxm ii )1()( .

Let number R is selected so that R is an integer. Then we set xmxm R~ .

Averaging method and Median smoothing method for xm calculation are considered. Our main aim is to elaborate a numerical method for cover probability calculation:

xmxmPx ~)(Pr . (1.3)

It means that we need to known a distribution of the R -th order statistic

xm R )( . That is a main problem that it necessary to solve.

2. AVERAGING METHOD At first we consider the method of kernel regression estimation [Hardle etc., 2004]. Let HK be any kernel function (Epanechnikov, Quartic and so on). Then Nadaraya-Watson point estimator xm is calculated by the formula

r

iiiHr

iiH

YxxKxxK

xm1

1

1

, (2.1)

where ix and

iY are the vectors of independent variables and dependent variable for the i -th elements of the resample, ri ...,,2,1 .

The resampling procedure gives us sequence xmxmxm R ...,,, 21 ,

r

iiiHr

iiH

j jYjxxKjxxK

xm1

1

1

(2.2)

where jxi and jYi

are the vectors of independent variables and dependent variable for the i -th elements of the j -th resample,

ri ...,,2,1 , Rj ...,,2,1 .

With respect to (1.1) we have:

jxmjxxKjxxK

jxxmE i

r

iiHr

iiH

j

1

1

1 ,

jxwjxxK

jxxK

jxxmVar i

r

iiH

r

iiH

j

1

22

1

2 ,

where jxjxjxjx r ...,,, 21 .

Then

z

r

iHr

iH

zjj zmzxK

zxKrk

zxmE

rk

xmE1

1

111 ,

(2.3) where the sums are taken on - a set of all r -samples without replacement from }....,,,{ 21

kxxx

Analogous expression we are able to write down for unconditional variance.

At first let us calculate the second moment:

zi

r

iiH

r

iiH

z

zjYzxKE

zxKrk

zxmE

rk

xmE

2

12

1

22

)(11

)(1)(

.)()(2

)()(11

1

1 1

22

1

22

1

r

i

r

ijjijHiH

zii

r

iiH

r

iiH

zmzmzxKzxK

zmzwzxK

zxKrk

Now the variance can be calculated by formula

.)()()( 22 xmExmExmVar (2.4)

Now we need to calculate the covariance between two various estimates xm j

and xm j ' . We have for j j’:

.)()()(

)()()()())(),((2

'

''

xmExmxmE

xmxmxmxmExmxmCov

jj

jj

d

jj

(2.5)

Therefore

z r

iiH

jjzxKr

kxmxmCov

1

22

)('),(

.)(1 2

1

v

r

vzzmmHr

iiH

m

zwzxKvxK

(2.7)

To avoid computational difficulties, it is possible to consider the following estimate instead of (2.1):

r

iiY

rxm

1

1)( (2.8)

and corresponding those sequence xmxmxm R ...,,, 21 .

Expectations, variances and covariance matrix for this sequence of random variables can be determined using the following lemmas.

Lemma 1 Let kZZZ ...,,, 21 be independent random variables with expectations

k ...,,, 21 and variances 222

21 ...,,, k . Let

rZZZ ...,,, 21 be a random sample of size r from kZZZ ...,,, 21 without replacement and S be their sum:

rZZZS ...21 . Then

kkrSE ...21 , (2.9)

1

1 12

1

22

12

k

j

k

jiji

k

jjj kk

rkrk

rkkrSVar . (2.10)

Lemma 2 For the conditions of the previous Lemma let sample

rZZZ ...,,, 21 be returned into set kZZZ ...,,, 21 and the described procedure be repeated, so that we have new sample

rZZZ ...,,, 21 and a corresponding sum rZZZS ...21 . Then the covariance between S and S is calculated by formula

k

ii

k

iii k

rkkk

rkkrSSCov

1

22

1

2

11, . (2.11)

3. MEDIAN SMOOTHINGAs it is noted by [Hardle etc, 2002] “Median smoothing may be described as the nearest-neighbor technique to solve the problem of estimating the conditional median function, rather than the conditional expectation function … . The conditional median function xYmed is more robust to outliers than the conditional expectation xYExm .”

For the resample rijYjx ii ...,,2,1;, the median smoother is defined as

jYjYjYmedxm rj ...,,, 21 . (3.1)

To evaluate the cover probability (1.3) we apply an approach that has been elaborated by [Andronov and Afanasyeva, 2004]. The idea of this approach is such: any median estimator coincides with some element of the initial sample. Therefore we need to calculate a corresponding probability for this element.

At first we consider the following problem. Let kYYY ...,,, 21 be the order statistics for initial dependent variables

kYYY ...,,, 21 and dxQ , be the probability of event 1)( dd YxmY :

1)(, dd YxmYPdxQ , nd ...,,1,0 , (3.2)

where 0Y , 1nY .

Formally, this probability can be expressed by formula

zvdxz zv

vmxmFvmxmFdxQ 1,),(

(3.3)

where dx, is a set of all d -samples dxxxz ...,,, 21 without

replacement from kxxx ...,,, 21 , xPxF is a distribution

function of the random term from formula (1.1).

Furthermore for the concrete r -resample jx jxjxjx r

...,,, 21 and corresponding median smoothing (3.1) we are able to calculate a probability that calculated median smoothing

xm j will be greater than xm : if d is fixed then we have a case of

hypergeometrical distribution:

2/)1(

0

1,r

i irdk

id

rk

xmxmPdxq (3.4)

(we remind that r is an odd number).

The conditional probability of interest (1.3) given fixed d is calculated by such way:

iRiR

Ridxqdxq

iR

dxmxmPdx

,1,~,Pr

. (3.5)

Finally we got the cover probability of interest:

2/1

0,Pr,~Pr

rk

ddxdxQxmxmPx . (3.6)

4. Case StudyWe consider a numerical example that illustrates the median smoothing. Let regression function be a such:

21.01 xxxm , < x < . (4.1)

Of course one is unknown for us, but we wish to calculate a true cover probability for this regression if we use the above described median smoothing. The following assumptions are supposed: random term has the normal distribution with zero mean and variance 2 Var = 4, a number of observations (a size of the initial sample) n = 22, values of the independent variables are ix i – 4, i = 1, 2, … , 22.

Further we assume that the confidence probability = 0.8, a number of considered nearest neighbors k = 9, a resample size r = 5.

Gotten results are presented in Table 1 that contains the values of regression function m(x) and cover probabilities )(Pr x . The last have been calculated with respect to above described procedure. Table 1

The cover probabilities

x -3 -2 -1 0 1 2 3 m(x) 4.9 3.4 2.1 1 0.1 -0.6 -1.1

)(Pr x 0.932 0.925 0.921 0.920 0.922 0.928 0.934

x 4 5 6 7 8 10

m(x) -1.4 -1.5 -1.4 -1.1 -0.6 1 )(Pr x 0.939 0.941 0.939 0.934 0.928 0.920

The presented results show that the considered median smoothing approach is the conservative estimation method because one sets the cover probabilities too high. We would like to remark too that cover probability )(Pr x is not monotone and convex function of argument x.

Proofs of the LemmasProof of Lemma 1Let 1j if the random variable jZ belongs to the sample

rZZZ ...,,, 21 and 0j otherwise. Of course k ...,,, 21 are

dependent random variables because rk ...21 . We have: krP j /1 , krP j /10 , krPE jj /1 , krkrVar j //1 , 1/11,1 kkrrPE jiji

for ji . Furthermore

i

k

iiZS

0 .

Random variables i and iZ are independent therefore

k

iii

k

ii

k

iii k

rZEEZESE111 ,

22222iiiiii k

rZEEZE ,

. 122

22222

kr

kr

kr

krZEZEZVar

ii

iiiiiiiii

(A.1)

Random variables iZ , jZ and ji for ji are independent too therefore

1

1

kkrrZEZEEZZE jijijijjii ,

11

1, 2

2

kk

rkrkr

kkrrZZCov jijijijjii . (A.2)

Formulas (A.1) and (A.2) give formula (2.10).

Proof of Lemma 2Let

i

k

iiZS

1 , j

k

jjZS

1 .

Then

.,,

,,),(

11

1 111

k

i ijjjii

k

ijjii

k

i

k

jjjiij

k

jji

k

ii

ZZCovZZCov

ZZCovZZCovSSCov

For i j random variables jiji ZZ ,,, are independent, therefore

jjii ZZCov , = 0. Further

.

,22

2

2

iiiiii

iiiiiiiiiii

kr

krZEZEE

ZEZEZEZZCov

Therefore

k

iik

rSSCov1

22

, .

REFERENCES[1] Andronov A., Afanasyeva H. Resampling-based

nonparametric statistical inferences about the distributions of order statistics. In: Transactions of XXIV International Seminar on Stability Problems for Stochastic Models, Transport and Telecommunication Institute, Riga, Latvia, 2004, pp. 300 – 307.

[2] DiCiccoT.J., Efron B. Bootstrap confidence intervals. Statistical Sciences, Vol.11, No.3, 1996, pp. 189 – 228.

[3] Hardle W., Muller M., Sperlich S., Werwatz A. Nonparametric and Semiparametric Models. Springer, Berlin, 2004.

[4] Wu C.F.J. Jackknife, Bootstrap and other resampling methods in regression analysis. The Annals of Statistics, Vol. 14, No. 3, 1986, pp. 1261 – 1295.

Thank You for your attention