simplex techniques for quantile regression model selection · applications of simplex techniques...
TRANSCRIPT
Simplex Techniques for Simplex Techniques for Quantile Regression Model Selection
Yonggang Yao
SAS Institute Inc.SAS Institute Inc.
8/01/2010
Nonparametric Statistics — 2010 JSM, Vancouver, Canada
Outline
k d
Outline
Background
Quantile Regression
Linear Programming
Model Selection and Simplex Tableau
G d M th dGreedy Methods
Penalty Methods
Resampling
Some Computing Issues
Background
•Three Formulations of Quantile Regression (QR) at level
Background
n1
n
ii
n
iii
xyxy
xyn
21)(
21-min
)(1min 1
nn
n
iii
i
aXaXay
n
]1,0[ and 1)1( s.t. max
22
1
1
where .
Li P i (LP)
ttt )1( )(
• Linear Programming (LP)
0 and s.t.min
zbAzc'z
Background
• Cast QR problem as LP problem
Background
IIXXA
nncz
'/)1(/ 0 0
)' (
0 and s.t.min
Form) (Standard LP
zbAzc'z
where c and z are m-vectors with m=2p+2n, A is a n-by-m matrix, and
Yb
is the residual vector. XY
Simplex Theory
• Let denote an index set. d i ibl b i f A
Simplex Theory
},,2,1{},,{ 1 nBBB n ][ AAA denote an invertible sub-matrix of A.is called a basic solution if satisfies:],,[
1 nBBB AAA
],,[ **1
*mzzz *z
bAz BB1*
Form)(StandardLP
• is an optimal solution if
Bmjz j \},,2,1{for 0*
1*
*z 0 and s.t.min
Form)(StandardLP
zbAzc'z
• Simplex Tableau0'
01
1*
BB
BB
cAAc
bAz
p
AAbA
AAcczc BBBB
11
1'''
AAbA BB
Model Selection and Simplex Tableau
• Cast QR model selection problem as LP problem
Model Selection and Simplex Tableau
)'00( XXAIIXXA
nncz
)- ( '0 0 /)1(/ 0 0 )'0 0 (
22*
11
where is forced to be a zero vector.2
0ands tmin
Form) (Standard LP
bAc'z
Yb
• Simplex Tableau for Model Selection
*111
*11 '''''AAAAbA
AAccAAcczc BBBBBB
0and s.t. zbAz
where is for , and is for . U i d l d l l i
AAAAbA BBB
A 1X *A 2X
}{ BBB• Use index set to control a model selection process. },,{ 1 nBBB
Key IdeasKey Ideas
A model is like a port. The simplex tableau is like a cargo ship.
Evaluate models with data.
Model Selection and Simplex Tableau• Greedy Methods (Forward, Backward, Stepwise)
Model Selection and Simplex Tableau
1 n
• Fit Criteria
)(vs.)(1)( 21 RMWARR F
.][1min )(
][1min )(
1model-reduced
1model-full
n
iiiR
n
iiiF
xyn
MWAR
xyn
MWAR
)(vs. )(
1)( RMWAR
RR
npMWARnSICpMWARnAIClog))(log(2)(
2))(log(2)(
log))(log(2)(sSawa'
2)1(2))(log(2)(
pnnMWARnBIC
pnnpMWARnAICC
2
log))(log(2)( sSawa'
pn
nMWARnBIC
scoreWaldratio Likelihood
score WaldscoreRank
SimulationSimulation
True Model: p=20 and n=1000
)10(~)(
32 1815121021
diiUnifxxX
exxxxxxy
... )25,0(~... )1,0(~),,( 101
diiNediiUnifxxX
SimulationSimulation
Forward Selection Summary at quantile level 0.5 EFFECT Objective p-value j pstep entered function (Wald Scores) QRR ADJQRR AIC SIC0 Intercept 0.875704 0.000000 -0.000500 -131.727 -125.819
-----------------------------------------------------------------------------1 x10 0.858893 0.00001 0.019198 0.018707 -151.111 -145.2032 x15 0.850475 0.00001 0.028811 0.027838 -159.960 -148.1453 x12 0.842748 0.00001 0.037634 0.036187 -168.087 -150.3644 x1 0.837204 0.00006 0.043965 0.042047 -173.687 -150.056*5 x5 0.832609 0.00058 0.049213 0.046827 -178.192 -148.6536 x18 0.830066 0.00138 0.052117 0.049260 -180.250 -144.804
* Optimal Value Of CriterionSelection stopped as the candidate for entry has p-value> 0.1.y
Stop DetailsCandidate Candidate CompareFor Effect Significance SignificanceEntry x3 0.13957 > 0.1000 (p-value on Wald Score)
SimulationSimulation
Effects: Intercept x10 x15 x12 x1 x5 x18p
Parameter Estimates at quantile level 0.5 Standard 95% Confidence
Parameter DF Estimate Error Limits t Value Pr > |t|Intercept 1 -0.0519 0.3907 -0.8186 0.7148 -0.13 0.8944x10 1 1.3327 0.3115 0.7215 1.9439 4.28 <.0001x15 1 1.2865 0.3086 0.6809 1.8921 4.17 <.0001x12 1 1.1219 0.3066 0.5203 1.7235 3.66 0.0003x1 1 0.8864 0.3010 0.2957 1.4770 2.94 0.0033
5 1 0 7483 0 3030 0 1536 1 3430 2 47 0 0137x5 1 0.7483 0.3030 0.1536 1.3430 2.47 0.0137x18 1 0.6918 0.3021 0.0990 1.2845 2.29 0.0222
Model Selection and Simplex Tableau
• Penalty Methods
Model Selection and Simplex Tableau
LASSO penalty, OSCAR penalty, Grouped LASSO penalty
• Manipulating Simplex Algorithm for Penalty MethodsManipulating Simplex Algorithm for Penalty MethodsFor example, LASSO penalty can be measured by using vector as follow:
AA 1'''
a
Form)Costc(ParametriLP
AAAAaaAAcc
bAzazc
B
BB
BB
B
BB
BB
1
1
1
1
''''
''
0 and s.t.'min
Form)Cost -c(ParametriLP
zbAzzac'z
where = (1, 1, 0, 0) according to .)'( za
SimulationSimulation
True Model: p=11 and n=1000.
3
1 1exy
g
p
igigi
g
... )50,0(~... )1,0(~),,(
(-3,2,-2))(0,0,0,0),((2,3,2), True
101
diiNediiNxxX
),(
Solution Path for LASSO QR Solution Path for LASSO QR
. :Penalty p
is 1i
(-3,2,-2))(0,0,0,0),((2,3,2), True
Solution Path for OSCAR QR Solution Path for OSCAR QR
(-3,2,-2))(0,0,0,0),((2,3,2), True
Solution Path for Grouped-LASSO QRSolution Path for Grouped LASSO QR
. ,....,max :Penalty1
1
G
gggg p
s 1g
(-3,2,-2))(0,0,0,0),((2,3,2), True
Applications of Simplex Techniques
• Resampling
Applications of Simplex Techniques
Cross-validation, Bootstrap
• Manipulating Simplex Algorithm for Resamplingp g p g p g1. Check whether an observation is active for a fitted model.2. Drive-out some observations by changing the objective function.
Key IdeasKey Ideas
Simplex Tableau can be used to:
• update an optimal partial model to another optimal partial model or full model.
dd t t i t d l • add extra constraints on a model.
• update an optimal model on a subset of a dataset to the optimal model on another subset of the datasetsubset of the dataset.
Computational GoalsComputational Goals
Hi h f i• High-performance computing
• Massive data processingp g
• Re-usable programs
Parallel Computing
• Parallel Computation can expedite Tableau Simplex algorithm on
Parallel Computing
1. Building initial tableau
2. Sorting/Ordering positive tableau rows
3. Changing the signs of tableau rows
4. Pivot Updating
Reference• Chen, C. and Wei Y. (2005), Computational Issues for Quantile Regression, The Indian Journal of Statistics, (67), pp.399-417.
Reference
• Koenker, R. (2005), Quantile Regression, Cambridge University Press.
• Koenker, R. and Machado, J.A.F. (1999), Goodness of fit and related inference processes for quantile regression, Journal of the American Statistician Association, (94), pp.1296-1310.
• Li, Y. and Zhu, J. (2008), L1-norm quantile regression, Journal of Computational & Graphical Statistics, (17), pp.163-185.Statistics, (17), pp.163 185.
• Sawa, T. (1978), Information criteria for discriminating among alternative regression models, Econometrica, (46), pp.1273–1282.
• Schwarz, G. (1978), Estimating the dimension of a model, Annals of Statistics, (6), pp.461–464.
• Yao, Y. and Lee, Y. (2007), Another look at linear programming for feature selection via methods of regularization. Technical Report No. 800, Department of Statistics, The Ohio State University.