stephen boyd steven diamond junzi zhang akshay agrawal ee &...
TRANSCRIPT
![Page 1: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/1.jpg)
Convex Optimization Applications
Stephen Boyd Steven DiamondJunzi Zhang Akshay Agrawal
EE & CS DepartmentsStanford University
1
![Page 2: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/2.jpg)
Outline
Portfolio Optimization
Worst-Case Risk Analysis
Optimal Advertising
Regression Variations
Model Fitting
2
![Page 3: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/3.jpg)
Outline
Portfolio Optimization
Worst-Case Risk Analysis
Optimal Advertising
Regression Variations
Model Fitting
Portfolio Optimization 3
![Page 4: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/4.jpg)
Portfolio allocation vector
I invest fraction wi in asset i , i = 1, . . . , nI w ∈ Rn is portfolio allocation vectorI 1T w = 1I wi < 0 means a short position in asset i
(borrow shares and sell now; must replace later)I w ≥ 0 is a long only portfolioI ‖w‖1 = 1T w+ + 1T w− is leverage
(many other definitions used . . . )
Portfolio Optimization 4
![Page 5: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/5.jpg)
Asset returns
I investments held for one periodI initial prices pi > 0; end of period prices p+
i > 0I asset (fractional) returns ri = (p+
i − pi )/piI portfolio (fractional) return R = rT wI common model: r is a random variable, with mean E r = µ,
covariance E(r − µ)(r − µ)T = ΣI so R is a RV with E R = µT w , var(R) = wT ΣwI E R is (mean) return of portfolioI var(R) is risk of portfolio
(risk also sometimes given as std(R) =√
var(R))
I two objectives: high return, low risk
Portfolio Optimization 5
![Page 6: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/6.jpg)
Classical (Markowitz) portfolio optimization
maximize µT w − γwT Σwsubject to 1T w = 1, w ∈ W
I variable w ∈ Rn
I W is set of allowed portfoliosI common case: W = Rn
+ (long only portfolio)I γ > 0 is the risk aversion parameterI µT w − γwT Σw is risk-adjusted returnI varying γ gives optimal risk-return trade-offI can also fix return and minimize risk, etc.
Portfolio Optimization 6
![Page 7: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/7.jpg)
Exampleoptimal risk-return trade-off for 10 assets, long only portfolio
Portfolio Optimization 7
![Page 8: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/8.jpg)
Examplereturn distributions for two risk aversion values
Portfolio Optimization 8
![Page 9: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/9.jpg)
Portfolio constraints
I W = Rn (simple analytical solution)I leverage limit: ‖w‖1 ≤ Lmax
I market neutral: mT Σw = 0I mi is capitalization of asset iI M = mT r is market returnI mT Σw = cov(M,R)
i.e., market neutral portfolio return is uncorrelated withmarket return
Portfolio Optimization 9
![Page 10: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/10.jpg)
Exampleoptimal risk-return trade-off curves for leverage limits 1, 2, 4
Portfolio Optimization 10
![Page 11: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/11.jpg)
Examplethree portfolios with wT Σw = 2, leverage limits L = 1, 2, 4
Portfolio Optimization 11
![Page 12: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/12.jpg)
Variations
I require µT w ≥ Rmin, minimize wT Σw or ‖Σ1/2w‖2I include (broker) cost of short positions,
sT (w)−, s ≥ 0
I include transaction cost (from previous portfolio wprev),
κT |w − wprev|η, κ ≥ 0
common models: η = 1, 3/2, 2
Portfolio Optimization 12
![Page 13: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/13.jpg)
Factor covariance model
Σ = F ΣF T + D
I F ∈ Rn×k , k � n is factor loading matrixI k is number of factors (or sectors), typically 10sI Fij is loading of asset i to factor jI D is diagonal matrix; Dii > 0 is idiosyncratic riskI Σ > 0 is the factor covariance matrix
I F T w ∈ Rk gives portfolio factor exposuresI portfolio is factor j neutral if (F T w)j = 0
Portfolio Optimization 13
![Page 14: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/14.jpg)
Portfolio optimization with factor covariance model
maximize µT w − γ(
f T Σf + wT Dw)
subject to 1T w = 1, f = F T ww ∈ W, f ∈ F
I variables w ∈ Rn (allocations), f ∈ Rk (factor exposures)I F gives factor exposure constraints
I computational advantage: O(nk2) vs. O(n3)
Portfolio Optimization 14
![Page 15: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/15.jpg)
Example
I 50 factors, 3000 assetsI leverage limit = 2I solve with covariance given as
I single matrixI factor model
I CVXPY/OSQP single thread time
covariance solve timesingle matrix 173.30 secfactor model 0.85 sec
Portfolio Optimization 15
![Page 16: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/16.jpg)
Outline
Portfolio Optimization
Worst-Case Risk Analysis
Optimal Advertising
Regression Variations
Model Fitting
Worst-Case Risk Analysis 16
![Page 17: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/17.jpg)
Covariance uncertainty
I single period Markowitz portfolio allocation problemI we have fixed portfolio allocation w ∈ Rn
I return covariance Σ not known, but we believe Σ ∈ SI S is convex set of possible covariance matricesI risk is wT Σw , a linear function of Σ
Worst-Case Risk Analysis 17
![Page 18: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/18.jpg)
Worst-case risk analysis
I what is the worst (maximum) risk, over all possiblecovariance matrices?
I worst-case risk analysis problem:
maximize wT Σwsubject to Σ ∈ S, Σ � 0
with variable ΣI . . . a convex problem with variable Σ
I if the worst-case risk is not too bad, you can worry lessI if not, you’ll confront your worst nightmare
Worst-Case Risk Analysis 18
![Page 19: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/19.jpg)
Example
I w = (−0.01, 0.13, 0.18, 0.88,−0.18)I optimized for Σnom, return 0.1, leverage limit 2I S = {Σnom + ∆ : |∆ii | = 0, |∆ij | ≤ 0.2},
Σnom =
0.58 0.2 0.57 −0.02 0.430.2 0.36 0.24 0 0.38
0.57 0.24 0.57 −0.01 0.47−0.02 0 −0.01 0.05 0.080.43 0.38 0.47 0.08 0.92
Worst-Case Risk Analysis 19
![Page 20: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/20.jpg)
Example
I nominal risk = 0.168I worst case risk = 0.422
worst case ∆ =
0 0.04 −0.2 −0. 0.2
0.04 0 0.2 0.09 −0.2−0.2 0.2 0 0.12 −0.2−0. 0.09 0.12 0 −0.180.2 −0.2 −0.2 −0.18 0
Worst-Case Risk Analysis 20
![Page 21: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/21.jpg)
Outline
Portfolio Optimization
Worst-Case Risk Analysis
Optimal Advertising
Regression Variations
Model Fitting
Optimal Advertising 21
![Page 22: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/22.jpg)
Ad display
I m advertisers/ads, i = 1, . . . ,mI n time slots, t = 1, . . . , nI Tt is total traffic in time slot tI Dit ≥ 0 is number of ad i displayed in period tI∑
i Dit ≤ TtI contracted minimum total displays:
∑t Dit ≥ ci
I goal: choose Dit
Optimal Advertising 22
![Page 23: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/23.jpg)
Clicks and revenue
I Cit is number of clicks on ad i in period tI click model: Cit = PitDit , Pit ∈ [0, 1]I payment: Ri > 0 per click for ad i , up to budget BiI ad revenue
Si = min{
Ri∑
tCit ,Bi
}. . . a concave function of D
Optimal Advertising 23
![Page 24: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/24.jpg)
Ad optimization
I choose displays to maximize revenue:
maximize∑
i Sisubject to D ≥ 0, DT 1 ≤ T , D1 ≥ c
I variable is D ∈ Rm×n
I data are T , c, R, B, P
Optimal Advertising 24
![Page 25: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/25.jpg)
ExampleI 24 hourly periods, 5 ads (A–E)I total traffic:
Optimal Advertising 25
![Page 26: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/26.jpg)
Example
I ad data:Ad A B C D Eci 61000 80000 61000 23000 64000Ri 0.15 1.18 0.57 2.08 2.43Bi 25000 12000 12000 11000 17000
Optimal Advertising 26
![Page 27: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/27.jpg)
ExamplePit
Optimal Advertising 27
![Page 28: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/28.jpg)
Exampleoptimal Dit
Optimal Advertising 28
![Page 29: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/29.jpg)
Example
ad revenue
Ad A B C D Eci 61000 80000 61000 23000 64000Ri 0.15 1.18 0.57 2.08 2.43Bi 25000 12000 12000 11000 17000∑
t Dit 61000 80000 148116 23000 167323Si 182 12000 12000 11000 7760
Optimal Advertising 29
![Page 30: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/30.jpg)
Outline
Portfolio Optimization
Worst-Case Risk Analysis
Optimal Advertising
Regression Variations
Model Fitting
Regression Variations 30
![Page 31: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/31.jpg)
Standard regression
I given data (xi , yi ) ∈ Rn × R, i = 1, . . . ,mI fit linear (affine) model yi = βT xi − v , β ∈ Rn, v ∈ RI residuals are ri = yi − yiI least-squares: choose β, v to minimize ‖r‖22 =
∑i r2
iI mean of optimal residuals is zeroI can add (Tychonov) regularization: with λ > 0,
minimize ‖r‖22 + λ‖β‖22
Regression Variations 31
![Page 32: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/32.jpg)
Robust (Huber) regression
I replace square with Huber function
φ(u) ={
u2 |u| ≤ M2Mu −M2 |u| > M
M > 0 is the Huber threshold
I same as least-squares for small residuals, but allows (some)large residuals
Regression Variations 32
![Page 33: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/33.jpg)
Example
I m = 450 measurements, n = 300 regressorsI choose βtrue; xi ∼ N (0, I)I set yi = (βtrue)T xi + εi , εi ∼ N (0, 1)I with probability p, replace yi with −yiI data has fraction p of (non-obvious) wrong measurementsI distribution of ‘good’ and ‘bad’ yi are the sameI try to recover βtrue ∈ Rn from measurements y ∈ Rm
I ‘prescient’ version: we know which measurements are wrong
Regression Variations 33
![Page 34: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/34.jpg)
Example50 problem instances, p varying from 0 to 0.15
Regression Variations 34
![Page 35: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/35.jpg)
Example
Regression Variations 35
![Page 36: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/36.jpg)
Quantile regression
I tilted `1 penalty: for τ ∈ (0, 1),
φ(u) = τ(u)+ + (1− τ)(u)− = (1/2)|u|+ (τ − 1/2)u
I quantile regression: choose β, v to minimize∑
i φ(ri )
I τ = 0.5: equal penalty for over- and under-estimatingI τ = 0.1: 9× more penalty for under-estimatingI τ = 0.9: 9× more penalty for over-estimating
Regression Variations 36
![Page 37: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/37.jpg)
Quantile regression
I for ri 6= 0,
∂∑
i φ(ri )∂v = τ |{i : ri > 0}| − (1− τ) |{i : ri < 0}|
I (roughly speaking) for optimal v we have
τ |{i : ri > 0}| = (1− τ) |{i : ri < 0}|
I and so for optimal v , τm = |{i : ri < 0}|I τ -quantile of optimal residuals is zeroI hence the name quantile regression
Regression Variations 37
![Page 38: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/38.jpg)
Example
I time series xt , t = 0, 1, 2, . . .I auto-regressive predictor:
xt+1 = βT (xt , . . . , xt−M)− v
I M = 10 is memory of predictorI use quantile regression for τ = 0.1, 0.5, 0.9I at each time t, gives three one-step-ahead predictions:
x0.1t+1, x0.5
t+1, x0.9t+1
Regression Variations 38
![Page 39: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/39.jpg)
Exampletime series xt
Regression Variations 39
![Page 40: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/40.jpg)
Examplext and predictions x0.1
t+1, x0.5t+1, x0.9
t+1 (training set, t = 0, . . . , 399)
Regression Variations 40
![Page 41: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/41.jpg)
Examplext and predictions x0.1
t+1, x0.5t+1, x0.9
t+1 (test set, t = 400, . . . , 449)
Regression Variations 41
![Page 42: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/42.jpg)
Exampleresidual distributions for τ = 0.9, 0.5, and 0.1 (training set)
Regression Variations 42
![Page 43: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/43.jpg)
Exampleresidual distributions for τ = 0.9, 0.5, and 0.1 (test set)
Regression Variations 43
![Page 44: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/44.jpg)
Outline
Portfolio Optimization
Worst-Case Risk Analysis
Optimal Advertising
Regression Variations
Model Fitting
Model Fitting 44
![Page 45: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/45.jpg)
Data model
I given data (xi , yi ) ∈ X × Y, i = 1, . . . ,mI for X = Rn, x is feature vectorI for Y = R, y is (real) outcome or labelI for Y = {−1, 1}, y is (boolean) outcome
I find model or predictor ψ : X → Y so that ψ(x) ≈ yfor data (x , y) that you haven’t seen
I for Y = R, ψ is a regression modelI for Y = {−1, 1}, ψ is a classifierI we choose ψ based on observed data, prior knowledge
Model Fitting 45
![Page 46: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/46.jpg)
Loss minimization model
I data model parametrized by θ ∈ Rn
I loss function L : X × Y × Rn → RI L(xi , yi , θ) is loss (miss-fit) for data point (xi , yi ),
using model parameter θI choose θ; then model is
ψ(x) = argminy
L(x , y , θ)
Model Fitting 46
![Page 47: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/47.jpg)
Model fitting via regularized loss minimization
I regularization r : Rn → R ∪ {∞}I r(θ) measures model complexity, enforces constraints, or
represents priorI choose θ by minimizing regularized loss
(1/m)∑
iL(xi , yi , θ) + r(θ)
I for many useful cases, this is a convex problemI model is ψ(x) = argminy L(x , y , θ)
Model Fitting 47
![Page 48: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/48.jpg)
Examples
model L(x , y , θ) ψ(x) r(θ)least-squares (θT x − y)2 θT x 0ridge regression (θT x − y)2 θT x λ‖θ‖22lasso (θT x − y)2 θT x λ‖θ‖1logistic classifier log(1 + exp(−yθT x)) sign(θT x) 0SVM (1− yθT x)+ sign(θT x) λ‖θ‖22
I λ > 0 scales regularizationI all lead to convex fitting problems
Model Fitting 48
![Page 49: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/49.jpg)
Example
I original (boolean) features z ∈ {0, 1}10
I (boolean) outcome y ∈ {−1, 1}I new feature vector x ∈ {0, 1}55 contains all products zi zj
(co-occurence of pairs of original features)I use logistic loss, `1 regularizerI training data has m = 200 examples; test on 100 examples
Model Fitting 49
![Page 50: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/50.jpg)
Example
Model Fitting 50
![Page 51: Stephen Boyd Steven Diamond Junzi Zhang Akshay Agrawal EE & …boyd/papers/pdf/cvx_applications.pdf · 2019-06-10 · Junzi Zhang Akshay Agrawal EE & CS Departments Stanford University](https://reader033.vdocuments.mx/reader033/viewer/2022050315/5f778b7f353a7b2d30101c79/html5/thumbnails/51.jpg)
Exampleselected features zi zj , λ = 0.01
Model Fitting 51