trust region method
TRANSCRIPT
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 1/18
Trust Region MethodsPart I
Andrew R. [email protected]
Mathematical SciencesIBM T.J. Watson Research Center
February 2007, Montreal
1
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 2/18
Overview
1 Trust-Region/Modelling Methods
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 3/18
Trust-Region/Modelling Methods
What is a trust-region method?
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Problem: Minimize
−10x 21 + 10x 22 + 4 sinx 1x 2
− 2x 1 + x
41
3
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 4/18
Trust-Region/Modelling Methods
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 0
T R i /M d lli M h d
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 5/18
Trust-Region/Modelling Methods
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 1
T t R i /M d lli M th d
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 6/18
Trust-Region/Modelling Methods
What is a trust-region method?
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 2
6
Trust Region/Modelling Methods
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 7/18
Trust-Region/Modelling Methods
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 3 = x 2
Trust Region/Modelling Methods
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 8/18
Trust-Region/Modelling Methods
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 4
8
Trust-Region/Modelling Methods
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 9/18
Trust-Region/Modelling Methods
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 5
9
Trust-Region/Modelling Methods
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 10/18
Trust Region/Modelling Methods
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 6 (x 7 = x ∗)
10
Trust-Region/Modelling Methods
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 11/18
st g o / o g t o s
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Six iterations —different initial point
11
Trust-Region/Modelling Methods
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 12/18
g / g
Basic Algorithm
Initialize: x 0, ∆
Compute Model: mk ( )
Compute Step: Compute s k from
mins ≤∆
mk (x k + s )
Trust-region Update: ρ = f (x k )−f (x k +s k )mk (x k )−mk (x k +s k )
If ρ > 0.75 ∆ ← 2.0∆
If 0.25 < ρ < 0.75 ∆ ← ∆
If ρ < 0.25 ∆ ← 0.5∆
Accept x k + s k
Accept x k + s k
Reject x k + s k
12
Trust-Region/Modelling Methods
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 13/18
/
Assumptions to prove convergence
On the problem
Smooth f ∈ C
2
Bounded Below f bounded below
Bounded Hessian xx f bounded above
13
Trust-Region/Modelling Methods
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 14/18
Assumptions to prove convergence
On the model (∀k )
Smooth mk ∈ C 2
Interpolatesm
k (x k ) =
f (x k )
Interpolates Gradient x mk (x k ) = x f (x k )
Bounded Hessianmaxx ∈Bk
xx mk (x ) bounded above where,Bk = {x ∈ n | x − x k k ≤ ∆k }
13
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 15/18
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 16/18
Trust-Region/Modelling Methods
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 17/18
Standard Model in the differentiable case
Typical trust region or line search method builds linear or
quadratic model of the objective function f .The model has to satisfy Taylor-like error bounds.Second Order
|f (x ) − m(x )| ≤ O(∆3)
|f
(x
) − m
(x
)| ≤ O(∆2
)|2
f (x ) − 2m(x )| ≤ O(∆)
In fact it typically is a first (or second) order Taylor series
approximation.In derivative based methods constants in O depend only on f
(and its derivatives).
By reducing the trust region or step size one guarantees better
accuracy.14
Trust-Region/Modelling Methods
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 18/18
Standard Model in the differentiable case
Typical trust region or line search method builds linear or
quadratic model of the objective function f .The model has to satisfy Taylor-like error bounds.Second Order
|f (x ) − m(x )| ≤ O(∆3)
|f
(x
) − m
(x
)| ≤ O(∆2
)|2
f (x ) − 2m(x )| ≤ O(∆)
In fact it typically is a first (or second) order Taylor series
approximation.In derivative based methods constants in O depend only on f
(and its derivatives).
By reducing the trust region or step size one guarantees better
accuracy.14