section 10: quadratic - argonne national laboratoryanitescu/classes/2011/lectures/... · section...
TRANSCRIPT
![Page 1: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/1.jpg)
Section 10: Quadratic Programming Reference: Chapter 16, Nocedal and Wright.
![Page 2: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/2.jpg)
10.1 GRADIENT PROJECTIONS FOR QPS WITH BOUND CONSTRAINTS
![Page 3: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/3.jpg)
Projection
�• The problem: �• Like in the trust-region case, we look for a Cauchy
point, based on a projection on the feasible set. �• G does not have to be psd (essential for AugLag) �• The projection operator:
![Page 4: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/4.jpg)
The search path
�• Create a piecewise linear path which is feasible (as opposed to the linear one in the unconstrained case) by projection of gradient.
![Page 5: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/5.jpg)
Computation of breakpoints
�• Can be done on each component individually
�• Then the search path becomes on each component:
![Page 6: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/6.jpg)
Line Search along piecewise linear path
�• Reorder the breakpoints eliminating duplicates and zero values to get
�• The path:
�• Whose direction is:
0 < t1 < t2 <�…
![Page 7: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/7.jpg)
Line Search (2)
�• Along each piece, find the minimum of the quadratic
�• This reduces to analyzing a one dimensional quadratic form of t on an interval.
�• If the minimum is on the right end of interval, we continue.
�• If not, we found the local minimum and the Cauchy point.
t j 1,t j
12xTGx + cT x
![Page 8: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/8.jpg)
Subspace Minimization �• Active set of Cauchy Point
�• Solve subspace minimization problem
�• No need to solve exactly. For example truncated CG with termination if one inactive variable reaches bound.
![Page 9: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/9.jpg)
Gradient Projection for QP
Or, equivalently, if projection does not advance from 0.
![Page 10: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/10.jpg)
Observations �– Gradient Projection
�• Note that the Projection �– Active set solve loop must be iterated to optimality.
�• What is the proper stopping criteria? How do we verify the KKT?
�• Idea: When projection does not progress ! That is, on each component, either the gradient is 0, or the breakpoint is 0.
![Page 11: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/11.jpg)
KKT conditions for Quadratic Programming with BC
![Page 12: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/12.jpg)
10.2 QUADRATIC PROGRAMMING WITH EQUALITY CONSTRAINTS
![Page 13: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/13.jpg)
Statement of Problem
�• Problem:
�• Optimality Conditions:
�• But, when is a solution of KKT a solution of the minimization problem?
![Page 14: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/14.jpg)
10.2.1 DIRECT (FACTORIZATION APPROACHES)
![Page 15: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/15.jpg)
Inertia of the KKT matrix
�• Separate the eigenvalues of a symmetric matrix by sign. Define:
�• Result: (K=kkt matrix)
�• But, how do I find inertia of K?, ideally while finding the solution of the KKT system?
![Page 16: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/16.jpg)
LDLT factorization
�• Formulation: �• Solving the KKT system with it:
�• Sylvester theorem: �• And B should have (n,m,0) inertia !!!
inertia(A)=inertia(CT DC)
![Page 17: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/17.jpg)
10.2.2 EXPLICIT USE OF FEASIBLE SET
![Page 18: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/18.jpg)
Reduction Strategies for Linear Constraints
�• Idea: Use a special for of the Implicit Function Theorem
�• In turn, this implies that and thus AY is full rank and invertible.
�• We can thus parametrize the feasible set along the components in the range of Y and Z.
![Page 19: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/19.jpg)
Using the Y,Z parameterization
�• This allows easy identification of the Y component of the feasible set:
�• The reduced optimization problem.
![Page 20: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/20.jpg)
How do we obtain a �“good �“ YZ parameterization?
Ax = b AT = Q1n×m
Q2n×(n m )
Rm×m
0
�• Idea: use a version of the QR factorization
�• After which, define
ATm×m
= Q1n×m
Q2n×(n m )
Rm×m
0Y =Q1,Z =Q2;Q1
TQ2 = 0m×(n m )
AZ[ ]T =Q2T AT = Q1 Q2 R
m×m
0= 0(n m )×(m )
AY[ ]T =Q1T AT = R AY = R T T
= RT
![Page 21: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/21.jpg)
10.2.3 NULL SPACE METHOD
![Page 22: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/22.jpg)
Affine decomposition �• Ansatz (AZ=0):
�• Consequence:
�• We can work out the Normal component as well:
pY = AY( ) 1 h
![Page 23: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/23.jpg)
Normal component
�• Multiply with Z transpose the equation:
�• Use Cholesky, get p_z and then (if not: second-order sufficient does not hold) �• Multiply first equation by Y^T to obtain
�• If I use QR, both are backsolves (O(n^2))!!!
![Page 24: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/24.jpg)
10.2.4 ITERATIVE METHODS: CG APPLIED TO REDUCED SYSTEM
![Page 25: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/25.jpg)
Reduced problem
�• Use affine decomposition:
�• Reduced problem
�• Associated linear system:
AYxY = b xY = AY( ) 1b
![Page 26: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/26.jpg)
Preconditioned Reduced CG
![Page 27: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/27.jpg)
Preconditioner
�• How do we do it? �• Idea:
Wzz ZTGZ( ) I; Wzz = Z
THZ;Wzz 0
![Page 28: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/28.jpg)
10.2.5 ITERATIVE METHODS: PROJECTED CG
![Page 29: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/29.jpg)
Rephrased reduced CG
�• Use the projection matrix:
�• The algorithm is equivalent to:
![Page 30: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/30.jpg)
10.3 INTERIOR-POINT METHODS FOR CONVEX QUADRATIC PROGRAM
![Page 31: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/31.jpg)
Setup
�• The form of the problem solved:
![Page 32: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/32.jpg)
Optimality Conditions
�• In original form:
�• With slacks:
![Page 33: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/33.jpg)
Idea: define an �“interior�” path to the solution
�• Define the perturbed KKT conditions as a nonlinear system:
�• Solve successively while taking mu to 0 solves KKT: F x µ( ), y µ( ), µ( );µ,( ) = 0; yi µ( ) i µ( ) = µ > 0
µ 0 x µ( ), y µ( ), µ( )( ) x*, y*, *( ) satisfies KKT
![Page 34: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/34.jpg)
How to solve it ?
�• Solution: apply Newton�’s method for fixed mu:
�• New iterate:
�• Enforce:
![Page 35: Section 10: Quadratic - Argonne National Laboratoryanitescu/CLASSES/2011/LECTURES/... · Section 10: Quadratic Reference: Chapter 16 ... 3 Note that the Projection Active set solve](https://reader033.vdocuments.mx/reader033/viewer/2022051723/5aae58e37f8b9aa8438bf646/html5/thumbnails/35.jpg)
How to PRACTICALLY solve it?
�• Eliminate the slacks:
�• Eliminate the multipliers (note, the 22 block is diagonal and invertible)
�• Solve (e.g by Cholesky if PD and modified Cholesky if not ).