Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Module 04 — Optimization ProblemsKKT Conditions & Solvers
Ahmad F. Taha
EE 5243: Introduction to Cyber-Physical Systems
Email: [email protected]
Webpage: http://engineering.utsa.edu/˜taha/index.html
September 14, 2015
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 1 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Module 04 Outline
In this module, we present basic mathematical optimization principles. Thefollowing topics are covered1:
General introduction to optimization
Convex optimization
Linear programming, SDP
Mixed-integer programming
Relaxations
KKT optimality conditions
Optimization problems solvers
1Much of the material presented in this module can be found in [Taylor, 2015; Boyd &Vandenberghe, 2004]
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 2 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Optimization — 1
Given a function to be minimized, f(x), x ∈ Rn
x0 is a global minimum of f(x)⇒ f(x0) ≤ f(x) for all x
x0 is a local minimum of f(x)⇒ f(x0) ≤ f(x) for{x ∈ Rn; ‖x− x0‖ ≤ ε, ε > 0}
Convexity:Function — f(x) is convex if:
f(αx+ (1− α)y) ≤ αf(x) + (1− α)f(y)for all 0 ≤ α ≤ 1
Set — X is convex if x, y ∈ X ⇒ αx+ (1− α)y ∈ X
If g(x) is convex, then X = {x | g(x) ≤ 0} is convex
Convex Optimization Problem
minx∈Rn
f(x)
s.t. gi(x) ≤ 0, i = 1, ...,m
f(x), gi(x) are all convex ⇒ any local minimum is a global minimum
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 3 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Examples of Convex Functions
Functions mapping R→ R:Affine (ax+ b), exponential (eax)
Powers (xa; a ≥ 1, a ≤ 0), powers of absolute value (|x|p; p ≥ 1)Functions mapping Rn → R:
Affine (a>x+ b)
Vector norms (‖x‖p =(∑n
i=1 |x|p) 1
p , p ≥ 1 )Functions mapping Rn×m → R:
Affine (f(X) = trace(A>X) + b)
Matrix norms (‖X‖2 = σmax(X) =√λmax(X>X))
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 4 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Tractability & Linear Programs
Computational tractability:Convex optimization: easy to solve, polynomial-time
Nonconvex optimization: difficult to solve, NP-hard2
Linear programming:f(x) = cTx
Affine constraints: gi(x) = aTi x− bi (usually as vector: Ax ≤ b)
Easiest type of optimization
Solvable in polynomial time
Quadratic programming with f(x) = xTQx is also easy if Q is symmetricpositive semi-definite
Is it convex? See next slide
2NP-hard: no polynomial-time (efficient) algorithm can exist©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 5 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Convexity of a Quadratic Function
Solution:Given that f(x) = x>Qx, we apply the definition of convex function.
αf(x) + (1− α)f(y)− f(αx+ (1− α)y) ≥ 0.
Substituting for f(x) into the LHS of the previous equation yields:
αx>Qx+ (1− α)y>Qy − (αx+ (1− α)y)>Q(αx+ (1− α)y)
= α(1−α)x>Qx−2α(1−α)x>Qy+α(1−α)y>Qy = α(1−α)(x−y)>Q(x−y)
Define z = x− y ⇒α(1− α)z>Qz
Since 0 ≤ α ≤ 1, Q = Q> � and ∀ z ⇒
α(1− α)z>Qz ≥ 0
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 6 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Mixed-Integer Programming
MIP
minx,y
f(x, y)
s.t. gi(x, y) ≤ 0, i = 1, ...,myi ∈ Z (the integers)
NP-hard even when f and gi are all linear
yi ∈ Z is nononvex
Very common in CPS planning problems
Even more relevant in smart grids: unit commitment problem, expansionmodels, . . .
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 7 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Semidefinite Programming
Hermitian: X = X∗ (conjugate transpose), X ∈ Cn×n
Definition: z∗Xz ≥ 0 for all z ∈ Rn
Equivalent: all eigs. of X nonnegative, all principal minors nonnegative
Notation: X � 0
X � 0 is convex constraint
Proof : Suppose X,Y � 0. Then
z∗(αX + (1− α)Y )z = αz∗Xz + (1− α)z∗Y z ≥ 0
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 8 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Semidefinite Programming — 2
Semidefinite Program (SDP)
minX
trace(CX)
s.t. trace(AiX) = bi , X � 0
Semidefinite Program (SDP) — Form 2
minx
c>x
s.t. F (x) = F0 +n∑i=1
Fixi � 0︸ ︷︷ ︸Linear Matrix Inequality (LMI), Fi=F>
i
SDP: linear cost function, LMI constraints — Convex, 1 minimum
Generalization of LP (don’t solve LP as SDP)
SDP’s can be solved in polynomial-time using interior point methods©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 9 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
LMIs
A system of LMIs F1(x), F2(x), . . . ,� 0 can be represented as a singleLMI:
F (x) =
F1(x)
F2(x). . .
Fm(x)
� 0
For an Rm×n matrix A, the inequality Ax ≤ b can be represented as mLMIs:
bi − a>i x ≥ 0, i = 1, 2, . . . ,m
Most optimization solvers cannot handle “ � ”⇒ replace it with “ � ”
Example: Lyapunov’s A>P + PA ≺ 0 is an LMI
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 10 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
LMI Example
Lyapunov’s TheoremReal parts of eig(A) are negative iff there exists a real symmetric positivedefinite matrix P = P> � 0 such that:
A>P + PA ≺ 0.
Can we write Lyapunov’s inequality as an LMI?
Define: P =
x1 x2 . . . xnx2 xn+1...xn x2n−1 . . . xm
, m = (n+1)n2 : # of Variables
P1 =
1 0 . . . 00 0...0 0 . . . 0
, P2 =
0 1 . . . 01 0...0 0 . . . 0
, . . . , Pm =
0 0 . . . 00 0...0 0 . . . 1
⇒ A>P + PA =
∑m
i=1 xi(A>Pi + PiA) = −x1F1 − x2F2 − . . . xmFm ≺ 0
is an lmi©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 11 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Example — Convex Quadratic Functions
Is the quadratic function
f(x) = x>
[1 2 20 1 20 0 0
]x
indefinite, positive definite, negative definite, positive semidefinite, or negativesemidefinite?
1 Start with finding the leading principal minors? NO!
2 Need to symmetrize f(x):
f(x) = 12x>(Q+Q>)x = 1
2x>
[2 2 22 2 22 2 0
]x
3 From the principal minors, we conclude that the quadratic form isindefinite
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 12 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Most Popular LMIs
LMIP — find a feasible x such that F (x) � 0
– Example: Lyapunov theorem
EVP/PDP — eigenvalue problem (EVP) is to minimize the maximumeigenvalue of a matrix A(x) that depends affinely on a variable subject toan LMI constraint
– Example: Finding the best H∞ robust controller: stabilization + goodperformance
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 13 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Example — Eigenvalue Optimization
Suppose A(x) ∈ Cn×n is a linear function of xObjective: minimize the maximum eigenvalue of A(x):
minx,λ
λ
s.t. λ is the largest eig. of A(x)
Eigenvalue:
A(x)v = λv ⇒ v∗A(x)v = λv∗v ⇒ v∗A(x)vv∗v
= λ
⇒ maxv∈Cn
v∗A(x)vv∗v
= λmax
⇒ λmaxv∗Iv ≥ v∗A(x)v ∀v
Hence, optimization problem can be equivalently written as:
minλ
λ minλ
λ
s.t. v∗(λI −A(x))v ≥ 0 ∀v s.t. λI −A(x) � 0
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 14 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Quadratic Optimization Problems
Quadratic Constrained Problem (QCP)
minx
x∗Cx
s.t. x∗Aix ≤ bi
Solvability:– If C � 0 and Ai � 0, solvable in polynomial time
– If any are not positive semi-definite (PSD), problem becomes NP-hardApplications:
– Binary constraints: x ∈ {0, 1} ⇔ x2 = x
– AC power flow in power systems
– Both examples are nonconvex
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 15 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Relaxations
Hard problem (exact):P1 : min
x∈Xf(x)
Easier, relaxed problem:P2 : min
x∈Yf(x), X ⊂ Y
Facts:– (Obj. of P2) ≤ (Obj. of P1)
– if x is optimal for relaxation and feasible for exact, x isoptimal for exact
* Proof: Suppose x is relaxed optimal and feasible suboptimal for exactproblem ⇒ ∃ y s.t. f(y) < f(x), y ∈ X. But by relaxation, y ∈ Y , andtherefore x is not relaxed optimal — a contradiction
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 16 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
SDP Relaxations
SDP can be written:
minx
trace(xx∗C)
s.t. trace(xx∗Ai) ≤ biX = xx∗ equivalent to: X � 0, rank(X) = 1
minX
trace(XC)
s.t. trace(XAi) ≤ biX � 0rank(X) = 1
Removing a constraint enlarges the feasible set, i.e. relaxation:
minX
trace(XC)
s.t. trace(XAi) ≤ biX � 0
If solution X∗ has rank 1, then relaxation is tight
Feasible, optimal exact solution is Cholesky: X = xx∗
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 17 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Linear Relaxation
Consider the following optimizaton problem:
minx
x1(x2 − 1)
s.t. x1 ≥ 1x2 ≥ 2x1x2 ≤ 3
Clearly, this problem is not convex (objective & a constraint)Relaxation: let y = x1x2, OP becomes:
minx,y
y − x1
s.t. x1 ≥ 1x2 ≥ 2y ≤ 3
y − 2x1 − x2 + 2︸ ︷︷ ︸=(x1−1)(x2−2)
≥ 0
Last constraint guarantees that y 6= −∞©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 18 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Solving Unconstrained OPs
Objective:minimizex∈Rn
f(x)
Necessary & Sufficinet Conditions for Optimalityx∗ is a local minimum of f(x) iff:
1 Zero gradient at x∗:∇xf(x∗) = 0
2 Hessian at x∗ is positive semi-definite:
∇2xf(x∗) � 0
For maximization, Hessian is negative semi-definite
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 19 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Solving Constrained OPs
Main objective: find/compute minimum or a maximum of an objectivefunction subject to equality and inequality constraints
Formally, problem defined as finding the optimal x∗:
minx
f(x)
subject to g(x) ≤ 0h(x) = 0
– x ∈ Rn
– f(x) is scalar function, possibly nonlinear
– g(x) ∈ Rm, h(x) ∈ Rl are vectors of constraints
Main PrincipleTo solve constrained optimization problems: transform constrainedproblems to unconstrained ones.
How?Augment the constraints to the cost function.
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 20 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
KKT Conditions
minx
f(x)
subject to g(x) ≤ 0h(x) = 0
Define the Lagrangian: L(x, λ, µ) = f(x) + λTh(x) + µT g(x)
Optimality ConditionsThe constrained optimization problem (above) has a local minimizer x∗ iffthere exists a unique µ∗ such that:
1 ∇xL(x∗, λ∗, µ∗) = ∇xf(x) + λ∗T∇xh(x∗) + µ∗T∇xg(x∗) = 0
2 µ∗j ≥ 0 for j = 1, . . . ,m
3 µ∗jgj(x∗) = 0 for j = 1, . . . ,m
4 gj(x∗) ≤ 0 for j = 1, . . . ,m
5 hi(x∗) = 0 for i = 1, . . . , l (if x∗, µ∗, λ∗ satisfy 1–5, they are candidates)
6 Second order necessary conditions (SONC): ∇2xL(x∗, λ∗, µ∗) � 0
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 21 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
KKT Conditions — Example3
Find the minimizer of the following optimization problem:
minimizex
f(x) = (x1 − 1)2 + x2 − 2
subject to g(x) = x1 + x2 − 2 ≤ 0h(x) = x2 − x1 − 1 = 0
First, find the Lagrangian function:
L(x, λ, µ) = (x1 − 1)2 + x2 − 2 + λ(x2 − x1 − 1) + µ(x1 + x2 − 2)
Second, find the conditions of optimality (from previous slide):1 ∇xL(x∗, λ∗, µ∗) =
[2x∗1 − 2− λ∗ + µ∗ 1 + λ∗ + µ∗
]>=[0 0
]>2 µ∗(x∗1 + x∗2 − 2) = 03 µ∗ ≥ 04 x∗1 + x∗2 − 2 ≤ 05 x∗2 − x
∗1 − 1 = 0
6 ∇2xL(x∗, λ∗, µ∗) = ∇2
xf(x∗) + λ∗∇2xh(x∗) + µ∗∇2
xg(x∗) � 0
=[
2 00 0
]+ λ∗
[0 00 0
]+ µ∗
[0 00 0
]� 0
3Example from [Chong & Zak, 2011]©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 22 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Example — Cont’d
To solve the system equations for the optimal x∗, λ∗, µ∗, we first tryµ∗ > 0.Given that, we solve the following set of equations:
1 2x∗1 − 2− λ∗ + µ∗ = 02 1 + λ∗ + µ∗ = 03 x∗1 + x∗2 − 2 = 04 x∗2 − x
∗1 − 1 = 0
⇒ x∗1 = 0.5, x∗2 = 1.5, λ∗ = −1, µ∗ = 0
But this solution contradicts the assumption that µ∗ > 0
Alternative: assume µ∗ = 0⇒ x∗1 = 0.5, x∗2 = 1.5, λ∗ = −1, µ∗ = 0
This solution satisfies g(x∗) ≤ 0 constraint, hence it’s a candidate forbeing a minimizer
We now verify the SONC: L(x∗, λ∗, µ∗) =[
2 00 0
]� 0
Thus, x∗ =[0.5 1.5
]> is a strict local minimizer
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 23 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
OPs Taxonomy
Figure from:http://www.neos-guide.org/content/optimization-introduction
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 24 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Solvers
Solving optimization problems require few things1 Modeling the problem2 Translating the problem model (constraints and objectives) into a modeling
language (AMPL, GAMS, MATLAB, YALMIP, CVX)3 Choosing optimization algorithms solvers (Simplex, Interior-Point,
Brand & Bound, Cutting Planes,...)4 Specifying tolerance, exit flags, flexible constraints, bounds, ...
Convex optimization problems: use cvx (super easy to install and code)
MATLAB’s fmincon is always handy too (too much overhead, often failsto converge for nonlinear optimization problems)
Visit http://www.neos-server.org/neos/solvers/index.html
Check http://www.neos-guide.org/ to learn more
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 25 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Complexity
Clearly, complexity of an OP depends on the solver used
Example: most LMI solvers use interior-point methods
Complexity: primal-dual interior-point has a worst-case complexityO(m2.75L1.5)
– m : #ofV ariables, L : #ofConstraints
– Applies to a set of L Lyapunov inequalities
– Typical performance: O(m2.1L1.2)
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 26 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
Questions And Suggestions?
Thank You!Please visit
engineering.utsa.edu/˜tahaIFF you want to know more ,
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 27 / 28
Optimization Intro LPs & MIPs SDPs QCP & Relaxations Optimality Conditions Optimization Solvers References
References I
Boyd, S., & Vandenberghe, L. (2004). Convex Optimization. New York, NY, USA: Cambridge University Press.
Chong, E., & Zak, S. (2011). An Introduction to Optimization. Wiley Series in Discrete Mathematics and Optimization. Wiley.URL https://books.google.com/books?id=THlxFmlEy_AC
Taylor, J. (2015). Convex Optimization of Power Systems. Cambridge University Press.
©Ahmad F. Taha Module 04 — Optimization and KKT Conditions 28 / 28