convex optimization problems (i)ย ยท optimization problem consider the problem not a convex...
TRANSCRIPT
Convex Optimization Problems (I)
Lijun [email protected]://cs.nju.edu.cn/zlj
Outline
Optimization Problems Basic Terminology Equivalent Problems Problem Descriptions
Convex Optimization Standard Form Local and Global Optima An Optimality Criterion Equivalent Convex Problems Quasiconvex Optimization
Outline
Optimization Problems Basic Terminology Equivalent Problems Problem Descriptions
Convex Optimization Standard Form Local and Global Optima An Optimality Criterion Equivalent Convex Problems Quasiconvex Optimization
Optimization Problems
Optimization variable: Objective function: Inequality constraints: ๐ ๐ฅ Inequality constraint functions: Equality constraints: โ ๐ฅ Equality constraint functions:
Unconstrainedwhen ๐ ๐ 0
Basic Terminology
Optimization Problems
Domain
is feasible if it satisfies all the constraints
The problem is feasible if there exists at least one feasible point
๐ dom ๐ โฉ dom โ
Basic Terminology
Optimal Value โ
Infeasible problem: โ
Unbounded below: if there exist with as , then โ
Optimal Points โ is feasible and โ โ
Optimal Set
โ is achieved if is nonempty
๐โ inf ๐ ๐ฅ |๐ ๐ฅ 0, ๐ 1, . . . , ๐, โ ๐ฅ 0, ๐ 1, . . . , ๐
๐ ๐ฅ|๐ ๐ฅ 0, ๐ 1, . . . , ๐, โ ๐ฅ 0, ๐ 1, . . . , ๐, ๐ ๐ฅ ๐โ
Basic Terminology
-suboptimal Points a feasible with โ
-suboptimal Set the set of all ๐-suboptimal points
Locally Optimal Points
๐ฅ is feasible and solves the above problem Globally Optimal Points
min ๐ ๐ง s. t. ๐ ๐ง 0, ๐ 1, . . . , ๐
โ ๐ง 0, ๐ 1, . . . , ๐ ๐ง ๐ฅ ๐
Basic Terminology
Basic Terminology
Types of Constraints If ๐ ๐ฅ 0, ๐ ๐ฅ 0 is active at ๐ฅ If ๐ ๐ฅ 0, ๐ ๐ฅ 0 is inactive at ๐ฅ โ ๐ฅ 0 is active at all feasible points Redundant constraint: deleting it does not
change the feasible set Examples on and ๐ ๐ฅ 1 ๐ฅโ : ๐โ 0, the optimal value is not
achieved ๐ ๐ฅ log ๐ฅ : ๐โ โ, unbounded blow ๐ ๐ฅ ๐ฅ log ๐ฅ : ๐โ 1 ๐โ , ๐ฅโ 1 ๐โ is optimal
Basic Terminology
Feasibility Problems
Determine whether constraints are consistent
Maximization Problems
It can be solved by minimizing ๐ Optimal Value ๐โ
find ๐ฅ s. t. ๐ ๐ฅ 0, ๐ 1, . . . , ๐
โ ๐ฅ 0, ๐ 1, . . . , ๐
max ๐ ๐ฅ s. t. ๐ ๐ฅ 0, ๐ 1, . . . , ๐
โ ๐ฅ 0, ๐ 1, . . . , ๐
๐โ sup ๐ ๐ฅ |๐ ๐ฅ 0, ๐ 1, . . . , ๐, โ ๐ฅ 0, ๐ 1, . . . , ๐
Basic Terminology
Standard Form
Box constraints
Reformulation
min ๐ ๐ฅ s. t. ๐ ๐ฅ ๐ข , ๐ 1, . . . , ๐
min ๐ ๐ฅ s. t. ๐ ๐ฅ 0, ๐ 1, . . . , ๐
๐ฅ ๐ข 0, ๐ 1, . . . , ๐
min ๐ ๐ฅ s. t. ๐ ๐ฅ 0, ๐ 1, . . . , ๐
โ ๐ฅ 0, ๐ 1, . . . , ๐
Outline
Optimization Problems Basic Terminology Equivalent Problems Problem Descriptions
Convex Optimization Standard Form Local and Global Optima An Optimality Criterion Equivalent Convex Problems Quasiconvex Optimization
Equivalent Problems Two Equivalent Problems If from a solution of one, a solution of the
other is readily found, and vice versa A Simple Example
๐ผ 0, ๐ 0, . . . , ๐ ๐ฝ 0, ๐ 1, . . . , ๐ Equivalent to the problem 1
Change of Variables
is one-to-one and dom , and define
An Equivalent Problem
If ๐ง solves it, ๐ฅ ๐ ๐ง solves the problem 1 If ๐ฅ solves 1 , ๐ง ๐ ๐ฅ solves it
min ๐ ๐ง s. t. ๐ ๐ง 0, ๐ 1, . . . , ๐
โ ๐ง 0, ๐ 1, . . . , ๐
๐ ๐ง ๐ ๐ ๐ง , ๐ 0, . . . , ๐โ ๐ง โ ๐ ๐ง , ๐ 1, . . . , ๐
Transformation of Functions ๐ : ๐ โ ๐ is monotone increasing ๐ , . . . , ๐ : ๐ โ ๐ satisfy ๐ ๐ข 0 if and only if
๐ข 0 ๐ , . . . , ๐ : ๐ โ ๐ satisfy ๐ ๐ข 0 if and only
if ๐ข 0 Define
An Equivalent Problem
๐ ๐ฅ ๐ ๐ ๐ฅ , ๐ 0, . . . , ๐โ ๐ฅ ๐ โ ๐ฅ , ๐ 1, . . . , ๐
min ๐ ๐ฅ s. t. ๐ ๐ฅ 0, ๐ 1, . . . , ๐
โ ๐ฅ 0, ๐ 1, . . . , ๐
Example
Least-norm Problems
Not differentiable at any with
Least-norm-squared Problems
Differentiable for all
Slack Variables if and only if there is an
that satisfies An Equivalent Problem
๐ is the slack variable associated with the inequality constraint ๐ ๐ฅ 0
๐ฅ is optimal for the problem 1 if and only if ๐ฅ, ๐ is optimal for the above problem, where
๐ ๐ ๐ฅ
min ๐ ๐ฅ s. t. ๐ 0, ๐ 1, . . . , ๐
๐ ๐ฅ ๐ 0, ๐ 1, . . . , ๐ โ ๐ฅ 0, ๐ 1, . . . , ๐
Eliminating Equality Constraints
Assume is such that satisfies
if and only if there is some such that
An Equivalent Problem
If is optimal for this problem, is optimal for the problem
If is optimal for , there is at least one which is optimal for this problem
โ ๐ฅ 0, ๐ 1, . . . , ๐
๐ฅ ๐ ๐ง
min ๐ ๐ง ๐ ๐ ๐ง s. t. ๐ ๐ง ๐ ๐ ๐ง 0, ๐ 1, . . . , ๐
Eliminating linear equality constraints
Assume the equality constraints are all linear , and is one solution
Let be any matrix with , then
An Equivalent Problem ( )
๐ ๐ rank ๐ด
min ๐ ๐น๐ง ๐ฅ s. t. ๐ ๐น๐ง ๐ฅ 0, ๐ 1, . . . , ๐
Linear algebra
Range and nullspace Let ๐ด โ ๐ , the range of ๐ด, denoted โ ๐ด ,
is the set of all vectors in ๐ that can be written as linear combinations of the columns of A:
โ ๐ด ๐ด๐ฅ|๐ฅ โ ๐ โ ๐ The nullspace (or kernel) of A, denoted
๐ฉ ๐ด , is the set of all vectors ๐ฅ mapped into zero by A:
๐ฉ ๐ด ๐ฅ|๐ด๐ฅ 0 โ ๐ if ๐ฑ is a subspace of ๐ , its orthogonal
complement, denoted ๐ฑ , is defined as:๐ฑ ๐ฅ|๐ง ๐ฅ 0 for all ๐ง โ ๐ฑ
Introducing Equality Constraints
Consider the problem
๐ฅ โ ๐ , ๐ด โ ๐ and ๐ : ๐ โ ๐
An Equivalent Problem
Introduce ๐ฆ โ ๐ and ๐ฆ ๐ด ๐ฅ ๐
min ๐ ๐ด ๐ฅ ๐ s. t. ๐ ๐ด ๐ฅ ๐ 0, ๐ 1, . . . , ๐ โ ๐ฅ 0, ๐ 1, . . . , ๐
min ๐ ๐ฆ s. t. ๐ ๐ฆ 0, ๐ 1, . . . , ๐
๐ฆ ๐ด ๐ฅ ๐ , ๐ 0, . . . , ๐ โ ๐ฅ 0, ๐ 1, . . . , ๐
Optimizing over Some Variables
Suppose is partitioned as , with and
Consider the problem
An Equivalent Problem
where
min ๐ ๐ฅ , ๐ฅ s. t. ๐ ๐ฅ 0, ๐ 1, . . . , ๐ ๐ ๐ฅ 0, ๐ 1, . . . , ๐
min ๐ ๐ฅ s. t. ๐ ๐ฅ 0, ๐ 1, . . . , ๐
Example
Minimize a Quadratic Function
Minimize over
An Equivalent Problem=
Epigraph Problem Form
Epigraph Form
Introduce a variable is optimal for this problem if and
only if is optimal for and The objective function of the epigraph
form problem is a linear function of
Epigraph Problem Form
Geometric Interpretation
Find the point in the epigraph that minimizes
Making Constraints Implicit Unconstrained problem
for
It has not make the problem any easier
It could make the problem more difficult, because is probably not differentiable
Making Constraints Explicit A Unconstrained Problem
where
An implicit equality constraint An Equivalent Problem
Objective and constraint functions are differentiable
Outline
Optimization Problems Basic Terminology Equivalent Problems Problem Descriptions
Convex Optimization Standard Form Local and Global Optima An Optimality Criterion Equivalent Convex Problems Quasiconvex Optimization
Problem Descriptions Parameter Problem Description Functions have some analytical or closed
form Example: , where
and Give the values of the parameters
Oracle Model (Black-box Model) Can only query the objective and
constraint functions by oracle Evaluate and its gradient Know some prior information (convexity)
Outline
Optimization Problems Basic Terminology Equivalent Problems Problem Descriptions
Convex Optimization Standard Form Local and Global Optima An Optimality Criterion Equivalent Convex Problems Quasiconvex Optimization
Convex Optimization Problems Standard Form
The objective function must be convex The inequality constraint functions must
be convex The equality constraint functions
must be affine
Convex Optimization Problems
Properties Feasible set of a convex optimization
problem is convex
Minimize a convex function over a convex set -suboptimal set is convex The optimal set is convex If the objective is strictly convex, then the
optimal set contains at most one point
dom ๐ โฉ ๐ฅ|๐ ๐ฅ 0 โฉ ๐ฅ|๐ ๐ฅ ๐
Concave Maximization Problems
Standard Form
It is referred as a convex optimization problem if is concave and are convex
It is readily solved by minimizing the convex objective function
Abstract From Convex Optimization Problem Consider the Problem
Not a convex optimization problem ๐ is not convex and โ is not affine
But the feasible set is indeed convex Abstract convex optimization problem
An Equivalent Convex Problem
min ๐ ๐ฅ ๐ฅ ๐ฅ s. t. ๐ ๐ฅ ๐ฅ 1 ๐ฅโ 0 โ ๐ฅ ๐ฅ ๐ฅ 0
min ๐ ๐ฅ ๐ฅ ๐ฅ s. t. ๐ ๐ฅ ๐ฅ 0 โ ๐ฅ ๐ฅ ๐ฅ 0
Outline
Optimization Problems Basic Terminology Equivalent Problems Problem Descriptions
Convex Optimization Standard Form Local and Global Optima An Optimality Criterion Equivalent Convex Problems Quasiconvex Optimization
Local and Global Optima
Any locally optimal point of a convex problem is also (globally) optimal
Proof by Contradiction is locally optimal implies
for some Suppose is not globally optimal, i.e.,
there exists and Define
๐ ๐ฅ inf ๐ ๐ง | ๐ง feasible, ๐ง ๐ฅ ๐
๐ง 1 ๐ ๐ฅ ๐๐ฆ, ๐๐
2 ๐ฆ ๐ฅ โ 0,1
Local and Global Optima By convexity of the feasible set
It is easy to check
By convexity of
which contradicts
๐ง is feasible
๐ง ๐ฅ ๐ ๐ฆ ๐ฅ๐ ๐ฆ ๐ฅ
2 ๐ฆ ๐ฅ๐ 2 ๐
๐ ๐ง 1 ๐ ๐ ๐ฅ ๐๐ ๐ฆ ๐ ๐ฅ
๐ ๐ฅ inf ๐ ๐ง | ๐ง feasible, ๐ง ๐ฅ ๐
Outline
Optimization Problems Basic Terminology Equivalent Problems Problem Descriptions
Convex Optimization Standard Form Local and Global Optima An Optimality Criterion Equivalent Convex Problems Quasiconvex Optimization
An Optimality Criterion for Differentiable
Suppose is differentiable ๐ ๐ฆ ๐ ๐ฅ ๐ป๐ ๐ฅ ๐ฆ ๐ฅ , โ๐ฅ, ๐ฆ โ dom ๐
An Optimality Criterion for Differentiable
Suppose is differentiable
Let denote the feasible set
is optimal if and only if and
๐ ๐ฆ ๐ ๐ฅ ๐ป๐ ๐ฅ ๐ฆ ๐ฅ , โ๐ฅ, ๐ฆ โ dom ๐
๐ ๐ฅ|๐ ๐ฅ 0, ๐ 1, , ๐, โ ๐ฅ 0, ๐ 1, . . . , ๐
๐ป๐ ๐ฅ ๐ฆ ๐ฅ 0 for all ๐ฆ โ ๐
An Optimality Criterion for Differentiable
is optimal if and only if and
defines a supporting hyperplane to the feasible set at
๐ป๐ ๐ฅ ๐ฆ ๐ฅ 0 for all ๐ฆ โ ๐
Proof of Optimality Condition
Sufficient Condition
Necessary Condition Suppose is optimal but
Define
So, for small positive ,
๐ป๐ ๐ฅ ๐ฆ ๐ฅ 0๐ ๐ฆ ๐ ๐ฅ ๐ป๐ ๐ฅ ๐ฆ ๐ฅ
โ ๐ ๐ฆ ๐ ๐ฅ
โ๐ฆ โ ๐, ๐ป๐ ๐ฅ ๐ฆ ๐ฅ 0
๐ ๐ง 0 ๐ ๐ฅ ,๐๐๐ก ๐ ๐ง ๐ก ๐ป๐ ๐ฅ ๐ฆ ๐ฅ 0
Unconstrained Problems is optimal if and only if Consider and When is small, is feasible
Unconstrained Quadratic Optimization
๐ฅ is optimal if and only if ๐ป๐ ๐ฅ ๐๐ฅ ๐ 0 If ๐ โ โ ๐ , no solution, ๐ is unbound below If ๐ โป 0, unique minimizer ๐ฅโ ๐ ๐ If ๐ is singular, but ๐ โ โ ๐ , ๐ ๐ ๐ ๐ฉ ๐
๐ป๐ ๐ฅ ๐ฆ ๐ฅ ๐ก ๐ป๐ ๐ฅ 0 โ ๐ป๐ ๐ฅ 0
min ๐ ๐ฅ 1 2โ ๐ฅ ๐๐ฅ ๐ ๐ฅ ๐, where ๐ โ ๐
Problems with Equality Constraints Only
Consider the Problem
is optimal if and only if๐ป๐ ๐ฅ ๐ฆ ๐ฅ 0, โ ๐ด๐ฆ ๐ ๐ฆ|๐ด๐ฆ ๐ ๐ฅ ๐ฃ ๐ฃ โ ๐ฉ ๐ด
Problems with Equality Constraints Only
Consider the Problem
is optimal if and only if
โ ๐ป๐ ๐ฅ ๐ฃ 0, โ ๐ฃ โ ๐ฉ ๐ด
โ ๐ป๐ ๐ฅ โฅ ๐ฉ ๐ด โ ๐ป๐ ๐ฅ โ ๐ฉ ๐ด โ ๐ด
โ ๐ป๐ ๐ฅ ๐ฃ 0, โ ๐ฃ โ ๐ฉ ๐ด
๐ป๐ ๐ฅ ๐ฆ ๐ฅ 0, โ ๐ด๐ฆ ๐ ๐ฆ|๐ด๐ฆ ๐ ๐ฅ ๐ฃ ๐ฃ โ ๐ฉ ๐ด
โ โ๐ฃ โ ๐ , ๐ป๐ ๐ฅ ๐ด ๐ฃ 0
Lagrange Multiplier Optimality Condition
๐ด๐ฅ ๐๐ป๐ ๐ฅ ๐ด ๐ฃ 0
Minimization over the Nonnegative Orthant Consider the Problem
is optimal if and only if
The Optimality Condition
The last condition is called complementarity
๐ป๐ ๐ฅ ๐ฆ ๐ฅ 0, โ๐ฆ โฝ 0
โบ๐ป๐ ๐ฅ โฝ 0๐ป๐ ๐ฅ ๐ฅ 0
๐ฅ โฝ 0, ๐ป๐ ๐ฅ โฝ 0, ๐ฅ ๐ป๐ ๐ฅ 0, ๐ 1, โฆ , ๐
โบ๐ป๐ ๐ฅ โฝ 0
๐ป๐ ๐ฅ ๐ฅ 0
Outline
Optimization Problems Basic Terminology Equivalent Problems Problem Descriptions
Convex Optimization Standard Form Local and Global Optima An Optimality Criterion Equivalent Convex Problems Quasiconvex Optimization
Equivalent Convex Problems Standard Form
Eliminating Equality Constraints
๐ด ๐ ; โฆ ; ๐ , ๐ ๐ ; โฆ ; ๐ ๐ด๐ฅ ๐, โ ๐น ๐ฉ ๐ด The composition of a convex function with an
affine function is convex
Equivalent Convex Problems Introducing Equality Constraints If an objective or constraint function has the
form ๐ ๐ด ๐ฅ ๐ , where ๐ด โ ๐ , we can replace it with ๐ ๐ฆ and add the constraint ๐ฆ๐ด ๐ฅ ๐ , where ๐ฆ โ ๐
Slack Variables Introduce new constraint ๐ ๐ฅ ๐ 0 and
requiring that ๐ is affine Introduce slack variables for linear inequalities
preserves convexity of a problem Minimizing over Some Variables It preserves convexity. ๐ ๐ฅ , ๐ฅ needs to be
jointly convex in ๐ฅ and ๐ฅ
Equivalent Convex Problems Epigraph Problem Form
The objective is linear (hence convex) The new constraint function ๐ ๐ฅ ๐ก is also
convex in ๐ฅ, ๐ก This problem is convex Any convex optimization problem is readily
transformed to one with linear objective.
Outline
Optimization Problems Basic Terminology Equivalent Problems Problem Descriptions
Convex Optimization Standard Form Local and Global Optima An Optimality Criterion Equivalent Convex Problems Quasiconvex Optimization
Quasiconvex Optimization Standard Form
๐ is quasiconvex and ๐ , โฆ , ๐ are convex Have locally optimal solutions that are not
(globally) optimal
min ๐ ๐ฅ s. t. ๐ ๐ฅ 0, ๐ 1, . . . , ๐ ๐ด๐ฅ ๐
Quasiconvex Optimization Standard Form
๐ is quasiconvex and ๐ , โฆ , ๐ are convex Have locally optimal solutions that are not
(globally) optimal Optimality Conditions for Differentiable Let ๐ denote the feasible set, ๐ฅ is optimal if
1. Only a sufficient condition2. Requires ๐ป๐ ๐ฅ to be nonzero
๐ฅ โ ๐, ๐ป๐ ๐ฅ ๐ฆ ๐ฅ 0 for all ๐ฆ โ ๐ \ ๐ฅ
min ๐ ๐ฅ s. t. ๐ ๐ฅ 0, ๐ 1, . . . , ๐ ๐ด๐ฅ ๐
Representation via family of convex functions Rerpesent the sublevel sets of a quasiconvex
function via inequalities of convex functions. is convex,
is a nonincreasing function of Examples
๐ ๐ฅ ๐ก โ ๐ ๐ฅ 0
๐ ๐ฅ 0 ๐ ๐ฅ ๐กโ otherwise
๐ ๐ฅ dist ๐ฅ, ๐ง|๐ ๐ง ๐ก
Quasiconvex Optimization via Convex Feasibility Problems Let , be a family of convex
functions such that
and for each , whenever Let ๐โ be the optimal value of quasiconvex problem Consider the feasibility problem
If it is feasible, ๐โ ๐ก. Conversely, ๐โ ๐ก
find ๐ฅ s. t. ๐ ๐ฅ 0
๐ ๐ฅ 0, ๐ 1, . . . , ๐ ๐ด๐ฅ ๐
Bisection for QuasiconvexOptimization Algorithm
The interval ๐, ๐ข is guaranteed to contain ๐โ
The length of the interval after ๐ iterations is 2 ๐ข ๐
log ๐ข ๐ /๐ iterations are required
given ๐ ๐โ, ๐ข ๐โ, tolerance ๐ 0repeat
1. ๐ก โ ๐ ๐ข /22. Solve the convex feasibility problem3. if it is feasible, ๐ข โ ๐ก; else ๐ โ ๐ก
until ๐ข ๐ ๐
Bisection for QuasiconvexOptimization
An โsuboptimal Solution โ
โ
โ โ โ
find ๐ฅ s. t. ๐ ๐ฅ 0
๐ ๐ฅ 0, ๐ 1, . . . , ๐ ๐ด๐ฅ ๐
Summary
Optimization Problems Basic Terminology Equivalent Problems Problem Descriptions
Convex Optimization Standard Form Local and Global Optima An Optimality Criterion Equivalent Convex Problems Quasiconvex Optimization