optimization problems
DESCRIPTION
Optimization Problems. 虞台文. 大同大學資工所 智慧型多媒體研究室. Content. Introduction Definitions Local and Global Optima Convex Sets and Functions Convex Programming Problems. Optimization Problems. Introduction. 大同大學資工所 智慧型多媒體研究室. General Nonlinear Programming Problems. objective function. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/1.jpg)
Optimization Problems
虞台文大同大學資工所智慧型多媒體研究室
![Page 2: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/2.jpg)
ContentIntroductionDefinitionsLocal and Global OptimaConvex Sets and FunctionsConvex Programming
Problems
![Page 3: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/3.jpg)
Optimization Problems
Introduction
大同大學資工所智慧型多媒體研究室
![Page 4: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/4.jpg)
General Nonlinear Programming Problems
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
objective function
constraints
![Page 5: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/5.jpg)
Local Minima vs. Global Minima
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
objective function
constraints
local minimum
global minimum
![Page 6: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/6.jpg)
Convex Programming Problems
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
objective function
constraints
f (x)
gi (x)
hj (x)
convex
concave
linear
Local optimality Global optimality
![Page 7: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/7.jpg)
Linear Programming Problems
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
objective function
constraints
f (x)
gi (x)
hj (x)
linear
linear
linear
Local optimality Global optimality
a special case of convex programming problems
![Page 8: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/8.jpg)
Linear Programming Problems
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
objective function
constraints
f (x)
gi (x)
hj (x)
linear
linear
linear
Local optimality Global optimality
![Page 9: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/9.jpg)
Integer Programming Problems
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx Z
objective function
constraints
f (x)
gi (x)
hj (x)
linear
linear
linear
![Page 10: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/10.jpg)
The Hierarchy of Optimization Problems
NonlinearPrograms
ConvexPrograms
LinearPrograms
(Polynomial) IntegerPrograms(NP-Hard)
Flowand
Matching
![Page 11: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/11.jpg)
Optimization Problems
General Nonlinear Programming Problems
Convex Programming Problems
Linear Programming Problems
Integer Linear Programming Problems
![Page 12: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/12.jpg)
Optimization Techniques
General Nonlinear Programming Problems
Convex Programming Problems
Linear Programming Problems
Integer Linear Programming Problems
ContinuousVariables
DiscreteVariables
ContinuousOptimization
CombinatorialOptimization
![Page 13: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/13.jpg)
Optimization Problems
Definitions
大同大學資工所智慧型多媒體研究室
![Page 14: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/14.jpg)
Optimization Problems
( )f xminimize
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
![Page 15: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/15.jpg)
( )f xminimize
Optimization Problems
( ) 0 1, ,ig x i m subject to
( ) 0 1, ,jh x j p nx R
Define the set of feasible points
F
Minimize cost c: FR1
![Page 16: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/16.jpg)
Definition:Instance of an Optimization Problem
(F, c) F: the domain of feasible points
c: F R1 cost function
Goal: To find f F such that
c( f ) c(g) for all gF.
A global optimum
![Page 17: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/17.jpg)
Definition:Optimization Problem
A set of instances of an optimization problem, e.g.– Traveling Salesman Problem (TSP)– Minimal Spanning Tree (MST)– Shortest Path (SP)– Linear Programming (LP)
![Page 18: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/18.jpg)
Traveling Salesman Problem (TSP)
![Page 19: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/19.jpg)
Traveling Salesman Problem (TSP)
Instance of the TSP – Given n cities and an n n distance matrix [dij], t
he problem is to find a Hamiltonian cycle with minimal total length.
on F n all cyclic permutations objects
( )1
n
j jj
c d
1 2 3 4 5 6 7 8
2 5 3 6 1 8 4 7
e.g.,
![Page 20: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/20.jpg)
Minimal Spanning Tree (MST)
![Page 21: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/21.jpg)
Minimal Spanning Tree (MST)
Instance of the MST – Given an integer n > 0 and an n n symmetric distance m
atrix [dij], the problem is to find a spanning tree on n vertices that has minimum total length of its edge.
( , ) {1,2, , }VF E V n all spanning trees with
( , )
: ( , ) iji j E
c V E d
![Page 22: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/22.jpg)
Linear Programming (LP)
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
![Page 23: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/23.jpg)
Linear Programming (LP)
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
1
2
n
c
cc
c
11 12 1
21 22 2
1 2
n
n
m m mn
a a a
a a aA
a a a
1
2
m
b
bb
b
1
2
n
x
xx
x
minimize
Subject to
c x
Ax b0x
![Page 24: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/24.jpg)
Linear Programming (LP)
, , 0nx x R AF x b x
:c x c x
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
minimize
Subject to
c x
Ax b0x
![Page 25: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/25.jpg)
Example:Linear Programming (LP)
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
1 2 34 2 3x x x
1 2 3
1 2 3
2
, , 0
x x x
x x x
4 2 3c
1 1 1A 2b
minimize
Subject to
![Page 26: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/26.jpg)
Example:Linear Programming (LP)
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
minimize 1 1 2 2 n nc x c x c x
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
m m mn n m
a x a x a x b
a x a x a x b
a x a x a x b
1 2, , , 0nx x x
Subject to
1 2 34 2 3x x x
1 2 3
1 2 3
2
, , 0
x x x
x x x
minimize
Subject to
x1
x2
x3
v1
v2
v3
c(v1) = 8
c(v2) = 4
c(v3) = 6
The optimum
The optimal point is at one of the vertices.
![Page 27: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/27.jpg)
Example:Minimal Spanning Tree (3 Nodes)
1 2 34 2 3x x x
1 2 3 2x x x
minimize
Subject to
c1=4
c3=3
c2=2
1 2 3, , {0,1}x x x
x1{0, 1}
x2{0, 1}
x3{0, 1}
Integer Programming
x1
x2
x3
![Page 28: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/28.jpg)
Example:Minimal Spanning Tree (3 Nodes)
1 2 34 2 3x x x
1 2 3 2x x x
minimize
Subject to
c1=4
c3=3
c2=2
x1{0, 1}
x2{0, 1}
x3{0, 1}
Linear Programming
x1
x2
x3
1 2 3, , 0x x x 1 2 3, , 1x x x
Some integer programs can be transformed into linear programs.
![Page 29: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/29.jpg)
Optimization Problems
Local and Global Optima
大同大學資工所智慧型多媒體研究室
![Page 30: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/30.jpg)
Neighborhoods
Given an optimization problem with instance
(F, c),
a neighborhood is a mapping
defined for each instance.
: 2FN F
For combinatorial optimization, the choice of N is critical.
![Page 31: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/31.jpg)
TSP (2-Change)
f F gN2(f )
2 ( ) N f g g F g and can be obtained as above
![Page 32: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/32.jpg)
TSP (k-Change)
( )
.k
g F gN f g
k f
and can be obtained
by changing edges of
![Page 33: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/33.jpg)
MST
f F gN(f )1. Adding an edge to form a cycle.2. Deleting any edge on the cycle.
( ) N f g g F g and can be obtained as above
![Page 34: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/34.jpg)
LP
minimize
Subject to
c x
Ax b0x
( ) , 0, N x y Ay b y y x and
![Page 35: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/35.jpg)
Local Optima
Given(F, c)
N
an instance of an optimization problem
neighborhood
f F is called locally optimum with respect to N (or simply
locally optimum whenever N is understood by context) if
c(f ) c(g) for all gN(f ).
![Page 36: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/36.jpg)
0 1 F
c
small
Local Optima
F = [0, 1] R1
( ) , 0, N f x x F y x f and
C
B
A Local minimum
Local minimum
Global minimum
![Page 37: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/37.jpg)
Decent Algorithm
f = initial feasible solution
While Improve(f ) do
f = any element in Improve(f )
return f
Improve( ) ( ) ( ) ( )f s s N f c s c f and
Decent algorithm is usually stuck at a
local minimum unless the neighborhood N
is exact.
![Page 38: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/38.jpg)
Exactness of Neighborhood
Neighborhood N is said to be exact if it makes
Local minimum Global Minimum
![Page 39: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/39.jpg)
Exactness of Neighborhood
0 1 F
c
F = [0, 1] R1
( ) , 0, N f x x F y x f and
C
B
A Local minimum
Local minimum
Global minimum
N is exact if 1.
![Page 40: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/40.jpg)
TSP
N2: not exact
Nn: exact
f F gN2(f )
![Page 41: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/41.jpg)
MST N is exact
f F gN(f )1. Adding an edge to form a cycle.2. Deleting any edge on the cycle.
( ) N f g g F g and can be obtained as above
![Page 42: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/42.jpg)
Optimization Problems
Convex Sets and Functions
大同大學資工所智慧型多媒體研究室
![Page 43: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/43.jpg)
Convex Combination
x, y Rn
0 1 z = x +(1)y
A convex combination of x, y.
A strict convex combination of x, y if 0, 1.
![Page 44: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/44.jpg)
Convex Sets
S Rn
z = x +(1)y
is convex if it contains all convex combinations of pairs x, y S.
convex nonconvex
0 1
![Page 45: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/45.jpg)
Convex Sets
S Rn
z = x +(1)y
is convex if it contains all convex combinations of pairs x, y S.
n = 1
S is convex iff S is an interval.
0 1
![Page 46: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/46.jpg)
Convex Sets
Fact: The intersection of any number of convex sets is convex.
![Page 47: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/47.jpg)
c
Convex Functions
x yx +(1)y
c(x)
c(y)c(x) + (1)c(y)
c(x +(1)y)
S Rn a convex set
c:S R a convex function if
c(x +(1)y) c(x) + (1)c(y), 0 1
Every linear function is convex.
![Page 48: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/48.jpg)
LemmaS
c(x)
t
a convex set
a convex function on S
a real number
( ) ,tS c x x Stx
is convex.
Pf) Let x, y St x +(1)y S
c(x +(1)y) c(x) + (1)c(y)
t + (1)t
= t
x +(1)y St
![Page 49: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/49.jpg)
Level Contours
c = 1
c = 2
c = 3
c = 4
c = 5
![Page 50: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/50.jpg)
Concave Functions
S Rn a convex set
c:S R a concave function if
c is a convex
Every linear function is concave as well as convex.
![Page 51: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/51.jpg)
Optimization Problems
Convex Programming Problems
大同大學資工所智慧型多媒體研究室
![Page 52: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/52.jpg)
Theorem
(F, c) an instance of optimization problem
a convex set
a convex function
Define ( )N x y y F x y and
( )N x is exact for every > 0.
![Page 53: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/53.jpg)
• Let x be a local minimum w.r.t. N for any fixed > 0.• Let yF be any other feasible point.
Theorem
(F, c) an instance of optimization probleman instance of optimization problem
a convex set
a convex function
Defi ne ( )N x y y F x y and
( )N x is exact f or every > 0.
(F, c) an instance of optimization probleman instance of optimization problem
a convex set
a convex function
Defi ne ( )N x y y F x y and
( )N x is exact f or every > 0.
Pf)
xF
( )N x
yNext, we now want to show that c(y) c(x).
![Page 54: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/54.jpg)
• Let x be a local minimum w.r.t. N for any fixed > 0.• Let yF be any other feasible point. <<1 such that• Since c is convex, we have
• Therefore,
Theorem
(F, c) an instance of optimization probleman instance of optimization problem
a convex set
a convex function
Defi ne ( )N x y y F x y and
( )N x is exact f or every > 0.
(F, c) an instance of optimization probleman instance of optimization problem
a convex set
a convex function
Defi ne ( )N x y y F x y and
( )N x is exact f or every > 0.
Pf)
xF
( )N x
yz
(1 ) ( ).x y xz z N and
( ) ( (1 ) )c c x yz
( ) (1 ) ( )c x c y
( ) ( )( )
1
zc c xc y
( ) ( )
1
c x c x
( )c x
( ) ( )zc c x
![Page 55: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/55.jpg)
Convex Programming Problems
(F, c)
Defined by ( ) 0, 1, ,ig x i m
: nig R R
Convex function
an instance of optimization problem
Important property:
Local minimum Global Minimum
Concave functions
![Page 56: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/56.jpg)
Convexity of Feasible Set
(F, c)
Defined by ( ) 0, 1, ,ig x i m
: nig R R
Convex function
an instance of optimization problem
Important property:
Local minimum Global Minimum
Concave functions
( ) : ig x convex
( ) : ig x concave
( ) 0 : ig x convex
( ) 0 : ig x convex
: iF convex
1
: m
ii
F F
convex
![Page 57: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/57.jpg)
Convex Programming Problems
(F, c)
Defined by ( ) 0, 1, ,ig x i m
: nig R R
Convex function
an instance of optimization problem
Important property:
Local minimum Global Minimum
Concave functionsConvex
![Page 58: Optimization Problems](https://reader035.vdocuments.mx/reader035/viewer/2022062321/568134d5550346895d9bfdbd/html5/thumbnails/58.jpg)
Theorem
In a convex programming problem, every
point locally optimal with respect to the
Euclidean distance neighborhood N is also
global optimal.