€¦ · 1 intr oduction in the moder n inter ior-point theor y, all difficult constr aints are...
TRANSCRIPT
Algorithm
sfor
Cone
Program
ming
(PartI)
Leven
tTu
ncel
Dep
t.o
fC
om
bin
atorics
and
Op
timizatio
n,
Faculty
of
Math
ematics,
Un
iversityo
fW
aterloo
,
Can
ada.
May
12,2004
1
1IN
TR
OD
UC
TIO
N
1In
trod
uctio
n
This
partofthecourse
isalm
ostcompletely
focusedon
Interior-PointM
ethods
anda
biton
The
Ellipsoid
Method
2
1IN
TR
OD
UC
TIO
N
Inthe
modern
interior-pointtheory,alldifficultconstraints
arepushed
into
theconvex
setconstraintsand/or
convexcone
constraints.T
hen,eachof
theseconvex
inclusionconstraints
aretreated
viaa
strictlyconvex
barrier
functionw
ithvery
specialproperties.
We
willpresentm
ostofourresults
inthe
fullgeneralityofan
arbitrary
convexset�
in���
oran
arbitraryconvex
cone�
in� �
.H
owever,
specialattentionw
illbepaid
toS
emidefinite
Program
ming
(SD
P).
Recall,
����
denotesthe
convexcone
of �
symm
etric,positive
semidefinite
matrices
overthe
reals.
����
isthe
interiorof� �
;i.e.,theconvex
coneof �
symm
etric,
positivedefinite
matrices
overthe
reals.
3
1IN
TR
OD
UC
TIO
N
�
Second-order
cone:
��� ����
���
� � � �
� � � �
����
SD
Pstands
forsem
idefiniteprogram
ming
where �
ism
adeup
fromdirect
sums
ofvarious� ��
(possiblyunder
some
linearisom
orphisms).
SO
CP
standsfor
second-ordercone
programm
ingw
here �
ism
adeup
fromdirectsum
sofvarious
��� �
�
(possiblyunder
some
linear
isomorphism
s).
4
1IN
TR
OD
UC
TIO
N
We
considerthe
semidefinite
programm
ing(S
DP
)problem
sin
the
following
primal� �
�
anddual���
�
forms.
� ��
inf
� ����
� ��
�
��
�
� �
����
sup����
��� ������
��
�� �
where
isa
linearoperator
from
� �
to ���
,sothat�
� � �
and �
denotesthe
adjointof
.
5
1IN
TR
OD
UC
TIO
N
Withoutloss
ofgenerality,we
assume
that
issurjective.
Ifnot,we
can
considerthe
representation� �����
����������� ������� ,w
here
�� � � �
forevery�
.
beingsurjective
isequivalentto
��� ��� ���� ��
beinglinearly
independent.T
helatter
canbe
assumed
withoutloss
of
generality,sinceifthey
arelinearly
dependent,theneither
thesystem
� ����� ����������� ���� ��
hasno
solution,orthere
aresom
e
redundantequationsw
hichcan
beelim
inated.In
thefirstcase,� �
�
is
infeasible.In
thesecond
case,allredundantequations,andcorresponding
��� ��
canbe
eliminated,to
arriveatan
equivalentproblemsatisfying
the
assumption.
6
1IN
TR
OD
UC
TIO
N
Under
thisassum
ption,forany
solution,� � � �� ,oftheequation
�� ��� � �� ���
�
,the �
partofthesolution
uniquelyidentifies
the
corresponding�.
Som
etimes,in
interior-pointalgorithms,itis
convenient
torefer
onlyto �
when
onem
entionsa
feasiblesolution
of���� .
The
abovesetting
oftheprim
al-dualSD
Ppair
canbe
embedded
inthe
following
more
generalsettingofconic
convexoptim
izationproblem
s:
� � ��
inf����
� �
�
��
���
where
isa
surjectivelinear
map
and�
isa
pointed,closed,convex
conew
ithnon-em
ptyinterior.
7
1IN
TR
OD
UC
TIO
N
We
definethe
dualof� � ��
as
� � ��
sup
� ��� �
�� ���
��
��
����
where �
isthe
dualofcone �w
ithrespectto���� � ,i.e.
�
� �
��� ��� �� ������
We
willrefer
tothis
settingas
theconic
convexoptim
izationsetting.
8
1IN
TR
OD
UC
TIO
N
SD
Pproblem
fitsinto
thisgeneralsetting
byletting
�� �� �� �
(thatis,�� �� ������
),
��� �� �
,
�� � �
� �� �� � .
9
1IN
TR
OD
UC
TIO
N
Under
thesedefinitions
we
have���
.I.e.,the
coneofsym
metric
positivesem
idefinitem
atricesis
self-dualunder
thetrace
innerproduct.
In
additionto
beingself-dual,the
cone� �
enjoysanother
symm
etry
property,inthatitis
homogeneous.
Thatis,the
setofnonsingularlinear
transformations
keeping� �the
same
(theautom
orphismgroup
of� �
)is
richenough
tocontain
lineartransform
ationsw
hichm
apany
fixedinterior
pointtoany
otherfixed
interiorpointof� �
.C
onvexcones
with
both
properties,i.e.hom
ogeneousself-dualcones,are
alsocalled
symm
etric.
10
1IN
TR
OD
UC
TIO
N
��
isself-dualifthere
existsan
inner-productunderw
hich� ���
.
��
ishom
ogeneousifA
ut� ��
actstransitively
onint� �
� .
��
issym
metric
if �is
homogeneous
andself-dual.
So,w
ehave
theunderlying
optimization
problems:
�
Sym
metric
Cone
Program
ming
(Sym
CP
)
�
Hom
ogeneousC
oneP
rogramm
ing(H
omC
P)
11
1IN
TR
OD
UC
TIO
N
�A
homogeneous
polynomial�
�� ����
ishyperbolic
inthe
direction��� �
,iftheunivariate
polynomial(in�
��
)
�� ����
hasonly
realrootsfor
every�� �
.
�
Aconvex
cone�
ishyperbolic
ifitis
�� ���� �
��� � � ������
fora
polynomial�
which
ishyperbolic
inthe
direction��� �
.
�
Hom
ogeneouscones
make
upa
propersubsetofhyperbolic
cones.
Hyperbolic
Cone
Program
ming
(HypC
P)
12
1IN
TR
OD
UC
TIO
N
Strictly
speakingw
ehave,
������ �
�� ��
��� �� �
��
��� �
��� � � �
�� ��
How
ever,insom
esense,
������ �
�� ���
�� �� ��
���� �
��� � � �
�� ��
Yetinan
anothersense,
���
��� �
�� ���
�� �� ��
���� �
��� � � �
�� ��
13
1IN
TR
OD
UC
TIO
N
Recallthe
weak
dualityrelation:
Pro
po
sition
1.1Let
��
befeasible
in� �� ,and�
�� ����
befeasible
in� �� .
Then
� ���
�
�
� ���
��
���
�� �
Sim
ilarlyfor
theconic
convexoptim
izationsetting...
Therefore,ifw
estart
with �
� �
and �� �
bothfeasible
intheir
respectiveproblem
s,then
decreasing� �� �
willgetus
closerto
optimality!14
1IN
TR
OD
UC
TIO
N
Since
thelinear
operator
issurjective,w
ecan
always
find
��� � �
suchthat
�
��� �� �
For
thedual,because
oftheform
we
chose,we
canalw
aysfind
���� �
,
��� � �
suchthat
��
�������
���
Denoting
�
���� �� � ��� �� � � �
we
claimthat� �
�
and����
areequivalentto
thefollow
ingpair.
15
1IN
TR
OD
UC
TIO
N
��
��
inf
������
�
��
���
�� �
�
� �
��
��
inf�
��� ��
��
���
��� �
�� �
here,
��
denotesthe
orthogonalcomplem
entof�
.16
1IN
TR
OD
UC
TIO
N
We
have
� ��
��
inf
����
�
���
���
��
and
� ��
��
inf� ��
�
�
�
��
�����
�
�
�
inthe
generalconicconvex
optimization
setting.To
establishthe
equivalence,firstnotethatthe
feasibleregions
arepreserved
(in� ��
��
we
onlyrefer
to
� ).
17
1IN
TR
OD
UC
TIO
N
Recallthe
proofofthew
eakduality
relation(P
roposition1.1).
For
every�� �
satisfying� � �� ,and
forevery� � � ��
satisfying
��� ����� ��
we
have
� ��
�
� ��� ��
� � � �
Ifwe
fix��� ����
suchthat
��
������� �
�
thenfor
all�� �
satisfying� � ��
we
have
� ��
�� �
�����������
where
theconstantis� ��
�� �
.T
herefore,minim
izing
� ��
subjecttoany
setofconstraints,containingthe
constraint� � ��
isequivalentto
minim
izing
� ���
subjecttothe
same
setofconstraints.
18
1IN
TR
OD
UC
TIO
N
Sim
ilarly,we
canestablish
theequivalence
ofthedualproblem
s.W
efix
�
suchthat
��� �
�then
forall� � � ��
satisfying �� �� �� ��
,we
have�� ��� ��
��� ���
�������
where
theconstantis
�����
� .T
herefore,maxim
izing� ��� �
subjectto
anysetofconstraints
containingthe
constraint �� �� �� ��
is
equivalenttom
inimizing�
�� �
subjecttothe
same
setofconstraints.
19
2E
LLIPS
OID
ME
TH
OD
2E
llipso
idM
etho
d�
�� �
isan
ellipsoidifthere
exist�� � �
(determining
thecenter)
and
�� � �
(determining
thesize
andthe
shape)such
that
������ �� �� � �
�� ���
��� ����
��
��� ��
�
We
canalternatively
expressthe
ellipsoidas
theim
ageofthe
unitballin
� �
(denotedby��
� � �� )
underan
affinem
appingas
follows:
�� �� �� �� �� ���� � �� ���
The
volume
oftheellipsoid
isproportionalto
thesquare-rootofthe
determinantofthe
positivedefinite
matrix
determining
itsshape:
vol� �� �� ��� �
���� �� vol� ��� � ��� �
20
2E
LLIPS
OID
ME
TH
OD
The
volume
ofthe
�
-dimensionalunitballis
vol� ��� � ��� �
� �� �
�� �� ��� �
where
�� ����� �� ���� ��
�� ,for .
We
firststudythe
ellipsoidm
ethodas
analgorithm
which
computes
apoint
inan
implicitly
describedconvex
set.In
thisbasic
setting,itiseasy
tosee
thattheellipsoid
method
isa
beautifulandtheoretically
verypow
erful
generalizationofthe
bisectionm
ethodfrom
�to� �
,foran
arbitrary
�
.
21
2E
LLIPS
OID
ME
TH
OD
2.1Ingredients:
Separation
Oracles,Inscribed
andC
ircumscribed
Ellipsoids
2.1In
gred
ients:
Sep
aration
Oracles,In
scribed
and
Circu
mscrib
edE
llipso
ids
Toappreciate
thefulltheoreticalpow
erofthe
Ellipsoid
Method,w
ew
ill
move
away
fromthe
explicitdescription
oftheconvex
optimization
problem
athand.Instead,w
ew
illassume
thatwe
aregiven
accessto
aseparation
oracle.Let�
�� �
bethe
convexsetw
eare
interestedin
optimizing
over
orless
ambitiously
justfindinga
pointinsideit(the
set�
).W
edefine
� -relaxationof�
asfollow
s:
relax� ��
�����
�
�� ���
�
�
�� ��
forsom
e
���
22
2E
LLIPS
OID
ME
TH
OD
2.1Ingredients:
Separation
Oracles,Inscribed
andC
ircumscribed
Ellipsoids
Aw
eakseparation
oraclefor�
takesas
input
��
� �
,
�
��
.It
eitheroutputs
“
��
relax� ��
�� ”or�
�� �
suchthat� �
� ���
and
� �
��
� �
� �
�� ��
relax� ��
�� �
Th
eorem
2.1F
orevery
compact,convex
setin���
with
nonempty
interior,
thereexists
aunique
minim
alvolume
ellipsoidcontaining
thatset.
Moreover,shrinking
thatellipsoid(around
itscenter)
bya
factorofatm
ost
�
givesan
ellipsoidcontained
inthe
convexset.
The
uniqueellipsoid
describedin
theabove
theoremis
usuallycalled
the
Lowner-John
ellipsoid.
The
factor
�
inthe
abovetheorem
isthe
best
possible.(T
he
�
-dimensionalsim
plexproves
theclaim
forevery
�
.)
23
2E
LLIPS
OID
ME
TH
OD
2.1Ingredients:
Separation
Oracles,Inscribed
andC
ircumscribed
Ellipsoids
Let
����
�� ���
��� �� �
��
��� ��
�
and
�����
� ���� ��� ��
� �
forsom
e�
� � ��
� � .W
ew
illassume
� �
.W
ew
ouldlike
to
constructthesm
allestvolume
ellipsoid �
containingthe
half-ellipsoid
��
.
Let�� � �
and
�� � �
denotethe
centerand
thepositive
definite
matrix
determining�
.T
hen
���
�
�
� ����
� ��
�
��
�
� �
� �� �
��� �
�
�
� ���� � ��
�
��� ��
�
�24
2E
LLIPS
OID
ME
TH
OD
2.1Ingredients:
Separation
Oracles,Inscribed
andC
ircumscribed
Ellipsoids
We
canexplicitly
compute
thevolum
eof�
interm
softhe
volume
of�
,
since�
isa
rank-1update
of�
.W
henw
etake
theim
ageof���
under
them
apping� �
�� �
,ourellipsoid�
becomes
theunitball.
Under
this
mapping
ourupdate
formula
becomes
��
� �� �
��� �
�
�
� ����
�� �
��
��
�� �
� �
where
��
� �� �� �
�
.T
heeigenvalues
of
� ��
��� �
��������
��
�� �
�
are � ��
� �
and�
(with
multiplicity� �
��� ).
Therefore,
���� �� �
����
� �
��
��� ��
�
� �� �
��� ��
� �� ���� �
�
25
2E
LLIPS
OID
ME
TH
OD
2.1Ingredients:
Separation
Oracles,Inscribed
andC
ircumscribed
Ellipsoids
Hence
thevolum
eof �
canbe
relatedto
thevolum
eof �
asfollow
s:
vol� �� �
�� �
��� ��
����
vol� �� �
Th
eorem
2.2W
ehave
��
��
and��
vol� ��
vol� ���
���
� �26
2E
LLIPS
OID
ME
TH
OD
2.2C
omplexity
Analysis
forthe
Ellipsoid
Method
2.2C
om
plexity
An
alysisfo
rth
eE
llipso
idM
etho
d
Suppose
vol� �� �
�
andw
ew
ant ��
suchthatvol� �
�� �� .
Then
by
thelasttheorem
,
��
vol� ���
vol� ���
�
��� �
and�
� ���
� �� ��
iterationssuffice.
Sim
ilarly,if�
isthe
radiusofthe
initialball(whose
image
underthe
mapping
��� �
�� �
containsthe
set)and
we
wantthe
stoppingcriterion
tobe
thattheradius
ofthecurrent
ball(whose
image
underthe
mapping
��� �
�� �
�
contains�
)is
at
most� ,then
using
vol� ���
vol� ��
��
��
��
�27
2E
LLIPS
OID
ME
TH
OD
2.2C
omplexity
Analysis
forthe
Ellipsoid
Method
we
findthat�
� � ���
� �� ��
iterationssuffice.
Th
eorem
2.3Let�
�� �
bea
convexsetsuch
that����� � �� .
Suppose
we
arealso
given
��
�
.Ifw
ehave
accessto
aw
eak
separationoracle
for�,then
inpolynom
ialtime
(polynomialin
� ����
� �� ��� ),w
ecan
compute
��
relax� �� ��
orprove
that
vol� �� �� .
These
resultscan
beextended
tooptim
izinga
convexfunction
over�
.W
e
needone
more
ingredienttodealw
iththe
objectivefunction.
Let
�
�� ����
bea
convexfunction.
28
2E
LLIPS
OID
ME
TH
OD
2.2C
omplexity
Analysis
forthe
Ellipsoid
Method
Defi
nitio
n2.1
Asubgradientoracle
for
�
takesas
input
��� �
and
returnsin
polynomialtim
e(polynom
ialin
�
andfor
aproper
definition,
size��� )
����
and��� �
suchthat
�� �
������
� ��
�
�� � �� � ��
Suppose
we
areinterested
insolving
theconvex
optimization
problem
��
�
��� ����
� �w
here
�
�� ����
isa
convexfunction.
29
2E
LLIPS
OID
ME
TH
OD
2.2C
omplexity
Analysis
forthe
Ellipsoid
Method
Th
eorem
2.4Let�
�� �
bea
convexsetand
�� �� �
begiven
suchthat
���
�� ����
���� � �� �
forsom
e
��� �
(here,
�
isn
ot
given).Let�
��
bealso
given.S
upposethata
subgradientoraclefor
�and
aw
eakseparation
oraclefor�
areavailable.
Then
after
�
� ��
��
���
��
�� �
��
iterations,theellipsoid
method
returnsa
feasiblesolution
���
suchthat
���� �
��
�
��� ����
� �� �
Inthe
above,� ���
����
��� �
��� ��
�
��
��� �
��� �� .
30
2E
LLIPS
OID
ME
TH
OD
2.3B
ibliographicalNotes
2.3B
iblio
grap
hicalN
otes
Until1979
veryfew
mathem
aticiansin
theW
estknewaboutthe
ellipsoid
method.
Khachiyan,in
1979proved
thattheE
llipsoidM
ethodcan
be
adaptedto
solvelinear
programm
ingproblem
sin
polynomialtim
e(hence
settlinga
longoutstanding
problem).
This
announcementcaused
an
unprecedentedreaction
fromthe
media.
The
Ellipsoid
Method
was
originallyproposed
byIudin
andN
emirovskiin
1976also
some
relatedw
orkis
dueto
Shor
in1977.
This
originalmethod
was
designedto
dealwith
essentiallyany
convexoptim
izationproblem
thatcanbe
posedin
afinite
dimensionalspace
bythe
potentialusageof
oracles(an
importantpointw
hichshould
beem
phasizedis
thatthe
functionsinvolved
indefining
thefeasible
solutionset,the
objective
31
2E
LLIPS
OID
ME
TH
OD
2.3B
ibliographicalNotes
functionneed
no
tbe
differentiable).F
ora
niceexposure
toE
llipsoid
Method
forlinear
programm
ingproblem
s,seethe
surveypaper
byB
land,
Goldfarb
andTodd,O
perationsR
esearch29
(1981)1039–1091.
Shortly
afterK
hachiyan’sresult,itw
asestablished
thatthism
ethodis
a
verypow
erfultoolindeterm
iningthe
computationalcom
plexitystatus
(hencethe
degreeofdifficulty)
ofvariouscom
binatorialoptimization
problems.
Agood
referenceis
Geom
etricalgorithm
sand
combinatorial
optimization
byM
.Grotschel,L.Lovasz
andA
.Schrijver.
32
2E
LLIPS
OID
ME
TH
OD
2.3B
ibliographicalNotes
Inthe
late1980’s,E
llipsoidM
ethodw
asapplied
tosom
eproblem
sin
System
andC
ontrolTheory
(seethe
bookLinear
Matrix
Inequalitiesin
System
andC
ontrolTheory
byB
oydetal.).
These
problems
were
small
butrelativelydifficultconvex
optimization
problems.
Since
late1980’s
and
early1990’s
interior-pointmethods
consistentlytook
overthe
solution
process.T
heseare
them
ethodsw
ediscuss
andanalyze
next.
33
3C
EN
TR
AL
PATH
3C
entralP
ath
One
ofthem
ostimportantconcepts
ininterior-pointm
ethodsis
thecentral
path.W
earrive
atthisconceptvia
anothercentalconcept:
thebarrier
forthe
difficultconstraints.
Let
�
�� ����
bea
logarithmically
homogeneous
self-concordant
barrierfor�
.
34
3C
EN
TR
AL
PATH
Defi
nitio
n3.1
(LHS
CB
)Let
�
�int� �� ��
bea���
-smooth
convex
functionsuch
that
�
isa
barrierfor �
(i.e.
�� � ��
as�
int� ��
approaches� �)
andthere
exists� �
suchthatfor
each� ,
�� ��
��� �
�
���
� �� �
and
� ��� �� �� �� �����
� ��
�� �� �� ��� �� �
forall
�
int� ��
andfor
all���
.T
hen�
iscalled
a�
-LHS
Cbarrier
for�
.
35
3C
EN
TR
AL
PATH
��� ��
� ��
��
�
�� int� �
� � � �� ��� ��
Legendre-FenchelC
onjugate
��
alsohas
theabove
mentioned
propertiesfor�
forthe
same
barrier
parameter�
.
�
isa
veryim
portantparameter
ofthesebarriers.
Currently,one
ofthe
bestiterationbounds
forinterior-pointm
ethodsfor
conicconvex
optimization
is
�
���
��
tocom
putean
� -optimalsolution.
36
3C
EN
TR
AL
PATH
Let� .
Consider
� � �� �
inf
� �� �
��� �
� �
�
��
�
��� �
and
� � �� �
inf
�� ��� ��
���� ��
��� ���
�
���
� �
��� �
37
3C
EN
TR
AL
PATH
Itisw
ell-known
that
Th
eorem
3.1S
uppose� � �� �� ��
�
int� ��
�
int� ��
feasiblein
� � ��
and� � ��
exists.T
hen� � �� �
and� � �� �
havea
unique
optimalsolution
pair� �� ,� �� �� � �� ��� ,foreach
� .
Defi
nitio
n3.2
� � � �� ��� �� � �� ��� �� �
iscalled
theprim
al-dual
centralpathfor
thepair� � �� �� � �
� .
Som
etimes
we
referonly
to� � �� � �� ��� .
38
3C
EN
TR
AL
PATH
3.1C
entralPath
forS
DP
3.1C
entralP
athfo
rS
DP
Let’sfocus
onS
DP
first.H
ere,�� ��
����
��
� ���� ��� �
��� �� �
�
��
� ���� ���
�
��� ��
�
�39
3C
EN
TR
AL
PATH
3.1C
entralPath
forS
DP
The
centralpathis
equivalentlycharacterized
asthe
uniquesolution
(via
necessaryand
sufficientconditionsfor
optimality)
oftheconvex
optimization
problem
� �� � :
� �� ����
�
�
� �� ��
�� �����
�� �
Let’sdo
thesubstitutions�
� ���
and ����
� � ��
.W
eobtain
the
system
� ��
�
�� ��
�
�� ������
��
��
� � ��
�
40
3C
EN
TR
AL
PATH
3.1C
entralPath
forS
DP
For
each
� ,the
uniquesolution
oftheabove
system� �� �� ��� �� � �� ���
definesthe
primal-dualcentralpath.
Path-F
ollowing
algorithms
“closely”or
“loosely”follow
thispath
tothe
setof
optimalsolutions.
(Note
thatunderthe
assumption
oftheexistence
of
strictlyfeasible
pointsfor
bothprim
alandthe
dual,bothproblem
sdo
have
optimalsolutions
andthere
isno
dualitygap.)
Potential-R
eductionalgorithm
sreduce
theproblem
tothatofm
inimizing
(ordriving
to
��
)a
combination
oftheobjective
function:
�����
�� ��
�� �� �
anda
barrier(e.g.,
�
��
� ���� ���
ora
measure
ofcentrality.
41
3C
EN
TR
AL
PATH
3.2N
eighbourhoodsofthe
CentralP
ath
3.2N
eigh
bo
urh
oo
ds
of
the
Cen
tralPath
Again,w
efirstfocus
onS
DP.To
followthe
centralpath,orto
understand
thepotential-reduction
algorithms
ina
unifyingw
ay,itisw
orthstudying
the
neighbourhoodsofthe
centralpath.In
theprim
al-dualsetting,thisis
quite
elegantandeffective.
Given
strictlyfeasible
points�and�
,define�� �� �� �
�
Let
�
denotethe
setofallstrictlyfeasible
solutionpairs� �
� ��
( �
satisfiesallthe
primalconstraints
andis
positivedefinite,�
satisfiesallthe
dualconstraintsand
ispositive
definite).T
henw
ecan
expressm
any
neighbourhoods.
42
3C
EN
TR
AL
PATH
3.3N
eighbourhoodsB
asedon
theA
lgebraicD
escriptionofthe
CentralP
ath
3.3N
eigh
bo
urh
oo
ds
Based
on
the
Alg
ebraic
Descrip
tion
of
the
Cen
tralPath
Let��� � ��
bean
absoluteconstant.
We
define
(i)so-called
wide
neighborhoods
��� ��
� �
� �� ���
������� � �� ���
�� �
� � �
�
�� ��
(ii)infinity-norm
neighborhoods
�� ��
���
� �� ���
��
��� �� � �� ���
�� �
��
�������
��
�
�
(ii) �orequivalently,
�� ��� �
� �� ���
��
����� ��� ���
�� �
�
�
� ������ ��
�
43
3C
EN
TR
AL
PATH
3.3N
eighbourhoodsB
asedon
theA
lgebraicD
escriptionofthe
CentralP
ath
(iii)so-called
tight(ornarrow
)neighborhoods
�� ��
���
� �� ���
��
����� ��� ���
�� �
�
�
� ������ �
�
�
Note
that(asis
well-know
n),
CentralP
ath
�
�� ��
�
�� ��
�
��� ��
��
�44
3C
EN
TR
AL
PATH
3.4N
eighbourhoodsB
asedon
theA
nalyticD
escriptionsofthe
CentralP
ath
3.4N
eigh
bo
urh
oo
ds
Based
on
the
An
alyticD
escriptio
ns
of
the
Cen
tralPath
We
definea
measure
ofcentralitybased
onthe
barriervalues:
�� �� ��� �
��
� �� �
�
��
� ���� ���
�
��
� ���� ��� �
Th
eorem
3.2F
orevery� �
� ��� � �
�
� �
,�� �� �� �
Moreover,the
equalityholds
aboveiff �
�� � �
�
forsom
e
� .
This
theoremgeneralizes
theA
rithmetic-G
eometric
Mean
Inequalityand
thecorresponding
characterizationfor
equality.
45
3C
EN
TR
AL
PATH
3.4N
eighbourhoodsB
asedon
theA
nalyticD
escriptionsofthe
CentralP
ath
We
canalso
definea
proximity
measure
basedon
thegradients
ofthe
barrier�
��
� ���� ��� ,
�
��
� ���� ��� :
Let
������ � �
�� � �
�
�
We
have
Th
eorem
3.3F
orevery�
�
,��
,�
�� ��
Equality
holdsabove
iff ��
� � ��
.
The
abovetheorem
generalizesthe
Arithm
etic-Harm
onicM
eanInequality
andthe
correspondingcharacterization
forequality.46
3C
EN
TR
AL
PATH
3.4N
eighbourhoodsB
asedon
theA
nalyticD
escriptionsofthe
CentralP
ath
Inthe
general,conicconvex
optimization
setting,thecentralpath
equation
� ��
��
�� �
replaces��
�� � �
�.
The
firstproximity
measure
isgeneralized
to:
�� � ��� ��
��
� � ��
��� � �
��� �� �� �
The
nexttheoremshow
sthatfor
everypair
ofinteriorsolutions� � �� ,the
proximity
measure
isnonnegative
anditis
equaltozero
ifandonly
ifthe
point� � ��
lieson
thecentralpath.
47
3C
EN
TR
AL
PATH
3.4N
eighbourhoodsB
asedon
theA
nalyticD
escriptionsofthe
CentralP
ath
Th
eorem
3.4Let
�
bea
LHS
CB
for �
with
parameter�
.T
hen�� � �� �
forall
�
int� �� ���
int� �� �
Moreover,the
inequalityabove
holdsas
equalityiff
� ���
��� � �
forsom
e� �
48
3C
EN
TR
AL
PATH
3.4N
eighbourhoodsB
asedon
theA
nalyticD
escriptionsofthe
CentralP
ath
For
convenience,we
write
����
��
�
�� ��
and
�����
��
�� � .O
necan
thinkof
�
and��
asthe
shadowiterates,as
��
int� ��
and
���
int� ��
andif� � ��
isa
feasiblepair,then
�� �
iff��� �
�
iff� � ��
lieson
thecentralpath.
We
alsodenote
��� ��
����� � �
Th
eorem
3.5F
orevery� � ��
�int� �
��
int� �� ,
���
��
Equality
holdsabove
iff ��
��
�
�� ��
(andhence
� ��
��
�� � ).
49
3C
EN
TR
AL
PATH
3.5P
ath-Follow
ing
3.5P
ath-F
ollo
win
g
While
mostelegantpotential-reduction
algorithms
mightnotm
akeany
referenceto
thecentralpath
(evenincluding
thetheoreticalanalysis),
path-following
andpotential-reduction
algorithms
arevery
closely
connected.
Currently,m
ostofthepracticalim
plementations
ofIPM
sboth
forLP
and
Sym
CP
arebased
onm
odificationsofpath-follow
ingalgorithm
s.
50
3C
EN
TR
AL
PATH
3.5P
ath-Follow
ing
There
arem
anystrategies
availableto
usfor
following
thecentralpath.
Our
algorithms
generatesearch
directions
��
and ��
andstep
sizes
��
and
��
andupdate
����
�� ���
and
��
���
� �� �
Intheory,itis
veryconvenientto
take
�� �
��
��
� �
51
3C
EN
TR
AL
PATH
3.5P
ath-Follow
ing
How
ever,thereare
possibleadvantages
inpractise
toallow
themto
take
differentvalues.
52
3C
EN
TR
AL
PATH
3.5P
ath-Follow
ing
Whatproperties
dow
eask
forin
thesearch
directions?
�
Improve
thecurrentduality
gap� �� �
�
Getcloser
tothe
centralpathw
ithoutincreasing� �� �
(verym
uch)
�
Asuitable
combination
ofthefirsttw
oabove!
We
canalso
mix
theseproperties
“externally”in
apredictor-corrector
methods.
53
3C
EN
TR
AL
PATH
3.5P
ath-Follow
ing
Let’sdefine
�� ��
� ���
� ���
and
�� ��
�����
� �� �
Given
�
�� � ��
(acentrality
parameter
definingsom
eofthe
propertiesof
thesearch
direction),thereare
many
searchdirections
achieving
� �� �� � �� �� �� �
�
�� ��
��� � �� � �
andfor
�
largeenough
(e.g.,
�� �� ,
�� �� � ,
�� ��
� ,...),
� �� �� � �� ���
staysin
asuitable
neighbourhoodofthe
centralpath.
More
onsearch
directionsfor
SD
Patthe
endofthis
lectureand
atthe
beginningofthe
next...
54
4P
RIM
AL-D
UA
LP
OT
EN
TIA
LF
UN
CT
ION
4P
rimal-D
ualP
oten
tialFu
nctio
n
Given�
�
��
���and�
���
���
apair
ofprimal-dualfeasible
andinterior
pairs,
we
would
liketo
havea
simple
way
ofcomparing
them.
We
havetw
o
criteria:
�
smaller
theduality
gap� �� �
isthe
better,
�
smaller
thedistance
tothe
centralpath(thatis,the
valueof
�� �� �� )
isthe
better.
55
4P
RIM
AL-D
UA
LP
OT
EN
TIA
LF
UN
CT
ION
The
nextfunction,calledthe
primal-dualpotentialfunction,serves
sucha
purposeand
allows
usto
designand
performthe
complexity
analysis
directlyon
it.
�� � �� ��� �
���
�� �� �� �
�� �� �� �
where
�
(we
willtake
�
���
in
ouranalysis).
56
4P
RIM
AL-D
UA
LP
OT
EN
TIA
LF
UN
CT
ION
Th
eorem
4.1S
upposew
ehave
� �� �� �� ��
feasiblefor� �
�
and����
suchthat
�� �� �� �� �� �
��
��
�
forsom
e
��� � �� �
Ifwe
generate� �� ��� �� ���
feasiblein� �
�
and����
suchthat
�� �� �
� ��� �� ��� �
�� �� �
� � ���� �� � �����
��
forevery
� ��
forsom
e
�
anabsolute
constant,thenfor
some
�� ���
��
� �� ��� ,w
ehave
� �� ��� �� �� �
�� �� �� �� � �
forevery
�
�� �57
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
5A
lgo
rithm
and
Co
mp
utatio
nalC
om
plexity
An
alysis
Based
onthe
lasttheorem,our
problemofproducing
anapproxim
ately
optimalsolution
pairis
reducedto
decreasingthe
potentialfunctionvalue
bya
constant,inevery
iteration.W
ew
anttoupdate� �
� ��� �� ���
to
� �� � ��� �� � ���
suchthat� �� �� , �� ������
�
, ��
,
��
areallm
aintained;moreover,� �
� �
isdecreased
and
�� �� ��
isnotincreased
alot(in
comparison
tothe
dualitygap).
We
canexpress
theupdate
from
� �� ��� �� ���
to� �� � ��� �� � ���
bya
pairofsearch
directions ��
, ��� � �
respectivelyand
astep
size
�� �
.W
e
dropthe
iteratenum
bersfor
thispartofour
studyand
define
�� ��� ���
� ���
and �� ��� ���
� �� �
58
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
Tom
aintainthe
feasibilityofthe
iterates,thesearch
directionsm
ustsatisfy
���
�� �
and �� �
� ����
� � �
forsom
e
��
����
;i.e., ��
mustbe
inthe
nullspaceof
����
and ��
mustbe
inthe
rangeof �
���� .
We
willderive
analgorithm
thatissym
metric
between
theprim
alandthe
dual(primal-dualsym
metry
),we
would
alsolike
tohave
ouralgorithm
invariantunderthe
symm
etriesofthe
coneconstrains
(scale-invariance).
Tobe
more
specific,insteadofform
alizingthese
vaguegoals,w
ew
ill
describean
approachw
hichattains
thesegoals.
59
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
For
every� �� ��� � �
�
� �
we
would
liketo
havea
self-adjoint,
positivedefinite
lineartransform
ation
�
�� ���� �
suchthat
��
�
Aut� � � � �
��� �� �
� ��
� �� �
���
��� � �
�� �
� ��
� � ��
� �� �
�
.
Ifwe
canfind
suchtransform
ation�
,thenw
ecan
map
ourprim
al-space
with
them
apping
� ��
andthe
dual-spacew
ith
�
.T
hism
odificationdoes
notchangeanything
significantly,exceptthatourcurrentprim
al-dual
iterateis
mapped
onto� ���� .
Let’selaborate:
60
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
Since
�(and
therefore
� ��
;because,
�
isself-adjoint)
isan
automorphism
ofthecone
ofpositivesem
idefinitem
atrices,them
ost
importantpartofthe
problem(for
thecurrentinterior-pointm
ethod
approach)is
unchanged.W
edefine
�����
���� ������
��� ��� ��
��
����� �
����
�� �
��
�� �����
�� �
61
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
Now
,� ��
and����
become
���
�inf
���
���
�� ��
�
��
�
�� �
�� � � � �� � �
��
��
sup� ��
���� �� ���
���
��
�� � � � �� ��
Inthese
scaledspaces,the
searchdirections
arestillorthogonal:
��
�
mustlie
inthe
nullspaceof
�����
and
��
�
mustlie
inthe
rangeof
� ����� .
62
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
Let’sanalyze
theduality
gap.W
ehave
� �� �� � �� ��
�
� �� � �
�� ����
�� �
��
�� �
Therefore,ifw
etake
as
��
�
and
��
�
theorthogonalprojection
of
�
�
ontothe
nullspaceof
�����and
rangeof
� �����
respectively,thenw
ew
ill
havethe
bestsearchdirection
toreduce
theduality
gapin
thissetting.
Now
,let’sturn
tothe
centralitypart
ofthepotentialfunction.
Whatkind
of
searchdirection
would
improve
thebarrier
functionvalues
inthis
setting?
We
utilizethe
following
technicallemm
aw
hichsum
marizes
many
ofthe
niceproperties
ofthebarrier
function
�
.
63
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
Lem
ma
5.1Let�
� � �
.S
uppose�� � �
satisfies
� �� �� ������� �
��� �
� �� ��
��
Then
�� �� �� �
�� �� � � �
�� ���� �
�� ���� �
�� �� �� �
�� �� � � �
� �� ��
�� �
�
� �� �� �
�
64
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
Rem
ark5.1
The
condition
� �� ���
(� �
)im
plies� ���� �
(
�
).T
hisis
clearfrom
thestatem
entofthelem
ma.
Butitcan
alsobe
directlyobserved
asfollow
s:
� � �� �
�� � �
�� ��� �
�� �
��
�
Therefore,� � �
�� ��� �
�� �
� ���
.B
utthisis
equivalentto
� �� �
�� ��� �
�� �
� .
Ifwe
applythe
automorphism�
�� ����� �
of
� �
toboth
sides,we
obtainequivalently �
��� .
65
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
Let’sfocus
onthe
firstorderupper
estimate
on
�� �� �����
�� �� ���
givenby
theabove
lemm
a.W
eobtain
� � ��� �
� � �� �� �
�� �� � ���
�
�� � ��
��
�� �
��
� �
Thus,as
inour
analysisofthe
dualitygap,ifw
etake
as
��
�
and
��
�
the
orthogonalprojectionof
� ��
ontothe
nullspaceof
�����
andrange
of
�������
respectively,thenw
ew
illhavethe
bestsearchdirection
toreduce
thefirstorder
termin
theupper
boundon
thevalue
ofthebarrier
terms
�� �� �
�� �� ,in
thissetting.
Therefore,to
reducethe
valueofthe
potentialfunction,itseems
desirable
tochoose
am
atrixw
hichis
anonnegative
linearcom
binationof
��
and
� ��
andthen
define
��
�
and
��
�
asthe
orthogonalprojectionsofthis
matrix
ontothe
nullspaceof
�� ��
andthe
rangeof
� �����
respectively.
66
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
This
isprecisely
whatw
edo
next.Let
��
� ��
� �
�
� �� �
��� �
��
Note
that���
��
�
iff
� ��
���
� ��
� �� �
��
.B
utthelatter
leadsto
a
contradiction(that �
�
)upon
takingthe
innerproductw
ith
�
of
bothsides.
Therefore,�
���
�
andw
edefine
�
� �
���
���
�
�In
fact,���
��
isconnected
toa
measure
ofcentrality.R
ecall
��� �� � �
�� � �
�
�� � �
��
� ��
�67
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
Then
��
� ��
�
� �
� �
� �� �
�
��� �
��
�
��
���
� � ���
������� �
Note
that� ���
���
is ���
times
thesquared
normofthe
errorin
the
equation��
� � ��
,where
thenorm
isw
ithrespectto
thelocalm
etric
inducedby
� �
.W
ehave
Co
rollary
5.1F
orevery�
� �� � �
,we
have
���
� ��
�� �
� �
The
equalityholds
aboveiff�
�� � �
�
.
68
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
Then
��
�
,
�� ,
��
�
make
upthe
uniquesolution
ofthesystem
:
��
��
���
� �� �
� ����
��
�
��
� ��
��
�
��
By
definition,
��
��� �� ��
��
�� ��
��
�� ��
���
Therefore,w
eim
mediately
concludethat
��
���
���
and�
��
��
����
Now
,we
analyze
� ��� �
and
� ��� �
.W
ehave
� ��� ��
�
������� �
��� � �
�
69
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
�
� �
��
����� �
��
��
�� � �
�
�
��
���� � � �
���
��
�� � � �
���
�
��� �
��
� �����
��
��� ��
�
��� �
��
� �����
�
�� ��� ��� �
�
Note
thatinthe
abovederivation,w
eencountered
thelinear
operator�� �
�
�� ���
�����
which
happensto
coincide(in
thiscase)
with
thelinear
operator
��� �
�� ��� �
�� �
�� �����70
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
Sim
ilarly,� �
�� ���
��� �
��
� �����
��
��� �
�
�
��� �
��
� �����
�
�� ��� ��� �
�
71
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
We
have,�� ��
�
�� � �
� �
���
� �� �� � �� ��
� �� �
�
����� �
�
�� �
� ��� ��
�� �
�
�� �
�� �� � �
� ��� ��
�� �
�
�� �
�� �� �
�
�� �
�
� �� �
����
�
����� �
�
��
� �
� �� ��� ��� �
�� �
�
�� �� ��� ���� �
�
��
���
�� �
� �
� �� ��� ��� �
� ��
�� �� ��� ���� �
�
�
�
���� ��
��� �
� �� ��� ��� �
� ��
�� �� ��� ���� �
72
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
�
�
�
�� �
�� �
�������� �
�
�
�
�� �
���
�
����
�
73
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
ALG
OR
ITH
M
Given� �
� �� �� ��
feasiblein� �
�
and����
suchthat�
� �
�
,�
� �
�
.A
lsogiven
is
��� � ��
suchthat
�� �� �� �� �� �
��
� �� �� .
���� .
While� �
� ��� �� ��
�� �� �� �� � �
� �� �� �
� ��� �
�� �
� � �� ��� �� ��
� ��
� �� ��� �� �
� �� �
� �� ��� �
�� �
�����
���� ��
��
��
�� � �
��� �
�
�
� ���
� ���
��
����
�
� �� �
� ��� �� ���
��� �
�
�
���
���
��� �
74
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
Solve
thesystem
��
��
���
� �� �
� ����
��
�
��
� ��
��
�
��
Com
pute
�����
���
�� �� �
� ���
��
��
��� �
� ���
�� �
��
��
� ��
��
�
.
Let�� � ��� ��
� ���
���
��
���
�� � ��� ��
� ���
��� �
��
��
� ��
�
����
���
.
end� While�
75
5A
LGO
RIT
HM
AN
DC
OM
PU
TATIO
NA
LC
OM
PLE
XIT
YA
NA
LYS
IS
The
aboveis
whatis
typicallycalled
apotentialreduction
algorithm.
We
provedthe
following
theorem.
Th
eorem
5.1T
heabove
algorithmterm
inatesin
atmost
��
��
� �� ��
iterationsw
ithfeasible �
� ��� �� ��
suchthat
� �� ��� �� �� �
�� �� �� �� � �
Even
thoughthe
algorithmrequires
theiterates
tolie
inthe
interiorofthe
underlyingcone
constraints,we
canrelax
theinitialfeasibility
assumption
byusing
auxiliaryoptim
izationproblem
s.
76
6IN
FE
AS
IBLE
-STA
RT
ALG
OR
ITH
MS
6In
feasible-S
tartAlg
orith
ms
Another
approachis
tow
orkin
thefram
ework
ofthealgorithm
thatwe
discussed(or
some
otherprim
al-dualinterior-pointalgorithm)
butmodify
thesearch
directionsso
thatthesearch
directionsalso
tryelim
inatethe
errorin
thelinear
equationsdefining
theprim
alanddualfeasible
regions.
Insteadofhaving
oursearch
directions��
and��
lyingin
thenullspace
of����
andrange
of ����� ,w
eask
thattheysatisfy
thefollow
ingsystem
ofequations:
���
�� ��
�
� �
� ���
and �� �
� � ��
� ��
�
�� �
� ���
�
�� ���
77
6IN
FE
AS
IBLE
-STA
RT
ALG
OR
ITH
MS
The
analysisbecom
esm
orecom
plicated,however,this
isone
ofthe
popularw
aysto
solveS
DP
problems
inpractise.
The
algorithms
needto
carefullym
onitorthe
progressin
attainingfeasibility,reducing� �
� �
as
wellas
theproxim
ityto
the“centralsurface.”
(Since
we
allowinfeasible
iterates,we
willbe
concernedw
iththe
distanceto
the“centralsurface”
ratherthan
thecentralpath.)
For
instance,thealgorithm
shouldnotallow
thefastreduction
of� �� �
unlessthe
iteratesare
gettingto
benear
feasibleatleastas
fast.
78
7O
TH
ER
INT
ER
IOR
-PO
INT
ALG
OR
ITH
MS
,GE
NE
RA
LR
EM
AR
KS
7O
ther
Interio
r-Po
int
Alg
orith
ms,G
eneralR
emarks
The
searchdirections
thatwe
discussedare
known
asthe
NT
direction(for
Nesterov-Todd).
These
algorithms
havebeen
generalizedto
convex
optimization
problems
overarbitrary
convexcones.
Other
primal-dualalgorithm
sthatare
usefulandpopular
relyon
search
directionsproposed
Helm
berg-Rendl-V
anderbei-Wolkow
icz/Kojim
a-Shindoh-H
ara/Monteiro
(HK
Mdirection)
andA
lizadeh-Haeberly-O
verton(A
HO
direction).A
llthese
directionscan
bedefined
andtreated
ina
unifiedw
ay(due
toY.Z
hang,
some
otherrelated
work
isdue
toM
onteiro-Y.Zhang):
79
7O
TH
ER
INT
ER
IOR
-PO
INT
ALG
OR
ITH
MS
,GE
NE
RA
LR
EM
AR
KS
Let�� � ���
.D
efine
���� ���
��� �
asfollow
s
��� ��
� ���� �
��� ��� �
� ��
This
������is
calledthe
symm
etrizedsim
ilaritytransform
ation.To
compute
thesearch
direction,we
solvethe
system
� �
���
��
� �
� ��� �
�� �
� � ��
�
�
��
�� �
� ���
�
�� ���
��� �
� ���
� ��
� �� ���
��
���
�
��� �
� ���� ���
�
where
�
�� � ��
aparam
eterfixed
bythe
user/algorithmand
� �� �
� ��� �� ���
asbefore.
80
7O
TH
ER
INT
ER
IOR
-PO
INT
ALG
OR
ITH
MS
,GE
NE
RA
LR
EM
AR
KS
Choosing�
� ��
givesthe
AH
Odirection,�
���� �
� ��� �� �
yieldsthe
HK
Mdirection,choosing
any �� � ���
suchthat
� ���
� �� ��� ��� �
� � �� ��� �� ��
� ��� �� ��� �� �
� �� �
� �� ��� ��� �
(forinstance,�
� �� � �
� ��� �� ��
� ��� �� ��� �� �
� ��
�� �� ��� ��� �
)gives
theN
Tdirection.
The
nextlecturestarts
with
adiscussion
ofthecom
putationalissues
relatedto
thesearch
directionsfor
SD
Pw
hichties
innicely
with
thebundle
methods .
81
7O
TH
ER
INT
ER
IOR
-PO
INT
ALG
OR
ITH
MS
,GE
NE
RA
LR
EM
AR
KS
We
canalso
designa
wide
rangeofprim
al-dualalgorithms
withoutthe
conicstructure
orlogarithm
ichom
ogeneity:
Polynom
ialtime
IPM
s
WIT
HO
RW
ITH
OU
T
theC
onicS
tructureand
Logarithmically
Hom
ogeneous
Barriers!
(Froma
recentpaperby
Nem
irovskiandT.)
82
7O
TH
ER
INT
ER
IOR
-PO
INT
ALG
OR
ITH
MS
,GE
NE
RA
LR
EM
AR
KS
We
aregiven
�
a�
-SC
B�
with
adom
ain
��
andthe
Legendre-Fenchelconjugate
��
(with
aslightdifference
fromthe
previousdefn.)
of
�
;thedom
ainof
��
isdenoted
��
.�
� is
acone:
��
��
�
���
�� ��
�
alinear
embedding
��� �
with
thenullspace
�� �
andthe
image
intersecting
��
;
�
avector�� � .
83
7O
TH
ER
INT
ER
IOR
-PO
INT
ALG
OR
ITH
MS
,GE
NE
RA
LR
EM
AR
KS
�
theoptim
izationproblem
��
�
�� ���� ���
� ���
� �� ��
��� �
we
areinterested
insolving;
�
thefunction
�� � �
�� � ��
which
isa�
-SC
Bfor
cl� �� .
84
7O
TH
ER
INT
ER
IOR
-PO
INT
ALG
OR
ITH
MS
,GE
NE
RA
LR
EM
AR
KS
Ashifted
centralpath
Lem
ma
7.1F
or� ,the
“primal-dualpair”� ��� �
� �� �� �� �� ���
is
uniquelydefined
bythe
relations� ��
��
�� ���
� ��
�� �� �
���
� ���
��� �� �� �� �
� ��
�� � ��� �
Moreover,
� �� �� �
argmin
� ���� ��
� �� �� �
���� �
85
7O
TH
ER
INT
ER
IOR
-PO
INT
ALG
OR
ITH
MS
,GE
NE
RA
LR
EM
AR
KS
7.1P
roximity
measure
7.1P
roximity
measu
re
Letusdefine
theproxim
itym
easureas
thefunction
�� ��� �
�� � �� �
��� ��
�
� � � � �
���
�� ��
(Legendre-Fenchelgap
between
�
and
��
).N
oticethatfor
every��
andevery�
���
,we
have�� ���
andfor
sucha
pair� ���
we
have
�� ��� �
iff� ��
�� � �� .
Using
thissetup
many
path-following
andpotential-reduction
algorithms
canbe
derivedand
analyzed.
86
RE
FE
RE
NC
ES
RE
FE
RE
NC
ES
Referen
ces
[1]F.A
lizadeh,“Interior
pointmethods
insem
idefiniteprogram
ming
with
applicationsto
combinatorialoptim
ization,”S
IAM
Journalon
Optim
ization5
(1995)13–51.
[2]F.A
lizadeh,J-P.A.H
aeberly,andM
.L.Overton,
Prim
al-dual
interior-pointmethods
forsem
idefiniteprogram
ming:
Convergence
rates,stabilityand
numericalresults,
SIA
MJ.O
ptim.,8:746–768,
1998.
[3]A
.Ben-Taland
A.N
emirovski,Lectures
onM
odernC
onvex
Optim
ization.Analysis,A
lgorithms,and
Engineering
Applications,
MP
S/S
IAM
Series
onO
ptimization,
Philadelphia,PA
,US
A,2001.
87
RE
FE
RE
NC
ES
RE
FE
RE
NC
ES
[4]C
.B.C
hua,Relating
homogeneous
conesand
positivedefinite
cones
via�
-algebras,SIA
MJ.O
pt.14(2004)
500-506.
[5]J.Farautand
A.K
oranyi,Analysis
onS
ymm
etricC
ones,Oxford
University
Press,N
Y,US
A,1994.
[6]M
.Grotschel,L.Lovasz
andA
.Schrijver,
Geom
etricalgorithm
sand
combinatorialoptim
ization(S
pringer,New
York,1988).
[7]O
.Guler,B
arrierfunctions
ininterior
pointmethods,M
ath.ofOper.
Res.21
(1996),pp.860–885.
[8]O
.Guler,H
yperbolicP
olynomials
andInterior
PointM
ethodsfor
Convex
Program
ming,M
ath.Oper.R
es.22(1997)
350–377.
88
RE
FE
RE
NC
ES
RE
FE
RE
NC
ES
[9]C
.Helm
berg,F.Rendl,R
.Vanderbeiand
H.W
olkowicz,A
n
interior-pointmethod
forsem
idefiniteprogram
ming.S
IAM
J.Optim
.6
(1996)342–361.
[10]R
.A.H
ornand
C.R
.Johnson,Topics
inm
atrixanalysis,
Cam
bridge
University
Press,C
ambridge,1994.
Corrected
reprintofthe1991
original.
[11]Jarre,F.Interior-P
ointMethods
viaS
elf-Concordance
orR
elative
LipschitzC
ondition,Habilitationsschrift,
University
ofWurzburg,July
1994.
[12]N
.Karm
arkar,A
newpolynom
ialtime
algorithmfor
linear
programm
ing.C
ombinatorica
4(1984)
373–395.
89
RE
FE
RE
NC
ES
RE
FE
RE
NC
ES
[13]M
.Kojim
a,M.S
hidaand
S.S
hindoh,S
earchdirections
inthe
SD
P
andthe
monotone
SD
LCP
:generalizationand
inexactcomputation.
Math.P
rogramm
ing,85:51–80,1999.
[14]M
.Kojim
a,S.S
hindohand
S.H
ara,Interior-pointmethods
forthe
monotone
linearcom
plementarity
problemin
symm
etricm
atrices.
SIA
MJ.O
ptim.7
(1997)86–125.
[15]R
.D.C
.Monteiro,F
irstandsecond
orderm
ethodsfor
semidefinite
programm
ing,Math.P
rog.B97
(2003)209–244.
[16]R
.D.C
.Monteiro,P
rimal-dualpath
following
algorithms
for
semidefinite
programm
ing.SIA
MJ.O
ptim.7
(1997)663–678.
90
RE
FE
RE
NC
ES
RE
FE
RE
NC
ES
[17]R
.D.C
.Monteiro
andT.T
suchiya,Polynom
ialconvergenceofa
new
family
ofprimal-dualalgorithm
sfor
semidefinite
programm
ing,SIA
M
J.Optim
.9(1999)
551–577.
[18]A
.Nem
irovskiandL.Tuncel,“C
one-free”prim
al-dualpath-following
andpotentialreduction
polynomialtim
einterior-pointm
ethods,
Research
Report
CO
RR
2002-32,Dept.ofC
ombinatorics
and
Optim
ization,FacultyofM
athematics,U
niversityofW
aterloo,
Canada,O
ctober2002.
[19]Yu.E
.Nesterov
andA
.S.N
emirovskii,
Interior-PointP
olynomial
Algorithm
sin
Convex
Program
ming,
SIA
M,P
hiladelphia,PA,U
SA
,
1994.
91
RE
FE
RE
NC
ES
RE
FE
RE
NC
ES
[20]N
esterov,Yu.E.and
M.J.Todd
(1997).Self-scaled
barriersand
interior-pointmethods
forconvex
programm
ing,Math.O
per.Res.22
1–46.
[21]Yu.E
.Nesterov
andM
.J.Todd,Prim
al-dualinterior-pointmethods
for
self-scaledcones,S
IAM
J.Optim
ization8
(1998)324–364.
[22]J.R
enegar,AM
athematicalV
iewofInterior-P
ointMethods
inC
onvex
Optim
ization,MP
S/S
IAM
Series
onO
ptimization,P
hiladelphia,PA,
US
A,2001.
[23]R
ockafellar,R.T.(1970).C
onvexA
nalysis,Princeton
University
Press,P
rinceton,New
Jersey.
[24]A
.Schrijver,
Theory
ofLinearand
IntegerP
rogramm
ing.John
Wiley
andS
ons,1986.
92
RE
FE
RE
NC
ES
RE
FE
RE
NC
ES
[25]A
.Shapiro,F
irstandsecond
orderanalysis
ofnonlinearsem
idefinite
programs,M
ath.Program
ming
B77
(1997)301–320.
[26]M
.J.Todd,On
searchdirections
ininterior-pointm
ethodsfor
semidefinite
programm
ing,Optim
izationM
ethodsand
Softw
are11
(1999)1–46.
[27]M
.J.Todd,Potential-reduction
methods
inm
athematical
programm
ing,Mathem
aticalProgram
ming
76(1997)
3–45.
[28]L.Tuncel,G
eneralizationofprim
al-dualinterior-pointmethods
to
convexoptim
izationproblem
sin
conicform
,Foundations
of
Com
putationalMathem
atics1
(2001)229–254.93
RE
FE
RE
NC
ES
RE
FE
RE
NC
ES
[29]L.Tuncel,P
otentialreductionand
primal-dualm
ethods,Handbook
of
Sem
idefiniteP
rogramm
ing:T
heory,Algorithm
sand
Applications,H
.
Wolkow
icz,R.S
aigalandL.V
andenberghe(eds.),K
luwer
Academ
ic
Publishers,B
oston,MA
,US
A,2000,pp.235–265.
[30]L.Tunceland
H.W
olkowicz,S
trengthenedexistence
anduniqueness
conditionsfor
searchdirections
insem
idefiniteprogram
ming,
Research
Report
CO
RR
2003-20,Dept.ofC
ombinatorics
and
Optim
ization,FacultyofM
athematics,U
niversityofW
aterloo,Ontario,
Canada,July
2003.
[31]L.V
andenbergheand
S.B
oyd,Aprim
al-dualpotentialreduction
method
forproblem
sinvolving
matrix
inequalities,Mathem
atical
Program
ming
69(1995)
205–236.
94
RE
FE
RE
NC
ES
RE
FE
RE
NC
ES
[32]L.V
andenbergheand
S.B
oyd,“S
emidefinite
Program
ming,”
SIA
M
Review
38(1996)
49–95.
[33]H
.Wolkow
icz,S
ome
applicationsofoptim
izationin
matrix
theory.
LinearA
lgebraand
itsA
pplications,40:101–118,1981.
[34]H
.Wolkow
icz,R.S
aigalandL.V
andenberghe,editors.H
andbookof
Sem
idefiniteP
rogramm
ing:T
heory,Algorithm
s,andA
pplications,
Kluw
erA
cademic
Publishers,B
oston,MA
,2000.xxvi+
654pages.
[35]Y.Ye,A
pproximating
quadraticprogram
ming
with
boundand
quadraticconstraints,M
ath.Program
ming
84(1999)
219–226.
[36]Y.Z
hang,O
nextending
some
primal-dualinterior-pointalgorithm
s
fromlinear
programm
ingto
semidefinite
programm
ing.S
IAM
J.
Optim
.,8:365–386,1998.
95