Chapter 6 Method of Successive Quadratic ProgrammingProfessor Shi-Shang Jang

National Tsing-Hua University

Chemical Engineering Department

6-1 Quadratic Programming Problems

Problem:

0

..

'2

1)(min

:formmatrix compact In

0

..

11

11111

222

2222

11211221112211

x

bAxts

Qxxcxxf

x

bxaxa

bxaxats

xhxxhxh

xxhxxhxhxcxcxcMin

MNMNM

NN

NNNNN

NNNN

Classical Constrained Regression Problem

Where β is an m×1 vector of regression coefficients to be estimated, e is an n×1 vector of error variables, Y is an n×1 observations on the dependent variable and X is an n×m matrix of observations on the independent variables. It is clear that the classical regression problem is a quadratic programming problem.

exYts

Iee

..

'min

Example- A numerical problem

211..

42

2406min

formmatrix compact

0,

2 ..

2226)(

2

1

2

121

2

1

21

21

2221

211

x

xts

x

xxx

x

xxf

xx

xxts

xxxxxxf

Example- A numerical problem

2

11

42

24

06

b

A

Q

c

Example-A scientist’s model A scientist has observed a certain quantity Q

as a function of t. She has a good reason to believe that there is a physical law relating t and Q that takes the form:

Q(t)=asint+bcost+c

She wants to have the “best” possible idea of the value of the coefficients a,b and c using the results of her n experiments:

Example-A scientist’s model (t1,q1; t2,q2;…, tn,qn). Taking into account

that her experiments are not perfect and also perhaps that the model is not rigorous enough, she does not expect to find a perfect fit. So defines an error term:

Ei=qi-q(ti)

Her idea is to minimize:

n

iieZ

1

2min

Example-A scientist’s modelWhat if a=3;b=2;c=5 with 15 experimental data?

0cossin

0cossin

0cossin..

),,,,,(min

2222

1111

1

21

nnnn

n

iin

ectbtaq

ectbtaq

ectbtaqts

eeecbaf

Classical Least Square Fitting Problem: Given a set of data (x1,y1; x2,y2;…,

xn,yn). Find a set of parameters ={a, b, c,…} such that:

ei2=yi-(axi+bxi

2+..) And iei

2 is minimized The solution of the unconstrained case: =(XTX)-1XTY=quasi_inverse(X)Y

Example: Curve Fitting>> x=linspace(0,5);>> for i=1:100y(i)=3*x(i)^2+2*x(i)+5;y(i)=y(i)+randn(1,1);xx(i,1)=x(i)*x(i);xx(i,2)=x(i);xx(i,3)=1;end>> theta=inv(xx'*xx)*xx'*y'

theta =

2.9983 2.1215 4.5839

>> for i=1:100yy(i)=theta(1)*x(i)^2+theta(2)*x(i)+theta(3);end>> plot(x,yy)

Example: Curve Fitting-Continued

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

10

20

30

40

50

60

70

80

90

100

Program QUADPROG QUADPROG Quadratic programming. X=QUADPROG(H,f,A,b) solves the quadratic programming problem: min 0.5*x'*H*x + f'*x subject to: A*x <= b x X=QUADPROG(H,f,A,b,Aeq,beq) solves the problem above while

additionally satisfying the equality constraints Aeq*x = beq. X=QUADPROG(H,f,A,b,Aeq,beq,LB,UB) defines a set of lower and upper bounds on the design variables, X, so that the solution is in the range LB <= X <= UB. Use empty matrices for LB and UB if no bounds exist. Set LB(i) = -Inf if X(i) is unbounded below; set UB(i) = Inf if X(i) is unbounded above.

Curve Fitting Using Quadprogx=linspace(0,5);for i=1:100 y(i)=3*x(i)^2+2*x(i)+5; y(i)=y(i)+randn(1); xx(i,1)=x(i)*x(i);xx(i,2)=x(i);xx(i,3)=1;enda=eye(100);a1=zeros(1,100);H=[a1; a1; a1; a];f=zeros(103,1);HH=[f f f H];Aeq=[xx eye(100)];beq=y';A=[];b=[];X=QUADPROG(HH,f,A,b,Aeq,beq) ;XX=X';XX(1)XX(2)XX(3)

The Solution Method-Complementary Pivot Problem K-T Condition for a QP:

0

0

0

0

u

ux

x

bAx

vAuQQxc TT

The Solution Method-Complementary Pivot Problem Find vectors w and z such that

0

0

0

zw

z

w

qMzw

T

The Solution Method-Complementary Pivot Problem The solution of the complementary pivot

problem is equivalent to find a K-T point of the original QP problem and hence the solution is sufficiently found since the QP is a convex objected (quadratic function) and qualified constrained (linear constraints).

6-2 The Successive QP Approach Consider a quadratic function:

The objective is to find d=(x-x0) such that the following problem can be minimized

002

00000 2

1; xxxfxxxxxfxfxxq T

0

0..

2

1min 2

dxgxg

dxhxhts

dxfddxf

Ttj

tj

Ttk

tk

tTTt

Example

0211

021..

244/25

4/258/3

2

1

4

47

4

23min

:found becan QP ingcorrespond The

12

01)(

02)(..

6)(min

0

21

21

212

121

d

dts

ddd

x

xxxg

xxxhts

xxxxxf

T

Example

Algorithm Step 1: Formulate the QP problem Step 2: Solve the QP problem, and set

x(t+1)=x(t)+*d Step 3: Check Convergency

Remarks Original SQP is subjected to the following

two problems:(1) The second derivative of the objective function is generally difficult to obtain.

(2) There is no guarantee that the second derivative of any function at any point is positive definite. In case the hessian is not positive definite, then the QP is failed.

6-3 Quadratic Approximation of the Lagrangian Function

0

0..

,,2

1min

:by edapproximat becan

0)(

0)(..

)(min

)()()()(

2

dxgxg

dxhxhts

dvuxLddxf

xg

xhts

xf

xugxvhxfxL

T

T

xTT

Algorithm-The Variable Metric Method Given initial estimates x0,u0,v0, and a symmetric

positive definitive matrix H0

Step 1: Solve the problem:

Step 2: Select the step size α along d(t), and set x(t+1)=x(t)+ α d(t)

0

0..

2

1min

dxgxg

dxhxhts

dHddxf

Ttt

Ttt

tTTt

Algorithm-The Variable Metric Method (VMCON) Step 3: Check convergence Step 4: Update H(t) using the gradient difference BSF update formulation:

wz

ww

zHz

HzzHHH

Finally

zHyw

then

yzzHz

zHzzHzyz

Define

vuxLvuxLy

xxz

T

T

tT

tTttt

t

TtT

tT

tTT

tttx

tttx

tt

1

111

1

1

otherwise 0.8

2.0 if 1

,,,,

Line Search on the SQP direction Given d from QP, instead of using the original

objective function, it is more useful to implement the following penalty function:

tj

tj

tj

tj

tk

tk

tk

tk

jjkk

J

jjj

K

kkk

J

jj

K

kk

uu

vv

uv

xgxhxf

xgxhRxfRxP

1

1

11

11

2

1,max

2

1,max

,iterationt subsequent allfor and

,

iterationfirst for the where

,0min()(

,0min()(),(

Numerical Example

T

T

TTT

d

d

dts

ddd

IH

ghfx

xxxg

xxxhts

xxxxxf

2 ,4

:givessolution The

0211

021..

10

01

2

1

4

47

4

23min

choosingby form the takerun willfirst The

1 1;21;4

47

4

23;12

01)(

02)(..

6)(min

0

1

0

21

21

212

121

MATLAB Program - VMCON% DEFINE GLOBAL VARIABLESglobal muglobal sigmaglobal vvglobal uuglobal ddglobal X0%% INITIALIZE THE PROBLEMH=eye(2);X0=[2;1];uu=0;vv=0;dx=100;mu=0;sigma=0;iter=0;%% THE WHILE LOOP%while abs(dx)>0.000001 iter=iter+1; % % FIND THE GRADIENT df % [df,dlgn]=sqp_Lagrangian(X0); % % DEFINE Aq,beq such that Aq*d=beq % A, b such that Ad<b

MATLAB Program – VMCON-Continued A=[-1 -1];b=(X0(1)+X0(2)-1); Aeq=[X0(2) X0(1)]; beq=(X0(1)*X0(2)-2);% % SOLVE THE SUB-QUADRATIC PROBLEM FOR d % [ss,FVAL,EXITFLAG,OUTPUT,LAMBDA]=QUADPROG(H,df,A,b,Aeq,beq); dd=ss'; % % GET THE LARAGNGE=MULTIPLIERS % uu=LAMBDA.ineqlin; vv=LAMBDA.eqlin; % % PREPARE THE LINE SEARCH % mu=max(abs(vv),0.5*(mu+abs(vv))); sigma=max(abs(uu),0.5*(sigma+abs(uu))); a0=0.1; % % LINE SEARCH ALONG dd WITH THE PENALTY FUNCTION AS THE OBJ % alopt = FMINSEARCH('sqp_penalty',a0);

MATLAB Program – VMCON-Continued% % GET THE NEW POINT % X=X0+alopt*dd'; % % PREPARE TO UPDATE THE APPROXIMATE HESSIAN %z=X-X0; [df,dlgn_1]=sqp_Lagrangian(X0); [df,dlgn_2]=sqp_Lagrangian(X); y=dlgn_2-dlgn_1; zz=z'*H*z; zzp=0.2*zz; zy=z'*y; if(zy>zzp) theta=1; else theta=0.8*z'*H*z/(zz-zy); end w=theta*y+(1-theta)*H*z; % % UPDATE THE APPROXIMATE HESSIAN % H=H-(H*z*z'*H)/zz+w*w'/(z'*w);

MATLAB Program – VMCON-Continued

% % CALCULATE THE OBJ AND EQUALITY CONSTRAINT AT THE NEW POINT % hh=X(1)*X(2)-2; ddx=X-X0; obj=sqp_obj(X); % % UPDATE THE NEW POINT % X0=X; % % CHECK THE CONVERGENCY % dx=sqrt(ddx(1)^2+ddx(2)^2);end%% OUTPUT THE RESULT%iterXobjhh

MATLAB Program Using Fmincon> help fmincon

FMINCON Finds a constrained minimum of a function of several variables.

FMINCON solves problems of the form: min F(X) subject to: A*X <= B, Aeq*X = Beq (linear

constraints) X C(X) <= 0, Ceq(X) = 0 (nonlinear

constraints) LB <= X <= UB

MATLAB Program Using Fmincon-Continued

X=FMINCON(FUN,X0,A,B,Aeq,Beq,LB,UB,NONLCON) subjects the minimization to the

constraints defined in NONLCON. The function NONLCON accepts X and returns

the vectors C and Ceq, representing the nonlinear inequalities and equalities

respectively. FMINCON minimizes FUN such that C(X)<=0 and Ceq(X)=0.

MATLAB Program Using Fmincon-ExampleX0=[2;1];A=[-1 -1];B=1;Aeq=[];Beq=[];LB=[];UB=[];X=FMINCON('sqp',X0,A,B,Aeq,Beq,LB,UB,'sqp_nlcon')

function [c,ceq]=sqp_nlcon(x)c=[];ceq=x(1)*x(2)-2;