System Optimization(1)

Liang Yu

Department of Biological Systems Engineering

Washington State University

04. 16. 2013

Outline

Background of Engineering Optimization

Application of Optimization

Introduction

Fundamentals of Optimization

Optimization Toolbox in Matlab

Optimization in Process Plants

Engineering applications of optimization

Some typical applications from different engineering disciplines

Design of water resources systems for maximum benefit

Design of pumps, turbines, and heat transfer equipment for maximum efficiency

Optimum design of chemical processing equipment and plants

Selection of a site for an industry

Optimum design of control systems

Application: Metabolic Flux AnalysisFlux Balance Analysis (FBA)

in silico simulation

Linear programming (LP)

Genome-scale

13C-assisted Metabolic Flux Analysis

in vivo search

Nonlinear programming (NLP)

Simplified model

maximize ∑ci ∙vi

s.t. S∙v = 0

lb < v < ub

minimize (MDVexp-MDVsim)2

s.t. S∙v = 0

IDV = f(v, IMM, IDV)

MDV = M∙IDV

lb < v < ub

Metabolic Steady state

Metabolic & isotopic Steady stateIDV: isotopomer distribution vectorMDV: mass distribution vector

Application for optimization of biorefinery configurations

Pham, Viet, and Mahmoud El‐Halwagi. "Process synthesis and optimization of biorefinery configurations." AIChE Journal 58.4 (2012): 1212-1221.

Part of the branching trees for the production of bio-alcohols from lignocellulosic bio-mass

Optimization Tree

IntroductionDefinition

Optimization is the act of obtaining the best result under given circumstances. It can be defined as the process of finding the conditions that give the maximum or minimum value of a function.

Goaleither to minimize the effort required or to maximize the desired benefit.

Optimization problem could be linear or non-linear.

Non –linear optimization is accomplished by numerical ‘Search Methods’.

Search methods are used iteratively before a solution is achieved.

The search procedure is termed as algorithm.

Introduction

Minimum of f (x) is same as maximum of −f (x)

Optimum solution is found while satisfying its constraint (derivative must be zero at optimum).

Introduction

Linear problem – solved by Simplex or Graphical methods.

The solution of the linear problem lies on boundaries of the feasible region.

Non-linear problem solution lies within and on the boundaries of the feasible region.

Solution of linear problem Three dimensional solution of non-linear problem

Introduction

Optimization Programming LanguagesGAMS - General Algebraic Modeling System

LINDO - Widely used in business applications

AMPL - A Mathematical Programming Language

Others: MPL, ILOG

Software with Optimization CapabilitiesExcel – SolverMATLABMathCADMathematicaMapleOthers

Statement of an optimization problem

An optimization or a mathematical programming problem can be stated

x1x2...xn

which minimizes f (X)Find X =

Subject to the constraints

gj (X) ≤ 0, j = 1, 2, . . . ,mlj (X) = 0, j = 1, 2, . . . , p

Fundamentals of Optimization

Single Objective function f(x)Maximization

Minimization

Design Variables, xi , i=0,1,2,3…..

ConstraintsInequality

Equality

Optimal pointsLocal minima/maxima points: A point or Solution x* is at local point if there is no other x in its Neighborhood less than x*

Global minima/maxima points: A point or Solution x** is at global point if there is no other x in entire search space less than x**

Example of design variables and constraints used in optimization.

Maximize X1 + 1.5 X2 Subject to:X1 + X2 ≤ 1500.25 X1 + 0.5 X2 ≤ 50X1 ≥ 50X2 ≥ 25X1 ≥0, X2 ≥0

Fundamentals of Optimization

Global versus local optimization. Local point is equal to global point if the function is convex.

Fundamentals of Optimization

Function f is convex if f(Xa) is less than value of the corresponding point joining f(X1) and f(X2).

Convexity condition – Hessian 2nd order derivative) matrix of function f must be positive semi definite ( Eigen values +ve or zero).

Convex and nonconvex set Convex function

Mathematical Background

Slop or gradient of the objective function f – represent the direction in which the function will decrease/increase most rapidly

Taylor series expansion

Jacobian – matrix of gradient of f with respect to several variables

x

f

x

xfxxf

dx

dfxx

00lim

)()(lim

.......)(!2

1)()( 2

2

2

xdx

fdx

dx

dfxxf

pp xxp

z

g

y

g

x

gz

f

y

f

x

f

J

Mathematical Background

Slope -First order Condition (FOC) – Provides function’s slope information

Hessian – Second derivative of function of several variables, Sign indicates max.(+ve) or min.(-ve)

Second order condition (SOC) Eigen values of H(X*) are all positive

Determinants of all lower order of H(X*) are +ve

2

22

2

2

2

y

f

yx

fxy

f

x

f

H

0*)( Xf

Optimization Algorithm

Deterministic - specific rules to move from one iteration to next , gradient, Hessian

Stochastic – probalistic rules are used for subsequent iteration

Optimal Design – Engineering Design based on optimization algorithm

Lagrangian method – sum of objective function and linear combination of the constraints.

Optimization Methods

DeterministicDirect Search – Use Objective function values to locate minimum

Gradient Based – first or second order of objective function.

Minimization objective function f(x) is used with –ve sign – f(x) for maximization problem.

Single VariableNewton – Raphson is Gradient based technique (FOC)

Golden Search – step size reducing iterative method

Multivariable Techniques ( Make use of Single variable Techniques specially Golden Section)

Unconstrained OptimizationPowell Method – Quadratic (degree 2) objective function polynomial is non-gradient based.

Gradient Based – Steepest Descent (FOC) or Least Square minimum (LMS)

Hessian Based -Conjugate Gradient (FOC) and BFGS (SOC)

Optimization Methods - ConstrainedConstrained Optimization

Indirect approach – by transforming into unconstrained problem.

Exterior Penalty Function (EPF) and Augmented Lagrange Multiplier

Direct Method Sequential Linear Programming (SLP), Sequential Quadratic Programming (SQP) and steepest Generalized Reduced Gradient Method (GRG)

Descent Gradient or LMS

Advanced Optimization Methods

Global Optimization – Stochastic techniques

Simulated Annealing (SA) method – minimum energy principle of cooling metal crystalline structure

Genetic Algorithm (GA) – Survival of the fittest principle based upon evolutionary theory

Optimization Toolbox in Matlab

Key featuresInteractive tools for defining and solving optimization problems and monitoring solution progress

Solvers for nonlinear and multiobjective optimization

Solvers for nonlinear least squares, data fitting, and nonlinear equations

Methods for solving quadratic and linear programming problems

Methods for solving binary integer programming problems

Parallel computing support in selected constrained nonlinear solvers

How to use Optimization Toolbox

Optimization FunctionsFunction files can be directly provided by M File

Syntax: [x,fval] = fminsearch(fun,x0)

Optimization Tool graphical user interface (GUI)Define and modify problems quickly

Use the correct syntax for optimization functions

Import and export from the MATLAB workspace

Generate code containing your configuration for a solver and options

Change parameters of an optimization during the execution of certain Global Optimization Toolbox functions

Function OptimizationOptimization concerns the minimization or maximization of functions

Standard Optimization Problem:

~

~minxf x

~

0jg x

~

0ih x

L Uk k kx x x

Equality ConstraintsSubject to:

Inequality Constraints

Side Constraints

~

f x is the objective function, which measure and evaluate the performance of a system. In a standard problem, we are minimizing the function. For maximization, it is equivalent to minimization of the –ve of the objective function.

Where:

~x is a column vector of design variables, which can

affect the performance of the system.

Function Optimization

Constraints – Limitation to the design space. Can be linear or nonlinear, explicit or implicit functions

~

0ih x

L Uk k kx x x

Equality Constraints

Inequality Constraints

Side Constraints

~

0jg x

Most algorithm require less than

Optimization Toolbox SolversMinimizers

This group of solvers attempts to find a local minimum of the objective function near a starting point x0. They address problems of unconstrained optimization, linear programming, quadratic programming, and general nonlinear programming.

Multiobjective minimizersThis group of solvers attempts to either minimize the maximum value of a set of functions (fminimax), or to find a location where a collection of functions is below some prespecified values (fgoalattain).

Equation solversThis group of solvers attempts to find a solution to a scalar- or vector-valued nonlinear equation f(x) = 0 near a starting point x0. Equation-solving can be considered a form of optimization because it is equivalent to finding the minimum norm of f(x) near x0.

Least-Squares (curve-fitting) solversThis group of solvers attempts to minimize a sum of squares. This type of problem frequently arises in fitting a model to data. The solvers address problems of finding nonnegative solutions, bounded or linearly constrained solutions, and fitting parameterized nonlinear models to data.

Objective Function

Linear

Quadratic

Sum-of-squares (Least squares)

Smooth nonlinear

Nonsmooth

Constraint Type

None (unconstrained)

Bound

Linear (including bound)

General smooth

Discrete (integer)

Select Solvers by Objective and Constraint

Constraint Type

Objective TypeLinear Quadratic Least Squares Smooth nonlinear Nonsmooth

None n/a (f = const, or min = )

quadprog, Theory, Examples

\, lsqcurvefit, lsqnonlin, Theory, Examples

fminsearch, fminunc, Theory, Examples

fminsearch, *

Bound linprog, Theory, Examples

quadprog, Theory, Examples

lsqcurvefit, lsqlin, lsqnonlin, lsqnonneg, Theory, Examples

fminbnd, fmincon, fseminf, Theory, Examples

*

Linear linprog, Theory, Examples

quadprog, Theory, Examples

lsqlin, Theory, Examples

fmincon, fseminf, Theory, Examples

*

General smooth fmincon, Theory, Examples

fmincon, Theory, Examples

fmincon, Theory, Examples

fmincon, fseminf, Theory, Examples

*

Discrete bintprog, Theory, Example

Minimization Algorithm

Minimization Algorithm (Cont.)

Equation Solving Algorithms

Least-Squares Algorithms

Implementing Optimization Toolbox

Most of these optimization routines require the definition of an M-file containing the function, f, to be minimized.

Maximization is achieved by supplying the routines with –f.

Optimization options passed to the routines change optimization parameters.

Default optimization parameters can be changed through an options structure.

Unconstrained Minimization

Consider the problem of finding a set of values [x1 x2]T that solves

Steps:Create an M-file that returns the function value (Objective Function). Call it objfun.m

Then, invoke the unconstrained minimization routine. Use fminunc

1

~

2 21 2 1 2 2

~min 4 2 4 2 1x

xf x e x x x x x

1 2~

Tx x x

Step 1 – Objective Function

function f = objfun(x)

f=exp(x(1))*(4*x(1)^2+2*x(2)^2+4*x(1)*x(2)+2*x(2)+1);

1 2~

Tx x x

Objective function

Step 2 – Invoke Routine

x0 = [-1,1];

options = optimset(‘LargeScale’,’off’);

[xmin,feval,exitflag,output]=

fminunc(‘objfun’,x0,options);

Output arguments Input arguments

Starting with a guess

Optimization parameters settings

Step 3 – Results

xmin =

0.5000 -1.0000

feval =

1.3028e-010

exitflag =

1

output =

iterations: 7

funcCount: 40

stepsize: 1

firstorderopt: 8.1998e-004

algorithm: 'medium-scale: Quasi-Newton line search'

Minimum point of design variables

Objective function value

Exitflag tells if the algorithm is converged.If exitflag > 0, then local minimum is found

Some other information

More on fminunc – Input

[xmin,feval,exitflag,output,grad,hessian]=

fminunc(fun,x0,options,P1,P2,…)

fun : Return a function of objective function.

x0 : Starts with an initial guess. The guess must be a vector

of size of number of design variables.

Option : To set some of the optimization parameters. (More after few slides)

P1,P2,… : To pass additional parameters.

More on fminunc – Output

[xmin,feval,exitflag,output,grad,hessian]=

fminunc(fun,x0,options,P1,P2,…)

• xmin: Vector of the minimum point (optimal point). The size is the number of design variables.

• feval: The objective function value of at the optimal point.

• exitflag: A value shows whether the optimization routine is terminated successfully. (converged if >0)

• Output: This structure gives more details about the optimization

• grad : The gradient value at the optimal point.

• hessian: The hessian value of at the optimal point

Next Class

Please take your laptop and install Matlab