optimization methods aleksey minin saint-petersburg state university student of acophys master...

Optimization methodsAleksey MininSaint-Petersburg State UniversityStudent of ACOPhys master program (10th semester)

Joint Advanced Students School

19.04.23

Applied and COmputational Physics

What is optimization? 19.04.23Joint Advanced Students School

Content:

19.04.23

Applications of optimization

• Advanced engineering designAdvanced engineering design• BiotechnologyBiotechnology• Data analysisData analysis• Environmental managementEnvironmental management• Financial planningFinancial planning• Process controlProcess control• Scientific modeling etcScientific modeling etc

19.04.23

Global or Local ?

19.04.23Joint Advanced Students School

What is global optimization?

•The objective of global optimization is to find the globally best solution of (possibly nonlinear) models, in the (possible or known) presence of multiple local optima.

Branch and bound 19.04.23Joint Advanced Students School

Branch and boundScientist are ready to carry out some experiments. But the quality of all of the varies depending on type of experiment according to next table:

Type of experiment

Scientist number

1 2 3 4

A 0.9 0.8 0.9 0.85

B 0.7 0.6 0.8 0.7

C 0.85 0.7 0.85 0.8

D 0.75 0.7 0.75 0.7

Branch and bound

RootAAAA0.55

Branch and bound

RootAAAA0.55

AADCC0.42

BBAAA0.42

CCAAA0.52

DDAAA0.45

Branch and bound

RootAAAA0.55

AADCC0.42

BBAAA0.42

CCAAA0.52

DDAAA0.45

BCBAA0.39

DCDAA0.45

ACABD0.38

Branch and bound

RootAAAA0.55

AADCC0.42

BBAAA0.42

CCAAA0.52

DDAAA0.45 D

CDAA0.45

BCBAA0.39

ACABD0.38 A

CBAD0.37

BCDBA0.40

Branch and bound

RootAAAA0.55

AADCC0.42

BBAAA0.42

CCAAA0.52

DDAAA0.45 D

CDAA0.45

BCBAA0.39

ACABD0.38

BCDBA0.40

ACBAD0.37

Branch and bound

Evolutionary algorithms

Simulated annealing 19.04.23Joint Advanced Students School

Apply small

perturbation

Solution found!

If T=0

Repeat until good solution

not found

Simulated annealingresults

Simulated annealing

Tree annealingdeveloped by Bilbro and Snyder [1991]

Swarm intelligence

Tabu Search

Taboo search implementation

Tabu search implementation

Taboo search implementation

Tabu Search

What is Local Optimization?

•The term LOCAL refers both to the fact that only information about the function from the neighborhood of the current approximation is used in updating the approximation as well as that we usually expect such methods to converge to whatever local extremum is closest to the starting approximation.

•Global structure of the objective function is unknown to a local method.

Local optimization

Gradient descent

Therefore we obtained: F(x0)<F(x1)<…<F(xn )

Quasi-Newton Methods

•These methods build up curvature information at eachiteration to formulate a quadratic model problem of the form:

where the Hessian matrix, H, is a positive definite symmetric matrix, c is a constant vector, and b is a constant.•The optimal solution for this problem occurs when the partial derivatives of x go to zero:

Quasi-Newton Methods

BFGS - algorithm 19.04.23Joint Advanced Students School

BFGS - algorithm

Gauss Newton algorithm

Given m functions f1 f2 … fm of n parameters p1 p2 .. Pn (m>n),and we want to minimize the sum:

Gauss Newton algorithm

Levenberg-Marquardt 19.04.23Joint Advanced Students School

This is an iterative procedure. Initial guess for pT = (1,1,…,1).

Levenberg-Marquardt

SQP – constrained minimization

Reformulation

SQP – constrained minimizationThe principal idea is the formulation of a QP sub-problem based on a quadratic approximation of the Lagrangian function:

SQP – constrained minimizationUpdating the Hessian matrix

Neural Net analysis

What is Neuron?

Typical formal neuron makes the elementary operation – weighs values of the inputs with the locally stored weights and makes above their sum nonlinear transformation:

y f u u w w xi ii , 0

u w w xi i 0

neuron makes nonlinear operation above a linear combination of inputs

Neural Net analysis

What is training?

W – set of synaptic W – set of synaptic weightsweightsE (W) – error functionE (W) – error function

What kind of optimization to choose?

19.04.23

Neural Network – any architecture

1 2 3 4

Error back propagation

How to optimize?

Objective function – is an Empirical error (should decay)Parameters to optimize - are weightsConstraints – are equalities (inequalities) for weights if exist

Neural Net analysis and constrained and unconstrained minimization

NB! For unconstrained optimization I applied Levenberg-Marquardt methodFor constrained case I applied SQP method

Thank you for your attention

optimization methods aleksey minin saint-petersburg state university student of acophys master...

joint advanced students

bound root slide

global optimization

optimization global

b cbaa

b baaa

local optimization

c caaa

Documents

eastern operations minin - tronox

valery minin -...

nikita minin porfolio

the beatrice property revelstoke minin divisiog n

dpatnt o n an minin maps publications

alberto di minin social innovation

shevcova minin

visual summary - aleksey k

high performance content hosting aleksey korzun 1

aleksey sechkin. verizon

ivanov aleksey petrovich

syllabus for technical assistant ap minin

aleksey androsov

aleksey mashanov rit

minin drawings

aleksey borsch, 31 webinar

photographer aleksey petrosian: italy

minin v commission - europa

aleksey koshkin

aleksey rehlov, external producer, creative mobile oÜ