1

GLOBAL OPTIMIZATION WITH BRANCH-AND-REDUCE

Nick SahinidisDepartment of Chemical Engineering

Carnegie Mellon Universitysahinidis@cmu.edu

EWO seminar, 23 October 2007

2

THE MULTIPLE-MINIMA DIFFICULTY IN OPTIMIZATION

• Classical optimality conditions are necessary but not sufficient

• Classical optimization provides the local minimum “closest” to the starting point used

f

x

3

COMMON FUNCTIONS IN MODELING

4

COMMON FUNCTIONS IN MODELING

5

AUTOMOTIVE REFRIGERANT DESIGN(Joback and Stephanopoulos, 1990)

• Higher enthalpy of vaporization (ΔHve) reduces the amount of refrigerant

• Lower liquid heat capacity (Cpla) reduces amount of vapor generated in expansion valve

• Maximize ΔHve/ Cpla, subject to: ΔHve ≥ 18.4, Cpla ≤ 32.2

6

FUNCTIONAL GROUPS CONSIDERED

7

PROPERTY PREDICTION

8

BRANCH-AND-BOUND

9

MOLECULAR DESIGNAFTER 150 CPU HOURS IN 1995

• One feasible solution identified• Optimality not proved

• First attempt:– IBM RS/6000 43P with 128 MB RAM

• Second attempt:– IBM SP/2 Single Processor with 2 GB RAM

10

MOLECULAR DESIGN IN 2000

In 30 CPU minutes

11

BREAST CANCER DIAGNOSIS• 200,000 cases diagnosed in the U.S. a year• 40,000 deaths a year

• Most breast cancers are first diagnosed by the patient as a lump in the breast

• Majority of breast lumps are benign

• Available diagnosis methods:– Mammography (68% to 79% correct)– Surgical biopsy (100% correct but invasive and costly)– Fine needle aspirate (FNA)

» With visual inspection: 65% to 98% correct» Automated diagnosis: 95% correct

• Linear programming techniques• Mangasarian and Wolberg in 1990s

12

WISCONSIN DIAGNOSTIC BREAST CANCER (WDBC) DATABASE

• 653 patients• 9 cytological characteristics:

– Clump thickness– Uniformity of cell size– Uniformity of cell shape– Marginal adhesion– Single epithelial cell size– Bare nuclei– Bland chromatin– Normal nucleoli– Mitoses

• Biopsy classified these 653 patients in two classes:– Benign– Malignant

From Wolberg, Street, & Mangasarian, 1993

13

BILINEAR (IN-)SEPARABILITYOF TWO SETS IN Rn

Requires the solution of three nonconvex bilinear programs

14

CHALLENGES INGLOBAL OPTIMIZATION

pn ZyRxyxgyxf

∈∈≤

,0),(s.t.

),(min

NP-HARD PROBLEMMultimodal objective

),( yxf

Integrality conditions

),( yxf

Nonconvex constraints

),( yxf

15

GLOBAL OPTIMIZATION ALGORITHMS• Stochastic and deterministic

algorithms• Branch-and-Bound

– Bound problem over successively refined partitions

» Falk and Soland, 1969» McCormick, 1976

• Convexification– Outer-approximate with

increasingly tighter convex programs

– Tuy, 1964– Sherali and Adams, 1994

• Horst and Tuy, Global Optimization: Deterministic Approaches, 1996

– Over 800 citations

• Our approach– Branch-and-Reduce

» Ryoo and Sahinidis, 1995, 1996

» Shectman and Sahinidis, 1998

– Constraint Propagation & Duality-Based Reduction

» Ryoo and Sahinidis, 1995, 1996

» Tawarmalani and Sahinidis, 2002

– Convexification» Tawarmalani and Sahinidis,

2001, 2002, 2004, 2005• Tawarmalani and Sahinidis,

Convexification and Global Optimization in Continuous and Mixed-Integer Nonlinear Programming, 2002

16

BOUNDING SEPARABLE PROGRAMS

1004060

8s.t.min

3

2

1

213

22

21

23

21

≤≤≤≤≤≤+=

≤−−−−=

xxx

xxxxxxxxf

17

BOUNDING SEPARABLE PROGRAMS

1004060

8s.t.min

3

2

1

213

22

21

23

21

≤≤≤≤≤≤+=

≤−−−−=

xxx

xxxxxxxxf

18

BOUNDING FACTORABLE PROGRAMS

Introduce variables for intermediate quantities whose envelopes are not known

19

TIGHT RELAXATIONS

Convex/concave envelopes often finitely generated

x x

Concave over-estimator

Convex under-estimator

)(xf )(xf Concave envelope

Convex envelope

)(xf

x

20

RATIO: TRADITIONAL RELAXATION

21

RATIO: THE GENERATING SET

22

DIFFERENCE BETWEEN ENVELOPE AND TRADITIONAL RELAXATION

Traditional

23

ENVELOPES OF MULTILINEAR FUNCTIONS

• Multilinear function over a box

• Generating set

• Polyhedral convex encloser follows trivially from polyhedral representation theorems

niUxLxaxxM iii

p

ii

ttn

t

,,1 , ,),...,(1

1 K=+∞<≤≤<∞−= ∏∑=

⎟⎟⎠

⎞⎜⎝

⎛∏=

],[vert1

ii

n

iUL

24

• Local NLP solvers essential for local search• Linear programs can be solved very efficiently• Outer-approximate convex relaxation by polyhedron

Tawarmalani and Sahinidis (Math. Progr., 2004, 2005)

POLYHEDRAL OUTER-APPROXIMATION

• Quadratically convergent sandwich algorithm• Cutting planes for functional compositions

25

RECURSIVE FUNCTIONAL COMPOSITIONS

• Consider h=g(f), where – g and f are multivariate convex functions – g is non-decreasing in the range of each nonlinear

component of f

• h is convex• Two outer approximations of the composite

function h:– S1: a single-step procedure that constructs supporting

hyperplanes of h at a predetermined number of points– S2: a two-step procedure that constructs supporting

hyperplanes for g and f at corresponding points

• Two-step is sharper than one-step– If f is affine, S2=S1– In general, the inclusion is strict

26

OUTER APPROXIMATION OF x2+y2

+

27

MARGINALS-BASEDRANGE REDUCTION

If a variable goes to its upper bound at the relaxed problem solution, this variable’s lower bound can be improved

Relaxed Value Function

z

xxUxL

UL

28

f.

c.

e.

b.a.

d.

REDUCTION VIACONSTRAINT PROPAGATION

29

FINITE VERSUS CONVERGENT BRANCH-AND-BOUND ALGORITHMS

Finite sequences

A potentially infinite sequence

30

FINITE BRANCHING RULE

• Variable selection:– Typically, select variable with largest underestimating gap– Occasionally, select variable corresponding to largest edge

• Point selection:– Typically, at the midpoint (exhaustiveness)– When possible, at the best currently known solution

• Finite isolation of global optimum• Finite termination in many cases

– Concave minimization over polytopes– 2-Stage stochastic integer programming

x*xx∗

f(x)

31

BRANCH-AND-REDUCE

STOP

START

Multistart search and reduction

Nodes? N

YSelect Node

Lower Bound

Inferior? DeleteNode

Y

N

Preprocess

Upper Bound

Postprocess

Reduced?N

Y

Branch

Feasibility-based reduction

Optimality-based reduction

32

Branch-And-Reduce Optimization NavigatorComponents• Modeling language• Preprocessor• Data organizer• I/O handler• Range reduction• Solver links• Interval arithmetic• Sparse matrix routines• Automatic differentiator• IEEE exception handler• Debugging facilities

Capabilities• Core module

– Application-independent– Expandable

• Fully automated MINLP solver

• Application modules– Multiplicative programs– Indefinite QPs– Fixed-charge programs– Mixed-integer SDPs– …

• Solve relaxations using– CPLEX, MINOS, SNOPT,

OSL, SDPA, …

• Available under GAMS and AIMMS• Available on NEOS server

33

26 PROBLEMS FROM globallib AND minlplib

93676CPU hrs

9813,772622,339Nodes in memory

99253,75423,031,434Nodes

% reductionWith cutsWithout cuts

634320Discrete variables

11510304Variables765132Constraints

AverageMaximumMinimum

EFFECT OF CUTTING PLANES

34

POOLING PROBLEM: p-FORMULATION

35

POOLING PROBLEM: q-FORMULATION

36

POOLING PROBLEM: pq-FORMULATION

37

PRODUCT DISAGGREGATION

Consider the function:

Let

Then

∑∑==

+++=n

kkk

n

kkkn ybxbxyaayyx

10

101 ),,;( Kφ

],[],[1

Uk

Lk

n

k

UL yyxxH Π=

×=

∑

∑

=×

=

+++=

n

kkkxxyy

n

kkkH

xyb

bxyaa

ULUk

Lk

1],[],[

01

0

)( convenv

convenv φ

Disaggregated formulations are tighter

38

LOCAL SEARCH WITH CONOPT

-4330.78Infeasiblert970-750haverly30-400haverly20-400haverly1

-6.5-7foulds5-6.5-6foulds4-6.5-6.5foulds3-600-1000foulds2

-2700-2900bental500bental4

-470.83-470.83adhya4-57.74-65adhya3

00adhya2-56.67-68.74adhya1

pq-formulationobjective

q-formulationobjective

Problem

39

GLOBAL SEARCH WITH BARON

0.561745629rt970103haverly301017haverly201025haverly11-1>1200>389foulds51-1>1200>326foulds45-1>1200>348foulds30-1161061foulds20-1>1200>6445bental5

0.510.5101bental411>1200>6129adhya4

1.531>1200>9248adhya30.51720501adhya20.52417573adhya1

CPU secNodesCPU secNodes

pq-formulationp-formulationProblem

40

ONGOING DEVELOPMENT OF BARONStructural

BioinformaticsSystems biology

X-ray imagingU

E(r)

Portfolio optimization

41

BARON IN APPLICATIONS

• Development of new Runge-Kutta methodsfor partial differential equations

– Ruuth and Spiteri, SIAM J. Numerical Analysis, 2004

• Energy policy making– Manne and Barreto, Energy Economics, 2004

• Design of metabolic pathways– Grossmann, Domach and others, Computers & Chemical

Engineering, 2005

• Model estimation for automatic control– Bemporand and Ljung, Automatica, 2004

• Agricultural economics– Cabrini et al., Manufacturing and Service Operations

Management, 2005

42

GLOBAL/MINLP SOFTWARE

• AlphaECP—Exploits pseudoconvexity• BARON—Branch-And-Reduce• BONMIN—Integer programming technology (CMU/IBM)• DICOPT—Decomposition• GlobSol—Interval arithmetic• Interval Solver (Frontline)—Interval solver; Excel• LaGO—Lagrangian relaxations (COIN/OR)• LGO—Stochastic search; black-box optimization• LINGO—Trigonometric functions; IF-THEN-ELSE; …• MSNLP, OQNLP—Stochastic search• SBB—Simple branch-and-bound

NLP/MINLP NLP MINLP

43

COMPARISONS ON MINLPLIB

BARON

44

GAMS SALESCommercial and academic users

0%10%20%30%40%50%60%70%80%90%

100%

1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

Global (BARON, LGO,MSNLP, OQNLP)

MINLP (DICOPT, SBB)

Local NLP (CONOPT,KNITRO, MINOS, PATH,SNOPT)

Data courtesy of Alex Meeraus

45

x*

Convexification

Range Reduction

Finiteness

BRANCH-AND-REDUCE

Engineeringdesign

Managementand Finance

Chem-, Bio-,

Medical Informatics