A First Course inLinear Optimization

— a dynamic book —

by

Jon Lee

Third Edition (Version 3.00)

ReEx PrEsS

Jon Lee

2013–2017

ReEx PrEsS

iii

This work is licensed under the

Creative Commons Attribution 3.0 Unported License

(CC BY 3.0) cb.

To view a copy of this license, visit

��������������� �����������������������

where you will see the summary information below and can click through to the fulllicense information.

Go Forward

This is a book on linear optimization, written in LATEX. I started it, aiming it at thecourse IOE 510, a masters-level course at the University of Michigan. Use it as is, oradapt it to your course! It is an ongoing project. It is alive! It can be used, modified(the LATEX source is available) and redistributed as anyonepleases, subject to the terms of the Creative Commons At-tribution 3.0 Unported License (CC BY 3.0) cb. Pleasetake special note that you can share (copy and redistributein any medium or format) and adapt (remix, transform, andbuild upon for any purpose, even commercially) this mate-rial, but you must give appropriate credit, provide a link tothe license, and indicate if changes were made. You may doso in any reasonable manner, but not in any way that sug-gests that I endorse you or your use. If you are interestedin endorsements, speak to my agent.

I started this material, but I don’t control so much whatyou do with it. Control is sometimes overrated — and I am a control freak, so I shouldknow!

I hope that you find this material useful. If not, I am happy to refund what you paidto me.

Jon Lee

Ann Arbor, Michiganstarted March 2013

v

Preface

This book is a treatment of linear optimization meant for students who are reasonablycomfortable with matrix algebra (or willing to get comfortable rapidly). It is not a goal ofmine to teach anyone how to solve small problems by hand. My goals are to introduce:(i) the mathematics and algorithmics of the subject at a beginning mathematical level,(ii) algorithmically-aware modeling techniques, and (iii) high-level computational toolsfor studying and developing optimization algorithms (in particular, ������ and ����).

Proofs are given when they are important in understanding the algorithmics. Imake free use of the inverse of a matrix. But it should be understood, for example,that B−1b is meant as a mathematical expression for the solution of the square linearsystem of equations Bx = b . I am not in any way suggesting that an efficient way tocalculate the solution of a large (often sparse) linear system is to calculate an inverse!Also, I avoid the dual simplex algorithm (e.g., even in describing branch-and-boundand cutting-plane algorithms), preferring to just think about the ordinary simplex al-gorithm applied to the dual problem. Again, my goal is not to describe the most efficientway to do matrix algebra!

Illustrations are woefully few. Though if Lagrange could not be bothered1, who amI to aim higher? Still, I am gradually improving this aspect, and many of the algorithmsare illustrated in the modern way, with computer code.

The material that I present was mostly well known by the 1960’s. As a student atCornell in the late 70’s and early 80’s, I learned and got excited about linear optimiza-tion from Bob Bland, Les Trotter and Lou Billera, using [1] and [5]. The present bookis a treatment of some of that material, with additional material on integer-linear op-timization, mostly which I originally learned from George Nemhauser and Les. Butthere is new material too; in particular, a “deconstructed post-modern” version of Go-mory pure and mixed-integer cuts. There is nothing here on interior-point algorithmsand the ellipsoid algorithm; don’t tell Mike Todd!

Jon Lee

Ann Arbor, Michiganstarted March 2013(or maybe really in Ithaca, NY in 1979)

vii

Serious Acknowledgments

Throw me some serious funding for this project, and I will acknowledge you — seri-ously!

Many of the pictures in this book were found floating around on the web. I am making”fair use” of them as they float through this document. Of course, I gratefully acknowl-edge those who own them.

Hearty thanks to many students and to Prof. Siqian Shen for pointing out typos in anearlier version.

ix

Dedication

For students (even Ohio students). Not for publishers — not this time. Maybe next time.

xi

The Nitty Gritty

You can always get all released editions of this book (in ���� format) from my webpage and the materials to produce them (LATEX source, etc.) from me.

If you decide that you need to recompile the LATEX to change something, then youshould be aware of two things. First, there is no user’s manual nor help desk. Second,I use a lot of include files, for incorporating code listings and output produced by soft-ware like ����� and ���. Make sure that you understand where everything is pulledfrom if you recompile the LATEX. In particular, if you change anything in the directorythat the LATEX file is in, something can well change in the ���� output. For example,����� and ��� scripts are pulled into the book, and likewise for the output of thosescripts. If you want to play, and do not want to change those parts of the book, play inanother directory.

I make significant use of software. Everything seems to work with:

� �� �����

��� ��������

����� ��������

����������� ��������

������ ����

��� �� ��

Use of older versions is inexcusable. Newer versions will surely break things. Nonethe-less, if you can report success or failure on newer versions, please let me know.

I use lots of LATEX packages (which, as you may know, makes things rather fragile).I could not possibly gather the version numbers of those — I do have a day job! (but Ido endeavor to keep my packages up to date).

xiii

Contents

1 Let’s Get Started 1

1.1 Linear Optimization and Standard Form . . . . . . . . . . . . . . . . . . . 1

1.2 A Standard-Form Problem and its Dual . . . . . . . . . . . . . . . . . . . 2

1.3 Linear-Algebra Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2 Modeling 11

2.1 A Production Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.2 Norm Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3 Network Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.4 An Optimization Modeling Language . . . . . . . . . . . . . . . . . . . . 15

2.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3 Algebra Versus Geometry 23

3.1 Basic Feasible Solutions and Extreme Points . . . . . . . . . . . . . . . . . 23

3.2 Basic Feasible Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.3 Basic Feasible Rays and Extreme Rays . . . . . . . . . . . . . . . . . . . . 30

3.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4 The Simplex Algorithm 33

4.1 A Sufficient Optimality Criterion . . . . . . . . . . . . . . . . . . . . . . . 33

4.2 The Simplex Algorithm with No Worries . . . . . . . . . . . . . . . . . . 35

4.3 Anticycling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.4 Obtaining a Basic Feasible Solution . . . . . . . . . . . . . . . . . . . . . . 42

4.4.1 Ignoring degeneracy . . . . . . . . . . . . . . . . . . . . . . . . . . 43

4.4.2 Not ignoring degeneracy . . . . . . . . . . . . . . . . . . . . . . . 45

4.5 The Simplex Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5 Duality 51

5.1 The Strong Duality Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 52

5.2 Complementary Slackness . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

5.3 Duality for General Linear-Optimization Problems . . . . . . . . . . . . . 54

5.4 Theorems of the Alternative . . . . . . . . . . . . . . . . . . . . . . . . . . 57

5.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

xv

xvi CONTENTS

6 Sensitivity Analysis 63

6.1 Right-Hand Side Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

6.1.1 Local analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

6.1.2 Global analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

6.1.3 A brief detour: the column geometry for the Simplex Algorithm . 67

6.1.4 Reduced costs as dual values . . . . . . . . . . . . . . . . . . . . . 68

6.2 Objective Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

6.2.1 Local analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

6.2.2 Global analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

6.2.3 Local sensitivity analysis with a modeling language . . . . . . . . 71

6.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

7 Large-Scale Linear Optimization 73

7.1 Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

7.1.1 The master reformulation . . . . . . . . . . . . . . . . . . . . . . . 74

7.1.2 Solution of the Master via the Simplex Algorithm . . . . . . . . . 76

7.2 Lagrangian Relaxation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

7.2.1 Lagrangian bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

7.2.2 Solving the Lagrangian Dual . . . . . . . . . . . . . . . . . . . . . 87

7.3 The Cutting-Stock Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 92

7.3.1 Formulation via cutting patterns . . . . . . . . . . . . . . . . . . . 93

7.3.2 Solution via continuous relaxation . . . . . . . . . . . . . . . . . . 93

7.3.3 The knapsack subproblem . . . . . . . . . . . . . . . . . . . . . . . 94

7.3.4 Applying the Simplex Algorithm . . . . . . . . . . . . . . . . . . . 96

7.3.5 A demonstration implementation . . . . . . . . . . . . . . . . . . 96

7.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

8 Integer-Linear Optimization 103

8.1 Integrality for Free . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

8.1.1 Some structured models . . . . . . . . . . . . . . . . . . . . . . . . 103

8.1.2 Unimodular basis matrices and total unimodularity . . . . . . . . 106

8.1.3 Consequences of total unimodularity . . . . . . . . . . . . . . . . 110

8.2 Modeling Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

8.2.1 Disjunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

8.2.2 Forcing constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

8.2.3 Piecewise-linear univariate functions . . . . . . . . . . . . . . . . 118

8.3 A Prelude to Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

8.4 Branch-and-Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

8.5 Cutting Planes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

8.5.1 Pure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

8.5.2 Mixed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

8.5.3 Finite termination . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

8.5.4 Branch-and-Cut . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

8.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

CONTENTS xvii

Appendices 141

A.1 LATEX template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143A.2 ������ for deconstructing the Simplex Algorithm . . . . . . . . . . . . . 149A.3 ������ for Gomory cuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159A.4 ���� for uncapacitated facility-location problem . . . . . . . . . . . . . . 161

End Notes 165

Bibliography 171

Index of definitions and only definitions — lazy and maintainable 174