OPTIMIZATION OF STOCHASTIC MODELS

The Interface Between Simulation and Optimization

THE KLUWER INTERNATIONAL SERIES IN ENGINEERING AND COMPUTER SCIENCE

DISCRETE EVENT DYNAMIC SYSTEMS Consulting Editor

Yu-Chi Ho Harvard University

GRADIENT ESTIMATION VIA PERTURBATION ANALYSIS, P. Glassennan ISBN: 0-7923-9095-4

PERTURBATION ANALYSIS OF DISCRETE EVENT DYNAMIC SYSTEMS, Yu-Chi Ho and Xi-Ren Cao

ISBN: 0-7923-9174-8

PETRI NET SYNTHESIS FOR DISCRETE EVENT CONTROL OF MANUFACTURING SYSTEMS, MengChu Zhou and Frank DiCesare

ISBN: 0-7923-9289-2

MODELING AND CONTROL OF LOGICAL DISCRETE EVENT SYSTEMS, Ratnesh Kumar and Vijay K. Garg

ISBN: 0-7923-9538-7

UNIFORM RANDOM NUMBERS: THEORY AND PRACTICE, Shu Tezuka ISBN: 0-7923-9572-7

OPTIMIZATION OF STOCHASTIC MODELS

The Interface Between Simulation and Optimization

by

Georg Ch. Pflug

~ . . , KLUWER ACADEMIC PUBLISHERS

Boston I Dordrecht I London

Distributors for North America: Kluwer Academic Publishers 101 Philip Drive Assinippi Park Norwell, Massachusetts 02061 USA

Distributors for all other countries: Kluwer Academic Publishers Group Distribution Centre Post Office Box 322 3300 AH Dordrecht, THE NETHERLANDS

Library of Congress Cataloging-in-Publication Data

A C.I.P. Catalogue record for this book is available from the Library of Congress.

The publisher offers discounts on this book when ordered in bulk quantities. For more information contact: Sales Department, Kluwer Academic Publishers,

101 Philip Drive, Assinippi Park, Norwell, MA 02061

ISBN-13: 978-1-4612-8631-8 e-ISBN-13: 978-1-4613-1449-3 DOl: 10.1007/978-1-4613-1449-3

Copyright © 1996 by Kluwer Academic Publishers. Fifth Printing, 1999. Reprint of the original edition 1996

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photo­copying, recording, or otherwise, without the prior written permission of the publisher, Kluwer Academic Publishers, 101 Philip Drive, Assinippi Park, Norwell, Massachusetts 02061

Printed on acid-free paper.

Second Printing, 1999.

Contents

1 Optimization

1.1 Stochastic optimization problems

1.1.1 Recourse problems

1.1.2 Stochastic systems

1.2 Approximations.... ..

1.2.1 The non-recursive method .

1.2.2 The recursive method

1.2.3 Change of measure ..

1.2.4 Recursive versus non-recursive methods

1.2.5 The black-box and the white-box approach

1.3 Bounds ....................... .

1.3.1 Variational inequalities and confidence bounds

1.4 Deterministic optimization procedures and their stochastic coun-

1

3

5

7

9

11

13

14

16

18

19

21

terparts . . . . . . . . . . . . . . . 22

1.4.1 Random search techniques.

1.4.2 Flexible polyhedron search

1.4.3 Line search methods ...

1.4.4 Steepest descent methods

1.4.5 Variable metric methods.

23

25

27

28

32

vi

1.4.6

1.4.7

1.4.8

1.4.9

Conjugate gradient methods.

Bundle methods ...... .

Penalty and barrier function methods

Methods of feasible directions

1.4.10 Methods involving duality.

1.5 Discrete Optimization . . . . . .

1.5.1 Branch and Bound search

1.5.2 Simulated annealing ...

CONTENTS

34

36

38

41

42

46

46

48

2 Discrete-Event processes 55

58

58

74

83

93

2.1 Markov chains with discrete time

2.1.1 Finite state space .....

2.1.2 Infinite denumerable state space

2.2 Markov chains with continuous time

2.2.1 The MIMll queue ..... .

2.2.2 The uniformization principle 100

2.3 MARKOV PROCESSES WITH ARBITRARY STATE SPACE 101

2.4 Semi-Markov processes . . . . . . . . 112

2.5 Generalized Semi-Markov processes 113

2.5.1 Lifetimes and hazard functions 120

2.5.2 The Markovian structure of a Discrete-Event process 121

2.5.3 Simulation of Generalized Semi-Markov processes 122

2.6 Queueing processes ...... .

2.6.1 The single server queue

2.6.2 Queueing networks . . .

123

123

132

CONTENTS vii

3 Derivatives 143

145

152

155

3.1 Derivatives of random processes.

3.1.1 Smoothing nondifferentiable integrands

3.2 Derivatives of probability measures

3.2.1 LP-derivatives of densities 155

3.2.2 Weak derivatives 157

3.2.3 Process derivatives 190

3.2.4 Hazard functions and process derivatives of minima 197

3.2.5 Summary.............. 200

3.3 Derivative concepts for Markov processes

3.3.1 Weak derivatives for Markov processes

3.3.2 Process derivatives of Markov processes

4 Simulation and sensitivity estimation

4.1 Simulation techniques ...... .

201

202

204

211

211

4.1.1 Random number generation 211

4.1.2 Simulation of Markov processes and the regenerative prop-erty . . . . . . . . . . . . . . . . . . . . . . . . 215

4.1.3 Making a non-regenerative process regenerative. 220

4.1.4 Variance reduction . . 221

4.1.5 Importance sampling. 227

4.2 Simulation of derivatives for random variables.

4.2.1 The score function method ...... .

4.2.2 The random generation of weak derivatives

4.2.3 The random generation of process derivatives

4.2.4 Summary ...

4.2.5 A comparison .

4.3 Simulation of sensitivities for Markov Processes

231

232

237

246

247

248

250

CONTENTS ix

G Duality and Lagrangians 341

H Probability spaces and random variables 345

I Convergence of random variables 351

J The Wasserstein distance 359

K Conditional expectations 365

L Martingales 369

M Choquet Theory 371

N Coupling 373

List of symbols

N the set of all positive integers No the set of all nonnegative integers Z the set of all integers lR the real line lR d the d-dimensional euclidean space A the closure of the set A AC the complement of the set A lA the indicator function of the set A:

{ lifuEA lA ( u) = 0 if u ~ A

.,J s the function

{ 0 if u E S .,J s ( u) = 00 if u ~ S

#(A) the cardinality of the set A xT the transpose of the vector x E lR d

xT . y inner product in lR d

(x, y) alternative notation for the inner product x .1 y the vector x is orthogonal to the vector y IIxll the euclidean norm of the vector x

IIxll = JL:i xl IIBII the euclidean operator norm of the matrix B,

i.e. IIBII = sUPllxllSl IIBxll, or equivalently the square root of the maximal eigenvalue of BT B

I/xliI the 1-norm of the vector x

I/xliI = 2:i IXil IIAIII the I-norm of the matrix A = (aij)

"Alb = sUPi Lj laijl x > 0 all components of the vector x are positive x > 0 all components of the vector x are nonnegative

xii

L(H) V' xF(x)

P(R)

M(R)

jlX\lp

IE(X) Var (X) Cov (X, Y) QDev (Y) J.1·H J.1n => J.1 J.1J..v

the diagonal matrix with diagonal elements (Cl, ... , Cd) the Lipschitz constant of the function H the gradient of F w.r.t. x

V'xF(x) = (:;:, ... , ::J

CONTENTS

The symbol V' x is also used, if x is one-dimensional. If there is no danger of confusion, the notation V' F(x) is used. the set of all probability measures on the Borel o--algebra of R, where R is a metric space the set of all signed measures on the Borel cr-algebra of R, where R is a metric space the random variable X is distributed according to the distribution J.1 the LP-norm of the random variable X \lX\lp = [IE(IXjP)F/p the expectation of the random variable X the variance of the random variable X the covariance of the two random variables X and Y the quadratic deviation of the estimate Y abbreviated notation for J H (w) dJ.1( w) weak convergence of the measures J.1n to J.1 the measures J.1 and v are orthogonal to each other i.e. there is a set A such that Il(A) = 0, v(N) = 0 point mass at u indicates the end of a proof (Halmos box)

Preface

Stochastic models are everywhere. In manufacturing, queueing models are used for modeling production processes, realistic inventory models are stochastic in nature. Stochastic models are considered in transportation and communica­tion. Marketing models use stochastic descriptions of the demands and buyer's behaviors, in finance market prices and exchange rates are assumed to be cer­tain stochastic processes, and insurance claims appear at random times with random amounts.

To each decision problem, a cost function is associated. Costs may be direct or indirect, like loss of time, quality deterioration, loss in production or dis­satisfaction of customers. In decision making under uncertainty, the goal is to minimize the expected costs. However, in practically all realistic models, the calculation of the expected costs is impossible due to the model complex­ity. Simulation is the only practicable way of getting insight into such models. Thus the problem of optimal decisions can be seen as getting simulation and optimization effectively combined.

The field is quite new and yet the number of publications is enormous. This book does not even try to touch all work done in this area. Instead, many concepts are presented and treated with mathematical rigor and necessary con­ditions for the correctness of various approached are stated.

The organization of the book is as follows. In chapter 1, optimization algo­rithms are reviewed and discussed whether they are also suitable for stochastic observations. In chapter 2, stochastic models are presented. Much emphasis is put on ergodicity of stochastic systems. This fundamental property guaran­tees that simulation "works" i.e. every infinitely long sample path reflects the properties of all sample paths. Geometric ergodicity allows a quantification of this property.

Chapter 3 is devoted to the mathematical theory of differentiation of random objects with respect to parameters. The different notions of differentiability

xiv PREFACE

may be used in sensitivity analysis or as the basis of optimization algorithms. The estimation of derivatives by random samples is treated in chapter 4. Fi­nally the most important recursive optimization technique, namely stochastic approximation, is discussed in chapter 5. Basic mathematic<,tl notions and re­sults which are used in the presentation are collected in the appendix.

Theorems, Lemmas, Definitions etc. are quoted with just the current number in the same chapter. If we make reference to such an item in a different chapter, the corresponding chapter number precedes the item number.

The book took more than three years of preparation. My part-time affiliation with the Institute of Applied Systems Analysis (IIASA) in Laxenburg, Austria offered many occasions to work on real world stochastic optimization problems and to have discussions with A. Ruszczynski, Yu. Ermoliev, S. Uryas'ev, Yu. Kaniovski and others. Large parts of the manuscript were written in Davis (California), Rennes (France) and Haifa (Israel). I am indebted to my hosts RWets, J. Deshayes and R Rubinstein for their hospitality and support. My special thank goes to R. Rubinstein for his advise and encouragement.

I also had shorter or longer discussions with L. Ho, P. Glynn, R. Suri, A. Gaivoronski, R Wets, B. Heidergott, A. Schwartz and many others. These discussions helped me a lot to get new perspectives in the area.

The manuscript was read by C. Cenker, A. Futschik, W. Gutjahr and G. Uchida. Their help in removing errors and typos is gratefully acknowledged. Despite their valuable help, I am unfortunately sure that the book contains errors. I have therefore established a list of corrections, which will be available under the WorldWideWeb URL

http://www.isoc.smc.univie.ac.at/-pflug/errors.

At the moment this list is empty but I am afraid that it will grow soon with the help of the reader. Please report found errors to pflugClsmc.univie.ac.at.

Vienna, june 1996 Georg Ch. Pflug