smart: a stochastic multiscale model for the analysis of ...powell et al.: a stochastic multiscale...

INFORMS Journal on ComputingArticles in Advance, pp. 1–18issn 1091-9856 �eissn 1526-5528 http://dx.doi.org/10.1287/ijoc.1110.0470

© 2011 INFORMS

SMART: A Stochastic Multiscale Model for theAnalysis of Energy Resources, Technology, and Policy

Warren B. Powell, Abraham George, Hugo Simão, Warren ScottDepartment of Operations Research and Financial Engineering, Princeton University, Princeton, New Jersey 08544

{[email protected], [email protected], [email protected], [email protected]}

Alan Lamont, Jeffrey StewartScience and Technology Directorate, Lawrence Livermore National Laboratory,

Livermore, California 94550 {[email protected], [email protected]}

We address the problem of modeling energy resource allocation, including dispatch, storage, and the long-term investments in new technologies, capturing different sources of uncertainty such as energy from

wind, demands, prices, and rainfall. We also wish to model long-term investment decisions in the presence ofuncertainty. Accurately modeling the value of all investments, such as wind turbines and solar panels, requireshandling fine-grained temporal variability and uncertainty in wind and solar in the presence of storage. We pro-pose a modeling and algorithmic strategy based on the framework of approximate dynamic programming (ADP)that can model these problems at hourly time increments over an entire year or several decades. We demon-strate the methodology using both spatially aggregate and disaggregate representations of energy supply anddemand. This paper describes the initial proof of concept experiments for an ADP-based model called SMART;we describe the modeling and algorithmic strategy and provide comparisons against a deterministic benchmarkas well as initial experiments on stochastic data sets.

Key words : artificial intelligence; simulation; statistical analysis; analysis of algorithms; queuesHistory : Accepted by David Woodruff, Area Editor for Heuristic Search and Learning; received August 2009;

revised August 2010, April 2011; accepted April 2011. Published online in Articles in Advance.

1. IntroductionOver the next few decades, the United States needs totransition to an energy system with much lower car-bon emissions and reduced dependence on unreliablesupplies. This process requires significant investmentin research and development (R&D) and new infras-tructure. The investments have to be made in thepresence of multiple sources of uncertainty that couldinfluence the future energy system, such as the mix ofprimary energies (nuclear, renewable, carbon seques-tration); the balance of energy carriers, including elec-tricity, hydrogen, and alcohol and synthetic fuels;and end-use strategies (level of demand, demandresponse). A major component will be renewableenergy that depends on intermittent sources such aswind and solar. The volatility of these sources can bemitigated with different types of storage such as bat-teries and pumped hydro.

The dynamics of our system are driven by exoge-nous factors (wind, solar, climate, technology, prices,supplies, and demands) and market decisions (howmuch energy should be produced by each source,how much of this energy should be used for eachtype of demand, how much energy should be stored,and how much capacity should be added or retired

each year in response to changing conditions). Tomodel investment decisions and government policies,the model has to extend over a multidecade horizon.To accurately represent the economics of energy tech-nologies, our model has to capture the hourly fluc-tuations of wind, daily solar cycles, seasonal climatechanges, and annual changes in energy technologies.Finally, it is widely understood that a key enablingtechnology for intermittent energy is storage, whichcomplicates the design of algorithms by coupling allthe time periods together.

Past models have represented market investmentdecisions by assuming that they can be reasonablyapproximated as optimal solutions to determinis-tic linear programs. These models have been effec-tive at capturing fundamental differences in supplyand demand, where the most famous example isthe groundbreaking policy conclusions from the PIESmodel (Hogan 1975). In recent years, there has beengrowing interest in the use of models to guide invest-ment and policy decisions in new technologies suchas wind and solar and to guide government poli-cies for carbon taxes and R&D to meet goals suchas 20% energy from renewables by 2030. It has longbeen recognized, however, that models that represent

1

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.

Published online ahead of print September 15, 2011

Powell et al.: A Stochastic Multiscale Model for the Analysis of Energy Resources, Technology, and Policy2 INFORMS Journal on Computing, Articles in Advance, pp. 1–18, © 2011 INFORMS

intermittent sources such as wind and solar overstatetheir value if they use averages (wind turbines aredramatically more productive if wind blows at a con-stant speed). The problem with intermittent sourcesis that they are not dispatchable (directly controllablein real time), requiring investments in other energytechnologies to make up for low periods, even if theaverage production is very high (see Lamont 2008for a thorough discussion of these issues). Energypolicy researchers have found that using time incre-ments longer than an hour begin to introduce signif-icant errors. Storage can overcome the limitations ofintermittent sources, but representing storage imposesadditional demands on a model.

This paper extends a deterministic energy planningmodel called META∗Net (see Lamont 1997, 2008),developed at Lawrence Livermore National Labora-tory, which has been used in two forms: a model thatcaptures hourly variations over a year and a modelthat extends for several decades in increments ofone to five years. META∗Net is an aggregate model,which captures approximately 13 different sources ofenergy to satisfy 9 types of demand, spanning differ-ent types of electric power as well as different energyforms for industrial and residential purposes. Aggre-gate planning models such as these stand betweensimple models that analyze the economics of differenttypes of energy generation in isolation and detailednetwork models used in operational planning for util-ities and grid operators.

Separate from the need to handle fine-grained timescales is the need to capture different types of uncer-tainty. This ranges from uncertainty about wind (windoperators often have to commit to a certain levelof production a day in advance) to uncertaintiesabout policies (imposition of a carbon tax), technology(whether will we be able to sequester carbon), prices(what the price of oil will be in five years), and climate(how global warming will affect snow accumulationand rainfall). These uncertainties can affect the risksincurred in an investment, significantly reducing theappeal of technologies that look attractive if the futureunfolds as expected.

Our goal is to develop a model that is stochas-tic (handling different sources of uncertainty) andmultiscale (handling different levels of spatial andtemporal granularity). We also need to incorporatestorage, which has the effect of linking different timeperiods together. Finally, we are interested in near-optimal policies, which we achieve using approximatedynamic programming (ADP). This dimension pro-vides a sharp contrast with other papers that may usea stochastic model but do not account for uncertaintywhen making a decision. We use deterministic bench-marks to demonstrate the quality of our solutions, but

we are limited to qualitative evaluations to assess ourability to properly reflect uncertainty in our policies.

These goals represent what we feel we are able toachieve in this paper, but there are other goals towhich we aspire. We are able to handle fine-grainedsources of uncertainty (e.g., wind, rainfall, prices,demand), but we would like to handle coarse-grainedforms of uncertainty, which include changes in taxpolicy or breakthroughs in technology. We can handlespatially disaggregate supply and demand patterns,but we cannot model individual power plants; thisrequires the introduction of integer variables that gov-ern when a plant is in use or not.

This paper describes an early implementation ofSMART (Stochastic Multiscale model for the Analysisof energy Resources, Technology, and policy). SMARTuses a control-theoretic framework to step forwardthrough time in hourly increments over a multidecadehorizon, which makes modeling energy storage pos-sible. Approximate dynamic programming is usedto produce policies that properly model the flowof information when making decisions. Althoughdesigned to handle stochastic problems, SMART canbe applied to deterministic problems, allowing us tocompare the performance of the model to a determin-istic linear programming model when such a modelcan be solved. We show that SMART closely matchesthe optimal solution from a deterministic linear pro-gram, which provides an important benchmark forour ADP algorithm. We also show that it exhibitsrobust behaviors when applied to a stochastic prob-lem. To the best of our knowledge, this is the firststochastic multiscale model that scales to handle hun-dreds of thousands of time periods while producingsolutions that closely match the results of optimalsolutions for deterministic formulations.

SMART can be used for both long-term policy stud-ies and economic analyses of portfolios of energyproduction and storage technologies. It can be usedto simulate energy investment decisions—under theassumption that a cost-minimizing model mimicsmarket behavior—or as a normative model to deter-mine how to reach specific energy targets. It can beused to quantify risk and to study the impact of dif-ferent energy investment strategies. It can also pro-vide accurate estimates of the value of intermittentenergy sources and the value of storage using realis-tic assumptions about our ability to forecast energysupply.

The rest of this paper is organized as follows. Sec-tion 2 reviews the literature for models on energypolicy, and it briefly touches on the literature forenergy storage and wind. Section 3 provides a sketchof our modeling and algorithmic strategy. An out-line of the mathematical model is described in §4,

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.

Powell et al.: A Stochastic Multiscale Model for the Analysis of Energy Resources, Technology, and PolicyINFORMS Journal on Computing, Articles in Advance, pp. 1–18, © 2011 INFORMS 3

but the full model is contained in the Online Sup-plement (available at http://joc.pubs.informs.org/ecompanion.html), which contains a number ofimportant equations to which the paper refers. Ourmodel captures both the physical dynamics andthe flow of information and decisions, but it doesnot provide a method for making decisions. Section5 provides two optimization formulations to drivethe process of making decisions: a deterministic lin-ear programming model (which we later use as abenchmark) and a stochastic optimization model. Sec-tion 6 describes an algorithmic strategy for solvingthe stochastic optimization problem based on approx-imate dynamic programming. In §7, we describebenchmarking experiments that compare the solutionobtained using approximate dynamic programmingto the optimal solution produced by a linear program-ming algorithm using a deterministic model. Experi-ments on stochastic applications are also reported todemonstrate specific behaviors that we would expectfrom a nonanticipative decision rule. Section 8 con-cludes the paper.

2. Literature ReviewThere are several bodies of literature that are relevantto our research: deterministic energy investment mod-els, stochastic optimization models in energy, modelsfor energy storage, and studies of the value of energyfrom wind.

Following the original PIES model (Hogan 1975), aseries of deterministic optimization models have beendeveloped to model energy investment decisions overmultidecade horizons. These include the NationalEnergy Modeling System (NEMS) (2003) that takes asinput demands and costs of various energy resourcesfrom a family of models. The MARKet ALlocation(MARKAL) model is a deterministic, optimization-based system that integrates the supply and end-usesectors of an economy (see Fishbone and Abilock1981, Loulou et al. 2004). MARKAL minimizes thediscounted total cost of the system, which includesinvestment, maintenance, and operations costs of thetechnologies and the purchase costs of fuels, and italso accounts for revenue generated. META∗Net (seeLamont 1997, 2008) combines the techniques used inNEMS as well as in MARKAL in that it performsmarket equilibration in a linear programming-basedmodel.

The early stochastic energy investment models canbe divided into two categories. We refer to the firstcategory as partially stochastic models, such as thestochastic version of MARKAL (Johnson et al. 2006)developed by the Environmental Protection Agency.In this version of MARKAL, a “stochastic wrapper” isused, wherein random parameters are sampled from

probability distributions, after which a determinis-tic optimization problem is solved. This process isrepeated a number of times, allowing the outputsto be analyzed statistically. This means that for eachrun on a sample realization, decisions are allowedto see events in the future. Kanudia and Loulou(1998) present a version of MARKAL that explicitlyincorporates multiple scenarios and includes as manyreplications of the MARKAL variables as the numberof scenarios considered. The optimization is done byminimizing total expected costs over all the scenariosbut where the solution for each scenario is indepen-dent of the other scenarios. Again, this strategy allowsa decision to see outcomes in the future for a partic-ular scenario.

Another example of a partially stochastic model isthe stochastic version of MESSAGE (Model for EnergySupply Strategy Alternatives and their General Envi-ronmental impact) from the International Institute forApplied Systems Analysis. The conventional MES-SAGE model (Messner and Strubegger 1995) usespoint estimates to forecast technology parameters.The stochastic MESSAGE model (Messner et al. 1996)aims to reduce the risk of underestimating the totalinvestment cost as a stochastic optimization problem,using a sufficiently large number of sample realiza-tions of the investment costs.

The second category of stochastic models uses theframework of stochastic programming to handle themodeling of random information and decisions (seeWallace and Fleten 2003 for an excellent introduc-tion to these models). Stochastic programming modelsuse the notion of scenario trees to represent uncer-tain information. The challenge with this frameworkis that the size of a scenario tree grows exponen-tially with the number of time periods, requiringthe use of aggregation strategies such as large timesteps and small numbers of scenarios per time period.See Birge and Louveaux (1997) for an in-depth treat-ment of this approach, and see Høyland and Wallace(2001), Gröwe-Kuska et al. (2003), and Kaut andWallace (2003) for research on generating scenariotrees. A competing strategy uses dual information togenerate cuts to approximate the impact of decisionsmade now on the future. This was originally intro-duced as the L-shaped method (Van Slyke and Wets1969) and was later adapted (for two-stage problems)using a Monte Carlo-based framework as stochasticdecomposition in Higle and Sen (1991) (see Higle andSen 1996 for a more in-depth treatment).

One limitation of models based on stochastic pro-gramming is that the scenarios (outcomes of ran-dom information) cannot depend on the sequence ofdecisions. For example, if decisions lead to energyshortages, prices may spike. Stochastic programmingrequires that scenarios be generated in advance, and

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


this limits their ability to capture the interactionbetween decisions and exogenous events. However,stochastic programming is able to capture complex,history-dependent processes such as the timing of acarbon tax or breakthroughs in technology.

The last class of stochastic energy investment mod-els can be viewed as pure simulation models. Per-haps the most visible example of this class is theStochastic Energy Deployment Systems (SEDS) (seehttp://seds.nrel.gov/), which uses classical MonteCarlo simulation to step forward through time. Let �n

represent the sample realization of all random vari-ables for the nth iteration of the model, and let cte4�n5be the cost of energy technology e ∈E in year t whilefollowing sample path �n. SEDS makes additionalinvestments, given by xn

te, according to the expression

xnte = �D

t

exp−�scte4�n5

∑

e′∈E exp−�scte′4�n51 (1)

where �Dt is a set of parameters that (over time) allows

investments to be scaled to demand, and �s is ascaling coefficient that controls the degree to whichinvestments are directed toward lower-cost technolo-gies. There is no attempt to formally optimize energyinvestments. A major goal of the model is to pro-vide fast estimates of the effects of policy changes ina fairly transparent way. The model works in yearly(or even multiyear) increments, so there is no attemptto capture fine-grained variations in wind and solarenergy or demand.

There is a separate literature on the use of storage,largely divided between the management of waterreservoirs and other forms of energy storage. Theliterature on water reservoir management is fairlyextensive (see, e.g., Foufoula-Georgiou and Kitanidis1988, Lamond and Sobel 1995, Archibald et al. 1997,and Cervellera et al. 2005). Algorithmic researchaddressing the problem of optimizing flow controlsunder uncertainty has progressed along two lines.The first uses dynamic programming, typically resort-ing to some sort of numerical or approximate strategyto handle the curse of dimensionality that arises withmultiple reservoirs (see Nandalal and Bogardi 2007and the references cited there). A major breakthroughin this approach involves approximating the impactof decisions made now on the future by using cutsgenerated from the dual of the linear program in thenext time period. Pereira and Pinto (1991) were thefirst to use this strategy in an energy setting underthe name of stochastic dual dynamic programming,modeling a large, multireservoir hydroelectric system.A separate modeling and algorithmic strategy withinthe stochastic programming community uses the con-cept of scenario trees, where sample paths of exoge-nous information are generated, retaining the entire

history at each node in the tree. Jacobs et al. (1995)and De Ladurantaye et al. (2007) give nice illustra-tions of this strategy in a hydroelectric setting. Heuris-tic approaches to reservoir management are describedin Johnson et al. (1991) and Edmunds and Bard(1990a, b).

A separate literature has evolved around the mod-eling of storage devices to handle either structuralor random variations between our ability to generatepower and the demand for power. Brunetto and Tina(2007) use a deterministic model to solve the problemof day-ahead electricity markets, completely ignoringthe high level of uncertainty that arises in this set-ting. Castronuovo and Lopes (2004) describe windas a stochastic process but then represent only thevariability of wind, not its uncertainty. Korpaas et al.(2004) and Paatero and Lund (2005) also use deter-ministic models to study storage for wind. Garcia-Gonzalez et al. (2008) and Brown et al. (2008) use theidea of stochastic wind scenarios, but allow decisionsto “see” the wind in the future within a particular sce-nario (violating what are known as nonanticipativityconditions).

There is a body of literature that is working tounderstand the impact of large amounts of wind onthe power grid. Sioshansi and Short (2009) use a two-day deterministic forecast of wind in a rolling hori-zon model to simulate decisions. Sioshansi (2010) usesa deterministic unit commitment model (requiring adeterministic forecast of wind) to study the impactof real-time pricing on the use of energy from wind.Holttinen et al. (2011) undertake a careful study ofthe effect of large quantities of wind (also see thesummary in Holttinen et al. 2009). However, no for-mal mathematical models are given (this includes thedoctoral dissertation on which this work is based—Holttinen 2004), and it does not appear that the studyuses a policy that explicitly accounts for the uncer-tainty in wind. Separate from the issue of handlinguncertainty is the challenge of capturing hourly vari-ations in long-term models. A modeling strategy thatavoids capturing every hour over a year (8,760 timeperiods) involves simulating only a sample of dif-ferent time periods (capturing, for example, differenttimes of the year and different times of the day), butsuch an approach makes it impossible to accuratelymodel storage (whether it is hydro, compressed air, orbatteries), which requires the ability to step forwardacross all time periods to calculate storage levels.

3. Strategy for a StochasticMultiscale Model

The goal of our research is to develop a model thatallows us to make short-term decisions about electricdispatch, energy resource allocation, and storage over

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


the short term (one year) as well as long-term invest-ment decisions in the presence of different sources ofuncertainty. We are interested in questions such as theimpact of increasing the fraction of energy generatedfrom wind and solar and in obtaining an accurateestimate of the marginal value of wind and solar inthe presence of storage. At the same time, we are notlooking to address decisions such as where to locate awind farm, or whether a particular coal or natural gasfacility should be used at a particular point in time.These details are important for operational problems,such as unit commitment, or specific investments,such as additions to the grid or energy facilities atspecific locations. Our model does not represent indi-vidual energy generators because the resulting modelwould have to include thousands of integer variablesper time period. Even without these details, we facea difficult algorithmic challenge.

We anticipate running our model over two timeframes. For studies demonstrating the effect of theimpact of storage on the marginal value of wind andsolar, it makes sense to run the model in hourly incre-ments over a year. For studies on the impact of taxsubsidies, commodity prices, and breakthroughs intechnology, we will want to extend the horizon overseveral decades while retaining the ability to modelvariations in wind, solar, and demand in hourlyincrements. For problems where we are only inter-ested in modeling a single year, we may be inter-ested in a spatially disaggregate model, whereas forlong-term investment decisions, a spatially aggregatemodel may be sufficient.

We use the modeling and algorithmic frameworkof approximate dynamic programming where dis-patch, storage, and investment decisions at hour hin year t are represented by the vector xth, which isdetermined by a decision function (or policy, in thelanguage of dynamic programming) X�4Sth5, whichdepends on the system state variable Sth. The termSth is designed to include only the information avail-able at time 4t1h5, and as a result, decisions are notallowed to anticipate events in the future.

Our strategy involves stepping forward throughtime as a simulation model would, but it repeats theprocess while learning from each iteration. Once ayear (corresponding to h = 0), we make investmentdecisions for new energy capacity that will arrive ina future year. Then, for the remainder of the year,we step forward to solve hourly dispatch problemsand thereby determine how much energy to producefrom each source to satisfy each type of demand.Energy demands comprise residential, commercial,and industrial demands for electricity and natural gas(residential uses include cooking and heating), as wellas freight transportation and aircraft demands forelectricity, hydrogen, and hydrocarbon fuels. Primary

energy resources include coal, petroleum, naturalgas, wind, solar, temperature gradients (geothermal,ocean, thermal), nuclear, biomass, and tides. Theseundergo conversion processes in plants to generateelectricity, hydrocarbon fuels, or hydrogen, each ofwhich is then channeled to the appropriate sourcesof demand. Whereas most of the conversion plantsthat handle carbon-based energy sources release CO2to the atmosphere, some technologies are able tosequester CO2. The dispatch problem also includesdecisions of how much to store and how much to con-vert to different types of energy (for example, naturalgas can be used to generate electricity, which in turncan be used to convert biomass to fuel or to compresshydrogen).

We demonstrate SMART using two networks. Thefirst is a spatially aggregate model, a portion of whichis illustrated in Figure 1, that represents 13 differ-ent types of energy supplies and 9 types of energydemand. If there is a shortage of energy supplied fromthese resources, there also exists the option of import-ing electricity. Each node in the network representsan aggregated version of a facility in a typical energynetwork. Resource nodes provide the supply of rawresources (such as coal, natural gas, wind, solar,biomass, and nuclear) at a certain cost. Conversionnodes take as inputs the raw resource and some typeof power needed to generate more power in someform such as electricity, natural gas, and transporta-tion fuel. The flows out of each of these nodes are con-strained by the capacity available at that node, whichis controlled by investment decisions. Demand nodescapture the demand for one of the various types ofpower. Market nodes accumulate energy from differ-ent sources to be distributed to the various types ofdemand (these can be sectors or locations). The net-work also captures interactions such as the need toproduce more electricity if it is needed to producehydrogen or ethanol. We may also add other nodesin this illustration to capture costs involved in thetransportation of the energy resource (for example,from the source to the conversion plant) and the dis-tribution of the final output (for example, electricitythrough transmission lines).

The second network model, used in a study ofintermittent energy by Tagher (2010), captures thegeneration of electricity from coal, nuclear, naturalgas, and wind from each of the 48 contiguous states tosatisfy different demand sectors in each of those statesplus the District of Columbia. Unlike our aggregatenetwork model, this model only considers electric-ity. The model did not have an explicit representationof the grid, but state-to-state distances were used tocompute transportation costs and transmission losses,using individual links for state-to-state transmission.This representation of power flows, which captures

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


Servicedemands

End-useconversions

Fuels/energycarriers

Primary

Resources

Resid.NG

Indust.elect.dmnd

Coal SW rsrcWind rsrc

Solar rsrc

BiomassNG

NG distr

Coal SW

NG CC CrbCpt

Biomasselect.

Elect.CA

Nuclear fuel

Nuc elect.Solar PV

Indust NGcomb

Resid.elect.dmnd

Hydro

Hydro

Geotherm

Geoth elect.

Nuclear repos

Coal pulv

Elect.

Elect. trans

Coal GenerSW mkt

Elect.

Elect.

Biom.

NG CT

Coal IGCC

Coal IGCC

Export.elect.dmnd

Import elect.

NG trans

Biom. Su

NG steam

NG oxy

Resid elect. distr Resid NG distr

CommNG

Commelect.dmnd

Comm elect. distr Comm NG

Comm NG comb

LqHC

Indust elect.

Nucl fuel

Indust elect. sum

HydroResrvr

WindTahachapi

WindSolanoelect.

Wind

Wind

Biom sto

NG elect.distr

NG elect.

Indust. Indust.NGdmnd dmnd

Resid. NG comb

SGorgonio

NG CCelect.

Coal pulvCarb captElect. SW

Indust NG sum

IndustNG distr

WindAltamontt

elect.

Elect. trans.NW

HydrFlow regl

Figure 1 Illustration of the Energy Distribution Network

point-to-point movements without explicitly model-ing the grid, is a common approximation made inpolicy studies. The linear program for a single timeperiod had approximately 40,000 rows and columns.

We do not model individual energy generators nordo we model the grid itself. This level of detail isrequired for operational models and for certain typesof regional planning models that require the identifi-cation of bottlenecks in the grid. Including this detailin a model that spans 20 years in hourly increments isa goal of the energy modeling community, but we arenot ready to claim that we can handle a model withthis granularity. We include transmission losses at anaggregate level, and we can model limits on interre-gional flows (as the CAISO model does). The modelcan be used to compare the economics of wind versusshale gas, coal, and nuclear under different taxationpolicies and targets for renewables.

Our modeling and algorithmic strategy involvessimulating our way through the time periods, solv-ing small optimization problems to handle dispatchand storage decisions (each hour) and investment

decisions (each year). As we step forward throughtime, we use Monte Carlo simulation to sample real-izations from any random variables. Our challengeis designing a decision function X�4Sth5 that pro-duces economically rational decisions without violat-ing restrictions on the availability of information.

We begin by providing an outline of the model ofthe problem in §4 (the complete model is given in§A.1 of the Online Supplement), followed by bothdeterministic and stochastic optimization formula-tions in §5. The mathematical model covers both theaggregate and disaggregate networks.

4. Outline of the Energy ModelWe let t ∈ T index years, and we let 0 ≤ h < 81760index the hours within a year. Our model consists ofthe following five fundamental components:

Sth = The state of the system at time4t1h5.

xth = The vector of decisions at time4t1h5.

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


Wth = The vector of random variablesrepresenting all the new infor-mation that just became avail-able in the hour prior to 4t1h5.

SM 4Sth1xth1Wt1h+15= The transition function (or sys-tem model) that governs theevolution from state Sth to St1h+1(or from St1H to St+110), givendecision xth and new informa-tion Wt1h+1.

Cth = The contribution function cap-turing rewards earned duringthe hour preceding 4t1h5.

The state variable consists of the current status ofour energy investments Rth, energy held in differentforms of storage yth, energy demands Dth, rainfall lev-els pth, and a generic “state-of-the-world” vector �th

that captures information about technology, climate,government policy, and the availability of intermittentenergy such as wind and solar.

We let the attributes of energy resources or storagebe represented by the vector a. In our aggregatemodel, a would represent the type of technology, itslocation, and age. For the disaggregate model, we addthe attribute “state” to capture geography. We let Abe the set of all possible attribute vectors, and we letAConv and AStor represent the sets of attribute vectorsthat correspond to conversion and storage resources,respectively. We let Rtha be the level of investment ina technology with attribute a ∈ A (for example, thiscould be the megawatt-hours of generating capacityof a wind farm), and we let Rth = 4Rtha5a∈A be the vec-tor of energy investments. The vector yth = 4ytha5a∈AStor

captures the amount of energy in each type of storage(note that our numerical work includes only hydro-storage). It is easy to see that the disaggregate model,with the inclusion of a single spatial attribute, dramat-ically increases the dimensionality of the vector Rth.

The decision variable xth consists of the vector ofinvestments in energy conversion capacity x

capth and

dispatch decisions xdispth . We assume that we make

decisions on energy investments only once a year athour h = 0, which means we are only interested inx

capt0 , with x

capth = 0 for h> 0. Energy dispatch decisions

govern the flow of energy from each source node (e.g.,a coal mine or a source of natural gas), through con-version nodes (coal, nuclear, and oil plants), throughvarious transmission and storage nodes, and finallyto demand nodes. A portion of the aggregate net-work is shown in Figure 1. This is the network ata point in time (a particular hour). These networksare linked over time solely through the deposits toand withdrawals from storage devices. We let a be theattributes of a node in the dispatch network, whichwould represent moving energy over a link in the net-work. We let xth1 aa′ be the amount of energy moved

from node a to node a′. We let �aa′ capture transmis-sion and storage losses when moving from a to a′; �can also handle changes in units (e.g., from barrels ofoil to megawatt-hours of energy).

Our decisions are governed by physical constraints.For example, there are conservation of flow con-straints in the dispatch network as well as upperbounds on the amount of energy that can be stored ina particular device or location at a point in time. Theflow conservation constraints are given by

∑

a′∈←−A 4a5

�a′axdispth1 a′a −

∑

a′′∈−→A 4a5

xdispth1 aa′′ = 0

for nonstorage locations1 (2)

ytha +∑

a′∈←−A 4a5

�a′axdispth1 a′a −

∑

a′′∈−→A 4a5

xdispth1 aa′′ ≥ 0

for storage locations0 (3)

We let Xth be the feasible region at time 4t1h5, and werequire that xth ∈Xth.

Once we have made a decision, the system thenevolves over time, with new information arrivingthat also changes the state of the system. We viewnew information as a random variable that representssome sort of exogenous change to one of our statevariables. Thus, Rth can be exogenous changes to ourenergy investments (e.g., Hurricane Katrina takingdrilling rigs out of commission), Dth can be randomchanges in demand for different types of electricity, pthcan be the random change in the amount of rainfall,and �th can be a change in technology, climate, or pol-icy (e.g., the government just imposed a carbon tax).These changes may take place on hourly time scalesbut will often occur on longer time scales (we modeltechnology as changing only once per year). We letWth capture the vector of all these sources of randominformation that occurred in the hour preceding 4t1h5.

We later need some notation for describing aset of sample realizations. Let ì be the set of allpossible realizations of the sequence 4W111 0 0 0 1W1H1W211 0 0 0 1W2H1 0 0 0 1WTH 5, where T is the number ofyears and H is the number of time periods in a year.We let Wth4�5 be a sample realization of the ran-dom variables in Wth when we are following samplepath �. If we are following sample path �n for itera-tion n, we will index variables such as the state vari-able using Sn

th to represent the actual state at time 4t1h5while following sample path �n.

All of the physics of the problem are captured bythe transition function (also known as the systemmodel, plant model, or model) using

St1h+1 = SM 4Sth1xth1Wt1h+151 h= 0111 0 0 0 1H − 11

St+110 = SM 4StH1xtH1Wt+11050

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


For example, changes in Rth and �th would bewritten as

Rt1h+1 =Rth + xcapth + Rt1h+11 (4)

�t1h+1 = �th + �t1h+10 (5)

Our model assumes that capacity investment deci-sions made at the beginning of one year arrive atthe beginning of the next year. In practice, an invest-ment may take several years before it comes online.Even assuming a one-year delay requires that wemake these decisions under uncertainty. If we wishto capture multiyear delays, we have an instance of alagged asset acquisition problem. This issue has beenaddressed in an ADP setting in Godfrey and Powell(2002b), Topaloglu and Powell (2006), and Nascimentoand Powell (2009).

We note that it is possible to capture the depen-dence of exogenous information on the state of thesystem. For example, if we overinvest in ethanolcapacity or wind turbines, we would expect thatprices might drop. We did not explicitly do this in ournumerical work, but the extension is a minor one. Wemay also have processes (e.g., prices, wind, demand)with autocorrelation. When this happens, we have tocapture at least some of the history of these processesin the state variable. It is easy to incorporate theseissues in the transition functions above, but it cancomplicate the challenge of designing good policies.

We measure our performance by adding up costsover time. Total capacity acquisition costs, dispatchcosts, fuel costs, and operating costs are given by

Ccapth 4Sth1x

capth 5= The incremental capital costs in

hour h of year t, andC

dispth 4Sth1x

dispth 5= The total costs resulting from the

dispatch of energy in hour h ofyear t.

We may express the total cost function in hour h ofyear t using

Cth4Sth1xth5=Ccapth 4Sth1x

capth 5+C

dispth 4Sth1x

dispth 50

We have not addressed the problem of how to makedecisions. In the next section, we present two mod-els, a deterministic linear programming model and astochastic model, that lead to different methods formaking a decision.

5. Optimization FormulationsOur modeling framework left unanswered the mech-anism for actually making decisions. Section 5.1describes a deterministic linear programming for-mulation that can be solved by using commercialpackages. Section 5.2 provides a stochastic optimiza-tion formulation for which exact solution algorithmsare not available. The remainder of this paper thenfocuses on designing an approximation strategy tosolve the stochastic optimization model.

5.1. A Deterministic Linear Programming ModelIf we assume that all the exogenous information isdeterministic (and known in advance), we can formu-late the decision problem as a single (albeit very large)linear program. The objective function is given by

minx

∑

t∈T

∑

h∈H

Cth4Sth1xth50

This has to be solved subject to two sets of constraints.The first set governs decisions made at a point intime, which is given by Equations (14)–(19) in §A.1of the Online Supplement. The second set of equa-tions are the transition equations that link activitiesover time (basically the transition function), which arecaptured by Equations (20)–(24) in §A.1 of the OnlineSupplement. It is significant that the linear program-ming model views all these constraints together overall points in time. This is why a linear programmingmodel (even with deterministic formulations) quicklybecomes too large to solve. For example, a 20-year,spatially aggregate model with hourly dispatch deci-sions produced a constraint matrix with more than 34million rows and 30 million columns.

5.2. A Stochastic Optimization ModelThe linear programming model suffers from twoimportant limitations. First, it is unable to handlelonger planning horizons while retaining the abil-ity to model intermittent supply and demand on anhourly level (along with storage). Second, it cannothandle the different forms of uncertainty that arisein energy policy analysis. For this reason, we posethe problem of controlling the stochastic version ofthe problem in terms of finding a policy (or decisionfunction) to make decisions about dispatch, storage,and investments over time. We represent the decisionfunction using

X�th4Sth5= A function that returns a decision xth ∈Xth,

given the state Sth under the policy � ∈ç.

The set ç refers to the set of potential decisionfunctions, or policies, whose meaning becomes clearerbelow. At this point, we refer abstractly to the spaceof all possible decision functions.

The problem being stochastic, our goal is to find theoptimal policy from among the set ç of policies bysolving

min�∈ç

Ɛ

{

∑

t∈T

∑

h∈H

Cth4Sth1X�th4Sth55

}

1 (6)

where St1h+1 = SM 4Sth1xth1Wt1h+15 (for h= 01 0 0 0 1H−1)and St+110 = SM 4StH1xtH1Wt+1105. The challenge, ofcourse, is designing the policy X�

th4Sth5.We note that by construction of our decision func-

tion (in particular, its dependence on the state vari-able), it is not allowed to depend on events in the

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


future. This seemingly essential property is violatedby the stochastic models that are allowed to make adecision as a function of a “scenario” that might spec-ify, for example, the entire sample path of prices orenergy from wind.

6. Algorithmic StrategyThere are a number of strategies that have beenused to solve stochastic optimization problems asstated in Equation (6). These include various typesof myopic policies (such as the logit investmentrule in SEDS); rolling horizon policies (which usea point forecast of the future to make decisionsnow); simulation optimization, where we optimizethe parameters of a myopic policy (Swisher et al.2000); stochastic programming (Higle and Sen 1996,Birge and Louveaux 1997, Wallace and Fleten 2003);and dynamic programming. We adopt the frame-work of dynamic programming in §6.1, which intro-duces a number of computational hurdles. Theseare addressed in §6.2 using approximate dynamicprogramming.

6.1. Dynamic Programming FormulationOur strategy to solve the energy model involves for-mulating the model as a dynamic program, solvingdecision problems for each hour h ∈ H in each yeart ∈T. We first define the value Vth4Sth5 of a state Sth asthe sum of the contributions that we expect to make,starting in state Sth, if we act optimally (make optimaldecisions) until the end of the time horizon. Bellman’sequation enables us to recursively compute the opti-mal value functions associated with each state,

Vth4Sth5=

maxxth∈X

disp1nth

(

−Cth4Sth1xth5+ Ɛ4Vt1h+14St1h+155 � Sth)

1 (7)

where St1h+1 = SM 4Sth1xth1Wt1h+15.As pointed out earlier, for a given year t, we assume

that capacity additions are made only at the begin-ning (h= 0) of the year with new capacity arriving insome future year; the capacity resource state remainsunchanged during the remaining hours. The valuefunction Vth4Sth5 in Equation (7) captures the informa-tion about the future state of the system while solvingthe problem for the current time, including the valueof water left behind in the reservoir after the decisionsin the current time period are implemented.

In the remainder of the paper, we consider onlya single form of storage—namely, the reservoir thatstores water collected from precipitation. This meansthat the set AStor consists of a single aggregate reser-voir, but it is straightforward to adapt our modelto multiple reservoirs, as well as multiple types ofstorage. The water from the reservoir may be used

to satisfy the energy demands now or stored to beused to satisfy demands later in time. Hydro energyis typically among the cheaper types of energy andtends to be used before more expensive energy types(such as coal or nuclear) if we do not provide forstorage. Ideally, our model of reservoir managementshould mimic real-life behavior where water wouldbe allowed to collect and be stored in the reservoirduring periods of heavy precipitation to be used laterduring dry periods.

Solving the dynamic program in Equation (7)is generally hard even for problems with smallstate spaces. The traditional dynamic programmingapproach for solving Equation (7) suffers fromthree curses of dimensionality arising from thestate space (8Sth9h∈H1 t∈T), the exogenous information(8Wth9h∈H1 t∈T), and the action space (8Xdisp

th 9h∈H1 t∈T) (seePowell 2007, Chapter 5). We now describe the tech-niques we use to overcome these.

We first define the postdecision state variable, Sxth =

SM1x4Sth1xth5. The postdecision state for hour h is thestate of the system immediately after making the deci-sions at hour h but before the occurrence of the ran-dom events (Wt1h+1) between hours h and h + 1. Inour analysis, we focus on two important componentsof the state variable, namely, the capacity resourcestate Rth and the reservoir level yth. The postdecisiontransitions of these components can be written as

Rxtha =Rtha + x

captha 1 a ∈AConv1

yxtha = ytha +

∑

a′∈←−A 4a5

�a′axdispth1 a′a1 a ∈AStor0

We then construct the value function around the post-decision state Sx

th = 4Rxth1y

xth5 using

V xth4S

xth5= Ɛ6Vt1h+14St1h+15 � Sx

th70 (8)

This allows us to avoid having to compute an expec-tation within the optimization formulation of Equa-tion (7). The modified dynamic program can bewritten as

Vth4Sth5= maxxth∈X

dispth

4−C4Sth1xth5+V xth4S

xth550

This leaves the problem of computing V xth4S

xth5. For

this, we turn to the techniques of approximatedynamic programming. It is important to realize thatour strategy represents a significant departure fromclassical modeling and algorithmic strategies usedin the approximate dynamic programming literature(notably Bertsekas and Tsitsiklis 1996 and Sutton andBarto 1998), which approximate the value functionaround the predecision state. This strategy still leavesus with the challenge of computing or approximatingthe expectation. We avoid this by directly approximat-ing the value function around the postdecision state.

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


6.2. An Approximate Dynamic ProgrammingAlgorithm

Our solution strategy involves choosing an appro-priate functional form for the value functions. Ouridea is very similar to that used in Pereira and Pinto(1991) for water reservoir management, with the sig-nificant difference that we are using separable, piece-wise linear approximations of the value of resourcesin the future, where resources can be the amount ofenergy in storage as well as energy investments. Thiscontrasts with the use in Pereira and Pinto (1991) ofmultidimensional cuts based on Benders decomposi-tion, which carries with it provable convergence prop-erties (examples of convergence proofs are given inHigle and Sen 1991 and Ruszczynski 1993). We havecompared both of these strategies and found that theseparable approximations produced much faster con-vergence than approximations based on Benders cuts(see Powell et al. 2004 for an analysis in the context oftwo-stage problems) and very accurate solutions forcomplex multistage problems that arise in transporta-tion (see Powell and Topaloglu 2003, Topaloglu andPowell 2006). However, both strategies use the samebasic idea of forming piecewise linear approximationsbased on dual information.

Starting with initial value function approximations,we step forward in time, solving the decision problemat each time using the value function approximationof the future states. We repeat this procedure over sev-eral iterations, updating the value function approxi-mations after each iteration. This iterative updatingenables us to form an approximation of the expecta-tion in Equation (8) (actually, we only need to approx-imate the slopes). Although this algorithmic strategyis not generally optimal, there are convergence proofsfor special versions of the problem (discussed in §6.4below), and in §7 we benchmark against the linearprogramming formulation for a deterministic versionof the problem that demonstrates very good agree-ment.

Using some policy �, we may write the objectivefunction for time t as follows:

F �th 4Sth5 = −Cth4Sth1X

�th5−

∑

h′>h

Cth′4Sth′1X�th′5

−∑

t′>t

∑

h∈H

Ct′h4St′h1X�t′h5

= −Cth4Sth1X�th5+V x

th4Sxth51 (9)

where

V xth4S

xth5= −

∑

h′>h

Cth′4Sth′1X�th′5−

∑

t′>t

∑

h∈H

Ct′h4St′h1X�t′h5

is the same as the postdecision value function inEquation (8). Because this function is unknown, weuse an approximation that we denote as V x

th4Sxth5. We

now introduce the approximation that the value func-tion is separable in the postdecision capacity resourcestate Rx

th and reservoir level yxth. We express this

approximation as the sum of components denotingthe value of the capacity resource state and the valueof water remaining in the reservoir:

V xth4S

xth5 = V

capth 4Rx

th5+ Vhydroth 4yx

th5

=∑

a∈AConv

Vcap

tha 4Rxtha5+ V

hydroth 4yx

th50

It is easy to show that V 4S5 is concave in R andy, and we approximate V 4R5 and V 4y5 using sepa-rable, piecewise linear functions (using a very finediscretization, because the quantities are continuous).It is important to recognize that we are only con-cerned with the derivative of V x

th4Sxth5 rather than the

actual value.The hourly dispatch problem involves solving the

optimization problem

xdisp1nth = arg max

xth∈Xdisp1nth

4−Cdispth 4Sn

th1xth5+ Vhydroth 4yx

th551 (10)

where Snth is the specific state that we are in at hour h

and year t, while following sample path �n. The vari-able yx

th is a function of both Snth and x

dispth . Equation

(10) defines our policy �, and the space of policiesç is the set of all possible value function approxi-mations. The dispatch problem at time t is illustratedin Figure 2, where the value function approximation(the value of water stored in the reservoir) is given bya piecewise linear function. It is important to empha-size that this is a very small linear program (for aspatially aggregate model), which scales fairly eas-ily to spatially disaggregate applications (see Pow-ell et al. 2002, Powell and Topaloglu 2005, Topalogluand Powell 2006, and Simão et al. 2009 for numer-ous transportation applications). This is importantbecause we have to solve 8,760 of these for each yearof planning.

The annual capacity acquisition problem, solved atthe beginning of each year, is given by

xcap1nt0 = arg max

xcapt0

4−Ccapt0 4Sn

t01xt05+ Vcapt0 4Rx

t055 (11)

subject to

Rxt0 =Rt0 + x

cap1nt0 1 (12)

xcap1nt0 ≥ 00 (13)

This problem is illustrated in Figure 3. Here, we use avalue function approximation for each type of energyresource. The only decision of how much to purchaseis handled by the piecewise linear value function giv-ing the value of resources in the future.

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


Hour h

Coal. pulv. crb. cpt. SW

Coal. pulv. elect. SW

Solar. rsrc

Wind. rsrc

Hydro. rsrvrHydro. elect.

Wind. elect.

Nucl. rsrc

Elect. trans. SW

Elect. trans. CV

Resid. elect. distr

Resid. elect. dmnd

Export. elect. dmnd

Comm. elect. dmnd

Indust. elect. dmnd

Indust. elect. distr

Intermediate node

Capt. CO2

Vent. CO2

Coal. SW. rsrc

hydro xVth (yth)

Comm. elect. distr

Nucl. elect.

Solar. elect.

Coal. IGCC. crb. cpt. SW

Coal. IGCC. elect. SW

Figure 2 The Hourly Optimization Problem with Value Function Approximations for the Reservoir Level

Using ADP, we learn the value function approx-imations using iterative updating operations. Letn denote the iteration counter while following samplepath �n. For resource allocation problems, we havefound that it is effective to compute the marginal

2011 2012

cap xVt, oil (Rt, oil)

cap xVt, wind (Rt, wind)

cap xVt, etha (Rt, etha)

cap xVt, coal (Rt, coal)

Figure 3 The Annual Capacity Acquisition Problem with ValueFunction Approximations for the Future CapacityResource State

value of an additional unit of a resource (this can beenergy resources Rth or the energy in storage yth). Let

vcap1ntha = A sample estimate of the marginal value of

increasing Rtha by one unit computed whilefollowing sample path �n,

vhydro1ntha = A sample estimate of the marginal value of

increasing ytha by one unit computed whilefollowing sample path �n.

vhydro1ntha is computed relatively simply by finding

the marginal impact of the value of additional energyin storage for a particular hour. Computing v

cap1ntha

requires calculating the incremental value of addi-tional energy investment over an entire year. Thisrequires finding the incremental value of, say, addi-tional wind turbine generating capacity for each hourover the year, taking into account the presence of stor-age. This calculation allows us to capture the impactof hourly, daily, and seasonal variability, as well as themarginal impact of additional storage.

In SMART, we used the updating equations givenin Powell (2007, §11.3). Note that the value v

hydro1nth

or vcap1nth found at time t and iteration n is used to

update the value function approximation around thepostdecision state variable in the previous time period

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


(see Powell 2007, §4.4). We represent general updatingoperations for the value functions as follows:

Vhydro1nt1h−1 4yx

t1h−15←U V 4Vhydro1n−1t1h−1 1yx1n

t1h−11 vhydro1nth 51

Vcap1nt1h−1 4R

xt1h−15←U V 4V

cap1n−1t1h−1 1Rx1n

t1h−11 vcap1nth 50

The details of the calculation of the derivativesv

hydro1nth and v

cap1nth are somewhat involved and are

given in §§A.2 and A.3, respectively, of the OnlineSupplement.

A complete summary of the approximate dynamicprogramming algorithm is given in Figure 7 inthe Online Supplement (moved because of spaceconstraints).

6.3. DiscussionThe essential feature of the algorithm is that it stepsforward through time, using a Monte Carlo sam-ple of any random variable, and solves sequences ofvery small linear programs. In this sense, the modelscales well to handle more complex information pro-cesses, or a spatial representation of the problem (forexample, dividing the country into 10 to perhaps 100regions). If the model is being used over a 30-yearhorizon, in hourly increments, we would have tomodel more than 250,000 time periods. This is quitemanageable for an aggregate model (as described inour experimental work), but CPU times would start tobecome an issue with a sufficiently fine-grained spa-tial model.

Simulating a policy forward in time makes it veryeasy to handle considerable complexity in the tran-sition function. We can handle complexities such asexogenous information such as prices, which dependon decisions about the supply of energy generat-ing capacity. We can also handle autocorrelations inwind, price, and demand processes. However, addingdimensions to the state variable can potentially com-plicate the process of making decisions. Ideally, wewould like to calculate a value function that hasall the dimensions of the state variable, but at thesame time, we have to recognize that these additionaldimensions may have little or no effect on the qualityof a decision. These issues represent important areasof research that are beyond the scope of this paper.

We have illustrated the logic using value functionapproximations that ignore other aspects of the statevariable such as the state of climate (whether globalwarming is progressing faster than expected), tech-nology (whether there was a significant change in theperformance of batteries or the cost of extracting andsequestering carbon), markets (whether the price ofoil is over $150), and government policy (whetherthere is a carbon tax).

We can handle these dimensions, in theory, by sim-ply replacing the separable piecewise linear value

functions Vcapth 4Rx

th5 and Vhydroth 4yx

th5 with functions thatalso depend on the state variable �th, as in V

capth 4Rx

th �

�th5 and Vhydroth 4yx

th � �th5, respectively. The algorithmwould be virtually unchanged (see the storage prob-lem described in Nascimento and Powell 2009 foran illustration). The vector �th evolves exogenouslyover time and is already being computed. All thatis needed is indexing the value functions by thisinformation.

Of course, this is much easier to write than toimplement. The number of possible values of �th isextremely large, and as a result, finding a value func-tion approximation V

capth 4Rx

th � �th5 for each �th is com-putationally impossible when �th has more than twoor three dimensions. There is a wide range of sta-tistical strategies for handling this problem (see, forexample, Hastie et al. 2009) that represent promisingavenues for research. There is also a growing base ofliterature in approximate dynamic programming thatdraws on techniques such as kernel regression (seeOrmoneit and Sen 2002, Fan and Gijbels 1996) thatoffer considerable promise.

6.4. Convergence AnalysisOur solution strategy is not optimal, but we offer boththeoretical arguments and, in §7, empirical evidenceto support the claim that the quality of the solution isextremely good.

First, the algorithm for optimizing storage over asingle year is an instance of a provably convergentalgorithm, as given in Nascimento and Powell (2011)(building on the same proof technique as that givenin Nascimento and Powell 2009). This proof techniqueapplies even if we index the value functions on theexogenous state vector �th. Central to this convergenceproof is that the storage (which is the only meansby which one subproblem interacts with another) isa scalar (the amount stored in a reservoir). Just thesame, optimizing the storage in water reservoirs isa particularly difficult problem, because we need todecide whether to store additional water in the rainymonths of February and March to satisfy needs laterin the summer, thousands of hours into the future.

Second, the use of separable, piecewise linearapproximations has been proven to be optimal fortwo-stage problems if the second stage is separa-ble and near optimal even when the second stageis nonseparable (see Powell et al. 2004). Separableapproximations are not, in general, optimal for non-separable, multistage resource allocation problems.However, we can obtain provably convergent algo-rithms for multistage problems. If we were to intro-duce other forms of storage, the algorithm would onlybe an approximation because of our use of separa-ble functions. For practical uses, however, separableapproximations have been found to scale very well tohigh-dimensional problems with fast convergence.

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


Theoretical convergence proofs, although reassur-ing, are no guarantee of empirical performance. Ourresearch has been preceded by extensive experimen-tal research with algorithms of this type for trans-portation applications (see Godfrey and Powell 2002a,Topaloglu and Powell 2006). In the next section, wereport on experimental comparisons of the approxi-mate dynamic programming algorithm on determin-istic and stochastic versions of the problem.

7. ExperimentsIn this section, we design experiments that focus onthe ability of the ADP algorithm in §6.2 to learn near-optimal policies in terms of how much to store andwhat investment decisions to make. We begin by com-paring ADP to linear programming (LP) for a deter-ministic version of the problem using an aggregatenational model, which is small enough that we cansolve the linear program. We also demonstrate thatthe algorithm can be run on a spatially disaggregatemodel and show that CPU times scale sublinearly asthe size of the network grows. We then demonstratethe algorithm on a stochastic problem. Our problem,with more than 175,000 time periods, is far beyondthe capabilities of any known algorithm for stochasticoptimization, and as a result, we do not have an opti-mal benchmark for this problem; we do show, how-ever, that SMART produces robust behaviors.

We begin in §7.1 by describing the performanceindices that we use to evaluate our algorithmic strat-egy. Section 7.2 reports on CPU times as we increasethe number of time periods and as we make thetransition from a spatially aggregate to a disaggre-gate network. In §7.3, we describe our experimentsin a purely deterministic setting, where we run testsover a single-year planning horizon with determinis-tic precipitation. In §7.4, we discuss experiments withstochastic precipitation scenarios. In §7.5, we focus onexperiments on capacity acquisition in a multidecadesetting.

7.1. Performance MeasuresIn this section, we design performance measures todetermine how closely the ADP algorithm matchesthe optimal solutions generated by an LP solver. Weuse three types of policies to construct our perfor-mance measures: ADP (the solution produced usingapproximate dynamic programming), OPT (the opti-mal solution of the deterministic problem producedby solving the linear program), and MYOPIC, whichis the result of a myopic policy (the same as ADPwith value function approximations set to zero). Themyopic policy is used only to compute one of theperformance metrics. Below, we let strategy s refer toADP, OPT, or MYOPIC.

We define the following terms that will be used forcomputing the statistics that we measure.

F s = The objective function value as computed usingstrategy s. For example, F MYOPIC denotes themyopic solution, and F OPT represents the opti-mal objective function value.

xstha = The total usage corresponding to resource node

with attribute vector a ∈ARes (aType ∈ 8coal, solar,nuclear, 0 0 09) in hour h of year t according tostrategy s

=∑

a′∈−→A 4a5

xsth1aa′ .

ysth = The reservoir level in hour h of year t obtained

by following strategy s.ca = Cost of adding a unit of capacity of type a.xta = Amount of capacity type a added in year t

using ADP.x∗ta = The optimal amount of capacity type a to be

added in year t.z∗a = The optimal amount of capacity type a that is

used in the starting year.

We use these to compute the following measures.General performance indices

Es1 = The error in the objective function (using strat-

egy s) from the optimal objective function as apercentage of the optimal

= F s − F OPT5/F OPT × 100%.Es

2 = The error in the objective function (using strat-egy s) as a percentage of the error in the objectivefunction for a myopic policy

= Es1/E

MYOPIC1 × 100%.

Performance indices for hydro storage

Rs1 =

∑

t1h1a �xstha − xOPT

thi �∑

t1h1a �xOPTtha �

1

Rs2 =

∑

t1h4ysth − yOPT

th 52

∑

t1h4yMYOPICth − yOPT

th 520

Indices for capacity matching

Rcap1 =

∑

a ca∑

t �xta − x∗ta�

T∑

a caz∗a

× 100%1

Rcap2 =

∑

a ca�∑

t xta −∑

t x∗ta�

T∑

a caz∗a

× 100%0

Es2 is a good indicator of the improvement in the

solution quality obtained by using strategy s com-pared to a myopic solution. Rs

1 measures the deviationof the solution from the optimal as a fraction of theoptimal solution. Rs

2 is a measure of how well strat-egy s improves over a myopic solution of the storageproblem.

7.2. Execution Times and ScalabilityA major feature of approximate dynamic program-ming is that it can scale to handle important dimen-sions such as fine-grained temporal and spatial

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


modeling, as well as uncertainty. In this section, wesummarize CPU times as we make the transition tohandle problems with more than 175,000 time peri-ods and as we move from a spatially aggregate to adisaggregate model. All runs were performed on a2.5 GHz, Intel Xeon processor with access to 64 GB ofmemory. The linear program was solved using CPLEXVersion 12. The model is coded in Java.

We first ran a series of tests to demonstrate theability of the ADP algorithm to handle a large num-ber of time periods, which we compare to solvinga single large deterministic model. We emphasizethat the energy modeling community feels that it isvery important to model wind at hourly increments,because longer increments have the effect of averag-ing out the peaks and valleys that plague this impor-tant energy source. Figure 4 shows the computationaltime in hours to solve the linear program as a functionof the number of years. With a horizon of four years(35,000 time periods), the linear program using theaggregate network required 20 hours, growing super-linearly with the horizon.

To compare ADP to the linear programming solu-tion, we first performed a series of comparisons of theADP objective function and solution (capacity invest-ment and the amount held in storage). We found,using the aggregate model and a deterministic dataset, that we could always obtain solutions within 1%(and often within a fraction of a percent) using 300iterations. Using this setting, Figure 4 shows that ADPcan solve the four-year horizon in under an hour.

We next turned to our spatially disaggregate model.This model represents energy generation and con-sumption at the level of the 48 states in the con-tinental United States, plus demand in the District

0.00

5.00

10.00

15.00

20.00

25.00

4321

CPU

tim

e (h

ours

)

Years in simulation horizon

Linear program (optimal)

Approximate dynamic programming

Figure 4 Time to Solve the Energy Model as a Function of the Numberof Years in the Simulation Model

Note. With the increasing number of years in the model, run times increasedramatically in order to obtain an optimal solution, whereas ADP producesnear-optimal results in a fraction of that time.

Table 1 Size of the Linear Programs for the Aggregate andDisaggregate Networks (for a Single Time Period), CPU Timesfor First Iteration Over a Year, for 20 Iterations, Average OverLast 19 Iterations, and the CPU Time per Year Normalized bythe Number of Rows in the Linear Program

Aggregate data set Disaggregate data set

Rows 198 401229Columns 174 391705CPU secs/first iteration 11073 1141202CPU secs/20 iterations 18007 22111908CPU secs/iteration 80893 1108909CPU secs/row/iteration 00045 00027

of Columbia. It covers four types of energy supply(coal, nuclear, natural gas, and wind) and four typesof energy demand (residential, commercial, industrial,and transportation). The network makes it possible toconsider different types of technologies (e.g., invest-ments in coal with carbon capture and sequestration).The resulting network has approximately 40,000 rowsand columns.

Table 1 shows the size of the linear programs for theaggregate and disaggregate data sets. It then showsthe CPU time (in seconds) for simulating a single year(8,760 subproblems), for 20 iterations (of a year), theCPU seconds per iteration (as an average over thelast 19 iterations), and the CPU seconds per iterationdivided by the number of rows in the linear program.The last statistic provides an indication of how theCPU times grow with the size of the linear program.We note that the model builds the full linear programand solves it from scratch only once each year; forthe remaining 8,759 subproblems, the only changesthat occur are in the costs and right-hand sides ofthe constraints, making it easy to use the previoussolution.

The table shows that the CPU time per row, peryear of simulation, actually drops as the problem sizegrows, from 0.045 CPU seconds/row (based on anaverage over the 20 year run) for the small aggregatedata set to 0.027 CPU seconds/row for the large dis-aggregate data set.

Because ADP scales linearly with the number ofiterations and the number of time periods, we canquickly estimate run times as we increase the num-ber of iterations. Three hundred iterations of the smallnetwork over a single year (which provided near-optimal solutions when we compared the small net-work to the optimal solution) can be run in under anhour, making it possible to perform studies of windpenetration with different assumptions on storage. Ifthe same run is extended over 20 years (to captureinvestment decisions), the run time grows to 15 hours,which is certainly manageable as an overnight run.Three hundred iterations of the large network over anentire year, although hopelessly intractable as a single

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


deterministic model (350 million rows and columns),can be run in approximately 90 hours using ADP. Thiscompares well to reports of run times of 30 days forsome strategic planning models that solve daily-unitcommitment models over a year, but at this point, itcertainly seems to be an opportunity for parallel com-putation, an avenue we intend to pursue.

7.3. Experiments on Hydro StorageThe first set of experiments focuses on testing howwell the hydro-usage behavior of the solution fromthe ADP algorithm matches the optimal solution fromthe LP model using our aggregate dispatch model,under the assumption of deterministic precipitation.We focus on hydro storage because this is the mostdifficult form of storage from an algorithmic perspec-tive. The reason for this complexity is that we mayput water in storage in March so that it can be usedin August, thousands of hours in the future, which isa far more complex problem than storing energy in abattery at night so that it can be used during the day.We only solve the problem over a single year, becausethe reservoir empties out at least once each year.

We conducted our experiments in a variety of deter-ministic settings. The demands were (1) constant,

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

Optimal from linear program

(a) Optimal: Constant demand (b) ADP: Constant demand

(c) Optimal: Sinusoidal demand (d) ADP: Sinusoidal demand

Reservoir level Reservoir level

Reservoirlevel

Reservoirlevel

Rainfall

Rainfall Rainfall

Rainfall

Demand

Demand

Demand

Demand

Approximate dynamic programming solution

Approximate dynamic programming solution

Optimal from linear program

Figure 5 Optimal Reservoir Levels (a) and (c) and Reservoir Levels from the ADP Algorithm (b) and (d) for Constant and Sinusoidal Demand

(2) sinusoidally varying, or (3) varying with no appar-ent pattern. Another setting that could be variedwas whether or not wind (varying by the hour)was included as a competing source of energy. Wetested our algorithm for a one-year horizon withstorage value functions computed for every 10-hourincrement. We then repeated the test for one-hourincrements. For some problems, this means that thedecision of how much water to draw from the reser-voir in March had to reflect usage behavior in July,thousands of hours into the future.

Figure 5 shows the reservoir level over the courseof a one-year period for a deterministic problem pro-duced by solving the linear program, which gives anoptimal solution (Figure 5, panels (a) and (c)), andfrom the ADP algorithm (Figure 5, panels (b) and(d)). These were run for constant demand (Figure 5,panels (a) and (b)) and a sinusoidal demand (Fig-ure 5, panels (c) and (d)). Note that the heaviest pre-cipitation occurs during late winter and spring. Themodel used a slight penalty for holding water in thereservoir so that the optimal solution was to buildup reserves as late as possible (this choice was madepurely for algorithmic testing purposes). With this

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


Table 2 Performance Statistics of the ADP Algorithm for Different Problem Instances of a One-Year EnergyModel with the Dispatch Problem Solved Every H Hours

Data set Demand type H (hrs/time period) Wind EADP1 (%) EADP

2 (%) RADP1 RADP

2

1 Constant 10 0 0.0008 2.26 0.02 0.0062 Constant 10 Ø 0.0003 0.63 0.04 0.0073 Sinusoidal 10 0 0.0004 0.02 0.04 0.0844 Sinusoidal 10 Ø 0.0613 2.67 0.04 0.0075 Fluctuating 10 0 0.0004 0.04 0.03 0.0116 Fluctuating 10 Ø 0.0000 0.01 0.03 0.0057 Constant 1 0 0.0001 0.06 0.06 0.0018 Constant 1 Ø 0.0001 0.08 0.07 0.0019 Sinusoidal 1 0 0.0012 0.12 0.03 0.002

10 Sinusoidal 1 Ø 0.0020 0.20 0.03 0.00111 Fluctuating 1 0 0.0000 0.01 0.02 0.00112 Fluctuating 1 Ø 0.0001 0.02 0.01 0.011

Note. A zero under wind means no wind, and a check mark means wind was included.

setup, the algorithm realized that it was necessary tobegin building up reservoirs in the March–April timeframe to meet needs that would occur thousands ofhours into the future. The results show an extremelyclose match between the optimal solution and thatproduced by the ADP algorithm.

In Table 2, we summarize the performance statis-tics of the ADP algorithm. We observe that the ADPalgorithm is able to produce solutions that are con-sistently within a very small margin of error of theoptimal solutions.

7.4. Hydro Storage with Stochastic PrecipitationWe now consider the case where the precipitation isstochastic. We generate 50 different precipitation sce-narios for the year, and we use these to train the valuefunction approximations for approximate dynamicprogramming. In Figure 6, we show the storage levelproduced by ADP for a single sample path, alongwith the storage level for each sample path whenoptimized knowing the entire sample path. The ADP

Res

ervo

ir le

vel

Time period

ADP at last iteration

Optimal for individualscenarios

80070060050040030020010000

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

Figure 6 Reservoir Level Produced by ADP (Last Iteration) forStochastic Precipitation, and Optimal Reservoir LevelsProduced by LP When Optimized Over Individual Scenarios

solution has the intuitive behavior of building up stor-age earlier in the season to prepare for potentiallylower rainfalls later in the season. By contrast, whenwe are allowed to optimize given the entire rainfalltrajectory, the reservoir levels are held at lower lev-els until later in the season, reflecting the benefits ofadvance information. We believe that this illustratesthe central weakness of models that sample uncertaininformation over the entire horizon, and we then findthe optimal solution given advance information.

7.5. Experiments on Capacity AcquisitionIn the final set of experiments, we test the ability ofthe ADP algorithm to optimize capacity acquisitiondecisions over a multidecade horizon under deter-ministic conditions. Given an initial set of capacitiesat the various conversion nodes, we allow the aver-age demand to increase over the years. Eventually, thecurrent level of capacities will be insufficient to coverthe growing demands for energy. The ADP algorithmwas found to produce a solution within 2.5% of theoptimal solution after 100 iterations, 0.70% after 200iterations, and 0.24% after 500 iterations.

In Table 3, we summarize the performance statisticsafter 500 iterations of the ADP algorithm for a numberof problem scenarios. Here again, the ADP algorithmis seen to approach within 1% of the optimal solutionquality for a variety of deterministic data sets.

Table 3 Performance Statistics of the ADP Algorithm for DifferentProblem Instances of a Multiyear Energy Model

Data set Demand type Wind Years EADP1 (%) EADP

2 (%) Rcap1 Rcap

2

1 Constant 0 5 0.03 0.85 0.008 0.0082 Constant 0 20 0.52 1.37 0.031 0.0673 Constant Ø 20 0.24 0.72 0.027 0.061

4 Fluctuating 0 20 0.34 0.92 0.029 0.0615 Fluctuating Ø 20 0.30 0.91 0.026 0.056

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


8. ConclusionsWe have presented a stochastic, multiscale energyresource planning model that handles competingenergy resources and demands, a wide range of formsof uncertainty, and the ability to model hourly varia-tions over a multidecade horizon. The computationalresearch shows that approximate dynamic program-ming can produce solutions that closely match theoptimal solution of a deterministic linear program-ming model while also providing solutions for prob-lems that are far larger than what can be solved as alinear program.

The ADP model uses a proper model of the flow ofinformation and decisions, and as a result, it avoidsthe shortcuts that have characterized many of theefforts at introducing uncertainty in optimization-based models. The experimental work shows thatADP produces more robust behaviors, such as storingrainfall earlier in the season, than any of the solutionsfrom a deterministic model acting on a specific sce-nario. It also exhibits nonanticipative behavior whenfaced with a major source of uncertainty, such as theintroduction of a carbon tax. Although our algorithmis supported by some important theoretical results,we do not have a provably optimal algorithm forstochastic data sets that can serve as a comparablebenchmark.

We show that the model can scale to handle spa-tially disaggregate problems, using a data set thatmodels each state in the continental United States. Weshow that the model scales sublinearly with respectto the number of rows in the linear program for eachtime period, using the results of two networks with200 and 40,000 rows.

The major unresolved algorithmic issue involvesthe handling of coarse-grained uncertainty, such asthe imposition of a carbon tax or a breakthroughin technology. ADP has no difficulty simulating dif-ferent scenarios describing these sources of uncer-tainty, but estimating a value function approximationthat takes these issues into account in a proper wayrepresents a new area of research. However, usingsuitably designed approximation strategies, ADP canproduce good solutions that are nonanticipative (donot peek into the future) and that properly capturethe dynamics in a realistic way.

ReferencesArchibald, T. W., K. I. M. McKinnon, L. C. Thomas. 1997. An

aggregate stochastic dynamic programming model of multi-reservoir systems. Water Resources Res. 33(2) 333–340.

Bertsekas, D. P., J. N. Tsitsiklis. 1996. Neuro-Dynamic Programming.Athena Scientific, Belmont, MA.

Birge, J. R., F. Louveaux. 1997. Introduction to Stochastic Program-ming. Springer-Verlag, New York.

Brown, P. D., J. A. Peas-Lopes, M. A. Matos. 2008. Optimizationof pumped storage capacity in an isolated power system withlarge renewable penetration. IEEE Trans. Power Systems 23(2)523–531.

Brunetto, C., G. Tina. 2007. Optimal hydrogen storage sizing forwind power plants in day ahead electricity market. IET Renew-able Power Generations 1(4) 220–226.

Castronuovo, E. D., J. A. P. Lopes. 2004. On the optimization ofthe daily operation of a wind-hydro power plant. IEEE Trans.Power Systems 19(3) 1599–1606.

Cervellera, C., V. C. P. Chen, A. Wen. 2005. Optimization of a large-scale water reservoir network by stochastic dynamic program-ming with efficient state space discretization. Eur. J. Oper. Res.171(3) 1139–1151.

De Ladurantaye, D., M. Gendreau, J.-Y. Potvin. 2007. Optimizingprofits from hydroelectricity production. Comput. Oper. Res.36(2) 499–529.

Edmunds, T. A., J. F. Bard. 1990a. A decomposition technique fordiscrete time optimal control problems with and applicationto water resources management. Math. Comput. Modelling 13(1)61–78.

Edmunds, T. A., J. F. Bard. 1990b. Time-axis decomposition of large-scale optimal control problems. J. Optim. Theory Appl. 67(2)259–277.

Fan, J., I. Gijbels. 1996. Local Polynomial Modelling and Its Applica-tions. Chapman & Hall, London.

Fishbone, L. G., H. Abilock. 1981. MARKAL—A linear program-ming model for energy systems analysis: Technical descriptionof the BNL version. Internat. J. Energy Res. 5(4) 353–375.

Foufoula-Georgiou, E., P. K. Kitanidis. 1988. Gradient dynamicprogramming for stochastic optimal control of multidimen-sional water resources systems. Water Resources Res. 24(8)1345–1359.

Garcia-Gonzalez, J., de la Muela, L. M. Santos, A. M. Gonzalez.2008. Stochastic joint optimization of wind generation andpumped-storage units in an electricity market. Power Systems,IEEE Trans. 23(2) 460–468.

Godfrey, G. A., W. B. Powell. 2002a. An adaptive dynamic pro-gramming algorithm for dynamic fleet management, I: Singleperiod travel times. Transportation Sci. 36(1) 21–39.

Godfrey, G. A., W. B. Powell. 2002b. An adaptive dynamic program-ming algorithm for dynamic fleet management, II: Multiperiodtravel times. Transportation Sci. 36(1) 40–54.

Gröwe-Kuska, N., H. Heitsch, W. Römisch. 2003. Scenario reductionand scenario tree construction for power management prob-lems. A. Borghetti, C. A. Nucci, M. Paolone, eds. IEEE BolognaPower Tech Conf. Proc., Vol. 3. IEEE Press, Los Alamitos, CA.

Hastie, T., R. Tibshirani, J. Friedman. 2009. The Elements of Statis-tical Learning: Data Mining, Inference and Prediction. Springer,New York.

Higle, J. L., S. Sen. 1991. Stochastic decomposition: An algorithmfor two-stage linear programs with recourse. Math. Oper. Res.16(3) 650–669.

Higle, J. L., S. Sen. 1996. Stochastic Decomposition: A Statistical Methodfor Large Scale Stochastic Linear Programming. Kluwer AcademicPublishers, Dordrecht, The Netherlands.

Hogan, W. W. 1975. Energy policy models for project independence.Comput. Oper. Res. 2(3–4) 251–271.

Holttinen, H. 2004. The impact of large scale wind. Ph.D. thesis,VTT Technical Research Center, Espoo, Finland.

Holttinen, H., P. Meibom, A. Orths, B. Lange, M. O’Malley, J. O.Tande, A. Estanqueiro et al. 2011. Impacts of large amounts ofwind power on design and operation of power systems, resultsof IEA collaboration. Wind Energy 14(2) 179–192.

Holttinen, H., P. Meibom, A. Orths, F. Van Hulle, B. Lange, M. O.Malley, J. Pierik et al. 2009. Design and operation of powersystems with large amounts of wind power: State-of-the-artreport. VTT Working Paper 82, VTT Technical Research Center,Espoo, Finland.

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.


Høyland, K., S. W. Wallace. 2001. Generating scenario trees for mul-tistage decision problems. Management Sci. 47(2) 295–307.

Jacobs, J., G. Freeman, J. Grygier, D. Morton, G. Schultz,K. Staschus, J. Stedinger. 1995. SOCRATES—A system forscheduling hydroelectric generation under uncertainty. Ann.Oper. Res. 59(1) 99–133.

Johnson, S. A., J. R. Stedinger, K. Staschus. 1991. Heuristic operatingpolicies for reservoir system simulation. Water Resources Res.27(5) 673–685.

Johnson, T. L., J. F. DeCarolis, C. L. Shay, D. H. Loughlin, C. L.Gage, S. Vijay. 2006. MARKAL scenario analyses of technologyoptions for the electric sector. Technical report, U.S. Environ-mental Protection Agency, Washington, DC.

Kanudia, A., R. Loulou. 1998. Robust responses to climate changevia stochastic MARKAL: The case of Québec. Eur. J. Oper. Res.106(1) 15–30.

Kaut, M., S. W. Wallace. 2003. Evaluation of scenario-generationmethods for stochastic programming. Pacific J. Optim. 3(2)257–271.

Korpaas, M., A. T. Holen, R. Hildrum. 2004. Operation and sizingof energy storage for wind power plants in a market system.Internat. J. Electrical Power Energy Systems 25(8) 599–606.

Lamond, B. F., M. J. Sobel. 1995. Exact and approximate solutionsof affine reservoir models. Oper. Res. 43(5) 771–780.

Lamont, A. 1997. User’s guide to the META∗Net economic model-ing system version 2.0. Technical report, Lawrence LivermoreNational Laboratory, Livermore, CA.

Lamont, A. D. 2008. Assessing the long-term system value of inter-mittent electric generation technologies. Energy Econom. 30(3)1208–1231.

Loulou, R., G. Goldstein, K. Noble. 2004. Documentation for theMARKAL family of models. Technical report, Energy Tech-nology Systems Analysis Programme, International EnergyAgency, Paris.

Messner, S., M. Strubegger. 1995. User’s guide for MESSAGE III.Technical report, International Institute for Applied SystemsAnalysis, Laxenburg, Austria.

Messner, S., A. Golodnikov, A. Gritsevskii. 1996. A stochastic ver-sion of the dynamic linear programming model MESSAGE III.Energy 21(9) 775–784.

Nandalal, K. D. W., J. J. Bogardi. 2007. Dynamic ProgrammingBased Operation of Reservoirs. Cambridge University Press,Cambridge, UK.

Nascimento, J. M., W. B. Powell. 2009. An optimal approximatedynamic programming algorithm for the lagged asset acquisi-tion problem. Math. Oper. Res. 34(1) 210–237.

Nascimento, J. M., W. B. Powell. 2011. An optimal approximatedynamic programming algorithm for the energy dispatchproblem with grid-level storage. Technical report, Departmentof Operations Research and Financial Engineering, PrincetonUniversity, Princeton, NJ.

NEMS. 2003. The National Energy Modeling System: An overview2003. Technical report, Energy Information Administration,Office of Integrated Analysis and Forecasting, U.S. Departmentof Energy, Washington, DC.

Ormoneit, D., S. Sen. 2002. Kernel-based reinforcement learning.Machine Learning 49(2–3) 161–178.

Paatero, J. V., P. D. Lund. 2005. Effect of energy storage on varia-tions in wind power. Wind Engrg. 8(4) 421–441.

Pereira, M. V. F., L. M. V. G. Pinto. 1991. Multistage stochasticoptimization applied to energy planning. Math. Programming52(1–3) 359–375.

Powell, W. B. 2007. Approximate Dynamic Programming: Solving theCurses of Dimensionality. John Wiley & Sons, Hoboken, NJ.

Powell, W. B., H. Topaloglu. 2003. Stochastic programming intransportation and logistics. A. Ruszczynski, A. Shapiro, eds.Stochastic Programming. Handbooks in Operations Research andManagement Science, Vol. 10. Elsevier, Amsterdam, 555–636.

Powell, W. B., H. Topaloglu. 2005. Fleet management. S. W.Wallace, W. T. Ziemba, eds. Applications of Stochastic Program-ming. Society for Industrial and Applied Mathematics/MPS,Philadelphia, 185–216.

Powell, W. B., A. Ruszczynski, H. Topaloglu. 2004. Learning algo-rithms for separable approximations of discrete stochastic opti-mization problems. Math. Oper. Res. 29(4) 814–836.

Powell, W. B., J. A. Shapiro, H. P. Simão. 2002. An adaptive dynamicprogramming algorithm for the heterogeneous resource alloca-tion problem. Transportation Sci. 36(2) 231–249.

Ruszczynski, A. 1993. Parallel decomposition of multistagestochastic programming problems. Math. Programming 58(1–3)201–228.

Simão, H. P., J. Day, A. P. George, T. Gifford, J. Nienow, W. B. Pow-ell. 2009. An approximate dynamic programming algorithm forlarge-scale fleet management: A case application. TransportationSci. 43(2) 178–197.

Sioshansi, R. 2010. Evaluating the impacts of real-time pricing onthe cost and value of wind generation. IEEE Trans. Power Sys-tems 25(2) 741–748.

Sioshansi, R., W. Short. 2009. Evaluating the impacts of real-timepricing on the usage of wind generation. IEEE Trans. PowerSystems 24(2) 516–524.

Sutton, R. S., A. G. Barto. 1998. Reinforcement Learning: An Introduc-tion. MIT Press, Cambridge, MA.

Swisher, J. R., P. D. Hyden, S. H. Jacobson, L. W. Schruben. 2000.A survey of simulation optimization techniques and proce-dures. Proc. Winter Simulation Conf., Orlando, FL, IEEE Press,Los Alamitos, CA, 119–128.

Tagher, N. P. 2010. Powering America: Optimizing electricity gen-eration for the United States until 2030. Ph.D. thesis, PrincetonUniversity, Princeton, NJ.

Topaloglu, H., W. B. Powell. 2006. Dynamic programming approx-imations for stochastic, time-staged integer multicommodity-flow problems. INFORMS J. Comput. 18(1) 31–42.

Van Slyke, R. M., R. Wets. 1969. L-Shaped linear programs withapplications to optimal control and stochastic programming.SIAM J. Appl. Math. 17(4) 638–663.

Wallace, S. W., S.-E. Fleten. 2003. Stochastic programming modelsin energy. A. Ruszczynski, A. Shapiro, eds. Stochastic Program-ming. Handbooks in Operations Research and ManagementScience, Vol. 10. Elsevier, Amsterdam, 637–677.

Copyright:

INFORMS

holdsco

pyrig

htto

this

Articlesin

Adv

ance

version,

which

ismad

eav

ailableto

subs

cribers.

The

filemay

notbe

posted

onan

yothe

rweb

site,includ

ing

the

author’s

site.Pleas

ese

ndan

yqu

estio

nsrega

rding

this

policyto

perm

ission

s@inform

s.org.

smart: a stochastic multiscale model for the analysis of ...powell et al.: a stochastic multiscale...

Documents