review of the ercot december 18, 2008 nodal cost benefit study

8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study

1/16

McCulloughResearch

ReviewoftheERCOT

December18,2008Nodal

CostBenefitStudy

January7,2009

RobertMcCullough


2/16

ReviewoftheDecember18,2008NodalCostBenefitStudy

January7,2009

RobertMcCullough

2|P a g e

OverviewOnDecember18,2008,CRA InternationalandReseroConsultingreleasedastudythatconcludedthat

the benefits of continuing the shift to nodal markets from the existing zonal system significantly

outweighedthecosts. Theyreportedbenefitsintwodifferentforms. First,theareaofTexasservedby

ERCOTwouldexperiencea$520millioncostreduction,whilerequiringonly$222million inadditional

computer programming costs. Second, the new market design would reduce costs to ERCOTs

consumersby$5.6billion.

Acarefulreviewrevealsseriousquestionsconcerningitsaccuracyandcraftsmanship. Thescopeofthe

new study is 2011 to 2020, although only the first two years are actually modeled. The period from

2013through2020issimplyarepeatedcutandpastefromtheresultsfor2011and2012. Moreover,

thebenefitsduetothesitingofnewgenerationaresimplyassumedfromtheresultsofthe2004cost

benefitstudy. Overall,only17%ofthe$520millionsavingsactuallyresultsfromthecomputermodel

described in the report. The remaining 83% of the benefits are the product of various ad hoc

adjustments.

Similarly, the $5.6 billion in consumer benefits are also based on modeling that stops in 2012 even

thoughresultsarereportedthrough2020. Thissurprisingresultthatashift incomputeralgorithms

will dramatically reduce revenues for electric generators reproduces the conclusions of the 2004

study. In both cases, the result relies on the assumption that generator bids are fixed at short term

marginal costs. Unfortunately, this is not the case inTexasandother areas thathave adoptednodal

pricing.

Strategicbidding

(bids

vastly

higher

than

marginal

costs)

is

acentral

feature

in

ERCOT.

In

any

market where bids can be easily changed to recapture the assumed revenue loss, such gains for the

consumerarebothspeculativeandeasilycorrectedbyadjustingthebids. The$5.6billionestimatealso

containsasignificanterroracriticalratioused inthecalculation isbasedonvaluestakenfromyears

beforenodalmarketsareactually implemented. Overall,the$5.6billiontracesonly17%tocomputer

modelingandwiththeresidual83%representingpoorlydocumentedadhocadjustments.

Noattemptwasmadetocalibratethestudytoactualconditions (knownasbackcasting). Nordidthe

authors estimate the sensitivity of their results to their assumptions. Since some assumptions were

unusual,the lackofsensitivitytesting isparticularlyproblematic. Thecostreduction inthetwoyears

actuallymodeled

is

only

.04%

of

annual

production

costs,

aslender

margin

in

aworld

where

the

cost

of

oilincreased100%in2008beforefalling75%bytheendoftheyear.

Theestimateof$222millioninadditionalcomputerprogrammingcostsisalsoproblematic. Inactuality,

theDecember18,2008studyassumes$429millioninadditionalcomputerprogrammingcoststhatare

offset by $207 million of additional computer programming costs to refresh the existing market

design. WhiletheDecemberstudyestimatesmaintenancecostsforthenewcomputerprogramas$14


3/16


4/16


January7,2009

RobertMcCullough

Atfirstglance,thistableappearstorepresentaneditorialdecisiontoomitthefullsetofresultsinthe

interestofbrevity. Acloserreadingmakesclearthattheseyearsaretheonlyyearsusedintheupdated

study:

TheNPVfrom2011to2020 isestimatedtobe$339million,assumingthatproduction

costsandresultingbenefitsobservedforthefirsttwoyearsofoperationremainatthe

samelevelonaveragethrough2020.2

ThecompletetablecanbereverseengineeredfromtheprojectedNPVonthefinallinesofTable5:

Table5:AnnualProductionCostbyScenario(inreal2008dollars)

ZonalCase NodalCase Benefit(Zonal Nodal)

($Million) ($Million) ($Million)

2009 12,928.6 12,892.0 36.6

2010 12,319.2 12,277.1 42.1

2011 12,211.2 12,164.4 46.8

2012 12,212.0 12,164.0 48.0

2013 12,211.2 12,164.4 46.82014 12,212.0 12,164.0 48.02015 12,211.2 12,164.4 46.82016 12,212.0 12,164.0 48.02017 12,211.2 12,164.4 46.82018 12,212.0 12,164.0 48.02019 12,211.2 12,164.4 46.8

2FinalReport,December18,2008,page9.

4|P a g e


5/16


January7,2009

RobertMcCullough

5|P a g e

2020 12,212.0 12,164.0 48.0AverageAnnual(20112012) 12,211.6 12,164.2 47.4

ProjectedNPV

(2011

2020)

86,378

86,039

339

Therowsadded in italicswerenotmodeled intheDecemberstudy,which insteadmodeled2011and

2012,andthenthesameresultsweresimplycutandpastedfortheremaining80%ofthestudyhorizon.

This unusual methodology is surprising given the problems that arose in the 2004 study in modeling

ERCOTsproductioncostsafter2012:

TCA believes that specific predictive conclusions should not be based on TCA results

obtained for 2013, and especially 2014. This is because the massive addition of new

generatingresourcesmodeledforoutyearsisnotsupportedbytransmissionupgrades.3

Thissame

conclusion

also

occurs

later

in

the

2004

report:

Themidtermtrendcontinuesforanotheryear(2012),butisreversedin2013and2014

duetodifficultiesassociatedwiththemodelingoffurthercapacityexpansiondecisions

intheabsenceoftransmissionupgrades.4

Table34intheNovember2004reportclearlyshowstheproblem:

3FinalReport,November24,2004,page316.



6/16


January7,2009

RobertMcCullough

In2013,thebenefitsofmovingtoanodalsystemarereducedbyover50%. Inthefollowingyearthe

resultsactuallyindicatethatthezonalsystemissuperiortothenodalsystem.Althoughitmaymerelybe

acoincidence,itisinterestingthattheDecember2008studystopsmodelingintheyearwhereproblems

aroseintheNovember2004study.

The second componentof nodalbenefits is the potentialadditional precisionofdata forpowerplant

siting.

The

December

2008

study

assumes

the

value

of

improved

generation

siting

by

a

particularly

arbitraryprocedure:

Assuming70% inadditionalbenefitsattributable to improvedgenerationsitingovera

periodof2013 through 2020 (when generation capacitywillbe needed inaddition to

6|P a g e


7/16


8/16


January7,2009

RobertMcCullough

8|P a g e

Only a small fraction of the reported results come from modeling using current loads, fuel prices,

generation, and transmission. The vast majority of the results come from ad hoc adjustments the

reuse of earlier years results in lieu of completing the modeling and proration of results from the

November2004study.ConsumerPaymentSavingsThe most significant numbers in the December 2008 study concern consumer benefits. The authors

indicate thatbetween2009and2012consumerswillseeanannualaverage$576.6million insavings

duetothereimbursementofcongestionrentsandotherbenefits. Thisvalueisthenratioedupby

the fraction that the average benefits had between 2005 and 2008 to total benefits estimated in

November2004.

This approach is extremely speculative. As noted above, the new study represents different years,

different resources, different fuel prices, and different transmission capabilities. To compound the

problems,theratiousedisbasedontwoyearswhenthenodalmarketwillnotbeoperating,sotheratio

itself, regardless of its legitimacy, is in error. Finally, the assumption addressed below, that ERCOT

generatorswillrestricttheirbidstoshorttermmarginalcost, isbothunrealisticand likelyto leadtoa

massiveoverestimateoftheconsumerpaymentsavings.

Thecalculationdescribedonpage34oftheDecember2008reportis:

1. Identifytheconsumerpaymentsavingsfor2005through2014fromtheNovember2004report$7.3billion.

2. Averagethecongestionrentsfor2009through2012fromtherevisedmodelusedintheDecember2008study.

3. Averagethecongestionrestsfor2005through2008fromtheoriginalmodelusedintheNovember2004report.

4. Dividethe2004modelresultbythe2008modelresultgettingafactor1.29.5. Dividethe2004valueforconsumerpaymentsavingsby1.29toget$5.6million.

Thiscalculation issounusualthat it isdifficulttoseeanyeconomic logic in itsderivation. There isno

reason

to

believe

that

there

is

any

logical

comparison

between

market

conditions

in

2005

through

2008

asestimated fouryearsagoandthecurrent forecast for2009 through2012. Even ifthemarkets

were similar, despite the changes in generation, fuel prices, and transmission, the appropriate

comparisonwoulduseyearsinwhichthenodalmarketwasactuallyoperating,not2009and2010when

ithasnotyetstarted.


9/16


January7,2009

RobertMcCullough

LocalCongestRentfor

2011and2012

17%

Ratioto2004Study's

Values

for

Consumer

Benefits

83%

AssumedValuesConstitutes83%ofTotalConsumerBenefitsfromShiftingtoNodalMarkets

Aside from the illogic of the calculation, the economics of this argument are also suspect. ERCOT is

characterized by a severely oligopolistic market structure of many buyers and few sellers. Strategic

biddingisacontinuingfeatureofERCOTsmarketswithbidsandpricesoftenatmultiplesofshortterm

marginal cost. If, as the authors assume, local congestion rents reduced the revenues of producers,

economic theory would lead one to expect that strategic bidding may increase to recover the

generatorsrevenues.

ThemajorgeneratorinERCOT,Luminant,routinelysubmitsbidsat$300/MWh. Wheninvestigatedby

theTexasPublicUtilitiesCommission,Luminantdefendeditsstrategicbiddingpracticesbyarguingthat

suchhighbidswererequiredtocoveritstotalcostsofgeneration. Whileitsargumentmighthavehada

poorgrounding ineconomics, it was a perfectly logical argument fora price leader in an oligopolistic

market. IfLuminantsactualcosts increasedafternodalmarketswereadoptedbyERCOT, it is logical

thatitwouldaddthecoststoitsstrategicbids,notpassivelyacceptareductioninincome.

9|P a g e


10/16


January7,2009

RobertMcCullough

10|P a g e

BackcastingAn important featureoftheNovember2004reportwasanattempttotest itsaccuracyagainstactual

pricesandoperationsfor2003. Thisisastandardstepforanycomplexcomputermodel. Backcasting

results for the November 2004 model were poor: overall ERCOT forecasts were 25.4 $/MWh against

actual prices of $29.1/MWh. This represents an error on theorderof15%,a fairly poorshowing for

forecastingthepreviousyear.

The December study dispenses with backcasting altogether. This omission is puzzling since the

December2008modelforecastedtwoyears(2009and2010)ofnospecialinteresttothependingpolicy

decision. These two years are not used in the calculations (with the exception of their erroneous

inclusionintheratiousedtoassumeconsumerbenefits)sincethenodalmarketisnotforecasteduntil

2011. Belowisanexplanationbyyear:

2007 Notmodeledasabackcast

2008 Notmodeledasabackcast

2009 Nodal market not in operation modeled but not used in the present value of

productioncostsavings

2010 Nodal market not in operation modeled but not used in the present value of

productioncostsavings

2011 Nodal market in operation modeled and used in present value of production cost

savings

2012 Nodal market in operation modeled and used in present value of production cost

savings

2013 Nodalmarketinoperationnotmodeledandresultsassumedtobeequalto20112012








11/16


January7,2009

RobertMcCullough

11|P a g e


Intheabsenceofabackcast,thereisnoevidencethattheDecemberstudyisabletoreproduceprices

andoperationsinERCOTwithanyaccuracywhatsoever.

AssumptionsTheauthorsstateonpage68:

A. Marginal Cost Bidding: All generation units are assumed to bid marginal cost(opportunity cost of fuel plus nonfuel VOM plus opportunity cost of tradable

permits).

To

the

extent

that

real

markets

are

not

perfectly

competitive,

the

model

tendstounderestimateprices.7

This decision to make this assumption is not surprising. The original November 2004 report

madeasimilarassumption:

The assumption of shortrun marginal cost bidding can be overridden, implementing

strategicbiddingbehavior,buttheeffortrequiredtodothisisconsiderable;priortothe

contractingprocess,theCBCGchosenottopursuethisapproach.Notethatthroughout

this report the use of the term marginal cost means refers to shortrun marginal

costs.8

Unfortunately,this isapoorassumptionforamodeldesignedtosimulatemarketprices. Ina

condition of perfect competition, bids will approximate marginal cost. The critical condition

precedentisperfectcompetition. ERCOTmeetsfewoftheconditionsofperfectcompetition,

andstrategicbiddingisacontinuingprobleminERCOTadministeredmarkets.

Recently,LuminantsettledacomplaintbyTexasregulatorsforstrategicbidding.9 Attheheart

of the investigation was the appropriateness of TXUs exercise of market power. In practice,

TXUtendstobidthevastmajorityofitsresourcesat$300/MWHregardlessofthemarginalcost

oftheunits.10



9NoticeofViolation,Docket34061,March28,2007.

10Ibid.,page26. Thetermsbidcurveandmarginalcostwereadded.


12/16


January7,2009

RobertMcCullough

TXUsstrategicbiddingwasobviously importantwitnessthecomplaintfiledbyPUCstaff. Moreover,

the deviation between the marginal cost of coal units and $300/MWh bids dwarfs any possible

efficiencyadvantageslikelytoresultfrommoreprecisepricinginthenodalmodel.

Modelingofstrategicbidding isachallenge. Oligopolisticpricingdecisionsarecomplexandvarywith

circumstances. As noted above, when the largest single market participant makes strategic bids for

thousands of megawatts, it is imprudent to ignore this information when modeling. In the example

takenfromthePUCcomplaint,thesignificantdifferenceinpricesisadifferenceof$200/MWh. Thebias

thisassumptionwillcreateinthemodelingdwarfstherelativelysmallproductioncostdifferencesinthe

December

2008

report.

The most important operational assumption after bidding behavior is the cost of natural gas. The

authorsdonotincludedetailedinformationonnaturalgasandcoalpricesusedintheirmodel.Thereis,

however,ashortdiscussionofthesourcesofthenaturalgasforecastsonpages68and69. Preparedin

a year of unprecedented fuel price volatility, the authors make the decision to base forecasts on the

2008 EIAs natural gas forecast published a year ago. The Energy Information Administration makes

forecastsonbothanannualandamonthlybasis. EIAsannualforecastsarereleasedinmidDecember,

12|P a g e


13/16


January7,2009

RobertMcCullough

thus making the 2008 forecast a full year out of date. While using forecasts from a year ago would

normally be defensible, 2008 was hardly a normal year. Within 2008 prices for natural gas used for

electricgeneration

increased

over

50%

by

July

2008

and

in

the

latter

portion

of

the

year,

they

fell

even

moreprecipitously.

$

$2.00

$4.00

$6.00

$8.00

$10.00

$12.00

$14.00

$/MWh

EIAShortTermandAnnualForecastsforNaturalGas

2008AEO December2008STEO

Forecasting fossil fuel prices has become increasingly difficult over the past few years and the use of

superannuatedforecastsisanextremelybadstepinanymodelingexercise. Notwithstanding,thedated

naturalgasforecast,theuseofasinglerelativelyoptimisticpointforecast isnolongerappropriate

regardless

of

its

source

or

age.

The

standard

approach

to

this

problem

is

to

prepare

sensitivities

for

high, medium, and low fossil fuel prices so that decisionmakers can see the impactof unpredictable

fossilfuelpricesonthemodelresults.

Additionalassumptionsandcalculationsarepoorlydocumentedormissing,including:

1. NPVcalculationsforsystemcosts2. NPVcalculationsforcustomersavings

13|P a g e


14/16


January7,2009

RobertMcCullough

3. UPLANresults4. GEMAPSresults5. SOxandNOxrequirementsandforecastedcosts6. Plantassumptions.

Theauthorsalsoneveraddresstherealdiscountrate,avaluethatoftendrivesresultsinmodelswhere

costsandbenefitsoccuratverydifferenttimesinthefuture. ThisistrueoftheDecemberstudywhere

computer programming costs largely occur before the onset of the nodal market. The estimated

benefits, of course, result only after nodal market implementation. It should be noted also that the

authorsadopt thediscount rate from theNovember2004study inspiteof theturmoil in theworlds

financialmarketsandtheglobalrecession.

ComputerProgrammingCostsPerhaps most mysterious is the expected cost to complete the computer programming for the nodal

market. Althoughnotexplicitlystated,thetotalexpectedbill isontheorderof$750million,ofwhich

$322.1 has already been spent.11 Table 13 identifies a present value of future expenditures of $430

million.12 Whilebeyondthescopeofthisreview,thepriceforthisrelativelystraightforwardcomputer

program is quite high by industry standards, especially for a single state with limited interties to the

surroundingmarkets. Table15showsthefullcostofthecomputerprogrammingexercise:

14|P a g e


12Ibid.,page40.


15/16


16/16


January7,2009

RobertMcCullough

16|P a g e

programasarelativebargainat$14.1millionperyearoverthe firstdecadeof itsoperation,which is

approximately onethird of the maintenance costs for the existing program. Assuming that the zonal

programsmaintenance

costs

have

been

overstated

by

afactor

of

three,

this

would

increase

the

net

cost

to continue the nodal program by $106 million, yielding a net going ahead cost of $326 million, as

opposed to $222 million. Alternatively, if maintenance costs of the nodal program are assumed to

matchthecostsofthezonalprogram,thetotalgoaheadcostswould increaseto$452milliondollars,

insteadofthereported$222million.

Ineithercase,theconclusioninTable15appearsvastlyunderstated.

ConclusionsThe December 18, 2008 study has a number of problems. These appear significant enough that the

conclusions of thestudy areeffectivelyspeculative. Inaddition, thestudy is poorlydocumentedand

dependsonanumberofunusualassumptions.

1. Thestudyhasonlymodeledtwoyearsofthetenyearperiodunderanalysis. Thisfailingaffectsboth the estimate of production costs and consumer benefits. The current modeling ends

beforetheyearwhenthe2004modelappearedtohavesignificantproblems.

2. Thestudyhasnotverifiedtheirmethodologybydoingabackcast. Intheabsenceofabackcast,theresultscannotbedependedupontoevenremotelyreproduceconditionsinERCOTmarkets.

3. Several assumptions are highly questionable. The study assumes marginal cost bidding bymarket participants even though this is not present in ERCOT. In addition, the study uses anatural gas forecast from a year ago a curious choice in such volatile times. Other

assumptions are undocumented or simply copied from a study four years out of date the

discountrate,forexample.

4. The costs of the additional computer programming appear high very high and theassumptions appear inconsistent. The high maintenance costs assumed for the existing

operationalzonalmodelappeartobejustafractionoftheassumedmaintenancecostsofthe

new,unfinishedandvastlymorecomplex,nodalmodel.

review of the ercot december 18, 2008 nodal cost benefit study

Documents