march 6, 2008 tac texas nodal market implementation program update jerry sullivan
TRANSCRIPT
March 6, 2008TAC
Texas Nodal Market Implementation
Program Update
Jerry Sullivan
2Texas Nodal Program UpdateMarch 6, 2008 2
AGENDA
• Program Status
• Key Risks
• IBM Review given to BoD in February
• Quality Results
• Schedule and the Program Level “Health Checks”
• Cost
The area of emphasis at today’s TAC Meeting
3Texas Nodal Program UpdateMarch 6, 2008 3
Program Status: Nodal Delivery
Nodal Delivery status remains amber; key checkpoints approaching
CostScheduleScope / Quality
• Revised budget approved by ERCOT BOD, however we will be RED until fee filing approved.
• Revised Fee Filing with the PUC this week.
• Need to decide on methodology for grayboxing and/or time-boxing the handling of:
current protocols,or
future NPRRs.
• Requirements for DAM in MMS are being revised.
• Key checkpoints in place to ensure ongoing viability of 12/1/08.
• Planning for the “168 Hour Test,” which is under 200 days away.
• EDS 2 is Red, but soon may be amber.
• EDS 4 is Red, will go amber when EMS 4/5 completed.
• Number of defects discovered in EDS were high, being resolved.
• Applying increased efforts to identify and resolve defects in pre-FAT and FAT
Green Amber RedAmber
Nodal Delivery
Legend
Red
Amber
Green Estimate at Complete = <$263m
Estimate at Complete = < $263m + 10%
Estimate at Complete = >$263m + 10%
Go-live = 12/1/08
Go-live = <30 days+
Go-live = >30 days+
Program aligned with current protocols
Program aligned to previous protocol version
Program not aligned to protocols
Current product quality is not sufficient for Nodal launch
Most Nodal products currently achieve quality standards
All Nodal products currently achieve quality standards
4Texas Nodal Program UpdateMarch 6, 2008 4
Program Status: ERCOT Readiness (3/15/2008)
ERCOT Readiness status remains amber, but a renewed awareness of the need for preparation is apparent
• Organization, Processes and Procedures, and Training show progress– Organization
• Completed review of Deep Dive presentations including staffing/ sizing department models for Post-Nodal implementation (2009).
• Draft Vision, Mission, Core Values and Strategic Plan completed. Transition sponsors are assigned.• January Executive Readiness Survey overall indicates Amber. Key Focus Areas to work on: adequate
staffing, procedures, transition planning.• Transition plans being documented for all key departments.
– Processes and Procedures• Business Process Model completed to procedure level and ready to leverage for testing and validation
during EDS and training.• Increased focus on the creation of procedures.
– Training• Training tracking and tailoring at the department/individual level.• Validation of current individual training plans
• Concerns Remain– Adequate staffing and resources for transition from project teams to employees
– Procedures still in flux until delivery of systems
– Completion of training in time for EDS
Priority for ERCOT Staff is to ensure readiness for the “168 hour test”
5Texas Nodal Program UpdateMarch 6, 2008 5
Program Status: Market Participants (2/28/2008)
• Registration and Qualification [green] - Resource Asset Registration quality, as measured by metric MP11, has improved. Two QSEs represent resources that are not current with submittals for resource registration:
• Barton Chapel Wind, Austin Energy
• Engagement continues to be main area of concern.
Engagement remains [amber] for all segments except QSEs with Resources.
Continuing to work with MPs to improve their engagement scores.
13 QSEs [red] - due to Engagement. The following remain red in the scorecard:
• Citadel, Clearview, Duke Energy Ohio, Econnergy, EnergyCo, JPMorgan Ventures Energy, Juice Energy, Kansas City Power and Light, Liberty Power, LPT, Mirant, Texpo, WCW International
Next survey to be released March 17. Negative responses will be treated as red in the next round.
• Training [green] – First pass of self reported training is green for all segments except LSEs
• Connectivity and Telemetry [green] – Connectivity metrics complete for most MPs. Outliers are new to the program and need to go through EDS process
Overall status* returns to amber
* Status current as of 2.28.08
6Texas Nodal Program UpdateMarch 6, 2008 6
Market Readiness
• Required by: ERCOT Nodal Transition Plan (5.4.4 (3))
• Purpose: Provide an opportunity for market participants to measure and improve the accuracy and stability of the Real Time Sequence (RTS).
• Accuracy: LMPs are reasonable given the input data, but the input data needs improvement. These are not “settlement grade” LMPs. ERCOT is limited in its capacity to handle all MP questions based on the quality and complexity of the data.
• Use: Should be used for trends and to assist the EDS team to improve the accuracy of the input data, but not for financial analysis.
• Actions: ERCOT is continuing to resolve a number of problems known to impact the SCED results: Working with QSE’s to resolve a number of SCED data input issues; similar exercise is in progress with TSP’s to improve the quality of the tele-metered data feeding the State Estimator.
Six month posting of LMPs initiated on February 15
7Texas Nodal Program UpdateMarch 6, 2008 7
Key Risks as 5 March 2008
These are the key, key risk
M
H
M
M
Likelihood
H
M
H
M
Impact on Go Live
ERCOT working with vendor to rectify defects/outstanding. issues Single entry model must be in place by July to verify go-live model
Single entry model could be lateNMMS
Prioritizing build out of environments and applications based on EDS schedule
Planning active / standby EDS environment (Austin / Taylor) to eliminate short-term ITest need
Building test environment in Blue Building server room Removing all non-essential equipment from Austin data center Compressing Nodal deployment footprint as much as possible
Working with 3 main vendors on multiple iterations of development and validation of CIM import/export
Investigating contingency plan to preserve April 1st start of DAM / RUC EDS trials
Building interfaces in advance of applications delivery based on available information expecting re-factoring
Testing application and integration in available, sub-optimal hardware / software environments
Mitigation
Limits speed and scale of environment build outs, impacts development projects, integration and EDS trials
Infrastructure constraints
(power, space, cooling)
Uncertainty over ability to integrate CIM models may impact data and system quality and testing
CIM
(Common Information
Model)
Lack of source systems and infrastructure limits ability to validate integration points, impacts re- factoring cycle times and costs
EIP (Enterprise Integration
Project)
DescriptionRisk
8Texas Nodal Program UpdateMarch 6, 2008 8
IBM Review - Dec 07 – Jan 08 - Some issues involve quality processes
Issue Nodal Team Comments
Resource Planning and Deployment. Attrition. More engagement needed from ERCOT.
Effort / Engagement will increase. Training necessarily follows delivery of software. Few key losses since October. We are relying on metrics and ownership which will drive behaviors.
Risk Management. Minor risks numerous, but duplicative by project. Few High program risks.
Relatively known risks. Risk log is comprehensive and quite exhaustive; Focus now is on the 5 or 6 top key risks rather than individual project perspectives.
Quality Reviews. Documentation lagging due to priority given to delivery.
Protocols covered. Priority given to fixing defects and trials.
Documentation of traceability is lagging behind delivery, we acknowledge documentation needs improvement
Timeline. Lagging measures in quality, need more Unit Testing visibility (before Pre FAT visibility.
High visibility given instead to Pre FAT at vendor site, to FAT at ERCOT sites, and to EDS. We view “Unit Testing” to be less reliable, and/or reporting delayed by most vendors.
PMO resources low (due to shifting resources to more critical projects)
Moved experienced personnel to other areas to fill needs; program leadership looks to optimize staffing in right places. Now adding staff to PMO and to Delivery Assurance Group (DAG)
9Texas Nodal Program UpdateMarch 6, 2008 9
Quality - Summary
• Successes– NMMS has fixed a number of high severity defects (2 fixed of 3)– CRR has reduced the overall number of defects; currently only 3 Severity 2
defects exist (No severity 1 defects currently)• Areas of Concern
– Defect resolution time– High percentage of EMS defects not detected till EDS
Mitigation Measures– Passive data center site with live data feed will allow for earlier defect detection
• Defect Classification
Quality has experienced recent successes but concerns remain
Severity 1 – Data Loss/Critical ErrorSeverity 2 – Loss of Functionality w/o workaroundSeverity 3 – Loss of functionality with workaroundSeverity 4 – Partial Loss of a feature setSeverity 5 – Cosmetic/Documentation ErrorSeverity 6 - Enhancement
10Texas Nodal Program UpdateMarch 6, 2008 10
Quality details: Nodal Test Results per Project
• Since December 2007, the Program undertook a significant documentation and tracking process.
• Significant visibility into quality status of each project:
• December 2007: 5 projects tracked
• March 2008: 14 projects tracked (others are not in the testing phase)
• Failure rates within acceptable limits, especially with Severity* 1 and 2 defects.
* Note: Severity 1 and 2 defects are the most significant of the 4 types.
December 17, 2007
March 4, 2008
11Texas Nodal Program UpdateMarch 6, 2008 11
Quality Details: Nodal FAT Active Defects by Severity by Project
(Note - Does not include Closed and Deferred)
Few severity 1 or 2 defects in the Market Management System (MMS)
No severity 1 defects in Energy Management System (EMS)
Commercial System Registration (COMS) is still early in development. In Pre FAT
The Network Model Management System (NMMS) has a few Sev 1 defects
Overall, active defects are being tracked by project and by classification.
The relative low number of Sev 1 and 2 defects is somewhat encouraging.
12Texas Nodal Program UpdateMarch 6, 2008 12
Quality Details: Nodal Active Defects by Severity Trend
Sev 3
Sev 4
Sev 1
Sev 2
• Severity 3 defect trend is rising due to focus on fixing higher priority severity 1 and 2 defects• Severity 2 trend is remaining steady which indicates “defect open rate” is only slightly higher than
defect closure rate
13Texas Nodal Program UpdateMarch 6, 2008 13
Quality Details: Average Days to Fix Defect by Vendor
The larger project vendors are doing fairly well in their turn around time for defect fixing. There are known and understood reasons for the other vendors.
What is not shown here is the turnaround by defect classification.
Overall, the time to fix defect in various categories gives us a good indication on:
how long the product will be in testing
what the next release will look like
how long will it take to get to market trials.
a good indication of ability to meet the 168 hour test.
14Texas Nodal Program UpdateMarch 6, 2008 14
Quality Details: Nodal Defects Reopened % (After Fixing by Vendor)
Indicates what percentage of fixes delivered by a vendor retested unsuccessfully.
ABB and Areva are fixing their products with a good low reopen rate.
Good indication that the Energy and the Market Management System are conducting good quality reviews at the vendor site prior to releases being given back to ERCOT and the market.
This metric is being tracked to determine trends.
15Texas Nodal Program UpdateMarch 6, 2008 15
CRREIPEMSIRTMMSMMS MM
Defect Summary Graph
Severity
1 D
ata
Loss
/Crit
ical
Err
or
2 Lo
ss o
f fun
ctio
nalit
y w
/o w
o..
3 Lo
ss o
f fun
ctio
nalit
y w
ith w
..
4 P
artia
l los
s of
a fe
atur
e se
..
5 C
osm
etic
/Doc
umen
tatio
n er
ror
6 E
nhan
cem
ent
Num
ber
of D
efec
ts
35
30
25
20
15
10
5
0
5
20
32
12
31
Quality Details: Defects found in the Early Delivery System
Severity 1 – Data Loss/Critical ErrorSeverity 2 – Loss of Functionality w/o workaroundSeverity 3 – Loss of functionality with workaroundSeverity 4 – Partial Loss of a feature setSeverity 5 – Cosmetic/Documentation ErrorSeverity 6 - Enhancement
We released EMS 2 through 3 too early, resulting in defects showing up in the Early Delivery System
No severity 1 defects in Energy Management System (EMS)
Commercial System Registration (COMS) is still early in development. In Pre FAT
The Network Model Management System (NMMS) has a few Sev 1 defects
Overall, active defects are being tracked by project and by classification.
The relative low number of Sev 1 and 2 defects is somewhat encouraging.
Severity
1
Severity
2
Severity
3
Severity
4
Severity
5
Severity
6
16Texas Nodal Program UpdateMarch 6, 2008 16
Schedule: “Health Checks”
Still targeting a 12/1/08 Go-Live date; critical path runs through EDS activities
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
EDS 1
EDS 2
EDS 3
EDS 4
Today
MP Trials & NM Verification
SCED & LMP Posting for 6 months
ICCP / P2P
SE Tuning
E1 R2
E2 R3E2 R4
E3 R5
E3 R6 LFC Testing
E3 R7 CRR Testing
E4 R9
CMM, DAM & RT Settlements, Verifiable Costs & Disputes
DAM, SASM & RUC execution, DAM Stability Test, EECP, DST, Compliance
E4 R8
Performance
Health Checks – measure of schedule
viability
Dictated by PRR 727: Market Readiness
Certification
168 Hr TestEDS 4 Go Live Sequence
CIMIntegration Defect
Go-Live (Go/No-Go)
Sept Oct Nov
TAC Meeting: 10/2
ERCOT BoD: 10/21
First Market Notice: 10/31 Second Market Notice: 11/21168 Hr Test Start
32 Days to complete Test3
2
1
Business Processes
17Texas Nodal Program UpdateMarch 6, 2008 17
Schedule: This is the metric status for each of the Early Delivery Systems
EDS activities are monitored and measured by 59 metrics
(xx) – Number of metrics associated with a particular Release
EDS1 – Complete
EDS2 – Overall Red resulting from ongoing issues with State Estimator
EDS3 – Yellow resulting from LMP performance and delays in LFC / ACE testing
EDS4 – Mostly unmeasured at this point, but inheriting Red status from recently moved Alarms Processing metric. Commercial Ops metrics being measured are Green
Overall Readiness – Yellow due to open issues on telemetry, modeling, and CIM
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
EDS 1
EDS 2
EDS 3
EDS 4
Today
MP Trials & NM Verification
SCED & LMP Posting for 6 months
ICCP / P2P
SE Tuning
E1 R2
E2 R3
E2 R4
E3 R5
E3 R6 LFC Testing
E3 R7 CRR Testing
E4 R9
CMM, DAM & RT Settlements, Verifiable Costs & Disputes
DAM, SASM & RUC execution, DAM Stability Test, EECP, DST, Compliance
E4 R8
168 Hr Test Go Live Sequence
(1 EDS, 2 MP)
(3 EDS)
(5 EDS, 1 MP)
(2 EDS, 1 MP)
(3 MP)
(2 EDS, 2 MP)
(12 EDS, 2 MP)
(5 EDS, 3 MP)
Overall Readiness: (15 metrics)
18Texas Nodal Program UpdateMarch 6, 2008 18
Outage Scheduler - Need for periodic duel entry of outages
• ERCOT will develop scripts to alleviate duel entry during most periods:– Between minor tests– Between the 168 hr. test and go-live
• There will be times, especially during the 168 hour test where zonal and nodal systems will require MPs to enter data into two systems
• Solid Bars below indicate duel entry requirement, estimated to be a total of 45 days
DecemberNovemberOctoberSeptemberAugust
Dual Entry to Support 168 Hour Test
Dual Entry just prior to Go – Live.
Minor tests
* (hashed bar if 168 hour test extends to the full month
19Texas Nodal Program UpdateMarch 6, 2008 19
Cost: Element Summary
EAC Revised Budget and Forecast remain constant; spending in January expected to be lower and should reverse much of the overrun through Dec.
Revised Var. (o)/u Revised Var. (o)/uCost Element Budget Actual Budget Budget Forecast Budget
Internal Labor $ 25.0 $ 25.3 $ (0.3) $ 47.9 $ 47.9 $ -
External Labor 56.6 59.8 (3.2) 94.9 94.9 -
Vendor 42.7 44.0 (1.3) 76.8 78.8 (2.0)
Hardware / Software 55.1 56.7 (1.6) 59.2 59.4 (0.2)
Taxes and Other 9.4 9.6 (0.2) 15.1 15.1 -
Subtotal $ 188.8 $ 195.4 $ (6.6) $ 293.9 $ 296.1 $ (2.2)
Contingency - - - 15.0 12.8 2.2
Total Nodal Program $ 188.8 $ 195.4 $ (6.6) $ 308.9 $ 308.9 $ -
Finance Charge 6.6 6.0 0.6 10.6 10.6 -
Total $ 195.4 $ 201.4 $ (6.0) $ 319.5 $ 319.5 $ -
2008 therefore bringing actual spending back in line with the Revised Budget NOTE: Overspending of the Revised Budget through December 2007 will be offset in January
Nodal ProgramCost Element Summary
($ Millions)
Through December 2007 Estimate at Complete (EAC)
January actuals are still unavailable as of 5 March