peter j. buerling · compliance with nerc reliability standards ... lessons learned, faqs &...

Peter J. BuerlingDirector, Records & Information Compliance

April 15, 2016

ReliabilityFirst Workshop


Opening Comments

■ Presentation Topic

■ Disclaimer

■ Presentation Support – Introductions

– Mark Koziel – Consultant, CIP Compliance

– Don Morrison – Manager,

Asset Operations

April 15, 2016FirstEnergy Critical Infrastructure Protection Program 2


FirstEnergy Facts at a Glance

■ Headquartered in Akron, Ohio

■ Among the largest investor-owned electric systems in

the U.S.

■ 6 million customers

■ More than $52 billion in assets

■ $15 billion in annual revenues

■ 15,800 employees


All data as of Dec. 31, 2015


FE Service Territories10 Operating Companies


Ohio Edison

The Illuminating Company

Toledo Edison

Penn Power

West Penn Power

MonPower

Potomac Edison

Potomac Edison

VA Transmission Zone

Met-Ed

Penelec

Jersey Central Power & Light


FE Transmission System

■ FirstEnergy’s transmission systems are located in the PJM region.

■ PJM is the Regional Transmission Organization (RTO) and is the

registered TOP, RC and BA

■ FirstEnergy transmission systems are operated within the ReliabilityFirst

(RF) Regional Reliability Organization territory

■ All-time coincident peak load:

– FirstEnergy reached 35,346 MW on July 21, 2011


Voltage Levels Miles

765 kV *N/A

500 kV 1,541

345 kV 1,360

230 kV 1,926

138 kV 7,195

115 kV 1,904

* FEU has one 765 kV transformer tie into the AEP 765kV system


FirstEnergy Diverse Generating SourcesOverview

Fully Regulated

Partially Regulated

Supercritical Coal 8,072 MW

Subcritical Coal 1,334

Nuclear 4,048

Gas/Oil 1,592

Renewable 1,906

Hydro 1,410

Wind 476

Solar 20

Total 16,952 MW8%

48%

24%

9%

11%Map excludes 99 MW of wind output in IL

* Includes generation from nominal gas/oil

units not shown on map

*

**


Updated as of Feb. 16, 2016


Compliance Ownership and Oversight

■ FERC Compliance – Responsible for independent oversight of

compliance with NERC Reliability Standards

■ Business Units – Responsible for compliance with NERC Reliability

Standards via process, procedures, training, etc.

■ Compliance Champion – Contact /liaison with FERC Compliance

and responsible to assist business units in managing and providing

BU oversight for all NERC applicable Reliability Standards


Compliance Champions

Compliance Oversight Compliance Ownership

Develop BU Supporting

Processes & Procedures

Conduct BU Training

& Testing

Coordinate Site and

Master CIP List

Follow Compliance

Policies & Programs

Collect and Retain BU

Documents & Reports

Executive Reliability Steering Committee

Review of Standards

Develop & Communicate

Compliance Policies

Facilitate Compliance

Process Development

Contact to External

Regulatory Groups

Independent Audit

Controls & Measures

FERC Compliance Operations Leadership

Business Units


Executive Reliability Steering Committee■ FirstEnergy Utilities

– Vice President Transmission

■ Internal Auditing

– Executive Director Internal Auditing

■ Information Technology

– Vice President IT Operation

– Senior Vice President Corporate Services & Chief

Information Officer

■ FERC Compliance

– Vice President Compliance and Regulated Services

& Chief FERC Compliance Officer

■ Enterprise-Wide Risk Management

– Vice President Corporate Risk & Chief Risk Officer

■ FENOC (Nuclear)

– Senior Vice President Fleet Engineering

■ Generation

– Vice President Fuel and Unit Dispatch

■ Fossil Operations

– Vice President Fossil Fleet Operations

■ Legal

– Associate General Counsel


Corporate

Security

IT

Infrastructure

FirstEnergy

Utilities

Fossil/

Generation


Compliance History

■ FirstEnergy has a single CIP Compliance Program

– All business units roll up to an overall corporate program

– Single CIP senior manager for FirstEnergy

– Common programs

– Use shared procedures across enterprise.

■ Audit 2010 – First CIP audit

■ Audit 2012 – Merged programs with

Allegheny

– 18 registered entities

■ Audit 2013

– 3 registered entities



Project Plan for Implementing CIPv5

■ Implementation was divided into 3 phases

– Phase I – BES Cyber System Identification and Project Planning

– Phase II – High and Medium Impact BES Cyber Systems

– Phase III – Low Impact BES Cyber Systems

■ Goal: Be compliant with Version 5 by Dec. 31, 2015

– High and Medium BES Cyber Systems

– Shakedown: Jan. 1 – March 31, 2016 (June 30, 2016)


January 1, 2014 December 31, 2015June 30, 2014 June 30, 2016 September 31, 2018

Phase I Phase II Shakedown

Phase III


Project Team – Core Team


Executive Reliability Steering

Committee

Peter Buerling

IT Compliance CIP Compliance IT OperationsEnergy Delivery

Planning & Protection

FES Dispatch Cyber Security TransmissionPhysicalSecurity

Generation

ERSC

Project

Manager

Track

Lead

Track

Lead

Track

Lead

Track

Lead

Track

Lead

Track

Lead

Track

Lead

Track

Lead

Track

Lead

Legal Internal Auditing Project Planning

ConsultantProject

PlannerConsultant


Challenges

■ Identifying BES cyber systems

– Developed a methodology

– Top-down approach

■ Unifying business units

– Maintaining a corporate approach

■ Different architectures

– Mergers

■ Outdated device inventories for new

in-scope devices

■ System switchovers

– Manual systems

■ Concept of external routable connectivity



Challenges

■ Working around outages

■ Lead time for nuclear units

■ Coordination with other utilities

■ Implementation of CIP v6

– Timing

■ Lessons Learned, FAQs & pilot – unreliable resources

– Timing

– Information

– Retraction – approved vs. unapproved



CIP Version 5 Landscape

■ As of Dec. 31, 2015

– 2 high-impact BES cyber

systems

– 119 medium-impact BES

cyber systems

– 895 low-impact assets

with low-impact BES

cyber systems



FE Transition Plan

■ NERC provided a flexible enforcement approach for entities

to start complying with some or all of V5 requirements

while maintaining compliance with V3 requirements

– Only V3 CIP Cyber Assets and V3 requirements are subject to

enforcement during the transition period

– Compliance with “mostly compatible” V5 requirement = V3

requirement compliance


Transition Period

Start Date Feb. 6, 2014 (FERC approves V5 standards)

End Date July 1, 2016 (Medium- and high-impact BES cyber systems)


FE Transition Plan

■ FE developed customized guidelines based on the NERC

Transition Plan and other applicable regulatory documents.

– Implementation plans for V5 and V6 standards

– Lessons learned, FAQs, and informal regulatory guidance

■ FAQ section provided guidance for specific internal

scenarios that arose during transition

– Compliance with specific V3 annual requirements

■ Initial versions encouraged early compliance with some V5

standards for new devices entering CIP program

– This piece meal compliance transition approach proved impractical

■ FE transitioned to compliance with all CIP V5 Standards

shortly before 12/31/2015



Maintaining Compliance During Transition

■ Maintain a good compliance culture during transition

– BUs identify compliance concerns with new CIP V5

processes/procedures and notify FE Compliance

– FE Compliance forwards compliance concerns to PV Review Board

when appropriate

■ PV Review Board evaluates issues against both CIP V3 and V5

standard requirements

– BUs need to be able to identify V3 CIP Cyber Assets throughout the

Transition Period

■ Compliance issues associated with BES cyber assets that are

not V3 critical cyber assets will result in no self-report

– Cause evaluations and corrective actions may be appropriate



IT Operations

■ Leadership

– Managers engaged at PMO (provide resources, tear-down issues)

– Directors engaged at steering committee

■ Leverage CIP v3 Work

– Control centers

■ Partnering with transmission for substations

– Establishing ownership of device types

– Device replacement

■ Implementation of Tripwire

– Baseline library

– Connected to more than 1,300 devices nightly to detect

configuration changes



IT Operations

■ Implementation of Intrusion Detection Systems and

software (18 sites)

■ Training

– Delta training for seasoned CIP v3 veterans

– Complete training for rookies

– More than 500 FE personnel and

contractors participated in 1 or

more of 15 modules.



Transmission

■ FE Transmission start point– no CIPv3 assets (devices)

■ CIPv5 Transmission Environment – >800 BES locations

– Filtered down to ~ 80 locations to evaluate individual assets

– Barcoded >20,000 assets (equipment, relays, meters, etc..)

– Information correlated for ~ 2,000 programmable CIPv5 devices/~

190 makes/models through review of barcoding data, asset

database, bill of materials, construction prints

■ Developed nearly 200 Security

Baseline Documents

– Security configurations

– Password strategy



Transmission (Continued)

■ Mobilized “Tiger Teams” August-December, 2015

– Team of “best” technicians and commissioning engineers for each

of 10 Operating Companies

■ CIPv5 “assets” tracked in separate database

■ Processes manually supported




■ Existing CIPv5 Sustainment

– New Organization created

– CIP Compliance Implementation

– Asset Tools

– Asset Operations

– Tight integrations with Design/Project Management/Commissioning

for new installations

– Weekly Change Control Meetings with all parties for all field

activities

– Field Training followed up with continued remote training

– Processes manual with incremental automation/efficiencies to

existing 2015 procedures




■ Future Improvements – Tools

– Multi-year focus on data integrity across entire footprint

– Upgrades to Asset Inventory system

– Upgrades to remote connectivity tool and field assets

– Purchase and installation a comprehensive “Operational

Technology Configuration Management” (OTCM) tool for all

configurable devices within a substation

including electromechanical




■ Future Improvements – Processes

– All new devices implemented with upgraded security

parameters/passwords

– Limiting new makes/models

– Upgrade security and passwords of existing devices at

maintenance cycles

– Data Governance Project

(10 Applications/60 Attributes)



Generation

■ Cyber system configuration determined compliance

approach

– Corporate Methodology used to define system rating

– Medium-impact cyber systems were analyzed to determine if they

could be reconfigured/split to be low-impact cyber system

– In-depth vendor studies used to further determine feasibility of

conversion to low impact

– Approximately 2-year-long effort for

analysis, design and implementation

– Some medium-impact cyber systems

were left as medium impact



Generation

■ Implementation of Plans

– All control work had detailed implementation

plans jointly developed between plants and

vendors to reduce outage duration

– Pre staged equipment and wiring

– Plant medium-impact cyber systems used corporate compliance

program to achieve compliance.

– Plant cyber security representative (PCSR) position

– Key person during implementation of all compliance efforts

– Has general knowledge of the CIP Standards as they relate to

plant equipment

– Has detailed knowledge of plant cyber systems

– Key person in change control process at plant



Corporate



Phase III – Low Impact BES Cyber Systems

■ Leverage project and governance structure put in place for

Phases I and II

■ Stood up strategy team for LEAP/LERC

■ Stood up strategy team for transient cyber assets and

removable media

■ Build out project plan

■ Tabletop exercise of connectivity prior to field visits



Controls


Currently 217 controls have been identified for CIP v5


Compliance Concern Process


ReliabilityFirst Workshop 31April 15, 2016FirstEnergy Critical Infrastructure Protection Program