intellibind top ten most violated standards presentation 2011 01 27 (f)
DESCRIPTION
This is the presentation made at the WECC CUG meeting in Feburuary 2011 and sponsored by LADWP. This presentation can also be found on the WECC website.TRANSCRIPT
WECC Compliance User Group Meeting
Top 10 Most Violated Standards in WECCApproaches to Compliance
February 9, 2011
Thanks to Los Angeles Department of Water and Power for sponsoring this session
Agenda
Speaker Intro Sources of Information Top 10 Most Violated Standards NERC and WECC Common Violation Findings Evidence and Proof of Compliance Details for 693 (Reliability) Standards
Primary Non-Compliance Factors Recommendations
Details for 706 (CIP) Standards Primary Non-Compliance Factors Recommendations
Introduction
Kevin Conway – Intellibind, LLC
26 Years in the Industry NERC Reliability Program NERC OC Representative NERC Functional Model
Workgroup NERC Standards Drafting
Team System Reliability
Manager Marketing and Trading NERC Certified System
Operator
Bill Addington – Intellibind, LLC
Over 20 years Cyber Security expertise
Involved in creation of original cyber security standard BS7799
Over 10 years with Electric Power Utilities
Principle author and speaker, NERC Cyber Security Workshop
Author of EPRI papers on Cyber Security
Former Interim Security Manager at ERCOT
Sources of Information
NERC published research and reviews
WECC documents Research into Violation Reports for
details and common causes Field Experience
Audit and readiness team experience across U.S.
Experience as SME’s in audit in various utilities in WECC
Interviews with utilities
NERC and WECC Top 10 Most Violated StandardsMay 1, 2009 to April 30, 2010 – Summary Table
1 2 3 4 5 6 7 8 9 10NERC PRC-005 CIP-004 CIP-007 CIP-001 EOP-005 CIP-003 FAC-008 CIP-006 FAC-001 CIP-002
WECC PRC-005 EOP-005 CIP-001 TOP-002 CIP-004 CIP-007 PER-002 EOP-001 CIP-003 COM-001
Violation RankRegion
Top Ten Most Violated WECC Standards
• PRC-005 Transmission Protection System Maintenance and Testing
• EOP-005 System Restoration Plans• CIP-001 Sabotage Reporting• TOP-002 Normal Operations Planning• CIP-004 Cyber Security – Personnel and training• CIP-007 Cyber Security – System Security
Management• PER-002 Operating Personnel Training• EOP-001 Emergency Operations Planning• CIP-003 Cyber Security – Security Management
Controls• COM-001 Telecommunications
Common Violation Findings
Poorly written or non-existent procedures and processes.
No evidence of testing procedures.
Documentation – inability to prove that processes where in place to prove compliance with the standard.
Common Violation Findings
Poorly written RSAWs RSAWS do not clearly describe
processes RSAWS do not describe the
relevance of evidence SME interviews do not match RSAW
statementsUnder or over documented
evidence
Evidence – Proof of Compliance
Procedures and Processes
Output of Procedures and Processes
Proof of Compliance (Evidence)
Top Most Violated Operational Standards (693)
Top Most Violated WECC Operational Standards
CIP-001 Sabotage Reporting COM-001Telecommunications EOP-001 Emergency Operations
Planning EOP-005 System Restoration Plans PER-002 Operating Personnel
Training PRC-005 Transmission Protection
System Maintenance and Testing
TOP-002 Normal Operations Planning
CIP-001 Sabotage Reporting
CIP-001 – Sabotage Reporting
Overall Non-Compliance Analysis Statement Problem areas identified for non-
compliance: ▪ Procedures▪ Reporting▪ Communication▪ Documentation
CIP-001 – Sabotage Reporting
CIP-001 Primary Non-Compliance Factors Lack of required records demonstrating
compliance. Procedures were missing or deficient for
reporting events of sabotage.
CIP-001 – Sabotage Reporting
CIP-001 Primary Non-Compliance Factors (cont.) Deficiencies were found in procedures to
communicate information regarding sabotage events to other appropriate personnel.
Contact list was too incomplete to report sabotage events to the local FBI and appropriate personnel.
CIP-001 – Sabotage Reporting
Recommendations Have a thoroughly documented and tested
procedure in place for dealing with sabotage events. This includes steps for recognizing and making sure relevant personnel and entities are informed of a sabotage event.
Perform periodic review of the communication reporting procedure and confirm that contact lists are complete and current. This includes making sure all “operating personnel” and local government contacts are correctly identified and included.
CIP-001 – Sabotage Reporting
Recommendations (cont.)
Verify that procedures are clearly identified and current in all Operator Procedure manuals.
COM-001 Telecommunications
COM-001 – Telecommunications
Overall Non-Compliance Analysis Statement Problem areas identified for non-
compliance: ▪ Procedures▪ Testing▪ Coordination▪ Documentation
COM-001 – Telecommunications
COM-001 Primary Non-Compliance Factors Lack of required documentation
demonstrating that entity was managing, alarming, testing, and monitoring its vital telecommunications facilities.
Failure to test its vital telecommunications facilities was evident with its documented testing procedures.
COM-001 – Telecommunications
COM-001 Primary Non-Compliance Factors (cont.) Entity unable to provide evidence that it
had the ability to investigate and recommend solutions to telecommunications problems within its own area and other areas, nor that it had procedures in place that confirm it would be able to continue operation of the system during a loss of telecommunications facilities.
COM-001 – Telecommunications
Recommendations Design, develop, implement and
maintain procedures addressing the process of managing, alarming, testing, and monitoring vital telecommunications facilities and methods of documentation for recording this process.
Supply evidence that you test vital telecommunications facilities consistent with the testing procedures and document test records at the time testing occurred.
COM-001 – Telecommunications
Recommendations (cont.) Design, develop, implement and
maintain written operating instructions and procedures to provide a means to coordinate telecommunications among respective areas.
This coordination shall include the ability to investigate and recommend solutions to telecommunications problems within the area and with other areas.
EOP-001 Emergency Operations Planning
EOP-001 – Emergency Operations Planning
Overall Non-Compliance Analysis Statement Problem areas identified for non-
compliance: ▪ Procedures▪ Documentation▪ Incomplete Emergency Operations Plans
EOP-001 – Emergency Operations Planning
EOP-001 Primary Non-Compliance Factors Failed to have in place an operating
agreement with provisions to obtain emergency assistance from remote and adjacent Balancing Authorities.
Emergency Plan did not directly address all of the necessary elements, such as, system restoration plans, communication protocol, mitigate operating emergencies, tasks to be coordinated and staffing levels during emergencies.
EOP-001 – Emergency Operations Planning
EOP-001 Primary Non-Compliance Factors Failure to provide evidence that it
reviews and annually updates its emergency plans.
Failure to provide updated emergency plans to all of the required entities at the time the plans were updated.
EOP-001 – Emergency Operations Planning
Recommendations Obtain an operating agreement with provisions
to obtain emergency assistance from remote and adjacent Balancing Authorities.
Design, develop, implement and maintain an Emergency Plan and procedures that directly address all of the necessary elements, such as, system restoration plans, communication protocol, mitigate operating emergencies, tasks to be coordinated and staffing levels during emergencies.
EOP-001 – Emergency Operations Planning
Recommendations (cont.) Design, develop, implement and
maintain a procedure that requires review and annually update to emergency plans.
Design, develop, implement and maintain a procedure to provide updated emergency plans to all of the required entities at the time the plans were updated.
EOP-005 System Restoration Plans
EOP-005 – System Restoration Plans
Overall Non-Compliance Analysis Statement Problem areas identified for non-
compliance: ▪ Procedures▪ Annual Review▪ Testing▪ Documentation▪ Coordination ▪ Training
EOP-005 – System Restoration Plans
EOP-005 Primary Non-Compliance Factors Failure to provide a Restoration Plan that
would re-establish its electric system in a stable and orderly manner in cases where there is a partial or total shutdown of its system.
Restoration Plan was not reviewed and updated at the least annually.
EOP-005 – System Restoration Plans
EOP-005 Primary Non-Compliance Factors (cont.) Failure to coordinate restoration plan with
the Generator Owners and Balancing Authorities within its area, its Reliability Coordinator, and neighboring Transmission Operators and Balancing Authorities.
Restoration Plan did not contain procedures for the loss of vital telecommunication channels and had not periodically tested its telecommunication facilities that are required to implement the restoration plan.
EOP-005 – System Restoration Plans
EOP-005 Primary Non-Compliance Factors (cont.) Failure to properly train operating
personnel on how to implement the restoration plan.
Restoration plan was not verified by actual testing or simulation.
EOP-005 – System Restoration Plans
Recommendations Design, develop, implement and maintain a
Restoration Plan that will re-establish its electric system in a stable and orderly manner in cases where there is a partial or total shutdown of its system.
Review and update the Restoration Plan at least annually and document this process.
Coordinate the Restoration Plan with the Generator Owners and Balancing Authorities within its area, its Reliability Coordinator, and neighboring Transmission Operators and Balancing Authorities.
EOP-005 – System Restoration Plans
Recommendations (cont.) Design, develop, implement and maintain a
Restoration Plan that contains operating instructions and procedures for the loss of vital telecommunication channels and periodically test its telecommunication facilities that are required to implement the restoration plan.
Properly train operating personnel on how to implement the restoration plan.
Verify restoration procedure by actual testing or by simulation.
PER-002 Operating Personnel Training
PER-002 – Operating Personnel Training
Overall Non-Compliance Analysis Statement Problem areas identified for non-
compliance: ▪ Documentation▪ Training ▪ Competency
PER-002 – Operating Personnel Training
PER-002 Primary Non-Compliance Factors Failure to staff operating personnel that
have direct impact on the real-time operation of the BES with adequately trained and qualified people.
Training program lacked specificity with respect to the training of operating personnel.
PER-002 – Operating Personnel Training
PER-002 Primary Non-Compliance Factors (cont.) Training program did not effectively
identify objectives, but merely provided a list of skills.
Failure to identify in the training plans the necessary knowledge, skills, or competencies for system operators to conduct reliable operations.
Failure to include a plan for initial and continuing training of operating personnel.
PER-002 – Operating Personnel Training
PER-002 Primary Non-Compliance Factors (cont.) Failure to include training time for
operating personnel. Lack of organized records for the
completion of training.
PER-002 – Operating Personnel Training
PER-002 Primary Non-Compliance Factors (cont.) Failure to prove the competencies of
training staff in both knowledge of system operations and/or formal training or other evidence of instructional skill competencies.
Failure to conduct annual training and drills using realistic simulations of system emergencies at least five days per year.
PER-002 – Operating Personnel Training Recommendations
Provide required training to staff operating personnel that have direct impact on the real-time operation of the BES.
A well-designed training program must start with the identification of job tasks. From this identification of the tasks required, learning objectives can be developed to give operators the abilities to perform the tasks. The knowledge, skills, and abilities are identified as required to meet the objectives, Training is then designed to the objectives and the related knowledge, skills, and abilities that are associated with them.
PER-002 – Operating Personnel Training
Recommendations (cont.) Program objectives should:
▪ Be based on NERC and regional Reliability Standards, entity operating procedures, and applicable regulatory requirements.
▪ Reference the knowledge and competencies needed to apply these standards, procedures and requirement.
▪ Consider normal, emergency, and restoration conditions.
PER-002 – Operating Personnel Training
Recommendations (cont.) Training programs must cover and allot
time for the operating personnel who have primary responsibility for real-time operations, or who are directly responsible for complying with NERC and regional Reliability Standards.
It is essential to carry out training according to plans for all operators to whom this reliability standard is applicable.
Document the records of all trained operators.
PER-002 – Operating Personnel Training Recommendations (cont.)
Provide training staff with training programs for instructional methods.
Annual practice sessions should use practice simulations of real emergency conditions, and records should be logged with date, participants, and events.
Overall training program should be based on a systemic approach to training, and address all aspects of the requirements, execute them and provide evidence.
PRC-005
Transmission Protection System Maintenance and Testing
PRC-005 - Transmission Protection System Maintenance and Testing
Overall Non-Compliance Analysis Statement Problem areas identified for non-
compliance: ▪ Understanding ▪ Documentation ▪ Organization
MOST FREQUENTLY VIOLATED STANDARD
PRC-005 - Transmission Protection System Maintenance and Testing
PRC-005 Primary Non-Compliance Factors Documentation of testing and
maintenance results missing or inadequate.
Not all components of the protection systems were identified or tested.
Inventory lists of applicable devices are incomplete and therefore, devices were not scheduled appropriately.
PRC-005 - Transmission Protection System Maintenance and Testing
PRC-005 Primary Non-Compliance Factors (cont.) Lacking basis to determine the
appropriate testing intervals. Failure to complete maintenance and
testing activities on time. Lack of complete and thorough
monitoring of testing and maintenance programs.
PRC-005 - Transmission Protection System Maintenance and Testing
Recommendations Entities subject to standard PRC-005 need to
have a thorough and rigorous documented maintenance and testing plan in place for devices that qualify as protection systems.
Perform periodic physical inventories, including walkthroughs where needed, to ensure the active device inventory list is complete and accurate, and all pertinent devices appear on maintenance and testing schedules. CHANGE MANAGEMENT.
PRC-005 - Transmission Protection System Maintenance and Testing
Recommendations (cont.) Verify that testing programs include the
appropriate basis of testing to ensure the reliability of the Bulk Electric System.
Complete maintenance and testing programs on schedule and within defined intervals.
Emphasis on the urgency to meet the specified time intervals must be made explicitly clear regardless of what situations the company may encounter that interfere with planned maintenance.
TOP-002 Normal Operations Planning
TOP-002 – Normal Operations Planning
Overall Non-Compliance Analysis Statement Problem areas identified for non-
compliance: ▪ Coordination▪ Notification▪ Documentation
TOP-002 – Normal Operations Planning
TOP-002 Primary Non-Compliance Factors Failure to coordinate current-day, next-
day, and seasonal operations with appropriate entities.
Failure to provide documentation showing that it was providing forecasts to appropriate entities.
Failure to coordinate planning and operations with neighboring entities
TOP-002 – Normal Operations Planning
TOP-002 Primary Non-Compliance Factors (cont.) Failure to notify appropriate entities of
changes in capabilities and characteristics, such as changes in real output capabilities.
Failure to provide documentation demonstrating that it used uniform line identifiers when discussing transmission facilities among a shared interconnect.
TOP-002 – Normal Operations Planning
Recommendations Design, develop, implement and maintain
procedures to coordinate current-day, next-day, and seasonal operations with appropriate entities.
Create a standardized document to be utilized for providing forecasts with a mutually agreed upon format when providing forecasts to appropriate entities. Create an electronic file system for storage of these documents.
Design, develop, implement and maintain procedures to coordinate planning and operations with neighboring entities.
TOP-002 – Normal Operations Planning
Recommendations (cont.) Confirm all line identifiers are consistent
with the identifiers used when discussing transmission facilities among a shared interconnect. Obtain a letter of agreement between entities on identifiers used when discussing transmission facilities among a shared interconnect or obtain an agreed upon one-line diagram demonstrating uniform line identifiers.
TOP-002 – Normal Operations Planning
Recommendations (cont.) Design, develop, implement and
maintain procedures to utilize uniform line identifiers when discussing transmission facilities among a shared interconnect.
Design, develop, implement and maintain procedures to notify appropriate entities of changes in capabilities and characteristics, such as changes in real output capabilities.
Top Violated Cyber Security CIP Standards (706)
Top Most Violated WECC Cyber Security Standards
CIP-003 Cyber Security - Security Management Controls
CIP-004 Cyber Security - Personnel and Training
CIP-007 Cyber Security - Systems Security Management
CIP-003 Cyber Security - Security Management Controls
CIP-003 Cyber Security – Security Management Controls
Overall Non-Compliance Analysis Statement Problem areas identified for non-
compliance: ▪ Documentation▪ Review
CIP-003 Cyber Security – Security Management Controls
CIP-003 Primary Non-Compliance Factors Failure to document or incomplete
documentation of designated CIP Sr. Manager or failure to update documentation upon changes to designated CIP Sr. Manager.
Failure to properly document exceptions to the Cyber Security Policy.
CIP-003 Cyber Security – Security Management Controls
CIP-003 Primary Non-Compliance Factors (cont.) Failure to properly review and approve
exceptions annually or failure to properly document the review.
Failure to perform or properly document annual review of Cyber Security Policy.
CIP-003 Cyber Security – Security Management Controls
Recommendations Document a procedure that ensures a
CIP Sr. Manager is designated by Name, Title, and Date of Designation. Ensure procedure requires an update to the documentation upon any changes.
Implementation of Cyber Security Policy review of procedures and develop methods such as a compliance calendar to ensure the review occurs annually.
CIP-003 Cyber Security – Security Management Controls
Recommendations (cont.) Procedures should exist to manage so
they are properly documented and reported.
Personnel should be trained on exception management procedures that ensure all responsible parties follow through with their obligations to identify, document and mitigate instances where exceptions to the Cyber Security Policy must be made.
CIP-004 Cyber Security - Personnel and Training
CIP-004 Cyber Security – Personnel & Training
Overall Non-Compliance Analysis Statement Problem areas identified for non-
compliance: ▪ Access▪ Training▪ Documentation▪ Risk Assessment
CIP-004 Cyber Security – Personnel & Training
CIP-004 Primary Non-Compliance Factors Lack of required records demonstrating
compliance. Employees or contractors were granted
access to critical cyber assets without documented proof of clearance or escorted access.
CIP-004 Cyber Security – Personnel & Training
CIP-004 Primary Non-Compliance Factors (cont.) Unable to prove training was offered
and/or completed in a timely manner by personnel.
Background checks for employees or contractors with access to critical cyber assets were missing or incomplete.
CIP-004 Cyber Security – Personnel & Training
Recommendations Ensure and verify that all employees with
access to Critical Cyber Assets, including contractors and service vendors, have the appropriate training prior to access. Train annually thereafter.
Entity procedures should have control points designed to prevent granting access to untrained individuals or individuals who have not passed the Personnel Risk Assessment as well as methods to ensure annual re-training requirements are met.
CIP-004 Cyber Security – Personnel & Training
Recommendations (cont.) Entities need to ensure and verify that risk
assessments on employees, contractors, and service vendors with access to Critical Cyber Assets are not only completed prior to access, but that the assessment focuses on relevant information.
Entities need to ensure and verify that the training provided to employees, contractors, and service vendors being granted access to Critical Cyber Assets focuses on the relevant information.
CIP-004 Cyber Security – Personnel & Training
Recommendations (cont.) Entities need to ensure that appropriate
changes are made to access lists upon termination, or transfer of employees from or to areas that contain Critical Cyber Assets, and that access lists are frequently updated to contain contractors or service vendors.
Procedure should exist to ensure all access lists are current and properly maintained within the provided timeframe required by the standard.
CIP-007 System Security Management
CIP-007 Cyber Security – System Security Management
Overall Non-Compliance Analysis Statement Problem areas identified for non-
compliance: ▪ Procedures▪ Documentation▪ Testing▪ Logical Account Management
CIP-007 Cyber Security – System Security Management
CIP-007 Primary Non-Compliance Factors Failure to demonstrate that testing is
conducted to ensure new Cyber Assets and significant changes to existing Cyber Assets within an ESP do not adversely affect ALL existing cyber security controls.
Documented procedures insufficient to prove only ports and services required for normal and emergency operations are enabled.
CIP-007 Cyber Security – System Security Management
CIP-007 Primary Non-Compliance Factors (cont.) Failure to document the assessment of
security patches and upgrade availability within thirty calendar days of availability of the patches or updates.
Failure to document and implement a process for the update of anti-virus and malware prevention tools (including “signatures”).
CIP-007 Cyber Security – System Security Management
CIP-007 Primary Non-Compliance Factors (cont.) Failure to properly document and control
access to shared and system accounts. Failure to enable logging on cyber assets
located within the ESP and/or lack of operational processes to manually monitor system events related to cyber security.
CIP-007 Cyber Security – System Security Management
CIP-007 Primary Non-Compliance Factors (cont.) Failure to destroy or erase data storage
media to prevent unauthorized retrieval of sensitive cyber security or reliability data.
Failure to document disposal/redeployment activities.
CIP-007 Cyber Security – System Security Management
CIP-007 Primary Non-Compliance Factors (cont.) Failure to perform cyber vulnerability
assessment of all Cyber Assets at least annually.
Failure to perform cyber vulnerability assessment of all Cyber Assets at least annually.
CIP-007 Cyber Security – System Security Management
Recommendations Require testing documentation be
retained to prove testing was performed in accordance with the test plans.
Testing should focus on the impact to cyber security controls rather than functionality. The standard does not require functionality testing but requires testing of ALL security controls.
CIP-007 Cyber Security – System Security Management
Recommendations (cont.) If tools are used for implementing changes,
these tools also need to be tested to ensure they will not adversely affect the systems.
Work with all vendors of systems and applications of applicable cyber assets to identify and document which ports and services are required for normal and emergency operations. Consider running manual or automated ports and services scans as part of normally scheduled maintenance on cyber assets (as part of the test plan for example).
CIP-007 Cyber Security – System Security Management
Recommendations (cont.) Entities should consider leveraging
corporate level patch management program for tracking, evaluating, testing, and installing applicable cyber security patches required for all Cyber Security Assets within the ESP.
Entities should understand the scope of their patch management programs and ensure that security patches for all applications, operating systems, databases and firmware is being actively tracked.
CIP-007 Cyber Security – System Security Management
Recommendations (cont.) For patches that cannot be tracked
automatically, develop a tracking form listing the URL where patches and updates are posted to help streamline the identification of new patches.
Develop procedures designed to identify and evaluate patches and updates.
CIP-007 Cyber Security – System Security Management
Recommendations (cont.) Ensure testing of anti-virus and malware
signatures is part of the process. Consider leveraging a corporate level program for updating anti-virus and malware if one does not exist. Successful installation of signatures on corporate systems can be leveraged to satisfy the testing requirements of the standard.
CIP-007 Cyber Security – System Security Management
Recommendations (cont.) Ensure technical and procedural controls
exist to minimize the risk of unauthorized system access by shared and system accounts and that these procedures are followed when specified events occur.
Employee training and accountability is critical to ensuring adherence to security practices.
CIP-007 Cyber Security – System Security Management
Recommendations (cont.) Test cyber asset logging capabilities in a
pre-production environment prior to moving to production to ensure the assets are properly configured to send automated events to a centralized logging server and/or generate event logs for manual review.
Implement automated review systems to reduce manual log reviews.
CIP-007 Cyber Security – System Security Management
Recommendations (cont.) Implement and document procedures
that ensure all cyber assets within the ESP are properly configured to log to a centralized location or identified as a system requiring manual review.
File a TFE if logging capabilities do not exist.
Documented, repeatable methods help ensure consistency in your compliance program.
CIP-007 Cyber Security – System Security Management
Recommendations (cont.) Implement documented procedures for
disposal or redeployment of sensitive electronic media which include the records proving erasure or destruction occurred.
Tracking electronic media (acquisition, deployment, destruction) is necessary to prove compliance.
Refer to NIST Special Publication 800-88; Guidelines for Media Sanitation for methods for destroying or erasing electronic media.
CIP-007 Cyber Security – System Security Management
Recommendations (cont.) Consider leveraging corporate level
vulnerability assessment programs where they exist so long as they meet the requirements of CIP-007 R8.
Vulnerability assessments can be included in the test plans to help minimize the scope of the annual assessment. As systems are changed and tested, vulnerability assessments are performed to satisfy the annual assessment requirement.
CIP-007 Cyber Security – System Security Management
Recommendations (cont.) Ensure the documented vulnerability
assessments explicitly cover the specific items required in the Standard.
Create comprehensive document review procedures to ensure review of all CIP-007 documentation is performed and implement methods to ensure the review occurs annually.
Summary
Need to have an overall approach to compliance.
There is no substitute for documented procedures.
Procedures must show implementation. Notable rise in Violations of CIP-006
Physical Security in 2010, heading for the Top 10 List.