misoperations workshop june 5 - 6, 2018 misoperations workshop … misoperations... ·...

Misoperations WorkshopJune 5 - 6, 2018

W e s t e r n E l e c t r i c i t y C o o r d i n a t i n g C o u n c i l

A discussion of issues and best practices with industry colleagues

Mis

op

era

tio

ns

Wo

rksh

op

Welcome to the WECC Misoperations Workshop

This workshop was started in 2017 to provide stakeholders and industry experts a forum to discuss issues and solutions related to Protection System Misoperations. We have gathered a wonderful group of your peers to discuss commissioning, root cause analysis, corrective action plans, model and settings, human performance, and knowledge transfer. We hope you enjoy the next two days and leave with connections, insights and actions to help you ensure continued excellence.

All WECC meetings are conducted in accordance with the WECC Antitrust Policy and the NERC Antitrust Compliance Guidelines. All participants must comply with the policy and guidelines. This meeting is public— confidential or proprietary information should not be discussed in open session. Please contact WECC legal counsel if you have any questions.

Workshop Dining

Get to know your fellow participants. Lunch will be provided on Tuesday. Although many participants may already be familiar with each other, some of you may find yourselves dining with strangers.

WECC has organized a dine o’round at nearby restaurants on Tuesday evening—we invite you to continue discussions of workshop topics with your new and familiar acquaintances.* A sign-in list will be available at the beginning of the workshop.

Breakfast will be provided on Wednesday morning. We encourage you to continue your discussions from the night before with your fellow participants as new friends.

On Wednesday afternoon lunch will be provided. Now, we hope that you consider your fellow participants teammates!

*Attendance for the dine o’round is on a volunteer basis. Each participant is responsible to purchase their food and beverages.

June 5

Noon - 1:00 Lunch with Strangers Join us in the atrium downstairs for lunch before the meeting.

1:00 - 1:30 Opening Remarks

1:30 - 3:30 Commissioning

Developing a Protection System Commissioning Process Dave Sydor, BC Hydro

Substation Equipment Commissioning Checklist and Modules Alberto Quinonez and Rafael Martinez-Chavez, TEP

Commissioning: Best Practices and Possible Traps Douglas Kirby, LDWP and Dan Shield, AESO

3:45 - 4:50 Root Cause Analysis

Understanding the Need for Root Cause Analysis Ed Ruck and Rick Hackman, NERC

4:50 - 5:00 Closing Remarks

5:00 Dinner with Acquaintances Sign-up at the registration

Safety

In case of emergency:

• Exit the building using the stairs.

• Follow a WECC employee in a florescent vest to the parking lot.

• Stay together for a headcount.

Microphone Use

Always use a microphone when speaking:

• The green light indicates the mic is on and unmuted.

• A yellow light indicates the microphone is muted.

• Press the power button once to mute and unmute.

Agen

da

June 6

7:30 - 8:00 Breakfast with Friends

8:00 - 8:15 Opening Remarks

8:15 - 9:45 Root Cause Analysis

Understanding the Need for Root Cause Analysis Ed Ruck and Rick Hackman, NERC

Why Root Cause Analysis? Discussion

10:00 - Noon Model & Settings

Improving Short Circuit Modeling Tyson Niemann, WECC

Ground Overcurrent Protection Discussion Randy Spacek, Avista

Relay Setting Validation and the Four Areas Where Relay Settings Go Wrong Dean Bender, BPA and David Beach, PGE

Noon - 1:00 Lunch with Teammates

1:00 - 2:15 Corrective Action Plan

How to Build a Successful Corrective Action Plan Curtis Sanden, SCE

Questions about Corrective Action Plan Reporting Discussion

2:30 - 3:45 Human Performance

Connecting as a Team Mark Savage, PSE

Misoperations: It’s Not Personal. It’s Personnel! Carla Holly, BP Wind Energy

3:45 - 5:00 Knowledge Transfer

Knowledge Transfer: More Than Just Hype, Learning How to Share Your Knowledge Robert Eubank, Peak and Deveny Bywaters, WECC

Biographies

David Beach, Principal Engineer, Portland General Electric

David has been with Portland General Electric (PGE) since the end of 2006, working in and with the System Protection Group the entire time. Presently David focuses on protection operations of the transmission system of today and the design of the transmission protection for tomorrow in the Operational Technology Support Services group.

David believes that there is a discernable cause for every misoperation; some are easy to discern and others are interesting. He is a contributor to the Electric Power & Transmission & Distribution forum on Eng-Tips.com, and has authored several papers for the Western Protective Relay Conference, one of which addressed the challenges of getting from rule of thumb zero sequence impedances to values that allow a model to match actual events.

David has a Bachelor’s Degree from California State University, Fresno and a Master’s Degree from the University of Idaho. He is a registered Professional Engineer in California, Oregon, and Washington as well as a Senior member of the IEEE. David has been participating in the WECC Relay Workgroup since 2008.

Dean Bender, Senior Engineer, Bonneville Power Administration

Dean Bender is a Senior Engineer for Bonneville Power Administration’s System Protection and Control Technical Services Group, with responsibilities including guidance and training for engineers throughout BPA’s system. Prior to his current role, Dean spent a couple of years traveling throughout the Pacific Northwest commissioning high voltage power system equipment as a Test and Energization Engineer, and later became a protection engineer for various locations within the BPA territory.

Dean has a B.S. in Electrical Engineering from Portland State University. When he’s not working, Dean spends his time scouring the mountains of the west for big game or plying the waters of the Pacific for salmon. Recently he has taken up accelerating down snow-covered slopes with boards strapped to his feet, trying his best to avoid colliding with immobile objects.

Deveny Bywaters, Training Manager, Western Electricity Coordinating Council (WECC)

Deveny Bywaters is the Training Manager for the Western Electricity Coordinating Council. With 28 years of experience in Training and Talent Development, Deveny finds Human Performance a fascinating element of training. What does it take to motivate a learner? What is the best way to facilitate knowledge transfer? How are hidden biases affecting performance? These are the kinds of questions that keep a trainer on their toes!

After a career in several different software training companies, Deveny entered the electric utility business at Bonneville Power Administration. She has trained hundreds of end users, developed corporate training programs, implemented a corporate online university, and managed cross-functional teams on topics ranging from electric utility GIS, substation, communications and protection asset management, billing, diversity, leadership, and performance management.

Biog

raph

ies

In her spare time, when Deveny is not reading about human performance or training techniques, she retreats to the garden for relaxation in the summer, and can be found on the ski slopes burning off energy in the winter. As often as possible, Deveny spends time with her three young grandchildren who prove to be a rich resource of inspiration for how people learn. Whether it is field training, human performance training, or six-year-olds training adults, Deveny is always exploring creative ways to learn.

Robert Eubank, RC Trainer, Peak Reliability

Robert has 39 years of utility experience. Starting with five year’s experience on distribution & transmission line crews working with 0kV to 345kV, 20 year’s experience as a distribution, transmission, generation system operator and 14 year’s experience as a system operations trainer. Robert was instrumental in revamping the WECC System Operator Training Program from a lecture based program to a simulation based program.

Robert’s training endeavors have included system operator training, field switchman training, and marketer training. He has provided training for NERC certification exams, continuing education training for NERC certification, system restoration training and training for personnel new to system operations and those personnel not familiar with system operations. Robert is a NERC RC Certified System Operator.

Robert has been recognized as a leading expert in Root Cause Analysis investigations related to switching errors and mis-operations on the BES. Robert has served as the lead system operations contact for commissioning of new generators, sub-stations and other equipment on the BES.

Throughout his career Robert has served on many industry committees including the NERC Personnel Sub-Committee, WECC Operations Training Sub-Committee, WSCC Certification Exam Workgroup, EPRI Operator Training Simulator Workgroup, EPRI Power Switching Safety Task Force (Chair and Co-Chair) and the Bismarck State College Power System Operations Advisory Committee.

Rick Hackman, Senior Reliability Advisor, North American Electric Reliability Corporation

Rick Hackman is with NERC Event Analysis, leading the Lessons Learned program and Failure Modes and Mechanisms development. Previously, he spent eight years with American Electric Power’s Transmission Substation Engineering and Regulatory Compliance groups. He built a complete Root Cause Training course with case studies and videos for AEP Transmission and NATF Operating Experience group. Before that, he had twenty-nine years of Nuclear Power experience including Licensed Reactor Operator, Radiochemist, Shift Technical Advisor, Nuclear Power Systems Trainer for Professionals, Engineering and Management, Contract Root Cause Analyst for Organizational, Management, Human Performance, and Equipment Failures, and Director Root Cause Analysis for Failure Prevention Incorporated. He wrote symptom based power plant equipment failure diagnostic assistance software for EPRI. He has a BS in Chemistry and Biology from Harding University Searcy, AR.

Biographies (cont.)

Carla Holly, Director, Regulatory Compliance, BP Wind Energy

Carla Holly joined BP Wind Energy in April 2008 as the Regulatory Compliance Specialist and is now leading the Electric Regulatory Compliance team in the role of Director, Regulatory Compliance. In her role as Director, she leads the management of BP Wind Energy’s Electric Regulatory Compliance Program, including the NERC Compliance Programs for twelve of BP Wind Energy’s registered entities. Carla is also responsible for maintaining compliance with FERC Regulations as well as with the market rules and protocols in the ERCOT, PJM, and SPP regions.

Carla has 17 years of experience in the power industry with a background in settlements, real-time trading and operations, and day-ahead power optimization. Carla holds a bachelor’s degree in Finance from Texas A&M University and is an MBA graduate from the University of Houston-Clear Lake. She also maintains a Reliability Coordinator certification with NERC. Carla currently serves as the chair for WECC’s Generator Operator Working Group as well as America Wind Energy Association’s (AWEA) NERC Working Group.

Douglas Kirby, Assistant Manager Substation Operations, L.A. Department of Water & Power

Douglas has over 32 years in the field of protective relay engineering, with expertise in operations and maintenance, commissioning capital projects, relay settings, and fault analysis. Douglas has been a protection supervisor with responsibilities across generation, transmission and distribution; and a project leader for Substation Automation projects, maintenance management implementations, and modernization of a pump storage generation facility. He co-authored a paper on a ground detection system for an ungrounded delta distribution network and served as the chairperson of LADWP’s Remedial Action Team. He is responsible for the operations of 170 substations and currently participates in the Remedial Action Reliability Subcommittee (RASRS) and the Relay Work Group (RWG).

Rafael Martinez-Chavez, T&D Engineering Supervisor, Tucson Electric Power

Rafael is a Supervisor of Substation Engineering at Tucson Electric Power and is responsible for the delivery of substation design, material procurement, construction management support and commissioning. Rafael has 11 years’ experience in the electric utility industry, and has played key roles in many areas including engineering, design, and construction. He has lead substation and line projects on voltages from 500kV to 13.8kV and has commissioned 100% of designed projects. Rafael holds a B.S. in Electrical Engineering from New Mexico State University with a Power emphasis.

Tyson Niemann, System Stability Engineer, Western Electricity Coordinating Council

Tyson Niemann is an Associate Engineer at Western Electricity Coordinating Council, with responsibilities including short circuit analysis, Under Frequency Load Shedding studies, and Remedial Action Scheme vendor development. As a participant in WECC’s Engineering Development Program, Tyson worked on projects across multiple departments, including compliance and System

Biog

raph

ies

Adequacy Planning, before accepting a permanent position on the System Stability Planning team.

Prior to WECC, Tyson worked for seven years as electrical technician, building and installing roller coasters in China. Tyson holds a Bachelor of Science degree in Electrical Engineering from Utah State University. Tyson enjoys riding around Moab in a RZR, and spending quality time outdoors with his family.

Alberto Quinonez, T&D Engineering Supervisor, Tucson Electric Power

Alberto is the Supervisor of the Substation Engineering group at Tucson Electric Power and is responsible for the delivery of substation design and material allocation. Alberto has 13 years’ experience in the electric utility industry, with key roles in areas including engineering, design, construction and project management. He has lead substation and line projects on voltages from 500kV to 13.8kV and has commissioned GIS substations, Series Capacitors and SVCs. Alberto holds a B.S. in Electrical Engineering from New Mexico State University with a Power emphasis.

Victoria Ravenscroft, Manager, Performance Analysis, Western Electricity Coordinating Council

Victoria manages WECC’s Performance Analysis Department, which is responsible for the analysis of historical performance and identification of broad and emerging risks to the Western Interconnection. Prior to joining WECC in 2012, Victoria worked for the Western Interconnection Reliability Advisory Body and the Western Interstate Energy Board as a policy advisor to western Governors and Public Utility Commissions on electricity matters. Before joining the electricity industry, Victoria worked as a founding member of the Center for Energy and Environmental Security at University of Colorado Law School. Here she worked on issues at the nexus of international energy law, environmental stewardship and human rights.

Victoria holds a Juris Doctorate degree from the University of Colorado Law School, where she focused on energy policy and international energy law. In addition to studies in political science and wildlife biology, Victoria rounds out her renaissance approach to life having worked as a professional stage manager for the National Theatre Conservatory.

Ed Ruck, Senior Reliability Engineer, North American Electric Reliability Corporation

Ed Ruck is a Senior Reliability Engineer and is responsible for performing event analyses of power system events and reviewing the Event Analysis reports written by the industry. Ed joined North American Electric Reliability Corporation (NERC) as a Regional Compliance Program Coordinator in October 2004 and was responsible for oversight of regional entities in their implementation of the mandatory compliance program, and since then has held roles in compliance auditing and compliance investigations prior to joining the Reliability Risk Management team.

Biographies (cont.)

Prior to joining NERC, he worked as a Senior Engineer at Mid-America Interconnected Network performing the Reliability Coordinator function. He also worked on EMS maintenance projects and regional planning studies. Ed has a Bachelor of Science degree in Electrical Engineering from the University of Illinois Champaign – Urbana.

Curtis Sanden, Senior Protection Engineer, Southern California Edison

Curtis Sanden, P.E. is a senior protection engineer in the engineering department of Southern California Edison. He received his BSEE from California State Polytechnic University, Pomona in December 1991. He joined Southern California Edison as a protection engineer in January 1992 and has 26 years of experience working in the transmission and generation protection utility industry. He is a registered professional engineer in the state of California and a member of the WECC Relay Work Group.

Mark Savage, System Protection Engineering, Puget Sound Energy

Mark graduated from the University of Washington with a BSME in 1979 and started working for the Pacific Gas & Electric Co. in the areas of energy conservation, customer connections and major customer technical contact. Learning about power quality and how large industry used electrical energy, he decided to pursue further education. In 1983 he enrolled at Washington State University earning a MSEE specializing in power systems and protection. In 1986 he returned to the Seattle area working for Puget Sound Energy in substation design and diagnostic testing of large apparatus. In 1988, Mark became a distribution engineer designing customer connections, relocations, feeder extensions and transmission design. He joined the System Protection Engineering Group in 1996 starting with line data preparation and distribution protection. Mark presently works with transmission protection where he enjoys ground fault studies, POTT schemes, event analysis, and coordinating PSE’s 115/230kV intertie lines. Mark is knowledgeable in the preparation of misoperation reports and Corrective Action Plans leading to improvements in grid reliability.

Dan Shield, Director of Transmission Engineering, Alberta Electric System Operator

Dan Shield, P. Eng., has over 27 years of experience in the field of High Voltage Design and standards development with a focus on protection and control systems for transmission, distribution and generation facilities. At the AESO, Mr. Shield and his department is accountable for the development of all project functional specifications for all transmission connections within Alberta, for the adoption of NERC mandatory reliability standards within Alberta’s regulatory framework, and for the development of all AESO technical rules (Protection Rule, SCADA Rule, Telecommunications Rule, etc).

Mr. Shield is one of AESO’s “Responsible Members” for AESO’s permit to practice Engineering. Prior to AESO, Mr. Shield was the Department Head of Electrical Design leading the Electrical, Telecommunications, SCADA, and Protection and Control Departments within SNC LAVALIN’s ATP division which was an EPC firm designing, procuring, and constructing transmission facilities. Prior

Biog

raph

ies

still, Mr. Shield work for TransAlta Utilities Corporation in the roles of Substation Design Lead and as a System Protection Engineer. Mr. Shield has applied his skill and knowledge on projects across Canada, in the United States, and New Zealand. He has been an active member of the WECC RWG since 2005.

Randy Spacek, System Protection Engineering Manager, Avista Corporation

Randy Spacek has over 30 years’ experience in the electric utility industry. He has performed the role as System Protection Engineering Manager since 2009 at Avista. As manager, he is responsible for NERC compliance activities in the area of Protection Relay and Control (PRC), design and developments of Protection and Integration systems for the Transmission, Distribution and Generation and the group’s technical support efforts of the utility. Randy has been a member of the WPRC steering committee, provided training at the Hands-On Relay School and presently chairs the WECC Relay Work Group.

David Sydor, Manager, P&C Support Services, BC Hydro and Power Authority

Dave Sydor has worked in the protection and control field with BC Hydro for more than 17 years. His primary focus areas have been the installation, testing and commissioning, and operation and maintenance of protection systems. Dave is a founding member of a protection and control team that developed and implemented a new commissioning process for BC Hydro’s protection and control, and telecommunications systems, and is responsible for the development and management of related commissioning standards.

Dave has an Electrical Engineering degree from the University of Victoria, in British Columbia, Canada. He is a Registered Professional Engineer in the Province of British Columbia. Dave is a loving husband, father of 3 kids, and family scapegoat from which all bad things originate. In his spare time, Dave enjoys being active with his family, participating in sports such as softball and hockey, and coaching youth sports.

Excellence must start at the very beginning.

Content Guiding Questions

The commissioning process is crucial to ensuring new equipment is installed correctly and functions as intended. Crossover and shared responsibilities between contractors and entity personnel increase the complexity of this process and introduce opportunities for error. Processes that account for potential complications can reduce the number of errors and resulting misoperations. Approaches that consider both the human factor—e.g., the makeup of a commissioning team—and methods for documenting the commissioning process can reduce issues resulting from the commissioning process. This section will provide perspectives on excellent practices for commissioning.

Does your company have a standardized approach to commissioning?

What works about your current process, and what could be improved?

Developing a Protection System Commissioning Process

David Sydor, BC Hydro

In order to ensure a protection system is ready for service, its installation and performance must be verified and validated in accordance with the design objectives, specifications and factory tests.

Substation Equipment Commissioning Checklist and Modules

Alberto Quinonez, TEPRafael Martinez-Chavez, TEP

Tucson Electric Power (TEP) developed checklists and modules to reduce equipment operations during commissioning. This presentation will explain how TEP uses its checklists to improve, standardize and maximize its available resources.

Commissioning: Best Practices and Possible TrapsDouglas Kirby, LDWPDan Shield, AESO

How can you ensure your commissioning practices are the best? This presentation will provide a sneak peak of the Relay Work Group’s white paper concerning best practices and possible traps associated with commissioning. Questions and input from participants are welcome! Co

mm

issi

onin

g

Commissioning

Do you have something similar to a commissioning checklist? Has it improved your commissioning testing process?

What processes would benefit from a checklist?

What information would be helpful to see the RWG white paper on commissioning?

How do you incorporate lessons learned from misoperations in your commissioning process?

NOTESDeveloping a Protection System Commissioning Process

Your Protection System Commissioning Process

Does your company have a documented Protection System Commissioning Process?

If yes, who owns it (e.g., a senior manager, a group of field technicians)?

Is the process followed consistently?

NOTES

Com

mis

sion

ing

Draw your Protection System Commissioning Process (if it is undocumented, draw it the way you understand it).

Does it make sense to you? Does it work well?

Exercise

Your Ideal Protection System Commissioning Process

Project and Quality Management

What are the key items you believe need to be part of your company’s Protection System Commissioning Process?

• Documentation• Review• Approval• Acceptance• Archival

What would your company’s senior management want to ensure?

Why are these things important?

NOTES

Com

mis

sion

ing

Draw your ideal process. You might consider the following:

• What are the key drivers for you?• Who would own this process?• Who are the key stakeholders?

Exercise

NOTESA Protection System Commissioning Process that Works for Your Company

Some questions to ask while building your company’s desired Protection System Commissioning Process:

• Is there a Professional who is taking responsibility for the Protection System design?

• Will there be consulting firms or contractors performing any component of the Protection System design or testing?

• Does your company have documented commissioning standards for Protection Systems?

• What type of relationship does your design/engineering team have with the site testing resources?

• What type of documentation do you or your company require?

• Are commissioning records used as a maintenance baseline, and will they be referred to during future Protection System maintenance?

• What does your operations team need for a new/modified Protection System to be placed in service?

• What does your asset management team need for a new/modified Protection System to be placed in service?

Com

mis

sion

ing

In 2017, 22% of misoperations were attributed to as-left

personnel errors and AC/DC systems, issues associated with

human performance during commissioning.

NOTESSubstation Equipment Commissioning Checklist and ModulesPre-Commisioning Meeting Checklist

Project Name:Project Number:Conference Meeting Date:

Per the conference meeting on the date above, the individuals below have indicated that their work is complete, pre-checked, and ready for commissioning.

EMSOMSElectronicsRelaySubstation C&MMeteringConstruction ManagerAutomation EngineeringProtection EngineeringProject EngineerAsset ManagementSystem Control and ReliabilityProject Management

“10 minutes can save you... 10 hours or more of a headache.”

Com

mis

sion

ing

Upgrade Checklist

When equipment needs to be upgraded (for example breakers), the use of an upgrade checklist helps to standardize the document packet used for the upgrade. This allows for efficient and effective resource allocation and staff familiarity with equipment to help reduce the probability of errors when upgrading equipment.

NOTES

Best Practices for Relay Commissioning

• Verify intended protection schematic is consistent with company design standards.

• Confirm that actual wiring matches schematic design standard. Resolve differences.

• Verification of relay panel wiring at panel shop or site before installation.

• Load production relay settings and test to make sure desired setting is installed.

• For first time installation of new ‘Standard’ settings, perform detailed testing and evaluation.

• If this is an upgrade to an existing installation, utilize methods to identify wiring that will be involved in isolation and removal. Color coding prints to identify circuits to remove and retain, as well as new circuits.

• Point to point wiring verification of all protection system wiring. Check against intended design.

• Verify all CT/PT ratios against setting sheets and document they are properly set.

• Do secondary tests on CT’s. Remove shorting wires on connected CT circuits.

• Check CT circuit for proper grounding. Single point of grounding is preferred to prevent ground loops.

• PT’s need to be tested when they are new using established PT testing procedures by your company. If existing or when initial testing is complete, the load test will be used to validate PT phasing and polarity against a known source such as an adjacent circuit.

• Use current as a secondary method to check that all CT wiring has been verified. For instance, push from the CT for each phase to neutral to verify that the neutral leg of the circuit has not been accidentally left open.

• Verify that proper settings have been loaded into the relay.

Commissioning: Best Practices and Possible Traps

NOTES

Com

mis

sion

ing

• Verify all telecommunication equipment is operational and programmed as desired.

• For systems with Telecommunications aided protection schemes it is advisable to generate dynamic test routines using state simulated events based upon relay system fault parameters. Satellite synchronized test equipment is necessary for testing at remote facilities.

• Trip check each relay for all phases and verify each expected output. Use schematic drawing as a reference to make sure all inputs/outputs are verified.

• Check all SCADA alarms to System Operator or monitored location for alarm/trip condition.

• Perform load test with system load to verify overall relay performance.

• Evaluate load test results against expected values. Flag any discrepancies and follow up.

• Submit complete test result package to Supervisor or O&M team for use as baseline information for future testing and PRC-005 compliance.

How much time do you spend on continuous improvement of your practices?Do you review events outside your system and develop lessons learned?Do you have practices that should be included in the best practices list?

Send your tips and tricks to the Relay Work Group for consideration and inclusion in this white paper!

[email protected]

(cont.)

NOTESOverall Best Work Practices

1. Make sure all links are closed (unless open is the desired state).

2. Establish work practice to push current through all links or connections that have been opened to verify that they are closed and there is a closed complete circuit.

3. Make sure CT’s are never open circuited and PT’s are never short circuited.

4. Verify that CT ratios are properly set and that any delta configurations for CT’s are made up properly as designed.

5. Be careful with ground connection interference between Relay Test equipment and Relay system under test.

6. Develop and apply standard relay settings across similar equipment and component types.

7. Peer Review of Relay Settings.

8. Training - develop a training program for Relay Engineers and Relay Technicians that focuses on ‘Teach it, Practice it, Test it.’

a. Fundamental knowledge of Power Systems. b. Knowledge of company standard relay settings for all systems they

are working on (Generation, Transmission, Distribution)c. Application training for devices that they are working on.d. Qualification certification of test personnel for the circuit they are

assigned.

How much time do you spend on continuous improvement of your practices?Do you review events outside your system and develop lessons learned?Do you have practices that should be included in the best practices list?

Send your tips and tricks to the Relay Work Group for consideration and inclusion in this white paper.

[email protected]

Com

mis

sion

ing

Are you enjoying this workshop? Continue the conversation at

the WECC Relay Work Group.

Visit https://www.wecc.biz/oc/Pages/RWG.aspx

for meeting dates, or email [email protected] for more

information.

If I had an hour to save the world, I would spend 55 minutes

defining the problem and 5 minutes solving it.

–Attributed to Albert Einstein


What is the value of Root Cause Analysis?

What is the goal of Root Cause Analysis?

What obstacles does your company face in performing Root Cause Analysis?

Understanding the Need for Root Cause AnalysisEd Ruck and Rick Hackman, NERC

When we hear about problems today, it’s often associated with “who is to blame?” or “what failed?” Yet, if we were to look deeper, we would find that many times, the human is not to blame, and there is a reason the part failed. This discussion will help us understand why we sometimes need to take a deeper look at things to help us understand what we as an organization did to set a person up for failure, or allowed a component to fail. We also look at why taking that deeper dive today, could allow us to prevent something worse from happening tomorrow.

Are misoperations putting you out of service? Get to the root of the problem. Root Cause Analysis and Corrective Action Plans help improve reliability by reducing the probability of repeat misoperations.

Many misoperation investigations begin and end with the discovery and resolution of an initiating cause. Although this approach may be sufficient to “patch” an immediate issue, it can also overlook underlying causes and the extent of condition on a system, which could lead to additional, preventable misoperations in the future. Investigations that incorporate Root Cause Analysis can identify latent errors and failure mechanisms, address issues at the source, and prevent repeat misoperations.

Root

Cau

se A

naly

sis

Root Cause Analysis

Method When to Use Advantages Disadvantages Remarks

Task Analysis When the problem appears to be the result of steps taken in a task (just about all the time).

Shows the steps that should have been taken.

Requires personnel and (possibly) equipment time to be performed correctly and completely.

Should be conducted as both a Cognitive Task Analysis (what was the person thinking while conducting the task) and a Contextual Task Analysis (what was going on while the task was being done).

Events and Causal Factor Analysis

For multi-faceted problems with long or complex causal factor chain.

Provides visual display of analysis process. Identifies probable contributors to condition.

Time-consuming and requires familiarity with process to be effective.

Requires a broad perspective of the event to identify unrelated problems. Helps to identify where deviations occurred from acceptable methods.

Change Analysis When cause is obscure. Especially useful in evaluating equipment failures

Simple 6-step process. Limited value because of the danger of accepting wrong “obvious” answer.

A singular problem technique that can be used in support of a larger investigation. All root causes may not be identified.

Barrier Analysis To identify barrier and equipment failures, and procedural or administrative problems.

Provides systematic approach. Requires familiarity with process to be effective.

This process is based on the MORT Hazard/Target concept.

MORT/Mini-MORT When there is a shortage of experts to ask the right questions and the problem is a recurring one. Helpful in solving programmatic problems.

Can be used with limited prior training. Provides a list of questions for specific control and management factors.

May only identify area of cause, not specific causes.

If this process fails to identify problem areas, seek additional help or use cause-and-effect analysis.

Human Performance Evaluations (HPE)

When people have been identified as being involved in the problem cause.

Thorough analysis. None if process is closely followed.

Requires HPE training.

Kepner-Tregoe For major concerns where all aspects need thorough analysis.

Highly structured approach focuses on all aspects of the occurrence and problem resolution.

More comprehensive than may be needed.

Requires Kepner-Tregoe training.

Fault Tree Analysis Normally for equipment-related problems.

Provides a visual display of causal relationships.

Does not work well when human actions are inserted as a cause.

Uses Boolean algebra symbology to show how the causes may combine for an effect.

Cause and Effect Charting (e.g., Reality Charting®)

For any type of problem. Visual display showing cause sequence.

Provides a direct approach to reach causes of primary effect(s). May be used with barrier/change analysis. Focus is on best solution generation.

May not provide entire background to understand a complex problem. Requires experience/knowledge to ask all the right questions.

Requires knowledge of the Apollo Root Cause Analysis techniques. RealityCharting® software may be used as a tool to aid problem resolution.

Cause Analysis Methods Matrix

NOTES

Root

Cau

se A

naly

sis

Method When to Use Advantages Disadvantages Remarks

Task Analysis When the problem appears to be the result of steps taken in a task (just about all the time).

Shows the steps that should have been taken.

Requires personnel and (possibly) equipment time to be performed correctly and completely.

Should be conducted as both a Cognitive Task Analysis (what was the person thinking while conducting the task) and a Contextual Task Analysis (what was going on while the task was being done).

Events and Causal Factor Analysis

For multi-faceted problems with long or complex causal factor chain.

Provides visual display of analysis process. Identifies probable contributors to condition.

Time-consuming and requires familiarity with process to be effective.

Requires a broad perspective of the event to identify unrelated problems. Helps to identify where deviations occurred from acceptable methods.

Change Analysis When cause is obscure. Especially useful in evaluating equipment failures

Simple 6-step process. Limited value because of the danger of accepting wrong “obvious” answer.

A singular problem technique that can be used in support of a larger investigation. All root causes may not be identified.

Barrier Analysis To identify barrier and equipment failures, and procedural or administrative problems.

Provides systematic approach. Requires familiarity with process to be effective.

This process is based on the MORT Hazard/Target concept.

MORT/Mini-MORT When there is a shortage of experts to ask the right questions and the problem is a recurring one. Helpful in solving programmatic problems.

Can be used with limited prior training. Provides a list of questions for specific control and management factors.

May only identify area of cause, not specific causes.

If this process fails to identify problem areas, seek additional help or use cause-and-effect analysis.

Human Performance Evaluations (HPE)

When people have been identified as being involved in the problem cause.

Thorough analysis. None if process is closely followed.

Requires HPE training.

Kepner-Tregoe For major concerns where all aspects need thorough analysis.

Highly structured approach focuses on all aspects of the occurrence and problem resolution.

More comprehensive than may be needed.

Requires Kepner-Tregoe training.

Fault Tree Analysis Normally for equipment-related problems.

Provides a visual display of causal relationships.

Does not work well when human actions are inserted as a cause.

Uses Boolean algebra symbology to show how the causes may combine for an effect.

Cause and Effect Charting (e.g., Reality Charting®)

For any type of problem. Visual display showing cause sequence.

Provides a direct approach to reach causes of primary effect(s). May be used with barrier/change analysis. Focus is on best solution generation.

May not provide entire background to understand a complex problem. Requires experience/knowledge to ask all the right questions.

Requires knowledge of the Apollo Root Cause Analysis techniques. RealityCharting® software may be used as a tool to aid problem resolution.

NOTESExample Event

Brief Description of Event

At 0645 an initiating fault on the 500 kV B-G #1 line resulted in:• The 500kV B-G #1 line locking out at both ends• The 500 kV C-B #1 and #2 lines being open-ended• Tripping of Generation C Units 1, 2, 3 and 4 on overspeed

Note: The B-G #1 line was patrolled to investigate and determine the cause of the initiating fault and no cause found. The line was successfully returned to service.

A single line to ground fault on 500kV line BG resulted in the line locking out after an unsuccessful reclose attempt at the station B end. Coincident with the initial trip, Station C breaker #5 tripped. Coincident with the reclose at Station B, the Station C #6 breaker tripped. Upon losing both lines from Generating Station C, all four units tripped on overspeed.

Identify contributing causes of the event to the extent known.

Identify any Protection System misoperations to the extent known.

Identify any GADS, DADS, TADS, or Protection System misoperations reports that will be submitted.

An initiating fault on the 500 kV B –G #1 line resulted in potential misoperation relay openings of CB #5 and CB #6 breakers at Station C. Station C CB #5 and #6 relays have been taken out of service and are being tested to confirm whether or not they misoperated.

Potential Relay Misoperation of CB #5 and CB #6 breakers at Station C.

GADS and TADS will be reported. Any misoperations identified will be reported.

1. If a one-line diagram is included, please provide an explanation.

Single lines showing Stations C, B and G.

Narrative

NOTES

Root

Cau

se A

naly

sis

Initial Line Up

06:45:00.000 Initial Fault

Generating Station C


Switching Station B

Switching Station B

Switching Station G

Switching Station G

What should happen next if...

1. The fault is “temporary” and clears:

2. The fault is “permanent” (does not clear):

Exercise

NOTES 06:45:00.000 Initial Fault

06:45:00.005



Switching Station B

Switching Station B

Switching Station G

Switching Station G

Is this what you expected?

Why or why not?

Exercise

NOTES

Root

Cau

se A

naly

sis

06:45:00.300 Testing Fault

06:45:00.305 Fault Continues

06:45:00.008 Fault Continues




Switching Station B

Switching Station B

Switching Station B

Switching Station G

Switching Station G

Switching Station G

NOTES

Identify the Problem

What happened and why?

1.

2.

3.

4.

Draft the Problem Statement

What?

When?

Where?

Significance:

Exercise

NOTES

Root

Cau

se A

naly

sis

Establish Cause and Effect between Conditions and Actions

What information do we have?

Where do we still have questions?

Where could we pursue a similar line of questioning for another action or condition?

Where could we pursue different avenues of investigation?

What are possible causes of these circuit breaker trips?

What could you do to prove or disprove any of these potential cause(s)?

Exercise

Food for Thought

Over dinner, consider the following questions...

• Why should we use Root Cause Analysis?• Why not use Apparent Cause Analysis (e.g., “The Why Staircase”)?

• What is the goal of Apparent Cause Analysis?• What is the goal of Root Cause Analysis?

• What prevents your company from performing Root Cause Analysis?

NOTESBreaker Failure Example

Station A

Station A

Station A

Station A

NOTES

Root

Cau

se A

naly

sis

Scenario 1


Final outcome for this scenario

Narrative

Nothing found. Put back into service.

On June 23, 2017, the Station A 138kV breaker B breaker failure protection operated when Capacitor Bank C was closed. The breaker failure operation cleared the 138kV bus by opening breakers A, C, D, E, and F.

What could cause this?

Exercise



The cause remains unknown.

Station A breaker and B breaker failure protection.

When shunt capacitor C was closed in, for voltage control, breaker B breaker failure operated de-energizing the Station A 138kV bus. After exhaustive testing, no cause was found and everything was put back into service.

NOTESScenario 2


Final Outcome for this scenario

Narrative

Nothing found. Put back into service.

On August 2, 2017, the Station A 138kV breaker B breaker failure protection operated when Capacitor Bank C was closed. The breaker failure operation cleared the 138kV bus by opening breakers A, C, D, E, and F.

What could cause this?

Exercise



The cause remains unknown.

Station A breaker and B breaker failure protection.

When shunt capacitor C was closed in, for voltage control, breaker B breaker failure operated de-energizing the Station A 138kV bus. After exhaustive testing, no cause was found and everything was put back in service.

NOTES

Root

Cau

se A

naly

sis

Scenario 3


Final outcome for this scenario

Findings

Breaker failure initiate contact from the SEL 321 relay was partially conducting at times. This contact is a hybrid solid state contact that would build up voltage and initiate breaker failure randomly.

A 10k ohm resistor was connected from the contact to negative to pull down this stray voltage build up, and not initiate breaker failure randomly.

Narrative

On August 2, 2017, the Station A 138kV breaker B breaker failure protection operated when Capacitor Bank C was closed. The breaker failure operation cleared the 138kV bus by opening breakers A, C, D, E, and F.



Leakage current through a solid state contact erroneously triggered the breaker B breaker failure.

Station A breaker B breaker failure protection.

When shunt capacitor C was closed in, for voltage control, breaker B breaker failure operated de-energizing the Station A 138kV bus. After extensive testing, it was determined that the breaker failure initiate contact from the SEL 321 relay was partially conducting at times. This contact is a hybrid solid state contact that would build up voltage and initiate breaker failure randomly. A 10k ohm resistor was connected from the contact to negative to pull down this stray voltage build up, and not initiate breaker failure randomly.

In 2017, 45% of misoperations were linked to

model and setting issues.


Incorrect settings, logic, or design are the cause of almost half of the misoperations in the Western Interconnection and are the leading cause of misoperations nationally. Many factors contribute to this issue, including model inconsistencies between neighbors and miscalculation of settings.

Established processes for updating and sharing short-circuit model information can improve setting consistency and accuracy by ensuring that settings are based on timely, accurate models. Validating settings prior to applying and testing them in the field creates opportunities to catch errors and prevent misoperations before they occur. This section provides information on inconsistencies between neighboring short-circuit models and an RWG initiative underway to create an evaluation checklist on settings validation.

How do you ensure your short-circuit model matches your neighbors for common elements?

Do you review your ground overcurrent settings?

What obstacles have you faced in setting your protection systems?

How does your company handle settings validation?

How does your company learn from incorrect settings when they are discovered?

Improving Short Circuit ModelingTyson Niemann, WECC

This presentation explains how WECC and industry are partnering to improve short-circuit modeling coordination, consistency, and conversion issues in the Western Interconnection.

Ground Overcurrent Protection DiscussionRandy Spacek, Avista

One-quarter of incorrect setting misoperations in the Western Interconnection are attributed to ground overcurrent settings. The presentation will provide a review of the 2016-2018 data to provide common reasons for ground overcurrent misoperations and a discussion for possible solutions.

Relay Setting Validation and the Four Areas Where Relay Settings Go Wrong

Dean Bender, BPAIn order to properly validate relay settings, you need to understand how and where improper settings occur. This presentation will outline four common traps that can lead to incorrect setting misoperations.

Mod

el &

Set

ting

s

Model & Settings

NOTESImproving Short Circuit Modeling

How consistent are short circuit models among neighboring entities in the Western Interconnection?

Modeled Reality

Where do differences occur?

• Do planning and protection engineers coordinate short circuit models consistently?

• Do neighboring utilities share and use each other’s models?

• What about sub-regional planning groups?

• Do software settings affect fault calculations?

• Does using a different software package matter?

NOTES

Mod

el &

Set

ting

s

WECC Short Circuit Analysis

• Requested data from entities on short circuit model cases

• Ran fault calculations for each individual entity

• Compared all fault calculations based on key identifiers (bus name + voltage)

• Identified inconsistencies between neighboring fault calculations

NOTESGround Overcurrent Protection Discussion

1. How often are ground settings and coordination checked?

2. Under what conditions are the settings checked, i.e. changes to system?

3. What criteria do you use for setting ground instantaneous overcurrent protection?

4. Are the remote terminal settings reviewed at the same time?

5. How many busses out do you model for coordination?

6. What tests do you perform in commissioning to verify proper operation? (directionality, polarizing, in-service checks)

Exercise

Best Practices

• Mutual coupling can have a significant impact on how ground fault conditions are perceived by protective relays. System modeling of mutual coupling and post-operation fault analysis are essential to verify that relay settings match with model assumptions.

• Ground instantaneous overcurrent (50G) is a common and well-understood protection scheme, but must be periodically reviewed for coordination with system changes and contingencies. The 50G element should be set greater than the maximum external fault current plus some margin.

• Ground time-overcurrent (51G) is a proven and effective method for system protection. GTO provides good sensitivity, non-directionality, simplicity of testing, and when used non-directionally an independence from the requirement for PT signals. Like its ground-instantaneous counterpart, coordination studies are required due to fault level variability, and the effects of mutual coupling must be considered.

NOTES

Mod

el &

Set

ting

s

Relay Setting Validation and the Four Areas Where Relay Settings Go Wrong

How we think it should respond

How it actually responds

Why it responds differently than expected

NOTES

Mod

el &

Set

ting

s

Can you think of a relay misoperation that occurred because the power system behaved differently than anticipated?

Were you able to take what you learned about the power system operation and use it in your future work?

Can you think of a misoperation that occurred because a relay or other protection system element operated differently than anticipated?

Can you think of a relay misoperation that occurred because a relay was not coordinated correctly with another relay?

Can you think of a relay misoperation that occurred because an error was made in getting the correct settings put on or entered into the relay?

Exercise

Corrective Action Plans combine the lessons learned from

Root Cause Analysis with actions to correct and prevent issues

from recurring.


Corrective Action Plans (CAP) are critical for implementing solutions following protection system misoperations. A robust CAP generally includes lessons learned and mitigating steps identified in a thorough Root Cause Analysis. Although entities in the Western Interconnection report completion of CAPs at a high rate (approximately 80%), there has not been an associated reduction in the number of misoperations. This suggests that CAPs may not be effective at reducing the number misoperations or that the lessons learned from misoperations are not being implemented. By improving the CAP process, entities can ensure adoption of lessons learned and may prevent future misoperations.

What elements make up a good CAP description?

How does a CAP help you in your daily work?

What tools do you need to implement a successful CAP?

How to Build a Successful Corrective Action PlanCurtis Sanden, SCE

Corrective Action Plans are designed to improve reliability by reducing the likelihood of repeat misoperations across a system. This presentation examines real CAP examples, outlines the steps necessary to build a successful Protection System CAP program, and provides some tools and best practices to get started.

Corr

ecti

ve A

ctio

n Pl

an

Corrective Action Plans

NOTESHow to Build a Successful Corrective Action Plan

What you need for a CAP process:

• Inventory of composite protection devices, with query search capabilities

ManufacturerPart numberSerial numberVoltage level

• Inventory of settings/design/logic

• Established prioritization of impactHigh impact locations (per CIP standard)Voltage levelPotential for dropping load

• Database or tracking toolActive and completed CAPsTasks and deadlinesCAP owners

Note: It is helpful if the CAP tracking tool aligns with the misops/event database/tracking tool your company uses.

NOTES

Corr

ecti

ve A

ctio

n Pl

an

CAP Steps

Best Practices

• Hold periodic meetings with affected staff to review and discuss the misoperation and status of the Corrective Action Plan. An advantage of these meetings is they keep the misoperations at the forefront of peoples minds. They also ensure that progress is being made and that multiple departments share accountability for the deadlines.

• Create short-term and long-term Corrective Action Plan goals.

• Engage multiple departments in the entire process to take ownership from investigation through to the completion of the Corrective Action Plan.

Half of misoperations in the Western Interconnection

are caused by human error.


Human error is the largest misoperation cause category, accounting for half of the misoperations reported in the Western Interconnection. This category includes misoperations due to errors in relay scheme logic and design, application of designed settings to equipment, and as-left personnel errors. Human error is especially prevalent with microprocessor relays, as these devices are extremely complex and provide ample opportunity for errors. Training, education, and awareness are critical elements to addressing human error related misoperations. This section explores communication and solution strategies for human performance issues.

Do you have technical peer groups in your company to share information?

How does your company handle and discuss human errors?

Do you know how many of your misoperations are caused by human error?

What control measures do you have in place to reduce human error?

Connecting as a TeamMark Savage, PSE

This presentation demonstrates the value of maintaining a cooperative technical environment within an internal group, between peer groups, and across an organization.

Misoperations: It’s Not Personal. It’s Personnel!Carla Holly, BP Wind Energy

This session will provide awareness of the trends in misoperations that are associated with human error and personnel. It will identify solutions to allow for continuous improvement with respect to reducing NERC’s overall misoperation rate. The session will also include best practices from a fellow Registered Entity pertaining to internal controls related to misoperations. After participating in this session, you will be able to apply new knowledge at your organization and realize that a misoperation isn’t personal. It’s personnel!

Hum

an P

erfo

rman

ce

Human Performance

NOTESConnecting as a Team

Internal Department Communication:Technical Meeting Agenda Creation

1. Weekly Technical Meeting

a. Technical agenda topics can be added by individual engineers and are gathered in a collaborative online tool.

b. Meetings are scheduled for longer than normal staff meetings to bring unity to the group and allow for common approaches in methodology.

c. Traditional staff meeting agenda such as investigations/misoperations are also discussed.

2. Pairing transmission protection engineers with distribution engineers allowing for cross over training and progression.

3. Misoperations are viewed as learning experiences, discussion is encouraged, and knowledge gained is applied to both existing and future relay settings.

NOTES

Hum

an P

erfo

rman

ce

Department-to-Department Communication:Relay Settings Commissioning Procedure

How can you improve information sharing between departments in your company?

List behaviors that support department-to-department communications in the Relay Setting Commissioning Procedure:

Exercise

Multi-Department and External Communication:Engineering and Operations Technical Forum

Best Practice - Engineering and Operations Technical Forum

• Began in the 1980 from the need for a cooperative approach to engineering, protection and operations asset and continues with full management support.

• Design attributes, technical issues, settings and testing feedback is the core of discussion

• Special technical topics are added as technologies advance (e.g. microprocessor relays and updated panel designs).

• Standards and other guests not core to technical issues are invited to participate as needed and available.

Hum

an P

erfo

rman

ce

Interested in learning more about human performance?

Join the conversation at the WECC Human Performance Work Group.

Visit https://www.wecc.biz/oc/Pages/HPWG.aspx

for more information.

NOTES

2017 Misoperations by Cause

What is your misoperation rate? Benchmark your organization!

Misoperations: It’s Not Personal. It’s Personnel!

NOTES

Hum

an P

erfo

rman

ce

Improvement Strategies Checklist

Training program for protection engineers

Participate in RWG (or equivalent group) to gather lessons learned in your Interconnection

Documented commissioning process for Protection Systems

Documented process for setting application

Audit settings periodically

Conduct fault studies

Apply CAP to all systems

Is the training recurring?Is it part of the onboarding process?Does it need to include contractors?

Does the process focus on relay settings?

Does it address firmware updates to the relays?Does it address settings during testing/maintenance process (as-found, as-set, as-left)?

Does it consider all NERC Standards (PRC-005, PRC-019, PRC-024)?

“My fundamental philosophy is that you owe it to society to

transfer to them any knowledge you have that might be useful.”

—Leroy Hood


Workforce turnover is an issue facing the entire industry. By 2024, nearly 1 in 4 people in the labor force are projected to be age 55 or over. Ten thousand baby boomers retire each day. As an increasing number of skilled technicians and engineers become eligible for retirement, the loss of system-specific experience and expertise is imminent. Without sufficient practices in place to capture and transfer this knowledge, valuable information can be lost. This section discusses techniques for transferring knowledge to ensure critical information is retained.

How does your company plan for knowledge transfer? Does your company have a knowledge transfer program?

Who handles knowledge transfer in your company?

What is your role in knowledge transfer?

Knowledge Transfer: More Than Just Hype, Learning How to Share Your Knowledge

Robert Eubank, PeakDeveny Bywaters, WECC

Knowledge transfer is the practical problem of transferring knowledge from one person to another (personalization). The biggest challenge isn’t the transfer of information or facts, rather the transfer of how to react or adapt to information or facts. This session explores how a skill or practice can be broken down and transferred, and provides techniques to personalize knowledge transfer in your own situations.

Know

ledg

e Tr

ansf

er

Knowledge Transfer

NOTESKnowledge Transfer: More than Just Hype, Learning How to Share Your Knowledge

Step 1: Identifying Critical Tasks and Activities

There are probably some aspects of your work that only you know how to do. In this step you are developing a list of those tasks and activities. It isn’t necessary to go into detail. Let the questions below stimulate your thinking:

• What are you known for? What are you the “go to” person for?

• What do only you know how to do?

• If you left your position today, what wouldn’t get done because no one else knows how to do it or what to do?

• When you return from a vacation, what work is usually waiting for you because no one else knows how to do it?

• When you have to be away from work, what do you worry about (what work isn’t getting done or what work isn’t being done well)?

• What does your organization rely on you for?

List the tasks and activities below:

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

NOTES

Know

ledg

e Tr

ansf

er

Step 2: Define Each Task and Activity

Think about the essence of the knowledge and experience required to complete each task or activity identified in Step 1. It is not necessary to inventory all your knowledge and experience; simply define the task/activity in more detail. What information or experience do you need to have in order to carry out this responsibility or task? Focus especially on things only you know and that others need to learn. Consider these areas to get you started:

• Knowing key contacts (customers, other entities, NERC, Regional and state entities, federal government contacts, business contacts, etc.)

• Having strong relationships with key customers or coworkers

• Knowing logistics or locations (training rooms, sub-stations, power plants, field offices, etc.)

• Knowing history (system events/issues, reason for procedures, business decisions, etc.)

• Knowing locations of critical files or information

• Knowing how to carry out a task or responsibility

Think through the steps necessary to complete the task and identify areas that are critical to your success. What do you know that others need to learn from you in order to be able to serve your organization as well as you do?

Task or Activity from Step 1:

List the critical knowledge, experience, or skills needed for this task:

Exercise


Critical knowledge, experience, or skills needed for this task:









Exercise

Know

ledg

e Tr

ansf

er

Thank you for attending the

2018 WECC Misoperations Workshop

Follow us

@weccreliability

misoperations workshop june 5 - 6, 2018 misoperations workshop … misoperations... ·...

Documents