application guidance · per-005 requirements are interrelated as part of a systematic approach to...

STANDARDS COMMITTEEM I D W E S T R E L I A B I L I T Y O R G A N I Z AT I O N

Promoting RELIABILITY and Mitigating RISKS to the Bulk Power System

APPLICATION GUIDANCEPER-005 System Personnel Training

MRO PER-005 Subject Matter Expert (SME)TeamDecember 15, 2011

Presenter

Presentation Notes

This presentation summarizes the PER-005 System Personnel Training Application Guide recently created by a volunteer Subject Matter Expert Team.

2

STANDARDS COMMITTEEM I D W E S T R E L I A B I L I T Y O R G A N I Z A T I O N

The MRO Subject Matter Expert Team is an industry stakeholder group which includes subject matter experts from MRO member organizations in various technical areas. Any materials, guidance, and views from stakeholder groups are meant to be helpful to industry participants; but should not be considered

approved or endorsed by MRO staff or its board of directors unless specified.

Disclaimer

The Midwest Reliability Organization (MRO) Standards Committee (SC) is committed to providing training and non-binding guidance to industry stakeholders regarding existing and emerging Reliability Standards. Any materials, including presentations, were developed through the Standards Committee by Subject Matter Experts from member organizations within MRO.

The materials have been reviewed by MRO staff and provide reasonable application guidance for the standard(s) addressed. Ultimately, demonstrating compliance depends on a number of factors including the precise language of the standard, the specific facts and circumstances, and quality of evidence.

These documents may be reproduced or distributed to any person or entity only in its entirety.

2

Presenter

Presentation Notes

The Midwest Reliability Organization Standards Committee is committed to providing training and non-binding guidance to industry stakeholders regarding existing and emerging Reliability Standards. Any materials, including presentations, were developed through the Standards Committee by Subject Matter Experts from member organizations within the MRO. The materials have been reviewed by MRO staff and provide reasonable application guidance for the standard(s) addressed. Ultimately, demonstrating compliance depends on a number of factors including the precise language of the standard, the specific facts and circumstances, and the quality of the evidence. These documents may be reproduced or distributed to any person or entity only in its entirety.

3




Guidance to meet a minimum level of compliance Clarify training terminology Explain how requirements are interrelated Identify key processes (to be documented) Identify key outcomes (to be documented)

Goals of the SME Team

3

Presenter

Presentation Notes

This Presentation summaries the SME team’s recommendations for meeting a minimum level of compliance. Our goals were to: Clarify training terminology Explain how requirements are interrelated Identify key processes (to be documented) Identify key outcomes (to be documented) �These are the recommendations that we plan to take back to our own companies. For more information, please review the Application Guide Document itself.

4



approved or endorsed by MRO staff or its board of directors unless specified. 4

PER-005 SME Team MembersCarla Fitzpatrick, Omaha Public Power DistrictChris Bellefy, Minnesota PowerJohn Pettingill, Xcel Energy – Northern States PowerPat Matson, Madison Gas and ElectricRich Cobb, Midwest ISO Robert Kieborz, Nebraska Public PowerRon Bauer, American Transmission CompanySteve Zimmerman, Wisconsin Public Service Will Behnke, Alliant Energy

Presenter

Presentation Notes

___ ______ ___________ ___ _____ _____ __ ________ ________ ____ __________ ____ _____ ____ _______ ____ ______ ________ ____ ___ _________ ___ _____ _______ _____ ____ __ _____ ______ __ ___ _______ ___ ___ _____ _____ ______ _______ _____ ____ __ ____ _______ ___________ ______ ____ ________ ________ _____ ______ ___ _____ ______ _____

5




Any questions so far?

5

6




1. Title: System Personnel Training

2. Number: PER-005-1

3. Purpose: To ensure that System Operators performing real-time, reliability-related tasks on the North American Bulk Electric System (BES) are competent to perform those reliability related tasks. The competency of System Operators is critical to the reliability of the North American Bulk Electric System.

4. Applicability:4.1. Functional Entities:

4.1.1 Reliability Coordinator.4.1.2 Balancing Authority.4.1.3 Transmission Operator.

Purpose & Applicability of PER-005

6

Presenter

Presentation Notes

The Statement of Purpose is a good starting place in analyzing a standard.. �� It states: “To ensure that System Operators performing real-time, reliability-related tasks on the North American Bulk Electric System (BES) are competent to perform those reliability related tasks. The competency of System Operators is critical to the reliability of the North American Bulk Electric System.” Note several key phrases: real time, reliability related tasks and competent to perform . We will be returning to these phases shortly. PER-005 will replace PER-002 in stages between April 2011 and April 2013. Note that certain PER-002 Training Program specifications are no longer explicitly mentioned. These PER-002 program requirements refer to Scheduling, Instructor qualifications, and the plan for initial and continuing training. These requirements are still relevant, however, since they are implicit in requirement R1 regarding following a systematic approach to training. FERC has also expressed this view in the final rule on this standard that was published in the Federal Register on November 26, 2010..

7




(ADDIE)(R.1.) Use a systematic

approach to training

Analysis – identifies the “requirements”

• Create list of Operator tasks• Conduct annual updating of task list

Design – identifies the “specifications”

• Create learning objectives (aligned with tasks)

Development – creates the “training solution”

• Create training materials (aligned with objectives)

Implementation – is the “actual delivery”

• Deliver training identified (aligned with objectives)• Provide 32 hours of EOP training annually• Train using simulator technology (where applicable)• Verify operator capabilities (on current tasks)• Verify operator capabilities (on new tasks)

Evaluation – is the “continuous improvement”

• Conduct annual evaluation of training program

Requirements of PER-005

7

Presenter

Presentation Notes

Requirement R1 indicates that we shall use a systematic approach to training. �The most widely accepted model for a training system is ADDIE. ADDIE stands for five phases in a systematic approach: Analysis, Design, Development, Implementation, and Evaluation. These phases are interrelated. The outputs of one phase provide the inputs to another phase. As the ADDIE phases are interrelated, so are the PER-005 requirements. In part because they reflect the ADDIE phases. The ten other PER-005 requirements are re-arranged in this list to reflect the sequence of the ADDIE training model. The requirements to create a list of operator tasks and to conduct annual updating of task list fall under “analysis” The requirement to create learning objectives falls under “design” The requirement to create training objectives falls under “development” The following requirements fall primarily under “implementation”: Deliver training identified Provide 32 hours of emergency operations training every 12 months Train using simulator technology Verify operator capabilities (on current tasks) Verify operator capabilities (on new tasks) And finally, the requirement to conduct an annual evaluation obviously falls under “evaluation” �The following slides will address these requirement in greater detail.

8




Complying with PER-005 requirements 1.1 through 3.1 provides most of the documentation you need for this requirement. Document your Processes Polices Procedures Plans Process maps, etc

Document your Outcomes Training records Schedules Evaluation feedback Curriculum, etc

R1. Systematic Approach to Training

8

Presenter

Presentation Notes

The evidence you create in meeting the other PER-005 requirements R1.1 through R3.1 will help provide most of the documentation you need for R1, Systematic Approach to Training. That is, of course,if you adequately document your training processes and outcomes. Your processes are documented with policies, procedures, plans, process maps, etc. Your outcomes are documented with training records, schedules, evaluation feedback, curriculum, etc. We do recommend some additional documentation to cover R1 Systematic Approach to Training. These are listed in the compliance roadmap provided in the appendix of the Application Guide.

9




List of task action statements (units of assigned work)Measurable: numerical results (e.g. within normal operating limits)Observable: can be verified as correct or not (e.g. in accordance with procedure)Suggest grouping related tasks (e.g. voltage control, emergency operations, etc)

R1.1 List of BES company-specificreliability-related tasks

9

Presenter

Presentation Notes

Requirement 1.1 specifies having a task list. At a minimum we need list of task action statements A task is a unit of assigned work. Each task statement usually starts with an action statement (such as evacuate control center; direct load shedding) . A task is observable or measurable. �Requirement 1.1 does not appear to require the Condition and Criteria for each task statement However, you will need this information for R2, Capability Verification. More on this shortly. Also we suggest grouping together related tasks. Tasks can be grouped by duty area or subject area. For example: Voltage control, Emergency Operations, etc.

10




Develop processes for how you: Determine if it occurs in real time Determine if reliability related Validate the tasks you have identified Determine which tasks need qualification training Determine which tasks need continuing training

R1.1. “reliability-related” tasks

10

Presenter

Presentation Notes

In maintaining compliance with R.1.1, develop processes for how you: Determine if it occurs in real time Determine if reliability related Validate the tasks you have identified Determine which tasks need qualification training Determine which tasks need continuing training The last three processes are not explicit in the requirement, but are considered essential in a Systematic Approach to training and are part or the Task Analysis process. OPTIONAL: Determining reliability related. One of the early outcomes in the job and task analysis process is a preliminary list of tasks. These generally come from observation on the job and/or interviews. To classify this list of tasks as reliability-related, determine if the system operator task is: Linked to the NERC Compliance Registry (BA, TOP, or RC)? Specified or implied in a NERC Standard? Linked to a relationship described in the NERC Functional Model? Specified in some other NERC document? Specified in RTO/ISO process that impacts BES?

11




Task Survey

Presenter

Presentation Notes

Task Surveying. As part of Requirement 1, the systematic approach to training ,you need to set priorities on which tasks must be trained on, what is covered in qualification training, what is covered in continuing training, and how often refresher training is delivered. This can be done by surveys, interviews, etc. An example of a tool used in analyzing tasks is a scaling survey such as a DIF Survey. DIF stands for Difficulty, Importance, Frequency The DOE Handbooks provide guidelines in making these decisions

12




Task Details Sheet

Presenter

Presentation Notes

Here is an example of a Job Performance Measure (also known as a Task Detail sheet or JPM). The top three rows are the task statement, condition, and criteria: Notice the addition of steps at the bottom. This type of documentation will help you with the Requirement #2, the Capability Verification You can use these steps in building a supervisor checksheet, training assessment, or other methods to verify capability. More on this in later slides.

13





13

14




Two parts to the requirementPart 1: address changes as they occur Create filter for identifying process or task changes Document the changes (e.g. tools, steps, references, criteria,

objectives, etc) Document your validation process Perform capability assessment within six months (see R2.1.)

Part 2: review the entire task list

R1.1.1 update BES company-specific reliability-related tasks…

14

Presenter

Presentation Notes

Requirment 1.1.1. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall update its list of BES company-specific reliability-related tasks performed by its System Operators each calendar year to identify new or modified tasks for inclusion in training. There are two parts to this requirement: Part one: address the modified or new tasks when they occur.. Create a filter for identifying process or task changes. Document the changes (e.g. tools, steps, references, criteria, objectives, etc.). Document your validation process (emails, surveys, observation, etc.) Perform capability assessment within six months (see Requirement 2.1.). More on this later. Part two: review the entire task list, at least annually, to ensure all tasks are current. Document your review process using emails, surveys, observations, and so on.

15




Objectives include a condition, action and criterion

Tasks drive the learning objectives

Learning objectives drive the training materials

R1.2 …design and develop learning objectives and training materials based on the task list created in R1.1

15

Presenter

Presentation Notes

Requirement 1.2 …states that we shall design and develop learning objectives and training materials based on the task list created in Requirement 1.1 Objectives are the knowledge, skills, and abilities that a student needs to master in order to perform the reliability-related tasks you have identified. Learning Objectives include a condition, action and criterion Here’s an Example: Given a substation one-line diagram, create a switching procedure for isolating a substation breaker, without errors.� Tasks drive the learning objectives while learning objectives drive the training materials. Processes you might consider as evidence of compliance:� Develop a procedure on objective writing; we suggest tying this to the NERC document on writing learning objectives.� Develop a procedure on curriculum development; this might include guidelines and templates on creating OJT guides, computer based training, slide decks, manuals, and so on. Outcomes you might consider as evidence of compliance: The Learning Objectives associated with each task (there is an example on the next slide) The Learning Objectives associated with each training (there is an example coming soon)

16




Task Detail Sheet

Presenter

Presentation Notes

For the Task Detail sheet mentioned earlier, we added the learning objectives that should prepare someone to perform that task. There are Instructional design processes, as shown in the DOE Handbooks, which will guide you in identifying learning objectives for a give task.

17





17

18




R1.2 Learning objectives & training materials

elearning Course – Evacuation Procedure Describe triggers requiring evacuation List items to bring from Control Center List who to contact before leaving Describe where to report

Drill – Evacuation to Backup Control Center Contact Security Operations Center (within 1 minute) Contact back-up control center site (within 1 minute) Transfer phones to Backup Control Center (within 1 minute) Gather items to carry from Control Center (within 2 minute)

18

Presenter

Presentation Notes

Here are two examples of learning objectives driving training materials (and learning strategy): Lower level knowledge objectives can be delivered via lecture or self-study. In this example regarding evacuation, the training is done using an eLearning course> Higher level knowledge and skill objectives can be delivered via simulation or drills and shown in this example. Note that there is a “system” or body of knowledge to writing learning objectives. NERC has clear ideas on what they are looking for. See the reference section of the Application Guide for resources on writing learning objectives.

19




“Deliver” represents the Implementation Phase of ADDIE Delivery can include scheduling of participants, classroom preparation, sign-in of participants, pre-assessment, student engagement, student assessment, making evaluations available, and recordkeeping

R1.3 …deliver the training established in R1.2

19

Presenter

Presentation Notes

Requirement 1.3 …states that we shall deliver the training established in Requirement 1.2 “Deliver” represents the Implementation Phase of ADDIE. The outcome of the successful Implementation phase is the creation of competent employees. Implementation involves the actual delivery of the training. Outcomes can include scheduling of participants, classroom preparation, sign-in of participants, pre-assessment, student engagement, student assessment, making evaluations available, and recordkeeping. Consider documenting processes such as your selection of Instructors and those assigned to conduct capability verification.

20




Evaluation Phase of ADDIEAggregate the course evaluationsOther input (surveys, interviews, advisors, etc)Show how feedback is used in program changes

R1.4 conduct an annual evaluation …to identify any needed changes to the training program and shall implement the changes identified

20

Presenter

Presentation Notes

Requirement 1.4 states that we shall conduct an annual evaluation …to identify any needed changes to the training program and shall implement the changes identified. This requirement represents the Evaluation Phase of the ADDIE Systematic Approach to Training. Evaluation is an ongoing process for identifying the changes needed in training setting, training activities, scheduling, course content, materials, etc. Do not just rely on individual course evaluations Look at both the qualification program and the continuing education program Examine how new needs identified and how these these needs addressed in training What are the impacts on individual performance and team performance. Input can be gathered using the course evaluations, surveys, interviews, advisors, and so on. Record where changes were made based on the input received. Documentation of your evaluation efforts can be found in revisions to training procedures, course tracking forms, course design, and program descriptions.

21





21

22




Capability Verification is part of ADDIE Implementation phase Processes: How to verify; who can perform verification; what to do when someone

fails a verification

Outcomes: Training records showing successful completion of tasks Supervisor check sheets showing task completed Results of learning assessments

R2. verify each of its System Operator’s capabilities to perform each assigned task…

22

Presenter

Presentation Notes

For Requirement 2. we must verify each of System Operator’s capabilities to perform each assigned task… “Capability Verification” is part of the implementation phase. It is a final assessment that training has been successful in creating competent employees. We need to determine how to verify; who can perform verification; and what to do when someone fails a verification. Identify the training settings and the on the job activities to assess an Operator’s performance of a task. Normal operating tasks can usually be assessed on the job. Emergency activities will usually be assessed in a simulated or training environment. As stated in the Standard, evidence can be documents such as training records showing successful completion of tasks with the employee name and date; supervisor check sheets showing the employee name, date, and task completed; or the results of learning assessments.

23




Elements: Condition, Action statement, CriteriaMeasurable: “Given a System Operating Limit violation on the transmission system, implement the correct procedure for the circumstances to mitigate loading to within normal operating limits.”Observable: “Given a tag submitted for scheduling, ensure that all transmission rights are assigned to the tag per the company Tariff and in compliance with NERC and NAESB standards.”

Performance Measure Examples

23

Presenter

Presentation Notes

Previously we described Requirement R1.1 where we need to identify the reliability related tasks performed in our companies. In Requirement two, we need performance measures to assess our System Operators’ ability to perform these tasks. These measures need at least three parts: condition, action statement, critieria (or standard). Here are two examples from the NERC PER-005 Reference Document. The first task regarding System Operating limits is measurable: “Given a System Operating Limit violation on the transmission system, implement the correct procedure for the circumstances to mitigate loading to within normal operating limits.” The second task on tagging is observable: “Given a tag submitted for scheduling, ensure that all transmission rights are assigned to the tag per the company Tariff and in compliance with NERC and NAESB standards.” OPTIONAL CONTENT IN PPT: Conditions: Given alarm indications Given a specific procedure to implement Upon request Criteria: Without error To normal limits Within a timely manner In accordance with procedures In accordance with NERC Standards

24




Capability Verification Checklist

Presenter

Presentation Notes

For Requirement number two, here is an example of a Capability Verification checklist (these are often called scorecards). It is built from the Job Performance Measure (or Task Details form) mentioned before. These Checklists often include the following information: task(s) to be assessed task steps tools, conditions, references needed if applicable standards of satisfactory performance for critical steps name of evaluator name of Operator signature for verification Each step of the task should be observed or assessed. If a step is omitted, a dated and initialed written explanation should be made on the form. All steps should be performed to the stated standard. Check the DOE-Handbooks and other instructional design references for guidance in the design and use of these assessments.

25




How do trainers assess capabilities? See the methodology described previously under requirement, R2. It describes how to determine that an existing task has changed sufficiently to require a new capability verification assessment.

What level of documentation is recommended? Use the same documentation used to meet requirement R1.1.1. (update task list) and requirement R2 (verify capabilities).

R2.1 Within six months…verify each of its System Operator’s capabilities to perform the new or modified tasks

25

Presenter

Presentation Notes

For Requirement R2.1 Within six months…verify each of its System Operator’s capabilities to perform the new or modified tasks How do trainers assess these capabilities? See the methodology described previously under requirement, R2. It describes how to determine when an existing task has changed sufficiently to require a new capability verification assessment. What level of documentation is recommended? Use the same documentation used to meet requirement R1.1.1. (in updating the task list) and requirement R2 (in verifying capabilities).

26




“At least every 12 months”“applicable to its organization”“reflects emergency operations topics”

R3. …provide each of its System Operators with at least 32 hours of emergency operations training

26

Presenter

Presentation Notes

For requirement R3. …provide System Operators with at least 32 hours of emergency operations training This requirement is similar to PER-002, R4, which was superseded by PER-002 on April 1, 2011. As to the interpretation of “every 12 months” most companies use the calendar year while some use the anniversary date of initial qualification. Most interpretations are that companies should continue to follow the same practice they followed under PER-002, Requirement R4. Regardless of their interpretation, Companies should define their methodology for determining this time period. A list of emergency topics is included in the NERC PER-005-1 Reference Document. These emergency topics cover the following areas: Recognition and Response to System Emergencies Operating Policies Related to Emergency Operation Power System Restoration Philosophy and Practices Interconnected Power System Operations Technologies and Tools Market Operations as They Relate to Emergency Operations

27




Applicability: Authority or control over facilities with IROLs Has established guides to mitigate IROL violations

Simulation Technology – replicates the operational behavior of the BES during normal and emergency conditions.

R3.1 …provide each System Operator with emergency operations training using simulation technology

27

Presenter

Presentation Notes

For requirement R3.1 …provide each System Operator with emergency operations training using simulation technology As with requirement R3, the evidence can include training records (rosters, drill scenarios, learning objectives, exercises, assessments, evaluations) associated with this training. As to the question of simulator technology, we took this at face value. The standards says the simulation technology replicates the Bulk Electric System. It does not say that it has to replicate the company-specific system. However, to perform the capability verification of emergency tasks without a company-specific system will require a great deal of instructional design.

28




(ADDIE)(R.1.) Use a systematic

approach to training

Analysis – identifies the “requirements”

• Create list of Operator tasks• Conduct annual updating of task list

Design – identifies the “specifications”

• Create learning objectives (aligned with tasks)

Development – creates the “training solution”

• Create training materials (aligned with objectives)

Implementation – is the “actual delivery”

• Deliver training identified (aligned with objectives)• Provide 32 hours of EOP training annually• Train using simulator technology (where applicable)• Verify operator capabilities (on current tasks)• Verify operator capabilities (on new tasks)

Evaluation – is the “continuous improvement”

• Conduct annual evaluation of training program

Requirements of PER-005

28

Presenter

Presentation Notes

In summary, Requirement R1 indicates that we shall use a systematic approach to training. �The most widely accepted model for a training system is ADDIE which stands for: Analysis, Design, Development, Implementation, and Evaluation. These phases are interrelated. The outputs of one phase provide the inputs to another phase. As the ADDIE phases are interrelated, so are the PER-005 requirements. In part because they reflect the ADDIE phases. The ten other PER-005 requirements are re-arranged in this list to reflect the sequence of the ADDIE training model. The requirements to create a list of operator tasks and to conduct annual updating of task list fall under “analysis” The requirement to create learning objectives falls under “design” The requirement to create training objectives falls under “development” The following fall primarily under “implementation”: Deliver training identified Provide 32 hours of emergency operations training every 12 months Train using simulator technology Verify operator capabilities (on current tasks) Verify operator capabilities (on new tasks) And finally, the requirement to conduct an annual evaluation falls under “evaluation”

29




ConclusionPER-005 requirements are interrelated as part of a systematic approach to training.

Consider going beyond the specifics of each individual requirement, when needed, to maintain this systematic approach.

29

Presenter

Presentation Notes

PER-005 requirements are interrelated as part of a systematic approach to training. � Trainers should consider going beyond the specifics of each individual requirement, when needed, to maintain this systematic approach. Please see the Checklist in the Application Guide for suggested processes and outcomes to use as evidence of compliance.

30




This presentation and white paper can be found on MRO’s website at:

http://www.midwestreliability.org/STA_standards_committee_sme.html

If you have any questions following this workshop, please send them to:

[email protected]

Questions

30

Presenter

Presentation Notes

This presentation and white paper can be found on MRO’s website. If you have any questions following this workshop, please send them to the following email address.

http://www.midwestreliability.org/STA_standards_committee_sme.html

mailto:[email protected]

application guidance · per-005 requirements are interrelated as part of a systematic approach to...

Documents