nureg/ia-0137 "a study of control room staffing levels for

NUREG/IA-0137

* • International Agreement Report

A Study of Control Room Staffing Levels for Advanced Reactors

Prepared by B. P. Hallbert1 , A. Sebok, Institutt for Energiteknikk, OECD Halden Reactor Project D. Morisseau, U.S. Nuclear Regulatory Commission 2

Institutt for Energiteknikk OECD Halden Reactor Project P.O. Box 173 N-1751 Halden NORWAY

1Now with the Idaho National Engineering & Environmental Laboratory 2Now with the Institutt for Energiteknikk, OECD Halden Reactor Project

Office of Nuclear Regulatory Research U.S. Nuclear Regulatory Commission Washington, DC 20555-0001

November 2000

Prepared as part of The Agreement on Research Participation and Technical Exchange under the International Code Application and Maintenance Program (CAMP)

Published by U.S. Nuclear Regulatory Commission

AVAILABILITY OF REFERENCE MATERIALS IN NRC PUBLICATIONS

NRC Reference Material

As of November 1999, you may electronically access NUREG-series publications and other NRC records at NRC's Public Electronic Reading Room at www. nrc. gov/N RC/ADAMS/index.html. Publicly released records include, to name afew, NUREG-series publications; Federal Register notices; applicant, licensee, and vendor documents and correspondence; NRC correspondence and internal memoranda; bulletins and information notices; inspection and investigative reports; licensee event reports; and Commission papers and their attachments.

NRC publications in the NUREG series, NRC regulations, and Title 10, Energy, in the Code of Federal Regulations may also be purchased from one of these two sources. 1. The Superintendent of Documents

U.S. Government Printing Office P. 0. Box 37082 Washington, DC 20402-9328 www.access.gpo.gov/su-docs 202-512-1800

2. The National Technical Information Service Springfield, VA 22161-0002 www.ntis.gov 1-800-533-6847 or, locally, 703-805-6000

A single copy of each NRC draft report for comment is available free, to the extent of supply, upon written request as follows: Address: Office of the Chief Information Officer,

Reproduction and Distribution Services Section

U.S. Nuclear Regulatory Commission Washington, DC 20555-0001

E-mail: [email protected] Facsimile: 301-415-2289

Some publications in the NUREG series that are posted at NRC's Web site address www.nrc.gov/NRC/NUREGS/indexnum.html are updated periodically and may differ from the last printed version. Although references to material found on a Web site bear the date the material was accessed, the material available on the date cited may subsequently be removed from the site.

Non-NRC Reference Material

Documents available from public and special technical libraries include all open literature items, such as books, journal articles, and transactions, Federal Register notices, Federal and State legislation, and congressional reports. Such documents as theses, dissertations, foreign reports and translations, and non-NRC conference proceedings may be purchased from their sponsoring organization.

Copies of industry codes and standards used in a substantive manner in the NRC regulatory process are maintained at

The NRC Technical Library Two White Flint North 11545 Rockville Pike Rockville, MD 20852-2738

These standards are available in the library for reference use by the public. Codes and standards are usually copyrighted and may be purchased from the originating organization or, if they are American National Standards, from

American National Standards Institute 11 West 4 2 n Street New York, NY 10036-8002 www.ansi.org 212-642-4900

The NUREG series comprises (1) technical and administrative reports and books prepared by the staff (NUREG-XXXX) or agency contractors (NUREG/CR-XXXX), (2) proceedings of conferences (NUREG/CP-XXXX), (3) reports resulting from international agreements (NUREG/IA-XXXX), (4) brochures (NUREG/BR-XXXX), and (5) compilations of legal decisions and orders of the Commission and Atomic and Safety Licensing Boards and of Directors' decisions under Section 2.206 of NRC's regulations (NUREG-0750).

DISCLAIMER: This report was prepared under an international cooperative agreement for the exchange of technical information. Neither the U.S. Government nor any agency thereof, nor any employee, makes any warranty, expressed or implied, or assumes any legal liability or responsibility for any third party's use, or the results of such use, of any information, apparatus, product or process disclosed in this publication, or represents that its use by such third party would not infringe privately owned rights.

I

NUREG/IA-0137

International Agreement Report

A Study of Control Room Staffing Levels for Advanced Reactors Prepared by B. P. Hallbert1, A. Sebok, Institutt for Energiteknikk, OECD Halden Reactor Project D. Morisseau, U.S. Nuclear Regulatory Commission 2

Institutt for Energiteknikk OECD Halden Reactor Project P.O. Box 173 N-1751 Halden NORWAY

'Now with the Idaho National Engineering & Environmental Laboratory 2Now with the Institutt for Energiteknikk, OECD Halden Reactor Project

Office of Nuclear Regulatory Research U.S. Nuclear Regulatory Commission Washington, DC 20555-0001

November 2000

Prepared as part of The Agreement on Research Participation and Technical Exchange under the International Code Application and Maintenance Program (CAMP)

Published by U.S. Nuclear Regulatory Commission

ABSTRACT

This report documents the results of an empirical study of operator and plant performance in simulator based settings. The simulator settings were designed to be representative of conventional and advanced plants. The advanced plant design employed passive systems. The control room architectures were also designed to represent both plant types. Two control room staffing configurations were employed in each plant setting: a staffing configuration reflecting the requirements of 10 CFR 50.54 (m); and a staffing configuration that involved a reduced number of control room operators. A series of five design basis scenarios, relevant for both conventional and advanced passive plants were chosen to evaluate the effects of plant type and crew size on operator performance. The scenarios were: 1) steam generator tube rupture with a stuck open steam generator safety relief valve in the affected steam generator, preceded by a fire in the turbine hall; 2) interfacing systems loss of coolant accident with compounded instrument failures due to the incident; 3) sustained total loss of feedwater; 4) loss of off-site power with a single steam generator safety relief valve stuck open, and; 5) steam generator overfill. Eight crews of operators from the Loviisa nuclear power station in Loviisa, Finland participated in the study: four crews in the conventional plant setting; and four crews in the advanced plant setting. Measures of objective performance were obtained, including ratings of crew performance and transient management. Measures of operator workload, situation awareness, and team interaction were obtained repeatedly during each scenario.. The findings of the study revealed a number of effects of crew size and plant type, and their combination on operator performance. This report documents the study and discusses the implications and issues raised by this performance-based evaluation of control room staffing requirements for advanced passive plants.

iii NUREG/IA-0137

CONTENTS

ABSTRACT ...................................................................................................... iii LIST OF FIGURES ........................................................................................... vii LIST OF TABLES ............................................................................................. viii EXECUTIVE SUM M ARY .................................................................................. ix ACKNOW LEDGEM ENTS .................................................................................... A ACRONYM S ..................................................................................................... xiii

1. INTRODUCTION ................................................................................................................................ 1 1.1 Purpose and Objectives ................................................................................................................... 1 1.2 Current Staffing Requirements for Operating Nuclear Power Units ......................................... 1 1.3 Review of New Reactor Designs ............................................................................................... 1 1.4 Report Structure .............................................................................................................................. 2

2. BACKGROUND .................................................................................................................................. 4 2.1 Review of Advanced Reactor Submittals .................................................................................. 4

2.1.1 Staffing Levels for the M ain Control Room ....................................................................... 4 2.1.2 Required Operator Activities ............................................................................................... 4 2.1.3 Control Room Instrumentation and Control (I&C) Systems and

Human-System Interfaces .................................................................................................. 6 2.1.4 Summary .................................................................................................................................. 7

2.2 Study Issues .................................................................................................................................... 7 2.3 Performance measures .................................................................................................................... 8

2.3.1 Evaluating Crew Size, Plant Performance and Crew Performance ..................................... 8 2.3.1.1 Objective Performance .................................................................................................. 8 2.3.1.2 Rated Crew Perform ance ............................................................................................ 10 2.3.1.3 Situation Awareness .................................................................................................... 10 2.3.1.4 W orkload ........................................................................................................................ 13 2.3.1.5 Team Interaction ............................................................................................................. 14

3. ME THOD ........................................................................................................................................... 17 3.1 Experimental Design ..................................................................................................................... 17

3.1.1 Participants ............................................................................................................................ 19 3.1.1.1 Training ........................................................................................................................... 20

3.1.2 Scenarios ................................................................................................................................ 20 3.1.3 Simulator Performance Characteristics .............................................................................. 21

4. ANALYSIS ......................................................................................................................................... 23 5. RESULTS ........................................................................................................................................... 24

5.1 Transient M anagement and Task Performance ......................................................................... 24 5.1.1 Crew Size and Plant Type Effects .................................................................................... 24

5.2 Rated Crew Perform ance .............................................................................................................. 26 5.2.1 Inter-Rater Reliability ............................................................................................................ 26 5.2.2 Crew Size and Plant Type Effects ..................................................................................... 26

5.3 Situation Awareness, Subjective Workload, and Team Interaction ......................................... 28 5.4 Situation Awareness ...................................................................................................................... 29

5.4.1 Scenario Type and Period Effects .................................................................................... 30 5.4.2 Scenario Effects ..................................................................................................................... 31 5A.3 Crew Size, Plant Type, and Position Effects ..................................................................... 32 5.4.4 Crew Size by Scenario Period Effects ............................................................................... 33 5.4.5 Position Effects ...................................................................................................................... 33 5.4.6 Crew Size, Plant Type and Scenario Type Effects ............................................................ 34

5.5 W orkload ...................................................................................................................................... 36 5.5.1 Scenario and Scenario Period Effects ................................................................................ 36 5.5.2 Crew Size, Plant Type, and Position Effects ..................................................................... 38 5.5.3 Crew Size and Plant Type Effects .................................................................................... 38 5.5.4 Plant Type and Position Effects ......................................................................................... 40

V NUTREG/IA-0137

5.5.5 Crew Size and Plant Type Effects .................................................................................... 41 5.6 Team Interaction ........................................................................................................................... 43

5.6.1 Inter-Rater Reliability ............................................................................................................ 43 5.6.2 Scenario and Scenario Period Effects ............................................................................... 44 5.6.3 Crew Size and Plant Type Effects .................................................................................... 45

6. DISCUSSION ..................................................................................................................................... 49 6.1 General Trends .............................................................................................................................. 49 6.2 Effects of Crew Size ..................................................................................................................... 50 6.3 Effects of Plant Type .................................................................................................................... 50 6.4 Interactions of Crew Size and Plant Type and Effects on Operator Performance ..................... 52

7. INSIGHTS .......................................................................................................................................... 55 8. REFERENCES ................................................................................................................................... 57 APPENDIX A: SCENARIO DESCRIPTIONS ............................................................................. A-1

A. 1 Scenario 1: Steam Generator Tube Rupture With A Fire In The Turbine Hall And A Stuck Open Steam Generator Safety Relief Valve ................................................. A-2

A.I.1 Synopsis ............................................................................................................................. A-2 A. 1.2 Probable course of events ................................................................................................... A-3

A.2 Scenario 2: Interfacing Systems Loss Of Coolant Accident: Primary Leakage Outside The Containment With Ensuing Control Room Instrument Malfunctions ............................................... A-7

A.2.1 Synopsis ............................................................................................................................. A-7 A.2.2 Probable course of events .................................................................................................. A-8

A.3 Scenario 3: Total Loss Of Feedwater ................................................................................ A-11 A.3.1 Synopsis ........................................................................................................................... A-11 A.3.2 Probable course of events .......................................................................................... A- 12

A.4 Scenario 4: Loss Of Off-site Power With A Stuck Open Steam Generator Safety Relief Valve ................................................................................................................. A-15

A.4.1 Synopsis ........................................................................................................................... A-15 A.4.2 Probable course of events .......................................................................................... A-16

A.5 Scenario 5. Steam Generator Over-fill ............................................................................. A-18 A.5.1 Synopsis .................................... : ...................................................................................... A-18 A.5.2 Probable course of events ................................................................................................ A-19

APPENDIX B: INSTRUCTIONS TO PARTICIPANTS ................................................................ B-1 B.1 General Instructions to Participants .......................................................................................... B-2 B.2 Instructions for Situation Awareness Questionnaire .............................................................. B4 B.3 Participant Instructions: W orkload Rating Scales ................................................................ B-5 B.4 Instructions for the Behaviorally Anchored Rating Scale ......................................................... B-6

APPENDIX C: INVENTORIES ..................................................................................................... C-1 C.A Situation Awareness Control Room Inventory Example Inventory ........................................... C-2 C.2 NASA TLX Rating Scale Definitions ................................................................................. C-4 C.3 Behaviorally Anchored Rating Scales Example Inventory ........................................................ C-6 C.4 Rated Crew Performance Example Inventory ......................................................................... C- 12

APPENDIX D: SITUATION AWARENESS CONTROL ROOM INVENTORY (SACRI) ...... D-1 D. 1 SACRI: The Inventory of Questions ........................................................................................ D-2 D.2 W ording of the SACRI Questions ............................................................................................. D-3 D.3 Answer Categories .................................................................................................................... D-3 D.4 Signal Detection Theory Calculations .................................................................................. D-4 D.5 Scoring the Situation Awareness Inventories ...................................................................... D-5

APPENDIX E: TRANSIENT MANAGEMENT PERFORMANCE RATING FORMS .................... E-1 APPENDIX F: DESCRIPTIVE STATISTICS ................................................................................ F-i

vi NUREG/IA-0137

LIST OF FIGURES No. Title Page

Figure 3.1 Side view of the Loviisa nuclear power station main control room ................................. 18 Figure 3.2 Side view of the HAMMLAB experimental control room .............................................. 19 Figure 5.1 Plot of transient management performance measures by crew size ................................ 25 Figure 5.2 Plot of transient management performance measures by plant type ................................ 25 Figure 5.3 Plot of rated crew performance by crew size and plant type ........................................... 27 Figure 5.4 Plot of average situation awareness by scenario period ................................................... 30 Figure 5.5 Plot of situation awareness by scenario type and period ................................................... 31 Figure 5.6 Plot of situation awareness by crew size, plant type, and scenario period ........................ 32 Figure 5.7 Plot of change in situation awareness from baseline ....................................................... 33 Figure 5.8 Plot of situation awareness by crew size and plant type .................................................. 34 Figure 5.9 Plot of situation awareness by crew size, plant type, and scenario type ......................... 35 Figure 5.10 Plot of average subjective workload across scenario periods ....................................... 37 Figure 5.11 Plot of average subjective workload by crew size and position ..................................... 38 Figure 5.12 Plot of subjective workload by plant type and position ................................................ 39 Figure 5.13 Plot of subjective workload by control room crew member ......................................... 39 Figure 5.14 Plot of operator workload by plant type, crew member, and scenario period ................ 40 Figure 5.15 Plot of operator workload by crew size, plant type and scenario period ....................... 41 Figure 5.16 Plot of operator workload change from baseline ........................................................... 42 Figure 5.17 Plot of average team interaction across scenario periods ............................................. 44 Figure 5.18 Plot of team interaction by crew size and plant type .................................................... 45 Figure 5.19 Plot of team interaction by plant type and scenario period ............................................ 46 Figure 5.20 Plot of team interaction by crew size, plant type, and scenario period .......................... 47 Figure 5.21 Plot of team interaction change from baseline .............................................................. 48

vii NUREG/IA-0137

LIST OF TABLES

No. Title Paeg

Table 2.1 Summary of advanced control room characteristics taken from vendor descriptions ...... 6 Table 3.1 Experimental study design ............................................................................................... 17 Table 5.1 Effects of scenario, crew size, and plant type on transient management and task

performance ............................ ........................................................................................... 24 Table 5.2 Effects of scenario, crew size, and plant type on rated crew performance ........................ 26 Table 5.3 Results of repeated measures MANOVA on situation awareness, workload,

and team interaction performance measures .................................................................... 28 Table 5.4 Effects of scenario type, crew size, plant type, position, and scenario period on

situation aw areness ............................................................................................................... 29 Table 5.5 Effects of scenario, crew size, plant type, position, and scenario period on operator

w orkload ................................................................................................................................ 36 Table 5.6 Effects of scenario, crew size, plant type, and scenario period on team interaction ...... 43 Table E- 1 Ratings of transient management tasks on the Steam Generator Tube

R upture scenario .................................................................................................................. E-2 Table E-2 Ratings of transient management tasks on the Interfacing Systems Loss of

Coolant scenario .................................................................................................................. E-3 Table E-3 Ratings of transient management tasks on the Loss of Feedwater scenario ........................ E-4 Table E-4 Ratings of transient management tasks on the Loss of Off-Site Power scenario ................. E-5 Table E-5 Ratings of transient management tasks on the Steam Generator overfill scenario ............ E-6 Table F-I Summary table of situation awareness data: scenario type by scenario period ................... F-2 Table F-2 Summary table of situation awareness data: crew size by scenario period ......................... F-2 Table F-3 Summary table of situation awareness data: plant type by scenario period ........................ F-2 Table F-4 Summary table of situation awareness data: crew size and plant type by

scenario period ..................................................................................................................... F-3 Table F-5 Summary table of situation awareness data: position and plant type by

scenario period ..................................................................................................................... F-3 Table F-6 Summary table of situation awareness data: crew size, plant type, and

scenario type by scenario period .......................................................................................... F-4 Table F-7 Summary table of subjective workload data: scenario by scenario period .......................... F-4 Table F-8 Summary Table of Subjective Workload data: Crew Size by Scenario Period .................. F-4 Table F-9 Summary table of subjective workload data: plant type by scenario period ................... F-5 Table F-10 Summary table of subjective workload data: position by scenario period .................... F-5 Table F-11 Summary table of subjective workload data: position by plant type by

scenario period ................................................................................................................... F-5 Table F-12 Summary table of workload data: crew size by plant type by scenario period ................. F-6 Table F-13 Summary table of subjective workload data: crew size by position by

scenario period ................................................................................................................... F-6 Table F-14 Summary table of team interaction data: scenario by scenario period .............................. F-6 Table F-15 Summary table of team interaction data: crew size by scenario period ............................ F-7 Table F- 16 Summary table of team interaction data: plant type by scenario period ........................... F-7 Table F-17 Summary table of team interaction data: crew size and plant type by

scenario period ................................................................................................................... F-7 Table F-18 Summary table of rated crew performance data by scenario ............................................. F-8 Table F-19 Summary table of rated crew performance data: crew size by plant type ......................... F-8 Table F-20 Summary table of objective performance rating data by scenario ..................................... F-9 Table F-21 Summary table of objective performance rating data: crew size and plant type

by scenario ......................................................................................................................... F-9

viii NUREG/IA-0137

EXECUTIVE SUMMARY

An empirical study was conducted to evaluate issues affecting staffing requirements in advanced nuclear power plants. New plant designs rely on passive systems to achieve safety objectives. With improvements in ease of plant operation, some vendors expect staffing requirements for advanced plants to differ from current regulations. Since the advanced passive plants under consideration are not yet built, no opportunity exists for observation of plant operation and the demands placed on the operating staff.

The study was conducted to provide data to support design review guidance. Two factors were evaluated across a range of plant operating conditions: 1) control room crew staffing sizes; and, 2) the effects of characteristics of the operating facility itself, whether employing conventional or advanced features.

Design factors influenced the performance of crews in this study. Crews in the conventional plant normal staffing complement exhibited better performance than minimum sized crews. Such plants have been designed with a larger control room staffing complement in mind. The functions, allocations, size, and automation in such control rooms require more crew members to maintain control of the plant. In the advanced plant, the minimum sized crews performed better than the normal sized crews. The advanced plant control room setting was designed for a smaller control room complement, being more compact and taking advantage of computer-based technologies more than the conventional plant. The compact control room was designed to support a more limited crew size than the conventional plant, being smaller, having greater automation, passive design features, and displays that better integrate control room information. Therefore, decisions about control room staffing should be based upon design features including: function allocation, automation, integration, and plant-specific characteristics (e.g., passive system performance). Validation and verification using measures of operator and crew performance are necessary to determine the staffing complement needed to operate the plant.

This study was carried out in two phases by collecting data on control room crew performance. The first phase was conducted at the Loviisa nuclear power station, in Loviisa, Finland. The second phase was carried out at the Halden Human-Machine Laboratory (HAMMLAB) at the OECD Halden Reactor Project in Halden, Norway. The Loviisa plant served as the conventional plant for this study; HAMMLAB served as the advanced plant. Both facilities are driven by a simulated plant model of the Loviisa nuclear power plant. Two crew configurations participated in each of the plant types: a crew consisting of four members, and a crew of 2 in the advanced plant; 3 in the conventional plant.

Eight crews of operators from the Loviisa nuclear power station participated in the study. All crews included a reactor operator (RO) and a balance of plant operator (BOP operator). In the 2-man crew, one operator performed supervisor functions. The 3-man crew included a dedicated Control Room Supervisor (CRS). The 4-man crew had a CRS as well as a control room technician (CT) who is a licensed control room operator.

Each crew participated in five scenarios: a Steam Generator Tube Rupture (SGTR), a sustained Total Loss of Feedwater (LOFW), a Loss of Off-site Power (LOOP), an Interfacing Systems Loss of Coolant Accident (ISLOCA), and a Steam Generator Overfill (SGOF). The scenarios were presented in random order. All scenarios contained several faults and events that occurred regardless of operator actions, but scenario outcome varied depending on the crew's responses. The scenarios required the crew to coordinate with external personnel, and required an hour to two-and-a-half hours to complete. All scenarios were divided into segments separated by data collection interrupts during which the simulator was frozen. During this pause, questionnaire data were collected from the crew and raters.

Scenarios required different problem solving techniques to mitigate disturbances. Three scenarios, the SGTR, LOFW, and LOOP were straight-forward to diagnose, each having a number of salient symptoms, and could be mitigated by following procedures. These are referred to as rule-based scenarios. The other two scenarios, the ISLOCA and the SGOF contained misleading or hidden symptoms, were difficult to

ix NUREGAIA-0137

diagnose, or included features that complicated operators' access to procedures (e.g., temporal demand for operator response, etc.). These are referred to as knowledge-based scenarios.

A number of metrics were used to assess operator and plant performance. Objective performance measures included task completion, critical task initiation time, and control of plant parameters. Other measures included situation awareness, operator workload, team interaction, and rated crew performance. Data were collected across time periods in the scenarios.

All crews received simulator-specific training on Instrumentation and Control (I&C) systems and plant performance characteristics. They also were informed about the overall purpose of the study, and were instructed on completing the inventories used in the study. The raters who evaluated team interaction and crew performance received inventory-specific training.

The minimum sized crews in the advanced plant demonstrated the best rated crew performance. In terms of objective performance, advanced plant crews performed better than conventional plant crews. While crews in the advanced plant experienced significantly higher workload than crews in the conventional plant, their performance was unimpaired.

Normal sized crews performed better than minimum size crews on cooldown and stabilization tasks. This difference may be explained by workload and task shedding. The normal sized crews experienced lower workload than the minimum sized crews, and thus were able to complete more tasks. The minimum sized crews shed tasks that were not perceived as critical. However, both crew sizes performed similarly on critical task completion.

Crews in the advanced plant setting achieved higher situation awareness than crews in the conventional plant. Minimum sized crews performed better than normal sized crews in the advanced plant; conversely, normal sized crews performed better than minimum sized crews in the conventional plant. This interaction may be explained by design aspects of the control rooms that better support different crew staffing complements.

Minimum sized crews experienced more workload than normal sized crews, with the control room supervisor experiencing most of the additional workload. Crew members are resources available to the control room supervisor, and their absence increases tasks for the supervisor. Advanced plant crews experienced significantly higher workload than their conventional plant counterparts. During debriefings, operators indicated that using the automated systems in HAMMLAB, different from their normal work environment, created additional workload. In the advanced plant setting, the control room supervisor also served as one of the operators, and this dual role CRS experienced much higher workload than all other operators. During the debriefings, the dual role CRSs expressed concerns about working effectively in such a stressful capacity.

Crews in the advanced plant setting exhibited better and more stable team interaction than crews in the conventional plant. This improvement possibly relates to design differences between the conventional and advanced plant settings used in this study. Many conventional plants have relatively large control rooms in which operators may be distant from one another. The advanced plant configuration provided close operator workstations centered around a common overview display, which served as a focus of operator discussions.

x

NUREG/IA-0137

ACKNOWLEDGEMENTS

The authors appreciate the dedication and effort extended to this study by a number of key persons; firstly, the staff of Imatran Voima OY's Loviisa nuclear power generating station. We thank all of the operators who volunteered to participate in this study; whose hard work under challenging circumstances made this work possible. We also thank Mr. Pekka Sammatti and Dr. Jussi K. Vaurio for their assistance in arranging for this study with the Operations group at the plant, and for their assistance throughout the entire study. Pekka Sammatti and Ray Saarni from the Loviisa Plant and Pekka Pyy from the Technical Research Center in Finland, VTT, provided invaluable assistance throughout this study, beginning with the development of the study scenarios, to issues of experimental design and realism, through data collection and analysis. They provided a much needed and valued liaison with the operators in this study on behalf of the researchers and assisted in the development of analysis techniques. The simulator staff at both facilities, the Loviisa training simulator and the Halden Human-Machine Laboratory, worked diligently to make the necessary modifications to the simulator models and I&C systems, and conducted operations expertly during data collection under very dynamic conditions. We noted many times that the simulator operators' workload appeared nearly to match that of the operators in the simulator during periods of high interaction and role play with the operators as they worked to maintain the realism of the scenarios. Finally, we acknowledge the assistance and support from Dr. Persensky of the U.S. Nuclear Regulatory Commission. We asked for and received a great deal of input on the aims and objectives, as well as background and design aspects of this study that greatly facilitated the work presented in this report.

xi NUREG/IA-0137

ACRONYMS

ABB AECL ANOVA AP600 BARS BOP BWR CANDU CFR CRS CRT CT EOP ERG FR GE HAMMLAB HILOAS I&C ISLOCA LOCA LOOP LOFW LOTS LWR MANOVA NASA NPP NRC NSSS OECD PIUS PWR RCS RO S/G SA SACRI SAR SBWR SGOF SGTR SME SRO SSAR STA TLX VDU Vrr

XiiiNUREG/IA-0137

Asea, Brown, Boveri Atomic Energy of Canada Limited Analysis of Variance Advanced Plant 600 Behaviorally-Anchored Rating Scale Balance of Plant Boiling Water Reactor Canadian Deuterium Uranium Code of Federal Regulations Control Room Supervisor Cathode Ray Tube Control Room Technician Emergency Operating Procedure Emergency Response Guidelines Federal Register General Electric Halden Human Machine Interaction Laboratory High Level Operator Action Strategies Instrumentation and Control Interfacing Systems Loss of Coolant Accident Loss of Coolant Accident Loss of Off-site Power Loss of Feedwater Loviisa Training Simulator Light Water Reactor Multivariate Analysis of Variance National Aeronautics and Space Administration Nuclear Power Plant Nuclear Regulatory Commission Nuclear Steam Supply System Organization for Economic Cooperation and Development Process Inherent Ultimately Safe Plant Pressurized Water Reactor Reactor Coolant System Reactor Operator Steam Generator Situation Awareness Situation Awareness Control Room Inventory Safety Analysis Report Simplified Boiling Water Reactor Steam Generator Overfill Steam Generator Tube Rupture Subject Matter Expert Senior Reactor Operator Standard Safety Analysis Report Shift Technical Advisor Task Loading Index Video Display Unit Technical Research Center of Finland

INTRODUCTION

1. INTRODUCTION

1.1 Purpose and Objectives

The purpose of this study was to evaluate the impact(s) of advanced passive plant design and staffing of control room crews on operator and team performance. It was conducted to contribute to the understanding of potential safety issues and provide data to support the review of proposed staffing levels for advanced passive reactors. Two factors were evaluated across a range of plant operating conditions: 1) control room crew staffing complement; and, 2) the effect of characteristics of the operating facility itself, whether employing conventional or advanced, passive features on operator performance.

This study was carried out by collecting data on control room crew performance. The first phase of this study was conducted at the Loviisa nuclear power station, in Loviisa, Finland. The second phase was conducted at the Halden Human-Machine Laboratory (HAMMLAB) at the OECD Halden Reactor Project in Halden, Norway. The Loviisa plant served as the conventional plant for this study; HAMMLAB served as the advanced plant. Both facilities are driven by a simulated plant model of the Loviisa nuclear power plant. The control room systems at Loviisa used in this study are representative of those in conventional control rooms, as are the time constants of the plant's thermal-hydraulic performance. The control room systems used in HAMMLAB consist of video display unit-based information and control systems, similar in form and function to those of advanced plant control rooms. In addition, some of the time constants of the thermal-hydraulic model in HAMMLAB were longer than at Loviisa to emulate the longer thermalhydraulic time constants in advanced plants. These longer time constants are due to the performance of passive systems or other advanced design features.

1.2 Current Staffing Requirements for Operating Nuclear Power Units.

The operation and management of a nuclear power plant require a sustained, coordinated effort to achieve both power production goals and compliance with established safety requirements. The focus of these efforts is usually the main control room, where operations are conducted by licensed personnel. The control room staff has the responsibility for oversight and decisions about the operation of the plant. Because of the dependence on these personnel, qualification requirements and minimum licensed operator staffing of nuclear power plants are specified by federal regulation, found in lOCFR 50.54(m).

1.3 Review of New Reactor Designs

In addition to ensuring compliance of existing nuclear units with regulatory requirements, the U.S. Nuclear Regulatory Commission (NRC) also conducts Design Certification reviews of proposed reactor designs to ensure the compliance of a vendor's proposed design with regulatory requirements. New plant designs have been produced by a number of vendors. Some of these advanced plant designs rely on new design or application approaches using passive systems to achieve increased redundancy and/or diversity of methods to achieve safety objectives. Although they use proven technologies to a large extent, these advanced passive plant designs also differ in many respects from conventional plant design.

The Advanced Reactor Policy Statement (51 FR 24643) and NUREG-1226, "Development and Utilization of the NRC Policy Statement on the Regulation of Advanced Nuclear Power Plants," define advanced reactors as those with innovative designs for which licensing requirements will differ significantly from existing light-water reactor (LWR) requirements. These documents also provide guidance for the

NUREG/IA-0137

INTRODUCTION

development of new regulatory requirements to support advanced designs. To the extent practical, staff reviews are based on existing regulations. When new requirements are necessary, there is a preference for performance-based rather than prescriptive regulations.

With improvements in ease of plant operation gained through passive design and digital instrumentation and control systems, vendors expect differences in operator performance needed to achieve plant safety objectives for credible events. These expected differences have led to a need to reconsider the requirements for minimum shift staffing of licensed Reactor Operators and Senior Reactor Operators contained in 10 CFR 50.54(m).

Since the advanced passive plants under consideration are not yet built, no opportunity exists for observation of plant operation and the demands placed on the operating staff. Operating experience with these plants and systems is limited to similar systems in other plants (e.g., BWR isolation condenser designs, PWR N2 injection accumulators) and designs incorporating similar features (i.e., introduction of

digital control systems into existing analog-based control rooms). Although the basic principles of operating these new plants are similar to existing plants, the range of operating conditions and response requirements may be quite different. Hence, the demands placed on the operating crew itself may be different.

1.4 Report Structure

The following sections present the background, method, analysis approach, results, conclusions, and a summary of this study. The Background section provides a summary of the review of advanced reactor submittals that was conducted for this study, and the key information that was obtained from this review. The Background section also presents the study issues that were identified for this research project, and a discussion of the performance measures that were used to assess the study issues.

The Method section includes a discussion of nuclear power plant staffing practices. Current staffing requirements and proposed advanced design staffing levels are compared. Issues such as crew size and team performance are addressed. Issues expected to influence staffing requirements in advanced plants, automation, passive plant features, and crew size, are also evaluated. The study design and dependent measures, the scenarios in which these measures were assessed, and how the data collection was performed are discussed.

The Analysis section provides a summary of the approach and rationale used to conduct the statistical analyses of data that were gathered. This section includes a discussion of how the level of statistical significance was adopted for this study, and the general approach to the evaluation of study issues using tests of statistical significance.

The Results section presents the results of the analyses of performance measures used to evaluate the effects of staffing and plant type on operator performance. Significant interactions are also discussed. The results of analyses are presented in tables that show the particular effects tested and the resulting test results. This section also contains a number of graphs of main results, to assist in the interpretation of study results.

The Discussion addresses the results of statistical analyses of the effects of crew size and plant technology on operator performance. This section contains separate discussions about general trends observed in the study, and the effects of crew size, plant type, and their interaction. Implications of these results are outlined in the Insights section, which discusses control room staffing issues for advanced plants that were identified by this study.

References and appendices are included at the end of the report. Appendices contain: ", synopses and detailed descriptions of the scenarios used in this study; "* instructions given to participants; "* copies of inventories administered;

2 NIJREGJEA-0137

INTRODUCTION

"* information regarding the development and scoring of the situation awareness inventory;

"* transient management performance rating forms; and "* descriptive statistics of performance measures collected in the study.

3 NUREG/IA-0137

BACKGROUND

2. BACKGROUND

2.1 Review of Advanced Reactor Submittals

Vendor submittals provide information to be used in the process of design certification and address a wide range of issues. To develop an approach for the study of crew size and crew performance, advanced reactor submittals were reviewed. The reviews were conducted to determine: the types of staffing levels envisioned; activities required of the main control room crew in operating these new plants; and the types of instrumentation and control systems employed in the human-system interface for carrying out information gathering and control activities. The advanced reference plant designs used include the Westinghouse AP600, ABB Atom PIUS Plant, General Electric Simplified Boiling Water Reactor (SBWR), and the Atomic Energy of Canada Limited CANDU 3 plant designs. The following summarizes the main conclusions of the review.

2.1.1 Staffing Levels for the Main Control Room

For most power operations, a reduced-staff control room crew is envisioned to be sufficient by most vendors. This is attributed to improvements gained through automation (e.g., function allocation) of many activities that are manually conducted in today's plants. In both the Westinghouse AP600 and GE SBWR plant designs, staffing considerations, including accident situations, follow the requirements of 10 CFR 50.54 (m) (Westinghouse AP600 SSAR Section 18.7.1 "Basis for Staffing Requirements," and GE SBWR SSAR Section 18.2 "Design Goals and Design Bases"). The AP600 submittal allows for a shift staffing complement that would be larger than that required by 10 CFR 50.54 (in). This includes additional licensed operating personnel to interface with other plant staff and a separate position for the shift technical advisor, resulting in up to six licensed operators (Westinghouse AP600 SSAR Section 18.7.2.1 "Main Control Area").

At present, the staffing requirements for the ABB PIUS plant differ from the current federal regulations. I&C systems in the ABB PIUS plant are designed to require only one operator in the control room for normal operation (Section 501.2.1). Furthermore, the design goal of the plant for operations is not envisioned to exceed two operators in the control room, though one should be fully adequate for controlling the plant (Section 511.2). Staffing issues were not addressed in AECL's CANDU 3 submittal.

Thus, the shift staffing plans vary considerably for different advanced plants. This variation in itself may not be at conflict with safety objectives, depending on the actual demands for human intervention in both preventing, detecting, and responding to accident conditions. Passive system operation and automation may reduce demands on the operating crew for direct intervention and minimize the likelihood for errors. Were this the case, a smaller staffing complement may be capable of operating a plant safely. Thus, staffing plans for advanced plants should also be considered in light of the activities required of the operating staff.

2.1.2 Required Operator Activities

One of the reasons given by vendors for reduced control room staffing is a general reduction in the number of tasks allocated to operating personnel. Increased allocation of functions to plant control systems, and improvements in the ease of performing remaining tasks due to advanced I&C systems reduce the number of staff required. This expectation appears more valid for activities carried out during normal operations than with emergency operations. Two of the vendor submittals contained information about operator activities required for emergencies: the Westinghouse and GE SSARs. To evaluate differences in the anticipated operator actions between advanced and conventional plants, procedures or operator action

4 NUREG/IIA-0137

BACKGROUND

strategies were compared. The High Level Operator Action Strategies (HILOAS) of the Westinghouse AP600 were compared with the EOPs of a conventional 3-loop Westinghouse PWR. A similar comparison had also been made by GE in the Emergency Response Guidelines (ERGs) for the SBWR with Revision 4 ERGs for conventional BWRs. Differences were found in operator activities between conventional and advanced plants.

Most importantly, the EOP structures of the advanced plants are similar to those of operating plants, often having the same EOP titles, general structure, and individual steps. A detailed comparison was made of operator activities contained in these documents. However, the HILOAS are activity descriptions, not actual procedures, and they may change when they are converted into procedures. Hence, this comparison of AP600 HILOAS with EOPs is actually a rough determination of activity differences. Some of the differences observed may thus be attributed to the level of detail provided in the conventional plant's procedures. They are written for an operational facility, in contrast to operator action strategies written at a stated "high level."

Bearing this in mind, a number of differences were observed between the conventional and advanced plant documents used to determine operator activities. The differences were characterized as wording changes, sequence changes, and activity changes. Wording changes do not result in significant changes in the activity of the operating crew, or in the intent of the step (i.e., "Check RCS Pressure" to "Check RCS Pressure within Plant Specific Limits"). A sequence change merely involves changing the order of a set of actions, though the balance of activities and, ostensibly, their objective remains the same. An activity change represents a likely change in the task(s) or intention of the activity. Examples of activity changes include the deletion of steps for establishing feed and bleed or once-through core cooling due to changes in plant operation, or the deletion of a task requiring a manual action to one requiring verification (e.g., "Check if Passive Containment Cooling is Required"). A total of 32 EOPs/ERGs was reviewed. On average, two to three wording changes, one to two sequence changes, and four activity changes were found in each EOP/ERG.

In summary, two of the vendors provided material about planned operator activities that was sufficiently detailed to allow comparison with operator activities in conventional plants. The strategies and safety functions for these advanced LWR plants appear analogous with those of the conventional LWR plants though some differences were found at the activity level. These differences are primarily due to changes in the systems employed in the advanced plants for maintaining critical plant functions, and not due to a change in the operational philosophies or functional response strategies. Unfortunately, the one vendor that expects the biggest differences in the necessary operational staff and systems and strategies for achieving plant safety did not furnish sufficient information to make a similar comparison of operator activities. Hence, the basis for expected differences in operator activities between conventional and advanced plants is limited to the insights gained from a review of material provided by two LWR vendors.

The tasks that operators must perform in the two types of plant may be more different than indicated by the activity descriptions. In conventional plants, analog information and manual control devices dominate the I&C systems in the control room. The use of advanced technology has a potential to change the ways in which operators gather information, the types of information presented, operator response modes (e.g., motor, perceptual, cognitive, etc.) and the types of operator-system interactions that occur when conducting control room activities. Therefore, vendor plans for main control room I&C systems were reviewed.

5 NUREG/IA-0137

BACKGROUND

2.1.3 Control Room Instrumentation and Control (I&C) Systems and Human-System Interfaces

All vendors described the general layout; work areas provided for crew members; information systems; methods of interaction; and, to some extent, automatic modes of operation provided in the main control room of their plant design. The control rooms in each of the advanced plants rely heavily upon advanced I&C technology, compared to conventional plants. Table 2.1 summarizes the main features of the control room as described by the vendors.

Table 2.1 Summary of advanced control room characteristics taken from vendor descriptions

Operator Control * Soft control units. "* Tracker ball, touch screens, keyboards, mouse. "* Function keyboards. "* Dedicated fixed-position or hard-wired function

switches. "* Display selection controls. "* Dedicated safety system controls. "* Functionally-arranged physical control panels for

backup (1 vendor only). "* Procedure displays (1 vendor only).

Information Systems VDUs presenting: a Physical mimic displays * Alarms • SPDS * Overview Displays (e.g., process mimics) with

important alarm information * Parameter trend VDUs.

Configurations a 2 Operator Workstations on a single integrated console or separate RO and BOP workstations.

* 1 supervisor's console, with or without control capabilities, located aft of operator workstations.

* Overview and alarm displays in a fixed wall position.

Alarm Processing 0 Processing techniques to reduce alarms including suppression or filtration via:

"* prioritization; "* mode; "* subsidiary; "* redundancy; "* system.

Operators interact with the information and control systems via tracker ball, mouse, or other dedicated input device (i.e., touch screen). Requests for information such as process formats and the navigation through the information system are largely accomplished by requests through either menus, touch screens, or function keyboards. VDUs provide information to the crew about process parameters, integrated as part of either plant mimic diagrams, detailed process diagrams, trends, or in connection with alarms. The integration of

6 NUREGJIA-0137

BACKGROUND

this information and the specific systems developed for providing this information are vendor-specific and differ somewhat from plant to plant.

Operations in the advanced plants will be carried out primarily using computer-based technologies to obtain information and to issue commands to the equipment and systems in the plant for both normal and emergency operations. Work areas for the operating crew are divided into separate computer-based workstation areas either physically separate from each other or functionally grouped for each operator. In contrast to existing control rooms, the control room supervisor also has a workstation for information gathering. Each operator typically has access to the same information and displays as the other operators. Each operator's work area or workstation typically has between three and seven VDUs for display and control, in addition to the wall panel overview and alarm displays. In most cases, operators also have the ability to obtain additional information about alarms using some of their VDUs. In some cases, specific VDUs are allocated in an operator's workstation or area for this purpose.

2.1.4 Summary

Due to the way advanced passive plant vendors expect these plants to be operated, compared to existing plants, some differences were noted in the amount of manual control required to safely operate these plants. In some cases, a change in the size of the minimum control room staff is suggested by a vendor. Some vendors, though, expect no change to the minimum control room staff compared to the current federal regulation. Descriptions of emergency operations indicate that the types of activities operators will carry out in the advanced plants are similar to activities carried out in existing plants. However, the vendor expecting the largest change in the amount of operator intervention to operate its plant did not provide sufficient information to allow comparison with either existing plant or other advanced plant activity descriptions. The ways in which operators will perform activities in the advanced plants will change significantly from existing plants due to the use of advanced computer-based systems for information display and control functions.

2.2 Study Issues

This study was concerned with the effects of advanced plant technologies and reductions in the minimum control room crew staffing complement on operator performance. Performance measures were defined for the study (see Section 2.3) and an experimental design employed that would permit collection of data to evaluate these effects. To evaluate the effects of the independent variables, a number of study issues were identified. These are described below.

" Do changes in the staffing complement of the main control room produce measurable differences in the performance of operators?

"* Compared to the reference (i.e., normal) crew size, do smaller sized crews exhibit poorer performance on transient management and mitigation activities?

"* Is the workload of crews in a smaller crew staffing complement different than that of a normal staffing complement? If so, does the workload of smaller sized crews exceed a level beyond which performance on other measures degrades?

"* Do any particular crew members appear to be affected most by a reduction in crew size? "* Are crew members able to maintain an awareness of the plant when one or more members of

the control room crew are removed from the control room complement? "* Are crew performance characteristics, i.e., how the crew works as a team, negatively affected

by a reduction in crew size?

"* Do changes in the type of plant automation and control systems produce measurable differences in the performance of operators?

7 NUREGAIA-0137

BACKGROUND

"* Are operators in the advanced plant able to manage the demands of the transients as well as operators in the conventional plant?

"* Is the workload of crews in the advanced plant different than the workload of crews in the conventional plant? If so, does it exceed or fall below an optimal level such that performance on other measures degrades?

"* Compared to the conventional plant, are operators in the advanced plant able to maintain their awareness of the plant?

"* Are any particular crew members affected most by a change in the task demands of operating the plant using a higher degree of automation?

"* Are crews able to maintain their teamwork skills when using the more highly automated systems?

* Does the combination of changes in control room staffing together with changes in plant automation and control systems produce measurable differences in the performance of operators?

* Does the interaction of crew size and plant type effect the performance of crews on transient management tasks?

0 Does the interaction of crew size and plant type effect the workload of crews? * Does the interaction of crew size and plant type effect the ability of crews to maintain an

awareness of the plant? 0 Does the interaction of crew size and plant type effect the way in which crews work together

as teams?

2.3 Performance measures

2.3.1 Evaluating Crew Size, Plant Performance and Crew Performance

To evaluate the effects of crew size and plant type, performance criteria were defined. Firstly, since one of the objectives of this study was to identify potential safety issues associated with either a reduced staffing complement or features of the advanced plants (e.g., passive plant performance, required operator activities, human-system interfaces) the study employed operating environments representative of both conventional and advanced plants. The issue of safety was to be partly addressed by a comparison of operator performance in the advanced plant setting with that in the conventional plant. A concern might exist, or warrant further consideration, if operator performance on some criteria in the advanced plant degraded below that exhibited by crews in the conventional plant setting, in different control room staffing complements, or by the interaction (e.g., combination) of plant type and crew size.

Secondly, measures of operator performance must be sensitive to changes that can be attributed to the different staffing and operating environments. They must also provide meaningful information about the ability of operators to manage the plant under simulated emergencies. The measurements taken should provide information about the objective performance of crews, and contextual (e.g., subjective) factors that influence their performance. The measurements should further provide information about operator and crew performance so that an assessment can be made about the effects that a reduction in crew size may have on the remaining operators in the team, and whether the effects of such reductions degrade performance.

Five metrics were used to assess crew performance: objective performance indicating the crew's management of transient mitigation activities; rated crew performance; situation awareness; operator workload; and team interaction. These are discussed in more detail below.

2.3.1.1 Objective Performance

8 NUREG/IA-0137

BACKGROUND

Data indicating crews' management of the scenarios were collected. Important tasks were identified prior to the simulator portion of the study via task analyses of each scenario (e.g., manual reactor trip, pressurizer cooldown) and initiation times for these tasks were obtained during the experiment. Plant parameters were logged from the simulator and stored for later analysis. Simulator event logs list key system events (e.g., reactor protection system signals, pressurizer heater bank actuations), automatic and manual system actions, and alarms. In addition, audio-video recordings of each crew's performance were stored on video cassette for later evaluation.

Task initiation times were obtained for important tasks identified prior to the simulator portion of the study from task analyses of each scenario (e.g., manual reactor trip, pressurizer cooldown). Plant parameters were logged from the simulator and stored for later analysis. Simulator event logs were also used to identify key system events (e.g., reactor protection system signals, simulator instructor-initiated faults and malfunctions, etc.), automatic and manual system actions, and alarms (e.g., accumulator level low level or gradient alarm, critical function alarms). In addition, audio-video recordings of each crew's performance were stored for later use.

A licensed shift supervisor (SRO) from the Loviisa plant reviewed the performance of crews using event logs from scenarios, plant parameter data, and audio and video tapes of crew performance during the scenarios. Three groups of tasks were identified, each necessary for successful management of transients: Announcements and Notifications, Critical Task Completion, and Stabilization and Cooldown. In all five scenarios, operators were required to make a standard set of announcements and notifications. Simulator instructors noted announcements and notifications made by the crew. Each scenario also contained separate Critical Tasks that had to be completed. For example, in the SGTR scenario, a critical task was "isolating the faulted steam generator." Similarly, Stabilization and Cooldown tasks had to be accomplished in each scenario. Specific details of how these were best accomplished varied among the scenarios, but the overall objectives were the same.

To perform the analyses, a list of sub-tasks was identified for each task group. For Announcements and Notifications, for example, a list of the required calls was developed based on operating procedures (e.g., emergency organization activation,), technical specifications (e.g., requirements for notifications to the safety authority,), and scenario-specific requirements (e.g., mobilizing the fire department, etc.). Each crew's performance of announcements and notifications was compared to this list. If a crew made a required call, they received a check for making that call. For a crew to receive credit for making a call, some announcements and notifications had to be made in a timely manner (e.g. fire notifications). Only calls made at an appropriate time were credited. In analyzing the performance of all the crews, the checks each crew received were tallied and the crew received a score.

Analysis of Critical Task Completion was conducted differently. Tasks necessary to mitigate the effects of the initiating events for each scenario were identified, resulting in a number of tasks in each scenario. Some tasks only needed to be completed (e.g., stop a pump); some needed to be completed within a certain time (e.g., close a valve before an overfill occurs); and some required a plant parameter to be controlled (e.g., safe filling of steam generators after dryout). Tasks that only needed to be completed were rated with a check if they were completed. Tasks that had to be performed within a certain period of time were rated based on occurrence and timeliness. Tasks that required complex control actions were rated more qualitatively, based on a process expert's evaluation of plant parameters. The process expert evaluated the parameters by ensuring that the crew controlled the plant by maintaining the parameters (e.g., margin of departure from nucleate boiling) in a safe range.

Each scenario placed a different set of challenges on the plant and on the control room crew's ability to mitigate these events. Some scenarios resulted in overheating of the primary circuit, and eventually inadequate core cooling (i.e., the total loss of feedwater scenario). Some resulted in overcooling of the reactor coolant system (RCS) (e.g., the steam generator overfill and loss of off-site power with Steam Generator (S/G) safety valve stuck open). Other scenarios resulted in a loss of RCS inventory, unfiltered radioactive release to the environment, and a threat to the availability and operability of equipment in both units due to a common cause external event (e.g., steam generator tube rupture with a stuck open S/G safety

9 NUREGIIA-0137

BACKGROUND

valve and a fire in the turbine hall). Thus, the necessary mitigation actions differed in each of the scenarios. For each scenario, the necessary actions were identified to mitigate the event and prevent additional challenges to equipment or failures, and crew performance was assessed in terms of accomplishing these actions.

Stabilization and Cooldown tasks were evaluated similarly to Critical Task Completion. All of the scenarios were of sufficient length that crews should have begun taking the plant from post-disturbance conditions to a hot standby or shutdown state. In the transition from important transient-specific event mitigation tasks to longer term stabilization activities, crews must be cognizant of many plant conditions. Cooldown rates, boiling margins, inventory levels in safety and auxiliary systems, and pressure differentials must be maintained within target ranges in order to prevent unintended challenges to equipment and systems. Plant parameters and event logs were studied to determine when stabilization activities were begun, and how well these tasks were accomplished. The specific tasks that had to be completed for each scenario were identified. Crews were evaluated (objectively) in terms of whether or not they completed the tasks and a weighting scale was applied (subjectively) in terms of the quality with which they controlled various critical plant parameters

Appendix E contains the transient management rating forms for each of the five scenarios. All crews' ratings are presented, as are the criteria for these ratings.

2.3.1.2 Rated Crew Performance

A crew performance rating technique was also used to evaluate how well the crew performed on a variety of dimensions. The dimensions on which crews were rated were deemed necessary to achieve and maintain control of plant performance. This rating technique was taken from a previous study that focused on the performance of nuclear power plant control room crews (Hanson et al., 1987). Process experts rated the crew's performance on four dimensions: solution path, control of plant, communication, and confidence. These dimensions are intuitively linked to the types of behavior required to achieve and maintain control of plant performance. Solution path refers to the crew's use of time in recognizing the event and selection of the correct mitigation procedure(s). Control of plant refers to the crew's demonstrated understanding of procedures in their analysis of the transient and the extent to which they challenged safety equipment. Communication refers to the extent to which the information exchange among the crew members facilitated transient mitigation. Confidence refers to the ease with which the crew completed transient mitigation without hesitation, and self-statements about the sureness of their own actions and decisions. Licensed operator examiner personnel created these dimensions through previous evaluations of crew performance.

This technique uses the observations of knowledgeable or trained observers to provide an evaluation of how well the crew performs on these dimensions. Immediately after a crew completed a scenario, three process experts rated the crew's performance. The process experts who provided ratings in this study include two licensed SRO's, and one engineer knowledgeable on plant operation and plant safety. Each dimension was rated on a scale of 1-10, with 1 being the worst performance, and 10 being optimal performance. An example of the rating form and a description of the behavioral categories is found in Appendix C.

2.3.1.3 Situation Awareness

Situation Awareness (SA) is a measure of the operator's understanding of the current process state. Current understanding of operator performance indicates that developing and maintaining SA is an integral and important aspect of operator activities (Adams, Tenney, & Pew, 1995). During routine monitoring, operators periodically gather information to assess the plant state or operations being conducted. Using their knowledge of the plant (i.e., their mental model of the process, systems and how they interact), operators integrate the information they gather to develop and maintain situation awareness. This part of

10 NUREG/IA-0137

BACKGROUND

the feedback loop is necessary to understand the state of the process at a given point in time, to determine the need for and type of control activities, and to predict the process state in the future. SA is necessary, but not sufficient, for optimal performance (Endsley, 1990). Although intuitively connected, understanding the process state (i.e., possessing good situation awareness) and taking the appropriate control actions are not inexorably linked. Decision-making in many instances may also require balancing economic goals with prescriptive requirements for operation (i.e., procedures, technical specifications, etc.). Situation awareness is required as an input to decision making, but must also be coupled with other types of knowledge to achieve safe and effective performance.

One may argue the need for situation awareness when the operator need only follow procedures. This may be especially so for emergency situations where extensive efforts have been directed to develop and ensure the adequacy of emergency operating procedures (EOPs) for all foreseeable circumstances (U.S. NRC 1980, 1981, 1982a, 1982b). Recent studies have shown, however, that active situation assessment is often a part of using EOPs . It enables operators to detect faults ahead of procedurally-driven checks, detect important information, and identify situations or problems that are not addressed by procedures (Roth, Mumaw, & Lewis, 1994). Although procedures are in principle applicable to a wide range of foreseeable events, the execution of procedures, especially during infrequent or rare circumstances (i.e., accidents) may involve significant planning and situation assessment to achieve procedural goals.

SA assessment methodologies have been applied in the aviation industry and serve as basic evaluation and major design criteria for the design of new pilot support systems (Selcon and Taylor, 1989). SA is measured to assess whether or not pilot support systems do, in fact, provide the pilot with an enhanced understanding of the state of the aircraft. SA is also relevant to the nuclear process control industry because of the need to evaluate new complex system designs in terms of how well they support operator SA (AIAA, 1992).

Changes in control room automation, evidenced by increased function allocation to machine systems or changes in the human-system interfaces used for information gathering and control have the potential to affect operator situation awareness. Some of these effects may be desirable. The way information is presented to operators may influence SA by increasing the amount of information operators can acquire visually. It may also minimize the potential for misunderstanding through improved integration of process measurements, and by presenting information in a manner more consistent with the operator's understanding of the plant and processes.

Improved I&C systems in advanced plants may also support SA by integrating information from various sources or by minimizing the amount of information an operator must attend to in processing disturbance information (e.g., alarm filtering features). Improved information presentation and less workload (e.g., via function allocation), especially in challenging situations, should result in greater reserve performance capacity of the operating crew. This reserve capacity should afford operators both the time needed and information necessary to build and maintain situation awareness. Hence, one of the advantages of advanced control room design is expected to be enhanced operator SA (O'Hara, 1990). However, no studies have assessed this expectation.

Increased automation may also create conflicts in maintaining SA. Greater function allocation to automated processes may place an additional mental burden on the operating crew to understand the functioning of automated systems. That is, their mental models also need to account for the activities of automation in order to correctly integrate information about the state of the plant and processes (Sarter & Woods, 1995). In other industries, a number of incidents have occurred in which misunderstanding or miscomprehension of automated systems have contributed directly to catastrophic outcomes (Rodgers, Mogford, and Mogford, 1995; Sarter &Woods, 1995; Strauch, 1995).

Increased automation may result in the operating crew being out of the control loop. As a result of design decisions to support operators by reducing workload, the operator may become side-lined. Operator performance has been shown to degrade in more highly-automated environments due to a diminished ability to detect system-level or automation errors and intervene to carry out the correct manual task (Carmody &

11 NIJREG/IA-0137

BACKGROUND

Gluckman, 1993; Wiener & Curry, 1980). This performance degradation may be due to a number of factors, including loss of operator vigilance, a loss of skills in performing tasks over time, or the failure of automated systems to provide relevant information to operators concerning automation failures (e.g., selfchecking or diagnostic information).

Thus, some factors influencing SA are likely to differ between conventional and advanced plants. Situation awareness may also be affected by changes in the staffing of control rooms, independent of the technologies employed for information presentation and control. Operators in a nuclear power plant control room work as a team. To some extent, crew members also work independently of each other, owing to a division of labor among crew members (e.g., reactor operator and balance of plant operator positions). As Endsley points out (Endsley, 1995) communication and sharing of SA-relevant information by individual members is crucial for the team to achieve SA, as well as for individual members to acquire responsibility-specific SA. Team interaction processes also serve as a vehicle by which the situation awareness of individual members is shaped (Salas et aL, 1995).

Given the relationship between individual members, the team, and situation awareness, changes in the way crews are staffed may also produce changes in operator situation awareness. Reducing the number of crew members from a nominal staffing level, and the accompanying increased workload demands placed on the remaining crew members may affect operator SA. Additionally, team processes and workload factors may be affected by changes in control room staffing, and may potentially affect SA.

As Endsley (1993) demonstrated in a series of studies, workload influences SA, though SA and workload differ in a number of ways. With low workload, operators may have low SA due to boredom, lack of attention, and vigilance problems. However, if an operator with low workload attends to information in the environment and if that information is presented in an easily-understood manner, the operator may have high SA. In high workload situations, the operator may have such low SA due to being so overburdened with tasks, the pace of incoming information, etc., that s/he is unable to formulate an accurate assessment of what is occurring in the plant. Differences in workload experienced by operators in the different staffing complements used in this study may also influence the situation awareness between the normal and minimum shift staffing complements.

The Situation Awareness Control Room Inventory, SACRI, a technique developed by Hogg et aL, 1994, was used to assess situation awareness. The questions in SACRI were specifically chosen to be relevant to nuclear power plant control rooms, and ask about the trends of different process parameters. Each administration contained 18 questions, and are divided six ways. Questions are asked about the past, present and future state of the plant and are divided between primary (i.e., reactor coolant system) and secondary (i.e., balance of plant) side parameters. The questions are evenly distributed so three questions are asked in each group.

The questions used in SACRI are in multiple-choice format. Depending on the type of question, either three or six possible answers are presented. If the SA question asks about a single parameter, such as the pressure in the pressurizer, three possible answers are presented: increase, decrease, or same. If the SA question asks about multiple parameters, such as the levels in the steam generators, six possible answers are presented: increase in one, increase in more than one, same, decrease in one, decrease in more than one, drift in both directions. The participants circle the answer they feel best characterizes the plant status for that parameter or group of parameters.

To grade the inventory, the operators' answers are compared with the applicable plant parameters during a given time interval. Simulator data were collected in event and variable logs. Event logs recorded when operator and simulator instructor actions occurred (such as freezing the simulator for a data-collection period). Variable logs recorded process parameters every 15 seconds during the scenario.

Using simulator data from the variable logs, a scoring copy of the SA test was generated. This provided a list of the correct answers for the inventory. SA inventories collected from operators during the scenarios were scored by comparing their responses to the correct answers. All answers were characterized as one of

12 NIJREG/IA-0137

BACKGROUND

four types: hit, correct acceptance, miss, or false alarm. A hit indicates that a change occurred in the process and the operator correctly recognized this change. A correct acceptance indicates that no change occurred in the process, and the operator correctly noted no change. A miss indicates that a change occurred in the process, but the operator said that no change occurred. A false alarm indicates that no change occurred in the process, but the operator said that a change had occurred.

This classification scheme creates some ambiguity in grading, so more specific guidelines were developed. In the case where a change occurs, and the operator notes that a change occurs, but has noted the wrong direction (e.g. the process parameter increased and the operator indicated "decrease"), the answer was characterized as a miss. Appendix D provides a more detailed description of the scoring criteria.

The Situation Awareness score is calculated based on the number of correct responses. The formula used to calculate SA is also presented in Appendix D. Two measures are calculated: operator sensitivity to the detection of process drifts and the response bias. Sensitivity and response bias scores were calculated for the primary side, the secondary side, and the entire process (i.e., both primary and secondary sides). For every questionnaire administered, these six scores were calculated. Analyses of data used only the overall sensitivity measure, which indicates how accurately the operator assessed the situation of the entire process (Hogg et al., 1994).

2.3.1.4 Workload

Workload is of particular importance in complex systems, especially in nuclear power plant control rooms, because of its effect on human error and performance. A certain amount of workload is considered optimal by both operators and designers. Some amount of workload is necessary to keep the operator engaged in overseeing the process and in the process control loop. If workload is much below this level, operators become under-stimulated and may become inattentive, lose vigilance, SA, and demonstrate degraded performance when the situation demands intervention. Also, overloaded operators generally suffer performance degradation as they are simply unable to match resources with the demands of the task. Some expert operators may be able to perform satisfactorily, at least temporarily, in an over-loaded condition but over time even their performance degrades. Between these two extremes is the optimal workload which provides sufficient challenge to keep operators stimulated without excessively taxing their capabilities (Huey and Wickens, 1993).

As nuclear power plant I&C systems become more automated, vendors expect that operator workload will decrease. Intuitively, this expectation seems reasonable: operators have less to do as automated systems are allocated more functions. However, the opposite effect may occur. Operators may have higher mental workload to understand a system with which they are not actively involved. In addition, the changes in I&C systems may shift the load in the types of resources required to interact with the systems. Some activities utilizing motor and physical capabilities of operators in conventional nuclear power plants may shift to perceptual and information processing functions of the operator in advanced plants. Although overall workload in some respects may be decreased, some processes (e.g., cognitive) may be more taxed in advanced plants.

The relationship between workload and performance is not linear or entirely predictable. As workload increases to a breaking point, operator performance may remain unchanged. Then suddenly, operator performance may deteriorate drastically (Bergstroem, 1993). In summarizing the determinants of workload and the consequences of workload on human performance, Huey and Wickens (1993) state:

"In some cases, it is apparent that human limitations reflect the consequences of poorly designed controls, displays, and automatic subsystems. In others, task demands simply exceed the operator's capabilities either momentarily or for extended periods. Despite their limitations, humans are remarkably flexible, adaptable, and capable. They can improvise, compensate for inadequate information and system or human failures, adjust to

13 NUREG/IA-0137

BACKGROUND

novel situations, exhibit graceful (rather than catastrophic) degradation, plan ahead, predict the outcome of familiar and unfamiliar events, and learn from experience. However, the consequence of extreme demands and requirements to act creatively and adaptively impose significant workload on the human operators of complex systems." (National Research Council, p. 85)

A further consideration about workload and performance is that changes in workload, especially sudden transitions from low to high workload, have been shown to produce the most pronounced decrements in performance (Kantowitz and Casper, 1988). This aspect of the relationship between workload and performance is especially significant for advanced NPP control room evaluation. In the advanced plants reviewed for this study, many routine operations of the plant during normal power operation will be automated. Sudden increases in workload when the crew is called upon to respond to an unanticipated situation may result in performance problems. A companion issue in the present study is whether the degree of workload produced by the staffing complements will produce marked differences in the workload experienced by the remaining members of the minimum control room crew complements. Members of the minimally-staffed crews which lack one or more licensed operators to assist during peak activity periods may experience an increase in workload.

The NASA Task Load Index (TLX) is a subjective workload measurement technique that has been validated in several studies and widely applied (Wierwille and Eggemeier, 1993; Hill, Iavecchia, Byers, Bittner, Zaklad, and Christ, 1992; and Moroney, Biers, Eggemeier, and Mitchell, 1992). This technique offers several advantages over other subjective measurement techniques. It is particularly suited for use in complex systems (Weirwille and Eggemeier, 1993) and applied settings (Nygren, 1991). It has been shown to be a globally sensitive measure of workload (Moroney et aL, 1992). Weirwille and Eggemeier (1993) recommend that TLX should be given strong consideration in test and evaluation applications. Furthermore, Hill et al. (1992) found TLX to be one of the best subjective workload assessment techniques in terms of operator acceptance and validity.

The NASA Task Load Index (TLX) subjective workload rating scale was used for the measurement of operator workload in this study. NASA TLX measures six components of workload: mental demand, physical demand, temporal demand, performance, effort, and frustration. Operators use a scale to rate workload for each component. The rating scale is from 0 to 100, with 0 being "no workload" and 100 being "highest possible workload." Participants were instructed to rate only the segment of the scenario prior to the interrupt. Participants were not given specific anchors, but were instructed to try to maintain consistency with their own use of the scale. Examples of the measurement scale and definitions provided participants are provided in Appendix C.

To analyze subjective workload, ratings of the six workload components were averaged. All six rating scales were weighted identically. One of the sub-scales, performance, was inverted for purposes of averaging. This is because the other workload component scales are generally at 0 indicating "no workload" and 100 indicating "extremely high workload." So, for most scales, a lower score is more desirable. However, in the "performance" scale, a higher score is more desirable. A low score indicates poor perceived performance. Since the NASA TLX rating scale was designed and validated in this manner, it was administered in this format.

2.3.1.5 Team Interaction

Most vendors of advanced plants envisage control room operators working in teams, though they differ to some extent in how many team members they consider necessary to safely operate their plants. The reduction by one or two crew members in a shift staffing complement represents a significant reduction in overall control room resources. The ultimate reference criterion about the adequacy of a staffing decision is the performance of the crew. Crew performance involves not only objective performance of the necessary control room tasks (e.g., procedural tasks), but also how well the crew works as a team in coordinating

14 NUREG/IA-0137

BACKGROUND

activities, developing goals and objectives, and communicating together and with others outside the control room.

Operators perform as members of a team in the control room of a nuclear power plant. Few studies have considered team interaction in complex, dynamic systems (Coury and Terranova, 1991). Those few studies that have looked at team interaction have shown teams to perform better than individuals in controlling a complex, dynamic system, and diagnosing failures. Teams control deviations better, require fewer control actions to achieve control, and are more efficient in information gathering than are individuals (Hooper, Coury, and Terranova, 1991; Montgomery and Hauth, 1991).

However, team interaction may have certain drawbacks. In a team situation, individual members may become complacent, relying on other members to do certain tasks without necessarily verifying that the tasks have been performed. Teams may also develop overly cohesive or dependent thinking habits in which members are hesitant to introduce or entertain discrepant information for fear of creating conflict or disturbing group cohesion. Pooling the abilities of a number of highly-skilled individuals should result in a better product, but this effect does not always occur. Group processes can be slow, cumbersome, inefficient, and lead to individual member frustration. Such problems must be considered in determining the correct balance between crew tasks and system dependence on humans (Foushee and Helmreich, 1983; Cooper, White, and Lauber, 1979).

Differences between conventional and advanced control rooms may influence the team interaction process. In conventional control rooms, for example, the work area typically occupies a physically larger area than in advanced control rooms. This larger area and the division of responsibilities between reactor and balance of plant operators may result in operators working at some distance from one another, although still within visual range. While the division of responsibilities may remain the same in advanced control rooms, the operators will not be working as far from one another due to the smaller control room areas in the designs reviewed for this study. Physically closer working areas may support team interaction. Alternatively, some advanced design features have been cited as potentially detracting from team performance. Intensive work with VDU-based systems may result in focusing or narrowing of an individual operator's attention on the VDU system, at the expense of information and activities outside this periphery. Such cognitive tunnel vision may be produced by some computer-based activities (e.g., via menu or system navigation) and result in reduced communication between operators and hence, decreased team interaction and performance.

Team interaction is important in this study because of the potential for variations in task load, staffing, and control room design to influence the adequacy of the performance of the control room crew. Several measurement techniques have been developed to assess the quality of team interaction. These measurement techniques use ratings of communication and other aspects of teams. One technique, the Behaviorally Anchored Rating Scale (BARS) has been used to assess team interaction though it is not a test per se (Meister, 1985). Using a BARS measurement technique, dimensions of team interaction have been assessed previously (Baker and Salas, 1992; Montgomery and Hauth, 1992; Montgomery, Gaddy, and Toquam, 1991). The actual crew interaction dimensions assessed vary in different studies, but in general they are team behaviors (e.g., communication, cooperation, task coordination, team spirit, maintaining task focus, and adaptability, etc.).

Team interaction was measured in this study by using a Behaviorally Anchored Rating Scale (BARS) developed with experienced plant personnel. Five behavioral categories were used to assess team interaction: communication, openness, coordination as a crew, team spirit, and task focus and decision making. Specific behaviors for each of these dimensions were obtained from previous studies that used similar dimensions (Glickman et al., 1987; Morgan et al., 1986). Each dimension was rated on a scale of 17, with one being the lowest rating and 7 being the highest. On the rating forms, specific examples of positive and negative behaviors were provided to the raters for reference purposes. Raters had these examples with them as they observed crew performance. Participants were rated as crews, not as individuals. Nonetheless, some crew members might exhibit behavior that would lead to a high rating on a specific dimension, while others might exhibit behavior that would lead to a low rating on the same

15 NUREG/AA-0137

BACKGROUND

dimension. Such situations could produce an intermediate rating of the behavioral category for the crew. An example of a BARS rating scale and form used by raters in this study is provided in Appendix C.

As with the workload data, the five team interaction scales were averaged to obtain an overall rating of team interaction. All BARS scales were designed so a higher score indicates more desirable team performance, so no inversions were necessary. Further, after checking the inter-rater reliability, the scores were averaged across raters.

16 NUREG/IA-0137

METHOD

3. METHOD

3.1 Experimental Design

Eight crews of operators participated in this study. Four crews participated in the normal staffing complement (4-person crews), and four in the minimum staffing complement (3-person crews in the conventional plant and 2-person crews in the advanced plant). Table 3.1 shows the experimental design of the study.

Table 3.1 Experimental study design

I Crew Size

Plant Type Normal Staffing Level Minimum Staffing Level

Conventional 4-person 3-person

Advanced 4-person 2-person

The normal staffing complement is based on current federal regulation (i.e., 10 CFR 50.54(m)) for staffing of a single operating unit at a single unit site, which is similar to the current staffing practice at the Loviisa plant. In the normal staffing complement for both plant types, the control room crew consisted of a control room supervisor/shift supervisor (CRS), a reactor operator (RO), balance of plant operator (BOP operator) and a control room technician (CT). The minimum staffing complement is based on the minimum crew size considered by an advanced passive plant vendor, or the minimum crew composition that would typically occur at Loviisa, which served as the conventional plant. In the conventional plant, the minimum crew complement of 3 included a CRS, RO, and BOP operator. In the advanced plant, the minimum crew complement of 2 included a CRS and either an RO or BOP operator, the CRS himself acting as a control room supervisor and an operator (either RO or BOP operator).

Four crews participated in the conventional plant condition, and four participated in the advanced plant condition. The conventional plant was modeled at the Loviisa Nuclear Power Station Training Simulator. The advanced plant was modeled in the HAlden Man-Machine LABoratory (HAMMLAB), in Halden, Norway. The Loviisa plant main control room, upon which the training simulator is modeled, and the HAMMLAB simulator control room are shown in Figures 3.1 and 3.2.

Each crew was presented with the same five scenarios over a period of two to three days. The scenarios were presented in random order to the crews. Scenario duration ranged from one hour to two and a half hours. All scenarios contained several faults and events that occurred regardless of operator actions, but scenario outcome could vary significantly depending on the crew's responses. Operator actions had the influence they would in the plant. The scenarios were nearly identical between both plant types, but modifications were made to simulate plant type-specific performance differences (i.e., passive system or increased automation).

Scenarios were divided into segments based on plant state and critical crew actions. The crews participated in the scenario, and, when a segment was complete, the simulator was temporarily frozen. During this pause, situation awareness and subjective workload data were collected from the crew, and team interaction data were gathered from the raters. These pauses lasted approximately 5-7 minutes. Observers who provided ratings of team interaction sat in the simulator area behind and to the side of the shift crew in an

17 NUREG/IA-0137

METHOD

area that would not interfere with the crew's activities. These two raters (one a licensed SRO from Loviisa, the other a systems analyst with a background in plant operations, thermal hydraulics, and risk assessment) assessed team interaction. At the end of the scenario, these two raters, together with a third rater (also a licensed SRO in charge of performance-based training at Loviisa) provided evaluations of the crew's performance.

Figure 3.1 Side view of the Loviisa nuclear power station main control room

Three of the scenarios employed in the study (i.e., the SGTR, ISLOCA, and LOFW scenarios) contained five data collection segments; the other two scenarios (i.e., the LOOP and SGOF scenarios) contained four segments (see Section 3.1.2 for a description of the scenarios). The first segment of each scenario contained a normal operating task, for example, load following. No disturbances occurred during this scenario segment. In the second segment, the simulated initiating event was introduced. After the crew noticed a malfunction in the plant and had begun taking steps to respond, the next interrupt occurred. During the ensuing segment(s), the crews proceeded with functional response strategies, event diagnosis (i.e., for eventbased procedure selection), and began stabilization and cooldown of the plant. Novel disturbances were generally not introduced, unless a crew's action resulted in a fault (e.g., operating a plant system in such a way that eventual damage or failure of equipment would result).

During each scenario, crews were required to make off-site notifications and to communicate with operators and other plant staff outside the main control room. The crews simulated performance of these notifications and communications by phoning or radioing to simulator instructors. The instructors made notes listing the agency or person to whom the call was made and the time the call was received. These data were later used in the evaluations of transient management.

18 NUREG/IA-0137

7777ýýL

METHOD

Figure 3.2 Side view of the HAMMLAB experimental control room

Performance measures were collected throughout the scenario. The measures collected during the first segment provided a baseline of operator performance data during normal operation. Operator performance was also assessed following the onset of the disturbance, during mitigation efforts, and during the resolution phase. Data collected across time periods in the scenario permitted comparisons of these types of design basis scenarios on baseline measures. Comparisons could also be made between crew staffing complements and plant types, and whether these differences increase as a function of scenario length, demands of specific scenarios, etc. The baselines also provided a measure for evaluating the comparability of crews, to ensure that any potential pre-study differences between crews did not affect the outcomes and inferences drawn from the study of performance data.

3.1.1 Participants

Eight crews of operators from the Loviisa nuclear power station in Loviisa, Finland, participated in the study. Crews were selected from actual operating crews at the plant. All participants were male and all were licensed plant operators, having the equivalent of an RO or SRO license. Control room/shift supervisors at Loviisa are degreed technical personnel (e.g., engineers). All operators spoke Finnish during the study.

All crews included a reactor operator (RO) and a balance of plant operator (BOP operator). In the 2-man crew, one of the operators was also a licensed supervisor and performed supervisor functions. The 3-man crew included a dedicated Control Room Supervisor (CRS) apart from the RO and BOP operators. The 4-man crew had a CRS as well as an additional licensed operator, a control room technician (CT). The CT

19 NUREG/IA-0137

METHOD

performed necessary supporting control room tasks as directed by the CRS, and assisted in monitoring the status of critical plant functions.

All crew activities and interactions were carried out in Finnish, the native language of the operators who participated in this study. Procedures that were used at both test sites, as well as all general instructions, and data collection inventories were written in or translated to Finnish.

3.1.1.1 Training

All crews were trained for participation in the study. Participants were informed about the overall purpose of the study and shown copies of the inventories used for data collection. They received verbal instruction on requirements for completing the SA and workload inventories, with emphasis on the need to work individually. Copies of the instructions to subjects are presented in Appendix B.

Further, crews received simulator-specific training. In the conventional plant setting, this training required approximately 2 hours and addressed modifications of the Instrumentation and Control (I&C) systems and plant performance characteristics. In the advanced plant condition, the crews received two and a half days of training on use of the HAMMLAB simulator and the interface. The difference in the amount of simulator-specific training between the two conditions was due to the operators' differing familiarity with the two simulators. Operators were trained on the Loviisa simulator, and work on the actual Loviisa plant. Only a couple of hours of instruction and practice on minor modifications was needed. In HAMMLAB, the simulator models the Loviisa process, but with some differences. Also, the interface in HAMMLAB differs significantly from Loviisa simulator interface. Operators needed a much longer training period for the advanced plant condition, modeled in HAMMLAB. Following the training, the operators had a 15 minute break prior to participating in the scenarios and data collection.

The raters of team interaction were trained on the use of these inventories. These raters had participated in the design of the team interaction assessment inventory and field-tested the inventory during a pilot study. A written copy of the verbal instructions the expert raters received is presented in Appendix B, and the inventory reference aid to which they referred during scenarios is presented in Appendix C.

3.1.2 Scenarios

The scenarios in which the crews participated were delivered on full-scale nuclear power plant simulators. The training simulator at the Loviisa Nuclear Power Plant served as the conventional plant; HAMMLAB served as the advanced plant. Data were collected during five design-basis scenarios.

"* Steam Generator Tube Rupture (SGTR) with a fire in the turbine building and a stuck-open steam generator safety relief valve in the affected steam generator;

"* Total Loss Of FeedWater (LOFW);

"* Loss Of Off-site Power (LOOP) with a stuck open steam generator safety relief valve;

"* Interfacing Systems Loss Of Coolant Accident (ISLOCA), and;

"* Steam Generator Over-Fill (SGOF).

The scenarios were chosen to cover a range of design basis conditions including overheating of the primary coolant system, overcooling of the primary coolant system, and loss of coolant from the primary system. These conditions correspond to those analyzed both in conventional plant safety analyses, and in the safety

20 NUREGAIA-0137

METHOD

analysis reports produced by the nuclear steam supply system (NSSS) vendors for the plant designs considered in this study.

The first three scenarios listed above were considered to be rule-based. That is, the crews at Loviisa are well trained on the procedures for these scenarios and have experienced them during training. These three scenarios are not difficult to diagnose, each having a number of salient symptoms.

The last two scenarios may be considered to be knowledge-based scenarios. The ISLOCA included a number of secondary instrument failures that were produced by the initiating event and that complicated the determination of the source of the leak. In the SGOF scenario, the failures of the feedwater flow control valves and ensuing steam generator level increase occurred in quick succession. The pace of these events may interfere with the crew's ability to identify and select the correct system procedure for responding to this scenario.

Several features of the scenarios improved the realism and the degree of challenge. The scenarios required the crew to coordinate with external personnel; the scenarios were of long duration; and by their control actions, the operators could influence the outcome of the scenario. Crews made radio and/or phone calls to simulator instructors who acted as field operators, maintenance personnel, instrumentation technicians, radiation protection personnel, and firefighters. Further, crews made off-site emergency notifications in the scenarios. The scenarios generally required an hour to two and a half hours. This duration allowed the researchers to observe and measure crew performance issues over time.

Appendix A provides more detail about the initial plant conditions, a brief synopsis of each scenario, the specific event sequence, expected or typical crew activities, and typical scenario pause times used for collection of SA, workload, and team interaction data.

3.1.3 Simulator Performance Characteristics

In specifying performance characteristics of the two simulators used for this research, the aim was to produce performance representative of the reference plants identified for the study. The Loviisa nuclear power plant model was used in the simulations both at Loviisa and in HAMMLAB. Since this nuclear power plant is a Russian light-water PWR design, the simulators were modified slightly to produce performance representative of both conventional and advanced plants. For the portion of the study conducted at Loviisa, U.S. conventional plant performance characteristics were identified from published information. For the portion of the study conducted in HAMMLAB, advanced passive plant performance characteristics were identified from advanced plants' safety analysis reports.

For the conventional plant model, key event and time constants for specific thermal-hydraulic behavior of conventional plants were identified from Wheatley, et al. (1987). The reference study provides information on three design basis scenarios similar to, or having features in common with, three of the scenarios used in this study. Wheatley, et al. include modeling of plant response to a total loss of feedwater, steam generator overfill, and small-break loss of coolant accident for a Babcock & Wilcox, Combustion Engineering, and Westinghouse plant. Using key events or thermal hydraulic time constants from the reference study meant that for the overheating and overcooling scenarios, the steam generator models in the Loviisa Training Simulator (LOTS) were reduced to about 1/3 of their actual capacity. This ensured that dryout of the steam generators and heatup of the primary coolant system in the loss of feedwater scenario, for example, occurred at about the same rate and in the same range of time as were predicted to occur in the conventional plants in the reference study.

For the advanced plant model, the safety analysis reports (SARs) provided to the NRC by the vendors were reviewed to find time constants, key events (e.g., passive system actuations) and a description of thermalhydraulic performance for scenarios similar to those used in this study. Of the four safety analyses used, only two describe design basis or accident analyses to a level of detail that allowed extrapolation of key

21 NUREG/IA-0137

METHOD

events or thermal hydraulic time constants to the level of detail required. Of these two safety analyses, only one is a PWR. However, the accident analyses in the SAR for the particular scenarios include different assumptions about equipment availability and recovery than the designed scenarios. As a result, best estimates, based on the advanced plant SAR were used to determine the significant characteristics to be used in emulating advanced plant performance. This meant, for example, that dryout of the steam generators and heatup of the primary coolant system in the loss of feedwater scenario would take approximately twice as long to occur in HAMMLAB than at LOTS. In addition, some safety system actuations were programmed to occur automatically (e.g., HPSI) when certain conditions existed in the plant (e.g., inventory loss) to emulate passive system actuations.

In addition to simulator thermal-hydraulic performance characteristics, the instrumentation and control (I&C) systems at Loviisa and in the Halden Man-Machine Laboratory (HAMMLAB) were reviewed and adjusted to make them representative of conventional and advanced plants, respectively.

22 NUREG/IA-0137

ANALYSIS

4. ANALYSIS

Analyses of data were performed on performance measures to determine whether overall performance of crews and specific aspects of crew performance were affected by staffing levels (i.e., normal vs. minimum crew complements) and plant type (i.e., conventional vs. advanced plants). Analyses were also performed to determine whether crew performance varied as a function of scenarios or scenario types (i.e., rule-based, knowledge-based). The experimental design of the study comprised a between-within design, having both between groups factors (i.e., crew size, plant type) and within-subjects factors (i.e., repeated measurement of subjective performance measures, scenarios).

The data analyses from this study were performed using the Statistica software package. Applying statistical tests to experimental data introduces the risk of two fundamental errors. A Type I error occurs when the researcher concludes that main experimental effects exist when actually none are present. In contrast, Type II errors occur when the researcher concludes that no experimental effects exist when such effects actually are present. In dynamic simulator studies such as these, the likelihood of committing a Type II error is larger than committing a Type I error. According to Wiener (1971), too much emphasis has been put on the significance level convention of 0.05 and 0.01:

"When the power of tests is likely to be low under these levels of significance (0.05 and 0.01), and when type I and type II errors are of approximately the same importance, then .30 and .20 levels of significance may be more appropriate than the .05 and .01 levels." (Wiener, 1971 page 14)

In this study, it was as important to avoid erroneously concluding that the performance of different-sized crews was equal when, in fact, it was different, as it was to avoid concluding that it was different when it was, in fact, equal. Thus, to avoid type II errors, results approaching the 0.20 level of statistical significance were also taken into account and are discussed.

The general approach to the analysis of data in this study was to first conduct multivariate analyses of variance (MANOVAs). Assuming that a significant difference was obtained in the MANOVA, univariate analyses of variance (ANOVAs) were conducted to estimate specific effects. Some of the specific study issues entailed a comparison of particular study groups. In such cases, following the univariate ANOVA, a post-hoc test was conducted.

To evaluate study issues, repeated measures Multivariate Analyses of Variance (MANOVA) were performed on the transient management, situation awareness, subjective workload, and team interaction data. As mentioned earlier, data on each of these measures were obtained a number of times during each scenario; these are the repeated measures in this analysis. The independent variables in the analysis were: scenario, crew size (normal or minimum crew complement), plant type (conventional or advanced), and scenario period (i.e., period in the scenario in which measures were obtained). Because of the potential for a large number of interactions between the main effects, and to simplify the explanation of results, a Type III approach was used to estimate the effects of the independent variables in all analyses of variance (SAS 1982; StatSoft, 1994). In the Type III approach, tests are conducted on specific effects, including interactions, after controlling for all other possible effects specified in the analysis. A significant effect represents a unique effect for a specific factor after controlling for all other effects or factors. Post hoc analyses of data were performed using the Tukey Test for Honestly Significant Differences for unequal sample sizes; the Spjotvoll and Stoline test (Spjotvoll & Stoline, 1973, StatSoft 1994, p. 1589).

23 NUREG/IA-0137

RESULTS

5. RESULTS

5.1 Transient Management and Task Performance

Crew performance was analyzed using objective criteria to evaluate their management of the transients and their performance of control room tasks. Analyses were performed on three performance categories: 1) announcements and notifications, 2) transient mitigation, and 3) cooldown and stabilization. Scenariospecific tasks were identified for each group of activities, and each crew received a rating from 1 (poor performance) to 5 (best performance) based on performance of these tasks. Ratings were combined and a score for each task category obtained for each crew. Ratings were subsequently analyzed. The independent variables used in the analysis were scenario, crew size, and plant type. A 3-way repeated measures (i.e., by

scenario) Multivariate Analysis of Variance (MANOVA) was conducted, and showed significant effects of crew size and plant type. The results are summarized in Table 5.1.

Table 5.1 Effects of scenario, crew size, and plant type on transient management and task performance

Summary of Effects:

Effect Rao's R/F df (effect) df (error) p-level (a)

Scenario 0.84 (F) 4 35 not significant Crew size 20.65 (Rao's R) 3 2 < 0.05

Plant Type 204.51 (Rao's R) 3 2 < 0.005

Objective performance measures did not vary significantly as a function of scenarios. Crews performed the necessary tasks about as well in all scenarios. This indicates that the crews coped equally with the study scenarios, regardless of the differences. This is similar to findings from the analyses of workload, team interaction, and crew performance ratings

5.1.1 Crew Size and Plant Type Effects

Both crew size and plant type exhibited a significant effect on the crews' performance of their tasks. In contrast to many of the other findings, there were no significant interactions of crew size and plant type on

these task performance measures. Further post hoc analyses of data were conducted on the individual measures of transient management and task performance for which significant results were found.

Figures 5.1 and 5.2 show plots of transient management performance measures for the two different staffing complements and plant types, respectively. Both crew complements performed important mitigation tasks equally well. However, the crews in the normal crew complement conducted cooldown and plant

stabilization tasks better than crews in the minimum crew size complement (p < 0.02). The normal sized crews also carried out announcement and notification tasks slightly better than the smaller crews, though the difference is not statistically significant.

Crews in the advanced plant showed improvements in all aspects of mitigation activities, relative to performance in the conventional plant. Crews showed a slight improvement in identification and notification tasks, and larger improvements in mitigation tasks and cooldown and stabilization tasks relative to the performance of crews in the conventional plant. The differences between the performance of the crews in the two plant settings are significant for both important task performance (p < 0.01) and cooldown and stabilization tasks (p < 0.001).

24 NUREG/IA-0137

RESULTS

3.4

3.2

3.1 .

3.2.9 ....... .........---------_._.......__.......-------

a.O

28 .............-

S2.7 . ....-.................................... . ... ".. . . ... . . . . .Announcements&.Notifications

-"-0- Important Task Performance 2.6 -. ....-. - Cooldown & Stabilization I

2.5 Minimum Normal

Crew Complement

Figure 5.1 Plot of transient management performance measures by crew size

4.5

4 .0 ....................... ................... .. .. .. ........................ .................... ... . . . . . . .............. ... . ............................. .. . .. .......... ..... .. .......... .................

3 .5 ............................ '. ...

350 ..... ...... .......... ...... ....... ... 4....... .....

23 . .......... ........ .... ...... ........ .. ........ ......................... .. ..........

, , [ -O-Announcements & Notifications:

i ./ / 2.0

2.0 ... .-- - ----.................... -l- Important Task Performance

- Cooldown & Stabilization

1.5', Conventional Plant Advanced Plant

Figure 5.2 Plot of transient management performance measures by plant type

25 NUREG/IA-0137

RESULTS

5.2 Rated Crew Performance

Ratings of crew performance obtained from the trained observers following each scenario were analyzed. The independent variables used in the analysis are: scenario, crew size, and plant type. A 3-way Analysis of Variance (ANOVA) was conducted on the data, and the results are summarized in Table 5.2.

Table 5.2 Effects of scenario, crew size, and plant type on rated crew performance

Summary of Effects: Effect F df (effect) df (error) p-level (cx) Scenario 0.81 4 98 not significant Crew size 4.55 1 98 < 0.05 Plant Type 23.26 1 98 < 0.001 Interaction Effect Crew size X plant type 17.72 1 98 < 0.001

5.2.1 Inter-Rater Reliability

Three trained observers made ratings of crew performance using the behavioral scales adopted in this study. Following the collection of data, the inter-rater reliability or degree of correspondence among ratings made by the raters was assessed. A Pearson product moment correlation coefficient of 0.65 was obtained among the raters.


The staffing levels, plant types, and their interaction exerted significant effects on crew performance. There were, however, no significant differences between ratings of crew performance for the different scenarios used in the study.

Figure 5.3 is a plot of the ratings obtained on crew performance measures for the different crew complements in the conventional and advanced plants. Ratings on the measures of crew performance range from I (i.e., poor performance) to 10 (i.e., excellent performance). The analyses indicate that normal sized crews performed better than minimum sized crews, and that crews in the advanced plant performed better than in the conventional plant. However, the interaction between these two factors, as shown in Figure 5.3, indicates that the minimum crew complement in the conventional plant primarily influences these outcomes. The crews in the other conditions demonstrated better control of the plant, adopted consistently better solution paths in mitigating the disturbance, communicated better, and displayed more confidence in their performance and decisions than did minimum sized crews in the conventional plant (p < .001).

The plot of rated crew performance indicates that the minimum sized crews achieved better performance in the advanced plant than in the conventional plant. Their performance was also slightly better than the normal sized crews in the advanced plant. Crews participating in the normal shift staffing complements maintained their performance levels on these performance criteria in the advanced plant.

26 NUREG/IA-0137

RESULTS

8.5

E750------ ......... ... .............. ...... . . . 6.0

.0

0 .5

"5.5 6.0 ------------------ - ... normal crew complement. ...

-[--- minimum crew complement

5.0 Conventional Plant Advanced Plant

Figure 5.3 Plot of rated crew performance by crew size and plant type

27

NUREG/IA-0137

RESULTS

5.3 Situation Awareness, Subjective Workload, and Team Interaction

The results of the analysis demonstrate significant effects of scenario, crew size, plant type, and scenario period on some or all of these performance measures. These are shown in Table 5.3. The conditions that are the focus of this study, crew size and plant type, appear to influence operator and crew performance. So, too, do the different scenarios developed for this study as well as the period within the scenario (e.g., stage of the transient in which the performance measures were obtained). The effects of plant type, crew size, scenario, and period are further modified by interactions with some or all of the performance measures. The significant crew size by plant type interaction indicates that the observed differences on some performance measure due to crew size were not constant across plant types. A specific crew complement that exhibited better performance on a measure in one plant setting may demonstrate either better or worse performance in the other plant setting. Further, this second order interaction (i.e., crew size by plant type) is modified by scenario period and the different scenarios themselves. The effects of crew size and plant type may therefore be different across scenarios and scenario periods. Each of the performance measures shows an interaction with scenario period, indicating significant deviations in the performance measures as a function of different stages of the transients. Means, standard deviations, and cell sizes for each of the performance measures analyzed are provided in Appendix F.

Table 5.3 Results of repeated measures MANOVA on situation awareness, workload, and team interaction performance measures

Summary of Effects:

Rao's R df (effect) df (error) p-level (a) Effect Scenario 2.41 12 47 < 0.02 Crew size 5.35 3 18 < 0.01 Plant type 11.78 3 18 < 0.001 Period 59.2 9 12 < 0.001 Interaction Effects

Crew size X plant type 9.64 3 18 < 0.001 Scenario X period 6.27 36 46 < 0.001 Crew size X period 3.32 9 12 < 0.05 Plant type X period 4.90 9 12 < 0.01 Scenario X plant type X 2.05 36 46 < 0.02 period Crew size X plant type X 3.90 9 12 < 0.02 period Scenario X crew size X plant type X 1.98 36 46 < 0.02 period I I

28NUREG/IA-0137

RESULTS

The shift staffing complements in the two plant settings differ on the dependent measures, though the results do not indicate how (i.e., which measures or which crew size complement had higher or lower scores). The significant interaction effects of scenario and scenario period indicate that performance measures do not differ across scenarios alone, but also across some portion(s) of the scenarios. This means that aggregating performance data across scenarios or scenario periods may not be warranted for some measures. In subsequent univariate analyses of individual performance measures, the effect of scenario is tested separately, prior to aggregating and analyzing data.

5.4 Situation Awareness

Previous analysis of situation awareness data (Hallbert et al., 1995) demonstrated significant differences between rule-based and knowledge-based scenarios. Prior to the analysis of main effects in this study, scenarios were therefore grouped into these two categories, which are referred to in the table of effects and ensuing discussion as "scenario type." The independent variables used in the analysis are: scenario type, crew size, plant type, crew member position (e.g., control room supervisor, reactor operator, balance of plant operator), and scenario period. A 5-way repeated measures Analysis of Variance (ANOVA) was conducted on the situation awareness data. Table 5.4 summarizes the results of this analysis.

Table 5.4 Effects of scenario type, crew size, plant type, position, and scenario period on situation awareness

Summary of Effects: Effect F df (effect) df (error) p-level (a) Scenario type 6.65 1 86 < 0.01 Crew size 0.40 1 86 not significant Plant type 2.10 1 86 < 0.20 Position 1.75 2 86 < 0.20 Period 2.68 3 258 < 0.05 Interaction Effect Scenario type X plant type 1.76 1 86 < 0.20 Scenario type X period 5.35 3 258 < 0.005 Crew size X plant type 25.23 1 86 < 0.001 Crew size X period 2.0 3 258 < 0.15 Scenario type X crew size X 2.05 1 86 < 0.20 plant type Scenario type X plant type X 1.96 3 258 < 0.15 period Position X plant type X 1.54 6 258 < 0.20 period

29NUREG/IA-0137

RESULTS

5.4.1 Scenario Type and Period Effects

The analyses show that the situation awareness of crews, in general, differed across scenario types, and scenario periods. Situation awareness follows a transition curve. This is shown in Figure 5.4. The situation awareness score ranges from a low of 0 (lacking situation awareness) to 1.0 (complete situation awareness). During normal conditions, crews possessed a relatively high level of situation awareness. Following disturbance onset, SA dropped markedly. The loss of SA from baseline conditions to its trough represents a loss of approximately 20% of the baseline SA. Following a sharp decline in SA, crew members gradually regained their SA. This occurred after they assessed the effects of the disturbance, began implementation of the necessary mitigation actions, and began to re-establish control of plant parameters. However, postaccident SA levels (i.e., period 5 measurements) were still lower than pre-accident SA levels. Figure 5.4 also includes the three study scenarios that contained five data collection periods (i.e., the SGTR, ISLOCA, and LOFW scenarios). In these scenarios, crews experienced a drop in situation awareness from period 4 to period 5. This drop, however was not statistically significant.

1 2 3

Scenario Period

4 5

Figure 5.4 Plot of average situation awareness by scenario period

30 NUREG/IA-0137

0.82

0.80

0.78

0.76

0.74

0.72

0.70

0.68

0.66

0 (U

0 U)

r

_a

---------------------- -------- .......

--- ------- ...... ------- -- .... .....................

.......... -------- - -- -------- ............

---------- ......... -- ---- -------. ....... ... ...... ------- ---- ------ --- ------ ......

- - - --- --- ---- -- ----------

.. ........ . . . . . . . . . . . ...... ............... .... .... ..... ..

............ ........ ...... .. . ............ .. .

RESULTS

5.4.2 Scenario Effects

Crews demonstrated better situation awareness on rule-based scenarios than knowledge-based scenarios. Table 5A also shows a significant interaction between scenario type and scenario period. Figure 5.5 shows a plot of SA for the two scenario types across scenario periods. The figure shows that crews experienced more variations in their situation awareness across scenario periods in rule-based scenarios than in knowledge-based scenarios. Post hoc analyses yielded significant differences between periods 1 and 2 (p < 0.001), 1 and 3 (p < 0.001), and periods 3 and 4 (p < 0.05) within the rule based scenarios. No significant differences were found between scenario periods within knowledge-based scenarios. One significant difference was also identified in period 1 between the rule-based and knowledge-based scenarios. Overall, crews possessed lower SA in knowledge-based scenarios than in rule-based scenarios. However, the transient-induced SA loss that accompanied the rule-based scenarios left the crews with roughly the same situation awareness as knowledge-based scenarios during the most disturbed phases of the scenario (i.e., periods 2-4).

0

0.850

0.825

0.800

0.775

0.750

0.725

0.700

0.675

0.650

0.625

0.6001 2 3 4

Scenario Period

Figure 5.5 Plot of situation awareness by scenario type and period

31 NUREG/IA-0137

RESULTS

5.4.3 Crew Size, Plant Type, and Position Effects

Figure 5.6 plots the situation awareness of crews in the different plant and staffing complements across scenario periods. Figure 5.7 plots the change in situation awareness from baseline of the crew complements in the two plant types across scenario periods. These two figures show a loss in SA experienced by all crews during the immediate post-disturbance period (i.e., period 2). Following the loss of SA, both the normal sized crews in the conventional plant and the minimum sized crews in the advanced plant tended to recover SA. The minimum crew complement in the conventional plant tended to lose SA over the scenario periods without recovery. Following an initial large drop, the normal crew complement in the advanced plant recovered SA.

In addition, both plant type and position had significant effects on operator situation awareness. By itself, crew size did not exert a significant effect on crew situation awareness, though differences in the SA of different crew complements were noted across scenario periods. Interactions among all three of these factors had significant effects on the situation awareness of operators in this study.

2 3 Conventional Plant

4 period 1 2 3 Advanced Plant

Figure 5.6 Plot of situation awareness by crew size, plant type, and scenario period

32 NUREG/IA-0137

0.85

0.80 --....-

0.75

0.7

0

P 0.70 ....

,0.65 ~0

0.60

0.55

0.50 . periodI

• _x /,

S...... '- -----..---- - ..

............ .............. ... ........... ........

i i \J61

4

RESULTS

6

4 -..... ---- - -----1 1---. ---- ......- -.. .. . -..... .. ......

2 * Normal crew complement 2 --- -- ~------- --.----- j--1 Minimum crew compleme I-----0 ........ --................... .... . . . ............ [ : • : :;iM i m : c e w o p m r i ::................ .• ..........

Z -2 ... ...------------.--.... . . . .. -----

-4~ ~ ~~~~~ ~~~ .. . . .. . ... . . . .. . . . . ...... .. .... .. ; e.0 * -2 ------ - -- - - - - --- ---

-8 -- - - - -2 --------

-1 0 ... .... ...... . ..... .. . .. . .... ------ .. - . ...... 4 -

-12~~~~ ------------- -------- ------ ---.-------- --------------- ------------

-14 .. . . . .. . . . .. . . . .. . .. .. . . . .--. . . . . . .- . ..... -. .. .... .. .. { .. . .... .. .. . •.. .. ....---

-16 .______I___ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ period 2 period 3 period 4 period 2 peoa3 period 4

Covnentlonal Plant Advanced Plant

Figure 5.7 Plot of change in situation awareness from baseline

5.4.4 Crew Size by Scenario Period Effects

Significant differences were found in the situation awareness of all crews across scenario periods. Post hoc analyses revealed the scenario periods for which SA differed. In the normal sized crews, SA was significantly different between periods 3 and 1 (p < 0.20) and between periods 3 and 4 (p < 0.05). Between the two crew sizes, situation awareness was significantly different in period 1 (p < 0.05). T7hese results were probably due to the considerable variation in situation awareness demonstrated by the normal sized crews in the advanced plant, dominating the performance of normal sized crews in the conventional plant. This effect is shown in Figure 5.6.

5.4.5 Position Effects

A significant difference was found between the SA of individual control room crew positions. Post hoc analyses revealed a significant difference between the SA of the shift supervisor and* that of the reactor operator position (p < 0. 15). However, as this difference was not demonstrated across scenarios, scenario periods, crew complements, or plant types it should be interpreted with caution.

33NUREG/IA-0137

RESULTS

5.4.6 Crew Size, Plant Type and Scenario Type Effects

Although crew size did not exhibit a main effect, in combination with plant type it exhibited an effect on situation awareness. Figure 5.8 shows a plot of the crew size by plant type interaction. Post hoc analyses identified significant differences between the SA of crews in the normal crew complement in the conventional plant with both the minimum crew complement in the conventional plant (p < 0.005) and with the normal crew complement in the advanced plant (p < 0.001). In addition, the SA of crews in the minimum crew complement in the advanced plant were significantly different than those in the minimum crew complement in the conventional plant (p < 0.15) and in the normal crew complement in the advanced plant (p < 0.001). Normal sized crews in the conventional plant and minimum sized crews in the advanced plant demonstrated similar levels of situation awareness, and were significantly higher than the other crews.

Conventional Plant Advanced Plant

Figure 5.8 Plot of situation awareness by crew size and plant type

34 NUREG/IA-0137

0= == 0=

0.80

0.78

0.76

0.74

0.72

0.70

0.68

0.66

0.64

0.62

. ~ ~ ..... ...... ----- ...... ----- -------

0- normal crew complement o -- minimum crew complement

RESULTS

Moreover, these trends in performance on SA extended to scenario type. Figure 5.9 shows the situation awareness of the different crew complements in both the conventional and advanced plant on rule- and knowledge-based scenarios. As can be seen in the plots, crews in the different crew complements and plant settings demonstrated lower situation awareness in knowledge-based scenarios than in rule-based scenarios. The exception to this is the minimum-sized crews in the advanced plant, which showed slightly better SA on knowledge-based scenarios. Post-hoc analyses showed that the normal sized crews in the conventional plant possessed higher situation awareness in rule-based scenarios than the normal sized crews in the advanced plant (p < 0.01). They also demonstrated higher SA on knowledge-based scenarios than both the minimum sized crews in the conventional plant (p < 0.15), and the normal sized crews in the advanced plant (p < 0.05). Crews in the minimum shift staffing complement in the advanced plant demonstrated significantly higher situation awareness on knowledge-based scenarios than normal sized crews in the advanced plant (p < 0.10) and minimum sized crews in the conventional plant (p < 0.20). No differences were found between the situation awareness of crews in the normal staffing complement in the conventional plant and the minimum crew complement in the advanced plant for either rule- or knowledge-based scenarios.

A significant effect was found in the interaction between scenario type and plant type. Post hoc analyses showed that crews in the conventional plant demonstrated better SA on rule-based scenarios than crews in the advanced plant (p < 0.15). In addition, their own SA was significantly higher on rule-based than knowledge-based scenarios (p < 0.05). Table 5.4 also shows a significant interaction between scenario type, plant type, and scenario period. Post hoc analyses, however, did not identify differences in SA on rule based scenarios between crews in the conventional plant and crews in the advanced plant across scenario periods. Neither were any significant differences found in the SA of crews on knowledge-based scenarios in the conventional and advanced plants. The interaction appears, rather, to be based on the variation in SA between rule- and knowledge-based scenarios (see Figure 5.5).

Rule-based Knowledge-based Conventional Plant

Rule-based Knowledge-based Advanced Plant

Figure 5.9 Plot of situation awareness by crew size, plant type, and scenario type

35 NUREG/IA-0137

0.85

0.80

' 0.75 0.

= 0.70

0

' 0.65

0.60

0.55

RESULTS

5.5 Workload

Subjective workload measures obtained from individual crew members during the data collection intervals from all scenarios were analyzed. The independent variables used in the analysis were: scenario, crew size, plant type, crew position, and scenario period. A 5-way repeated measures Analysis of Variance (ANOVA) was conducted on the subjective workload data. Table 5.5 summarizes the results of these analyses.

Table 5.5 Effects of scenario, crew size, plant type, position, and scenario period on operator workload

5.5.1 Scenario and Scenario Period Effects

The results of the analysis show that the scenarios used in this study, in general, did not produce significantly different levels of workload on crews. All of the scenarios used in this study were design basis events, and appear to have created roughly equivalent amounts of workload. Some of the crew members experienced more demand in some scenarios than other crew members (i.e., the significant Scenario X Position interaction). This makes intuitive sense, since some scenarios resulted in challenges to the primary systems, while others were initiated in the balance of the plant. Thus, specific crew members should experience some difference in the amount of workload placed upon them by the demands of the scenario. For example, post hoc tests showed significant differences in the subjective workload experienced by the RO in the ISLOCA and SGOF scenarios. In the ISLOCA scenario the RO experienced more workload. This was likely due to the fact that the ISLOCA produced more problems in the primary circuit than the SGOF, which was a balance of plant-induced transient. Similarly, differences in workload were observed between crew members in a few cases. The ISLOCA scenario, for example, placed greater demand on the RO than the BOP operator. This was shown in these analyses.

36 NUREG/IA-0137

Summary of Effects:

Effect F df (effect) df (error) p-level (c) Scenario 1.45 4 50 not significant Crew size 5.15 1 50 < 0.05 Plant type 39.24 1 50 < 0.001 Position 4.52 2 50 < 0.02 Period 220.7 3 150 < 0.001 Interaction Effects Scenario X position 1.96 8 50 < 0.10 Scenario X period 9.64 12 150 < 0.001 Crew size X position 4.24 2 50 < 0.02 Plant type Xperiod 10.59 3 150 < 0.001 Plant type X position 4.09 2 50 < 0.05 Crew size X

position X period 2.17 6 150 < 0.05 Crew size X plant type X period 2.86 3 150 < 0.05 Position X plant type X period 1.95 6 150 < 0.10 Crew size X position X 1.92 6 150 < 0.10 plant type X period

RESULTS

The effect of scenario period was found to be significant. Figure 5.10, a plot of the subjective workload for all crews in the study, shows the development of workload over scenario periods. This shows a marked effect of workload transition produced by the demands of the scenarios. The subjective workload scale has a range from 0 (total absence of workload) to 100 (extremely high workload). As can be seen, workload increased dramatically following the onset of the disturbance. The amount of subjective workload experienced by the operator following a major disturbance is approximately twice the amount experienced during normal operations. After reaching a peak during the middle of the scenario, workload subsided, but did not fall to pre-disturbance levels.

In summary, the results indicate that the scenarios produced roughly the same amount of physical, mental, and temporal demand on the crews. As these were all design basis scenarios requiring significant crew efforts to manage their effects, this result is not surprising. Two additional findings related to position and scenario period effects were observed. These indicate that the technique used for measuring subjective workload was sensitive to differences in workload experienced by different crew members in some scenarios, and during different periods of the scenarios. Although no differences in subjective workload were observed between the different scenarios on the whole, workload was higher in some scenario periods than in others.

1 2 3

Scenario Period

4 5

Figure 5.10 Plot of average subjective workload across scenario periods

37 NUREG/IA-0137

50

45

40

0

0 S35

30 0

25

20

.... ... .......... ------ ....... ------ ----- ---------

............ .............. . . . . .. . . . . . . . . . . . . ... ...... .. . . . . . .. .. . . . . . . . .. .............. . . . . . . . . . . . . . . . . . . .................. . . . . . . . . .. . . . . . . . . . .. . .. . .. . . . . . . . . . . . . ..... ...........

........ .... . . . . .. . . . ... ..... ..... .. . . . . . . . . .. . .. . . . .. ....... . .................... ..z ............ . . . . . . . . . . . . .. .. . . ............. ...... . . . . . .....................

---- ------- ------------------- .. ......

RESULTS

5.51 Crew Size, Plant Type, and Position Effects

The analyses show that crew size, plant type, and position themselves and by their interaction (i.e., crew size by position and plant type by position interactions) significantly affected the workload of operators in this study. The interactions between these factors were also modified by scenario period. This is demonstrated by the significant interaction between crew size by position by period, plant type by position by period, and crew size by plant type by period. The four-way interaction between crew size, plant type, position, and scenario period was also significant.


Crew members in the normal crew complement experienced less workload on average than crew members in the minimum staffing complement. Figure 5.11 shows the average subjective workload of the control room crew members in the two crew complements. The figure shows that the shift supervisor experienced significantly more workload in the minimum crew complement than in the normal crew complement (p < 0.001). However, this relationship did not hold true between the conventional and advanced plants: Figure 5.12 shows that shift supervisors in the two plants experienced similar demand. However, differences in workload for both RO and the BOP operators were observed between the conventional and advanced plants (p < 0.001 and p < 0.05, respectively). Both ROs and BOP operators experienced significantly more workload in the advanced plant than in the conventional plant.

A significant interaction was also found between position, crew complement, and scenario. Differences were limited to the workload of shift supervisors in the conventional and advanced plants during the first 3 periods of the scenarios (p < 0.001, p <0.001, and p <0.20, respectively). Both the ROs and BOP operators experienced similar amounts of workload across scenario periods in the normal and minimum staffing complements.

52

50

48

46

44

42

40

38

36Supervisor Reactor Operator BOP Operator

Control Room Crew Member

Figure 5.11 Plot of average subjective workload by crew size and position

38NUREG/IA-0137

0

S 0

CL 0

RESULTS

55

50

45

40

35

30Supervisor Reactor Operator BOP Operator


Figure 5.12 Plot of subjective workload by plant type and position

A particular issue in the study concerned the workload of the control room supervisor serving in the twoperson crew. The workload of the dual-role CRS was compared with that of other crew members, and contrasted specifically with that of other CRSs in other staffing complements. When compared with other CRSs, ROs, and BOP operators, a significant difference was observed in the amount of workload these operators experienced (F 3,106 = 5.02, p < .005), a planned comparison showing that the dual role CRS experienced significantly more workload than all others. In contrast to all other control room supervisors, a planned comparison showed that the dual role CRS also experienced higher workload than other control room supervisors (F 1,106 = 6.22, p < .02). Figure 5.13 shows a plot of the workload of individual control room crew members.

YO BOP operator


Dual Role CRS

Figure 5.13 Plot of subjective workload by control room crew member

39 NUREG/IA-0137

0

0. 0

5C

V

01

E 4z.

0

44C

34

RESULTS

5.5.4 Plant Type and Position Effects

Workload of operators in the advanced plant was significantly higher across scenario periods. Post hoc analyses of the plant type by scenario period interactions revealed significant differences between crew members' workload across scenario periods: in each scenario period most crew members experienced more workload in the advanced plant than in the conventional plant. However, this relationship does not hold for all crew members. Figure 5.14 shows the subjective workload of the different control room crew members in both the conventional and advanced plants across the different scenario periods. Post hoc analyses showed that differences in subjective workload experienced by the shift supervisor were statistically significant only in period 4 of the scenario (p < 0.001). Significant differences exist in workload experienced by the BOP operator in periods 2 and 4 of the scenario (p < 0.05 and p < 0.001, respectively). The reactor operators in the study, however, experienced significantly higher workload in the advanced plant across all portions of the scenarios (i.e., from period 1 to 4: p < 0.05, p < 0.001, p < 0.001, and p < 0.001).

65

60 ----- ... ..

55 -- ----- --- ....

50

t4O .... . . ...... .. --

1 2 3 4 1 2 3 4 Supervisor Reactor Operator

Figure 5.14 Plot of operator workload by plant type, crew member, and scenario period

40 NUREG/IA-0137

RESULTS


Figure 5.15 shows a plot of the interaction between crew size, plant type, and scenario period on subjective workload. Post hoe analyses showed that crews in the normal complement in the conventional plant experienced significantly lower workload than both crew complements in the advanced plant across all scenario periods (p between < 0.01 and < 0.001). Minimum sized crews in the conventional plant experienced significantly less workload than minimum sized crews in the advanced plant following disturbance onset (i.e., scenario period 2) (p < 0.01). They also experienced less workload than both crew complements in the advanced plant during scenario periods 3 and 4 (p < 0.001). The only intra-plant type workload differences were found to be between the normal and minimum crew complements in the conventional plant, but only for scenario periods I and 2 (p < 0.01 and p < 0.10, respectively). No significant differences were detected between the normal and minimum sized crews in the advanced plant, though the crews serving in the minimum crew complement appear to have experienced slightly more workload.

Figure 5.16 shows a plot of the change in workload experienced by the different crews across scenario periods, as compared to their baseline levels of workload. This figure shows that the crews in the advanced plant experienced higher workload transition than crews in the conventional plant (e.g., period 2 change from baseline). The results also show that the workload of crews in the advanced plant tended to increase almost linearly over the scenario. Workload of crews in the conventional plant tended to stabilize, and even decrease in the case of the normal crew complements.

2 3 4 Period Conventional Plant

j •z normal crew complement •ck minimum crew complement

1 2 3 4 Advanced Plant

Figure 5.15 Plot of operator workload by crew size, plant type and scenario period

41 NUREGAIA-0137

65

60

55

50

m45 0

040

0 35

0 30

25

20

15

10 1. Period

RESULTS

perid 2 perod 3 peiod 4 period 2 period 3 period 4 Conventional Plant Advanced Plant

Figure 5.16 Plot of operator workload change from baseline

42NUREG/IA-0137

40

35

30

0

025

20

10

15 15

RESULTS

5.6 Team Interaction

Team interaction measures were based on ratings obtained from the trained observers during the data collection intervals from all scenarios. The independent variables used in the analysis were: scenario, crew size, plant type, and scenario period. A 4-way repeated measures Analysis of Variance (ANOVA) was conducted on the team interaction data. Table 5.6. summarizes the results of this analysis.

Table 5.6 Effects of scenario, crew size, plant type, and scenario period on team interaction

Summary of Effects: Effect F df (effect) df (error) p-level (a) Scenario 1.41 4 60 not significant Crew size 1.13 1 76 not significant Plant type 18.49 1 76 < 0.001 Period 3.70 3 228 < 0.02 Interaction Effect Crew size X plant type 3.59 1 76 < 0.06 Crew size X period 2.58 3 228 < 0.06 Plant type X period 2.59 3 228 < 0.06 Crew size X plant type X 2.54 3 228 < 0.06 period

5.6.1 Inter-Rater Reliability

Two trained observers rated crew interactions. As previously discussed, the raters also had participated in the development of the rating scales and had examples of behavioral anchors for reference purposes. Each observer rated crew interactions during the same scenario periods. After collecting data from both phases of the study, the inter-rater reliability was assessed. A Pearson-product moment correlation coefficient was used to evaluate the reliability or degree of correspondence between the two raters. Analysis of data indicated a moderate degree of correspondence or inter-rater reliability (r=0.55). This is due to variation in ratings made on the individual crew interaction sub-scales. Due to this variability, no analyses of team interaction sub-scales were performed. This is because some of the variability that might influence results and inferences drawn could be due to either honest differences of opinions held by the raters, criterion shift, or both.

43 NUREG/IA-0137

RESULTS

5.6.2 Scenario and Scenario Period Effects

The results show that team interaction did not differ significantly across scenarios, but as with other performance measures, it differed significantly across scenario periods. This is shown in Figure 5.17. Team interaction was rated on a scale from 1 (poor team interaction) to 7 (optimum team interaction). The data indicate that during normal conditions, crews showed a moderate level of team interaction (i.e., a score between 4 and 5 on a scale from 1 to 7). Following the initiating event, crews mobilized, and team interaction increased. This is probably because during normal operating activities, tasks do not require as close a coupling of activities. During an abnormal event, especially during a design basis event, the challenges to the plant require much closer and more sustained coordination of activities. Following this initial mobilization, team interaction gradually declined. By the fourth data collection period, team interaction dropped to pre-disturbance levels. By the end of the scenario, team interaction decreased to slightly below pre-disturbance levels.

I 2 3 4 5

Scenario Period

Figure 5.17 Plot of average team interaction across scenario periods

44 NUREG/IA-0137

6.0

5.5

5.0

4.5

4.0

3.5

3.0

C

S 0 C 2: E

........................... .......

.............. ............. ---- ----- ......... . . .........

----------------- -------- --- ------------ -- -- -------

- ----- --- ------ -- - ----

.......... -- - --------------- - ------- ------ -- --- ------------------ - - -----

RESULTS


The results of the analysis show a main effect for plant type, but not for crew size. However, a number of

interactions between crew size, plant type, and scenario period occurred that affected team interaction.

Figure 5.18 shows the interaction of crew complement and plant type on team interaction. The figure shows

an improvement in team interaction in the advanced plant as compared to the conventional plant. In

addition, the most improvement in team interaction was achieved by the minimum crew complement. Post

hoc analyses demonstrated significant differences between: 1) normal and minimum sized crews in the

conventional plant (p < 0.20); 2) the normal sized crews in the conventional plant and the minimum sized

crews in the advanced plant (p < 0.10); and, 3) the minimum shift staffing complement in the conventional

plant and both the normal and minimum sized crews in the advanced plant (p < 0.005 for both). Crews

tended to perform better as a team in the advanced plant setting; the more so for minimum sized crews.

Whereas the smaller crew complements exhibited poorer team interaction in the conventional plant in

comparison to the larger crews, in the advanced plant they performed at least as well as a team.

C a 0 S S C

E U S I-

5.6

5.4

5.2

5.0

4.8

4.6

4.4

4.2

4.0Conventional Plant Advanced Plant

Figure 5.18 Plot of team interaction by crew size and plant type

45 NUREG/IA-0137

RESULTS

The data indicate that the effects of crew size and plant type varied across scenario periods (i.e., the crew size by scenario period and plant type by scenario period interactions). Figure 5.19 is a plot of the interaction effect of plant type and scenario period on team interaction. This figure shows that crews in the advanced plant exhibited better team interaction than their counterparts in the conventional plant. Post hoc analyses indicated that this S,'ý,-.;,urred across each of the scenario periods (for periods 1-4: p< 0.001, p< 0.02, p< 0.001, and p< 0.001, respectively). During some scenario periods, crews in both plants demonstrate slightly better team interaction relative to other scenario periods, but these differences were not statistically significant.

5.8

5.6

5.4

5.2

5.0

4.8

4.6

4.4

4.2

4.01 2 3 4

Scenario Period

Figure 5.19 Plot of team interaction by plant type and scenario period

46 NUREG/IA-0137

C

0

E

RESULTS

Both crew size and plant type interacted with scenario period (i.e., the crew size by plant type by scenario period interaction). Figure 5.20 plots the team interaction for the different crew complements in the conventional and advanced plants across scenario periods. Figure 5.21 plots the change in team interaction of crews in the two plants from baseline conditions. The better team interaction ratings of the crews in the advanced plant can be seen in these figures. The crews in the conventional plant exhibited more variation in their team interaction over the course of the scenario, especially towards the latter portions of the scenario. Post hoc analyses confirm this. Analyses yielded significant differences in team interaction between scenario periods 2 and 3, and periods 2 and 4 for the minimum crew complement (p < 0.05, and p < 0.15, respectively). Significant differences were also found between periods 3 and 4 for the normal crew complement in the conventional plant (p < 0.10). Analyses also indicate a difference in team interaction between the normal and minimum sized crews in the conventional plant during scenario period 3 (p < 0.001) and scenario period 4 (p < 0.10). No significant differences were observed in team interaction across scenario periods or between either crew complement in the advanced plant.

Further analyses identified team interaction differences of crew complements during some scenario periods across plant types. Both crew complements in the advanced plant exhibited better team interaction during the first and third periods of the scenarios than the minimum crew complement in the conventional plant (p < 0.01 and p < 0.05, respectively). During the second scenario period the minimum crew complements in the advanced plant exhibited better team interaction than the minimum sized crews in the conventional plant (p < 0.05). The minimum sized crews in the advanced plant showed better team interaction than both crew complements in the conventional plant during the fourth period of the scenarios (p < 0.001).

2 3 Conventional Plant

4

Sz normal crew complement 1 minimum crew complement

1 2 3 Advanced Plant

4

Figure 5.20 Plot of team interaction by crew size, plant type, and scenario period

47 NUREG/TA-0137

6.0

5.5

0 5.0 0

0

0

.4.5

4.0

3.51

period 3 period 4 period 2 period 3 Conventional Plant Advanced Plant

Figure 5.21 Plot of team interaction change from baseline

48

period 4

NUREG/IA-0137

RESULTS

8

6

4

2

0

S-2

C U

-4

-8

-10

... .. ..... ... ........ ........... } .. . - -*-*------- - -~-*----------

S . . .> . . ..... . .... ...... . . . . . . . . . . t . . . . - - .

-- -- --- --- -- --- --- 0

period 2

----------------------- -- -- ---

S.................. ....... i .. .. ... ........... ......

0

/ Normaicrew .omplement

S... , Minimum crew compleren .........

DISCUSSION

6. DISCUSSION

6.1 General Trends

General trends in the performance measures of situation awareness, workload, and team interaction were observed and merit consideration. The findings demonstrate the effect of large plant disturbances on the performance of operating crews. Over the course of these scenarios, situation awareness varied significantly: following a disturbance, crews, on average experience a marked loss in situation awareness, i.e., approximately a 20 percent loss of baseline SA. Crews gradually regain SA though this recovery does not approach the level of SA found in pre-disturbance conditions. In some of the longer scenarios that included five data collection periods, SA dropped in the last period. Thus, the recovery of SA, both in terms of the amount and rate of recovery, may be affected by scenario-specific factors such as length, complexity and task demand. Although the differences in SA between the last and preceding period were not statistically significant, the trend by itself is noteworthy. Crews may be required to make critical decisions after one or more hours following the initiating event in a real situation, and this requires, among other things, adequate situation awareness.

The technique used to measure situation awareness demonstrated sensitivity to differences in the type of scenario, (i.e., whether rule-based or knowledge-based). Crews possessed better SA in rule-based than knowledge-based scenarios on the whole. The results further show that transient-induced SA losses in rulebased scenarios may leave crews with approximately the same levels of SA during the most disturbed phases as in knowledge-based scenarios. This demonstrates, again, that the loss and recovery of SA is not invariant. Rather, it appears to be subject to many things, such as: a crew's previous experience with similar types of scenarios; their ability to identify procedural information to assist in stabilizing the plant; whether they direct their attention to relevant information; and the rate of change in plant parameters.

On average, ratings of workload during transient mitigation were nearly double the ratings in baseline conditions. Most of the workload change increase occurs during the onset of the disturbance, and persists over the scenario. Even later in the scenario, after critical mitigation actions have been accomplished, crews experienced a high degree of workload as they brought the plant to a more stable, safer state. Thus, crews worked in a condition of sustained, high demand over the length of the scenarios in this study. Although each scenario was different, each produced similar workload and workload transition effects. This finding is not surprising since all of the scenarios used in this study were design basis events, requiring significant intervention to minimize challenges to the safety of the plant and, in some cases, the environment.

Team interaction also demonstrates the effects of these challenging and lengthy scenarios. During the initial phase of these scenarios that involved routine tasks, crews demonstrated a moderate degree of team interaction. The coupling between the primary and secondary sides of the nuclear power plant require this coordination and communication between crew members to balance the generation and transfer of energy while carrying out plant evolutions. Following a disturbance, crews mobilize by increasing communication, coordination, and emphasis on task requirements (i.e., establishing goals and objectives, discussing options, and decisions). Following this mobilization, team interaction subsides somewhat over the course of the scenario. In most situations, this is probably because the individual crew members focus on specific activities related to their area of responsibility (e.g., primary circuit) and interact to communicate instructions, information, and status as needed.

This trend merits consideration. While the demands of energy generation and transfer during normal activities require close coordination among crew members, there is likely to be more need during large plant upsets. Not all crews sustained team activities over the course of these scenarios. Perhaps this was due to task demands, design, or other factors. Nevertheless, team interaction is one of the ways that crews develop

49 NUREG/IA-0137

DISCUSSION

and maintain situation awareness, and is necessary to achieve effective performance in a nuclear power plant control room.

6.2 Effects of Crew Size

Although most other metrics did not demonstrate significant effects due to crew size alone, one aspect of transient management, cooldown and stabilization did. Control room crews are accustomed to working under different workload conditions, and learn to adapt their performance to match the demands of the operation being conducted. Under conditions of elevated workload, they, like skilled operators of other technological processes, may be able to perform for a period with no notable degradation in performance. The effect of workload may evidence itself not only early in a transient as workload transition, but also later as a cumulative effect. This may explain the difference between the different crew complements in carrying out cooldown and stabilization tasks, since no differences between the two crew complements were observed in their performance of other mitigation tasks. The minimum and normal crew complements both identified the correct procedural activities to mitigate the transients. However, the cooldown and plant stabilization activities were carried out under conditions of workload that were sustained typically for an hour or more. The absence of some crew member(s) may have affected the smaller crews' abilities to bring the plant to safe shutdown either in as timely a manner or within technical constraints (i.e., while maintaining boiling margins, pressure differentials between systems, etc.).

Minimum sized crews experienced more workload over the course of scenarios than normal sized crews. Further analyses revealed that the control room supervisor experienced the bulk of the additional workload. Crew members are resources available to the control room supervisor. Their absence results in a significant increase in activities that he is either not able to allocate to others, or which require additional planning and interaction on his part to accomplish. This may interfere with the effective completion of the control room supervisor's tasks, coordination of resources, or simply place the control room supervisor in a sustained situation of higher workload.

6.3 Effects of Plant Type

In all aspects of transient management, crews in the advanced plant setting performed better than crews in the conventional plant, particularly so for transient mitigation and cooldown and stabilization tasks. These findings indicate that although crews in the advanced plant experienced significantly higher workload than crews in the conventional plant, it did not exceed a threshold beyond which performance degraded.

All of the operators in the study experienced an increase in workload extending across scenario periods. By the fourth scenario period, all crew members in the advanced plant setting experienced significantly higher workload than their counterparts in the conventional plant setting. During debriefings, operators indicated that they had some difficulties in accomplishing their work using the different automated systems in HAMMLAB. Their relative unfamiliarity with the interfaces and automated systems in HAMMLAB, despite the training they received, may have affected their workload. This seems likely, since the largest demonstrated differences in workload were found among the RO and BOP operators between the two plant settings. These are the two crew members who interact most directly with the plant. Hence, some of the difference in workload observed between the conventional and advanced plant is probably related to the additional effort expended by operators to accomplish their tasks using different interfaces and automated systems.

Approximately two to three days training was provided to crews coming to Halden to participate in the advanced plant phase of the study. During that time, each crew learned how to interact with the advanced I&C systems in the simulator. Baseline measures of workload showed that crews in both plant settings experience roughly equivalent levels of workload. However, this did not hold true during the transient

50 NUREG/IA-0137

DISCUSSION

phases of scenarios. Then, there was evidence of differences between HAMMLAB and their own plant where they have developed their skills and habits in operating a nuclear power plant. Crews experienced more workload in the advanced plant over all disturbance phases. Moreover, their experience of workload transition was greater and did not level off later in the scenarios as did the workload of crews in the conventional plant. A question remaining is whether these differences in workload are temporary and would, over time and trials, subside. This question is beyond the scope of the present study.

A further concern was identified with the workload of the dual role control room supervisor/operator. In the advanced plant setting, the control room supervisor also served as one of the operators. The study design included conditions in which this dual role CRS served as the RO in one crew and the BOP operator in the other. Comparisons of his workload with that of other operators in the study showed that the dual role CRS experienced much higher workload. This includes CRSs in the conventional plant. During the debriefings, both of the licensed shift supervisors who served as dual role CRSs expressed concerns about working in this capacity. The shift supervisors expressed concerns about: being able to maintain an overview of the entire operation and its goals while being in the midst of the mitigation efforts; demands of communication with external personnel (e.g., local and national authorities, plant management and the emergency organization); and occasions in which the shift supervisor was left alone in the control room while the other licensed operator went to check something in the plant.

One of the concerns associated with the finding of higher workload in the advanced plant is the effect this may have on crews' abilities to manage the demands of the disturbance. Crews in the advanced plant, however, demonstrated better performance than crews in the conventional plant. Two separate measures of task performance were used in this study. One was a subjective evaluation based on criteria suggested by licensed operator examiners and used previously in other studies. The other was a method that divided transient management into distinct activities, each having tasks and goals that were more objectively evaluated. On measures of rated crew performance, the minimum sized crews in the advanced plant demonstrated the best performance. These crews also experienced the greatest workload levels in the study.

Crews in the advanced plant setting exhibited better team interaction than crews in the conventional plant. These differences extended across all scenario periods, during both normal and disturbance phases of scenarios. The averaged plot of team interaction of all crews across scenario periods (Figure 5.17) shows a drop in team performance following mobilization. However, the plot of team interaction across scenario periods in both plant settings (Figure 5.19) shows that this must be due to lower ratings of crews' team performance in the conventional plant. Crews in the advanced plant, on average, demonstrated more stable team interaction. They also showed no marked drop in performance later in the scenarios. This effect was modified by crew size and will be discussed further in the discussion of interactions.

One explanation for the better and more consistent team interaction of crews in the advanced plant relates to design differences between the conventional and advanced plant settings used in this study. Many conventional plants have relatively large control rooms in which operators carry out activities. The division of labor in NPPs typically dictates that the RO and BOP operators work on different, though complementary activities (i.e., different tasks, same goals). Depending on the design of the control room, this could place the operators at some distance from one another. The advanced plant complement used in this study embodied a wrap-around design in which each operator sat close to the other around a central overview and SPDS. Each operator could see what the other did and what displays were used. The control room supervisor in the advanced plant setting often left his workstation to sit between and a little behind the RO and BOP operators. He stayed very close to them for much, if not all, of the scenario. The crews in the advanced plant also used the common overview display as a focus of their discussions, both during the initial phases of the disturbance and during recovery. This specific type of interaction did not occur in the conventional plant due to the absence of such a display.

All of the crews in this study experienced situations of intense workload transition. During this transition, crews also experienced a loss of situation awareness. This can mean that some aspects of decision making are likely to be impaired. Symptom-based emergency operating procedures represent an advance in the prescriptive guidance provided to control room crews since they require much less determination of causes

51 NUREG/IA-0137

DISCUSSION

to be applied. However, operators actively engage in re-establishing and maintaining their SA following disturbances, and SA is itself an important element in using EOPs. Operators in the advanced plant, on average, possessed higher SA than crews in the conventional plant, though this was shown to depend on crew size.

6.4 Interactions of Crew Size and Plant Type and Effects on Operator Performance

Some of the measures used to assess the effects of crew performance in normal and minimum staffing complements in the two plant types revealed significant interactions (i.e., situation awareness, team interaction, rated crew performance). In these cases this means that the effect of crew size and plant type cannot be considered separately. The differences in performances on these measures were due, rather, to a combination of the two factors.

Rated crew performance on mitigation tasks nearly mirror the interaction of team interaction, discussed later. In the conventional plant, the normal sized crews demonstrated better performance on task-oriented performance measures than the minimum sized crews. The minimum sized crews in the conventional plant received the lowest ratings on these performance measures of all the crews. In the advanced plant setting, the normal sized crews demonstrated performance on these measures at levels equivalent to normal sized crews in the conventional plant. The minimum sized crews in the advanced plant, though, showed significantly better selection of mitigation paths, control of the plant, communication and confidence in their decision-making compared to minimum sized crews in the conventional plant. In fact, the smaller sized crews in the advanced plant performed slightly better than the normal sized crews in both plant settings. This difference, though, was not statistically significant.

Crews in the advanced plant setting achieved higher situation awareness on average than crews in the conventional plant. However, the difference between the situation awareness of the minimum sized crews in the conventional and advanced plant settings contributed significantly to this result (see Figure 5.8). In the conventional plant, the minimum sized crews demonstrated lower SA than normal sized crews. In the advanced plant, the minimum sized crews possessed significantly higher SA than the normal sized crews. Hence, there were marked differences in the SA of the crew complements between the conventional and advanced plants. The normal sized crews in the conventional plant and the minimum sized crews in the advanced plant demonstrated roughly equivalent levels of SA (i.e., the differences between them were not statistically significant). Their SA was higher than that of the other crews.

All crews experienced a drop in SA following transient initiation. However, crews demonstrated different amounts of SA loss and rates of SA recovery. The normal sized crews in the conventional plant and the minimum sized crews in the advanced plant both experienced a drop in their SA following the onset of and during the peak of the disturbances. Over the course of the scenario, however, both groups regained their SA. Figures 5.6 and 5.7 show the change in SA of the crews from their pre-disturbance SA.

In contrast, the minimum sized crews in the conventional plant lost SA over the course of the scenario without recovery. These crews' SA, in fact, showed a down-turn at the last scenario period. The normal sized crews in the advanced plant, on the other hand, recovered the SA they lost during the most disturbed portions of the scenarios. They also possessed slightly better SA at the end of the scenarios than at the beginning. However, this group also experienced the largest loss of SA of all the crews. At it's lowest, their SA approached a level near which their assessments of plant performance (e.g., detections of change and direction, predictions into the future, etc.) were only slightly better than chance.

The differences in the SA of the normal sized crews between the conventional and advanced plant settings was unexpected. HAMMLAB was designed for a somewhat smaller shift complement, though CRTs and other operator support systems were provided for each crew member in this study commensurate with the plans suggested by advanced plant vendors. However, the normal sized crews in the conventional plant, and the minimum sized crews in the advanced plant possessed higher SA than the other groups. Hence,

52 NUREG/IA-0137

DISCUSSION

there may be some design aspects in each of the control rooms that better support different crew staffing complements.

These results extend to performance on rule- and knowledge-based scenarios. Most crew complements demonstrated lower SA on knowledge-based scenarios than rule-based scenarios. The exceptions to this are the minimum sized crews in the advanced plant who demonstrated equivalent levels of SA in both types of scenarios.

Maintaining situation awareness is an integral part of operator performance, during both simulated and actual emergencies. Situation awareness is needed by crew members to evaluate progress in procedurallydirected activities, to evaluate different alternatives in the various planning phases that occur during transient mitigation, to establish appropriate performance goals, and to use as input to decision making. Situation awareness of the control room crew may also influence the quality of information provided to external agencies. During an emergency, crews must direct not only on-site activities towards mitigating the disturbance but also communicate and provide information to local and federal authorities. These agencies use the information provided by the plant as part of the basis for making decisions about important collateral activities. This includes the activation of emergency contingencies, public evacuation, dose rate or source term prediction (in the case of the most serious accidents). Some of this information may come directly to agencies via data links with the plant where such links exist. Some information will almost certainly come from the verbal reports by control room crew members. In some phases of a design basis event, reported information may be subject to large uncertainties depending on whether it taps into SArelated information or not. The findings of this study show that control room crew staffing and plant design features may contribute to such uncertainties.

The largest differences in the ratings of team interaction were found between the minimum sized crews in the conventional and advanced plant settings. In the conventional plant, the minimum sized crews demonstrated fewer team interaction characteristics than the normal sized crews. In the advanced plant, though, both the normal sized crews and the minimum sized crews exhibited equivalent team interaction characteristics.

The trends in the team interaction over scenario periods are also relevant. Crews in the conventional plant demonstrated more variability in their team interaction than crews in the advanced plant. The minimum sized crews in the conventional plant demonstrated more variability in team interaction during the latter portions of the scenarios, and exhibited poorer team performance than other crews in the conventional plant. The differences in team interaction compared to baseline further underscore these results. Figure 5.21, a plot of the change in team interaction from baseline, shows that both the normal and minimum crews in the conventional plant mobilized to about the same extent. Following this mobilization, the minimum sized crews showed a significant drop in team interaction. They also showed little improvement at the end of the scenario. The normal sized crews demonstrated a slight increase in team interaction following mobilization, but did not maintain team performance throughout the scenario. This was shown by a drop in team interaction during the last phase of the disturbance. In contrast, both crew complements in the advanced plant exhibited similar team interaction. Their team interaction was better than in the conventional plant, and exhibited less variability across periods of the scenarios. On average, crews in the advanced plant maintained team interaction at about the same levels during the scenarios

An objective of control room design is to support the performance of individual operators as a team. Design features should be pursued that lead to improvements in crew coordination, and in the availability and presentation of information needed to support crew task performance and decision making. An aim of such design improvements should be to achieve levels of team interaction that result in acceptable performance. Such performance characteristics can be considered in relative terms (i.e., compared to another operating environment) as well as in terms of the consistency crews demonstrate in their performance. The findings of the study show that performance as a team is affected by both crew size and control room design features. Crews interacted better as a team in the advanced plant setting, especially the smaller sized crews who demonstrated poorer performance in the conventional plant setting. Variability between the different crew complements was reduced in the advanced plant setting. The interaction of

53 NUREG/IA-0137

DISCUSSION

effects, thus, appears to have more to do with equalizing the performance between the different crew complements (i.e., minimizing differences between them). Moreover, the levels of performance on these measures were, in the advanced plant, higher than in the conventional plant. In this respect, the interactions indicate that some design objectives were better achieved in the advanced plant, evidenced by the interaction and performance of the crews.

54 NUREG/IA-0137

INSIGHTS

7. INSIGHTS

This study involved the analysis of crew performance in normal and minimum crew staffing complements, carried out in simulator settings that were made to be representative of both conventional and advanced passive plants. Modifications were made to the plant model and part of the instrumentation and control systems at the Loviisa simulator prior to data collection to provide a control room environment more similar to U.S. plants. In HAMMLAB, the instrumentation, control, and automated systems were configured to produce a work environment similar to those that operators in advanced passive plants are expected to encounter. Results obtained on all of the performance measures used in this study revealed a number of effects of crew size, plant type, and their interaction.

In considering the effects of crew size and plant type on performance measures, several issues concerning the validity of the study itself should be mentioned. Simulator studies are sometimes criticized because they take place outside the real control room, involve scenarios that crews are highly trained on, don't include off-site notifications and communications with outside control room staff, or performance of activities outside the control room. Simulator studies, however, represent the best approximation of the actual control room environment, and are the environment in which crew performance is tested as part of licensing and requalification. The scenarios used in this study represented a broad range of design basis events, involving scenarios for which crews were well trained, and some for which they had less training. In addition, all necessary notification and communication tasks with external personnel, whether on-site or off-site, were included in the scope of these scenarios to maintain a high degree of realism in the scenarios. Feedback from crew members and training personnel experienced with observing crews supports the conclusion that the scenarios contained a high degree of realism and challenge.

Considering analyses of the task performance, situation awareness, subjective workload, and team interaction measures, these scenarios were deemed to challenge the resources of the crews to mobilize and mitigate the disturbances. Situation awareness, subjective workload, and team interaction all demonstrated the effects of transitions from normal to abnormal operating conditions. The effects were further associated with the crew size and plant settings used in this study. Simple effects of these factors on operator, crew, or plant performance were observed in some cases. In many cases, though, significant effects were observed by the combination of these factors.

The findings of this study, thus, demonstrate significant effects of control room crew staffing and plant technology on crew performance. Some of these are in accord with the expectations of previous research into human performance with advanced technologies (i.e., better situation awareness). Some were not (i.e., lower SA of normal sized crews in advanced plant relative to that of same-sized crews in the conventional plant). Though improvements occurred in many aspects of performance based on differences between conventional and advanced plant features, the improvements did not extend equally to the different crew complements.

The findings underscore the benefit of using a broad range of performance measures to evaluate issues that involve human performance, such as control room staffing and design issues. Were one to consider the differences in workload alone between the two plant types and staffing complements, then perhaps the current minimum staffing complement in a conventional plant might have appeared adequate based on statistically significant differences alone. However, differences in team interaction, situation awareness, and task performance in the advanced plant underscore the importance of using both subjective and objective performance measures in evaluating potential changes that affect aspects of the work environment and that may, therefore, affect the performance of the crew.

The designs of new nuclear steam supply systems that incorporate passive system performance, together with new control room technologies offer the possibility to automate much of what is done manually today.

55 NUREG/IA-0137

INSIGHTS

However to the extent that this is done, consideration must be given to the best methods of involving nuclear power plant operators in the control of the process. The reasons for this are threefold, at least. Firstly, the operator serves as both a first and last line of defense in case of automation failure, and may, perhaps, sometimes be an initiator of it. Maintaining an awareness of the process is the most important part of the operator's role, and is the basis for many important control room decisions and actions. Maintaining an awareness of the plant may become more difficult if the operator becomes removed from the process control loop through automation of operator activities.

Secondly, inappropriate function allocation may produce a set of activities that may be difficult or errorprone. This may be the result of allocating to machines those functions that are easiest in terms of engineering approaches, and leaving those that are not easy to allocate to the control room crew. Not all of the vendor submittals that served as the basis of this study (which, admittedly are at different stages of completion) included the decision criteria used for allocation of control room functions. Hence, the final set of control room activities for which operators in advanced passive plants will be responsible is still uncertain.

Thirdly, even though many functions may be under automated system control, intervention by the operating crew must be anticipated. As discussed earlier, operators in advanced passive plants will carry out many activities similar to those in plants today, though the timing and methods for carrying them out may change considerably. The ability to intervene with automated systems is necessary if the crew is to serve as a backup to automation, and to initiate automation sequences (e.g., change from one mode of operation to another). To supervise automated systems, the crew must understand the goals of automation (e.g., programmed sequences) and how it functions (e.g., receive the appropriate feedback from the system). Else, operator intervention may be ill-timed or erroneous, based on incorrect assumptions, and knowledge of goals and means.

At the same time that changes in the design and operation of nuclear power plants are under consideration, some vendors also anticipate that a smaller control room complement may be sufficient to control the nuclear power production process. Some of the design factors that have led to proposed reductions in staffing relate to changes in allocation of functions (i.e., activities usually performed by operators allocated to machine systems) or introduction of passive systems. To date, none of the vendor submittals cite empirical studies or demonstrations of their systems that show the ability of crews to operate their systems as envisioned by the vendors. The findings of this study demonstrate that changes in the control room crew complement must take into account the balance of activities allocated to the control room crew for foreseeable circumstances. The findings also underscore the importance of adopting appropriate measures of operator performance. Such measures are needed to ensure that the demands imposed by the system are manageable by the control room crew and do not have a negative impact on important aspects of crew and plant performance.

In summary, design factors influenced the performance of crews in this study. Crews in the conventional plant normal staffing complement exhibited better performance than minimum sized crews. Such plants have been designed with a larger control room staffing complement in mind. The functions, allocations, size, and automation in such control rooms require more crew members to maintain control of the plant. In the advanced plant, the minimum sized crews performed better than the normal sized crews. The advanced plant control room setting was more compact, designed for a smaller control room complement. It made greater use of computer-based technologies and automation than the conventional plant, including passive design features and displays that better integrate control room information. Therefore, decisions about control room staffing should be based upon design features including function allocation, automation, integration, and plant-specific characteristics (e.g., passive system performance). Validation and verification using measures of operator and crew performance are necessary to determine the staffing complement needed to operate the plant.

56 NUIREG/IA-0137

REFERENCES

8. REFERENCES

"Policy Statement on Engineering Expertise on Shift," Federal Register (50 FR 53621), October 21, 1985.

10 CFR 50.54(m), "Energy: Conditions of Licensees," Code of Federal Regulations, Washington, D.C., January 1, 1992.

ABB Atom Preliminary Safety Information Document (PSID). PIUS: The PIUS Arrangement Principle for a Simplified Passive Advanced Light Water Reactor Plant., December, 1989.

Adams, M.J., Tenney, Y.J., & Pew, R.W. "Situation awareness and the cognitive management of complex systems." Human Factors, 37 (1), pp. 85-104, 1995.

American Institute of Aeronautics and Astronautics. Guide to Human Performance Measurement. American National Standards Institute. BSRIAIAA G-035-1992.

Atomic Energy of Canada Limited, CANDU Operations. The CANDU 3 Technical Description. Technical Report 74-01371-TED-001, Rev. 3, September 15, 1989.

Atomic Energy of Canada Limited, CANDU Operations. The CANDU 3 Technical Outline. Technical Report 74-01010-TED-001, Rev. 9, September 15, 1989.

Baker, D.P. & Salas, E. Principles for measuring teamwork skills. Human Factors. 34(4). pp. 469-475, 1992.

Bergstroem, B. Task demand, workload, and performance. National Defense Research Establishment Department for Human Studies. FOA report A 50016-5.2. ISSN 0281-0239, 1993.

Carmody, M.A., & Gluckman, J.P. "Task-specific effects of automation and automation failure on performance, workload and situational awareness." In R.S. Jensen and D. Neumeister (Eds.), Proceedings of the Seventh International Symposium on Aviation Psychology (pp. 167-17 1). Columbus, OH: Ohio State University, 1993.

Casali, J.G. & Wierwille, W.W. "A comparison of rating scale, secondary-task, physiological, and primarytask workload estimation techniques in a simulated flight task emphasizing communications load." Human Factors. 25(6). pp. 623-64, 1993.

Cooper, G.E., White, M.D. & Lauber, J.K. (Eds.). Resource Management on the flight deck. NASA Conference Publication 2120, Moffet Field, CA: NASA-Ames Research Center, 1979.

Coury, B. G., & Terranova, M. "Collaborative Decision Making in Dynamic Systems." In Proceedings of the Human Factors Society 35th Annual Meeting, September 2-6, San Francisco, CA, 1991.

Endsley, M.R. "Toward a theory of situation awareness in dynamic systems." Human Factors, 37(1), pp. 32-64, 1995.

Endsley, M.R. "Design and evaluation for situation awareness enhancement." Proceedings of the Human Factors Society 32nd Annual Meeting. pp. 97-101, 1988-A.

Endsley, M.R. "Situation awareness global assessment technique (SAGAT)." Proceedings of the IEEE National Aerospace and Electronics Conference, NAECON. Dayton, Ohio. pp. 789-795, 1988-B.

Endsley, M.R. "Pilot situation awareness: the challenge for the training community." Proceedings of the 11th Interservice llndustry Training Systems Conference. November, 1989. pp. 111-117, 1989.

57 NUREG/IA-0137

REFERENCES

Endsley, M.R. "Predictive utility of an objective measure of situation awareness." Proceedings of the Human Factors Society 34th Annual Meeting. pp.4 1-4 5 , 1990.

Endsley, M.R. "A survey of situation awareness requirements in air-to-air combat fighters." The International Journal of Aviation Psychology. 3(2). pp. 157-168, 1993-A.

Endsley, M.R. Towards a Theory of Situation Awareness in Dynamic Systems. TTU-IE-93-02. Lubbock, TX: Texas Tech University, Department of Industrial Engineering (internal technical report), 1993-B.

Endsley, M.R. Measurement of Situation Awareness in Dynamic Systems. TTU-IE-94-04. Lubbock, TX: Texas Tech University, Department of Industrial Engineering (internal technical report), 1994.

Endsley, M.R. "Predictive Utility of an Objective Measure of Situation Awareness." In Proceedings of the Human Factors Society 34th Annual Meeting, pp. 41-45, 1992.

Endsley, M.R. "The application of human factors to the development of expert systems for advanced cockpits." Proceedings of the Human Factors Society, 31st Annual Meeting. pp. 1388-1392, 1987.

Endsley, M.R. & Bolstad, C.A. "Human capabilities and limitations in situation awareness. Individual differences in pilot situation awareness." International Journal of Aviation Psychology. pp. 19-1 - 1910, 1992.

Ericsson, K.A. & Simon, H.A. Protocol Analysis: Verbal Reports as Data. London, England: The MIT Press, 1984.

Fontenelle, G. & Laughery, K.R. "A workload assessment aid for human engineering design." Proceedings of the Human Factors Society - 32nd Annual Meeting, 1988.

Foushee, H.C. "Dyads and triads at 35,000 feet: factors affecting group process and aircrew performance." American Psychologist. 39(8). pp. 885-893, 1984.

Foushee, H.C., & Helmreich, R.L. Group Interaction and Flight Crew Performance. in Human Factors in Aviation, E. Wiener & D. Nagel (eds.), New York, NY: Academic Press, 1988.

Fracker, M.L. "A theory of situation assessment: Implications for situation awareness." Proceedings of the Human Factors Society 32nd Annual Meeting. pp.102-106, 1988.

Fracker, M.L. "Attention gradients in situation awareness." NATO AGARD Aerospace Medical Panel Symposium. pp. 6-1 - 6-10, 1989.

GE Nuclear Energy. SBWR Standard Safety Analysis Report. Report 25A5113 Rev. A, August 1992.

Glickman, A.S.; Zimmer, S.; Montero, R.C.; Guerette, P.J.; Campbell, W.J.; Morgan, B.B., Jr.; Salas, E. The Evolution of Teamwork Skills: An Empirical Assessment with Implications for Training. Technical Report 87-016. Prepared for the Office of Naval Research and Human Factors Division (Naval Training Systems Center). Orlando, FL, 1987.

Hahler, B.; Dahl, S.; Laughery, R.; Lockett, J.; & Thein, B. "Crewcut - A tool for modeling the effects of high workload on human performance." Proceedings of the Human Factors Society 35th Annual Meeting, pp. 1210-1214, 1991.

Hallbert, B.P., Sebok, A.L., Haugset, K., Morisseau, D.S., & Persensky, J.L "Interim Results of the Study of Control Room Crew Staffing for Advanced Passive Reactor Plants." In Proceedings of the 23rd

58 NUREG/IA-0137

REFERENCES

Water Reactor Safety Meeting, U.S. Nuclear Regulatory Commission, October 23-25, 1995, Bethesda, Maryland.

Hanson, D.J., Meyer, O.R., Blackman, H.S., Nelson, W.R., & Hallbert, B.P. Evaluation of Operation Safety at Babcock and Wilcox Plants Volume I - Results Overview. NUREG/CR-4966, Volume 1, U.S. Nuclear Regulatory Commission, Washington, D.C., October 1987.

Hart, S.G. & Hauser, J.R. (1987). "Inflight application of three pilot workload measurement techniques." Aviation, Space and Environmental Medicine, pp. 402-410 May, 1987.

Hendy, K.C.; Hamilton, K.M.; & Landry, L.N. "Measuring subjective workload: When is one scale better than many?" Human Factors. 35(4). pp. 579-60, 1993.

Hill, S.G.; Iavecchia, H.P.; Byers, J.C.; Bittner, A.C.; Zaklad, A.L.; Christ, R.E. "Comparison of four subjective workload rating scales." Human Factors. 34(4). pp. 429-439, 1992.

Hogg, D.N.; Folles0, K.; Torralba, B.; & Volden, F.S. "Development of a situation awareness measure to evaluate advanced alarm systems in nuclear power plant control rooms." Ergonomics. 38(11), pp. 2394-2413, 1995.

Hooper, K. N., Coury, B.G., & Terranova, M. "Team Performance in Dynamic Systems." Proceedings of the Human Factors Society 3 5Ih Annual Meeting, September 2-6, San Francisco, CA, 1991.

Huey, B.M. & Wickens, C.D. (Eds.). Workload Transition: Implications for Individual and Team Performance. Washington, DC: National Academy Press, 1993.

Idaszak, J.R. "Human operators in automated systems: The impact of active participation and communication." Proceedings of the Human Factors Society 33rd Annual Meeting. pp. 778-782, 1989.

James, L.R.; Demaree, R.G.; & Wolf, G. "Estimating within-group interrater reliability with and without response bias." Journal of Applied Psychology. 69(1). pp. 85-98, 1984.

Kantowitz, B.H. & Casper, P.A. "Human workload in aviation." In Human Factors in Aviation (Ed. Weiner, E. &Nagel, D.) San Diego, CA: Academic Press. pp.157-187, 1988.

Kibbe, M.P. "Information transfer from intelligent EW displays." Proceedings of the Human Factors Society 32nd Annual Meeting. pp.107-110, 1988.

Komaki, J.L.; Desselles, M.L.; & Bowman, E.D. "Definitely not a breeze: Extending an operant model of effective supervision to teams." Journal of Applied Psychology. Vol. 74, No. 3, pp. 522-529, 1989.

Meister, D. Behavioral Analysis and Measurement Methods. New York: John Wiley and Sons, Inc, 1985.

Melber, B.; Roussel, A.; Baker, K.; Durbin, N.; Hunt, P.; & Hauth, J. Staffing decision processes and issues: Case studies of seven US Nuclear Power Plants. NUREG/CR-6122. Washington, DC: Nuclear Regulatory Commission, 1994.

Montgomery, J.; Gaddy, C.; & Toquam, J. "Team interaction skills evaluation criteria for nuclear power plan control room operators." Proceedings of the Human Factors Society 35th Annual Meeting. pp. 918-922, 1991.

Montgomery, J.C., & Hauth, J.T. Team Interaction Skills Evaluation Criteria for Nuclear Power Plant Control Room Operators. NUREG/CP-0 114, Vol. 1, 1991.

59 NUREGAIA-0137

REFERENCES

Montgomery, J.C.; Hauth, J.T. "Team interaction skills evaluation criteria for nuclear power plant control room operators." Proceedings of the U.S. Nuclear Regulator Commission Eighteenth Water Reactor Safety Information Meeting. U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory Research, 1992.

Moray, N. "Subjective mental workload." Human Factors. 24(1). pp. 25-40, 1982.

Morgan, B.B., Jr.; Glickman, A.S.; Woodard, E.A.; Salas, E.; Blaiwes, A. Measurement of Team Behaviors in a Navy Training Environment. Technical Report 86-014. Prepared for the Office of Naval Research and Human Factors Division (Naval Training Systems Center). Orlando, FL, 1986.

Moroney, W.F., Biers, D.W., Eggemeier, F.T, & Mitchell, F.A. "A Comparison of Two Scoring Procedures with the NASA Task Load Index in a Simulated Flight Task." IEEE Proceedings, pp. 734740, 1992.

Mosier, K.L. & Chidester, T.R. "Situation Assessment and Situation Awareness in a Team Setting." In Y. Queinnec & F. Daniellou (Eds.), Designing for Everyone. London: Taylor and Francis. pp. 798-800, 1991.

Murphy, K.R.; Martin, C. & Garcia, M. "Do Behavioral Observation Scales Measure Observation?" Journal of Applied Psychology. Vol. 67, No. 5, 562-567, 1982.

NUREG-0660. NRC action plan developed as a result of the TMI-2 accident, Volume 1. Washington, D.C.: U.S. Nuclear Regulatory Commission, 1980.

NUREG-0737. Requirements for emergency response capability, supplement 1 (Generic Letter 82-33). Washington, D.C.: U.S. Nuclear Regulatory Commission, 1982.

NUREG-0799. Draft criteria for preparation of emergency operating procedures. Washington, D.C.: U.S. Nuclear Regulatory Commission, 1981.

NUREG-0899. Guidelines for the preparation of emergency operating procedures. Washington, D.C.: U.S. Nuclear Regulatory Commission, 1982.

Nygren, T.E. "Psychometric properties of subjective workload measurement techniques: implications for their use in the assessment of perceived mental workload." Human Factors, 33(1), pp. 17-31, 1991.

O'Hara, J.M. "Advanced human-system interface design review guidelines." Proceedings of the U.S. Nuclear Regulatory Commission Eighteenth Water Reactor Safety Information Meeting. Prepared by Brookhaven National Laboratory. Holiday Inn Crowne Plaza, Rockville, MD. October 22-24, 1990.

Olsen, L.M. "Operator modeling of a loss-of-pumping accident using MicroSAINT." Proceedings of the ANS Annual Meeting, Boston, 1992.

Parasuraman, R.; Molloy, R. & Singh, I.L. "Performance Consequences of automation-induced "complacency." International Journal of Aviation Psychology, 3(1), 1-23, 1993.

Regal, D.M.; Rogers, W.H.; Boucek, G.P., Jr. "Situational awareness in the commercial flight deck: definition, measurement, and enhancement." SAE Technical Paper Series: Aerospace Technological Conference and Exposition. Anaheim Convention Center, California October 3-6, 1988. pp. 1-5, 1988.

60 NLTREG/IA-0137

REFERENCES

Roth, E.M., Mumaw, R.J., & Lewis, P.M. An empirical investigation of operator performance in cognitively demanding simulated emergencies. NUREG/CR-6208, U.S. Nuclear Regulatory Commission, Washington, D.C., July 1994.

Sanders, M.S. & McCormick, E.J. Human Factors in Engineering and Design, Sixth Edition. London: McGraw-Hill Book Company, 1987.

Sarter, N.B., & Woods, D.D. "How in the world did we ever get into that mode? Mode error and awareness in supervisory control." Human Factors, 37(11), pp. 5-19, 1995.

SAS Institute, Inc. SAS User's Guide: Statistics. SAS Institute, Inc., Box 8000, Cary, North Carolina, 1982.

Salas, E., Prince, C., Baker, D.P., & Shrestha, L. "Situation awareness in team performance: Implications for measurement and training." Human Factors, 37(1), pp. 123-136, 1995.

Sebok, A.L., Hallbert, B.P. & Morisseau, D.S. "Crew Performance Issues in Nuclear Power Plant Process Disturbances." Proceedings of the 1996 American Nuclear Society International Topical Meeting on Nuclear Plant Instrumentation, Control, and Human-Machine Interface Technologies (NPIC & HMIT). Pennsylvania State University, State College, Pennsylvania. pp. 645-651, 1996.

Selcon, S.J. & Taylor, R.M. "Evaluation of the situational awareness rating technique (SART) as a tool for aircrew systems design." NATO AGARD Aerospace Medical Panel Symposium. pp. 5-1 - 5-8, 1989.

Selcon, S.J.; Taylor, R.M.; & Koritsas, E. "Workload or situational awareness?: TLX vs. SART for aerospace systems design evaluation." Proceedings of the Human Factors Society 35th Annual Meeting. pp. 62-66, 1991.

Spjotvoll, E., & Stoline, M.R. "An Extension of the T-Method of Multiple Comparison to Include the Cases with Unequal Sample Sizes." Journal of the American Statistical Association, 68, pp. 976-978, 1973.

Spurgin, A.J. & Beveridge, R.L. "A model of a control-room crew." Power Plant Dynamics, Control and Testing. pp. 69.01-69.11, 1986.

Tsang, P.S. & Johnson, W.W. "Cognitive demands in automation." Aviation, Space, and Environmental Medicine. February, 1989. pp. 130-135, 1989.

Vidulich, M.A. & Hughes, E.R. "Testing a subjective metric of situation awareness." Proceedings of the Human Factors Society 35th Annual Meeting. pp. 1307-1311, 1991.

Westinghouse Electric Corporation. AP600 Standard Safety Analysis Report, Rev. 0, Simplified Passive Advanced Light Water Reactor Plant Program Report DE-AC03-90SF18495, Prepared for U.S. Department of Energy, San Francisco Operations Office, June 16, 1992.

Wheatley, P.D.; Davis, C.B.; Callow, R.A.; Fletcher, C.D.; Dobbe, C.A.; & Beelman, R.J. (1987). Evaluation of Operational Safety at Babcock and Wilcox Plants, Volume 2 - Thermal-Hydraulic Results. NUREG/CR-4966, EGG-2515, U.S. Nuclear Regulatory Commission, Washington, D:C:, November, 1987.

Wierwille, W.W. & Eggemeier, F.T. "Recommendations for mental workload measurement in a test and evaluation environment." Human Factors. 35(2). pp. 263-281, 1993.

61 NUREG/IA-0137

REFERENCES

Wiener, E.L., & Curry, R.E. "Flight deck automation: Promises and problems." Ergonomics, 23, pp. 9951101, 1980.

Winer, B.J. Statistical Principles in Experimental Design (2nd Edition). McGraw-Hill Book Company: New York, 1971.

62 NUREGAIA-0137

APPENDIX A: SCENARIO DESCRIPTIONS

A-1NUREG/IA-0137


A.1 Scenario 1: Steam Generator Tube Rupture With A Fire In The Turbine Hall And A Stuck Open Steam Generator Safety Relief Valve.

A.1.1 Synopsis

The scenario begins after a shift turnover. A control rod has inadvertently dropped earlier. An electrical fault has been repaired and main control room (MCR) staff will start returning the plant to full power.

Turbine oil separation had been started before the shift change, so the level of the oil tank is rising slightly.

The only symptom of the leak in the oil tank, apparent to operators in the MCR after separation is complete, is that the oil level of the main oil tank is not rising. After pumping is stopped, a decreasing oil level

indicates the leak. A Field Operator (FO) will inform the MCR of the fire. Very soon after getting

information about the leak, operators will probably trip the turbine.

The fire in the turbine hall will be initiated simultaneously with the turbine trip (due to electric spark and oil

fumes). The oil leak will continue until the turbine is tripped. Thereafter the scenario continued as a small

turbine hall fire. In the event of a turbine hall fire, the reactor should be scrammed according to procedure.

Just after the scram a SGTR (5 kg/s) will occur increasing to 35 kg/s ten minutes later. All indicators work normally, and the crew should have no difficulty to diagnose this transient.

Subsequent to the steam generator leakage, a steam generator safety relief valve will fail open, after cycling open and closed a few times. This results in an event that leads to an unfiltered release of primary circuit radiation to the atmosphere.

A-2 NUREG/IA-0137


A.1.2 Probable course of events

Note: Entries preceded by an * symbol indicate planned malfunctions implemented by the simulator instructor.

Time (min.) 34 Description

Initial condition (IC):

Smaller steam generators; secondary volume is 1/3 nominal size (10 nP each). Measurement fault in SC1OL003 gradient, its parameter +3 mmn/nin.

Power level is about 90 % due to control rod drop. MCR staff start withdrawal of the rod and increasing the output to full power over 1 1/2 hours.

(11: 00 p.m. simulated time of day)

0 min. Low level alarm of the main oil tank < 0 mm. It is between 10-50 mm below normal level. An alarm has been active for some hours but this is normal during oil separation.

1 Oil separation is going on (during IC), and normally increases the level of oil tank, +3 mm/min. This task is conducted by the FO. When oil tank is full, pumping will be regenerated back to main oil tank from leaking oil tank at a rate of +50 I/min. over a 15 minute period.

5 *A small control oil pipe (return oil line) under turbine 1 breaks. Leaking oil drops on a steam line of super pre-heater and a portion of it will vaporize.

Oil leak initiation, SCIOL03 = 0 mnmi/min. Level will not rise because of the leak, nor will it decrease. The leakage is masked by simultaneous oil pumping from the oil separator which pumps purified oil back to the main oil tank compensating for the level decrease

Leak onto the floor is at a rate of between 40-50 /mrin.

Low level alarm does not disappear despite oil pumping back to tank. MCR staff may detect this and suspect something strange.

7 Separation is over and the leak will now start decreasing the level of the oil tank at a rate of -3 mra/min.

8 Pause for data collection

3 Times may be different between LOTS and HAMMLAB due to plant performance differences, e.g., emulation of passive system performance.

4 Times for data collection are approximate and, in some cases, depend on a crew activity

A-3 NUREG/IA-0137


11 Alarm in MCR: "SC 10L803 (gradient) -3 > -1 mm/min."

12 BOP operator probably asks FO: "Is there something unusual in oil separation?", and will get an answer: "Everything is OK."

14 After the first low level alarm of the oil tank BOP operator will ask the field operator to check the turbine hall for leakages.

16 FO finds the leak and reports the situation like: 'There is a rather big leak under turbine 1." It is very difficult for FO to determine which of several control oil pipes is leaking.

17 CRS gives oil fire fighting alarm to the fire brigade.

18 CRS orders trip of the Number 1 turbine.

18 Turbine I tripped.

19 Oil fumes will catch fire in the basement. A small oil fire starts in the turbine hall on level +3 - 0 m.

20 alarm *Automatic fire alarm.

20 Oil leak is terminated by turbine trip

20 FO informs BOP operator by radio telephone about the fire.

21 CRS performs routine procedures for activating fire brigade.

23 CRS probably orders RO to trip the reactor according to the EOP, "Fire in turbine hall".

23 Turbine 2 trips automatically due to reactor trip.

24 CRS should now use symptom based "General Emergency Operating Procedure" (GEOP) to make post trip checks.

24 *SGTR, parameter 0.54 cm2 activation in SG2 (YB52WO0). Just after the scram SGTR (5 kg/s) will occur.

24 RL72S03 closed (automatic on high level).

25 SC1OL803 (gradient) LG <-1 mm/min. alarm off.

25 Extra makeup pump (2 of 3)started.

26 Extra makeup pump (3 of 3) started.

30 Fire is over. MCR is informed by fire brigade.


34 *SGTR leak increases to 4 cm 2, 35 kg/s.

A-4 NUREG/IA-0137


34 alarm Radioactivity in evacuation pumps of condensers.

34 alarm RA52R001, high radiation 0.637 > 0.01 mSv/h

35 HPSI on

36 The staff will probably transition to event based EOP, "Steam Generator Tube Rupture."

36 Stop of RCP, YD12D001.

36 *An attempt to close the main primary gate valves is unsuccessful due to YA12SO01 torque failure, stuck open 70%.

36 Automatic closure of SG steam and feedwater isolation valves in affected SG (activated on high SG level).

37 Manual start of high volume boron injection.

38 Steam dumping to the condenser on both turbines; by-pass 2x26 % open due to reduced SG volumes in conventional plant.

38 Staff should start primary circuit depressurization and cooldown by PZR-spray from make-up-system to minimize the integrated break flow from primary to secondary circuit, and to environment.

38 Primary pressure decrease by opening PORV (1.3 kg/s).

40 *After automatic isolation of the affected steam line when the pressure rises and the SG safety valve had opened several times, the SG safety relief valve finally sticks open.

41 Manual override of steam isolation signal from 30 bars and change-over to gradient mode at 5 bars/min.

41 Containment general isolation due to low level of pressurizer (< 1.4 m) and low primary pressure (<100 bar). RCPs trip.


46 Steam dumping will probably be decreased to between 2-10% (valve position) due to low steam pressure. Stuck open safety relief valve causes high primary circuit cooldown rate.

50 Staff begin to maintain emergency cooling water from exceptional reserve pools (this activity is taken outside of MCR).

50 The second HPSI pump will probably be stopped.

50 Decision to try manually closing the sticking main gate valve.

51 Core cooling accumulators isolated manually.

52 Staff should start cooling of the feedwater tanks based on the temperature of the primary circuit.

A-5 NUREG/IA-0137


55 The third HPSI pump will probably be stopped.

55 Restart of intermediate cooling system (TF60) to facilitate the letdownflow.

56 Restart of service water system (VF62) to cool TF60.

56 Opening of the letdown isolation valves.

57 Let-down will probably be opened for controlling the level in the Pressurizer.

60 Core exit temperature is about 235°C, pressure 45 bars, PZR-level 8 m, steam pressure 10 bars.



90 Cooling down shall continue to cold shut-down.

A-6 NUREG/IA-0137


A.2 Scenario 2: Interfacing Systems Loss Of Coolant Accident: Primary Leakage Outside The Containment With Ensuing Control Room Instrument Malfunctions

A.2.1 Synopsis In initial conditions the turbines are operating at full power, PZR-level 4.6 m. An instrument line breaks in the instrumentation room (IR) in the reactor building basement. Rupture of the reference line in the pressurizer steam space means that a steam leak takes place as long as the PZR-level is under 5.3 meters; thereafter it is in liquid phase or theoretically as two phase flow. Leak flow was permanently set at 0.88 kg/s as steam. Additional demands on the MCR staff are caused by a defective measurement YD14FO1, measurement fault (i.e., RCP seal injection flow) with gradient -0.1 kg/s. This occurs because the steam leakage in the instrument room damages this instrument. This results in a spurious signal starting the emergency injection water pump, YD30D01, and tripping RCP YD14DO1. This incident will probably lead the staff to suspect a leak in RCP injection water system.

After all this process information the CRS or RO will send a FO to check the rooms in the reactor building basement to find the possible leak. The Loviisa staff will expect some symptoms (alarms) in the ventilation and floor drain well levels. The FO will pass the defected instrumentation room on the way to YD-injection water room on the lower levels. Some steam may penetrate through the door seals, which are not leak-tight. The assumption in this early phase is that the FO does not notice anything strange.

After the FO has checked the basement, the simulator instructor will call to the RO and report that he found nothing.

Simultaneously with the call from the FO, the MCR staff will receive a specific fire alarm from the instrumentation room. The same signal will automatically close the exit ventilation line. It will further increase temperature and the pressure rise in the instrument room resulting in visible steam penetration through the door.

The room-specific humidity alarm would be the first alarm. A steam leak of 1 kg/s will probably increase the humidity (alarm limit > 60%, normal value is 6 %) very early. Closed trap of the floor will also produce a specific flood alarm > 30 mm. The latter alarm was assumed to come later than the humidity alarm in the case of a steam leak.

The exit ventilation line of the instrumentation room has a humidity measurement alarm down-stream, but it probably is slower than the other alarms. There are also radiation measurements, which will cause a changeover in the vent line from exhaust to filtration in case of high radiation.

Additional instrumentation faults occur as a consequence of continued steam spraying into the instrument room, e.g., injection water disturbance of the same kind as earlier or spurious low level signal of an emergency accumulator closing the discharge line. These have little importance for the operation of the plant, but will produce uncertainties in the control room.

A-7 NUREG/IA-0137

APPENDIX A: SCENARIO DESCRIPTONS

A.22 Probable course of events


Time (min.) 5'6 Description

(7:57 p.m. simulated time of day)

0 Unit at full power.

3 * Leak of PZR-level measurement +line (steam space), parameter 1 cm2 . * Measurement fault YP10L001 +0.3 i/min. over 12 minutes.

3 alarm High PZR-level > +200 mm.

3 alarm Unrecognized leak > 1 kg/s. This is a common alarm, even in normal power operation or small evolutions at the plant.

4 alarm PZR-heater banks automatically actuated.

4 Normal letdown valve opened on high level signal.

5 Normal level control switched to manual mode by RO.

5 RO probably notices that letdown should not be open and closes the valve.

6 *Measurement fault YD 14FO1, gradient -0.1 kg/s. Alarm and pump trip in 3 minutes.

7 RO takes the makeup pump automatics off to make it possible to operate makeup pumps.

7 RO probably starts both TK52/53D001 additional makeup pumps.

8 Emergency RCP injection water initiated.

9 alarm YD14FO1 (RCP YD14 seal injection flow) < 0.19 kg/s, YD14DO1 (RCP) trip.

9 Plant automatic power reduction to 80 %.


5 Times may be different between LOTS and HAMMLAB due to plant performance differences, e.g., emulation of passive system performance.

6Times for data collection are approximate and, in some cases, depend on a crew activity.

A-8 NUREG/IA-0137


11 alarm *Fire alarm in the instrumentation room UX00K401.

12 Fire alarm sounds and CRS goes to the fire monitoring panel and checks the identification code of fire location. CRS alerts the fire department by the loud speaker, and they will access the reactor building.

13 *TH80L003 (accumulator) measurement fault, gradient -I m/main. Alarm will be produced in 5 minutes. Erroneous instrumentation signal due to high humidity/temperature. This will close the accumulator discharge line valve in five minutes. Alarm will indicate that accumulators have been injecting to primary circuit, though primary circuit is above accumulator injection pressure.

18 TH80L003 Accumulator gradient alarm.

19 *YB54L061(Steam Generator Low level) level measurement fault due to humidity problems in instrumentation room. This plant protection signal initiates inadvertent EFW startup into the YB54 steam generator. Because it is a plant protection signal, BOP operator may follow up to attempt to determine the cause of the alarm.


24 Firemen inform MCR about what they see in the reactor building basement: "There is no fire, but the instrumentation room is full of steam". They cannot see anything nor access the compartment.

26 Firemen have a radiation meter. They will ask CRS instructions what to do, i.e., Are they allowed to access the instrument room? CRS will probably ask how high the radiation is. If it is > 10 mSv/h, access is not allowed.

In case of access, they have respirators, and protective clothes are available in the fire truck.

CRS will probably ask firemen to rapidly check the room inside and identify the leaking pipeline, if they have appropriate clothing and respirators.

28 The MCR staff may diagnose an ISLOCA, though the source of the leak will not yet be clear. Earlier RCP trip due to low flow may indicate leaking flow injection line. However, the temperature of RCP seal water is not high enough to produce steam, so they will have to check all possibilities, and look through alarm logs, instrument diagrams, and procedures.

CRS takes the event based procedure of small LOCA.

29 The firemen cannot see clearly enough to tell the code of the leaking line.

29 Manual start of boration.

A-9 NUREG/IA-0137


32 Reactor manual scram.

Pause for data collection (Manual SCRAM plus 5 minutes, or at -32 minutes).

35 Cooling down by turbine by-pass according to the EOP.

33-40 Operation of PZR-level control to adjust the level to normal.

40 YZ70 manual overriding to gradient mode with the limit of 5 bars. This is done when steam pressure < 35 bars. This may be done later depending on the shutdown rate, etc.

50 Leak may be locally identified in the SG compartment of the containment. Access into the compartment is possible immediately after reactor scram. But the crew must decide that this is the place to look. The isolation is not complete, YP1OLO continues leaking, 0.1 cmr2 .

Pause for data collection

53 Start of the pressurizer cooling by spraying.

55 Shut down goes on.

60 Attempts to stop the rest of leak, crimping the leaking pipe flat, for example.


A-10 NUTREG/IA-0137


A.3 Scenario 3: Total Loss Of Feedwater

A.3.1 Synopsis

Initial conditions: "* Unit is at 100 % power, "* Reactor status: End Of Cycle (or high decay heat rate) "* Emergency Main Feedwater-pump RL92DO1 is out of service. "* MFW-pump RL91DO1 is out of service. "* Lubricating oil pump RL22D03 out of service.

In this scenario, a main feedwater line leakage of 20 kg/s occurs in the room where the main feedwater pump lubricating oil pumps are located. This is a common-cause fault (i.e., a single fault that results in multiple failures) that produces electrical short-circuiting of the lubricating oil pumps. As a consequence, individual oil pumps trip until, one by one, all of the main feedwater pumps have tripped. Only one of the emergency feedwater pumps is available, since the other is out of service for some kind of maintenance or repair.

The operating crew will use the remaining EFW pump to supply feedwater to the steam generators. However, due to a recurrent over-speed fault, even the remaining EFW pump is not able to supply the necessary feedwater to remove decay heat. Over the course of the scenario, the crew will inject water from emergency sources into the RCS while they attempt to recover at least one feedwater or emergency feedwater pump. Heatup of the RCS will occur. It is important to note here that, unlike U.S. plants, the Loviisa plant (like other Russian-designed PWRs of it's type) does not have sufficient PORV capacity to use once-through core cooling or "feed and bleed" cooling to successfully remove core decay heat. Hence, the crew is dependent upon recovery of a secondary system heat sink to remove decay heat.

After approximately 1 hour (conventional plant) to 2 ½ hours (advanced plant) later, as maintenance and electrician personnel arrive at the plant on the morning shift, a main feedwater pump lubricating oil pump will become available to place back in service. This occurs only after the MCR staff isolate the feedwater leak and direct personnel to dry the pump. Thereafter, MCR staff must evaluate a situation for which procedural guidance is unclear: the recommissioning of an empty steam generator. The crew must determine the best way to supply water to the empty steam generator(s) without inducing thermal shock to the collector sheet in the steam generator. This latter situation (i.e., uncontrolled feed of cold water to the collector sheet) could result in a tube rupture.

A-11 NUREG/IA-0137




Time (min.sec) 7,8 Description (simulated time of day)

-10.0 Unit on full power. (06:10 a.m.)

0.0 *Leak of main feed water line RL20, parameter 1.3 = 20 kg/s (06:20 a.m.)

2.00

5.10 alarm

5.17

5.18 alarm

6.11 alarm alarm

6.29 alarm

7.07 alarm

7.13

7.40

7.42 alarm

7.47 alarm

9.13 alarm

10.06 alarm

Decreasing level of the FW-tanks. Increasing secondary makeup water flow. Pause for data collection

RL22D02 oil pump tripped. This is caused by electrical short circuit because of wetting from the leak.

MFW-pump RL22DO1 tripped, due to no lubrication.

MFW-pressure < 65 bar.

Fire alarm due to hot steam and water spraying in room. Fire extinguisher pumps automatically started.

RL21D02 oil pump tripped (electrical short circuit, because of wetting). RL21D03 reserve oil pump starts automatically.

RL21D03 tripped (electrical short circuit, because of wetting).

RL21D01, MFW-pump tripped, due to no lubrication oil.

Steam pressure > 47 bar.

MFW-pressure > 55 bars.

RC (turbine bypass) > 25% open, due to automatic turbine power decrease.

RL61D02 oil pump tripped. (electrical short circuit because of wetting). RL61D03 started automatically.

RL61D03 tripped. (electrical short circuit, because of wetting).

7 Times are different between LOTS and HAMMLAB due to plant performance differences, e.g., emulation of passive system performance. 8 Times for data collection are approximate and, in some cases, depend on a crew activity.

A-12 NUREG/IA-0137


alarm

alarm

SPDS-alarm

prior alarm

10.12

10.13

10.24

10.25

10.25

10.40

10.35

10.50

11.14

11.55

14.07

16.16

17.17

24.04

25.13

29.07

30.06

37.35

(Note:

40

42

A-13NUREG/IA-0137

RL61DO1 MFW-pump tripped, due to no lubrication.

RL62DO1 tripped because of low MFW-pressure < 52 bars. There is now a loss of MFW.

The leak decreased from 20 kg/s to 6 kg/s after Loss of MFW.

Manual reactor trip.

EFW-pump RL93DO1 started automatically, on low SG level.

Turbine trips due to reactor scram.

YZ51 started EFW-signal due to low level -140 ram.

Primary cooling is abnormal, category 1.

SG-level < 1.95 m.

SPDS-alarm Low SG level

SPDS-alarm SG level 2/6 -300 mm.

SPDS-alarm Automatic diagnosis: Leak of secondary circuit (suppressed).

alarm *Over-current trip of EFW-pump, RL93D01. Theie is now a total loss of feedwater.


alarm All RCPs tripped due to low SG levels, -500 mm.

SPDS-alarm Primary cooling is abnormal, category 2. SG-level -800 mm and T < 300"C.

SPDS-alarm SG-levels < 1.2 m (= <-900 mm below nominal).

RL93DO1 started again.

*RL93D01 trips again. (Starting and tripping occurs repeatedly,

depending on attempts to re-start requested by MCR staff).


RO Initiates large volume boration.

Time scale between conventional and advanced plants differ now)

Primary circuit heatup 20°C/h.

SPDS-alarm Primary cooling is abnormal, category 3. SG-level -800 mm and T > 300°C.



60 Startup of RL62D03, MFW oil pump after drying the electrical switching box of the pump motor.

62 Startup of RL62DO1, MFW pump

64 Initiation of MFW to one SG with small flow rate in the beginning.


A-14 NUREG/IA-0137


A.4 Scenario 4: Loss Of Off-site Power With A Stuck Open Steam Generator Safety Relief Valve

A.4.1 Synopsis Initial conditions: * 90% power, load following. * Reactor status: End Of Cycle, boron concentration 77 PPM, burnup 322 full power days. * 400 kV grid in overhaul, circle line of Southern Finland is disconnected at Korea switch yard. * A core exit temperature measurement of reactor protection is faulted. It has one day earlier exceeded

the fast trip limit, 3 10°C. Reactor protection takes place by 2/3 logic.

The scenario begins with a request from the dispatch center for the plant to go to full power. After beginning this task, a load rejection on one of the plant's two turbine generators occurs due to a controller failure. The staff will be able to repair this malfunction rather easily, and begin a return to power on both turbines. During the return to power, the external supply of power to the plant is lost.

The loss of off-site power produces a number of alarms, mainly on the secondary side of the process. In addition to the normal alarms, an additional malfunction occurs: a stuck-open steam generator safety relief valve (SRV). Normally an over-cooling event, the stuck open SRV will increase the rate of over-cooling. An important task of the MCR staff is to detect this spurious valve failure and isolate it.

Because of a pre-existing reactor protection signal fault, the reactor will trip. Even after the station lost it's external power supply, it was supplied with power by it's own power production. With the trip of the reactor due to the protection signal fault, the station is without power, aside from it's emergency diesel generators. These will provide the necessary electrical supply for the MCR staff to bring the plant to a safe and stable shutdown state.

A-15 NUREG/IA-0137




Time (min.sec) 9,10 Description

0 (06:00 a.m. simulated time of day)

7

8

alarm10

14

15

20

25

26

32

34

34.25

35.00

35.15

35.16

alarms

Unit power at 90% power.

Dispatch center asks Unit to go back to full power, increasing at 100 MW/h.

BOP operator begins to carry out the task. The electrical output starts increasing.


*Turbine power controller alarm.

BOP operator/CRS acknowledges the alarm in the instrumentation room.

Turbine 2 power reduction due to the power controller failure. The turbine output is 30-40 MW controlled by the frequency controller.

Staff will repair the malfunction and begin the return to power. Pause for data collection

*Loss of off-site power, 400 & 100 kV grids are lost.

*RA52S02, SG safety relief valve sticks open simultaneously with

secondary pressure spike.

Reactor fast trip due to faulted core exit temperature over 31 0VC, 2/3 logic. (Reactor scram is not a consequence of turbine trips.)

Power imbalance I, between reactor and turbines, +5 bar addition to the fresh steam set value of the turbine power controller.

Turbine trips due to power imbalance II.

RO starts emergency boration.

Low voltage alarms of auxiliary power supplies.

Frequency = 45 Hz, generator trips.

9 Times may be different between LOTS and HAMMLAB due to plant performance differences, e.g., emulation of passive system performance. ' Times for data collection are approximate and, in some cases, depend on a crew activity.

A-16 NUREG/IA-0137


35.18

35.20

35.20

35.22

SPDS-alarm

RCP trips due to low voltage.

6 and 0.4 kV loads trip, breakers of busbars open.

Reactor trip due to loss of RCPs.

Inventory alarm, PZR-level < 1.7 m

Diesel generators start.

Automatic restart of safety related loads.

HPSI initiation.

At this time approximately the staff will probably recognize stuck open SG safety relief valve.

EFW automatic initiation due to loss of main feedwater.

RO probably starts closing YA12SO01, the main gate valve and also the corresponding TC-valves connected to the loop with the affected SG.


EFW-pump trip due to pressure spike of makeup water in the pump suction.

Steam pressure < 35 bar.

YZ51 plant protection signal on reinitiating EFW.

BOP isolates the faulted SG resulting in a fast pressure drop in the SG.

Automatic isolation of SG, YB52 due to steam pressure difference > 5 bars between YB52 and the main steam header.

Now the staff will dry out the faulted SG and let it cool down.

RO should stop HPSI, 2/4, TJ12DO01 and TJ52D00l.

TJ51DO01 stopped by RO.


TJ1 1DO01 stopped by RO.

Process is stable and cooling is maintained by natural circulation and using atmospheric steam header relief valves.

A-17 NUREG/IA-0137

35.22-35

36.30

36.36

36.40

36.46

37.10

.37.28

37.44

40.11

47.18

51.04


A.5 Scenario 5. Steam Generator Over-fill

A.5.1 Synopsis

Initial condition: Full power, 100%, normal size of SG.

The scenario begins after a shift change-over has taken place. A small de-watering line leak in the high pressure pre-heaters has been detected earlier and it is necessary to shut down one train of the high pressure pre-heater system (RD50) for repair. It is a written order given by the operations engineer.

During the shut-down of the affected train of high pressure pre-heaters, an alarm is received in the MCR. A faulty current has resulted in an interruption of the normal power supply to one of the RCPs. The pump has not tripped, but is supplied by station batteries. When the RO goes to the instrument room and acknowledges the alarm, a trip of the RCP occurs together with the ensuing power run-back of the reactor. After some time, the cause of the malfunction is repaired and the crew places the RCP back on-line.

During the preparations for, and together with the re-start of the RCP, the BOP operator's task is to supply feedwater to the steam generator in the loop of the affected RCP. While doing so, a malfunction of the steam generator feedwater regulating supply valve system occurs. The regulating valve sticks open at 80%, and subsequent supply valve failures produce a situation in which the steam generator is supplied with water in excess of the capacity to boil it off. This results in a rising steam generator level in the affected steam generator. There is no specific procedure for this malfunction, only system knowledge and system-level operating procedures.

When the BOP and other MCR staff realize that they have an uncontrolled supply of feedwater to a steam generator, they will begin attempts, both in and outside the control room to manually isolate the steam generator. This is needed to prevent water from entering the steam lines, and causing thermal shock to the highly-heated pipes and to further guard against the possibility of water approaching the turbine generators which may be destroyed by such an event.

It is important, while attempting to isolate the affected steam generator, that water continues to be supplied to some of the unaffected steam generators to remove core decay heat.

A-18 NUREG/IA-0137



Time (min.sec) "," Description (10:00 p.m. simulated time of day)

0 Unit on full power. Instrument technician and electrician are on-site during evening shift.

3 BOP operator starts decreasing turbine 2 output by 8 MW to match high pressure pre-heater train RD50 shut-down.

5 BOP gives an automatic shutdown command for RD50.

8 He opens the FW-bypass of RD50 and closes the normal lineup (manual command of remote control valves.)

9 The staff probably start planning the remaining isolations needed for shut down and repairs. Pause for data collection

10 alarm (very *During the isolation planning, RCP YD12DO1 gives an alarm: important) "Magnet is supplied by batteries". The reason is a faulty current

measurement. RO should go to the instrument room in the same building, lower level to check the status of the magnet.

13 The simulator instructor describes the situation to the RO: "The RCP relief magnet is now supplied by the batteries."

15 When the RO acknowledges the alarm it results in a trip of the whole relief magnet and RCP, too.

15- Automatic power decrease occurs. 20 The staff monitor that everything goes as planned:

- reactor power reduction to 75-80 % - no reverse rotation of the RCP (1-2 minute after the trip) - electric power reduction to 350-370 MW - Plant parameters, i.e.:

- SG-levels - Steam pressure - Primary pressure, etc.

20- Electrical and instrumentation technician diagnose the fault and 30 correct it replacing the defective instrumentation card.


'1 Times may be different between LOTS and HAMMLAB due to plant performance differences, e.g., emulation of passive system performance.

12 Times for data collection are approximate and, in some cases, depend on a crew activity.

A-19 NUREG/IA-0137


BOP operator adjusts the set value of the power controller to match the new situation.

He also acknowledges an additional pressure signal from the reactor power limitation due to the RCP trip. His duty is now especially to monitor the power balance between reactor and the turbines.

32 - The staff starts preparing for the RCP restart:

- PZR-level increased to +50 mm. - SG-level decreased -50 mm to avoid overfill during start-up. This is done by opening the periodic SG-blow down for a while. - BOP operator switches the SG-level controller to manual mode and closed.

35.39 RO restarts the tripped reactor coolant pump.

35.51 BOP maintains the SG-level by operating manually the FW-control valve.

36.46 RL72S002 (feedwater flow control valve) is 70 % open.

36.54 65% open.

37.10 When the level corresponds to the set value, BOP switches the level controller to automatic mode.

37.20 BOP gives the power controller a new set value, 458 MW, and a gradient of 300 MW/h, which is used up to 405 MW; after that 120 MW/h.

37.36 *Simultaneously with power adjustment the specific main feedwater flow controller of SG52 inadvertently opens the control valve and on the way it sticks open 80% due to mechanical failure of the actuator.

37.57 alarm SG level is rising (high level)

38.12 RL72S03 block valve (operation time is 19 sec.) closes automatically due to high level, +100 mm.

After RL72S03 closure the SG level drops fast and at +50 mm, the valve is opened again by automatics.


38.24 BOP switches the control circuit to manual mode and tries to close the control valve, but is unsuccessful.

38.41 *The valve cycles 2-3 times and finally sticks 30 % open due to a torque trip.

39 BOP will now try to close the serial block valve, RL72SO01 (operation time is 78 sec.)

A-20 NLUREG/IA-0137


39.45 *RL72SO01 sticks 13 % open, torque trip.

41.09 *Torque trip will finally stick RL72S03 to the position of 30-50% resulting in further rising level, three stuck open serial valves.

41.45 Turbine trips will occur at a level of +20 Mm.

42.24 At the level +340 mm a new protection signal will isolate the steam line, MFW-line (RL72S03, but it stuck open 30 %) and trip the RCP.

No reactor trip takes place. Reactor power will be decreased automatically (if in automatic mode) or the RO may do it manually faster, if quick enough.

The SG level continues rising.

43.00 BOP tries to avoid the torque trip of RL72S03 by trying to open it, no success.

43.30 BOP tries to avoid the torque trip of RL72SO1 by trying to open and reclose it, no success.

43.45 BOP sends FO to locally and manually close RL72S03. The closure will take about in 10 minutes. The SG level will continue to rise during this time, gradually approaching the level of the MFW lines.

44.00 RO trips the reactor.

44.00 BOP stops all the MFW-pumps to avoid excessive overfilling.

SG safety valve may open, depending on the time scale.

44 An alternative: BOP can try to split the MFW-system to two sections (closing RL61S001, operation time 125 sec.) and stop half of the MFWpumps and corresponding RCPs leaving the other half running and maintaining 50 % power.

45 After MFW-pumps stop it is possible that the crew try to re-open and reclose the faulted block valve. It will be logical to let them succeed in closing because the pressure difference across the valves is gone.


46 Scenario continues until valves are returned to normal (e.g., 10-20 minutes)

A-21 NUREG/IA-0137

APPENDIX B: INSTRUCTIONS TO PARTICIPANTS

B-1 NUREG/IA-0137

B.1 General Instructions to Participants

The Halden Reactor Project together with the Loviisa Nuclear Power Station are conducting a research project for the U.S. Nuclear Regulatory Commission concerning the staffing of advanced passive plant control rooms. The purpose of this research is to evaluate operator performance in representative, conventional plant environments and advanced plant-type environments. For similar challenging situations we wish to evaluate the effect of different staffing complements on crew performance. Eight crews from the Loviisa Nuclear Power Station are assisting us with this research, together with personnel from the Loviisa Nuclear Power Station simulator facility. Four of the crews will participate in the study at the Loviisa Nuclear Power Station. The other four crews will come to Halden (HAMMLAB - HAlden Man-Machine Laboratory) to participate in the study.

(The following paragraph was only in the General Instructions given to Loviisa /Conventional plant participants) At Loviisa some changes will be made in the simulation facility as you are accustomed to seeing it. This is being done to make Loviisa more representative of conventional western style PWR-control rooms and plant performance. Before the study that you are about to participate in, you will also receive some additional training to acquaint you with some of these differences and to become more familiarized with these differences. All of your previous training, experience and procedures for operating the plant will still be applicable during the study. The main differences are that some of the information systems in the control room, e.g., the advanced computerized systems, will be somewhat unavailable. After the training we will go through a series of five challenging scenarios. The purpose of the scenarios is to gather a baseline of operator performance in a conventional type plant complement for comparison purposes with advanced plant type complements, which will be conducted in the autumn at Halden.

(The following paragraph was only in the General Instructions given to Halden/Advanced plant participants) Before the study in HAMMLAB, you will receive a couple of days of training. The process modeled here is the Loviisa process, with minor modifications. Your training will focus on the differences between the interfaces, the systems in HAMMLAB which may be unfamiliar to you, and the way to interact with the simulator. After the training we will go through a series of five challenging scenarios. The purpose of the scenarios is to collect data on operator performance in an advanced type plant complement for comparison purposes with conventional plant type complements, which was conducted in the spring at Loviisa.

During each scenario, interruptions will be made in which members of our staff will give you a set of questionnaires with some very straight-forward questions. In the interruptions, please simply answer the questions on the questionnaires; do not discuss questions with the other crew members. During the actual scenarios themselves, though, the part in which you are actually operating together as a crew, discussions should be carried out as normal.

Following each scenario you will have a break. During the break there will be a short debrief in which we will ask you for some feedback about the scenarios, your impressions and thoughts about how specific things like procedures and training and the instrumentation in the control room helped you in controlling the situation. We will also ask you some general questions about what factors made the scenario difficult or easy to control. There are no right or wrong answers to the questions that we will ask you during the debriefing. We are only asking for your opinions about things which you perceive as affecting your performance.

B-2 NUREG/IA-0137

At the end of the study in the autumn, all the data will be analyzed from both Loviisa and from Halden, and a report produced. The report will be sent to Loviisa for review and analysis, before it will be sent to the NRC. If you are curious about the results of the different crews' performance in both the conventional and the advanced plant type complements, please contact (name of the plant-specific contact person) for details or even perhaps a copy of the report.

One word should be mentioned about confidentiality. It is very important for obvious reasons that other members of the plant not be aware of the specific scenarios we are using. We ask, therefore, that you do not discuss the scenarios with other members of the plant or the operations staff, or in the areas where other crew members could overhear your conversations about this research.

To guarantee your anonymity, data will not be provided which could be used to identify crew members. There will be no direct link between crew performance as presented in our final report and the crews which participated in the study nor individual crew members.

Thank you for your participation, support and offering us your expertise in carrying out this very important research for a very important issue for advanced reactors. If you have specific questions about the study or about data from the study, please ask either (name of the plant-specific contact person) or (name of the study leader).

B-3 NUTREG/IA-0137

B.2 Instructions for Situation Awareness Questionnaire

This questionnaire has been developed in Halden with assistance from Loviisa operators to investigate how operators form and maintain a good overview of the process status. We typically use it in situations where we want to see if different process control systems are good or poor at giving the operator overview of the process.

During the scenario, the simulator will be frozen several times and you will be given a couple of questionnaires. Please turn away from the screens: do not look at the panels. During these interruptions you will be given a questionnaire with a set of 18 questions. Each set of questions will be new for each interruption, although some questions might be repeated several times during a scenario. Please note that:

"* For each interruption, the questions are picked at random from a large question-base. "* The questions sometimes focus on important parameters related to the disturbance, other times

the parameters are not related to the disturbance.

The questions will be given to you in written form. Each of the questions will be focused on a set of key process parameters (such as pressures and flows) and components (such as pumps and valves). The first 6 questions are related to how the process has developed the last three minutes, the next 6 questions focus on the current state of the process in comparison to the first five minutes of the scenario, and the last 6 questions are related to how you think the plant will develop over the next three minutes. For each of the questions, there is a limited number of answers to choose from. The alternatives are:

"* The parameter remains the same. "* The parameter has decreased. "* The parameter has increased.

Some questions ask you about several parameters at the same time (e.g. steam generators), and you can then answer deviations in one, more than one, or deviations in both directions. Please circle the answer category you think is appropriate for each question.

Sometimes it can be difficult to decide if a parameter is just fluctuating or actually decreasing or increasing. We want to report changes only when:

"* A parameter is deviating from its normal value "* There is a steady increasing or decreasing trend in the parameter.

We will ask you how parameters have developed from the 'recent past' and how they will develop in the 'near future'. By 'recent past' and 'near future' we mean approximately three minutes ago, or approximately three minutes into the future. After the experiment, all your answers will be compared to the simulator log.

Please answer all questions. We want you to give your best judgment about the status of the parameter and component, regardless if you know the actual status of the component, or you have to infer this from other information you have. If you have no idea about the actual parameter, simply give us your best guess.

3-4 NUREG/IA-0137

B.3 Participant Instructions: Workload Rating Scales

We are not only interested in assessing your performance but also the experiences you had during the different task conditions. Right now we are going to describe the technique that will be used to examine your experiences. In the most general sense we are examining the "workload" you experienced. Workload is a difficult concept to define precisely, but a simple one to understand generally. The factors that influence your experience of workload may come from the task itself, your feelings about your own performance, how much effort you put in, or the stress and frustration you felt. The workload contributed by different task elements may change as you get more familiar with a task, perform easier or harder versions of it, or move from one task to another. Physical components of workload are relatively easy to conceptualize and evaluate. However, the mental components of workload may be more difficult to measure.

Since workload is something that is experienced individually by each person, there are no effective "rulers" that can be used to estimate the workload of different activities. One way to find out about workload is to ask people to describe the feelings they experienced. Because workload may be caused by many different factors, we would like you to evaluate several of them individually rather than lumping them into a single global evaluation of overall workload. This set of six rating scales was developed for you to use in evaluating your experiences during different tasks. Please read the descriptions of the scales carefully. If you have a question about any of the scales in the table, please ask me about it. It is extremely important that they be clear to you. You may keep the descriptions with you for reference during the experiment.

After performing each of the tasks, you will be given a sheet of rating scales. You will evaluate the task by putting an "X" on each of the six scales at the point which matches your experience. Each line has two endpoint descriptors that describe the scale. Please consider your responses carefully, and try to use the scales to distinguish among the different task con-ditions. Consider each scale individually. The ratings you give us will play an important role in the evaluation being conducted.

Your active participation is essential to the success of this experiment. Your willingness to participate is greatly appreciated by all of us.

B-5 NUREG/IA-0137

B.4 Instructions for the Behaviorally Anchored Rating Scale

During this experiment, you will watch eight crews of operators performing in five process disturbance scenarios. All crews will participate in the same five scenarios. You will be seated in the control room with the crew, so you can hear and see their communications and behaviors. You will be seated at the side of the room, so you do not disturb the crews. Please remain seated and do not talk with the crew members during the sessions. Each scenario will be divided by interrupts. During these interrupts, the crew will complete questionnaires. Also, during these interrupts, please complete the rating form for evaluating the crew's team interaction. Please only evaluate the crew's performance on the last segment of the scenario (from the last interrupt to the present).

The rating form divides team interaction into five dimensions: task focus/decision making, coordination as a crew, communication, openness, and team spirit. For each of these dimensions, a table lists behaviors to watch for when evaluating the crews. Please make notes as you feel relevant, to help you accurately evaluate the crew.

At the bottom of each page (for each dimension of team interaction) is a rating scale, from 1 (worst) to 7 (best). Please draw an X over the part of the scale which you feel best represents the crew's performance on that dimension for the scenario segment immediately preceding the interrupt. To help you make your evaluation, each rating scale contains examples of behaviors that would be rated as 1, 4, or 7.

Thank you for your participation and assistance.

B-6 NUREG/IA-0137

APPENDIX C: INVENTORIES

C-1NUREG/IA-0137


C.1 Situation Awareness Control Room Inventory Example Inventory

Test subject: RO Scenario:01 Administration :1 Team: 1

HISTORY

In comparison with the recent past, how has the insertion of L rods (strong adjustment effect) developed?

Increase Same Decrease

In comparison with the recent past, how has the number of active primary circulation pumps developed?


In comparison with the recent past, how has the number of active pumps in the TK make up system developed?


In comparison with the recent past, how has the temperatures after the low-pressure pre-heaters developed?

Increase > 1 Increase 1 Same Decrease 1 Decrease > 1 Drift In Both Directions

In comparison with the recent past, how has the number of active condensate pumps developed?


In comparison with the recent past, how has the pressures in the condensers developed?


NOW

In comparison with the normal status, how would you describe the current temperatures in the hot legs of the primary circuit ?


In comparison with the normal status, how would you describe the current neutron flux of the reactor (power range detectors) ?


In comparison with the normal status, how would you describe the current number of active pressurizer heater banks ?

C-2 NUREG/IA-0137



In comparison with the normal status, how would you describe the current temperatures in the condensers ?


In comparison with the normal status, how would you describe the current flows into the steam generators ?


In comparison with the normal status, how would you describe the current electrical power outputs from the generators ?

Increase > 1 Increase I Same Decrease I Decrease > 1 Drift In Both Directions

FUTURE

In comparison with now, predict how the insertion of D rods (weak adjustment effect) will develop?


In comparison with now, predict how the level in the pressurizer will develop?


In comparison with now, predict how the temperatures in the cold legs of the primary circuit will develop?

Increase > 1 Increase 1 Same Decrease 1 Decrease > I Drift In Both Directions

In comparison with now, predict how the steam pressures in the secondary loops will develop?


In comparison with now, predict how the levels in the condensers will develop?

Increase > 1 Increase 1 Same Decrease I Decrease > I Drift In Both Directions

In comparison with now, predict how the openings of the turbine bypass valves will develop?


C-3 NUREG/IA-0137


C.2 NASA TLX Rating Scale Definitions

Title

Mental Demand

Physical Demand

Temporal Demand

Performance

Effort

Frustration

Endpoints

Low / High

Low / High

Low / High

Failure / Perfect

Low / High

Low / High

Description

How much mental and perceptual activity was required (thinking, deciding, calculating, remembering?)

How much physical activity was required (e.g. pushing, pulling, turning, controlling, activating, etc.)? Was the task slow or brisk, slack or strenuous, restful or laborious?

How much time pressure did you feel due to the rate or pace at which the task elements occurred? Was the pace slow and leisurely or rapid and frantic?

How successful do you think you were in accomplishing the goals of the task set by the experimenter (or yourself)? How satisfied were you with your performance in accomplishing these goals?

How hard did you have to work (mentally and physically) to accomplish your level of performance?

How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task?

C-4 NUREG/IA-0137


Job Title in Scenario: Date: Name of examiner:

Scenario Number: Pause Number:

Mental Demand

L =1Low 50 High

Physical Demand

I- I I

Low 50

.1High

Temporal Demand

I I1-

Low 50I I

-IHigh

Performance

i Failure

I I I I I I I

50

Effort

I- I I I I I

Low 50

Frustration Level

I I I I I I I IHigh

C-5NUREGAA-0137

Perfect

-IHigh

!.

Low 50

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

II

I II

!I

II

II

II

I II

II

II

II

II

I I!

II

II

II

II

II

I II

II


C.3 Behaviorally Anchored Rating Scales Example Inventory

Scenario

Segment of scenario

Date Time

Crew

Evaluator

C-6 NUREGAIA-0137


During the last set of exercises that you observed, did you see any of these things happen that significantly affected work outcome? (1) X the positions of the team members who were involved; (2) Circle the X of the individuals who did what you marked.

Communication Reactor Turbine Shift Control Room Operator Operator Supervisor Technician

1. Sufficient amount of communication information that needed to be stated was stated.

2. Information was informative, not chatter.

3. Information was relevant to the situation.

4. Information was correct.

5. Information had the desired effect - the targeted person responded appropriately.

6. Information was given at the appropriate time in the scenario.

If the incident took place more than once, please indicate the number of occurrences.

Did you observe any examples of extremely effective or extremely poor communication? (describe below I other side)

Please place an X over the rating which you feel best describes the crew's performance on this dimension.

I I I I I I

Communication1

I I I 4

I I 7 7

1: failed to communicate or communicated misleading information 4: communicated about important issues, but neglected some 7: optimal communication - communication was informative, effective, timely

C-7NUREG/IA-0137

I



Team Spirit Reactor Turbine Shift Supervisor Control Room Operator Operator Technician

1. Joking in a positive manner.

2. Appropriate degree of informal discussion.

3. Silences due to hostility. (- negative)

4. Gestures of hostility. (- negative)

5. Arguing over trivial issues. (- negative)


Did you observe any examples of extremely good or extremely poor team spirit? (describe below/other side)


I I I I I ITeam Spirit

1: were openly negative, hostile 4: were neutral 7: demonstrated encouraging, positive and reinforcing behaviors

C-8NUREG/IA-0137

1 4 7

| | | | | |II



Openness Reactor Turbine Shift Control Room Operator Operator Supervisor Technician

1. Helpful in giving information to crew members.

2. Asking for/providing assistance when needed.

3. Seeking confirmation when appropriate.


Did you observe any examples of extremely open or extremely closed team openness? (describe below / other side)


Openness

I I I I I I

4 7

1: were not willing to give assistance or ask for assistance 4: moderate openness - gave/received some assistance, but more would have been better 7: showed optimal openness - giving/receiving assistance as often as needed

C-9 NUREG/IA-0137

1

I I I I II



Coordination as a Crew Reactor Turbine Shift Control Room Operator Operator Supervisor Technician

1. Good ability to work together.

2. Able to shift roles when needed.

3. Work together to find facts needed for decision making.

4. Appropriate degree of delegation of tasks.

5. Good utilization of technical support.


Did you observe any examples of extremely good or extremely poor crew coordination? (describe below / other side)


Coordination as a Crew I I I IIII I I

I I I 4

I I I 7

1: unable to work together 4: moderately effective as a crew - they worked together reasonably effectively 7: delegated wisely, shifted roles as needed - they worked together highly effectively

C-10NUREG/IA-0137

1



Task Focus / Decision Making Reactor Turbine Shift Control Room Operator Operator Supervisor Technician

1. Considering alternative actions.

2. Staying with chosen primary task (remaining focused on main task).

3. Prioritizing tasks.

4. Solving the correct problem.

5. Decisions appropriate to diagnosis.


Did you observe any examples of extremely good or extremely poor task focus/decision making? (describe below / other side)


Task Focus / Decision Making

I I I 4

' I 7

1: unfocused, poor decision making 4: moderately effective decision making 7: optimal degree of consideration of alternatives and focus on task

C-11NUREG/IA-0137

1

I I I

I IJIII


C.4 Rated Crew Performance Example Inventory

Additional Performance Factors

Explanation: Solution Path refers to the crew's use of time in recognizing the event (e.g., loss of heat transfer, etc.) and selection of the correct mitigation procedure(s).

Control of plant refers to the crew's understanding of procedures in their analysis of the transient and the extent to which they challenged safety equipment (e.g., PORVs, code safety valves, etc.).

Communication refers to the extent to which the information exchange among the crew members facilitated transient mitigation.

Confidence refers to the ease with which the crew completed transient mitigation without hesitation, and self-statements about the sureness of their own actions and decisions.

Please circle the number below which corresponds best to your rating of the crew on the above factors for this scenario only. A score of 1 represents a low score (poor performance on the factor) while a 10 is a high score (best performance on the factor).

Solution Path

Control of Plant

Communication

Confidence

Crew Number: Rater:

1 2 3 4 5 6 7 8 9 10

1 2 3 4 5 6 7 8 9 10

1 2 3 4 5 6 7 8 9 10

1 2 3 4 5 6 7 8 9 10

Scenario:

C-12NUREG/IA-0137

APPENDIX D: SITUATION AWARENESS CONTROL ROOM INVENTORY (SACRI)

D-1 NUIREG/IA-0137


D.1 SACRI: The Inventory of Questions

Parameters included in the inventory

Primary Circuit "* Average reactor temperature "* Flows through the TC purification systems "* Insertion of D rods (weak adjustment effect) "* Insertion of L rods (strong adjustment effect) "* Level in the pressurizer "* Level in the TE let-down system tank "* Level in the TH emergency water supply tank "* Margin of departure from Nucleate Boiling (DNB) "* Neutron flux of the reactor (power range monitors) "* Number of active pressurizer heater banks "* Number of active primary circulation pumps "* Number of active pumps in the TK make-up system "* Pressure in the containment "* Pressure in the pressurizer "* Pressure in the primary circuit "* Temperature in the pressurizer "* Temperatures in the cold legs of the primary circuit "* Temperatures in the hot legs of the primary circuit

Secondary Circuit "* Electrical power outputs from the generators "* Flows into the steam generators "* Levels in the condensers "* Levels in the feedwater tanks "* Levels in the steam generators "* Number of active main ejectors "• Number of active condensate pumps "* Number of active emergency feedwater pumps "• Number of active main feedwater pumps "* Openings of the condensate systems' three-way control valves "• Openings of the turbine bypass valves "* Pressures in the condensers "* Steam line pressure "* Steam line temperature "* Temperatures after the high-pressure pre-heaters "* Temperatures after the low-pressure pre-heaters "• Temperatures in the condensers

D-2NUREG/IA-0137


D.2 Wording of the SACRI Questions

The wording of the questions is as follows (with the parameter inserted in the blank space): Questions comparing the present with the recent past e In comparison with the recent past, how have the __ developed?

Questions comparing the present with the status during normal operations o In comparison with the normal status, how would you describe the -?

Questions comparing the present with the predicted near future o In comparison with now, predict how the - will develop over the next few minutes.

D.3 Answer Categories

The answer categories used are: * Increase / Decrease I Same

* Increase in more than one / Increase in one / Same / Decrease in one / Decrease in more than one / Drift in both directions.

A computer program generates a series of 18 questions, including 9 from the primary side and 9 from the secondary side. The 9 questions can then be subdivided into 3 groups of 3 questions each, asking about the past, present, and future states of the plant. A parameter is only selected once for each questionnaire generation. Each question uses the appropriate answer category.

D-3 NUREG/IA-0137


D.4 Signal Detection Theory Calculations

Abbreviations used in the formulas

pCA probability of Correct Acceptance: number of correct acceptances divided by the total number of correct acceptances and false alarms

pFA probability of False Alarm: number of false alarms divided by the total number of correct acceptances and false alarms

pH probability of Hit: number of hits divided by total number of hits and misses

pM probability of Miss: number of misses divided by total number of hits and misses

pDe probability of Deviation: total number of hits and misses divided by total number of questions

pNDe probability of Non Deviations: total number of correct acceptances and false alarms divided by the total number of questions

A' Non parametric measure of operator's ability to discriminate stable parameters from those which are fluctuating

A' = I - 0.25 {(pFA/pH) + (pM/pCA)}

An A' score of 1.0 indicates that the operator was perfect in his/her ability to discriminate parameter drifts from stability, i.e. s/he has made no errors. A performance score of 0.5 meant the operator performed no better than if s/he had been responding according to chance. A performance score of below 0.5 meant the operator's responses were worse than if they had been determined by chance. In fact, it is mathematically possible to derive a negative score: however, due to psychological factors involved in the detection of parameter deviations (on which the underlying principles of Signal Detection Theory are based), an expert operator who is genuinely responding to the best of his/her ability is unlikely to score below the 0.5 mark.

R:S Ratio Non-parametric measure of response bias

R:S = { (pNDe * pFA) + (pDe * pH)} / pDe

A value of I means the operator showed no bias towards either under-estimating or overestimating the extent of non-stable parameters; below 1, the operator under-estimate; above 1, the operator overestimated.

D-4 NUREG/AI-0137


D.5 Scoring the Situation Awareness Inventories

Operator responses should be scored as one of the following four categories: hit, miss, false alarm, or correct acceptance. These categories are based on the following situation:

Process Parameter

Change

Hit

No change

False Alarm

Correct Acceptance

Use the following table for more specific guidance.

The operator says...

Same

The master copy says...

Same

Score

Correct Acceptance

Increase, Decrease, Increase >1, Increase in only one, Decrease > 1, Decrease in only one, Drift in both directions (a change)

Same

Increase, Decrease, Increase >1, Increase in only one, Decrease > 1, Decrease in only one, Drift in both directions (same change)

any change

any change Same

Drift in both directions

any other change, Same

Increase, Increase > 1, Increase in only one, Same

Decrease, Decrease > 1, Decrease in only one, Same

Increase >1, Decrease >1

Increase in only one, Decrease in only one

any other change, Same

Drift in both directions

Decrease, Decrease > 1, Decrease in only one

Increase, Increase > 1, Increase in only one

Increase in only one, Decrease in only one Increase >1, Decrease >1

D-5

Hit Hit Hit Hit Hit Hit Hit (score as Hit)

Miss

False Alarm

False Alarm

Miss

Miss

Miss

False Alarm

Miss

NUREG/IA-0137

Operator response

Change

No change Miss

APPENDIX E: TRANSIENT MANAGEMENT PERFORMANCE RATING FORMS

E-1 NUREG/IA-0137


Table E-1 Ratings of transient management tasks on the Steam Generator Tube Rupture scenario

Announcements and Notifications Critical Task Completion Stabilization and Cooldown

Crew 1 2 3 413 5 6 7 8 Crew 122 31456 718 Crew 1 2 3 4 56 7 8

Initiate Response 3 4 4 3 3 4 4 5 Identification of SGTR 2 4 1 2 3 4 2 5 -Handling of Cooldown 1 4 3 2 3 5 3 1 Sound Fire Alarm XXXXXX X X Response toSGTR 3 5 1 2 4 5 2 5 Pressurizer Level 2 2 2 1 45 4 5 Alert Fire Brigade -- XX-- -- X XX Stop Pump X X -- X X X-- X Primary Pressure 1 2 3 2 3 3 3 1 Make Fire Announcement 3 4 3 3 2 4 5 5 Close Valves X X XX X Xj-- X RateofCooling 2 4 3 1 215 2 1 Declare Plant Emergency X X XIX -- X X X Reactor Scram 3 4 3 3 2 4 4 5 Departure from Nucleate 1 4 3 2 3 3 3 1

1_ Boiling Notify Regulatory Agency X XX X -- -- X X InitiateBoration 4 2 3 2 4 4 4 41 Alert Shift Technical Advisor X X X X X X X X Alert Emergency Preparedness X X ----- X X X Organization Alert Dispatch Center -- X X X -- X X X Call Radiation Personnel X -r ..... Overall rating: 34332445 Overall rating: 3422314 35 Overall rating: 1 4 3 2 35 3 1

Key Rating Scale: 1= minimal performance, 5=optimal performance X = task completed -- = task not completed

Crews 1-4: Crews 5-8: Conventional Plant Advanced Plant

Crews 1 & 2: 4-man crews Crews 5 & 6: 4-man Crews 3 & 4: 3-man crews Crews 7 & 8: 2-man

E-2NUREG/IA-0137


Table E-2 Ratings of transient management tasks on the Interfacing Systems Loss of Coolant scenario

Announcements & Notifications Critical Task Completion Stabilization & Cooldown

Crew 1 2 3 4 5 6 7 8 Crew 1 213141516178 Crew 1 2 3 4516 7 8

Sound Fire Alarm --........- X X X X Identify Leak in Primary 4 1 3 1 2 1 2 4 Handling of Cooldown 3 4 1 11313 4 4 Circuit

Alert Fire Brigade Verification of Pump 2 1 3 3 2 3 4 3 Primary Pressure 3 4 1 21514 5 4 Stop

Make Fire Announcement Investigation of Other 3 2 2 2 4 3 2 4 Pressurizer Level 3 3 1 1 5 5 5 5 Signals

Declare Plant Emergency X X -- X X -- X Reactor Scram 1 1 3 5 3 3 4 5 Rate of Cooldown 3 4-- 4 4 5-Notify Regulatory Agency X X X -- X -- X X Initiate Boration 4 4 4 4 4 4 4 -- Departure from Nucleate 4 4 --... 4 2 2 4

Boiling Alert Shift Technical Advisor X X XXX -- X X Isolating Leak 4 1 2 1 .--........ Alert Emergency Preparedness X X ---- X X X X Organization Alert Dispatch Center -- X ..... X -.... X Call Radiation Personnel -- X X -- X I--I--IOverall rating: 3 5 3 1 5 3 3 4 Overall rating: 2 1313 3 3 4 5 -Overall rating: 314 1 154 4 4

Key: Rating Scale: 1= minimal performance, 5=optimal performance X = task completed -- = task not completed



E-3NUREG/IA-0137


Table E-3 Ratings of transient management tasks on the Loss of Feedwater scenario


Crew 1 2314156718 Crew 1234 5 6 7 8 Crew 1 2 3 4 5 6 7 8

Sound Fire Alarm ..--------- X X X X Identifying leak before 1 1 1 --------- Handling of cooldown 2 3 2 2 4 3 4 5 pump trip __PrimaryPr

Alert Fire Brigade ---------------------- Identifying leak location 1 1 1 1-- ---- Primary Pressure 2 3 2 2 4 4 4 4

Make Fire Announcement ------------------ Isolating leak 2 2 2 1 1 2 2 1 Pressurizer Level 1 3 1 1 4 3 4 4

Declare Plant Emergency -- X X---- X-- Restarting of RL-pumps 3 3 2 2 3 4 4 3 Cooldown gradient

Notify Regulatory Agency X X X .-- - X X X Reactor scram 4 5 4 4 5 5 5 5 Departure from Nucleate 3 3 1 2 3 4 4 4 Boiling

Alert Shift Technical Advisor X X X X X X X X InitiateBoration 4 4 4 2 5 4 4 4

Alert Emergency Preparedness -- X X -- X X X X Refilling of dry steam 1 3 1 2 3 4 4 5 Organization generator(s)

Alert Dispatch Center XlXl--X-- X X X Call Maintenance Personnel I-- -- 1---- ---

Overall rating: 2 5 5 1 3 4 4 5 0verallrating: 1*3 2 2 3 4 4 5 Overall rating: 2 3 2 2 4 4 4 5




E-4NUREGJIA-0137


Table E-4 Ratings of transient management tasks on the Loss of Off-Site Power scenario


Crew 1 2 3 4 5 6 7 8 Crew 123456 7 8 Crew 1 2 6

Make Fire Announcement --.--.--.--.--.--.--.-- Smooth Power Reduction 2 1 2 1 4 4 4 4 Handling of Cooldown 1 1 4 4 4 4

Declare Plant Emergency -- X X ..... X X X Assessing LOOP, 2 1 2 1 4 4 3 3 Primary Pressure 1 111 2 4 4 4 Attempting Back-up Power Sources

Notify Regulatory Agency X X X ..... X X-- Reactor scram (from 5 1 5 1 5 5 5 5 Pressurizer Level 2 1 3 4 4 4 protection vs. by hand)

Alert Shift Technical X X X -- X X X X Initiate Boration 3 1 3 1 3 5 4 4 Rate of Cooldown 1 111 3 3 3 3 Advisor I Alert Emergency X X X ..... X X X Identify andAttemptto 2 1 2 1 5 5 5 5 Departure from 2 1 2 1 3 3 3 3

Preparedness Close Stuck-Open Safety Nucleate Boiling Organization Relief Valve I Alert Dispatch Center -- X X X -- X X X Minimize Loss Through 2 1 2 1 5 5 4 2

Valve: Stop Pump and Close Valve

Call Radiation Personnel -------------k.. .--Overall rating: 3 4 4 1 1 5 5 3 Overallrating: 2 1 2 1 4 5 4 4 Overall rating: 2 1213 4 4 4

Key: Rating Scale: 1= minimal performance, 5=optimal performance X = task completed -- = task not completed Crews 1-4: Crews 5-8: Conventional Plant Advanced Plant

Crews 1 & 2: 4-man Crews 5 & 6: 4-man Crews'3 & 4: 3-man Crews 7 & 8: 2-man

E-5NUREG/IA-0137


Table E-5 Ratings of transient management tasks on the Steam Generator Over-fill scenario

Announcements and Notifications Critical Task Completion Stabilization and Cooldown

Crew 1 2 3 4 5 6 7 8 Crew 1 213 4 5 6 7 8 Crew 1 213 4 5 6 7 8

Make Fire Announcement .................- Identify and Attempt to Close Stuck- --.--.-- 1 4 4 4 4 Handling of Cooldown 4 4 4 4 4 4 4 4 Open Valve

Declare Plant Emergency Manually Closing Alternative 1 4 4 4 4 Primary Pressure 4 2 2 2 5 4 4 4 Valves

Notify Regulatory Agency X X X ------- X Reactor Scram 2-- 1 1 4 4 3 4 Pressurizer Level 4 3 3 3 3 3 3 3 Alert Shift Technical Advisor X X X X X X X X Stopping Feed Water Pumps 2 2 1 1 1 4 2 4 Rate of Cooldown I ....

Alert Emergency Preparedness -- X X --------- Stop Overfilling by Separating 2 2 1 1 -- 4 4 Departure from Nucleate 4 4 4 4 4 4 4 4 Organization Feedwater Lines Boiling

Alert Dispatch Center Initiate Boration F-- I----

Call Maintenance Personnel -- X X ...... X -- Prevent Safety Relief Valve From 1 2 2 1 3 4 1 4 Hammering

Overall rating: 2 4 41 1 1 2 1 2 Overall rating: 2 2 1 1 3 4 3 4 Overall rating: 4 21 2 2 4 33 33



Crews I & 2: 4-man crews Crews 5 & 6: 4-man crews Crews 3 & 4: 3-man crews Crews 7 & 8: 2-man crews

E-6NUREG/IA-0137

APPENDIX F: DESCRIPTIVE STATISTICS

F-1NUREG/IA-0137


Table F-1 Summary table of situation awareness data: scenario type by scenario period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Scenario Type Mean I N I SD Mean I N SD Mean I N [SD Mean I N [SD Mean N SD Rule-based 0.83 66 0.14 0.69 66 0.14 0.65 66 0.24 0.77 66 0.15 0.65 41 0.22 Knowledge-based 0.66 44 0.24 0.69 44 0.19 0.69 44 0.19 0.68 44 0.26 0.90 22 0.06 All Scenarios 0.76 110 0.21 10.69 110 0.16 0.67 110 0.22 10.73 110 0.21 0.74 63* 0.22

*Indicates a loss of 3 records from the data set. This was a 3 person (minimum crew size) crew from the conventional plant in the last scenario period of the Loss of Feedwater scenario. No parametric statistical analyses used data from this fifth scenario period.

Table F-2 Summary table of situation awareness data: crew size by scenario period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 CrewSize Mean I N ISD Mean I N I SD Mean I N I SD Mean I N I SD Mean N SD Normal 0.75 60 0.23 0.69 60 0.17 0.64 60 0.26 0.76 60 0.20 0.76 36 0.20 Minimum 0.78 50 0.17 0.70 50 0.15 0.70 50 0.17 0.69 50 0.21 0.71 27 0.25 All Groups 0.76 110 0.21 0.69 110 0.16 10.67 110 0.22 0.73 110 0.21 10.74 63* 0.22

Table F-3 Summary table of situation awareness data: plant type by scenario period

Scenario Period I Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Plant Type Mean N ISD Mean I N I SD Mean N SD Mean IN ISD Mean I N ISD Conventional 0.79 60 0.22 0.71 60 0.15 0.71 60 0.17 0.72 60 0.19 0.77 33 0.21 Advanced 0.73 50 0.19 0.67 50 0.17 0.62 50 0.27 0.75 50 0.23 0.70 30 0.23 All Groups 0.76 110 0.21 0.69 110 0.16 0.67 110 0.22 10.73 110 0.21 0.74 63* 0.22

F-2

NUREG/IA-0137


Table F-4 Summary table of situation awareness data: crew size and plant type by scenario period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Crew Size Plant Type Mean N SD Mean I N SD Mean N N SD Mean I N SD Mean N SD Normal Conventional 0.81 30 0.23 0.74 30 0.11 0.75 30 0.15 0.79 30 0.14 0.80 18 0.15 Normal Advanced 0.69 30 0.22 0.64 30 0.21 0.53 30 0.29 0.73 30 0.25 0.72 18 0.24 Minimum Conventional 0.77 30 0.21 0.69 30 0.18 0.67 30 0.17 0.65 30 0.22 0.74 15 0.26 Minimum Advanced 0.79 20 0.09 0.71 20 0.10 0.76 20 0.14 0.76 20 0.20 0.67 12 0.24

AllGroups 1 0.76 110 0.21 0.69 110 0.16 0.67 110 0.22 0.73 110 0.21 0.74 63* 0.22

Table F-5 Summary table of situation awareness data: position and plant type by scenario period

Scenario Period I Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Operator Plant Type Mean I N SD Mean I N I SD Mean I N ISD Mean I N I SD Mean I N SD CRS Conventional 0.84 20 0.14 0.72 20 0.17 0.70 20 0.18 0.74 20 0.12 0.81 11 0.14 CRS Advanced 0.73 10 0.14 0.66 10 0.16 0.62 10 0.19 0.79 10 0.09 0.78 6 0.14 RO Conventional 0.73 20 0.29 0.68 20 0.17 0.70 20 0.17 0.70 20 0.24 0.80 11 0.14 RO Advanced 0.78 15 0.11 0.68 15 0.18 0.61 15 0.28 0.66 15 0.35 0.64 9 0.27 BOP Conventional 0.81 20 0.19 0.74 20 0.10 0.73 20 0.16 0.72 20 0.20 0.71 11 0.30 BOP Advanced 0.64 15 0.28 0.64 15 0.22 0.52 15 0.34 0.79 15 0.11 0.71 9 0.27 Dual Role Advanced 0.79 10 0.07 0,70 10 0.10 0.79 10 0.07 0.76 10 0.21 0.69 6 0.21 CRS

AllGroups 0.76 110 0.21 .69 110 0.16 0.67 110 0.22 0.73 110 0.21 0.74 63* 0.22

F-3NUREG/IA-0137


Table F-6 Summary table of situation awareness data: crew size, plant type, and scenario type by scenario period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Crew Size Plant Type Scenario Type Mean N SD MeanI N I SD Mean N SD MeanI N I SD Mean N I SD Normal Conventional Rule 0.91 18 0.13 0.73 18 0.11 0.74 18 0.18 0.79 18 0.14 0.74 12 0.15 Normal Conventional Knowledge 0.66 12 0.27 0.74 12 0.11 0.77 12 0.11 0.80 12 0.13 0.94 6 0.01 Normal Advanced Rule 0.74 18 0.14 0.63 18 0.17 0.52 18 0.33 0.79 18 0.15 0.61 12 0.23 Normal Advanced Knowledge 0.62 12 0.30 0.65 12 0.26 0.54 12 0.23 0.65 12 0.33 0.92 6 0.05 Minimum Conventional Rule 0.87 18 0.13 0.73 18 0.16 0.65 18 0.18 0.69 18 0.17 0.66 9 0.31 Minimum Conventional Knowledge 0.62 12 0.22 0.63 12 0.20 0.70 12 0.17 0.59 12 0.26 0.85 6 0.06 Minimum Advanced Rule 0.80 12 0.08 0.68 12 0.09 0.74 12 0.17 0.81 12 0.10 0.54 8 0.17 Minimum Advanced Knowledge 0.77 8 0.11 0.76 8 0.10 0.79 8 0.08 0.70 8 0.28 0.93 4 0.06

AllGroups 0.76 110 0.21 0.69 110 .016 0.67 110 0.22 0.73 110 0.21 0.74 63* 0.22

Table F-7 Summary table of subjective workload data: scenario by scenario period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Scenario Mean I N SD_ MeanI N SD MeanI N TSD Mean I N SD SGTR 23.36 22 10.14 46.15 22 13.23 49.64 22 8.62 50.14 22 12.38 47.63 19 10.35 ISLOCA 37.09 22 12.85 43.04 22 15.84 44.42 22 13.49 43.87 22 13.26 42.16 22 14.40 LOOP 18.22 22 10.24 41.89 22 12.85 46.88 22 13.46 48.61 22 10.57 45.34 22 11.23 LOFW 20.76 22 10.29 38.67 22 14.15 50.41 22 13.48 51.84 22 12.59 -- 0 0.00 SGOF 26.17 22 11.98 33.22 22 11.90 42.56 22 12.41 45.67 22 15.49 41.67 3 10.14 All Scenarios 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07

Table F-8 Summary Table of Subjective Workload data: Crew Size by Scenario Period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Crew Size Mean I N ISD Mean N ISD Mean I N ISD Mean I N -SD MeanI N .1 SD Normal 23.40 60 12.43 38.17 60 13.94 46.09 60 12.93 46.82 60 13.61 42.18 39 12.86 Minimum 27.18 50 13.00 43.50 50 13.91 47.61 50 12.20 49.48 50 12.32 48.52 27 9.90 AllGroups 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07

F-4NUREG/IA-0137


Table F-9 Summary table of subjective workload data: plant type by scenario period

Scenario Period I Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 PlantType Mean I N I SD Mean I N I SD Mean I N I SD Mean N FSD Mean I N I SD Conventional 22.98 60 12.97 34.80 60 11.98 41.91 60 11.63 41.29 60 10.30 38.21 33 8.97 Advanced 27.68 50 12.16 47.55 50 13.42 52.63 50 11.17 56.12 50 11.34 51.34 33 11.26 AllGroups 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07

Table F-10 Summary table of subjective workload data: position by scenario period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Position Meanl N I SD Mean 7-N K SD Mean I N I SD MeanI N ISD Mean N SD CRS 26.73 40 14.77 45.13 40 14.22 50.94 40 12.67 52.21 40 12.11 45.03 24 10.31 RO 24.74 35 12.22 39.06 35 16.44 45.93 35 12.91 46.57 35 13.90 47.42 21 14.29 TO 23.66 35 10.86 36.95 35 9.83 42.89 35 10.92 44.70 35 12.26 41.83 21 11.43 All Groups 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07

Table F-11 Summary table of subjective workload data: position by plant type by scenario period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Position Plant Type Mean N SD Mean I N I SD Mean I N SD Mean N SD Mean I N ISD CRS Conventional 28.50 20 16.07 42.25 20 11.97 47.25 20 12.51 46.42 20 10.61 39.70 11 8.29 CRS Advanced 24.96 20 13.54 48.00 20 15.95 54.63 20 12.01 58.00 20 10.86 49.55 13 9.91 RO Conventional 20.13 20 11.12 29.27 20 10.63 38.46 20 8.31 39.00 20 10.39 39.17 11 8.50 RO Advanced 30.89 15 11.12 52.11 15 13.54 55.89 15 11.17 56.67 15 11.46 56.50 10 14.08 BOP Conventional 20.33 20 9.63 32.88 20 9.74 40.02 20 12.17 38.44 20 8,19 35.76 11 10.29 BOP Advanced 28.11 15 11.10 42.39 15 7.12 46.71 15 7.83 53.06 15 11.96 48.50 10 8.84

AllGroups 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07

F-5NUREG/IA-0137


Table F-12 Summary table of workload data: crew size by plant type by scenario period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Crew Size Plant Type Mean I N ISD Mean I N ISD MeanI N SD Mean N SD Mean I N ISD Normal Conventional 18.67 30 10.22 31.18 30 11.12 40.71 30 12.86 39.36 30 10.94 33.75 18 6.32 Normal Advanced 28.14 30 12.78 45.17 30 13.06 51.47 30 10.72 54.28 30 11.91 49.40 21 12.71 Minimum Conventional 27.30 30 14.12 38.42 30 11.88 43.11 30 10.33 43.21 30 9.41 43.56 15 8.89 Minimum Advanced 27.00 20 11.46 51.13 20 13.48 54.37 20 11.87 58.87 20 10.07 54.72 12 7.46

AllGroups 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07

Table F-13 Summary table of subjective workload data: crew size by position by scenario period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Crew Size Operator Mean I N I SD Mean I N S MSD Mean N SD Mean I N Mean IN SD Normal CRS 19.96 20 10.94 38.42 20 12.23 46.54 20 11.85 49.08 20 13.63 39.68 13 9.95 Normal RO 24.33 20 13.82 39.10 20 17.89 47.75 20 14.19 46.92 20 16.34 45.38 13 17.32 Normal BOP 25.92 20 12.19 37.00 20 11.47 43.98 20 13.02 44.46 20 10.52 41.47 13 10.30 Minimum CRS 33.50 20 15.23 51.83 20 13.06 55.33 20 12.17 55.33 20 9.74 51.36 11 6.66 Minimum RO 25.28 15 10.13 39.00 15 14.89 43.50 15 10.97 46.11 15 10.30 50.73 8 6.98 Minimum BOP 20.66 15 8.22 36.89 15 7.50 41.42 15 7.44 45.03 15 14.64 42.40 8 13.83

AllGroups 25.12 110 12.77 40.59 110 14.12 46.78 110 12.57 48.03 110 13.05 44.77 66 12.07

Table F-14 Summary table of team interaction data: scenario by scenario period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Scenario Mean I N ISD Mean I N ]SD MeanI N I SD Mean I N I SD Mean I N ISD. SGTR 5.20 16 1.05 5.59 16 0,88 5.56 16 1.23 4.55 16 1.56 4.44 14 1.38 ISLOCA 4.72 16 1.28 4.72 16 0.98 4.98 16 1.07 4.83 16 1,11 4.54 16 1.28 LOFW 4.84 16 0.98 5.46 16 0.83 5.06 16 1.12 5.22 16 1.21 4.80 16 1.16 LOOP 5.04 16 1.17 4.61 16 1.22 4.38 16 1.60 4.43 16 1.37 -- 0 -SGOF 5.02 16 1.00 5.25 16 0.84 5.36 16 0.88 4.85 16 1.01 -- 0 -All Scenarios 4.97 90 1.09 5.13 90 1.02 5.07 90 1.24 4.78 90 1.26 4,60 46 1.25

F-6NUREG/IA-0137


Table F-15 Summary table of team interaction data: crew size by scenario period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Crew Size Mean N SD Mean I N I SD Mean I N ISD Mean I N I SD Mean I N J'SD Normal 5.05 40 0.94 5.14 40 0.98 5.36 40 1.09 4.79 40 1.14 4.59 24 1.17 Minimum 4.87 40 1.22 5.11 40 1.07 4.78 40 1.33 4.76 40 1.39 4.61 22 1.36 All Groups 4.97 80 1.09 5.13 80 1.02 5.07 80 1.24 4.78 80 1.26 4.60 46 1.25

Table F-16 Summary table of team interaction data: plant type by scenario period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Plant Type Mean I N ISD Mean N SD Mean N SD Mean N SD Mean N SD

Conventional 4.62 40 1.00 4.86 40 1.13 4.56 40 1.39 4.24 40 1.34 3.84 22 1.13 Advanced 5.32 40 1.07 5.39 40 0.83 5.57 40 0.82 5.31 40 0.93 5.30 24 0.92 All Groups , 4.97 80 1.09 5.13 80 1.02 5.07 80 1.24 4.78 80 1.26 4.60 46 1.25

Table F-17 Summary table of team interaction data: crew size and plant type by scenario period

Scenario Period 1 Scenario Period 2 Scenario Period 3 Scenario Period 4 Scenario Period 5 Crew Size Plant Type Mean I N I SD Mean I N ISD Mean I N SD Mean I N [SD Mean I N I SD Normal Conventional 4.75 20 0.99 5.01 20 1.22 5.19 20 1.41 4.47 20 1.42 3.98 12 1.23 Normal Advanced 5.36 20 0.80 5.28 20 0.65 5.52 20 0.63 5.11 20 0.66 5.20 12 0.73 Minimum Conventional 4.48 20 1.01 4.72 20 1.03 3.93 20 1.06 4.01 20 1.25 3.67 10 1.03 Minimum Advanced 5.27 20 1.31 5.50 20 0.97 5.63 20 0.99 5.51 20 1.11 5.40 12 1.10

All Groups 4.97 80 1.09 5.13 80 1.02 5.07 80 1.24 4.78 80 1.26 4.60 46 1.25

F-7NUREG/IA-0137


Table F-18 Summary table of rated crew performance data by scenario

Scenario Mean N SD SGTR 6.97 23 2.03 ISLOCA 6.95 23 1.52 LOFW 7.50 24 1.60 LOOP 6.78 24 2.26 SGOF 7.19 24 1.62 All Scenarios 7.08 118 1.81

Table F-19 Summary table of rated crew performance data: crew size by plant type

Crew Size Plant Type Mean N SD Normal Conventional 7.30 30 1.47 Normal Advanced 7.47 30 1.28 Minimum Conventional 5.57 30 2.01 Minimum Advanced 8.04 28 1.46

All Groups 7.08 118 1.81

F-8NUREG/IA-0137


Table F-20 Summary table of objective performance rating data by scenario

Announcements and Notifications Important Task Performance Cooldown and Stabilization Scenario Mean N SD Mean N SD Mean N SD SGTR 3.63 8 0.74 3.13 8 1.55 2.75 8 1.39 ISLOCA 3.38 8 1.30 3.00 8 1.20 3.25 8 1.49 LOFW 3.63 8 1.51 3.00 8 1.31 3.25 8 1.16 LOOP 3.25 8 1.58 2.88 8 1.55 2.63 8 1.30 SGOF 2.13 8 1.25 2.50 8 1.20 2.88 8 0.83 All Scenarios 3.20 40 1.36 2.90 40 1.32 2.95 40 1.22

Table F-21 Summary table of objective performance rating data: crew size and plant type by scenario

Announcements and Notifications Important Task Performance Cooldown and Stabilization Crew Size Plant Type Mean N SD Mean N SD Mean N SD Minimum Conventional 2.60 10 1.51 1.80 10 0.79 1.80 10 0.63 Normal Conventional 3.50 10 1.08 2.00 10 0.94 2.60 10 1.17 Minimum Advanced 3.60 10 1.35 4.00 10 0.94 3.50 10 1.08 Normal Advanced 3.10 10 1.45 3.80 10 0.79 3.90 10 0.74

All Groups 3.20 40 1.36 2.90 40 1.32 2.95 40 1,22

F-9NUREG/IA-0137

NRC FORM 335 U.S. NUCLEAR REGULATORY COMMISSION 1. REPORT NUMBER

(2-89) (Assigned by NRC, Add Vol., Supp., Rev., NRCM 1102, and Addendum Numbers, If any.) 3201,3202 BIBLIOGRAPHIC DATA SHEET

(See instnuctions on the reverse)

2. TITLE AND SUBTITLE NUREG/IA-0137

A Study of Control Room Staffing Levels for Advanced Reactors 3. DATE REPORT PUBLISHED

MONTH I YEAR

November 2000 4. FIN OR GRANT NUMBER

5. AUTHOR(S) 6. TYPE OF REPORT

B. P. Hallbert, A. L. Sebok, D. Morisseau Technical

7. PERIOD COVERED (inclusive Dates)

8. PERFORMING ORGANIZATION - NAME AND ADDRESS (If NRC, provide Division. Office or Region, U.S. Nuclear Regulatory Commission. and mailing address. if contractor, provide name and mailing address.)

Institutt for Energiteknikk Kjell Hangset P.O. Box 173, N-1751 Halden NORWAY

9. SPONSORING ORGANIZATION - NAME AND ADDRESS (If NRC, type "Same as above'; ifcontractor, provide NRC Division, Office or Region, U.S. Nuclear Regulatory Commission, and mailing address.)

Divsion of Systems Analysis and Regulatory Effectiveness

Office of Nuclear Regulatory Research U.S. Nuclear Regulatory Commission

Washington, DC 20555-0001 10. SUPPLEMENTARY NOTES

11. ABSTRACT (200 words or less)

This report documents the results of an empirical study of operator and plant performance in simulator based settings. The simulator settings were designed to be representative of conventional and advanced plants. The advanced plant design employed passive systems. The control room architectures were also designed to represent both plant types. Two control room staffing configurations were employed in each plant setting: a staffing configuration reflecting the requirements of 10 CFR 50.54 (m); and a staffing configuration that involved a reduced number of control room operators. A series of five design basis scenarios, relevant for both conventional and advanced passive plants were chosen to evaluate the effects of plant type and crew size on operator performance. The scenarios were: 1) steam generator tube rupture with a stuck open steam generator safety relief valve in the affected steam generator, preceded by a fire in the turbine hall; 2) interfacing systems loss of coolant accident with compounded instrument failures due to the incident; 3) sustained total loss of feedwater; 4) loss of off-site power with a single steam generator safety relief valve stuck open, and; 5) steam generator overfill. Eight crews of operators from the Loviisa nuclear power station in Loviisa, Finland participated in the study: four crews in the conventional plant setting; and four crews in the advanced plant setting. Measures of objective performance were obtained, including ratings of crew performance and transient management. Measures of operator workload, situation awareness, and team interaction were obtained repeatedly during each scenario. The findings of the study revealed a number of effects of crew size and plant type, and their combination on operator performance. This report documents the study and discusses the implications and issues raised by this performance-based evaluation of control room staffing requirements for advanced passive plants.

12. KEY WORDS/DESCRIPTORS (List words or phrases that will assist researchers in locating the report.) 13. AVAILABILITY STATEMENT

OECD Halden unlimited

human factors 14. SECURITY CLASSIFICATION

control room (This Page)

crew size unclassified

human performance (This Report)

unclassified 15. NUMBER OF PAGES

16. PRICE

NRC FORM 335 (2-89) This form was electronically produced by Elite Federal Forms, Inc.

kpaPrinted

1n recycled• pFaper

Federal Recycling Program

UNITED STATES NUCLEAR REGULATORY COMMISSION

WASHINGTON, D.C. 20555-0001

years