UNITED STATES NUCLEAR REGULATORY COMMISSION

REGION III 2443 WARRENVILLE ROAD, SUITE 210

LISLE, IL 60532-4352

November 29, 2011 EA-11-243 Mr. Anthony Vitale Vice-President, Operations Entergy Nuclear Operations, Inc. Palisades Nuclear Plant 27780 Blue Star Memorial Highway Covert, MI 49043-9530 SUBJECT: PALISADES NUCLEAR PLANT - NRC SPECIAL INSPECTION TEAM (SIT)

REPORT 05000255/2011014 PRELIMINARY YELLOW FINDING

Dear Mr. Vitale:

On October 28, 2011, the U.S. Nuclear Regulatory Commission (NRC) completed a Special Inspection at your Palisades Nuclear Plant. The inspection was conducted to evaluate the circumstances surrounding the loss of the left train of direct current (DC) power on September 25, 2011. Based on the risk and deterministic criteria specified in Management Directive 8.3, “NRC Incident Investigation Program,” a special inspection was initiated in accordance with Inspection Procedure 93812, “Special Inspection.”

The Special Inspection Charter (Attachment 2 of the Enclosure) provides the basis and focus areas for the inspection.

The enclosed inspection report documents the inspection results, which were discussed with you and other members of your staff at the exit meeting on October 28, 2011. The determination that the special inspection would be conducted was made on September 27, 2011, and the on-site inspection commenced the same day.

The inspection examined activities conducted under your license as they relate to safety, compliance with the Commission’s rules and regulations, and with the conditions of your license. The inspectors reviewed selected procedures and records, conducted field walkdowns, and interviewed personnel.

This report documents a finding that has preliminarily been determined to be Yellow or a finding of substantial safety significance. As documented in Section 4OA5 of this report, the loss of the left train of DC power (one of two trains of DC power onsite) on September 25, 2011, caused the loss of two preferred alternating current (AC) busses which caused a reactor trip and transient, and was directly related to maintenance activities your staff performed on DC Distribution Panel D11-2. Based on our assessment of the information, the maintenance work instructions for the scheduled work were not adequate and were not followed by your staff for the work performed. This finding was assessed based on the best available information,

A. Vitale - 2 -

including influential assumptions, using the applicable Significance Determination Process (SDP). During the maintenance activities, a horizontal bus bar slipped out of an electrician’s hand, causing an electrical fault and the loss of one of the two trains of DC power distribution (reference Figure 7 in Attachment 5 of the Enclosure). Each train was comprised of a set of batteries, two chargers, instrumentation, and two inverters (reference Figure 1 in Attachment 4 of the Enclosure). The inverters provided AC power to the preferred AC busses, which in turn provided power to approximately 50 percent of the control room indications and controls.

In addition to the reactor trip and turbine trip that occurred at 3:06 p.m. on September 25, 2011, the loss of the left train of DC power coincident with the loss of both preferred AC busses led to a safety injection actuation signal, main steam isolation signal, containment high radiation signal, containment isolation signal, auxiliary feedwater actuation signal, and containment high pressure alarm. The protection circuitry actuated as a result of the loss of the left train of DC power and was not required to mitigate a degraded or abnormal condition of the reactor. The reactor operators followed the emergency operating procedures and general operating procedures to restore primary and secondary systems to normal, which occurred at approximately 11:48 p.m. on September 25, 2011. The reactor was placed in Hot Shutdown (Mode 4), at 11:06 p.m. on Monday, September 26, 2011. Because of the actions taken following the event, no current safety concern exists.

This finding is also an apparent violation of NRC requirements and is being considered for escalated enforcement action in accordance with the NRC Enforcement Policy. The current Enforcement Policy can be found at the NRC’s Web site at http://www.nrc.gov/reading-rm/doc-collections/enforcement.

In accordance with Inspection Manual Chapter (IMC) 0609, we intend to complete our evaluation using the best available information and issue our final determination of safety significance within 90 days of the date of this letter. The SDP encourages an open dialogue between the NRC staff and the licensee; however, the dialogue should not impact the timeliness of the staff’s final determination.

Before the NRC makes its enforcement decision, we are providing you an opportunity to either: (1) present to the NRC your perspectives on the facts and assumptions used by the NRC to arrive at the finding and its significance at a Regulatory Conference, or (2) submit your position on the finding to the NRC in writing. If you request a Regulatory Conference, it should be held within 30 days of the receipt of this letter and we encourage you to submit supporting documentation at least one week prior to the conference in an effort to make the conference more efficient and effective. If a conference is held, it will be open for public observation. The NRC will also issue a press release to announce the conference. If you decide to submit only a written response, such submittal should be sent to the NRC within 30 days of the receipt of this letter. If you decline to request a Regulatory Conference or to submit a written response, you relinquish your right to appeal the final SDP determination; in that, by not doing either you fail to meet the appeal requirements stated in the Prerequisite and Limitation Sections of Attachment 2 of IMC 0609.

A. Vitale - 3 -

Please contact John Giessner at (630) 829-9619 and in writing within 10 days of the date of this letter to notify the NRC of your intended response. If we have not heard from you within 10 days, we will continue with our significance determination and enforcement decision. The final resolution of this matter will be conveyed in separate correspondence. Since the NRC has not made a final determination in this matter, no Notice of Violation is being issued for this inspection finding at this time. Please be advised that the number and characterization of the apparent violation described in the enclosed inspection report may change as a result of further NRC review.

In addition to the preliminary Yellow finding, one NRC-identified Severity Level IV violation, one self-revealed finding, and five NRC-identified findings were identified during the Special Inspection. The Severity Level IV violation and four findings involved violations of NRC requirements, and because of the very low safety significance and because they are entered into your corrective action program, the NRC is treating these findings as non-cited violations (NCVs), in accordance with Section 2.3.2 of the NRC Enforcement Policy.

If you contest the subject or severity of these NCVs, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001, with a copy to the Regional Administrator, U.S. Nuclear Regulatory Commission - Region III, 2443 Warrenville Road, Suite 210, Lisle, IL 60532-4352; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the Resident Inspector Office at the Palisades Nuclear Plant. In addition, if you disagree with the cross-cutting aspect assigned to any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region III, and the NRC Resident Inspector at the Palisades Nuclear Plant.

Finally, based on the facts gathered during the special inspection, the inspectors determined that a possible cause for the September 25, 2011, event was that the Palisades organization did not establish safety policies amongst the management team and employees, which reinforced that nuclear safety was an overriding priority. Specifically, several of the organizational decisions demonstrated in this event were not consistent with the established nuclear safety policies and procedures at the site. In addition, production and schedule goals were not developed, communicated, and implemented in a manner that reinforced nuclear safety on September 25, 2011, as demonstrated by the organization’s performance during the execution of this emergent work.

A. Vitale - 4 -

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records System (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely, /RA/ Steven West, Director Division of Reactor Projects

Docket No. 50-255 License No. DPR-20

Enclosure: Inspection Report 05000255/2011014 w/Attachments: 1. Supplemental Information 2. Special Inspection Team Charter 3. Palisades Event Timeline 4. Simplified Diagrams of Palisades 125-Volt DC System 5. Images Of Palisades 125-Volt Dc System During/Following Maintenance 6. Permission to Utilize Graphics/Visual Aids 7. List of Major Affected Equipment 8. Phase 3 Significance Determination Process Detailed Analysis for the Failure

to Have Adequate Work Instructions

cc w/encl: Distribution via ListServ

Enclosure

U. S. NUCLEAR REGULATORY COMMISSION

REGION III

Docket No: 50-255 License No: DPR-20

Report No: 05000255/2011014

Licensee: Entergy Nuclear Operations, Inc.

Facility: Palisades Nuclear Plant

Location: Covert, MI

Dates: September 27, 2011, through October 28, 2011

Inspectors: R. Krsek, Senior Resident Inspector, Kewaunee (Lead) A. Dahbur, Senior Reactor Engineer A. Scarbeary, Reactor Engineer

Approved by: John B. Giessner, Chief Branch 4 Division of Reactor Projects

Enclosure

TABLE OF CONTENTS

SUMMARY OF FINDINGS .......................................................................................................... 1

REPORT DETAILS ..................................................................................................................... 9

Summary of the Plant Event ................................................................................................... 9 Inspection Scope ...................................................................................................................10

4OA5 Other Activities – Special Inspection (93812) ...................................................10 4OA6 Meetings, Including Exit....................................................................................51

ATTACHMENT 1

SUPPLEMENTAL INFORMATION ............................................................................................. 1 KEY POINTS OF CONTACT .................................................................................................. 1 LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED ....................................................... 2 LIST OF DOCUMENTS REVIEWED ...................................................................................... 3

ATTACHMENT 2

PALISADES SPECIAL INSPECTION CHARTER ....................................................................... 1

ATTACHMENT 3

PALISADES EVENT TIMELINE .................................................................................................. 1

ATTACHMENT 4

SIMPLIFIED DIAGRAMS OF PALISADES 125-Volt DC System ................................................. 1 Figure 1 - Simplified Diagram of Palisades 125-Volt DC System ............................................ 1 Figure 2 - Simplified Diagram of Left Train 125-Volt DC Bus De-energized During Event ....... 2

ATTACHMENT 5

IMAGES OF PALISADES 125-Volt DC SYSTEM DURING/FOLLOWING MAINTENANCE ........ 1 Figure 3 - DC Distribution Panel D11-2 With Breaker 72-120 Removed ................................. 1 Figure 4 - DC Distribution Panel D11-2 With Breakers 72-119 and 72-120 Removed ............. 2 Figure 5 - DC Distribution Panel D11-2 With Breakers 72-119, 72-120 and 72-121 Removed 3 Figure 6 - DC Distribution Panel 11-2 With Breakers 72-119, 72-120, 72-121 and 72-123

Removed ................................................................................................................ 4 Figure 7 - DC Distribution Panel D11-2 Immediately Following the Electrical Fault ................. 5 Figure 8 - DC Distribution Panel D11-2 Inspection and Repair Several Days After the Event . 6

Enclosure

ATTACHMENT 6

PERMISSION TO UTILIZE GRAPHICS/VISUAL AIDS ............................................................... 1

ATTACHMENT 7

LlST OF MAJOR AFFECTED EQUIPMENT ............................................................................... 1

REVIEW OF DE-ENERGIZED EQUIPMENT ON 9/25/11 PLANT TRIP ................................. 1

ATTACHMENT 8

PHASE 3 SIGNIFICANCE DETERMINATION PROCESS DETAILED ANALYSIS FOR THE FAILURE TO HAVE ADEQUATE WORK INSTRUCTIONS ........................................................ 1

1 Enclosure

SUMMARY OF FINDINGS

IR 05000255/2011014, 09/27/2011 10/28/2011; Palisades Nuclear Plant; Inspection Procedure 93812, Special Inspection.

This report covers a 32-day period (September 27 – October 28, 2011) of on-site inspection and in-office review through October 28, 2011. A team, comprised of three regional inspectors, conducted this special inspection. The inspectors identified one finding preliminarily determined to be Yellow or a finding of substantial safety significance, which is also an apparent violation. In addition, the inspectors identified one NRC-Identified Severity Level IV violation, five NRC-Identified findings, and one self-revealed finding of very low safety significance. The Severity Level IV violation and four findings involved violations of NRC requirements. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter (IMC) 0609, “Significance Determination Process” (SDP). Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRC’s program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, “Reactor Oversight Process,” Revision 4, dated December 2006.

A.

Cornerstone: Initiating Events

NRC-Identified and Self-Revealed Findings

• Preliminary Yellow

The inspectors determined that the finding was more than minor in accordance with Inspection Manual Chapter (IMC) 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," dated December 24, 2009, because it was associated with the Procedure Quality and Human Performance attributes of the Initiating Events

. A preliminary finding of substantial safety significance (Yellow) and an associated apparent violation of Title 10 of the Code of Federal Regulations (10 CFR) Part 50, Appendix B, Criterion V, “Instructions, Procedures, and Drawings,” was self-revealed on September 25, 2011. The licensee failed to ensure that the work instructions on safety-related 125-Volt direct current (DC) Distribution Panel D11-2 through Work Orders (WO) 291194-01, 291210-01, and 291123-03, all activities that affected quality, were adequate for the scheduled work; and the licensee failed to ensure the work instructions were followed by your staff for the affected activity. As a result of these deficiencies, during the work in the field on the energized Panel D11-2, a positive horizontal bus bar rotated and contacted a negative horizontal bus bar. This in turn, caused an electrical fault in Panel D11-2 and a complete loss of the left train 125-Volt DC safety-related system coincident with both 120-Volt preferred alternating current (AC) power sources, busses Y-10 and Y-30. These electrical losses resulted in a reactor and turbine trip at approximately 3:06 p.m. on September 25, 2011, coincident with a Safety Injection Actuation Signal, Main Steam Isolation Signal, Containment High Radiation Signal, Containment Isolation Signal, Auxiliary Feedwater Actuation Signal, and Containment High Pressure Alarm (no actuation signal). This issue was documented in the licensee’s corrective action program as CR-PLP-2011-04822 and at the end of this inspection, the licensee continued to perform a root cause evaluation to determine the causes of the event and develop corrective actions. As a remedial corrective action on September 25, 2011, the licensee repaired the damage caused to Panel D11-2 to restore it to service and addressed the operability and effect of the transient on other components.

2 Enclosure

Cornerstone, and adversely affected the cornerstone objective to limit the likelihood of those events, that upset plant stability and challenge critical safety functions during power operations. Specifically, the failure to create work orders in accordance with procedures and the failure to perform work in accordance with prescribed instructions directly resulted in the loss of the left train of 125-Volt DC coincident with two preferred AC power sources. The Phase 1 Significance Determination Process (SDP) evaluation determined that the finding contributed to both the likelihood of a reactor trip and the likelihood that mitigation equipment or functions would not be available. Therefore, the finding required a Phase 2 evaluation using IMC 0609 Appendix A, “Determining the Significance of At-Power Reactor Inspection Findings,” which determined the significance was a Yellow Finding. The SRAs used the Palisades SPAR [Simplified Plant Analysis Risk] model, Revision 8.17, for the SDP Phase 3 evaluation. The result of the Phase 3 SDP is a preliminary finding of substantial safety significance (Yellow) with an estimated conditional core damage probability (CCDP) of 1.6E-5. The inspectors also determined this finding had a cross-cutting aspect in the area of human performance, work practices, because the licensee failed to communicate and ensure human error prevention techniques were used, such as holding formal pre-job briefings, self and peer checking, and proper documentation of activities. The licensee also failed to ensure that these techniques were used commensurate with the risk of the assigned task, such that work activities are performed safely. Finally, during these maintenance activities, the inspectors concluded that licensee personnel proceeded in the face of uncertainty or unexpected circumstances (H.4(a)). (Section 4OA5.3.b.1)

• Green

The inspectors determined that the finding was more than minor in accordance with IMC 0612 "Power Reactor Inspection Reports," Appendix B, "Issue Screening," because, if left uncorrected, the performance deficiency could have the potential to lead

. A finding of very low safety significance and associated non-cited violation of Technical Specification 5.4.1 was identified by the inspectors for the failure to implement procedures for combating emergencies and other significant events as required by Regulatory Guide (RG) 1.33, Section 6. Specifically, during the performance of EOP-1.0, “Standard Post-Trip Actions,” in response to a loss of the left train 125-Volt DC bus and subsequent plant trip, the control room reactor operators failed to immediately take the contingency action in the “response not obtained” column for an immediate action step that could not be met due to the partial loss of control room indications. Procedure EOP-1.0, Step 2.b. of Section 4.0, “Immediate Actions,” required the reactor operator in the control room to verify that the Main Generator was disconnected from the grid, and if that step cannot be completed, then the operator was required to connect a jumper across the corresponding relay terminals in the control room panel to open the output breakers. These actions were not immediately taken by the control room staff at the time of this event. Once the control room staff was aware of the “closed” status of the Main Generator output breakers from an update provided by an extra reactor operator who was in contact with transmission system operator, the action step was then taken by the turbine-side reactor operator to jumper the relay terminals in the control room panel to open the breakers. This issue was documented in the licensee’s corrective action program as CR-PLP-2011-06081 and at the end of the special inspection the licensee was still performing an evaluation to determine the causes and to develop corrective actions. As a remedial corrective action on October 28, 2011, each operations crew received a briefing about operator expectations, the usage of human performance tools and procedures, and an overview of the recent events.

3 Enclosure

to a more significant safety concern. In particular, this loss of 125-Volt DC event could have become a more significant event with further complications and plant issues. The inspectors determined that the finding could be evaluated using the SDP in accordance with IMC 0609, “Significance Determination Process,” Attachment 0609.04, “Phase 1 - Initial Screening and Characterization of Findings,” Table 4a, for the Initiating Events Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Transient Initiator question of contributing to both the likelihood of a reactor trip and the likelihood that mitigating equipment or functions would not be available and screened the finding as having very low safety significance (Green). The finding had a cross-cutting aspect in the area of human performance related to the cross-cutting component of Work Practices, in that the licensee communicates human error prevention techniques, such as peer-checking, and that these techniques are used commensurate with the risk of the assigned task, such that work activities are performed safely (H.4(a)). (Section 4OA5.4.b.1)

Cornerstone: Mitigating Systems

• Green

The inspectors determined that the finding was more than minor in accordance with IMC 0612, “Power Reactor Inspection Reports,” Appendix B, “Issue Screening,” dated December 24, 2009, because it was associated with the Procedure Quality and Human Performance attributes of the Mitigating Systems Cornerstone. This adversely affected the cornerstone objective, in that, the failure to utilize human error reduction tools impacted the availability, reliability and capability of systems that responded to initiating events to prevent undesirable consequences. Specifically, the failure to utilize human performance tools directly contributed to the inadequate work planning and preparation scheduled for Panel D11-2 on September 25, 2011. The inspectors determined that the finding could be evaluated using the significance determination process in accordance with IMC 0609, “Significance Determination Process,” Attachment 0609.04, “Phase 1 - Initial Screening and Characterization of Findings,” Table 4a, for the Mitigating Systems Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Mitigating Systems questions and screened the finding as having very low safety significance (Green). The finding has a cross-cutting aspect in the area of human performance, work practices, because the licensee failed to ensure personnel work practices supported human performance through defining and effectively communicating expectations regarding procedural compliance coincident with plant personnel following procedures.

. A finding of very low significance was identified by the inspectors for the licensee’s failure to implement Procedure EN-HU-102, “Human Performance Tools,” which established standards and expectations for the use of specific human performance tools with the goal to improve personnel and plant performance through human error reduction. The inspectors identified that Procedure EN-HU-102 was not implemented for the work performed on September 25, 2011, to install a temporary modification and to address a non-conforming condition associated with Panel D11-2. Implementation of the procedure for Panel D11-2 scheduled work required the use of Procedure EN-OP-116, “Infrequently Performed Tests or Evolutions,” and performance of an infrequently performed tests and evolution pre-job brief, which the inspectors determined was not performed for the work on September 25, 2011. No violation of NRC requirements occurred. The licensee documented this condition in its corrective action program as CR-PLP-2011-04822 and CR-PLP-2011-04981. At the end of this inspection, the licensee continued to perform a root cause evaluation to determine the causes of the event and develop corrective actions.

4 Enclosure

Specifically, the licensee personnel failed to reference or implement procedures with human performance tools, which, if implemented, would have required an IPTE brief for the work performed on Panel D11-2 on September 25, 2011 (H.4(b)). (Section 4OA5.3.b.2)

• Green

The issue affected the Mitigating Systems Cornerstone because the 125-Volt DC system work plan development was overseen by the non-covered workers. The inspectors determined that the finding was more than minor in accordance with IMC 0612, “Power Reactor Inspection Reports,” Appendix B, “Issue Screening,” dated December 24, 2009, because it revealed weaknesses that, if left uncorrected, could lead to more significant safety concerns associated with overseeing work on safety-related equipment. In addition, the inspectors concluded that the failure to implement working hour limitations for non-covered workers in Procedure EN-FAP-OM-006 was more than an isolated instance. The inspectors and Senior Reactor Analyst concluded that the use of IMC 0609, Appendix M, “Significance Determination Process Using Qualitative Criteria,” was the appropriate method for determining the significance. In accordance with IMC 0609, Appendix M, management review of this issue determined that this finding was of very low safety significance since the performance deficiency did not directly contribute to the event, as the non-covered workers were involved with the planning and not actual implementation of the work performed on September 25, 2011, on Panel D11-2. The finding has a cross-cutting aspect in the area of human performance, resources, because the licensee failed to ensure that personnel and other resources were available and adequate to assure nuclear safety; specifically, sufficient qualified personnel were available to maintain work hours within working hour guidelines (H.2(b)). (Section 4OA5.3.b.3)

. A finding of very low significance was identified by the inspectors for the licensee’s failure to implement Procedure EN-FAP-OM-006, “Working Hour Limits for Non-Covered Workers,” which established standard fleet guidance for working hour limits for Entergy non-covered (not covered under 10 CFR 26) workers as defined in EN-OM-123, “Working Hour Limits.” The inspectors identified that at least two non-covered managers on the nightshift, involved with the work planning and oversight of troubleshooting repair efforts for Panel D11-2, had not followed the standards for work hour limits and did not initiate condition reports when the work hour limits were exceeded, as required by Procedure EN-FAP-OM-006. Specifically, the inspectors identified that the Duty Station Manager worked approximately 25 consecutive hours from September 23 through September 24, and greater than 72 hours in a 7-day period. The electrical superintendent exceeded the administrative limits of 16 hours in 24-hour period, 26 hours in 48-hour period, 72 hours in a 7 day period, and greater than a 10-hour break between work periods over a consecutive 19-day period of work. No violation of NRC requirements occurred. The licensee documented this condition in its corrective action program as CR-PLP-2011-05095 and CR-PLP-2011-05116. At the end of this inspection, the licensee continued to perform an apparent cause evaluation and extent of condition to determine extent of the problem and causes for the performance deficiency in order to develop corrective actions.

• Green. A finding of very low safety significance and associated NCV of TS 5.4.1 was identified by the inspectors for the failure to establish a procedure for combating emergencies and other significant events as required by RG 1.33, Section 6. Specifically, Section 6 states, in part, that the loss of electrical power (and/or degraded power sources) is a safety-related activity that should be covered by written procedures,

5 Enclosure

and TS 5.4.1 required, in part, that written procedures be established, implemented, and maintained to cover the activities in RG 1.33. The design and licensing basis of the plant includes the loss of a single train of DC power. Although the site has multiple procedures to address the loss of the DC system and individual preferred AC sources, the procedures did not integrate to provide a response that minimized challenges to plant safety. The site has three separate procedures that were used in this event for the loss of one DC bus and loss of one preferred AC source (two sources were lost during the event, hence two of these procedures were used); but not one inclusive procedure to cover the loss of both preferred AC sources simultaneously. The procedures that the crew worked through were inadequate to respond in a timely fashion to changing plant conditions caused by the loss of the left train of DC power. This issue was documented in the licensee’s corrective action program as CR-PLP-2011-06209 and, at the end of the special inspection, the licensee was still performing an evaluation to determine the causes and to develop corrective actions.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," because the finding was associated with the Mitigating Systems Cornerstone attribute of Procedure Quality, and adversely impacted the objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the attribute of procedure quality, areas to measure, lists operating (post-event) procedures such as abnormal operating procedures, standard operating procedures, emergency operating procedures, and can include off-normal procedures, as being items that should be established and maintained to ensure the cornerstone objective is met. The inspectors determined that the finding could be evaluated using the significance determination process in accordance with IMC 0609, “Significance Determination Process,” Attachment 0609.04, “Phase 1 - Initial Screening and Characterization of Findings,” Table 4a, for the Mitigating Systems Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Mitigating Systems questions and screened the finding as having very low safety significance (Green). The finding does not have an associated cross-cutting aspect since the last known operating experience for a loss of the 125-Volt DC system occurred in 1981 at the Millstone Nuclear Generating Station. (Section 4OA5.4.b.2)

• Green. A finding of very low safety significance and associated non-cited violation of Title 10 of the Code of Federal Regulations (10 CFR) Part 50, Appendix B, Criterion V, “Instructions, Procedures, and Drawings,” was identified by the inspectors for the failure to implement a procedure for an activity affecting quality. Procedure EN-OP-104, “Operability Determination Process,” required an assessment of the operability for structures, systems, and components (SSCs) when degraded or non-conforming conditions were identified and establishment of compensatory measures were needed to, “ensure, maintain, and enhance future operability.” Specifically, the inspectors identified that the operability evaluation for the 125-Volt DC system, completed on September 30, 2011, did not contain two compensatory measures necessary to ensure the operability of the system. It was also identified that the 50.59 pre-screening (process applicability determination) for the temporary modification, which was also a compensatory measure for the operability evaluation, was not clearly written and did not adequately describe the evaluation of the modification or the bases for this decision. This issue was documented in the licensee’s corrective action program as CR-PLP-2011-04988 and CR-PLP-2011-04965 and at the end of the special inspection the licensee was still performing an evaluation to determine the causes and to develop

6 Enclosure

corrective actions. The licensee’s remedial corrective actions included revising the 50.59 pre-screening to clearly address the effect of the compensatory measures on other aspects of the facility, prohibiting maintenance on the energized 125-Volt DC busses, and issuing additional site guidance for the operation of battery chargers.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," because the finding was associated with the Mitigating Systems cornerstone attribute of Equipment Performance, and adversely impacted the objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the attribute of equipment performance impacted the availability and reliability of the 125-Volt DC system. The inspectors determined that the finding could be evaluated using the SDP in accordance with IMC 0609, “Significance Determination Process,” Attachment 0609.04, “Phase 1 - Initial Screening and Characterization of Findings,” Table 4a, for the Mitigating Systems Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Mitigating Systems questions and screened the finding as having very low safety significance (Green). The finding had a cross-cutting aspect in the area of human performance related to the cross-cutting component of Decision-Making, because the licensee did not adequately conduct an effectiveness review of a safety-significant decision to verify the validity of the underlying assumptions and identify possible unintended consequences, as necessary (H.1(b)). (Section 4OA5.5.b.1)

• Green

The inspectors determined that the finding was more than minor in accordance with IMC 0612, “Power Reactor Inspection Reports,” Appendix B, “Issue Screening,” dated December 24, 2009, because it was associated with the Mitigating Systems Cornerstone attribute of Equipment Performance, and affected the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, lack of coordination between

. A self-revealed finding of very low safety significance (Green) and associated NCV of Title 10 CFR Part 50, Appendix B, Criterion III, “Design Control,” and Criterion IV, “Procurement Document Control,” was identified for the licensee’s failure to establish measures to ensure that the applicable regulatory requirements and design bases were correctly translated into specifications and instructions. In addition, the licensee failed to establish measures to assure that the applicable regulatory requirements and design bases, which were necessary to assure adequate quality, were suitably included or referenced in the documents for procurement of equipment. Specifically, 125-Volt DC Breakers 72-01 and 72-02 were purchased and installed with thermal overloads and instantaneous trips enabled. The design basis stated that the breakers were non-automatic and only actuated manually. As a result, on September 25, 2011, when an electrical fault occurred on Panel D11-2, the left train 125-Volt DC bus was lost, because the instantaneous trip device on Breaker 72-01 automatically actuated, propagating the fault through the bus, which resulted in a reactor and turbine trip, and plant transient. This issue was documented in the licensee’s corrective action program as CR-PLP-2011-4835 and CR-PLP-2011-4965 and at the end of the special inspection the licensee was still performing an evaluation to determine the causes and to develop corrective actions. As a remedial corrective action prior to plant startup, the licensee implemented a temporary modification to increase the breaker instantaneous trips and performed an operability evaluation, with compensatory actions for the 125-Volt DC system.

7 Enclosure

Panel D11-2 protective device (FUZ/D11-2) and Breaker 72-01 resulted in the loss of the left 125-Volt DC bus and two preferred AC power sources and complicated plant shutdown during the reactor trip on September 25, 2011, when an electrical fault occurred while working on Panel D11-2. The risk assessment associated with the event on September 25, and the complication caused by the breaker opening, is evaluated and described in the preliminary Yellow AV. The inspectors determined the finding, related to the design deficiency, could be evaluated using the SDP in accordance with IMC 0609, “Significance Determination Process,” Attachment 0609.04, “Phase 1 - Initial Screening and Characterization of findings,” Table 4a for the Mitigating Systems cornerstone. The inspectors answered “Yes” to Question 1 in Column 2. Therefore, the inspectors determined that this finding could be screened as having very low safety significance (Green), because the finding was a design deficiency confirmed not to result in loss of operability or functionality of a system safety function. In addition, the inspectors also determined that the finding affected the fire protection safe shutdown strategies. Therefore, screening under IMC 0609, Appendix F, “Fire Protection Significance Determination Process,” was required. Based on review of IMC 0609, the inspectors concluded that the finding represented a moderate degradation within the post-fire safe shutdown category and performed a Phase 2 analysis. Based on the licensee’s evaluation for the loads the inspectors determined that this finding screened as having very low safety significance (Green) per Task 2.3.5, screening check for lack of fire ignition sources and fire scenarios. The inspectors did not identify a cross-cutting aspect associated with this finding because Breakers 72-01 and 72-02 were procured and installed in 1981 and therefore, the finding was not reflective of licensee’s current performance. (Section 4OA5.6.b.1)

• Severity Level IV

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," dated December 24, 2009, Block 7, Figure 2, because reporting failure violations are considered to be violations that potentially impact the regulatory process and are dispositioned using traditional enforcement. The underlying technical issue was required to be evaluated using the SDP and is assessed separately in Section 4OA5.6.b.1 of this report as a separate Green finding. In accordance with Section 6.1.d.2 of the NRC Enforcement Policy, this violation was categorized as Severity Level IV because the underlying technical issue was evaluated by the SDP and determined to be of very

. A Severity Level (SL) IV non-cited violation of 10 CFR 50.72(b)(3)(ii)(B) was identified by the inspectors for the failure to notify the NRC as soon as practical and in all cases within eight hours of the occurrence of any event or condition that results in the nuclear power plant being in an unanalyzed condition that significantly degrades plant safety. Specifically, the licensee failed to report on September 26, 2011, within eight hours an Appendix R noncompliance that was identified in DC shunt trip Breakers 72-01 and 72-02 for the 125-Volt DC system following the reactor trip that occurred on September 25, 2011. The licensee’s preliminary analysis demonstrated that if a shunt trip breaker automatically opened due to fire induced fault currents, then the licensee’s Appendix R credited equipment may have been lost unexpectedly, an unanalyzed condition that significantly degrades plant safety. This issue was documented in the licensee’s corrective action program as CR-PLP-2011-05263 and at the end of the special inspection, the licensee continued to perform a causal evaluation in order to develop corrective actions. As a remedial corrective action, the licensee made the required event notification in Event Notification Number 47322 on October 5, 2011.

8 Enclosure

low safety significance. In addition, NRC Enforcement Policy, dated July 12, 2011, Section 6.9.d.9, states, in part, that an example of an SL IV violation is the licensee’s failure to make a report required by 10 CFR 50.72. Cross-cutting aspects were evaluated in the underlying ROP finding (4OA5.6.b.1). (Section 4OA5.2.b.1)

B.

No findings were identified.

Licensee-Identified Violations

9 Enclosure

REPORT DETAILS

At 3:06 p.m. on Sunday, September 25, 2011, the licensee experienced an automatic reactor and turbine trip due to the loss of one of two trains of 125-Volt direct current (DC) power. Specifically, the loss of the left train DC busses D-10L and D-10R caused the loss of preferred alternating current (AC) busses Y-10 and Y-30 (reference Figures 1 and 2 in Attachment 4 of this report). At the time of the reactor trip, licensee maintenance personnel were working on the left train 125-Volt DC system on DC Distribution Panel D11-2 (Panel D11-2) in order to repair a previously identified issue with individual breakers inside the panel.

Summary of the Plant Event

During the work, a horizontal bus bar slipped out of an electricians hand, causing an electrical fault on the left train of DC power distribution (reference Figure 7 in Attachment 5 of this report). Each train was comprised of a set of batteries, two chargers, instrumentation, and two inverters (reference Figure 1 in Attachment 4 of this report). The inverters provided AC power to the preferred AC busses, which in turn provided power to approximately 50 percent of the control room indications and controls.

In addition to the reactor trip and turbine trip, the loss of the left train of DC power coincident with the loss of both preferred AC busses led to a Safety Injection Actuation Signal (SIAS), Main Steam Isolation Signal (MSIS), Containment High Radiation Signal, Containment Isolation Signal (CIS), Auxiliary Feedwater Actuation Signal (AFAS), and Containment High Pressure Alarm (no actuation signal). The protection circuitry actuated as a result of the loss of the left train of DC power and was not required to mitigate a degraded or abnormal condition of the reactor. The MSIS caused the condenser to not be available for decay heat removal and the follow-up performed by the inspectors determined that for approximately the first hour of the event, secondary side steam pressure was controlled by the secondary side code safety valves. Reactor operators (ROs) implemented the Emergency Operating Procedures (EOPs), while maintenance personnel performed troubleshooting activities, diagnosed and corrected the loss of the 125-Volt DC system, which took approximately an hour. Once power was restored to the preferred AC busses, secondary side steam pressure was controlled through the use of the atmospheric steam dump valves (ASDVs).

During the transient, the ROs encountered additional complications that included: a rising containment sump level with an increasing, unidentified primary coolant system (PCS) leak rate of less than 10 gallons per minute (gpm), that was later determined to be from the actuation of a chemical and volume control system relief valve in containment; increasing PCS level in the pressurizer that reached a maximum of 98 percent (the PCS was approximately 9 minutes from being placed in a solid condition); increasing steam generator (SG) ‘A’ level, which reached approximately 98 percent; and, the actuation of suction and discharge pressure relief valves for the charging pumps, which displaced volume control tank water into the charging pump cubicles located in the auxiliary building.

The ROs followed the EOPs, Off-Normal Procedures (ONPs) and General Operating Procedures (GOPs) to restore the primary and secondary systems to normal, which occurred at approximately 11:48 p.m. on September 25, 2011. The reactor was placed in Hot Shutdown (Mode 4), at 11:06 p.m. on Monday, September 26, 2011.

10 Enclosure

Based on the deterministic and conditional risk criteria specified in Management Directive 8.3, “NRC Incident Investigation Program,” a special inspection was initiated in accordance with NRC Inspection Procedure (IP) 93812, “Special Inspection Team.” The team reviewed technical and design documents, control room and engineering logs, procedures, maintenance records, and corrective action documents; interviewed site personnel and consultants; and, performed plant walkdowns of plant equipment. All documents reviewed by the inspectors during the inspection are listed in Attachment 1 of this report.

Inspection Scope

This report contains the following additional attachments to assist in the understanding of the events that occurred on September 25, 2011:

• Attachment 2 is the Special Inspection Charter; • Attachment 3 is a timeline of events developed by the inspectors; • Attachment 4 contains simplified diagrams of the Palisades 125-Volt DC system; • Attachment 5 contains images of Palisades DC Distribution Panel D11-2 during

and after the maintenance activities that took place on September 25, 2011; • Attachment 6 provides permission to utilize the graphics and images contained in

the report; • Attachment 7 provides a list of major plant equipment affected by the event; and, • Attachment 8 provides the SDP analysis.

4OA5 Other Activities – Special Inspection

In accordance with the Charter, the following items were reviewed:

(93812)

.1

a.

Establish A Historical Sequence Of Events Related To Reactor Trip On September 25, 2011, Including The Maintenance Activities Which Led To The Planned Work On September 23, 2011.

The inspectors reviewed control room logs, plant parameter recordings, plant procedures, corrective action documents, maintenance work order (WO) and work request history, and engineering design changes as part of the inspection activities. The inspectors also conducted interviews with plant ROs who responded to the event and other individuals who were directly involved to conduct fact-finding and determine the relevant sequence of events associated with the maintenance activities and plant response on September 25, 2011.

Inspection Scope

A narrative of the facts regarding the event that occurred is detailed below in Section 4OA5.1.b.1. Licensee performance deficiencies detailed in the narrative are addressed in other sections of this report. In addition, a detailed historical timeline of activities developed by the inspectors is contained in Attachment 3 of this report.

The inspectors emphasized fact-finding to fully understand the circumstances surrounding the event and probable causes. NRC IP 93812, “Special Inspection,” Section 03.01.b required the inspectors to assess the safety culture component considerations, as detailed in Inspection Manual Chapter (IMC) 0310, “Components Within the Cross-Cutting Areas,” specifically, the other safety culture components.

11 Enclosure

Section 4OA5.1.b.2 contains the inspectors’ assessment of the safety culture component considerations.

b.

b.1

Findings and Observations

During Refueling Outage (RFO) 21 in the fall of 2010, the licensee performed extensive maintenance on Panel D11-2, which included the replacement of 10 breakers inside the panel, as well as other maintenance activities. Any performance deficiencies associated with the maintenance conducted during RFO 21, which led to the instrument air transient that occurred on September 23, 2011, will be addressed in the fourth quarter NRC Integrated Inspection Report (IR) (05000255/2011005).

Detailed Narrative of the Plant Event

On Thursday, September 22, 2011, the licensee commenced a WO to troubleshoot the inoperative green indicating lights for Door MZ-50 (Emergency Airlock Lights). Through this investigation, all interlocks, indication lights, and limit switches for this door were found to be satisfactory. Since this door was due for its technical specification (TS) required surveillance test on Monday, September 26, 2011, the decision was made to conduct more troubleshooting activities to identify the cause of the indicating light issue.

On Friday, September 23, 2011, the licensee concluded through its troubleshooting efforts that DC Breaker 72-123 on Panel D11-2, which powered the indicating lights for Door MZ-50, the emergency airlock, required replacement. The breaker was then successfully replaced. While restoring the foreign material exclusion (FME) barrier for Panel D11-2 (a metal strip down the center of the panel), control room ROs observed flickering of the turbine generator (TG) voltage regulator, a generator over-excitation alarm, a loss of indication for multiple containment isolation valves (CIVs), and a loss of power for the instrument air compressors standby start feature in the plant (the instrument air compressors internal “sleep mode” feature remained available to automatically start the air compressors). The feedwater (FW) purity air compressor continued to supply the necessary air to equipment through a control valve that failed open upon the intermittent loss of power and cross-connected the two systems, as designed. These alarms and indications were experienced due to a suspected intermittent connection issue with breakers on Panel D11-2. Additional troubleshooting Friday night and early Saturday morning revealed no voltage on the load side of Breaker 72-119 (Main Control Room Panel EC-13 Loads), misalignments in the mounting of Breakers 72-119, 72-121 (Exciter Control Panel E01), and 72-123. Troubleshooting also identified a small air gap between the positive horizontal bus bar and the positive connection on Breaker 72-119 that resulted in a slightly elevated temperature (approximately two degrees Fahrenheit hotter than the other breakers in Panel D11-2) as evidenced by thermography.

On Saturday, September 24, 2011, the licensee continued troubleshooting activities for these breakers, developed a temporary modification (TM) to power the TG voltage regulator (which needed its power source maintained when Breaker 72-121 was removed), and developed a work package to implement the TM. In order to correct the discrepancies identified during the troubleshooting, the licensee concluded that breakers in Panel D11-2 required removal. In order to maintain power to the TG voltage regulator during breaker removal, the licensee developed the TM to provide power to this

12 Enclosure

component through a spare breaker in Panel D11-2. These troubleshooting and development work activities continued into the night shift.

On Sunday, September 25, 2011, at approximately 5:00 a.m., the TM work package for Breaker 72-121 was completed and a medium risk pre-job brief was held. The electrical maintenance superintendent felt that the workers were too tired to execute the TM work, and was concerned about a gap in front line electrical supervision coverage from 6:00 a.m. to 8:00 a.m., due to work hour limits for covered workers. Therefore, the electrical superintendent decided the work would not occur on the current nightshift, but the workers and managers continued with the pre-job brief and WO review, to ensure any issues were identified and addressed before the work commenced on dayshift.

At approximately 7:00 a.m., the off-going electrical superintendent performed a turnover with the oncoming mechanical superintendent (also the acting maintenance manager for that day). The turnover focused on the TM pre-job brief, the importance of the removal sequence of Breakers 72-119, 72-120, 72-121 and 72-123, the importance of insulating the negative horizontal bus bar after the removal of the second breaker (72-119), and addressing the as-found condition of the horizontal breaker bus bars. During the turnover, the oncoming mechanical superintendent questioned why the off-going electrical superintendent did not upgrade an electrician to a lead, via an administrative process, so that the TM work could proceed without waiting for a qualified front line supervisor (FLS) to arrive at the site later at 8:00 a.m. The electrical superintendent stated that this action would not be done because of the importance of having an FLS at the job site, since the work being performed might result in a plant transient or turbine trip. The electrical superintendent then stayed at the plant and re-performed the previously conducted turnover with the newly-arrived electrical FLS. Sometime prior to the turnovers, the electrical superintendent realized that the operational risk associated with this work might be greater than a medium risk evolution; however, this concern was never verbalized during the turnover with either the oncoming mechanical superintendent or electrical FLS.

Sometime after 8 a.m., a pre-job brief was performed for the TM WO for Breaker 72-121, that included the FLS, three electricians, Duty Station Manager (DSM), and engineering staff. The workers utilized the medium risk pre-job brief form that had been filled out previously at 5:00 a.m., rather than a new form, as required by site procedure. Operations staff were then briefed separately by the maintenance crew on the work to be performed, and the ROs prepared contingencies to address a failure of Panel D11-2, should this occur during the maintenance. At approximately 11:00 a.m., TM Number 31973 was installed to power breaker loads from Breaker 71-121, “Main Generator Voltage Regulator Control Power,” from the spare Breaker 72-127. At this point preparation of the WO package for the removal, inspection, and repair of Breakers 72-119, 72-120, 72-121, and 72-123 was not completed.

The WO packages were ready for use and delivered to the maintenance break area at approximately 12:45 p.m., where an informal pre-job brief commenced for this work. Those present at the informal pre-job brief included the three electricians performing the work, the FLS, the mechanical superintendent, the DSM, and three engineers. The informal brief was conducted as a reverse-brief, meaning the electricians led the discussion and emphasized the importance of the breaker removal sequence, electrical safety, insulating the exposed horizontal bus bars, and the steps of the WO.

13 Enclosure

The informal pre-job brief did not include a detailed discussion of critical steps or how, if discovered, galled threads on the horizontal bus bars would be addressed, either through chasing of the screw holes (to re-establish the screw hole threads) in the bus or removal of the horizontal bus stabs. The conduct of the informal briefing, including not using the required pre-job brief checklist, was not challenged by the supervisors or workers present at the time.

Following the informal pre-job brief, the electricians separately discussed with the FLS their concern that the chasing of the horizontal bus bar screw holes could not be done safely in the energized bus. The FLS agreed with their concern, and the decision was made to remove any damaged horizontal bus bars and work on them in the electrical maintenance shop. The FLS had an expectation that, following the removal of the breakers, work would stop prior to the repair of the horizontal bus bars. This expectation was not verbalized at the pre-job brief, nor was it communicated to the workers prior to the initiation of the field work. The electricians also challenged whether the three WOs governing the work adequately addressed the work they just discussed, specifically removal of the bus bars, and the FLS responded the work order did not need revision.

After discussing these issues, the workers proceeded to the field to begin work, accompanied by their FLS, the mechanical superintendent, three engineers, the DSM, and the Shift Manager (SM). Panel D11-2 was energized and considered operable during all the maintenance that was scheduled to be performed on the four breakers. After the removal of each breaker, work was stopped and the observing individuals (FLS, superintendent, DSM, SM, and engineers) entered into the work area to look at what had been performed and to take pictures (reference Attachment 5). Following the removal of the second breaker (72-119), at approximately 2:15 p.m., the negative horizontal bus bar was energized and fully exposed. The negative horizontal bus bar should have been insulated at this time, but was not. Gapping between the bus bar and breaker stab, minor indications of arcing, and evidence of cross-threading at the bus bar hole were identified at this time for Breaker 72-119. The electricians, FLS, DSM, or engineers did not question why the negative bus bars were not insulated after the removal of Breaker 72-119.

Following the removal of the fourth breaker and after the observers inspected Panel D11-2, the electricians began removal of the first positive horizontal bus stab located in the upper left-hand corner of the bus (reference Figure 4 in Attachment 5). The workers were concerned that the single screw holding the horizontal bus stab would fall into the bus cubicle, so they devised a plan to hold onto the horizontal bus stab with one hand, loosen the screw with the other hand, and slowly back out the screw. At the time, the workers believed they met the intent of a note in the WO package to insulate the bus stab because the electrician performing the work was wearing insulated gloves. The electrician felt positive control was maintained by holding the bus bar.

At approximately 3:06 p.m. on September 25, after initially loosening the screw for the horizontal bus bar, the worker saw a flash from the area of the screw. Upon seeing a flash, the worker’s hands instinctively recoiled from the work area for personal protection, thereby letting go of the horizontal positive bus bar. The horizontal positive bus bar was loosened enough so that it rotated downward and contacted the negative bus bar causing a significant arc and melting of the bus bars (reference Figure 7 of Attachment 5). The worker was not injured during the incident because the appropriate protective clothing was worn.

14 Enclosure

An electrical fault occurred from this contact between the positive and negative bus bars. This fault revealed a latent design deficiency of the installed shunt trip Breaker 72-01, in that, the installed breaker had thermal and instantaneous trip settings actuate due to the fault. The installed thermal and instantaneous trip settings on the shunt trip breaker were not listed as a characteristic of the breaker in the plant’s updated final safety analysis report (UFSAR), drawings, or calculations. The design and licensing basis documentation described the shunt trip breakers as only a non-automatic, manually-operated breaker without any thermal or instantaneous protection devices. Per the system design, the transient, that occurred due to the electrical fault, should have only resulted in the loss of Panel D11-2, due to the actuation of fuse FUZ-D11-2 (which should have isolated the fault from the rest of the loads on the bus). However, because the shunt trip Breaker 72-01, which was installed in 1981 for both the left and right trains of 125-Volt DC busses, had thermal and instantaneous protection for overcurrent, the breaker tripped free on an overcurrent setting when the electrical fault on Panel D11-2 occurred and the entire left train 125-Volt DC bus was lost (reference Figure 2 of Attachment 4).

Subsequent to the loss of the DC bus, the electrical transient resulted in the failure of the in-service left train battery charger (due to its high current contribution to the fault) and associated left train inverters. This led to the loss of two preferred 120-Volt alternating current (AC) power sources (busses Y-10 and Y-30) that supply power for annunciation and instrumentation in the control room.

The loss of the left train 125-Volt DC bus and two preferred AC power sources resulted in an instantaneous reactor and turbine trip caused by a Reactor Protection System (RPS) actuation. These trips were coincident with the automatic actuation of the following systems and components: a Safety Injection Actuation Signal (automatically started right train Emergency Core Cooling Systems (ECCS)); Main Steam Isolation Signal (closed main steam isolation valves (MSIVs) and main feedwater regulating valves); Containment High Radiation Signal (started control room heating, ventilation, and air conditioning (HVAC) system); Containment Isolation Signal (closed right channel containment isolation valves); Auxiliary Feedwater (AFW) Actuation Signal (started AFW Pumps P-8B and P-8C, and opened all flow control valves to the full open position); and a Containment High Pressure Alarm (no actuation signal). The protection circuitry actuated as a result of the loss of the left train of DC power and was not required to mitigate a degraded or abnormal condition of the reactor. Also, at this time, the following major equipment was also affected by the loss of the left train 125-Volt DC system: the Primary Coolant Pumps (PCPs) ‘A’ and ‘C’ coasted down; nonsafety-related busses 1A and 1E did not fast transfer on loss of load, and hence their loads lost power; and the atmospheric steam dump valve master controller lost power, which made them unavailable to provide a steaming path to reduce primary side pressure (reference Attachment 7 for a detailed list of major plant equipment affected by the event).

The ROs responded to the reactor trip and turbine trip by immediately entering Emergency Operating Procedure (EOP) 1.0, “Standard Post-Trip Actions.” The ROs failed to perform an immediate action contingency step to verify the main generator output breakers were open; however, this was identified 10 minutes after the turbine trip and the actions were appropriately taken at that time. Following the completion of EOP 1.0, the ROs were directed to EOP 9.0, “Functional Recovery,” due to the loss of

15 Enclosure

both preferred AC busses Y-10 and Y-30. The ROs were directed into several ONPs from EOP-9.0, which prioritized the ROs’ recovery actions.

Throughout this event, the ROs’ response was complicated by the fact that the loss of power resulted in a loss of indications and annunciators in the control room, which would have aided in diagnosing issues in both the primary and secondary systems, and containment. ROs implemented the EOPs, while maintenance personnel troubleshot, diagnosed and corrected the loss of the 125-Volt DC system, which took approximately 50 minutes. Once power was restored to the preferred AC busses, secondary side steam pressure was controlled through the use of the ASDVs.

During the transient, the ROs encountered additional complications that included: a rising containment sump level with an increasing, unidentified PCS leak rate of less than 10 gpm, that was later determined to be from the actuation of a chemical and volume control system relief valve in containment; increasing PCS level in the pressurizer that reached a maximum of 98 percent (the PCS was approximately 9 minutes from being placed in a solid condition); increasing SG ‘A’ level, which reached approximately 98 percent; fire alarms in the Turbine Driven AFW pump room due to steam emission from the fully open steam admission valve to the turbine; and the actuation of suction and discharge pressure relief valves for the running charging pumps, which displaced volume control tank water into the charging pump cubicles located in the auxiliary building. The operations crew thought that the secondary side steam pressure during the transient was controlled by the two remaining Train ‘A’ ASDVs via the quick open feature. However, the inspectors noted during the inspection, that not only was the primary coolant system average temperature not high enough to cause actuation of the quick open feature, due to the additional cooling of the steam generators by the AFW system, but also, the ASDV circuitry did not have power due to the loss of the left train of 125-Volt DC. Upon review of the plant process computer data, the inspectors confirmed that for approximately the first hour of the event, secondary side steam pressure was controlled by the secondary side code safety valves.

At 7:46 p.m., the plant met the requirements to exit EOP 9.0 with the restoration of the two preferred 120-Volt AC power sources and all necessary systems meeting their safety function checks. Upon the exit of EOP 9.0, the ROs were able to reset the Safety Injection Actuation Signal, and restore systems and components that were in an abnormal configuration due to the losses of power during the event. Later that evening, with the left train loads restored, pressurizer level was reduced below the TS limit. Finally, at 11:48 p.m.,pressurizer level was returned to normal.

On Monday, September 26, a WO was initiated to remove Breaker 72-122 and use those bus tie stabs to replace the ones on Breaker 72-119 that were damaged during the event. This work was executed without incident. The plant began a PCS cooldown to Mode 4 (Hot Shutdown) on Monday and entered Mode 5 (Cold Shutdown) at 6:33 a.m. on Tuesday, September 27, 2011. Throughout the remainder of the week, the licensee completed work to install, restore, and verify the condition of Panel D11-2, and to inspect, validate, correct and test all of the components associated with the left train 125-Volt DC system affected by the transient. Following the completion of additional analysis and engineering evaluations of the 125-Volt DC system, the reactor was returned to full power on Monday, October 3, 2011.

16 Enclosure

b.2

The inspectors assessed the licensee’s safety-conscious work environment (SCWE) during interviews of over 30 plant employees from the operations, maintenance, outage and planning, and engineering organizations. The individuals interviewed provided a distribution across the various departments at all levels of the organization. The inspectors concluded, based on the interviews, that the licensee had an environment where people were free to raise issues without fear of retaliation.

Assessment of Safety Culture Component Considerations

The inspectors also considered the other safety culture components contained in IMC 0310, “Components Within the Cross-Cutting Areas,” in the assessment of the circumstances surrounding the September 25, 2011, event and probable causes. The inspectors’ assessment included the components of accountability, continuous learning environment, organizational change management and safety policies.

The inspectors concluded that one of the possible causes for the event that occurred on September 25, 2011, was related to safety policies.

Specifically, safety policies and related training establish and reinforce that nuclear safety is an overriding priority, in that, organizational decisions and actions at all levels of the organization were consistent with the policies. Production, cost and schedule goals were developed, communicated, and implemented in a manner that reinforces the importance of nuclear safety. Facts gained by the inspectors as a result of the inspection, supported an assessment that organizational decisions and actions at all levels were not consistent with plant policies at times, and that the production and schedule goals were not developed, communicated or implemented in a manner that consistently reinforced the importance of nuclear safety. Some examples include the following:

• On September 25, 2011, the acting maintenance manager questioned the off-going nightshift electrical superintendent, as to whether an electrician scheduled to perform the work could be “stepped up to a lead,” so that the electricians could immediately begin field work without a qualified front line supervisor (FLS). A 2-hour gap in FLS coverage existed due to covered workers, work hour rules. The Electrical Superintendent remained at the site to directly conduct a turnover with the oncoming FLS, in order to ensure work did not begin prior to the FLS’s arrival. The inspectors determined that the acting maintenance manager’s actions demonstrated that meeting the schedule was potentially more important than having a qualified FLS for these critical evolutions to ensure nuclear safety;

• On September 25, 2011, a formal pre-job brief was not conducted for the breaker maintenance, as required by Procedure EN-HU-102, “Human Performance Tools.” The inspectors determined that no one present at the informal brief, which took place on September 25, reviewed or required the use of the human performance tools procedure. Those individuals included the acting maintenance manager, DSM, three engineering observers, the FLS, the Operations SM, and the three electricians, all of whom were present at the informal brief prior to the start of work. The inspectors determined that the managers and workers actions demonstrated that the organizational decisions and actions in preparation for this

17 Enclosure

work were driven primarily by the schedule, rather than taking the additional time necessary to validate the work preparation was done correctly;

• On September 25, 2011, upon removal of the second breaker from Panel D11-2,

the DSM, acting maintenance manager, SM, FLS, and three engineering staff, all visually observed that both the positive and negative bus bars were exposed on the energized electrical Panel D11-2. However, no individual present at the job site questioned the lack of insulation of the positive and negative horizontal bus bars on Panel D11-2 and work proceeded to the removal of the third breaker. Another opportunity was available to question this practice following the removal of the third breaker from Panel D11-2. Visual evidence of this is provided in Attachment 5, which were the pictures taken by engineering staff in chronological order, starting at the beginning of work. There were several times when the work in the field should or could have been stopped following individual breaker removals; however, the inspectors concluded that the electricians rationalized in the field why work could continue and management observers did not question the workers. The inspectors determined that these actions demonstrated the organizational focus for the work in the field was on meeting the schedule and the work was not consistently implemented in a manner that reinforced the importance of nuclear safety consistent with plant policies and procedures;

• On September 25, 2011, during the performance of work in the field prior to the electrical fault, the DSM recommended to the workers that a spare horizontal bus bar be swapped in Panel D11-2; however, the DSM did not require a revision to the work order prior to the start of work, additional worker briefings or reconsideration of the operational risk to reflect these changes to the existing work orders utilized in the field. The inspectors determined that these actions demonstrated that the management observer’s in-field interactions with the workers, without validating plant policies and procedures were followed for changes to work orders, emphasized schedule adherence instead of implementing site procedures to ensure nuclear safety; and,

• On September 25, 2011, the work orders utilized for the work to remove the

breakers from Panel D11-2 did not actually reflect the work being performed in the field; however, the field work continued without either the workers, FLS or management observers ensuring that plant procedures were followed for work orders and work control for the work to be performed. The inspectors did determine that prior to the start of the work, the electricians questioned the FLS, as to whether the removal of the breaker horizontal bus stabs was allowed by the existing work order, even though there was not an action step describing this process. The FLS told the electricians that no changes to the work order were needed. The inspectors determined that the managers and workers actions in the field were driven by the schedule and the organization did not take the time to revise the work orders, in accordance with site procedures, to ensure that plant policies and procedures were followed.

Based on the facts detailed above, the inspectors concluded that the work on September 25, 2011, was performed with a focus on completion of the tasks on schedule, without ensuring all nuclear safety policies were followed. The inspectors determined that the work scope developed following the September 23, 2011, transient was placed on a schedule timeframe that was not commensurate with the significance of

18 Enclosure

the issues. Specifically, the work required to be performed on the breakers for Panel D11-2 was not of such an urgent nature that it was required to be performed quickly without additional time to validate and revalidate assumptions and contingencies.

Therefore, based on the facts gathered during the inspection, the inspectors determined that a possible cause for the September 25, 2011, event was that the Palisades organization did not establish safety policies amongst the management team and employees, which reinforced that nuclear safety was an overriding priority. Specifically, several of the organizational decisions demonstrated in this event were not consistent with the established nuclear safety policies and procedures at the site. In addition, production and schedule goals were not developed, communicated, and implemented in a manner that reinforced nuclear safety on September 25, 2011, as demonstrated by the organization’s performance during the execution of this emergent work.

.2

a.

Review of Reportability Requirements To Confirm Necessary Notifications Were Made Per 10 CFR 50.72 And 10 CFR 50.73 And Possible Emergency Action Levels

The inspectors reviewed control room logs, plant parameter recordings, plant procedures related to reportable events and emergency procedures, corrective action documents, and engineering design and evaluation documents as part of the inspection activities. The inspectors also conducted interviews with plant ROs who responded to the event and individuals directly involved in the event to conduct fact-finding and determine the relevant sequence of events associated with the maintenance and plant response to the event on September 25, 2011.

Inspection Scope

b.

b.1

Findings and Observations

Failure to Report a 10 CFR 50.72 Notification for an 8-hour Non-Emergency Report

Introduction: A Severity Level (SL) IV NCV of Title 10 of the Code of Federal Regulations (10 CFR) Part 50.72(b)(3)(ii)(B) was identified by the inspectors for the failure to notify the NRC, as soon as practical and in all cases within eight hours, of the occurrence of any event or condition that results in the nuclear power plant being in an unanalyzed condition that significantly degrades plant safety. Specifically, the licensee failed to report on September 26, 2011, within eight hours, an Appendix R noncompliance that was identified in DC shunt trip Breakers 72-01 and 72-02, for the 125-Volt DC system following the reactor trip that occurred on September 25, 2011. The licensee’s preliminary analysis demonstrated that if a shunt trip breaker automatically opened due to fire induced fault currents, then the licensee’s Appendix R credited equipment may have been lost unexpectedly, an unanalyzed condition that significantly degrades plant safety. Following the inspectors questions, the licensee made the required event notification (EN) in EN 47322 on October 5, 2011.

Description: The inspectors reviewed the events and engineering evaluations developed as a result of the transient on September 25, 2011. The inspectors determined that immediately following the event on September 25, 2011, the licensee discovered that the shunt trip Breakers 72-01 and 72-02, associated with the left and right train 125-Volt DC busses, respectively, had an instantaneous trip feature enabled, which resulted in a lack of coordination on the associated 125-Volt DC busses. The licensee’s documented Appendix R analysis for the fire protection program assumed that these two breakers

19 Enclosure

were only manually actuated and that no instantaneous trip features existed. The licensee’s preliminary analysis demonstrated that if a shunt trip breaker automatically opened due to fire induced fault currents, then the licensee’s Appendix R equipment credited to address a fire in the plant may have been lost unexpectedly.

On Tuesday October 4, the inspectors questioned the licensee as to why the NRC had not been notified of this potentially unanalyzed condition in accordance with Title 10 CFR 50.72(b)(3)(ii)(B) upon discovery of the condition on September 26, 2011. The licensee’s analysis, based on the information available at that time, concluded that Appendix R credited equipment may have been lost unexpectedly due to this condition, which constituted an unanalyzed condition that significantly degraded plant safety. Licensee personnel subsequently notified the NRC, as required, on October 5 at 6:00 p.m.

Analysis

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," dated December 24, 2009, Block 7, Figure 2, because reporting failure violations are considered to be violations that potentially impact the regulatory process and must be compared to the examples in the NRC Enforcement Policy. The underlying technical issue was required to be evaluated using the SDP and is assessed separately in Section 4OA5.6.b.1 of this report as a separate Green finding.

: The inspectors determined that the failure to report the condition in accordance with 10 CFR 50.72 was a performance deficiency warranting a significance determination.

In accordance with Section 6.1.d.2 of the NRC Enforcement Policy, this violation was categorized as Severity Level IV because the underlying technical issue was evaluated by the SDP and determined to be of very low safety significance. In addition, NRC Enforcement Policy, dated July 12, 2011, Section 6.9.d.9, states, in part, that an example of an SL IV violation is the licensee’s failure to make a report required by 10 CFR 50.72. Cross-cutting aspects were evaluated in the underlying ROP finding.

Enforcement

Contrary to the above, as of September 27, 2011, the licensee failed to report, within eight hours, an Appendix R noncompliance that was identified on September 26, 2011. Specifically, the DC shunt trip Breakers 72-01 and 72-02 for the 125-Volt DC system had an instantaneous trip featured enabled. Had a shunt trip breaker automatically opened due to fire induced fault currents, then the licensee’s Appendix R credited equipment may have been lost unexpectedly. This would result in an unanalyzed condition that significantly degraded plant safety. Because this violation was of very low safety significance, and was entered into the licensee’s corrective action program (CAP), as CR-PLP-2011-05263, this violation is being treated as an SL IV NCV, consistent with Section 2.3.2 of the NRC Enforcement Policy (NCV 05000255/2014011-01; Failure to Report a 10 CFR 50.72 Notification for an 8-hour Non-Emergency Report).

: Title 10 CFR 50.72(b)(3)(ii)(B) requires, in part, that the licensee shall notify the NRC, as soon as practical and in all cases within eight hours, of the occurrence of any event or condition that results in the nuclear power plant being in an unanalyzed condition that significantly degrades plant safety.

20 Enclosure

At the end of this inspection, the licensee continued to perform a causal evaluation for the performance deficiency in order to develop corrective actions. As a remedial corrective action, the licensee made the required event notification in EN 47322 on October 5, 2011.

b.2

The inspectors reviewed the licensee’s actions in response to the September 25, 2011, event and the implementation of the Site Emergency Plan on that day. The SM, who was the Emergency Director, utilized Procedure EI-1, “Emergency Classification and Actions,” to assess whether the plant condition necessitated the activation of the Site Emergency Plan. The SM concluded that based on the plant condition and the Palisades Emergency Action Levels (EALs) and Emergency Action Level Basis document that activation of the Site Emergency Plan was not warranted.

Assessment of Licensee Implementation of the Site Emergency Plan

The inspectors interviewed the SM and Shift Technical Advisor, in addition to the Palisades organization emergency response personnel. The inspectors also reviewed the Site Emergency Plan, Palisades EALs, Procedure EI-1, and Procedure EAL Basis, “Emergency Action Level Technical Basis.” The loss of the left train of 125-Volt DC and subsequent loss of both preferred AC sources from that bus (reference Figure 1 in Attachment 4) resulted in the loss of half of the safety-related systems, approximately 50 percent of the control room indications, less than 15 percent of the control room annunciators and the loss of the plant process computer terminal at the control room supervisors station only. From a review of the Palisades EALs and EAL Basis procedure the inspectors concluded the applicable EALs were in “Category S – System Malfunction,” with subcategories 2, “Loss of DC Power,” and 5, “Loss of instrumentation.”

The only EAL associated with System Malfunction for the Loss of DC Power was EAL SS2.1, which was a Site Area Emergency for less than 105-Volt DC on DC bus Number 1 (D-10) and DC Bus Number 2 (D-20) for greater than or equal to 15 minutes. Based on the event that occurred, the inspectors determined the licensee did not meet this EAL criteria for notification.

The first applicable EAL associated with System Malfunction for Instrumentation was EAL SU5.1, which was an Unusual Event for the unplanned loss of greater than 75 percent of annunciation or indication, Table S-2, on the main control boards for greater than or equal to 15 minutes. Based on the event that occurred, which was only a 50 percent loss of indication and less than 15 percent loss of annunciation, the inspectors determined the licensee did not meet this EAL criteria for notification.

The second applicable EAL associated with System Malfunction for Instrumentation was EAL SA5.1, which was an Alert for the unplanned loss of greater than 75 percent of annunciation or indication, Table S-2, on the main control boards for greater than or equal to 15 minutes and a significant transient in progress, Table S-3. Table S-3 defined a significant transient as one of the following: a turbine runback greater than or equal to 25 percent thermal power; a reactor trip; or, a safety injection actuation signal actuation. Based on the event that occurred, which was only a 50 percent loss of indication and less than 15 percent loss of annunciation, the inspectors determined the licensee did not meet this EAL criteria for notification.

21 Enclosure

The inspectors reviewed and compared the licensee’s NRC approved EALs to the following documents and consulted a regional NRC Regional Emergency Preparedness Specialist:

• Nuclear Energy Institute (NEI) 99-01, Revision 5 Final, “Methodology of Emergency Action Levels,” February 2008, NRC ADAMS Accession Number ML080450149; and

• NRC Regulatory Issue Summary (RIS) 2003-18, Supplement 2, “Use of NEI 99-01, Methodology for Development of Emergency Action Levels, Revision 4, Dated January 2003,” Dated December 12, 2005.

The inspectors and regional Emergency Preparedness Specialist concluded that although the licensee entered Emergency Operating Procedure (EOP) EOP-9, “Functional Recovery,” which had a basic assumption that the facility Emergency Response Organization was activated, the licensee correctly implemented the Palisades Site Emergency Plan for the September 25, 2011, event.

.3

a.

Review The Activities And Human Performance Related To The Maintenance Of The DC Bus To Ensure All Required Plant Procedures And Work Instructions Were Followed

The inspectors reviewed the maintenance and planning activities that led to the maintenance performed on September 25, 2011, on Panel D11-2. The inspectors interviewed licensee personnel from the operations, maintenance, and planning and scheduling departments, as well as, the management team responsible for overseeing the work performed. The inspectors also reviewed the licensee’s procedures that prescribed plant operational risk assessments, maintenance rule risk evaluations, work order planning and development, pre-job briefs, human performance tools and work hour rules for both covered and non-covered workers. A detailed review was performed of the completed work orders for the work performed on September 25, 2011.

Inspection Scope

The inspectors did not include in their review, an analysis of any performance deficiencies associated with the maintenance conducted during RFO 21, which led to the instrument air transient that occurred on September 23, 2011, which precipitated the maintenance performed on September 25, 2011. These activities will be reviewed by the resident inspectors and any performance deficiencies identified will be addressed in the fourth quarter 2011 NRC Integrated Inspection Report (05000255/2011005).

b.

b.1

Findings and Observations

Failure to Have Adequate Work Instructions for Work Performed on Panel D11-2

Introduction: A preliminary finding of substantial safety significance (Yellow) and an associated apparent violation of 10 CFR Part 50, Appendix B, Criterion V, “Instructions, Procedures, and Drawings,” was self-revealed on September 25, 2011, when the licensee failed to ensure that the work performed on safety-related 125-Volt DC Panel D11-2 through Work Orders (WO) 291194-01, 291210-01, and 291123-03, all activities that affected quality, was prescribed by documented instructions or procedures of a type appropriate to the circumstances and accomplished in accordance with the instructions or procedures. As a result of these deficiencies, during the work in the field on the energized Panel D11-2, a positive horizontal bus bar rotated

22 Enclosure

and contacted a negative horizontal bus bar. This in turn, caused an electrical fault in Panel D11-2 and a complete loss of the left train 125-Volt DC safety-related system coincident with both 120-Volt preferred AC power sources, busses Y-10 and Y-30. These electrical losses resulted in a reactor and turbine trip at approximately 3:06 p.m. on September 25, 2011, coincident with a Safety Injection Actuation Signal, Main Steam Isolation Signal, Containment High Radiation Signal, Containment Isolation Signal, Auxiliary Feedwater Actuation Signal, and Containment High Pressure Alarm.

Description

On Saturday, September 24, 2011, the licensee continued troubleshooting activities for these breakers and developed a TM to power the TG voltage regulator (which needed the power source maintained when the associated 72-121 breaker was removed). Licensee personnel also developed a work package to implement the TM. This troubleshooting and development work continued through the night shift into early Sunday morning.

: On Friday, September 23, 2011, the licensee performed troubleshooting and replacement of DC Breaker 72-123 on Panel D11-2 due to the loss of indication lights for Door MZ-50, the emergency airlock. During maintenance restoration of the FME barrier for Panel D11-2 (a metal strip down the center of the panel), control room alarms were received for the TG voltage regulator, generator over-excitation, and a loss of indication for multiple containment isolation valves. In addition, a loss of normal instrument air was experienced in the plant. Troubleshooting performed by electricians Friday night and early Saturday morning identified a 1/16-inch gap between the Panel D11-2 positive horizontal bus stab and the contacts on Breaker 72-119, which could cause the intermittent power loss to the breakers in Panel D11-2.

On Sunday, September 25, 2011, at approximately 5:00 a.m., the TM WO 291209-01, “Install Temporary Modification 31973,” development was completed and a medium risk pre-job brief was held. Due to worker fatigue concerns during the pre-job brief and a gap in FLS coverage from 6:00 a.m. to 8:00 a.m., the decision was made to implement the work package with a new maintenance crew later Sunday morning. However, the pre-job brief and work order review did not identify any additional concerns with this work. Around this time, preparation began on WO 291194-01, “Inspect and Replace Breakers 72-119 and 72-120,” WO 291210-01, “Replace Breaker 72-121,” and WO 291123-03, “Inspect Breaker 72-123 Installation,” to remove and inspect Breakers 72-119, 72-120, 72-121, and 72-123, and repair the Panel D11-2 horizontal bus stabs.

Sometime after 8:00 a.m., a dayshift pre-job brief was performed for the TM work order that included the FLS, three electricians, DSM, and engineering staff. At approximately 11:00 a.m., TM EC 31973 was installed to power breaker loads from Breaker 71-121, “Main Generator Voltage Regulator Control Power,” from the spare Breaker, 72-127. A review by the inspectors did not identify any deficiencies in WO 291209-01, with respect to Procedure EN-WM-105, “Planning,” and the associated procedures referenced for work order planning. The inspectors noted that work order steps were in a logical order to ensure successful work task completion and that critical steps were appropriately identified in the work order. The precautions and limitations appropriately highlighted that tripping of the main generator was a risk if the work steps were not implemented in the correct order. Upon completion of the TM, the work package for the removal, inspection, and repair of Breakers 72-119, 72-120, 72-121 and 72-123 was not yet prepared.

23 Enclosure

Work Order 291194-01, WO 291210-01, and WO 291123-03 were ready and delivered to the maintenance break area at approximately 12:45 p.m., where an informal pre-job brief was held that was not in accordance with EN-HU-102, “Human Performance Tools,” (See Section 4OA5.3.b.2 for additional details). Following the informal pre-job brief, the electricians discussed with the FLS their concern that chasing of the horizontal bus bar screw holes could not be done safely in an energized bus. The FLS agreed with the concern, and the decision was made to remove any damaged horizontal bus bars and work on them in the electrical maintenance shop. The inspectors determined through interviews that at this time, the electricians challenged whether the three work orders governing the work adequately addressed the work just discussed. The FLS responded there was no need to change the work orders. The inspectors determined through interviews that the acting maintenance manager was informed of this change after the start of the work.

The workers proceeded to begin field work and were accompanied by their FLS and several observers. The observers included the acting maintenance manager (normally mechanical superintendent), three engineers, the DSM and the SM. Panel D11-2 was energized and considered operable during all the maintenance that was scheduled to be performed on the four breakers. During the actual performance of work by the electricians, the observers were outside of the established electrical safety boundary and were not able to directly observe the electricians working inside Panel D11-2. After the removal of each breaker, work was stopped, the electrical safety boundary was repositioned and the observers entered the work area to take pictures and observe the as-found condition (actual photographs are contained in Attachment 5). Following the removal of the second breaker, Breaker 72-119, the negative horizontal bus bar was fully exposed and should have been insulated, but was not. The electricians, FLS, DSM, or engineers did not question why the negative bus bars were not insulated following removal of the second breaker, as evidenced by the photographs in Attachment 5.

Interviews with the FLS revealed that he had an expectation that following the removal of the four breakers, work would stop prior to the repair of the horizontal bus bars. However, this expectation was not verbalized at the pre-job brief, nor was it communicated to the workers prior to or during the work, and the work order did not contain a step to halt work following breaker removal. Following the removal of the fourth breaker and after the observers inspected Panel D11-2, the electricians began removal of the first positive horizontal bus stab located in the upper left hand corner of the bus (reference Figure 4 of Attachment 5). The workers were concerned that the single screw holding the positive horizontal bus stab would fall into the energized bus cubicle, so they devised a plan to hold onto the horizontal bus stab with one gloved hand, loosen the screw with the other gloved hand, and then slowly remove the screw. During interviews with the inspectors, the electricians stated that at the time they felt the bus bar was insulated with the workers glove and supported by the worker's other gloved hand.

At approximately 3:06 p.m. on September 25, 2011, after initially loosening the screw for the horizontal bus bar, the worker saw a flash from the area of the cap of the screw after the break-away torque was reached. Upon seeing a flash, the worker’s hands instinctively recoiled from the work area to protect himself, thereby letting go of the horizontal positive bus bar. The horizontal positive bus bar was loosened enough such that it rotated downward and contacted the negative bus bar causing a significant arc and melting of the bus bars as seen in Figure 7 of Attachment 5. The worker was not

24 Enclosure

injured during the incident because the appropriate protective clothing was worn. The electrical fault and subsequent loss of the left train 125-Volt DC bus and two preferred AC power sources resulted in an instantaneous reactor and turbine trip coincident with a Safety Injection Actuation Signal, Main Steam Isolation Signal, Containment High Radiation Signal, Containment Isolation Signal, Auxiliary Feedwater (AFW) Actuation Signal, and Containment High Pressure Alarm (no actuation signal).

Following the event the inspectors interviewed the workers, observers and work planners and reviewed the site procedures that governed work planning. The quality-related procedure that prescribed the actions necessary for safety-related work planning was Procedure EN-WM-105, “Planning,” Revision 9. In addition, several additional procedures that governed work order planning were reviewed including Procedures EN-IS-123, “Electrical Safety,” EN-FAP-WM-011, “Work Planning Standard,” EN-DC-136, “Temporary Modifications,” EN-MA-101, “Conduct of Maintenance,” EN-MA-125, “Troubleshooting Control of Maintenance Activities,” EN-WM-100, “Work Request Generation, Screening and Classification,” EN-WM-101, “On-Line Work Management Process,” EN-WM-102, “Work Implementation and Closeout,” 5.18, “Control of Work Order Instructions, and 10.51, “Writer’s Guideline for Site Procedures.”

The inspectors identified the following deficiencies associated with the written work orders for the work performed on Breakers 72-119, 72-120, 72-121 and 72-123:

• The work instructions did not provide a logical step progression and format throughout the written instructions to minimize confusion in the field. The work instructions were not written in the order performed. Specifically, the work on the four breakers was contained in three work orders that did not identify a logical step progression and format. The instructions were not clear for the work order packages as to the sequence of breaker removal. The work performed in the field was done using all three work orders simultaneously, which was not logical (Step 4.b of Procedure EN-WM-105);

• A critical task analysis was not performed and critical steps were not identified for the work instructions. Work Order 291194-01, WO 291210-01, and WO 291123-03 all contained critical steps that were not identified because the analysis was not performed (Step 3 of EN-FAP-WM-011 invoked by Procedure EN-WM-105);

• Action steps to insulate the horizontal bus bars were contained in a work instruction “Note,” which was not allowed. The work planning standard explicitly stated that action steps were not included in “Notes, Cautions or Warnings,” (Step 23 of EN-FAP-WM-011 invoked by Procedure EN-WM-105);

• Work order steps did not authorize removal of the horizontal bus bars. The work order steps stated, “As needed, CLEAN and TIGHTEN (hand tight) load side bus bar connections, including chasing threads and replacing fasteners.” While performing the work, the electricians attempted to remove the horizontal bus bars, which was not prescribed in the work order (WO 291194-01, WO 291210-01, and WO 291123-03); and,

• Work order steps did not prescribe the proper instructions for checking the tightness of the horizontal bus bar screws, which were required to be torqued to 45-foot pounds (WO 291194-01, WO 291210-01, and WO 291123-03).

25 Enclosure

Consequently, the inspectors determined the that licensee failed to create work orders for the work on Panel D11-2 in accordance with the licensee’s planning process and the work performed in the field was not accomplished in accordance with the prescribed instructions that were provided. Specifically, Procedure EN-WM-105, required, in part, that Work Orders 291194-01, 291210-01, and 291123-03 provided logical step progression and format throughout the written instructions for the work to be performed, identify critical steps to highlight significant action steps contained in the work instructions and not include action steps in Notes, Warnings and Cautions. The three work orders did not provide a logical step progression and format, and did not contain all the requisite steps that were to be performed in the field. Although all three work orders contained critical steps in the work instructions, none of the critical steps were identified in the work orders. Finally, the work orders included action steps in the work instruction Notes to, “Insulate or support load side bus bars to ensure they do not short.” In addition, the electricians performing work in the field, attempted to remove a positive horizontal bus bar in Panel D11-2, which was not a prescribed step in the work instructions. As a result of these deficiencies, during the work in the field the positive horizontal bus bar rotated and contacted the negative horizontal bus bar that in turn caused an electrical short and a loss of the left train 125-Volt DC safety-related system. This resulted in a reactor and turbine trip at approximately 3:06 p.m. on September 25, 2011, coincident with a Safety Injection Actuation Signal, Main Steam Isolation Signal, Containment High Radiation Signal, Containment Isolation Signal, Auxiliary Feedwater Actuation Signal, and Containment High Pressure Alarm (no actuation signal).

Analysis

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," dated December 24, 2009, because it was associated with the Procedure Quality and Human Performance attributes of the Initiating Events Cornerstone, and adversely affected the cornerstone objective to limit the likelihood of those events, that upset plant stability and challenge critical safety functions during power operations. Specifically, the failure to create work orders in accordance with procedures, and the failure to perform work in accordance with prescribed instructions directly resulted in the loss of the left train of 125-Volt DC coincident with two preferred AC power sources. The complications caused by the as-found, latent design deficiency associated with the instantaneous trips for shunt trip Breaker 72-01 are evaluated in this SDP. The inspectors evaluated the finding under the Initiating Events Cornerstone using IMC 0609, "Significance Determination Process," Attachment 0609.04, "Phase 1 - Initial Screening and Characterization of Findings." Using Table 4a, “Characterization Worksheet for IE, MS, and BI Cornerstones,” dated January 10, 2008. The Phase 1 SDP evaluation determined that the finding contributed to both the likelihood of a reactor trip and the likelihood that mitigation equipment or functions would not be available. Therefore, the finding required a Phase 2 evaluation using IMC 0609 Appendix A, “Determining the Significance of At-Power Reactor Inspection Findings.”

: The inspectors determined that the failure to create work orders for the work on Panel D11-2 that were in accordance with the licensee’s procedures and the failure to perform the field work in accordance with the prescribed instructions, was a performance deficiency warranting a significance evaluation.

26 Enclosure

The Senior Reactor Analysts (SRAs) used the Palisades Risk-Informed Inspection Notebook, Revision 2.1a, to perform the Phase 2 evaluation. The Loss of DC Bus 10 worksheet (LDC10) was solved assuming that the initiator occurred (IEL = 0), the turbine-driven AFW (TDAFW) pump was failed because the ROs had to close the steam admission valve, and recovery was possible. The result was a Yellow finding. The dominant sequence was a loss of DC Bus 10, followed by failure of AFW and high pressure recirculation, or failure of feed and bleed. The SRAs determined that a Phase 3 evaluation was necessary because the Phase 2 worksheet did not model additional complications observed during the event as a result of losing DC Bus 10.

The finding directly caused the loss of: DC busses 10L and 10R; DC Panels D11-1 and D11-2; DC Battery Charger 1; Inverters 1 and 3; and, preferred AC busses Y-10 and Y-30. The SDP Phase 3 evaluation estimates the conditional core damage probability (CCDP) of the event that was caused by the performance deficiency. The CCDP value represents the risk increase to the plant of one event occurrence from no event occurrence when there is no duration associated with the event.

The SRAs used the Palisades SPAR [Simplified Plant Analysis Risk] model, Revision 8.17, for the SDP Phase 3 evaluation. The detailed SDP Phase 3 evaluation is included as Attachment 8 of this report. The result of the Phase 3 SDP is a preliminary finding of substantial safety significance (Yellow) with an estimated CCDP of 1.6E-5.

This inspectors also determined this finding had a cross-cutting aspect in the area of human performance, work practices, because the licensee failed to communicate and ensure human error prevention techniques were used, such as holding formal pre-job briefings, self and peer checking, and proper documentation of activities. The licensee also failed to ensure that these techniques were used commensurate with the risk of the assigned task, such that work activities are performed safely. Finally, during these maintenance activities, the inspectors concluded that licensee personnel proceeded in the face of uncertainty or unexpected circumstances (H.4(a)).

Enforcement

Procedure EN-WM-105, “Planning,” Revision 9, was designated as a quality related procedure used to ensure that quality related work is planned in a manner consistent with its importance to plant safety. Quality Related Work Orders 291194-01, “Inspect and Replace Breakers 72-119 and 72-120,” 291210-01, “Replace Breaker 72-121,” and 291123-03, “Inspect 72-123 Installation,” all dated September 25, 2011, were developed and implemented to perform work on safety-related Panel D11-2, associated with the left train 125-Volt Direct Current (DC) safety-related system.

: Title 10 of the Code of Federal Regulations (CFR), Part 50, Appendix B, Criterion V, “Instructions, Procedures, and Drawings,” states, in part, that activities affecting quality shall be prescribed by documented instructions, procedures or drawings of a type appropriate to the circumstances and shall be accomplished in accordance with these instructions, procedures or drawings.

An Apparent Violation (AV) of 10 CFR 50, Criterion V, “Instructions, Procedures, and Drawings,” has been identified, in that, on September 25, 2011, the licensee failed to ensure that the work performed on Panel D11-2 through Work Orders 291194-01, 291210-01, and 291123-03, all activities that affected quality, was prescribed by documented instructions or procedures of a type appropriate to the circumstances and accomplished in accordance with the instructions or procedures. The licensee documented the apparent violation in its corrective action program as

27 Enclosure

CR-PLP-2011-04822 (AV 05000255/2014011-02; Failure to Have Adequate Work Instructions For Work Performed on Panel D11-2).

At the end of this inspection, the licensee continued to perform a root cause evaluation to determine the causes of the event and develop corrective actions. As a remedial corrective action on September 25, 2011, the licensee repaired the damage caused to Panel D11-2 to restore it to service and addressed the operability and effect of the transient on other components, as further discussed in Section 4OA5.5 of this report.

b.2 Finding for the Failure to Implement Human Performance Tools and to Perform an Infrequently Performed Test or Evolution (IPTE) Brief

Introduction: A finding of very low significance was identified by the inspectors for the licensee’s failure to implement Procedure EN-HU-102, “Human Performance Tools,” which established standards and expectations for the use of specific human performance tools with the goal to improve personnel and plant performance through human error reduction. The inspectors identified that Procedure EN-HU-102 was not implemented for the work performed on September 25, 2011, to install a TM and address a non-conforming condition associated with Panel D11-2. Implementation of the procedure for Panel D11-2 scheduled work required the use of Procedure EN-OP-116, “Infrequently Performed Tests or Evolutions,” and performance of an IPTE pre-job brief, which the inspectors determined was not performed for the work on September 25, 2011.

Description

• Precaution 2.2.4 to install the TM in WO 291209-01, stated, in part that failure to complete the WO steps in order would potentially trip the main generator;

: On September 27, 2011, the inspectors reviewed completed WO 291209-01, “Install Temporary Modification,” WO 291194-01, “Inspect and Replace Breakers 72-119 and 72-120,” WO 291210-01, “Replace Breaker 72-121,” and WO 291123-03, “Inspect Breaker 72-123 Installation,” and associated work packages. The inspectors also began interviewing workers involved with the incident. The inspectors established the following facts related to the job preparation and work performed on Panel D11-2 on Sunday September 25, 2011:

• The work performed via WO 291194-01, WO 291210-01, and WO 291123-03 contained step 2.2.3, which stated that shorting bus bars in Panel D11-2 may cause physical injury, damage to equipment and/or a trip the plant;

• A pre-job brief for WO 291209-01 was held at approximately 5:00 a.m. for the night shift electricians, in which Attachment 9.5, “Medium Risk (Standard) Pre-job Brief,” was used to conduct the briefing, the night shift did not begin work on this WO;

• Sometime following the 5:00 a.m. pre-job brief, but before the nightshift to dayshift maintenance crew turnover, the nightshift electrical superintendent recognized that the scheduled work on Panel D11-2 might be considered a high operational risk and may require a high risk pre-job brief. However, this realization was not verbalized during turnover from the nightshift to dayshift maintenance crews;

• Sometime after 8:00 a.m. a pre-job brief was held with the dayshift electrical maintenance crew. The crew re-used the previous medium pre-job brief form completed as part of the 5:00 a.m. nightshift pre-job brief and did not complete a new form, as required;

28 Enclosure

• At approximately 12:45 p.m., an informal pre-job brief was held for WO 291194-01, WO 291210-01, and WO 291123-03 to inspect and repair breakers associated with Panel D11-2. No pre-job brief form was utilized, as required by Procedure EN-HU-102. The workers conducted a reverse-brief utilizing the work orders. Based on interviews with the workers, supervisors, engineers and managers present, not all aspects of the work were covered during the informal pre-job brief. Critical steps were not discussed, nor was a plan discussed for addressing potentially cross-threaded screws in the horizontal bus bars. Following the event, one manager described the informal pre-job brief felt like an alignment meeting; however, this was not challenged by any of the licensee personnel present at the informal pre-job brief; and,

• On September 25, 2011, no licensee personnel, supervision or management reviewed or referenced Procedure EN-HU-102, “Human Performance Tools,” or Procedure EN-OP-116, “Infrequently Performed Tests or Evolutions,” even though the work was scheduled on a safety-related, energized, operable 125-Volt DC bus, the work was not routine or frequently performed, and the risk for a plant transient was recognized in the work order precautions and limitations.

The inspectors reviewed Attachment 9.1, “Worker Human Performance Tools,” of Procedure EN-HU-102, and evaluated the pre-job decision flowchart. The inspectors determined that based on the work orders and information known prior to the event, licensee personnel established that an error in the work could have a significance consequence to the plant or personnel. The inspectors also noted that a special test procedure was not required and that some aspects of the work could be considered complex. The inspectors also concluded, based on a historical review of work orders for the 125-Volt DC system, that scheduled work on a safety-related, energized, and operable 125-Volt DC bus was not a task that was performed frequently. The inspectors concluded that if the flowchart was utilized, licensee personnel were directed by Procedure EN-HU-102 to determine if the scheduled work met IPTE criteria per Procedure EN-OP-116.

The inspectors reviewed Procedure EN-OP-116, and determined the scheduled work met the definition of an IPTE based on the information available prior to the plant event. Specifically, the work order tasks were an evolution, which if not properly conducted or if unexpected results were obtained had the potential to significantly reduce margins of safety, introduce operational transients, or introduce reactor trips. Also, the work was not covered by an existing approved procedure, as the work was being performed via work orders. The inspectors determined that if the IPTE screening checklist in Attachment 9.2 of Procedure EN-OP-116 were utilized, the first four questions would were answered "yes," based on the information available at the time, and IPTE controls were required.

The inspectors interviewed several maintenance workers and supervisors representing all departments and determined that the pre-job brief process described in Procedure EN-HU-102 was significantly revised with the issuance of a new revision on September 1, 2011. Prior to this date, maintenance personnel utilized Attachment 9.1 of Procedure EN-MA-101, “Fundamentals of Maintenance,” for all work-related pre-job briefs. The inspectors determined that no onsite personnel received any formal briefings or training related to the newly revised human performance procedure, which contained significantly different standards for the conduct of pre-job briefs. The change was communicated via an email on August 29, 2011, from the maintenance manager.

29 Enclosure

Although the previous pre-job brief form in Procedure EN-MA-101 was still active, it was not allowed for use, even as a supplement to the new pre-job briefs contained in EN-HU-102. Several workers interviewed by the inspectors expressed confusion with respect to the new pre-job brief process.

The inspectors concluded that the licensee’s human performance tools contained in Procedure EN-HU-102 were adequate processes and the failure to implement the procedure, as required, directly contributed to the events that occurred on September 25, 2011.

Analysis

The inspectors determined that the finding was more than minor in accordance with IMC 0612, “Power Reactor Inspection Reports,” Appendix B, “Issue Screening,” dated December 24, 2009, because it was associated with the Procedure Quality and Human Performance attributes of the Mitigating Systems Cornerstone. This adversely affected the cornerstone objective, in that, the failure to utilize human error reduction tools impacted the availability, reliability and capability of systems that responded to initiating events to prevent undesirable consequences. Specifically, the failure to utilize human performance tools directly contributed to the inadequate work planning and preparation scheduled for Panel D11-2 on September 25, 2011. The inspectors determined that the finding could be evaluated using the SDP in accordance with IMC 0609, “Significance Determination Process,” Attachment 0609.04, “Phase 1 - Initial Screening and Characterization of Findings,” Table 4a, for the Mitigating Systems Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Mitigating Systems questions and screened the finding as having very low safety significance (Green).

: The inspectors determined that the licensee’s failure to implement approved procedures for the use of human performance tools was a performance deficiency warranting a significance evaluation.

The finding has a cross-cutting aspect in the area of human performance, work practices, because the licensee failed to ensure personnel work practices supported human performance through defining and effectively communicating expectations regarding procedural compliance coincident with plant personnel following procedures. Specifically, the licensee personnel failed to reference or implement procedures with human performance tools, which, if implemented would have required an IPTE brief for the work performed on Panel D11-2 on September 25, 2011 (H.4(b)).

Enforcement

The licensee documented this condition in its corrective action program as CR-PLP-2011-04822 and CR-PLP-2011-04981. At the end of this inspection, the licensee continued to perform a root cause evaluation to determine the causes of the event and develop corrective actions.

: No violation of regulatory requirements occurred since the prejob brief process is not required by NRC regulations, but the inspectors did identify a finding (FIN) (FIN 05000255/2014011-03, Failure to Implement Human Performance Tools and to Perform an Infrequently Performed Test or Evolution Brief).

b.3 Finding for the Failure to Comply With Work Hour Rules for Non-Covered Workers

Introduction: A finding of very low significance was identified by the inspectors for the licensee’s failure to implement Procedure EN-FAP-OM-006, “Working Hour Limits for Non-Covered Workers,” which established standard fleet guidance for working hour

30 Enclosure

limits for Entergy non-covered (not covered by 10 CFR 26) workers as defined in EN-OM-123, “Working Hour Limits.” The inspectors identified that at least two non-covered managers on the nightshift, involved with the work planning and oversight of troubleshooting repair efforts for Panel D11-2, had not followed the standards for work hour limits and did not initiate condition reports when the work hour limits were exceeded, as required by Procedure EN-FAP-OM-006. Specifically, the inspectors identified that the DSM worked approximately 24.5 consecutive hours from September 23 through September 24, and greater than 72 hours in a 7-day period. The electrical superintendent exceeded the administrative limits of 16 hours in a 24-hour period, 26 hours in a 48-hour period, 72 hours in a 7-day period, and greater than a 10-hour break between work periods over a consecutive 19-day period of work.

Description

Procedure EN-FAP-OM-006, Section 3.2, stated, in part, that it is the expectation that all Entergy workers who were non-covered will adhere to the following working hour limitations:

: The inspectors assessed the licensee’s implementation of working hour limits for covered workers under 10 CFR 26, “Fitness For Duty Programs,” and non-covered Entergy workers as defined in Entergy Procedures EN-FAP-OM-006, “Working Hour Limits for Non-Covered Workers,” and EN-OM-123, “Working Hour Limits.” The inspectors determined that the licensee appropriately implemented the requirements of 10 CFR 26 for covered workers and did not identify any performance deficiencies associated with the covered workers performing work on Panel D11-2 on September 25, 2011.

• ≤ 16 hours in any 24-hour period; • ≤ 26 hours in any 48-hour period; • ≤ 72 hours in any 7-day period; and, • ≥ 10-hour breaks between work periods.

Section 3.3, “Documentation,” stated, in part, that if working hour limits described by Procedure EN-FAP-OM-006 were exceeded, a condition report was required to be generated by the worker. Section 3.4, “Process,” stated, in part, that if limits were to be exceeded that: the worker seeks supervisor approval to exceed working hour limits prior to working beyond the limits for a non-covered worker; the supervisor of a non-covered worker approves any instances where working hour limits for non-covered workers were exceeded; documentation of approval was to be recorded as a notation in the payroll system; and the non-covered worker documented excession of the work hour limits by initiating a condition report.

The inspectors initially queried six random supervisors and managers associated with the dayshift and nightshift activities that took place on Panel D11-2 from September 23 through September 26, 2011. At the time of the query, no condition reports had been initiated by non-covered workers for exceeding the work hour limits for non-covered workers. Based on the review of the data provided to the inspectors, the inspectors determined the following:

• Four of the supervisors and managers did not exceed the work hour limits for non-covered workers;

31 Enclosure

• The nightshift DSM had logged into the site owner controlled area on Friday, September 23, 2011, at 7:35 a.m., and exited the site owner controlled area on Saturday September 24, 2011, at 08:22, a consecutive time period of 24.5 hours;

• The nightshift DSM had worked greater than 72 hours in a 7-day period starting on September 19, 2011; and,

• The nightshift electrical superintendent had exceeded all the working hour limits established in EN-FAP-OM-006 and had worked a 19-day consecutive period.

Based on interviews with the individuals, the inspectors affirmed the timeframes established were correct. The inspectors also confirmed that approval documentation in the form of a standing memo, as allowed by Section 3.2 of Procedure EN-FAP-OM-006, did not exist. Interviews of the all the supervisors and managers confirmed a general lack of knowledge with respect to the requirements of Procedure EN-FAP-OM-006, and no training for new supervisors or managers that covered the provisions of EN-FAP-OM-006. The inspectors concluded that while the supervisors of the non-covered workers were generally aware of the hours their employees worked, prior approval to exceeding work hours was not performed under the auspices of Procedure EN-FAP-OM-006 and documentation of approval was not generally recorded as a notation in the payroll system. Finally, condition reports were not initiated when work hour limits were exceeded; however, condition reports were initiated for the individuals discussed above following discussions of the requirements of Procedure EN-FAP-OM-006 with the inspectors.

The inspectors determined the failure of management personnel to adhere to the working hour limitations for non-covered workers in Procedure EN-FAP-OM-006 reduced their effectiveness for oversight and direction in the implementation of licensee procedures for work control and human performance tools for the work scheduled on Panel D11-2.

Analysis

The issue affected the Mitigating Systems Cornerstone because the 125-Volt DC system work plan development was overseen by the non-covered workers. The inspectors determined that the finding was more than minor in accordance with IMC 0612, “Power Reactor Inspection Reports,” Appendix B, “Issue Screening,” dated December 24, 2009, because it revealed weaknesses that, if left uncorrected, could lead to more significant safety concerns associated with overseeing work on safety-related equipment. In addition, the inspectors concluded that the failure to implement working hour limitations for non-covered workers in Procedure EN-FAP-OM-006 was more than an isolated instance. The inspectors contacted a regional SRA for additional assistance in determining the risk significance of this finding, since the significance could not be assessed using probabilistic risk assessment methods and tools. The SRA concurred with the inspectors that the use of IMC 0609, Appendix M, “Significance Determination Process Using Qualitative Criteria,” was the appropriate method for determining the significance and that the finding was not potentially greater than green because the attributes contained in Step 4.2 were not affected. In accordance with IMC 0609, Appendix M, management review of this issue determined that this finding was of very low safety significance since the performance deficiency did not directly contribute to the

: The inspectors determined the failure of management personnel to adhere to the working hour limitations for non-covered workers in Procedure EN-FAP-OM-006 was a performance deficiency warranting a significance evaluation.

32 Enclosure

event, as the non-covered workers were involved with the planning and not actual implementation of the work performed on September 25, 2011, on Panel D11-2.

The finding has a cross-cutting aspect in the area of human performance, resources, because the licensee failed to ensure that personnel and other resources were available and adequate to assure nuclear safety, specifically, those necessary for training of personnel and sufficient qualified personnel to maintain work hours within working hour guidelines (H.2(b)).

Enforcement

The licensee documented this condition in its corrective action program as CR-PLP-2011-05095 and CR-PLP-2011-05116. At the end of this inspection, the licensee continued to perform apparent cause and extent of condition evaluations to determine the causes and the extent of the problem for the performance deficiency, in order to develop corrective actions.

: No violation of regulatory requirements occurred since the issue involves workers not covered by 10 CFR 26 regulations, but the inspectors did identify a FIN (FIN 05000255/2014011-04, Failure to Comply With Work Hour Rules for Non-Covered Workers).

b.4

The inspectors reviewed the licensee’s risk analysis performed for the scheduled work on Panel D11-2 for September 25, 2011. The inspectors also reviewed licensee Procedures EN-WM-104, “On Line Risk Assessment,’ Revision 4 and 4.02, “Control of Equipment,” Revision 59.

Assessment of Licensee Implementation for Managing and Assessment of Maintenance Risk

On Sunday September 25, 2011, the licensee performed a quantitative risk assessment using the quantitative risk tool and knowledge of the 125-Volt DC system available at that time. The only breaker modeled in the licensee’s quantitative risk tool was Breaker 72-119, which was entered into the quantitative risk tool, along with other out of service equipment. The calculated risk achievement worth was calculated to be 1.03, which was low. In accordance, with Procedure EN-WM-104, operations personnel also protected the right train of plant equipment, including the right train of 125-Volt DC system. Interviews with the ROs also indicated that a qualitative risk assessment was performed, in the event Panel D11-2 was lost. The qualitative assessment led operations personnel to develop a contingency plan and conduct an approximately 30-minute briefing on the loss of Panel D11-2, should that occur during maintenance. At the time, operations staff was unaware of the latent design deficiency associated with the instantaneous trips for shunt trip Breaker 72-01, which was discovered immediately following the event on September 25, 2011.

Therefore, based on the information available at the time, the inspectors concluded that operations personnel appropriately assessed and managed the increase in risk in compliance with 10 CFR 50.65(a)(4) of the Maintenance Rule. However, the inspectors did identify a minor violation, in that, operations personnel did not appropriately document the qualitative risk assessment in accordance with licensee Procedure EN-WM-104. The licensee documented this minor violation in its corrective action program as CR-PLP-2011-04822 and at the end of the inspection continued to perform a root cause evaluation and develop corrective actions. This failure to comply

33 Enclosure

with EN-WM-104 constitutes a violation of minor significance that is not subject to enforcement action in accordance with the NRC=s Enforcement Policy.

.4

a.

Evaluate Operator Response To The Transient That Occurred On September 25, 2011, As It Related To The Implementation Of Licensee’s Procedures And Processes For Evaluating/Assessing Operator Performance

The inspectors reviewed control room logs, plant parameter recordings, plant procedures, corrective action documents, maintenance work order (WO) and work request history, and engineering design changes as part of the inspection activities. The inspectors also conducted interviews with plant ROs who responded to the event and individuals directly involved in the event to conduct fact-finding and determine the relevant sequence of events associated with the RO response to the event on September 25, 2011.

Inspection Scope

The inspectors independently assessed the operations personnel use of procedures during the event. In response to the event, ROs entered the following EOPs, ONPs, and General Operating Procedures (GOPs) in chronological order:

• EOP-1.0, “Standard Post-Trip Actions”; • EOP-9.0, “Functional Recovery”; • ONP-2.3, “Loss of DC”; • ONP-24.1, “Loss of Preferred AC Bus No. 1 (Y-10)”; • ONP-24.3, “Loss of Preferred AC Bus No. 3 (Y-30)”; • ONP-7.1, “Loss of Instrument Air”; • ONP-14.1, “Spurious Containment Isolation”; and • GOP-8, “ Power Reduction and Plant Shutdown to Mode 2 or Mode 3 ≥ 525oF.”

The inspectors independently concluded that the operations staff utilized the correct procedures to respond to the event, and that the licensee’s procedures authorized the actions the ROs subsequently took both in the field and in the control room on September 25, 2011. The inspectors did identify two performance deficiencies with respect to the licensee’s implementation and content of approved procedures that are discussed below.

b.

b.1

Findings and Observations

Failure to Implement Emergency Operating Procedures Immediate Actions

Introduction: A finding of very low safety significance and associated non-cited violation (NCV) of TS 5.4 was identified by the inspectors for failure to implement procedures for combating emergencies and other significant events as required by Regulatory Guide (RG) 1.33, Section 6. Specifically, during the performance of EOP-1.0, “Standard Post-Trip Actions,” in response to a loss of the left train 125-Volt DC bus and subsequent plant trip, the ROs in the control room failed to immediately take the contingency action in the “response not obtained” column for an immediate action step that could not be met due to the partial loss of control room indication. Step 2.b. of

34 Enclosure

Section 4.0, “Immediate Actions,” had a reactor RO in the control room verify that the main generator was disconnected from the grid. If that cannot be verified, then the RO was required to connect a jumper across the corresponding relay terminals in the control room panel. These actions were not immediately taken by the control room staff at the time of this event.

Description

The contingency actions under Step 2.b.1 (“response not obtained” column) directed ROs to perform any of the following, if the verification step could not be met: “1) OPEN Main Generator Output Breakers (25F7, 25H9) at Control Panel C-01; or, 2) CONNECT jumper between terminals 1 and 10 on Relay 487U (Y Phase) inside Panel C-04.” After overhearing the exchange between the turbine-side RO and the CRS, the SM engaged the turbine-side RO on the response given, due to questions the SM had regarding indications available to the turbine-side RO. The SM then took additional actions by dispatching an Auxiliary Operator (AO) to the switchyard to visually verify that the disconnects were open. The appropriate direction from the SM upon verifying no indication should have been to direct the turbine-side RO to implement the contingency actions.

: On Sunday, September 25, 2011, the plant experienced a loss of the left train 125-Volt DC bus which resulted in a reactor trip. The control room RO entered EOP-1.0, “Standard Post-Trip Actions,” as required. While working through Section 4.0, “Immediate Actions,” Step 2.b. directed the ROs to, “VERIFY that the Main Generator is disconnected from grid,” by verification that the main generator output Breakers 25F7 and 25H9 were open or the motor-operated disconnect 26H5 was open. When asked by the control room supervisor (CRS) for verbal verification of this condition, the turbine-side RO responded that the breakers were open, when, in fact, the indication for these breakers was lost during the event. The CRS did not challenge the RO’s response and continued through the procedure. This information was corroborated with the completed copy of EOP-1.0 that was reviewed by the inspectors after the event.

Upon hearing the conversation between the turbine-side RO and the SM, an extra RO in the control room, called the transmission system operator to inquire about the status of the plant. Through that communication, approximately 10 minutes after the reactor trip, the extra RO learned that the main generator output breakers were still closed. The extra RO then promptly updated the control room operations crew of the current status of the breakers. The contingency action of jumpering between Terminals 1 and 10 on Relay 487U inside Panel C-04 was then taken by the turbine-side RO.

Analysis

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," because, if left uncorrected, the performance deficiency could have the potential to lead to a more significant safety concern. In particular, this loss of 125-Volt DC event could have become a more significant event with further complications and plant issues. The inspectors determined that the finding could be evaluated using the SDP in accordance with IMC 0609, “Significance Determination Process,” Attachment 0609.04, “Phase 1 - Initial Screening and Characterization of Findings,” Table 4a, for the Initiating

: The inspectors determined that the failure to take the contingency actions immediately after being unable to verify an immediate action step, and, therefore, appropriately implement EOP-1.0, was a performance deficiency that warranted a significance determination.

35 Enclosure

Events Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Transient Initiator question of contributing to both the likelihood of a reactor trip and the likelihood that mitigating equipment or functions would not be available and screened the finding as having very low safety significance (Green).

The finding had a cross-cutting aspect in the area of human performance related to the cross-cutting component of Work Practices, in that the licensee communicates human error prevention techniques, such as peer-checking, and that these techniques are used commensurate with the risk of the assigned task, such that work activities are performed safely. In this finding, the licensee had opportunities to peer-check/challenge the turbine-side RO’s response to the verbal verification step of the procedure, but did not immediately follow up with the appropriate steps from the procedure to ensure that the plant was in a safe condition (H.4(a)).

Enforcement

Contrary to the above, on September 25, 2011, the licensee failed to implement the contingency action of an immediate action step in EOP-1.0, “Standard Post-Trip Actions.” Specifically, in response to a loss of the left train 125-Volt DC bus and subsequent plant trip, the operators in the control room could not complete Step 2.b. of Section 4.0, “Immediate Actions,” which had the operator verify that the main generator was disconnected from the grid, due to the unavailable indication for the output breakers from the loss of 125-Volt DC event. The operators then failed to immediately take the contingency action in the “response not obtained” column, Step 2.b.1, which directed the operator to connect a jumper across the corresponding relay terminals in the control room panel to open the breaker. Because this violation was of very low safety significance, and was entered into the licensee’s CAP, as CR-PLP-2011-06081, this violation is being treated as an NCV, consistent with Section 2.3.2 of the NRC Enforcement Policy (NCV 05000255/2011014-05; Failure to Implement Emergency Operating Procedure Immediate Actions).

: The TS 5.4.1 requires that written procedures shall be established, implemented, and maintained covering the activities in RG 1.33, Revision 2, Appendix A, dated February 1978. RG 1.33, Appendix A, Section 6, specifies procedures for combating emergencies and other significant events, including a reactor trip.

At the end of the special inspection, the licensee was still performing a root cause evaluation to determine the causes of the event and to develop corrective actions. As a remedial corrective action on October 28, 2011, each operations crew received a briefing from site operations management about operator expectations, the usage of human performance tools/procedures, and an overview of the recent events and any shortcomings associated with these events by the operations department.

b.2 Failure to Establish a Procedure for the Loss of a DC Bus and the Simultaneous Loss of Two Preferred AC Power Sources

Introduction: A finding of very low safety significance and associated NCV of TS 5.4.1 was identified by the inspectors for the failure to establish a procedure for combating emergencies and other significant events as required by RG 1.33, Section 6. Specifically, Section 6 states, in part, that the loss of electrical power (and/or degraded power sources) is a safety-related activity that should be covered by written procedures, and TS 5.4.1 required, in part, that written procedures be established, implemented, and maintained to cover the activities in RG 1.33. The design and licensing basis of the plant includes the loss of a single train of DC power. Although the site has multiple

36 Enclosure

procedures to address the loss of the DC system and individual preferred AC sources, the procedures did not integrate to provide a response that minimized challenges to plant safety. The site has three separate procedures that were used in this event for the loss of one DC bus and loss of one preferred AC source (two sources were lost during the event, hence two of these procedures were used); but not one inclusive procedure to cover the loss of both preferred AC sources simultaneously.

Description

As operators responded to the event, they worked through various EOPs and ONPs to protect the reactor and establish appropriate safe shutdown conditions, including Procedure ONP-2.3, “Loss of DC Bus.” This procedure was created as a commitment from past operating experience (a 1981 loss of 125-Volt DC event at the Millstone Nuclear Generating Station) to try to prescribe steps to combat a loss of 125-Volt DC event. However, ONP 2.3 did not contain steps that adequately covered a total loss of a single train of the 125-Volt DC system, which ultimately resulted in the loss of two preferred AC sources. This simultaneous loss of a single train of DC and two preferred AC power sources was not explicitly addressed under any licensee procedure. Instead, ONP 2.3 directed ROs to two separate ONPs (ONP 24.1, “Loss of Preferred AC Bus No. 1,” (Y-10), and ONP 24.3, “Loss of Preferred AC Bus No. 3” (Y-30)), each for the loss of a single preferred AC source. All of these procedures independently covered their respective loss of power events adequately; but the design and licensing basis of the site is the loss of a single train of 125-Volt DC power. This loss of a single 125-Volt DC train cannot be appropriately analyzed without taking into account the subsequent loss of the preferred AC busses associated with the DC bus, and the licensee’s procedures did not explicitly address this concurrent event. In addition, since the loss of only one preferred AC bus was addressed in each procedure, no single integrated strategy was available to prioritize and manage specific safety parameters.

: On Sunday, September 25, 2011, while maintenance work was being conducted inside Panel D11-2, an error occurred, that caused an electrical fault and loss of the panel. The electrical perturbation from the fault on Panel D11-2, caused Bus shunt trip Breaker 72-01 to open, which, in turn caused the loss of 125-Volt DC busses D-10L and D-10R. These losses de-energized the two preferred 120-Volt AC power sources associated with busses Y-10 and Y-30. The loss of two out of four of the preferred AC power sources caused a loss of power to two RPS channels and initiated a reactor and turbine trip (two-out-of-four logic was made up).

Based on the inspection and interviews conducted by the inspectors, the complex network of procedures that needed to be utilized during the events of September 25, 2011, complicated the operations crew’s response to the transient. The procedures that the crew worked through were inadequate to respond in a timely fashion to changing plant conditions caused by the loss of the left train of DC power. As a result, an abnormally high pressurizer level of approximately 98 percent was reached, which exceeded the TS limit of 62.8 percent, and exceeded the desired operational band. If the pressurizer were to have gone completely solid (filled with water), the integrity of the PCS would have been challenged, and the pressurizer safety valves could have lifted, creating a path for water to leave the PCS. From the inspectors’ review of the plant data associated with the loss of 125-Volt DC event, it was calculated that the site was less than 9 minutes away from the aforementioned condition. In addition, the water level in SG ’A’ increased to approximately 98 percent, which was also outside the desired operating band. If the SG were completely filled with water, the water would have

37 Enclosure

entered the main steam lines, which would have challenged the structural integrity of the piping, since it is not designed to handle water.

Analysis

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," because the finding was associated with the Mitigating Systems cornerstone attribute of Procedure Quality, and adversely impacted the objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the attribute of procedure quality, areas to measure, lists operating (post-event) procedures such as AOPs, SOPs, EOPs, and ONPs, as being items that should be established and maintained to ensure the cornerstone objective is met. The inspectors determined that the finding could be evaluated using the SDP in accordance with IMC 0609, “Significance Determination Process,” Attachment 0609.04, “Phase 1 - Initial Screening and Characterization of Findings,” Table 4a, for the Mitigating Systems Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Mitigating Systems questions and screened the finding as having very low safety significance (Green).

: The inspectors determined that the failure to establish a procedure for combating emergencies and other significant events, as required by RG 1.33, Section 6, was a performance deficiency that warranted a significance determination. Specifically, the event that occurred on September 25, 2011, which involved the loss of a safety-related DC bus and simultaneous loss of two preferred AC sources, was not covered by a written procedure for the loss of electrical power (and/or degraded power sources), although RG 1.33, Section 6.c, identified loss of electrical power as an area which required procedures to address.

The finding does not have an associated cross-cutting aspect since the last known operating experience for a loss of the 125-Volt DC system occurred in 1981 at the Millstone Nuclear Generating Station.

Enforcement

Contrary to the above, prior to September 25, 2011, the licensee did not have a procedure to cover the loss of a DC bus concurrent with the loss of two preferred AC power sources. The design and licensing basis of the plant was the loss of a single train of DC power. Conversely, the ROs had to utilize two separate ONPs to combat the simultaneous loss of both preferred AC power sources associated with this event, which complicated and impacted the operations crew’s response to changing plant conditions during this transient. Because this violation was of very low safety significance, and was entered into the licensee’s CAP, as CR-PLP-2011-06209, this violation is being treated as an NCV, consistent with Section 2.3.2 of the NRC Enforcement Policy (NCV 05000255/2011014-06; Failure to Establish a Procedure for the Loss of a DC Bus and the Simultaneous Loss of Two Preferred AC Power Sources).

: The TS 5.4.1 requires that written procedures shall be established, implemented, and maintained covering the activities in RG 1.33, Revision 2, Appendix A, dated February 1978. RG 1.33, Appendix A, Section 6, specifies procedures for combating emergencies and other significant events, including a loss of electrical power.

At the end of the special inspection, the licensee was still performing a root cause evaluation to determine the causes of the event and to develop corrective actions.

38 Enclosure

.5

a.

Review Any Corrective Actions Taken Including Operability Evaluations And Direction Provided By The Licensee In Response To The Transient And Equipment Failures

The inspectors reviewed the licensee’s condition reports generated from the loss of left train 125-Volt DC event to ensure that all issues were captured and that appropriate corrective actions were assigned to address those issues. The team also reviewed the licensee’s list of equipment that was adversely impacted due to the loss of 125-Volt DC or 120-Volt preferred AC power, and any associated repair work that required completion during the forced outage. The inspectors, in conjunction with the resident inspectors onsite, attended daily shift turnover meetings and the plant’s start-up review meeting, which assessed all outstanding equipment issues and addressed items that needed to be fixed prior to mode changes.

Inspection Scope

The inspection team did not identify any equipment issues that precluded start-up or mode changes. The licensee’s operability evaluation and compensatory measures for the electrical coordination issues on Panel D11-2 (and subsequently Panels D11-1, D21-1, and D21-2), and the associated issues with the design deficiencies identified with shunt trip Breaker 72-01 were also reviewed by the inspectors. The team’s observations and assessment of this operability evaluation and the issues identified are described below.

Through the review of corrective action documents and forced outage work, the inspectors identified one item that needed to be addressed prior to re-starting the plant. The issue was that the licensee did not take all of the appropriate actions to assess the left train safety-related battery, DC Battery 1, D-01, to ensure the battery was operable following the transient. According to the UFSAR, Battery D-01 was a qualified component per the Institute of Electrical and Electronics Engineers (IEEE) standard. The IEEE Standard 450, “IEEE Recommended Practice for Maintenance, Testing, and Replacement of Vented Lead Acid Batteries for Stationary Applications,” Section 4.3.4, “Special Inspections,” stated, in part, that if the battery had experienced an abnormal condition, such as severe discharge or overcharge, an inspection should be made to ensure that the battery was not damaged. After this information was presented to the site, the station’s onsite and corporate engineering staff concluded that the transient, which occurred on September 25, 2011, made the battery experience an abnormal condition per IEEE Standard 450. The licensee documented this in its corrective action program as CR-PLP-2011-04974 and determined testing was necessary. The inspectors determined that because the battery tested satisfactorily per the IEEE standard this was a minor violation. This failure to comply with IEEE standards, initially, constitutes a violation of minor significance that is not subject to enforcement action in accordance with the NRC=s Enforcement Policy.

b.

b.1

Findings and Observations

Failure to perform an Adequate Operability Evaluation

Introduction: A finding of very low safety significance and associated NCV of 10 CFR Part 50, Appendix B, Criterion V, “Instructions, Procedures, and Drawings,” was identified by the inspectors for the failure to implement a procedure for an activity

39 Enclosure

affecting quality. Procedure EN-OP-104, “Operability Determination Process,” required an assessment of the operability for structures, systems, and components (SSCs) when degraded or non-conforming conditions were identified and establishment of compensatory measures were needed to, “ensure, maintain, and enhance future operability.” Specifically, the inspectors identified the operability evaluation for the 125-Volt DC system, completed on September 30, 2011, did not contain two compensatory measures necessary to ensure the operability of the system. It was also identified that the 50.59 pre-screening (process applicability determination) for the TM, which was also a compensatory measure for the operability evaluation, was not clearly written and did not adequately describe the evaluation of the modification or the bases for this decision.

Description

The plant’s UFSAR Section 8.3.5, stated that, “a non-automatic circuit breaker with a shunt trip is provided in the circuit between the battery fuse and its bus…[the circuit breaker] does not contain fault detectors and is not intended to interrupt fault currents…[it is] manually operated open or closed with the capability of being opened remotely via the shunt trip device.” The expected outcome of the fault on Panel D11-2 was that the battery fuse, FUZ/D11-2, would have tripped to isolate the fault. This event revealed the design issue with shunt trip Breaker 72-01, electrical coordination issues with the circuit breakers and fuses within the bus, and an extent-of-condition review revealed similar issues with Circuit Breaker 72-02 and the right train DC bus.

: On Sunday, September 25, 2011, maintenance activities were conducted inside Panel D11-2 to repair identified issues with breakers in the panel. During this maintenance work, a human error caused a fault in the panel and resulted in the loss of Panel D11-2. This electrical transient within the bus caused the isolation breaker to DC Battery 1, D-01, which is shunt trip Breaker 72-01, to open due to the breaker having internal instantaneous and thermal protective elements that were unknown to the licensee prior to this event. The opening of this breaker then resulted in the de-energization of a single train of the 125-Volt DC system (the left train), the loss of two preferred 120-Volt AC power sources, and ultimately, a reactor and turbine trip. The extent of the transient on the plant was an unexpected occurrence from the electrical fault on Panel D11-2 due to the revealed design deficiency of shunt trip Breaker 72-01, having thermal and instantaneous protective circuits.

This design deficiency for Breakers 72-01 and 72-02 was captured in the site’s corrective action program (CAP) as CR-PLP-2011-04835 and CR-PLP-2011-04965) and identified as a non-conforming condition. Per site procedure, EN-OP-104, “Operability Determination Process,” a non-conforming condition is “a condition of an SSC that involved a failure to meet the current licensing basis (CLB) of the plant or a situation in which quality has been reduced because of factors such as improper design, testing, construction, or modification.” When a non-conforming condition was identified, an operability evaluation (or functionality assessment) was required to be performed, per the procedure, to document technical information that will determine if there was a reasonable expectation that the SSC can perform its specified safety function(s). The operability evaluation also identified and developed compensatory measures that may, “restore, enhance, or maintain future operability of an SSC that has a degraded or nonconforming condition.” Also, according to the Section 5.5 flowchart, if an SSC was considered operable with compensatory measures, then the site should perform a process applicability determination (50.59 pre-screening) per site procedure, EN-LI-100, to determine the effects of the compensatory measures on other aspects of the facility.

40 Enclosure

On September 30, 2011, the operability evaluation for the 125-Volt DC system was completed and reviewed by the inspectors. The compensatory actions identified in the evaluation involved implementing: TM EC 32028 to raise the magnetic setpoints on Breakers 72-01 and 72-02; opening and caution tagging the DC Oil Lift Pump supply breakers on both trains; and, securing open the breaker for the public address (PA) system and a motor generator on the left train. A long-term action was also identified to implement a modification for addressing the electrical coordination issues between Breakers 72-01 and 72-02 and the associated downstream equipment on the respective busses. The inspectors identified that the operability evaluation did not contain the following two compensatory measures necessary to ensure the future operability of the 125-Volt DC system:

• There was no prohibition put in place to prevent maintenance on busses D11-1, D11-2, D21-1, or D21-2 (which is what caused the transient event on September 25, 2011); and,

• The assumptions used in the calculation presented in the operability evaluation for the electrical coordination issues credited only one battery charger on a single DC bus. The inspectors noted that per Procedure SOP-30, operators were allowed to place two battery chargers on a single DC bus in Modes 1, 2, or 3. A compensatory measure for prohibiting this procedural step in SOP-30 should have been initiated to ensure the correct electrical configuration of the DC bus assumed in the operability evaluation.

The inspectors also identified that the 50.59 pre-screen, written for TM EC 32028 and identified as a compensatory measure, did not clearly or adequately describe the 50.59 applicability of the modification, the basis for this modification, or the impact of it on other operating plant equipment. Furthermore, it was noted that the evaluation did not contain a discussion on the DC oil lift pumps for the PCPs, whose supply breakers were opened and tagged out as a compensatory measure, and which were discussed in the UFSAR Sections 3.8, 4.4, and 14.7.

The inspectors’ identified concerns were subsequently documented in CR-PLP-2011-04988 and CR-PLP-2011-04965, and revisions were made to the operability evaluation and 50.59 pre-screening. The inspectors reviewed these revisions and did not identify any additional deficiencies.

Analysis

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," because the finding was associated with the Mitigating Systems Cornerstone attribute of Equipment Performance, and adversely impacted the objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the attribute of equipment performance impacted the availability and reliability of the 125-Volt DC system. The inspectors determined that the finding could be evaluated using the SDP in accordance with IMC 0609, “Significance Determination Process,” Attachment 0609.04, “Phase 1 - Initial Screening and Characterization of Findings,” Table 4a, for the Mitigating Systems Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Mitigating

: The inspectors determined that the failure to implement a procedure affecting quality was a performance deficiency warranting a significance evaluation.

41 Enclosure

Systems questions and screened the finding as having very low safety significance (Green).

The finding has a cross-cutting aspect in the area of Human Performance, Decision-Making, because the licensee did not adequately conduct an effectiveness review of a safety-significant decision to verify the validity of the underlying assumptions and identify possible unintended consequences, as necessary. This includes properly evaluating for operability, addressing non-conforming/degraded conditions that are adverse to quality, and constructing adequate compensatory measures that do not adversely affect operating plant equipment or lineups (H.1(b)).

Enforcement

Contrary to this, on September 30, 2011, the licensee failed to identify and develop the compensatory measures to restore operability of an SSC that has a degraded or non-conforming condition, in accordance with Procedure EN-OP-104, Section 5.4. Specifically, the licensee failed to put a prohibition on maintenance activities for busses D11-1, D11-2, D21-1, and D21-2, which had electrical coordination issues that could result in inoperability of a single train of 125-Volt DC power; properly credit battery chargers in the operability calculation; and did not adequately evaluate the impact of the TM on other operating plant components in the 50.59 pre-screening. Because this violation was of very low safety significance, and was entered into the licensee’s CAP as CR-PLP-2011-04988 and CR-PLP-2011-04965, this violation is being treated as an NCV, consistent with Section 2.3.2 of the NRC Enforcement Policy (NCV 05000255/2011014-07; Failure to Perform an Adequate Operability Evaluation).

: Title 10 CFR Part 50, Appendix B, Criterion V, “Instructions, Procedures, and Drawings,” requires, in part, that activities affecting quality shall be prescribed by documented instructions, procedures, or drawings of a type appropriate to the circumstances and shall be accomplished in accordance with these instructions, procedures, or drawings. EN-OP-104 is a quality procedure which provides for the implementation of operability evaluations, including implementation of compensatory actions, at the site. Specifically, Section 5.4, required, in part, that the operability evaluation identify and develop the compensatory measures to restore operability of an SSC that has a degraded or non-conforming condition.

The licensee’s corrective actions included revising the 50.59 pre-screening to clearly address the effect of the compensatory measures on other aspects of the facility, prohibiting maintenance on the energized 125-Volt DC busses, and issuing additional site guidance for the operation of battery chargers.

.6

a.

Review The Electrical Tripping Scheme For The DC Bus And Evaluate The Transient To Determine If The Equipment Operated Per Design

The inspectors reviewed engineering analyses, vendor and procurement documents, and other plant documents related to the 125-Volt DC and 120-Volt AC systems (reference Attachment 4), the left and right train DC busses, and the protective devices and loads connected to each bus. The inspectors also reviewed electrical calculations related to protective devices’ coordination analysis to evaluate the adequacy of coordination between the upstream protective devices and the downstream devices connected to each DC bus. The inspectors also reviewed corrective action documents

Inspection Scope

42 Enclosure

issued as a result of the September 25, 2011, event to ensure that appropriate repairs and testing were conducted on components that failed to operate as a result of the electrical fault. Finally, the inspectors observed field maintenance and testing activities performed on Breaker 72-01 to ensure no physical damage resulted from the fault current interruption.

b.

b.1

Findings and Observations

Failure to Maintain Design and Procurement Control of the 125-Volt DC Systems

Introduction: A self-revealed finding of very low safety significance (Green) and associated NCV of 10 CFR Part 50, Appendix B, Criterion III, “Design Control,” and Criterion IV, “Procurement Document Control,” was identified for the licensee’s failure to establish measures to ensure that the applicable regulatory requirements and design bases were correctly translated into specifications and instructions. In addition, the licensee failed to establish measures to assure that the applicable regulatory requirements and design bases, which were necessary to assure adequate quality, were suitably included or referenced in the documents for procurement of equipment. Specifically, 125-Volt DC Breakers 72-01 and 72-02 were purchased and installed with thermal overloads and instantaneous trips enabled. The design basis stated that the breakers were non-automatic and only actuated manually. As a result, on September 25, 2011, when an electrical fault occurred on Panel D11-2, the left train 125-Volt DC bus was lost because the instantaneous trip device on Breaker 72-01 automatically actuated, propagating the fault through the bus, which resulted in a reactor and turbine trip, and plant transient.

Description

UFSAR Section 8.3.5, “DC and Preferred AC Systems,” described these breakers as non-automatic circuit breakers a with shunt trip. The shunt trip device of these breakers was a trip coil that could be energized by battery voltage via 125-Volt DC Panels D11A and D21A for Batteries 1 and 2 respectively. The UFSAR section also indicated that these breakers did not contain fault detectors and were not intended to interrupt fault currents. The breakers were manually-operated open or closed and capable of being opened remotely via their associated shunt trip device. Additionally, electrical coordination between these circuit breakers and downstream bus loads’ protective devices were not considered and analyzed in the Appendix R analysis, EA-APR-95-004, “10 CFR Part 50, Appendix R, Shutdown Associated Circuit Analysis for Common Power Supplies and Common Enclosure,” Revision 4, due to the assumption that these circuit breakers were non-automatic and did not have any fault detectors.

: Isolation circuit Breakers 72-01 and 72-02 were installed in 1981 per plant facility change modification Package FC-407-14C, “Design Basis and Criteria for the 125-Volt DC Distribution Systems.” The modification was implemented to ensure the site’s capability of achieving safe shutdown in the event of a design basis fire in any safety-related fire areas, in compliance the requirements of 10 CFR Part 50, Appendix R. The circuit breakers procured were Gould-ITE Catalog No. KM3B-800, Class 1E, Seismic Category I qualified, equipped with a shunt trip and auxiliary switch. These circuit breakers were depicted on the site’s single line meter and relay diagram as a single switch contact. Each breaker was installed between the battery fuse and its respective DC bus on both trains of the 125-Volt DC distribution system.

43 Enclosure

On September 25, 2011, while technicians performed work activities on Panel D11-2, a human performance error caused a reactor trip. During the work inside the panel, while removing a section of bus bar, the horizontal bus bar rotated and the positive and negative bus bars contacted. This caused a short circuit and the subsequent loss of the left train 125-Volt DC bus and two preferred AC power sources, busses Y-10 and Y-30. The left train busses D-10R and D-10L were electrically separated from Battery D01 due to the opening of Breaker 72-01 as a result of the fault.

Following the September 25, 2011, event, troubleshooting activities revealed that Breakers 72-01 and 72-02 contained thermal overload and instantaneous protective elements, contrary to what was described in the UFSAR, single-line diagram, and coordination analysis; however, it was consistent with the information from the vendor regarding installed Model No. KM3B800. The thermal elements (overload trips) for Breakers 72-01 and 72-02 were tested and found satisfactory. The instantaneous element for Breaker 72-01 was tested and the as-found result showed the breaker was set at the low setting value of 3400 Amps [Amperes]. The instantaneous element for Breaker 72-02 was tested and the as-found result showed the breaker was set at the maximum setting value of 5212 Amps.

Per Calculation EA-ELEC-FT-005, “Short Circuit Analysis for Palisades Class 1E Station Batteries D01 and D02,” the available fault current at Panel D11-2 was 12,889 Amps. Fuse FUZ/D11-2, which protected Panel D11-2, had an instantaneous trip of approximately 8000 Amps. The as-found setting on Breaker 72-01 which was installed upstream of FUZ/D11-2, was approximately 3400 Amps. These values verified what happened during the event on September 25, 2011, when the electrical fault occurred on Panel D11-2. Panel D11-2 was not isolated from the left train 125-Volt DC bus because the instantaneous trip device on Breaker 72-01 actuated upon the Panel D11-2 fault, which resulted in the loss of the left train 125-Volt DC bus and its associated preferred AC power sources.

Following the event, the licensee raised the instantaneous trip device setpoint on Breaker 72-01 as a compensatory measure to improve electrical coordination between the breaker and the downstream protective devices. The setpoint change was implemented per TM EC 32028. The instantaneous trip setpoint of Breaker 72-01 was subsequently left at a value of 4902 Amps. Breaker 72-02 was found initially set at the high instantaneous setting, so no setpoint change was required. The TM improved coordination issues, but both breakers still did not fully coordinate with the downstream protective devices connected to each bus. Therefore, the licensee had to take additional compensatory measures to ensure operability of the 125-Volt DC system per the operability evaluation for CR-PLP-2011-4835 and CR-PLP-2011-4965. These compensatory measures included: 1) prohibiting all work within Panels D11-1, D11-2, D21-1 and D21-2, and on cables connected to these panels within the cable spreading room (CSR); 2) allowing only one battery charger connected to an operable DC bus; and, 3) opening nonsafety-related load Breaker 72-17 for the public address (PA) system and Breakers 72-13, 72-14, 72-23 and 72-24 for the DC oil lift pumps on the DC busses. The inspectors reviewed the operability evaluation for CR-PLP-2011-4835 and CR-PLP-2011-4965, Revision 1K and did not identify any new issues.

Analysis: The inspectors determined that the licensee’s failure to correctly translate design basis into procedures or instructions, and failure to assure adequate quality were suitably included in procurement documents for safety-related material was contrary to

44 Enclosure

10 CFR Part 50, Appendix B, Criterion III, “Design Control,” and Criterion IV, “Procurement Document Control,” respectively and was a performance deficiency.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, “Power Reactor Inspection Reports,” Appendix B, “Issue Screening,” dated December 24, 2009, because it was associated with the Mitigating Systems Cornerstone attribute of Equipment Performance, and affected the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, lack of coordination between Panel D11-2 Fuse FUZ/D11-2 and Breaker 72-01 resulted in the loss of the left 125-Volt DC bus and two preferred AC power sources and complicated plant shutdown during the reactor trip on September 25, 2011, when an electrical fault occurred while working on Panel D11-2. The risk assessment for the September 25 event, and the complications caused by the as-found, latent design deficiency associated with the instantaneous trips for shunt trip Breaker 72-01 are evaluated in AV 2011014-02 documented in Section 4OA5.3.b.1 of this report.

The inspectors determined the finding could be evaluated using the SDP in accordance with IMC 0609, “Significance Determination Process,” Attachment 0609.04, “Phase 1 - Initial Screening and Characterization of findings,” Table 4a for the Mitigating Systems Cornerstone. The inspectors answered “Yes” to Question 1 in Column 2. Therefore, the inspectors determined that this finding could be screened as having very low safety significance (Green), because the finding was a design deficiency confirmed not to result in loss of operability or functionality of a system safety function.

In addition, the inspectors also determined that the finding affected the fire protection safe shutdown strategies. Therefore, screening under IMC 0609, Appendix F, “Fire Protection Significance Determination Process,” was required. Based on review of IMC 0609, the inspectors concluded that the finding represented a moderate degradation within the post-fire safe shutdown category and performed a Phase 2 analysis. Based on the licensee’s evaluation for the loads connected to DC busses D10 and D20, the turbine building (Fire Area 23) was the only fire area identified as being affected by the shunt trip breaker design deficiency. The safe shutdown-credited trains for the remaining fire areas were not affected. Breakers 72-13, 72-14, and 72-101 associated with PCP DC oil lift Pumps P-81A and P-81C and 4160 Switchgear EA-21 all had DC power cables routed in the turbine building which may experience fault currents that could potentially result in the loss of DC Bus D10. The loss of DC bus D10 for a turbine building fire was not identified in the Appendix R safe shutdown analysis. However, these cables only extended a few feet into the turbine building; therefore, the inspectors could not postulate a credible fire scenario that could damage these cables. The inspectors determined that this finding screened as having very low safety significance (Green) per Task 2.3.5, screening check for lack of fire ignition sources and fire scenarios.

The inspectors determined there was no cross-cutting aspect associated with this finding because the Breakers 72-01 and 72-02 were installed in 1981, and there was no indication from either engineering or maintenance documentation reviewed during the inspection that plant personnel recognized or should have been aware of this deficiency and, therefore, was not indicative of the licensee’s current performance.

45 Enclosure

Enforcement

Title 10 CFR Part 50, Appendix B, Criterion IV, “Procurement Document Control,” requires, in part, that measures shall be established to assure that applicable regulatory requirements, design bases, and other requirements, which are necessary to assure that adequate quality are suitably included or referenced in the documents for procurement of material, equipment, and services.

: Title 10 CFR Part 50, Appendix B, Criterion III, “Design Control” requires, in part, that measures shall be established to assure that applicable regulatory requirements and the design basis are correctly translated into specifications, drawings, procedures and instructions. Design control measures provide for verifying or checking the adequacy of design, such as by the performance of design reviews, by use of alternate or simplified calculational methods, or by the performance of suitable testing program.

Contrary to the above requirements, since the installation of the safety-related shunt trip Breakers 72-01 and 72-02 with thermal overloads on the 125-Volt DC safety-related system in 1981, the licensee failed to establish design control measures to assure that the applicable regulatory requirements and design bases were correctly translated into specifications and instructions. The licensee also failed to provide adequate measures for verifying and checking the adequacy of design. Additionally, the licensee failed to establish measures to assure that the applicable regulatory requirements and design basis, which were necessary to assure adequate quality, were suitably included or referenced in the documents for procurement of Breakers 72-01 and 72-02. Specifically, Breakers 72-01 and 72-02 were purchased and installed in 1981 with thermal overloads and instantaneous elements enabled, which did not meet the applicable regulatory requirements and design basis that stated the breakers would only be actuated manually. The specifications for procurement did not assure that the applicable regulatory requirements and design basis, which were necessary to assure adequate quality, were included or referenced in procurement documents. As a result, on September 25, 2011, when an electrical fault occurred on Panel D11-2, it was not isolated from the left train 125-Volt DC bus because the instantaneous trip device on Breaker 72-01 automatically actuated, resulting in the loss of the left train 125-Volt DC bus and two preferred 120-Volt AC power sources Y-10 and Y-30. Because this violation was of very low safety significance, and was entered into the licensee’s CAP as CR-PLP-2011-4835 and CR-PLP-2011-4965, this violation is being treated as an NCV, consistent with Section 2.3.2 of the NRC Enforcement Policy (NCV 05000255/2011014-08; Failure to Maintain Design and Procurement Control of the 125-Volt DC System).

At the end of this inspection, the licensee was still performing a root cause evaluation to determine the causes of the event and develop corrective actions. As a remedial corrective action prior to plant startup, the licensee implemented a TM to increase the breaker instantaneous trips and performed an operability evaluation of the 125-Volt DC system.

46 Enclosure

.7

a.

Evaluate The Extent Of The Electrical Transient On The Safety-Related Bus To Determine Additional Equipment Impacts

The inspectors reviewed documentation related to the 125-Volt DC and 120-Volt AC systems to assure compliance with their design and licensing basis. The documentation included the UFSAR, TSs, vendor documents, WOs, and corrective actions generated as a result of the electrical fault that occurred on September 25, 2011. The inspectors also interviewed plant and vendor personnel to gather information related to the failure of Breaker 72-01, Battery Charger D-15, and Inverter D-06. The following observations were related to the three major electrical components that were impacted by the electrical fault and transient on September 25, 2011.

Inspection Scope

b.

b.1

Findings and Observations

Assessment of the 125-Volt DC Systems

The 125-Volt DC electrical power system consisted of two independent and redundant safety-related Class 1E DC power sources. Each DC system consisted of battery, switchgear, distribution panels, two chargers and instrumentation (reference Attachment 4). Fuse FUZ/D018 was installed between Battery D-01 and its Bus D-10. The fuse was rated for 1200 Amps and provided adequate coordination with the downstream protective devices per EA-APR-95-004, Revision 4.

Left Train DC Bus D-10:

Shunt trip Breaker 72-01 was installed downstream of Fuse FUZ/D018 in 1981 to be used in conjunction with a shunt trip switch to isolate the balance of the left channel DC circuit from Panel D11A for a fire in the cable spreading room (CSR). The breaker was rated for 800 Amps and was provided with thermal overcurrent and instantaneous protective devices, which was not recognized by licensee personnel until the September 25, 2011, event. Following the event on September 25, 2011, the thermal element was satisfactorily tested to assure coordination. The instantaneous trip element was also tested and the as-found results showed the breaker was set at the low setting value of 3400 Amps. This low setting did not provide adequate coordination between this breaker and downstream protective devices. The licensee implemented TM EC-32028 and raised the trip setpoint for the shunt trip breaker to its high setting of 4902 Amps to improve coordination.

The licensee’s extent-of-condition inspection for Breaker 72-02 found the breaker’s instantaneous element set at the maximum setting, which was 5212 Amps.

Although the TM improved electrical coordination, it still did not provide adequate coordination between Breakers 72-01 and 72-02 and downstream devices. The PA system DC supply Breaker 72-17 did not coordinate with Breaker 72-01 such that a fault at the nonsafety-related load would cause a loss of the entire left train bus D10. The PCP motor DC oil lift Pumps P-81A and P-81C supplied right train Breakers 72-13 and 72-14, and did not coordinate with Breaker 72-01 such that some cable faults for these nonsafety-related loads could cause loss of Bus D10. The PCP DC oil lift Pumps P-81B and P-81D supplied Breakers 72-23 and 72-24 also did not coordinate with 72-02

47 Enclosure

such that some cable faults for these nonsafety-related loads could cause loss of Bus D20l.

Additional compensatory measures were implemented to assure that under a fault condition on downstream nonsafety-related loads, a similar event would not result in the loss of a full train of 125-Volt DC power on either the left or right train due to a lack of electrical coordination. These additional compensatory measures included: 1) prohibiting all work within Panels D11-1, D11-2, D21-1 and D21-2, and on cables connected to these panels within the cable spreading room (CSR); 2) allowing only one battery charger connected to an operable DC bus; and, 3) opening nonsafety-related load Breaker 72-17 for the public address (PA) system and Breakers 72-13, 72-14, 72-23 and 72-24 for the DC oil lift pumps on the DC busses. The inspectors did not have any additional operability concerns with the 125-Volt DC system following the implementation of the compensatory measures.

Each station battery had two associated battery chargers. One charger was powered by the train-specific AC power distribution system (i.e., the directly-connected chargers), and the other charger was powered from the opposite train’s AC power distribution system (the cross-connected chargers). The cross-connected chargers were not credited to meet TSs, specifically Limiting Condition for Operation (LCO) LCO 3.8.4. The battery chargers normally operated in pairs, either both directly connected chargers or both cross-connected chargers were in operation, to assure a diverse AC supply. During normal operation, the 125-Volt DC loads were powered from the battery chargers with the batteries floating on the system. In case of loss of normal power from the battery charger, the DC loads continue to be powered from the station batteries.

Battery Charger D-15:

On September 25, 2011, when the fault occurred on Panel D11-2, which resulted in a reactor and turbine trip and de-energiziation of Bus D-10, Battery Charger No. 1, D-15, was the inservice charger and attempted to feed the electrical fault. Protective devices installed in the system actuated to protect the charger from damage. During the event, Battery Charger D-15 experienced a large current surge due to the battery charger’s attempt to supply power to a faulted DC bus. As a result of the surge, the battery charger’s internal fuse blew open. The inspectors determined that it was also likely that the DC output breaker tripped. These devices actuated to protect the battery charger from damage, as designed.

According to Ametek, the manufacturer of the battery charger, the current-limiting feature of the battery chargers were not immediate (38 millisecond time delay) and were not designed to protect against large, instantaneous DC faults. Because the fault current was very high, the battery charger tripped before the current limiter was able to respond and compensate. This phenomenon was also discussed in IEEE-946-2004, “IEEE Recommended Practice for the Design of DC Auxiliary Power Systems for Generating Station.”

Following the event, troubleshooting per WO 291319 discovered that the charger’s internal Fuse F302 was blown. Fuse F302 was one of three AC phase inputs to the battery charger silicon-controlled rectifiers. With this fuse blown in the charger, the charger was only operating on two AC phase inputs. Battery voltage was oscillating because not all silicon-controlled rectifiers were operating as designed. Hence, the charger was not capable of providing its full design current with the fuse blown.

48 Enclosure

Fuses F301, F302, and F303 were all replaced and the charger was returned to service. Additionally, the X302 control board on the charger was also replaced during the troubleshooting as a defense in depth measure. The inspectors did not have any additional operability concerns with Battery Charger D-15 upon completion of the licensee’s corrective actions.

Inverters D-06, D-07, D-08, and D-09 were the normal source of power for preferred AC busses Y10, Y20, Y30, and Y40, respectively. The 120-Volt preferred AC system provided power for the four separate RPS channels. The function of the inverter was to provide continuous 120-Volt AC electrical power to the preferred AC busses, even in the event of an interruption to the normal AC power distribution system. A preferred AC bus could be powered from the AC power distribution system via the bypass regulator if its associated inverter was out of service. An interlock prevented supplying more than one preferred AC bus from the bypass regulator at any time.

Inverter D-06 to Preferred AC Bus Y-10:

On September 25, 2011, the fault that occurred on Panel D11-2 resulted in a reactor and turbine trip, and de-energiziation of Bus D-10. Breaker 72-37, which supplied DC power to Inverter D-06, was found tripped. According to the manufacturer, Ametek, the inverters were capable of reverse-feeding DC short circuits for short durations, which could have caused Breaker 72-37 to trip. This was possible since the inverter had four 7700 microFarad capacitors in parallel on the DC side of the inverter, which, during a DC short circuit, the capacitors would rapidly discharge the capacitors and feed the fault.

Breaker 72-37 was a Westinghouse Model HFD breaker and had a rating of 100 Amps for the thermal setting and 700 Amps for the magnetic setting. According to the manufacturer an approximation for an inverter DC fault current contribution was about 1100 Amps per capacitor; therefore, this was approximately a total of 4400 Amps for Inverter D-06. This exceeded the magnetic rating of the breaker and explained why the breaker tripped during the fault condition.

Following the event, all other breakers and internal fuses on Inverter D-06 were checked, and found to be satisfactory. Following the restoration of Bus D-10, Breaker 72-37 was closed, and Inverter D-06 was returned to service.

b.2 Unresolved Item: Potential Loss of Preferred AC Sources in Harsh Environment

Introduction: The inspectors identified an unresolved item (URI) during review of the failure of Inverter D-06 as a result of the fault condition on Panel D11-2 on September 25, 2011. Specifically, the inspectors were concerned with the following: 1) that several nonsafety-related and non-qualified cables associated with the four PCP DC oil lift pumps were routed in a harsh environment and were supplied from the safety-related busses; and, 2) that without further analysis a low probability condition could exist, which could result in the loss of all safety-related inverters and preferred AC sources.

Description: On September 25, 2011, a fault occurred on Panel D11-2, which resulted in reactor and turbine trip, and de-energiziation of Bus D-10. Breaker 72-37, which supplied DC power to Inverter D-06, was found tripped. According to the manufacturer, the inverters were capable of reverse-feeding DC short circuits for short durations and this could have caused Breaker 72-37 to trip. This was possible because the inverter

49 Enclosure

had four 7700 microFarad parallel capacitors on the DC side of the inverter. During a DC short circuit, the capacitors would rapidly discharge and feed the fault.

Breaker 72-37 had a rating of 100 Amps for the thermal setting and 700 Amps for the magnetic setting. According to the manufacturer an approximation for an inverter DC fault current contribution was about 1100 Amps per capacitor; therefore, this was approximately a total of 4400 Amps for Inverter D-06. This exceeded the magnetic rating of the breaker and explained why the breaker tripped during the fault condition.

The PCP motor DC oil lift Pumps P-81A and P-81C were nonsafety-related loads, which received power from Bus D-10 via safety-related Breakers 72-13 and 72-14, respectively. The PCP motor DC oil lift Pumps P-81B and P-81D were also nonsafety-related loads that received power from D-20 via safety-related Breakers 72-23 and 72-24, respectively. The cabling for these loads was not environmentally qualified and was routed through containment, which could be susceptible to failure due to a harsh environment. The inspectors were concerned that if all four nonsafety-related cables for these pumps faulted due to a harsh environment during a design basis event, this could result in the loss of all preferred AC power busses due to the internal capacitors contributing to the fault as seen by each DC bus. However, without further analysis of the design and licensing basis, the inspectors could not determine if a postulated harsh environment affecting all four cables during a design basis event was a credible event. Therefore, the inspectors’ initial conclusion, based on the available information was that this event may not be credible; however, further analysis was required. In addition, all four PCP motor DC oil lift pump breakers were opened as one of the compensatory measures for the operability of the 125-Volt DC system. Therefore, this is not a current safety concern.

Title 10 CFR 50.49, “Environmental Qualification of Electrical Equipment Important to Safety for Nuclear Power Plants,” Section b(2), requires nonsafety-related electric equipment to be environmentally qualified if the failure of the nonsafety-related electric equipment under postulated environmental conditions could prevent satisfactory accomplishment of safety functions specified in subparagraphs (b)(1)(i) (A) through (C) of paragraph (b)(1) of this section by the safety-related equipment. The inspectors were concerned that the cables associated with the PCP motor DC oil lift Pumps P-81A, P-81B, P-81C, and P-81D were not evaluated for the effect on the safety-related equipment specifically the safety-related inverters and their associated preferred AC sources.

The licensee entered this issue into their CAP as CR-PLP-2011-6210. This issue is a URI pending the licensee evaluation, and the inspectors’ review of the licensees design and licensing basis, and evaluation to determine if a performance deficiency existed (URI 05000255/2011014-09; Potential Loss of Preferred AC Sources in Harsh Environment).

.8

a.

Determine The Impact Of The Loss Of Left Train DC Busses On Capability/Functionality Of The AFW System And Atmospheric Dump Valves

The inspectors reviewed the loss of left train DC event with respect to the operation of the AFW and the ASDVs. The plant process computer (PPC) data, control room chart

Inspection Scope

50 Enclosure

recordings, and post-event trip report were assessed to determine the AFW and ASDV systems’ responses during this event. This data was then compared to the plant’s licensing basis documents to ensure that the systems operated as designed and that no adverse impacts were encountered during the transient. Through this review, the inspectors concluded that the AFW and ASDV systems operated as designed given the parameters of this event. The data gained from this review also aided in the probabilistic risk analysis conducted by the NRC’s regional SRA for an overall assessment of the risk associated with this transient.

b.

With respect to the operation of these systems, some complications were seen during the loss of left train DC bus event but none that jeopardized the safe shutdown or cooldown of the reactor or primary and secondary systems. Upon receipt of the SIAS from the loss of the two preferred 120-Volt AC sources, Y-10 and Y-30, the right channel equipment was actuated, and AFW Pump P-8C received a start signal, as designed. The pump actually started approximately one minute after receipt of the start signal due to a time delay built into the SIAS and AFAS logic for Pump P-8C. The AFW Pump P-8B started when power was lost to its steam supply Valve CV-0522B, which failed open on the loss of DC power from Panel D11-1. An AFAS signal was also received due to a loss of the two preferred AC sources, Y-10 and Y-30. The AFW Pump P-8A did not start, however, because it lost power with the loss of the left train. The two running AFW pumps during this event (P-8B and P-8C) both fed SGs ‘A’ and ‘B’ while in operation.

Findings and Observations

Approximately 25 minutes into the event, an AO was dispatched to the field to respond to a fire alarm in the AFW pump room (where AFW Pumps P-8A and P-8B were located). During this response, it was discovered that the turbine steam supply Valve CV-0522B for Pump ‘B’ was fully open and admitting steam into the room. The AO was directed by control room supervision to manually close this valve, isolating AFW Pump ‘B’ since the pump was not needed to respond to the transient. With Pump P-8B isolated and Pump P-8A unavailable due to the loss of power, the entire left train of AFW was inoperable, and the appropriate TS LCO was entered. At that time, the levels in both SGs were within or above their required operating bands and the additional pump capacity was not needed to respond to the transient. Following the event on September 25, 2011, power was restored to the preferred AC sources, rendering Pump ‘A’ operable, and Pump ‘B’ was restored following the re-opening of Valve CV-0522B, and the controller being placed in the “automatic” operation mode.

Approximately 25 minutes after this, the level in SG ‘A’ was recorded at a maximum value of approximately 98 percent, with the only FW source being AFW Pump P-8C. There were no overall consequences of the high level in SG ‘A’; however, had the level continued to increase, water could have entered the main steam lines.

The ASDVs were also affected by the loss of DC power. With the loss of the preferred AC source, Y-10, the four ASDVs lost power to the master controller, as well as the quick-open capability (i.e., Bus Y-10 powered the relay in the logic needed for this action). Since the ASDVs were unable to be operated either manually or automatically, only the main steam safety valves were available for secondary side pressure control. The operators could also have utilized the turbine bypass valve that could have been manually operated from the control room; however, condenser vacuum would first have

51 Enclosure

to be restored, which was allowed by site procedures. From a review of PPC data and control room chart recordings, the inspectors determined that until power was restored to Bus Y-10, the first set of main steam safety valves controlled secondary side pressure because the setpoint pressures were reached. The lifting of the main steam safety valves was documented in the licensee’s CAP as CR-PLP-2011-04939. Once power was restored to Y-10 later that day, the ASDVs were considered operable and were able to function automatically. The inspectors did not identify any performance deficiencies.

.9

a.

Review The Licensee’s Root Cause Evaluation Plan And Schedule. Include The Schedule For Performing Testing Of Any Components That Failed During The Transient. Evaluate Whether The Root Cause Evaluation Plan Is Of Sufficient Depth And Breadth. Confirm That The Time Allowed To Perform The Root Cause Evaluation Is Commensurate With The Safety Significance Of This Issue.

The inspectors reviewed the root cause charter, schedule, team make-up and action plan, and discussed actions with the root cause team.

Inspection Scope

b.

No findings were identified. The inspectors found the root cause team was comprised of site individuals with an independent reviewer. Some of the team members had been involved with the September 25, 2011, event. Team members included representatives of the maintenance, training, system engineering, human performance/industrial safety, and operations departments. The root cause timeline was scheduled to be completed within a timeframe commensurate with the significance of the issue and was in accordance with licensee procedures.

Findings and Observations

The root cause investigation charter was determined to be of adequate depth and breadth to be successful in determining the actual root cause.

4OA6

On October 28, 2011, the special inspection team leader presented the preliminary inspection results to Mr. A. Vitale and members of his staff. No proprietary information is included in this inspection report.

Meetings, Including Exit

ATTACHMENTS: 1. Supplemental Information 2. Special Inspection Team Charter 3. Palisades Event Timeline 4. Simplified Diagrams of Palisades 125-Volt DC System 5. Images Of Palisades 125-Volt Dc System During/Following Maintenance 6. Permission to Utilize Graphics/Visual Aids 7. List of Major Affected Equipment 8. Phase 3 Significance Determination Process Detailed Analysis for the Failure to

Have Adequate Work Instructions

1 Attachment 1

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

A. Vitale, Site Vice President

Licensee Personnel

D. Hamilton, General Manager Plant Operations C. Arnone, Nuclear Safety Assurance Director A. Blind, Engineering Director O. Gustafson, Licensing Manager P. Russell, PS&O Manager B. Baker, Maintenance Manager D. Corbin, Acting Operations Manager R. White, Assistant Operations Manager J. Haumersen, Systems Engineering Manager S. Heffler, Electrical Maintenance Superintendent D. Lucy, Online Maintenance Scheduling Superintendent L. Marvin, Human Resources Manager J. Miksa, Programs & Components Engineering Manager B. Nixon, Training Supervisor C. Plachta, QA Manager D. Rogers, Site VP Coordinator M. Savage, Sr. Lead Communications Specialist C. Sherman, Radiation Protection Manager B. Dotson, Licensing Specialist IV N. Lane, Manager of Projects E. Weinkam (telecon), Sr. Licensing Manager P. Schmidt, Training Superintendent T. Reddy, MP&C Manager T. Mulford, Shift Manager G. Sleeper, Assistant Operations Manager T. O’Leary, CA&A Manager D. Berkenpas, Security Manager L. Engelke, Engineering Supervisor D. MacMaster, Acting Design Engineering Manager R. VanWagner, DFS Manager

S. West, Director, DRP

NRC Personnel

J. Giessner, Branch Chief J. Ellegood, Sr. Resident Inspector T. Taylor, Resident Inspector A. Scarbeary, Resident Inspector R. Krsek, Sr. Resident Inspector-Kewaunee A. Dahbur, RIII Inspector C. Zoia, RIII Inspector J. Jandovitz, RIII Inspector R. Lerch (telecon), RIII Project Engineer D. Passehl (telecon), RIII SRA

2 Attachment 1

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

05000255/2011014-01

Opened

NCV Failure to Report a 10 CFR 50.72 Notification for an 8-hour Non-Emergency Report (Section 4OA5.2.b.1)

05000255/2011014-02 AV Failure to Have Adequate Work Instructions For Work Performed on Panel D11-2 (Section 4OA5.3.b.1)

05000255/2011014-03 FIN Failure to Implement Human Performance Tools and to Perform an Infrequently Performed Test or Evolution Brief (Section 4OA5.3.b.2)

05000255/2011014-04 FIN Failure to Comply With Work Hour Rules for Non-Covered Workers (Section 4OA5.3.b.3)

05000255/2011014-05 NCV Failure to implement Emergency Operating Procedure Immediate Actions (Section 4OA5.4.b.1)

05000255/2011014-06 NCV Failure to Establish a Procedure for the Loss of a DC Bus and the Simultaneous Loss of Two Preferred AC Power Sources (Section 4OA5.4.b.2)

05000255/2011014-07 NCV Failure to perform an Adequate Operability Evaluation (Section 4OA5.5.b.1)

05000255/2011014-08 NCV Failure to Maintain Design and Procurement Control of the 125-Volt DC System (Section 4OA5.6.b.1)

05000255/2011014-09 URI Potential Loss of Preferred AC Sources in Harsh Environment (Section 4OA5.7.b.2)

05000255/2011014-01

Closed

NCV Failure to Report a 10 CFR 50.72 Notification for an 8-hour Non-Emergency Report (Section 4OA5.2.b.1)

05000255/2011014-03 FIN Failure to Implement Human Performance Tools and to Perform an Infrequently Performed Test or Evolution Brief (Section 4OA5.3.b.2)

05000255/2011014-04 FIN Failure to Comply With Work Hour Rules for Non-Covered Workers (Section 4OA5.3.b.3)

05000255/2011014-05 NCV Failure to implement Emergency Operating Procedure Immediate Actions (Section 4OA5.4.b.1)

05000255/2011014-06 NCV Failure to Establish a Procedure for the Loss of a DC Bus and the Simultaneous Loss of Two Preferred AC Power Sources (Section 4OA5.4.b.2)

05000255/2011014-07 NCV Failure to perform an Adequate Operability Evaluation (Section 4OA5.5.b.1)

05000255/2011014-08 NCV Failure to Maintain Design and Procurement Control of the 125-Volt DC System (Section 4OA5.6.b.1)

None.

Discussed

3 Attachment 1

LIST OF DOCUMENTS REVIEWED

- (AMMS) WO 24000315; Verification Of C/R Alarm Window K05-48

4OA5 Other Activities – Special Inspection

- (AMMS) WO 24323547; Verify Alarm K05-48 From Left Channel - (AMMS) WO 24422813; Verify Alarm K05-48 From Right Channel - (AMMS) WO 24614371; Verification Of C/R Alarm Window K05-48 - 10.51; Writer’s Guideline For Site Procedures; Revision 19 - 108 Trend Plot; Shift Hourly Trends; Group 11; PAL September 25, 2011 And

September 29, 2011 - 292 Event Log; Area 1; September 25, 2011 19:53:48 To 20:01:48 - 3.8.4; DC Sources – Operating; Palisades Nuclear Plant Amendment No. 189 - 4.00; Shift Relief and Turnover – Attachment 13; Revision 43 - 4.02; Control Of Equipment; Revision 59 - 4.08; Plant Personnel Statements; Attachment 2; Completed September 25, 2011 - 4.08; Post Event Review Requirements; Completed September 29, 2011 - 4.08; Post Event Review Requirements; Revision 6 - 5.18; Control Of Work Instructions; Revision 7 - B 3.8.4; DC Sources – Operating; Palisades Nuclear Plant Revised July 13, 2006 - Characteristic Curve Sheet And Pump Data For P-8C (Auxiliary Feedwater Pump ‘C’);

Bingham Pump Co.; September 10, 1968 - Chemistry Log; Day Shift; September 24, 2011 - Chemistry Log; Night Shift; September 24, 2011 - Component/Plant Impact For WO 291194-01; Replace Defective 72-119 Breaker With New

Breaker - Component/Plant Impact For WO 291210-01; Replace Defective 72-121 Breaker With New

Breaker - CR-PLP-2011-04288; Palisades September 25, 2011, Reactor Trip, Corporate Event

Response Team Report; dated October 12, 2011 - CR-PLP-2011-04566; Pre-Job Brief Conducted On September 13, 2011 For Use Of MANTA

Test Set - CR-PLP-2011-04821; Reportable As LER Only Under 10CFR50.73 - CR-PLP-2011-04822; Unplanned, Automatic Reactor Trip - CR-PLP-2011-04823; DC Input Breaker To Inverter ED-06 Was Found Tripped Free;

September 25, 2011 - CR-PLP-2011-04824; Main Generator Output Breakers Did Not Open Automatically - CR-PLP-2011-04826; Battery Charger Number 1 Had Zero Output Following The Event;

September 25, 2011 - CR-PLP-2011-04829; CRS PPC Lost Indication - CR-PLP-2011-04835; Discrepancy With Breaker 72-01, Isolation Breaker To DC Battery No. 1 - CR-PLP-2011-04858; Pressurizer Level Exceeded The TS Limit Of 62.8% - CR-PLP-2011-04864; Control Room Switchyard Supervisory Panel (EC-07) Has No Display

Of Switchyard Breaker Status - CR-PLP-2011-04865; ‘B’ Steam Generator Level Rose Sharply Then Slowly Returned - CR-PLP-2011-04872; PCS Loop To Pressurizer Delta T Exceeded 200 Degrees F During

Plant Cooldown - CR-PLP-2011-04879; Main Steam Isolation Valves Closed At Full Load Following Plant Trip - CR-PLP-2011-04890; Control Room Supervisor Authorized A Control Band For Pressurizer

Level Outside Of Recommended Band In SOP-1B - CR-PLP-2011-04897; ED-15, Station Battery Charger Number 1 Was Declared Inoperable

4 Attachment 1

- CR-PLP-2011-04908; SRO And NCO Turnover Items Check Sheets Not Completed - CR-PLP-2011-04920; Fuse F302 Found Open In Battery Charger ED-15; September 29, 2011 - CR-PLP-2011-04929; Auxiliary Feedwater P-8C Tripped While Performing Line Purging IAW

SOP-12, Att. 5 - CR-PLP-2011-04931; Potential Trend In Operator Control Of The Plant - CR-PLP-2011-04939; Post Trip Data Review Indicates Steam Generator Code Safety Relief

Valves May Have Opened - CR-PLP-2011-04965; Appendix R Protective Device Coordination Is Not Maintained For Some

Short Circuit Scenarios - CR-PLP-2011-04968; Comp Measures for Op Eval CR-PLP-2011-04835 Tagged Out Breaker

72-17, “PA System/Motor Generator,” And Results In Alarm EK-0548 Being Locked-In In The Control Room

- CR-PLP-2011-04972; IEEE 450-1995 Cell Parameter Readings And Battery Inspection Not Performed On Battery ED-01

- CR-PLP-2011-04974; IEEE 450-1995 Cell Parameter Reading and Battery Inspection Were Not Performed On Station Battery ED-01 Following The Fault That Occurred In DC Panel ED11-2

- CR-PLP-2011-04981; Failure To Use A Pre-Job Brief Form In Accordance With EN-HU-102 - CR-PLP-2011-04988; 50.59 Screening Performed For Temporary Modification To Implement

Compensatory Measures For EC 32028 Was Not Adequately Documented - CR-PLP-2011-05116; Duty Station Manager Exceeded Non-Covered Administrative Limits - CR-PLP-2011-05132; Unable to Find Condition Report For A Material Deficiency

(NRC Identified) - CR-PLP-2011-05154; Near Miss For Equipment Integrity – Main Generator Breakers (25H9

And 25F7) Did Not Immediately Open - CR-PLP-2011-05994; P-8C, AFW Pump, Was Run On Recirc For A Period Of Time That

Exceeded The SOP-12 Limit - CR-PLP-2011-06021; Fatigue Assessment Not Performed Per EN-OM-123 - CR-PLP-2011-06079; Logkeeping During Event Did Not Meet Standards Of EN-OP-115-09 Or

Admin 4.00 - CR-PLP-2011-06080; Post-Event Review Report Did Not Contain Sufficient Detail - CR-PLP-2011-06081; Operating Crew’s Actions When The Main Generator Output Breakers

Remained Closed Following The Turbine Trip Were Inadequate - CR-PLP-2011-06082; Crew And Station Knowledge Of TS LCO 3.8.6, Battery Cell

Parameters, Is Weak - CR-PLP-2011-06083; Simulator Is Unable To Adequately Re-Create The Transient - CR-PLP-2011-06084; Post-Event Organizational Response Was Inadequate - CR-PLP-2011-06085; Routine Simulator Training For LOI/LOR Dofes Not Include Full

Evaluation Of All Required Administrative Tasks - CR-PLP-2011-06086; Crew Initiated A PCS Leak When Swapping PZR Level And Pressure

Controllers - CR-PLP-2011-06209; NRC SIT Green Finding For Lack Of A Procedure For The Loss Of Both

Preferred AC Busses - CR-PLP-2011-06210; NRC SIT URI For Nonsafety-related Electrical Equipment Preventing

Satisfactory Accomplishment Of Safety Functions - Design Basis Document-1.03; Auxiliary Feedwater System; Revision 8 - Design Basis Document-1.09; Main Steam System; Revision 3 - Design Basis Document-2.04; Primary Coolant System; Revision 7 - Drawing; Main Steam System; HIC-0780A Tave Error Signal Program - Drawing; Main Steam System; Steam Dump Control - E-116; Schematic Diagram For Generator A.C.B.’s Interposing Control; Revision 8

5 Attachment 1

- E-117; Schematic Diagram For Switchyard & Power Plant Interposing Relays; Sheet 1; Revision 35

- E-8, Sheet 1; Single Line Meter And Relay Diagram – 125V DC, 120V Instrument And Preferred AC System; Revision 57

- E-8, Sheet 2; Single Line Meter And Relay Diagram – 125V DC, 120V Instrument And Preferred AC System; Revision 54

- EA-APR-95-004; 10CFR 50, Appendix R Shutdown Associated Circuit Analysis For Common Power Supplies And Common Enclosure; Revision 4

- EA-ELEC-FLT-005; Short Circuit Analysis For Palisades Class 1E Station Batteries D01 And D02; Revision 0

- EAL BASIS; Emergency Action Level Technical Bases; Revision 4 - EA-SC-98-012-02; Analysis Of Electrical Critical Characteristics For Replacing The P-50D

Primary Coolant Pump Motor; Completed January 26, 2000 - EC 32028; Circuit Breakers 72-01 And 72-02; Revision 1 - EC-18181; Replace/Test DC Breakers 72-119 Thru 72-136; Closed November 4, 2010 - EI-1; Emergency Classification And Actions; Revision 54 - EM-04-08; Shutdown Margin Requirements; Revision 30 - Email From Bret Baker; Subject: Non-Conforming Material; August 29, 2011 - Email From Charles Arnone; Subject: Site Standdown Documents; September 26, 2011 - Emergent Issues List; September 29, 2011 - Emergent Issues Open Items List; Printed September 28, 2011 - EN 47322; Appendix R Non-Compliance Issue Identified; Event Date September 26, 2011 - EN-FAP-OM-006; Working Hour Limits For Non-Covered Workers; Revision 2 - EN-FAP-WM-011; Work Planning Standard; Revision 1 - EN-HU-102; Attachment 9.5; Human Performance Tools; Revision 6; Medium Risk (Standard)

Pre-Job Brief Completed September 25, 2011 - EN-HU-102; Human Performance Tools; Revision 7 - EN-HU-103; Human Performance Error Reviews; Completed September 27, 2011 - EN-IS-123; Electrical Safety; Revision 8 - EN-LI-100; Process Applicability Determination; Revision 10 - EN-LI-102; Corrective Action Process; Revision 16 - EN-MA-101; Fundamentals Of Maintenance; Revision 9 - EN-MA-118; Foreign Material Exclusion; Completed September 23, 2011 - EN-MA-125; Attachment 9.3; Troubleshooting Control Form; Completed September 24, 2011 - EN-MA-125; Troubleshooting Control Of Maintenance Activities; Revision 8 - EN-NS-102; Fitness For Duty Program; Revision 9 - EN-OP-104; Operability Determination Process; Revision 5 - EN-OP-116; Infrequently Performed Tests Or Evolutions; Revisions 008 And 009 - EN-WM-100; Work Request (WR) Generation, Screening And Classification; Revision 6 - EN-WM-101; On-Line Work Management Process; Revision 7 - EN-WM-102; Work Implementation And Closeout; Revision 6 - EN-WM-104; On Line Risk Assessment; Revision 4 - EN-WM-105; Planning; Revision 9 - EOP Supplement 19; Alternate Auxiliary Feedwater Methods; Revision 10 - EOP-1.0; Standard Post-Trip Actions; Revision 13 - EOP-9.0; Functional Recovery Procedure; Attachment 1; Revision 21 - EOP-9.0; Functional Recovery Procedure; Revision 21 - EOP-9.0; Success Path HR-2; Functional Recovery Procedure; Revision 22 - EOP-9.0; Supplement A; Reactivity Control; Revision 19 - EOP-9.0; Supplement B; Maintenance Of Vital Auxiliaries – Electric; Revision 19 - EOP-9.0; Supplement C; PCS Inventory Control; Revision 19

6 Attachment 1

- EOP-9.0; Supplement CA-1; Containment Air Coolers (Normal Mode); Revision 19 - EOP-9.0; Supplement CA-2; Containment Air Coolers (Emergency Mode); Revision 20 - EOP-9.0; Supplement CA-3; Containment Spray; Revision 21 - EOP-9.0; Supplement CA-CA; Continuing Actions; Revision 20 - EOP-9.0; Supplement CI-1; Automatic/Manual Isolation; Revision 19 - EOP-9.0; Supplement CI-CA; Continuing Actions; Revision 19 - EOP-9.0; Supplement D; PCS Pressure Control; Revision 19 - EOP-9.0; Supplement E; PCS/Core Heat Removal; Revision 19 - EOP-9.0; Supplement F; Containment Isolation; Revision 19 - EOP-9.0; Supplement G; Containment Atmosphere; Revision 19 - EOP-9.0; Supplement G-1; Alternate Indications/Actions For Inoperable Instrumentation

Located Inside Containment; Revision 19 - EOP-9.0; Supplement G-2; Installing Temporary Pressure Gauges On P-8B Steam Supply

Line; Revision 19 - EOP-9.0; Supplement H; Maintenance Of Vital Auxiliaries – Water; Revision 19 - EOP-9.0; Supplement HR-1; Via S/G With SIS NOT In Operation; Revision 19 - EOP-9.0; Supplement HR-2; Via S/G With SIS In Operation; Revision 22 - EOP-9.0; Supplement HR-CA; Continuing Actions; Revision 19 - EOP-9.0; Supplement I; Maintenance Of Vital Auxiliaries – Air; Revision 19 - EOP-9.0; Supplement IC-1; CVCS Or SI Throttled; Revision 19 - EOP-9.0; Supplement IC-2; Safety Injection; Revision 22 - EOP-9.0; Supplement IC-CA; Continuing Actions; Revision 19 - EOP-9.0; Supplement LTA; Long Term Actions; Revision 19 - EOP-9.0; Supplement MVAA-1; Instrument Air Compressors; Revision 19 - EOP-9.0; Supplement MVAA-2; FWP Building Air; Revision 19 - EOP-9.0; Supplement MVAA-CA; Continuing Actions; Revision 19 - EOP-9.0; Supplement MVAE-AC-1; Offsite Power; Revision 19 - EOP-9.0; Supplement MVAE-AC-2; Diesel Generators; Revision 19 - EOP-9.0; Supplement MVAE-AC-3; Backfeeding Main XFMR; Revision 19 - EOP-9.0; Supplement MVAE-AC-CA; Continuing Actions; Revision 19 - EOP-9.0; Supplement MVAE-AC-CA; Continuing Actions; Revision 19 - EOP-9.0; Supplement MVAE-AC-DC-1; Battery Chargers/Station Batteries; Revision 20 - EOP-9.0; Supplement MVAW-1; Service Water And Component Cooling Water; Revision 20 - EOP-9.0; Supplement MVAW-CA; Continuing Actions; Revision 19 - EOP-9.0; Supplement PC-1; Subcooled Pressure Control; Revision 21 - EOP-9.0; Supplement PC-2; PORVs; Revision 19 - EOP-9.0; Supplement PC-3; Saturated Pressure Control; Revision 21 - EOP-9.0; Supplement PC-CA; Continuing Actions; Revision 19 - EOP-9.0; Supplement RC-1; Control Rod Insertion; Revision 19 - EOP-9.0; Supplement RC-2; Boration Using CVCS; Revision 20 - EOP-9.0; Supplement RC-3; Boration Using SIS; Revision 207 - EOP-9.0; Supplement RC-CA; Continuing Actions; Revision 19 - FC-407-14C; Design Basis and Criteria for the 125-Volt DC Distribution System;

January 19, 1981 - FE-SA; Modified Performance Test – Battery No. ED-01; Revision 17 - Fleet Engineering Review Of Palisades Forced Outage Response - UFSAR Chapter 1 – Introduction & General Description Of Plant; Revision 28 - UFSAR Chapter 14 – Safety Analysis; Figures 14.7.1-1 To 14.7.2.5; Revision 21 - UFSAR Chapter 14 – Safety Analysis; Revision 26 - UFSAR Chapter 4 – Primary Coolant System; Revision 25 - UFSAR Chapter 4 – Primary Coolant System; Revision 28

7 Attachment 1

- UFSAR Chapter 8 – Electrical Systems; Revision 29 - GOP-14; Shutdown Cooling Operations; Revision 43 - GOP-8; Power Reduction And Plant Shutdown To Mode 2 Or Mode 3 ≥ 525°F; Revision 28 - GOP-9; Mode 3 ≥ 525°F To Mode 4 Or Mode 5; Revision 31 - IEEE Std 946-2004; IEEE Recommended Practice For The Design Of DC Auxiliary Power

Systems For Generating Stations - LCO Board Log; September 25, 2011 - LEL-PLP-2011-080010; Performed Observation Of PJB; September 25, 2011 - MSE-E-23; Battery ED-01 Tested September 30, 2011 - NCO Shift Turnover Checklists; September 25-27, 2011 - NEI 96-07; Revision 1; November 2000 - NRC RIS 2005-20, Operability Determinations And Functionality Assessments For Resolution

Of Degraded Or Nonconforming Conditions Adverse To Quality Or Safety; Revision 1 - OCC Turnover; Day 5; September 30, 2011 1800 Hours - ONP-2.3; Loss Of DC Power; Revisions 16 And 17 - ONP-23.1; Primary Coolant Leak; Revision 25 - ONP-24.1; Loss Of Preferred AC Bus Y10; Revision 24 - ONP-24.3; Loss Of Preferred AC Bus Y30; Revision 24 - ONP-25.2; Alternate Safe Shutdown Procedure – Attachment 5; Revision 23 - ONP-4.1; Spurious Containment Isolation; Revision 20 - ONP-7.1; Loss Of Instrument Air; Revision 13 - Operability Evaluation No. CR-PLP-2011-4835 And 4965; Circuit Breakers 72-01 And 72-02;

September 30, 2011 - Operations Log; Day Shift; September 24-25, 2011; October 2, 2011 - Operations Log; Night Shift; September 24, 2011; October 1, 2011 - Palisades Open Mode Restraint List; September 29, 2011 - Palisades Root Cause Team Detailed Timeline - Plant Process Computer Reports; September 25, 2011 - PL-MEL-304-002L; Operate Manta Test Equipment; Completed September 21, 2011 - PLP-1F2203CS-E10; Pal CSD Forced Outage 1F2203S (Mode 5); September 27, 2011 And

September 30, 2011 - Preliminary Review Of De-Energized Equipment On 09/25/11 Plant Trip - QO-21C; Surveillance Test Data For P-8C; January 2011 To July 2011 - Radiation Protection Log; September 23-25, 2011 - RE-83A; Service Test – Battery No. ED-01; Revision 18 - RM-29A; Attachment 1; Data Sheet; Revision 0 - SA4; System Malfunction – Initiating Condition - Alert; Revised January 2003 - Shift Logs; September 23–25, 2011 - SOER Number 81-15; Operations: Loss Of DC And Transfer Of Essential AC; Palisades;

1981 - SOP-12; Feedwater System; Revision 60 - SOP-13; Air Ejector, gland Steam Condenser And Condenser Vacuum Pump; Revision 30 - SOP-1B; Attachment 6; Opening Of PORV Isolation Valves; Revision 11 - SOP-1B; Attachment 8; PCS Equipment Elevations And Volumes; Revision 11 - SOP-1B; Primary Coolant System – Cooldown; Revision 11 - SOP-2A; Chemical And Volume Control System; Revision 73 - SOP-30; Station Power; Revision 63 - SOP-38; Gaseous Process Monitoring System; Revision 30 - SPS-E-23; Testing Of AC Or DC Molded Case Circuit Breakers Without Static Trip Devices;

Completed September 23, 2011 - Station Log; Night Shift; September 30, 2011

8 Attachment 1

- SU3; System Malfunction – Notification Of Unusual Event; Revised January 2003 - Temporary Modification Log; September 24-25, 2011 - TM EC-32028; – Breaker 72-01 and 72-02 Setting Change; - TM EC-32028; Breaker 72-01 And 72-02 Setting Change - Updated Final Safety Analysis Report; Revision 27 - Various Responses To 125 Requests For Information From NRC To Licensee - VEN E11B, Sheet 16; Schematic Diagram – 6KVA Inverter; Revision 2 - Vendor Manual 950Y179*M1-D; Sheet 999; Primary Coolant Pipe And Fittings;

November 1968 - Vendor Manual M0001EB Sheet 2034 (VTD-0001-0042); Oil Lift System - Vendor Manual M0001EB Sheet 856 (VTD-0001-0042); Oil Lift System - WO 212303; ED-11-2 Breaker Testing For ED-01 - WO 214401; Verification Of C/R Alarm Window EK-0548 - WO 214878; Verify Alarm EK-0548 From Right Channel - WO 291123; Breaker 72-123 Found To Have Open Phase - WO 291209; 72-121: Install TM To Power 72-121 From Alternate Source - WO 291210; 72-121: Replace Breaker/ED-11-2; Restore Bus; September 27, 2011 - WO 291601; 72-01: Check As-Founds, Adjust Trip Settings, Check As-Lefts - WO 291603; 72-02: Check As-Founds, Adjust Trip Settings, Check As-Lefts - WO 291904; ED-01

- CR-PLP-2011-04819; Procedure EN-OM-123 Requires A CR To Be Generated As Part Of A Fatigue Assessment After An Event

Condition Reports Generated As A Result Of September 26, 2011, Incident

- CR-PLP-2011-04820; 72-127: Relay Testing Delayed Approximately Two Hours Due To No Electrical Supervisor Coverage

- CR-PLP-2011-04821; Spurious Containment High Rad Initiation, SI Initiation, Due To Loss Of Power/Preferred AC Bus EY-10/30

- CR-PLP-2011-04822; Unplanned Reactor Trip During Maintenance On DC Supply Panel - CR-PLP-2011-04823; DC Input Breaker To Inverter ED-06 To Preferred AC Bus Y-10 Tripped

Free - CR-PLP-2011-04824; Following Reactor Trip, Main Generator Output Breakers Did Not Open

Automatically - CR-PLP-2011-04825; Letdown Relief Valve RV-2006 Lifted During Containment Isolation

During Reactor Trip - CR-PLP-2011-04826; During Reactor Trip, Battery Charger #1 On Battery #1 Appears To

Have Zero Output - CR-PLP-2011-04827; P-55B Charging Pump Suction Relief Valve Lifted And Did Not Reseat - CR-PLP-2011-04828; P-55A Charging Pump Suction Relief Valve Lifted And Did Not Reseat - CR-PLP-2011-04829; During Unplanned Reactor Trip, CRS PPC List Indication - CR-PLP-2011-04830; During Plant Trip Operations Noted Inboard And Outboard Seals For “B”

Main Feed Pump Leaking - CR-PLP-2011-04832; Waiver Of Restrictions Of OM-123 For Five Persons On An Operating

Shift - CR-PLP-2011-04835; Plant Documentation Has Not Recognized That Breaker 72-01 Will Also

Actuate Upon A Fault Current - CR-PLP-2011-04838; Damaged Positive And Negative Bus Bars Removed For Panel ED-11-2

Are Being Staged In Electrical Maintenance Non-Conforming Parts Holding Area Pending Disposition

9 Attachment 1

- CR-PLP-2011-04839; Breaker 72-119 Removed From ED-11-2 Is Being Staged In Electrical Maintenance Non-Conforming Parts Holding Area Pending Disposition

- CR-PLP-2011-04840; Breaker 72-121 Removed From ED-11-2 Is Being Staged In Electrical Maintenance Non-Conforming Parts Holding Area Pending Disposition

- CR-PLP-2011-04848; During Containment Walkdown, A Small Puddle Of What Appears To Be Oil Found On The Ground Near Primary Coolant Pump P-50C

- CR-PLP-2011-04849; During Containment Walkdown, A Small Amount Of Boric Acid Identified On The Seal Area Of Primary Coolant Pump P-50D

- CR-PLP-2011-04850; RV-2082 Is Leaking By Approximately 2.5 Gpm To T-74 - CR-PLP-2011-04852; The Control Room Unexpectedly Received Alarm EK-0702, Relief Valve

2006 Disch Hi Temp - CR-PLP-2011-04853; MO-1043A, Pressurizer PORV Isolation Valve Did Not Open Normally

During Procedure For SOP-1B - CR-PLP-2011-04858; Following Unplanned Reactor Trip, The Pressurizer Level Exceeded

The TS Limit Of 62.8% - CR-PLP-2011-04859; Containment 590’ Elevation Floor Had Standing Water During Post-Trip

Engineering Walkdown - CR-PLP-2011-04864; Control Room Switchyard Supervisory Panel (EC-07) Has No Display

Of Switchyard Breaker Status - CR-PLP-2011-04865; The “B” Steam Generator Level Rose Sharply From 68% To 75%, Then

Slowly Returned To 68% Over A Period Of Approximately 8 Minutes - CR-PLP-2011-04866; In Between Plant Trips, CRD-4 Experienced A Step Increase In

Temperature After Plant Startup From The Trip On September 16, 2011 - CR-PLP-2011-04872; PCS Loop To Pressurizer Delta T Exceeded 200 Degrees F During The

Plant Cooldown - CR-PLP-2011-04879; Main Steam Isolation Valves Closed At Full Load Following Plant Trip

Of September 25, 2011 - CR-PLP-2011-04881; Radionuclides Missing From Containment Purge Batch Card - CR-PLP-2011-04882; DC Bus Ground Fault Current Meters 64-D1 And 64-D2 Are Incorrectly

Set At Plus And Minus 5 Madc - CR-PLP-2011-04890; NRC Resident Identified That During Cooldown To Mode 5, The Control

Room Supervisor Authorized A Control Band For Pressurizer Level Outside Of The Recommended Band In SOP-1B

- CR-PLP-2011-04897; ED-15, #1 Station Battery Charger, Was Declared Inoperable And Removed From Service Due To Lack Of Confidence In Its Ability To Control DC Bus/Battery Voltage Stable

- CR-PLP-2011-04908; SRO And NCO Turnover Items Check Sheets Not Completed - CR-PLP-2011-04920; While Performing WO 00291319-02, Battery Charger No. 1 ED-15

Found Fuse F302 Open - CR-PLP-2011-04939; Review Of Post Trip Data From 9/25 Plant Trip Indicates That Some Of

The Steam Generator Code Safety Relief Valves May Have Opened - CR-PLP-2011-04946; While Performing WO #291601-01, The Left Phase Magnetic Trip Dial

On Breaker 72-01 Broke While Changing Setting From Low To High During Testing - CR-PLP-2011-04958; While Performing WO 291603-01, Breaker 72-02 Would Trip Free When

Trying To Close - CR-PLP-2011-04965; During Performance Of Operability Evaluation Regarding DC Shunt

Breakers 72-01 And 72-02, It Was Identified That Appendix R Protective Device Coordination Is Not Maintained For Some Short Circuit Scenarios

- CR-PLP-2011-04968; As Part Of The Comp Measures For The Op Eval, 72-17 PA System/Motor Generator Has Been Tagged Out

10 Attachment 1

- CR-PLP-2011-04970; The Post Event Review Requirements Presented To The OSRC On 9/30/11 Was Of A Quality Not Acceptable For Review

- CR-PLP-2011-04974; IEEE 450-1995 Cell Parameter Reading And Battery Inspections Were Not Performed On Station Batter ED-01 Following Fault On DC Panel ED-11-2

- CR-PLP-2011-04981; Failure To Use Pre-Job Brief In Accordance With EN-HU-102 - CR-PLP-2011-04988; The 50.59 Screening Performed To Address A

Degraded/Non-Conforming Condition Under EC 32028 Did Not Adequately Document The Extent To Which The Temporary Modification Was Required To Be Reviewed

- CR-PLP-2011-05016; 2011 Plant Trip Special Inspection RFI 61 Response Identified The Condition That Coordination For 72-01 And The Battery Charger As A Load Does Not Exist

- CR-PLP-2011-05085; The Station Has Exhibited A Weakness With Respect To Risk Recognition/Removal/Mitigation

- CR-PLP-2011-05095; Per NE-FAP-OM-006, Maintenance Superintendent Exceeded Administrative Limits Due To Back To Back Forced Outages

- CR-PLP-2011-05116; Duty Station Manager Exceeded Non-Covered Administrative Limits September 23-25, 2011

- CR-PLP-2011-05132; Unable To Find A Condition Report For A Material Deficiency (NRC Identified) That CRD-01 Matrix Light SPI Was Delayed

- CR-PLP-2011-05154; During Plant Trip Of 9/25/11, Main Generator Breakers 25H9 And 25F7 Did Not Immediately Open

- CR-PLP-2011-05171; OCC Day Shift Outage Director During The D-11-2 Trip Recovery Effort, Exceeded Non-Covered Worker Administration Limits

- CR-PLP-2011-05263; On September 26, 2011, A Late 8-Hour Non-Emergency Event Notification #43722, Was Made To The NRC

11 Attachment 1

LIST OF ACRONYMS USED AC Alternating Current ADAMS Agencywide Document Access Management System AFAS Auxiliary Feedwater Actuation System AFW Auxiliary Feedwater AO Auxiliary Operator ASDV Atmospheric Steam Dump Valve AV Apparent Violation BI Barrier Integrity CAP Corrective Action Program CCDP Conditional Core Damage Probability CFR Code of Federal Regulations CHR Containment High Radiation CIS Containment Isolation Signal CIV Containment Isolation Valve CLB Current Licensing Basis CRS Control Room Supervisor CSR Cable Spreading Room DC Direct Current DSM Duty Station Manager EAL Emergency Action Level ECCS Emergency Core Cooling Systems EN Event Notification EOP Emergency Operating Procedure °F Degrees Fahrenheit FIN Finding FLS Front Line Supervisor FME Foreign Material Exclusion FW Feedwater gpm Gallons Per Minute GOP General Operating Procedure HEP Human Error Probabilities HPSI High Pressure Safety Injection HVAC Heating, Ventilation, and Air Conditioning IE Initiating Event IEEE Institute of Electrical and Electronics Engineers IMC Inspection Manual Chapter INL Idaho National Laboratories IP Inspection Procedure IPTE Infrequently Performed Tests And Evolution IR Inspection Report LCO Limiting Condition for Operation LOCHS Loss of Condenser Heat Sink MD Management Directive MS Mitigating Systems MSIS Main Steam Isolation Signal MSIV Main Steam Isolation Valve NCV Non-Cited Violation NEI Nuclear Energy Institute NRC U.S. Nuclear Regulatory Commission

12 Attachment 1

ONP Off-Normal Procedure PA Public Address PARS Publicly Available Records System PCP Primary Coolant Pump PCS Primary Coolant System PPC Plant Process Computer PSF Performance Shaping Factors psia Per Square Inch Absolute RFO Refueling Outage RG Regulatory Guide RIS Regulatory Issue Summary RO Reactor Operator ROP Reactor Oversight Process RPS Reactor Protective System SCWE Safety-Conscious Work Environment SDP Significance Determination Process SG Steam Generator SI Safety Injection SI Structural Integrity SIAS Safety Injection Actuation Signal SL Severity Level SM Shift Manager SPAR Simplified Plant Analysis Risk SRA Senior Reactor Analyst SSC Structure, System, and Component SW Service Water TBV Turbine Bypass Valve TDAFW Turbine-Driven Auxiliary Feedwater TG Turbine Generator TM Temporary Modification TS Technical Specification UFSAR Updated Final Safety Analysis Review URI Unresolved Item WO Work Order WR Work Request

1 Attachment 2

PALISADES SPECIAL INSPECTION CHARTER

September 27, 2011 MEMORANDUM TO: Robert Krsek, Senior Resident Inspector

Kewaunee Power Plant; Branch 5 Division of Reactor Projects

FROM: Steven West, Director /RA/

Division of Reactor Projects SUBJECT: SPECIAL INSPECTION CHARTER FOR PALISADES NUCLEAR

PLANT REACTOR TRIP DUE TO LOSS OF LEFT TRAIN DC BUSSES D-10L AND D-10R ON SEPTEMBER 25, 2011

Palisades experienced a complicated Reactor Trip and Safety Injection at 1506 on September 25, 2011, due to a loss of the left train DC busses D-10L and D-10R that caused the loss of preferred AC busses Y-10 and Y-30. The failure of the left train DC busses is the subject of a Special Inspection that you have been identified to lead. A short discussion of the event follows. On September 25, 2011, electricians were working on the left train DC bus when a bus bar slipped causing an arc and a loss of the left train DC busses. This resulted in the loss of two preferred AC busses, causing a reactor trip, a safety injection signal, auxiliary feedwater actuation signal, containment high radiation isolation signal, and main steam isolation signal. The licensee entered several Limiting Conditions for Operation during this transient. During the transient, licensed operators lost various indications in the control room, as expected with this type of transient. The licensee’s initial assessment is that all systems responded as expected. Electric power has been restored to most of the affected equipment. The plant is stable and in Mode 5 per required Technical Specifications. Shutdown cooling has been placed in service. Pressurizer level trended high during the event due to loss of letdown that resulted from the containment isolation signal. Based upon our initial review, in addition to the human performance error by licensee’s maintenance personnel, we are concerned that several pieces of equipment did not operate as required during the transient, namely, #1 inverter DC input breaker tripped free, #1 battery charger failed, and possible failure of the fuse(s) downstream of the battery shunt breaker during the DC bus short circuit. CONTACT: John Giessner

630-829-9619

2 Attachment 2

R. Krsek -2- The sequence of events and the root and contributing causes for this issue are being investigated by the licensee.

Based on the deterministic and risk-based criteria in Management Directive 8.3, a Special Inspection at Palisades will commence September 27, 2011. The Special Inspection Team, which is being led by you, will include April Scarbeary and Alan Dahbur. Other members may be assigned if specific needs are identified.

The special inspection will determine the sequence of events, and will evaluate the facts, circumstances, and the licensee’s actions surrounding this issue. The Special Inspection Charter for you and your team is enclosed.

Enclosure: As Stated

Distribution w/encl: G. Shear S. Reynolds K. O’Brien D. Roberts, RI J. Clifford, RI P. Wilson, RI S. Weerakkody, RI R. Croteau, RII W. Jones, RII J. Munday, RII H. Christensen, RII K. Kennedy, RIV T. Pruett, RIV A. Vegel, RIV J. Lara, RIII V. Mitlying P. Chandrathil N. Valos D. Passehl D. Merzke RidsNrrPMPalisadesResource NRR Reactive Inspection@nrc.gov

3 Attachment 2

PALISADES SPECIAL INSPECTION CHARTER

This Special Inspection Team is chartered to assess the circumstances surrounding the Reactor Trip and failure of left train DC bus on September 25, 2011. The Special Inspection will be conducted in accordance with Inspection Procedure 93812, “Special Inspection.” The special inspection will include, but not be limited to, the items listed below. This charter may be revised based on the results and findings of the inspection.

1. Establish a historical sequence of events related to Reactor Trip on September 25, 2011, including the maintenance activities which led to the planned work on September 23, 2011.

2. Review licensee’s reportability requirements to confirm necessary notifications were made per 10CFR50.72 and 10CFR50.73 and possible Emergency Action Levels.

3. Review the activities and human performance related to the maintenance of the DC bus to ensure all required plant procedures and work instructions were followed.

4. Evaluate operator response to the transient that occurred on September 25, 2011, as it related to the implementation of licensee’s procedures and processes for evaluating/assessing operator performance.

5. Review any corrective actions taken including operability evaluations and direction provided by the licensee in response to the transient and equipment failures.

6. Review the electrical tripping scheme for the DC bus and evaluate the transient to determine if the equipment operated per design.

7. Evaluate the extent of the electrical transient on the safety-related bus to determine additional equipment impacts.

8. Determine the impact of the loss of left train DC busses on capability/functionality of the AFW system and atmospheric dump valves.

9. Review the licensee’s root cause evaluation plan and schedule. Include the schedule for performing testing of any components that failed during the transient. Evaluate whether the root cause evaluation plan is of sufficient depth and breadth. Confirm that the time allowed to perform the root cause evaluation is commensurate with the safety significance of this issue. Communicate to the licensee that the NRC will inspect the completed root cause evaluation and the associated corrective actions as part of our normal inspection activities.

Additional Inspection Requirements

1. Determine if there are any lessons learned from this Special Inspection.

4 Attachment 2

Charter Approval /RA/ 9/27/11 J. Giessner, Chief, Branch 4, Division of Reactor Projects _____/RA/ 9/27/11___ S. West, Director, Division of Reactor Projects

1 Attachment 3

PALISADES EVENT TIMELINE

HISTORICAL SEQUENCE of EVENTS

The timeline developed was created independently by the inspectors, with best estimates based on all available information. Items that are approximate times are preceded with “~” prior to the listed time. During the development of this timeline all times were referenced back to the control room clock, which was the official time and differed from the plant process computer and sequence event recorder times. The term “days” refers to activities that were conducted on the dayshift. The times listed below are based on the 24-hour clock.

RFO21 During Refueling Outage 21, 10 breakers were replaced by maintenance personnel inside electrical Panel D11-2, associated with the left train 125-Volt DC system (reference Figure 2, Attachment 4).

October 2010

Days Maintenance personnel began work on work order (WO) 248834-01 to troubleshoot the inoperative green indication lights for Door MZ-50, the emergency air lock. All interlocks, indication lights, and limit switches were found to be satisfactory; more troubleshooting was planned for this door indication light issue.

Thursday, September 22, 2011

Days Maintenance personnel completed WO 291123-01 to troubleshoot Breaker 72-123 in Panel D11-2. Maintenance personnel identified that there was no load side voltage phase to phase (this feeds power to the Door MZ-50 indicating lights).

Friday, September 23, 2011

15:26 Maintenance personnel completed WO 291123-03 to successfully replace Breaker 72-123. Restoration activities included re-installing cover panels inside Panel D11-2.

16:07 Control room alarms were received by reactor operators (ROs) for the “Generator Field Forcing/Over-Excitation” cycling; and for red indication lights flickering for the “Voltage Regulator Control Switch” and “Turbine Generator Exciter Field Breaker Control” alarms. Breaker 72-121, Main Generator Voltage Regulator Control Power, experienced an intermittent connection during these restoration activities of Panel D11-2.

16:17 The ROs experienced a loss of indication for multiple containment isolation valves (CIVs) due to an intermittent loss of power from Breaker 72-119. The ROs entered Technical Specification Action Conditions (TSAC) for Limiting Condition for Operation (LCO) 3.3.7 (30-day TSAC for CIV indication) and LCO 3.6.3 for all valves (4-hour TSAC to administratively lock the valves closed).

16:35 The ROs entered Off-Normal Procedure (ONP) ONP-7.1, “Loss of Instrument Air.” The DC power for a junction box common to all three instrument air compressors was a load associated with Breaker 72-119. The intermittent loss of power affected the instrument air compressors standby start feature (the instrument air compressors internal “sleep mode” feature remained available to automatically start the air

2 Attachment 3

compressors). The feedwater (FW) purity air compressor continued to supply the necessary air to equipment through a control valve that failed open upon the intermittent loss of power and cross-connected the two systems, as designed. The running instrument air Compressor C-2A was automatically placed in sleep mode while higher pressure air was supplied by feedwater purity air Compressor C-903B.

~21:30 Maintenance personnel commenced a new troubleshooting plan and identified: no voltage on the load side of Breaker 72-119; misalignments on Breakers 72-119, 72-120, 72-121, and 72-123; and, a 1/16-inch air gap between the horizontal positive bus bar and the line side positive connection on Breaker 72-119. Maintenance personnel also discovered that: the positive feed wire to DC Panel D11-2, was 2°degrees Fahrenheit (°F) hotter than the negative wire; the bus had a slight ground; and, each breaker’s positive horizontal bus bars were hotter than the negative horizontal bus bars.

22:23 The ROs exited ONP-7.1 when instrument air Compressors C-2A, C-2B and C-2C were identified as available for manual start.

Days Licensee personnel continued with troubleshooting activities, challenge boards, work package reviews, and Temporary Modification (TM) EC 31973 development for Breaker 72-121, due to Friday’s events.

Saturday, September 24, 2011

05:00 Nightshift maintenance personnel held a pre-job brief for TM EC 31973 to discuss implementation of WO 291209-01 to implement the TM. The electrical superintendent made the decision not to have the nightshift electricians begin work.

Sunday, September 25, 2011

~07:00 Turnover between electrical superintendent and mechanical superintendent (acting maintenance manager) discussed the upcoming evolution to commence work on WO 291209-01 to implement the TM and for work on Breakers 72-119, 72-120, 72-121, and 72-123. The turnover highlighted the steps of insulating the bus tie stabs and conducting the evolution in the prescribed sequence for breaker removal to keep positive control over the bus tie stabs.

~08:00 Turnover between electrical superintendent and electrical front line supervisor (FLS) discussed the upcoming evolution to commence work on WO 291209-01 to implement the TM and for work on Breakers 72-119, 72-120, 72-121, and 72-123. The turnover highlighted the steps of insulating the bus tie stabs and conducting the evolution in the prescribed sequence for breaker removal to keep positive control over the horizontal bus tie stabs.

~08:30 A pre-job brief for performing work on WO 291209-01 was held.

11:03 Dayshift maintenance personnel installed TM EC 31973 to power breaker loads from Breaker 72-121, Main Generator Voltage Regulator Control Power, from the spare Breaker 72-127.

~12:45 Dayshift maintenance personnel performed an informal pre-job brief for implementing WO 291194-01, WO 291210-01, and WO 291123-03 for work on Breakers 72-119,

3 Attachment 3

72-120, 72-121, and 72-123. The workers and management observers then proceeded to Panel D11-2 for fieldwork.

14:14 The Duty Station Manager (DSM) updated plant management on the breaker work via an email that stated: “Breaker 72-119 (top breaker in panel) was removed, Breaker 72-120 (spare breaker) removed, and an approximately 1/16-inch gap was found between the copper bus bar and breaker stab was identified as well as minor indications of arcing in this area, and the bus bar hole showed evidence of cross-threading.”

15:03 Palisades Plant Status

• Reactor power was approximately 98.5 percent;

:

• Steam Generator (SG) ‘A’ Level was 65.15 percent; • SG ‘A’ Pressure was 970.26 pounds per square inch absolute (psia); • SG ‘B’ Level was 63.96 percent; • SG ‘B’ Pressure was 983.44 psia; • Pressurizer level was 57.86 percent; • Pressurizer pressure was 2063.35 psia; • Primary coolant system (PCS) average temperature was 559.84°F; and, • Letdown flow from the PCS was 43.45 gpm.

15:06 Reactor and Turbine Trip occurred.

15:06 Electrical fault on Panel D11-2 caused the shunt trip Breaker 72-01 to open (reference Figure 2 of Attachment 4).

During the work inside 125-Volt DC Panel D11-2, while removing a section of bus bar, the bar rotated and contact was established between the positive and negative horizontal bus bars, which caused an electrical fault.

15:06 Opening of the shunt trip Breaker 72-01 de-energized the left train 125-Volt DC, D-10L, and D-10R.

15:06 Loss of D-10L and D-10R de-energized 120-Volt preferred alternating current (AC) busses Y-10 and Y-30.

15:06 Inverter input Breaker 72-37 tripped.

15:06 The loss of two out of the four preferred AC busses caused a loss of power to two reactor protection system (RPS) channels (RPS is a two-out-of-four logic).

15:06 The RPS trip signal caused RPS Breakers 3 and 4 to actuate resulting in a reactor trip (a two-out-of-four RPS logic).

15:06 Reactor trip initiated a turbine trip.

15:06 The ROs entered EOP-1.0, “Standard Post-Trip Actions.”

15:06 All controls rods verified inserted into the core by ROs (only the control room supervisor plant process computer lost power, all other RO stations were available, in addition, left train indications to PPC were lost due to the loss of the left train 125-Volt DC).

4 Attachment 3

15:06 Main Steam Isolation Signal (MSIS) initiated the right channel based on a two-out-of-four logic made-up for the loss of 120-Volt preferred AC busses Y-10 and Y-30 (low SG pressure sensed).

15:06 The right channel MSIS signal initiated closure of the right train Main Steam Isolation Valve (MSIV). The left train MSIV closed due to the closure of the first MSIV.

15:06 Safety Injection Actuation Signal (SIAS) occurred based on a two-out-of-four logic made-up for the loss of 120-Volt preferred AC busses Y-10 and Y-30 (sensed low pressurizer pressure). Right channel initiated and started the following pumps: High pressure safety injection (HPSI) ‘A’; low pressure safety injection (LPSI) ‘A’; Auxiliary Feedwater (AFW) Pump 8C; and, charging Pumps ‘A’ and ‘B’.

15:06 The LPSI ‘A’ and HPSI ‘A’ pumps do NOT inject due to the PCS pressure being greater than the pumps’ shutoff head.

15:06 Containment High Radiation (CHR) signal received based on a two-out-of-four logic made-up from loss of 120-Volt preferred AC busses Y-10 and Y-30. This initiated the following: both trains of control room heating, ventilation, and air conditioning (HVAC) in emergency mode; only the right train (‘B’) started, since the left train (‘A’) had no power; primary coolant pump (PCP) bleedoff and letdown isolation control Valve CV-2099 closed; and both SG bottom blowdown line control Valves CV-767 and CV-768 closed.

15:06 Containment Isolation Signal (CIS) initiated based on a two-out-of-four logic made-up for loss of 120-Volt preferred AC busses Y-10 and Y-30. This closed all of the right channel containment isolation valves (CIVs), which included the letdown control valves on the pressurizer.

15:06 Containment high pressure alarm occurred, but not an actuation signal. The alarm was seen on the left channel based on a two-out-of-four logic, but since the downstream relays in this logic had no power, a containment high pressure actuation signal was not initiated. The right channel did not receive any signals.

15:06 Turbine Driven Auxiliary Feedwater (TDAFW) Pump P-8B started due to its steam supply control Valve CV-0522B failing open on loss of DC power (powered by Panel D11-1) and the AFW Actuation Signal (AFAS), which overrode the low suction pressure trip signal caused by the loss of the left train 125-Volt DC system.

15:06 The AFAS was received due to a loss of 120-Volt preferred AC busses Y-10 and Y-30 (sensed low SG water level), which made up the two-out-of-four logic. The AFW Pump P-8A did not start due to the loss of power to the control circuits associated with the low suction pressure trip.

15:06 The ROs verified that safety-related AC busses 1D and 1C (safety-related 2400-Volt) were available due to loss of AC Bus 1E (nonsafety 2400-Volt).

15:06 Busses 1A (nonsafety 4160-Volt) and 1F (nonsafety 2400-Volt) did not fast transfer to station transformer (received fast transfer signal with loss of power; however, the loss of the left train 125-Volt DC prevented the fast transfer from occurring).

5 Attachment 3

15:06 The PCPs ‘A’ and ‘C’ started a slow coastdown due to the loss of power from Bus 1A (bus still had some energy due to main generator not being fully disconnected immediately). PCPs ‘B’ and ‘D’ continued to run.

15:06 The FW Purity Air Compressor C-903B was lost due to the loss of Bus 1E (at the time Compressor C-903B was supplying air to the instrument air header, due to the September 23, 2011, event). Instrument air Compressor C-2A was in “sleep mode” and auto started upon a lowering instrument air header pressure.

Inoperable Technical Specification (TS) Related Equipment and TSACs entered by ROs

• Preferred AC Bus No. 1, Y-10, TSAC 3.8.9 (B);

:

• Preferred AC Bus No. 3, Y-30, TSAC 3.8.9 (B); • Inverter No. 3, D-08, TSAC 3.8.7 (A.1); • Inverter No. 1, D-06, TSAC 3.8.7 (A.1); • The TS 3.0.3 was entered due to two preferred AC busses INOPERABLE and

two inverters INOPERABLE. The ROs exited this at 19:12; • Left train 125-Volt DC busses D-10L and D-10R, TSAC 3.8.9 (C); • Four atmospheric steam dump valves (ASDVs) lost power due to the master

controller being powered from Bus Y-10 and lost the quick-open capability (relay lost power with loss of Y-10), TSAC 3.7.4 (A.1 and A.2); and,

• The PCS unidentified leakage TSAC 3.4.13 ( unidentified leakage was >1 gpm for PCP-controlled bleedoff being isolated).

15:07 The AFW Pump P-8C started due to AFAS (one minute later due to time delay built in to logic).

~15:16 The ROs manually switched (per their ONP) pressurizer pressure and level indication instruments over to Channel ‘A’ due to the loss of indication from the loss of power on Channel ‘B’ and actual increased level and pressure seen in the pressurizer. With the failure of the controller, the pressurizer control systems were at maximum charging, no letdown (letdown orifice valves were isolated), and no pressurizer spray. Charging Pumps ‘A’ and ‘B’ were running because the pumps started on the SIAS.

~15:16 Pressurizer spray was able to operate with swapping of controllers. Primary system pressure is stable at ~2063 psia.

15:17 The turbine-side RO in control room manually jumpered main generator output breakers to the “open” position (Breakers 25F7 and 25H9).

~15:20 Main feedwater Pumps ‘A’ and ‘B’ were tripped by the ROs and their respective turbines were tripped. Condensate Pump ‘A’ was tripped by the ROs. Condensate Pump ‘B’ was functioning.

15:27 The ROs Entered EOP-9.0, “Functional Recovery,” due to the loss of two preferred AC busses upon completion of EOP-1.0.

6 Attachment 3

15:27 Plant Status

• Reactor power was 0 percent;

:

• SG ‘A’ Level was 65.15 percent; • SG ‘A’ Pressure was 925.42 psia; • SG ‘B’ Level was 55.56 percent; • SG ‘B’ Pressure was 969.67 psia; • Pressurizer level was approximately 66.3 percent; • Pressurizer pressure was approximately 2140 psia; • The PCS average temperature was 536.27°F; and • Charging flow to the pressurizer was 133 gpm (approximate indication).

~15:30 The ROs entered ONP-2.3, “Loss of DC.”

15:31 An AO was dispatched to the field to respond to a fire alarm in the AFW pump room. The AO was also directed to manually close CV-0522B (AFW ‘B’ steam supply control valve) for isolation of TDAFW Pump P-8B. Level in SG ‘A’ was approximately 67 percent and level in SG ‘B’ was approximately 58.6 percent. This rendered the left train of AFW INOPERABLE and the ROs entered TSAC 3.7.5.

15:37 Pressurizer pressure increased to a maximum of 2206 psia (indicated on PTR-0122). This was below the first pressurizer code safety valve setting of 2500 psia (The pressurizer power operated relief valves were isolated at Palisades during normal operations).

15:37 The ROs entered ONP-24.1, “Loss of Preferred AC Bus No. 1 (Y-10).”

The ROs entered ONP-24.3, “Loss of Preferred AC Bus No. 3 (Y-30).”

15:42 Letdown heat exchanger inlet safety relief Valve RV-2006, was isolated after not re-seating correctly during the event.

15:49 Bus 1E (nonsafety 2400-Volt AC that was lost during event) was restored by maintenance and operations personnel in field (load shed on SIAS).

~15:51 Main Steam Safety Valve(s) maintained secondary side pressures, which subsequently maintained PCS temperature, from the start of the event.

15:53 The plant process computer (PPC) for control room supervisor was restored.

15:55 Pressurizer level reached greater than 62.8 percent, which was the TS limit. The ROs entered TSAC 3.4.9(A.1) and (A.2) to reduce levels to less than the limit. Pressurizer level was approximately 81 percent at this time.

15:57 In EOP-9.0, Attachment 5, “Safety Injection Throttling Criteria,” was met so the ROs throttled reduced flow on the charging pumps in an attempt to lower the PCS level in the pressurizer; however, the letdown system was still isolated.

7 Attachment 3

15:57 Plant Status

• SG ‘A’ Level was 97.02 percent;

:

• SG ‘A’ Pressure was 853.33 psia; • SG ‘B’ Level was 63.96 percent; • SG ‘B’ Pressure was 965.86 psia; • Pressurizer level was 81.17 percent; • Pressurizer pressure was 2046.04 psia; and, • The PCS average temperature was 532.77°F.

15:57 120-Volt Preferred AC Bus No. 3 (Y-30) was OPERABLE on the bypass regulator. Bus 1E (nonsafety 2400 Volt AC) was lost with these actions.

15:57 Busses D-10L and D-10R, 125-Volt DC Left Train, were OPERABLE due to Y-30 being restored and the shunt trip Breaker 72-01 re-closed. Upon restoration generator field Breaker 341 automatically opened and instrument air Compressor C-2A tripped for an unknown reason.

15:57 The SG ‘A’ reaches a maximum level of ~97.02 percent (per PPC).

16:02 Charging Pump ‘B’ (P-55B) suction relief Valve RV-2096 lifted and did not properly re-seat. This caused volume control tank water to fill up the equipment drain tank and spill-over onto the floor in pump Cubicle ‘B’ (backed-up floor drain).

~16:02 Main steam safety valve(s) continue to lift to maintain secondary side pressures, which subsequently maintained PCS temperature.

16:15 Pressurizer level reached a maximum of approximately 98 percent.

16:21 Procedure ONP-7.1, “Loss of Instrument Air,” entered since instrument air compressor C-2A tripped at 15:57 upon restoration of the 125-Volt DC left train. Instrument air Compressors C-2B and C-2C were placed in service by the AOs.

16:30 Operators in the field manually isolated charging Pump P-55B, by closing the discharge and suction isolation valves. This was necessary due an abundance of water in the cubicle from the improperly seated relief Valve RV-2096.

16:34 The HPSI and LPSI Pumps ‘A’ were secured due to SIAS throttling criteria being met (were never injecting but started on SIAS signal).

16:44 The SG ‘B’ level reached a maximum of approximately 69.06 percent.

16:44 Plant Status

• SG ‘A’ level was 90.45 percent;

:

• SG ‘A’ pressure was 932.45 psia; • SG ‘B’ level was 69.06 percent; • SG ‘B’ pressure was 930.70 psia; • Pressurizer level was 91.94 percent; • Pressurizer pressure was 1864.13 psia; and, • The PCS average temperature was 539.48°F.

8 Attachment 3

16:46 120-Volt preferred AC Bus No. 1 (Y-10) was OPERABLE on bypass regulator. 120-Volt Preferred AC Bus No. 3 (Y-30) was taken off of the bypass regulator and powered from the inverter.

16:46 All four ASDVs were OPERABLE with the return of 120-Volt Preferred AC power source No. 1, Y-10 (power restored to controller).

17:20 Procedure ONP-4.1, “Spurious Containment Isolation,” was entered due to loss of preferred AC busses Y-10 and Y-30 causing a CIS.

17:46 The ROs exited EOP-9.0 with restoration of the preferred AC busses and entered GOP-8, “Power Reduction and Plant Shutdown to Mode 2 or Mode 3 ≤525oF.”

18:00 Once the ROs exited EOP-9.0, the criteria was met to reset the SIAS.

18:00 Cooling was restored to Spent Fuel Pool (SFP) Heat Exchanger (lost during loss of power). The temperature in the SFP at 15:00 was 83.4°F and the temperature of the pool at the time of restoration of the heat exchanger was 87.4°F.

18:52 The AFW Pump P-8B was declared OPERABLE when steam supply control Valve CV-0522B was re-opened and controller placed in AUTO.

19:09 The ROs exited ONP-24.1, “Loss of Preferred AC Bus No. 1,” with restoration of Y-10 and associated loads.

19:11 The ROs exited ONP-24.3, “Loss of Preferred AC Bus No. 3,” with restoration of Y-30 and associated loads.

19:12 The ROs declared Inverter No. 3, D-08, OPERABLE which enabled the exit of TSAC 3.0.3 with busses Y-10, Y-30, and Inverter D-08 restored.

19:12 The ROs exited ONP-7.1, “Loss of Instrument Air,” when power was returned to the right channel controller.

19:23 Battery Charger No. 1 D-15 was still INOPERABLE and TSAC 3.8.4(A.2) was entered by the ROs.

19:23 Main Station Battery left Channel D-01 was still INOPERABLE and TSAC 3.8.4(B.1) and 3.8.6(A.1 and A.2) were entered by the ROs due to not being connected to a charger.

19:33 The ROs connected battery Charger No. 3, D-17, to the 125-Volt DC bus to charge main station battery left channel D-01.

20:16 Main station battery left channel D-01 met the TSAC requirement 3.8.6 (A.1) and its terminal voltage was greater than 125-Volt; however the ROs were still in TSAC 3.8.6(A.2).

23:48 The ROs restored pressurizer level to less than 62.8 percent (TS limit) which enabled the exiting of TSAC 3.4.9 (A.1 and A.2).

9 Attachment 3

01:23 WO 291210-03 started to remove Breaker 72-122 to use those bus tie stabs to replace the ones on Breaker 72-119 that were damaged during the event.

Monday, September 26, 2011

01:56 Charging Pump P-55B was declared OPERABLE by the ROs after leaking suction relief Valve RV-2096 was verified to function and water was cleaned up in cubicle.

03:00 Breakers 72-119 and 72-120 were installed and restored.

04:41 Main station battery left channel D-01 was declared OPERABLE by the ROs and TSACs 3.8.4 and 3.8.6 were exited.

06:40 Power was restored back to Breaker 72-119 loads and thermography was completed satisfactorily on all of the restored breakers, with no anomalies identified.

11:58 Charging Pump P-55B was started to initiate double charging and letdown to aid in PCS cooldown and transition to Mode 4 (Hot Shutdown).

16:09 The ROs exited ONP-4.1, “Spurious Containment Isolation.”

16:30 The ROs commenced a PCS cooldown with turbine bypass valve.

23:06 Reactor entered Mode 4.

04:30 Shutdown cooling was placed in-service per GOP-9 and GOP-14 when PCS pressure was less than 265 psia and PCS temperature was less than 300°F.

Tuesday, September 27, 2011

06:33 Reactor entered Mode 5 (Cold Shutdown).

20:05 Revision 1 of the operability evaluation for the 125-Volt DC system was accepted by operations.

Friday, September 30, 2011

21:31 Reactor entered Mode 4.

02:48 Reactor entered Mode 3.

Saturday, October 1, 2011

23:30 Reactor entered Mode 2.

01:35 Initial criticality achieved with Group 3 rods at 99.3 inches.

Sunday, October 2, 2011

02:26 Achieved the Point of Adding Heat.

03:27 The MSIVs were opened with no issues on operation of valves.

07:37 AFW Pump P-8C was secured with no issues.

10 Attachment 3

08:24 Reactor entered Mode 1.

10:20 Generator output breakers closed.

11:50 Reactor power was at 100 percent.

Monday, October 3, 2011

1 Attachment 4

SIMPLIFIED DIAGRAMS OF PALISADES 125-VOLT DC SYSTEM

FIGURE 1 - SIMPLIFIED DIAGRAM OF PALISADES 125-VOLT DC SYSTEM

Left Train Right Train

2 Attachment 4

FIGURE 2 - SIMPLIFIED DIAGRAM OF LEFT TRAIN 125-VOLT DC BUS DE-ENERGIZED DURING EVENT

1 Attachment 5

IMAGES OF PALISADES 125-VOLT DC SYSTEM DURING/FOLLOWING MAINTENANCE

FIGURE 3 - DC DISTRIBUTION PANEL D11-2 WITH BREAKER 72-120 REMOVED

2 Attachment 5

FIGURE 4 - DC DISTRIBUTION PANEL D11-2 WITH BREAKERS 72-119 AND 72-120 REMOVED

(Note that with both breakers removed the positive horizontal bus bar (top left copper stab) and the negative horizontal bus bar (right copper stab) were not insulated.)

3 Attachment 5

FIGURE 5 - DC DISTRIBUTION PANEL D11-2 WITH BREAKERS 72-119, 72-120 AND 72-121 REMOVED

(Note that with three breakers removed the positive horizontal bus bar (top left copper stab) and the negative horizontal bus bar (upper right copper stab) were not insulated.)

4 Attachment 5

FIGURE 6

DC DISTRIBUTION PANEL D11-2 WITH BREAKERS 72-119, 72-120, 72-121 AND 72-123 REMOVED

(Note that with all 4 breakers removed the positive horizontal bus bar (top left copper stab) and the negative horizontal bus bar (right copper stab) were not insulated.)

5 Attachment 5

FIGURE 7 - DC DISTRIBUTION PANEL D11-2 IMMEDIATELY FOLLOWING THE ELECTRICAL FAULT

(Note that when the positive bus bar, top left copper stab, was loosened and the electrician let go of it, the bus bar rotated down contacting the negative bus bar, which initiated the electrical fault on the left train 125-Volt DC system.)

6 Attachment 5

FIGURE 8 - DC DISTRIBUTION PANEL D11-2 INSPECTION AND REPAIR SEVERAL DAYS AFTER THE EVENT

1 Attachment 6

PERMISSION TO UTILIZE GRAPHICS/VISUAL AIDS

1 Attachment 7

LIST OF MAJOR AFFECTED EQUIPMENT

REVIEW OF DE-ENERGIZED EQUIPMENT ON 9/25/11 PLANT TRIP

This is a preliminary review of equipment response to the plant trip on 9/25/11. The initial set of components lost on 9/25 is D11-1, D11-2, D-10R, D-10L. The loss of D-10R and D-10L led to the loss of Y-10 and Y-30, which led to the plant trip. This table identifies the loss of major components. Other components may have been lost, but did not have a significant impact on mitigating the event in the short term. The instrument air system was in an abnormal lineup at the time, with Feedwater Purity Air cross-tied to plant air. ‘A’ Channel of Pressurizer Pressure and Level control systems were in-service.

Affected Component Actual Component State Following Transient

Additional Information

Loss of DC Power

SV-0522G & SV-0522H, Air Control to Steam Supply Valve for ‘B’ AFW pump

De-energized Fails open ‘B’ AFW pump steam supply valve CV-0522B

CV-1212, Service Air Header Isolation

Failed closed Loss of Service Air

25F7, Main Generator Output Breaker

Did not auto trip (stayed closed)

Required to relay terminals to be jumpered in control room panel to Open

25H9, Main Generator Output Breaker

Did not auto trip (stayed closed)

Required to relay terminals to be jumpered in control room panel to Open

Main Generator Field Breaker, 341

Did not open (should open on turbine trip)

Locally tripped open

Bus 1A, Non-Safety 4160V

De-energized and did not fast transfer to Station Transformer (from Start-up Transformer)

Lost control power for all breakers and indicating lights

Bus 1F, Non-Safety 4160V

De-energized and did not fast transfer to Station Transformer (from Start-up Transformer)

Lost control power for all breakers and indicating lights

Load Center -11 (480V AC Safety-related)

Lost control power for all breakers (with loss of DC bus)

Local manual control available

Load Center -19 (480V AC Safety-related)

Lost control power for all breakers (with loss of DC bus)

Local manual control available

Load Center -17 (480V AC Safety-related)

Lost control power for all breakers (with loss of DC bus)

Local manual control available

Load Center -77 (480V AC Safety-related)

Lost control power for all breakers (with loss of DC bus)

Local manual control available

2 Attachment 7

Affected Component Actual Component State Following Transient

Additional Information

CV-2009, Letdown Containment Isolation Valve

Failed closed Caused Letdown Heat Exchanger Inlet Relief Valve, RV-2006, to lift

CV-2083, Primary Coolant Pumps (P-50A/B/C/D) Controlled Bleedoff Control Valve

Failed closed Controlled Bleedoff instead went to Primary System Drain Tank via Relief Valve, RV-2082

Instrument Air Compressors, C-2A/B/C

Lost standby start feature, internal “sleep mode” feature still available

Manual Start capability available

CV-1212, Service Air Header Isolation Valve

Failed closed Service Air was not needed during this event

CV-1221, Feedwater Purity Air Cross-Tie to Plant Air Valve

Failed open Feedwater Purity Air System fed air to the Instrument Air System loads

EK-02, Alarms on Control Room panel C-11A (Radiation Control Room HVAC panel)

Lost alarm scheme due to loss of power

EK-21, Left Channel alarms on Safety Injection Signal sequencer display

Lost alarm scheme due to loss of power

EK-24, alarms on Isophase Bus Panel

Lost alarm scheme due to loss of power

EK-33, alarms on Control Room panel C-106 (Cooling Tower Master Supervisory and Control Cabinet)

Lost alarm scheme due to loss of power

EK-35, alarms on Control Room panel C-126 (Circulation Water and Iodine Removal Panel)

Lost alarm scheme due to loss of power

Various Containment Isolation and Radwaste Valves

Failed closed and lost position indication due to loss of DC power

3 Attachment 7

Affected Component Actual Component State Following Transient

Additional Information

Loss of Y-10 and Y-30

Safety Injection Actuation Signal

2 out of 4 channels received actuation signal which meets circuit logic start criteria

Right Channel logic was met – Left Channel lost power to its relays when Y-30 was lost, therefore Charging Pump P-55C was unavailable

Containment High Radiation Signal

2 out of 4 channels received actuation signal which meets circuit logic start criteria

Right Channel logic was met – Left Channel lost power to its relays when Y-10 was lost

Containment High Pressure Alarm

Alarmed in Control Room Left Channel created alarm – no actuation initiated due to relays losing power when Y-10 was lost

Main Steam Isolation Signal

2 out of 4 channels received actuation signal which meets circuit logic start criteria

Right Channel logic was met – Left Channel lost power to its relays when Y-30 was lost

‘A’ Channel of Pressurizer Pressure Control (in Control Room)

Lost power with loss of AC

Pressurizer pressure control systems responded by the Heaters going to maximum capacity and the Spray not actuating

‘A’ Channel of Pressurizer Level Control (in Control Room)

Lost power with loss of AC

Pressurizer level control systems responded by having maximum Charging flow from the available charging pumps and minimum Letdown capability by closing the Letdown Isolation Valves

Auxiliary Feedwater Pumps P-8A/B receive low suction pressure trip

2 out of 3 channels received trip signal which meets circuit logic trip criteria

‘A’ AFW Pump P-8A did not have power, ‘B’ AFW Pump P-8B was running at full capacity due to AFAS that overrode the low suction pressure trip and was manually isolated during the event by operators

HIC-0780A/B & HIC-0781B, Atmospheric Steam Dump Valve Controllers (in Control Room)

Lost power with loss of AC

Could not manually or automatically control ASDVs (valves were not available for use during the event)

1 Attachment 8

PHASE 3 SIGNIFICANCE DETERMINATION PROCESS DETAILED ANALYSIS FOR THE FAILURE TO HAVE ADEQUATE WORK INSTRUCTIONS

The Senior Reactor Analysts (SRAs) used the Palisades SPAR [Simplified Plant Analysis Risk] model, Revision 8.17, for the significance determination process (SDP) Phase 3 evaluation. The model was modified by Idaho National Laboratories (INL) and the SRAs to model: 1) a changed success criteria for feed and bleed scenarios to one-of-two pressurizer power-operated relief valves instead of two-of-two; 2) recovery of the direct current (DC) bus; 3) reactor operator (RO) action for the turbine driven auxiliary feedwater pump (TDAFW) pump; 4) potential for the pressurizer safety relief valve to be challenged and failure to reseat; 5) allow for secondary side cooldown success for loss of condenser heat sink (LOCHS) events with auxiliary feedwater (AFW) success; and, 6) load shed and required recovery of Bus 1E for an LOCHS event.

The SPAR model does not have a loss of DC Bus 10 initiating event. As a substitute, the SRAs modeled the effects of the event using the “Loss of Condenser Heat Sink” initiating event, and set its frequency to 1.0. The SPAR model also does not have individual basic events representing failures of the various DC panels or the preferred alternating current (AC) busses. To model the impact of loss of power to these components, the basic event for the loss of power to DC Bus 11, and the basic event for Battery Charger 1, were failed in the model. The failure of a DC bus in the model essentially fails one train of equipment. Restoration of the DC bus was modeled to recover these functions.

During the event, the loss of the DC power caused the steam admission valve to the TDAFW pump to fail open, resulting in the inability to control the flow of the pump from the control room. ROs manually closed the valve in the plant, causing the TDAFW pump to become unavailable without further manual action. Additionally, if ROs had not closed the steam admission valve, there was a potential to overfill the SG, which would also result in failure of the TDAFW pump.

Also during the event, a Safety Injection (SI) Actuation Signal (SIAS) occurred due to the loss of preferred AC power busses Y10 and Y30. As a result, charging flow was maximized and letdown isolated. The SI pumps did not inject because primary coolant system (PCS) pressure remained above the shut-off head of the pump. However, charging continued, which caused pressurizer level to increase to approximately 98 percent before ROs took control and reduced flow.

The SIAS also caused load shedding of 2400-Volt AC Bus 1E. During the event, this bus was restored but then tripped again during the restoration of the preferred AC busses. This bus is important because it provides power to components necessary for long term makeup to the condensate storage tank for AFW system operation. The RO action to recover Bus 1E is modeled in the SDP.

As a result of the loss of preferred AC busses Y-10 and Y-30, the turbine bypass valve (TBV) and the atmospheric steam dump valves (ASDVs) failed closed. In the Phase 3 evaluation, this was modeled by using the LOCHS initiator and setting the basic event representing RO action to depressurize the steam generators (SGs) to “True.” This represented the failure of the ASDVs and TBV in the short-term to depressurize the SGs and allow for condensate injection if AFW fails.

2 Attachment 8

Human reliability is an important aspect of this Phase 3 SDP. The following basic events were added to the model to assess the risk significance of the performance deficiency. These events were evaluated using the SPAR-H human reliability analysis method. For all of the human error probabilities (HEPs), the performance shaping factors (PSFs) of stress and complexity were assumed to be performance drivers. For several of the HEPs, time, experience, and/or procedures were also determined to be performance drivers.

AFW-XHE-XM-TDPCNTRL

: This basic event represents the required RO action to manually operate the TDAFW pump locally in response to the loss of DC Bus 10 event. During the event, ROs manually closed the steam supply valve to the turbine because SG levels were increasing rapidly. The Off Normal Procedure (ONP) ONP 2.3 “Loss of DC Power,” directed ROs to System Operating Procedure (SOP) SOP-12, “Feedwater System,” Section 7.2.3, which provided instructions to manually close the steam supply valve. However, during the event, ROs did not use SOP-12; the ROs used Emergency Operating Procedure (EOP) Supplement 19, “Alternate Auxiliary Feedwater Methods,” Section 4.0.2, which prescribed the same steps as Section 7.2.3 of SOP-12, for erratic operation of the TDAFW pump following startup. Procedure EOP Supplement 19 also provided instructions for manually operating this valve and other flow control valves locally in the event the steam supply valve failed in the closed position.

The SRAs assumed that success of the TDAFW pump would require initial isolation of the steam supply valve using SOP-12 followed by manual operation of the valve using EOP Supplement 19. The estimated HEP for these combined manual actions is 0.13. The SRAs assumed the performance drivers for diagnosis were stress and complexity. Stress was assumed to be high and complexity moderate. For the action, the SRAs assumed that in addition to stress, complexity was high given the requirement to use multiple procedures locally to manually operate a number of valves. Time was also considered to be a performance driver. The time available to manually operate the TDAFW pump was assumed to be approximately equal to the time required. DCP-XHE-XL-DCBUS11

: This basic event represented recovery of the DC Panels 11-1 and 11-2 by re-closing the Breaker 72-01 to restore power to DC Bus 10. The diagnosis of a loss of 125-Volt DC event is covered in EOP 1.0, “Standard Post-Trip Actions,” and in ONP 2.3 “Loss of DC Power.” The action to restore a DC bus or panel is directed by ONP 2.3. For diagnosis, the SRAs assumed the performance drivers were stress and complexity, which were evaluated as high and moderate, respectively. For the action portion of the HEP, in addition to stress and complexity, the SRAs assumed experience/training and procedures were performance drivers. The estimated HEP is 0.1.

ROs have limited training and no experience with responding to the loss of an entire train of DC power. ROs received training on the off-normal procedures. There are no simulator exercises that model this event. The experience/training PSF was rated as low. The procedure’s PSF was considered to be available, but poor. After exiting EOP 1.0, “Standard Post-Trip Actions,” ROs entered EOP 9.0, “Functional Recovery Procedures.” Since D-21A and D21-1 remained energized, the criteria for 125-Volt DC were met and the focus of the functional recovery became the preferred 120-Volt AC busses. No part

3 Attachment 8

of the EOP network specifically directs the operator to ONP 2.3, “Loss of DC Power,” which is the required procedure to recover the left train 125-Volt DC bus from this event. Procedure ONP 2.3 was entered during the event because the entry criteria were met. The procedure is structured to address recovery of individual sections of the DC power system and does not present an integrated approach to the event that occurred. Separate procedures are also required to restore power to each of the preferred AC busses, which was directed from EOP 9.0, Success Path MVAE-DC-1. PPR-PZR-SOLID

The following human reliability basic events were in the base model. The values were changed to better reflect the risk of this finding.

: This basic event represented ROs failing to control pressurizer level such that the pressurizer safety valves are challenged and open. Stress and complexity were assumed to be the performance drivers and were rated high and moderate, respectively, for both diagnosis and action. For action only, time was also considered to be a performance driver. The time available to perform the action was assumed to be approximately equal to the time required to perform the action. The estimated HEP was 8E-2.

MSS-XHE-XM-DEPRESS

: This basic event represented the ROs failing to depressurize SGs for condensate injection. This event was set to “True” to represent the failure of preferred AC Bus Y-10, which fails the ability of the ROs to use the ASDVs to depressurize the SGs.

PCS-XHE-XM-COOLDOWN

: This basic event represented the ROs failing to initiate a cooldown to allow the use of shutdown cooling in the sequences where the pressurizer safety valves lift and do not reclose and high pressure injection is successful.

Stress and complexity were assumed to be the PSFs that were the performance drivers for both diagnosis and action, and were rated as high and moderately complex, respectively. The estimated HEP was 4.4E-2. ACP-XHE-XL-BUS1E

The dominant core damage sequence is a loss of condenser heat sink followed by ROs failing to control pressurizer level. One or more safety relief valves on the pressurizer are challenged and fail to reclose. High pressure injection fails due to the failure to restore the DC bus combined with random failures of the opposite train of high-pressure injection.

: This basic event represented the failure to restore Bus 1E after it is load shed following an SIAS. The SRAs used the HEP value from the licensee’s model, 2.6E-3.

The second dominant sequence is a loss of condenser heat sink followed by failure of AFW, failure to depressurize the SGs for condensate injection, and failure of once-through cooling. AFW fails due to random failures, failure to control the TDAFW pump, and the failure to restore the DC bus.

Other sequences contributing are similar to the dominant sequence involving one or more open safety relief valves. In these sequences, high-pressure injection is successful but either secondary side cooldown or shutdown cooling fail.

The result of the Phase 3 SDP is a finding of substantial safety significance (Yellow) with an estimated CCDP of 1.6E-5.

A. Vitale -2- In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records System (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely, /RA/ Steven West, Director Division of Reactor Projects

Docket No. 50-255 License No. DPR-20

Enclosure: Inspection Report 05000255/2011014 w/Attachments: 1. Supplemental Information 2. Special Inspection Team Charter 3. Palisades Event Timeline 4. Simplified Diagrams of Palisades 125-Volt DC System 5. Images Of Palisades 125-Volt Dc System During/Following Maintenance 6. Permission to Utilize Graphics/Visual Aids 7. List of Major Affected Equipment 8. Phase 3 Significance Determination Process Detailed Analysis for the Failure

to Have Adequate Work Instructions

cc w/encl: Distribution via ListServ

DISTRIBUTIONSee next page

:

DOCUMENT NAME: G:\DRPIII\1-Secy\1-Work In Progress\Palisades SIT 2011014.docx

Publicly Available Non-Publicly Available Sensitive Non-Sensitive To receive a copy of this document, indicate in the concurrence box "C" = Copy without attach/encl "E" = Copy with attach/encl "N" = No copy

OFFICE RIII RIII RIII RIII RIII NAME RKrsek*Per

Telecon JGiessner:cs/dp LKozak*Per

email SOrth SWest

DATE 11/28/11 11/28/11 11/29/11 11/28 /11 11/29/11

OFFICIAL RECORD COPY

Letter to A. Vitale from S. West dated November 29, 2011. SUBJECT: PALISADES NUCLEAR PLANT - NRC SPECIAL INSPECTION TEAM (SIT)

REPORT 05000255/2011014 PRELIMINARY YELLOW FINDING DISTRIBUTION: Amy Snyder RidsNrrPMPalisades Resource RidsNrrDorlLpl3-1 Resource RidsNrrDirsIrib Resource Cynthia Pederson Jennifer Uhle Steven Orth Jared Heck Allan Barker Carole Ariano Linda Linn DRPIII DRSIII Patricia Buckley Tammy Tomczak ROPreports Resource