schneider electric - nuclear navy and reducing human error

The US Nuclear Navy: Eradicating Human Error

Domenic Alcaro, LT, USN, Ret

Schneider Electric

Schneider Electric 2- Division - Name – Date

Agenda

●Engineering Plant Overview●Picking the Right People●Training the Right People●Procedures-Zero Deviation Allowed●Policies●System Design●War Stories●Cool Pictures●Non-Military Applications


The Nuclear Submarine


Engineering Plant Overview(Each plant below made up of multiple, smaller, complex subsystems)

●Reactor Plant●Steam Plant●Electrical Plant●Hydraulic Plant●Cooling Plants●Fresh Water Make-Up Plants●Multiple Auxiliary Plants (I.e., High Pressure Air, Low Pressure Air, Refuse Removal, etc.)

●Etc…(and this is just the back half of the sub)


Picking the Right People

●The Infamous Admiral Rickover, “Father of the Nuclear Navy”

●Technical Competency Screening●Leadership Screening●Psychological Screening (claustrophobia is rarely an issue)

●Reliability and “Quality of Character” Screening – The PRP, Personnel Reliability Program

●Applied to both Officer and Enlisted Ranks


Training the Right People:Prior to Arriving On Board

●Nuclear Power School

●Nuclear Prototype Training

●General Submarine Training

●If time permits, other Navy Schools


Training the Right People:After Arriving On Board

●Qualification Cards (Qual Cards)●General Submarine Qual Cards●Individual Watch Station Qual Cards

●Under Instruction (UI) Watches (the RIGHT way to do OJT)

●Extensive Testing (Written and Oral)●On Board Continuing Training Program●Off Ship Navy Schools●Drills, Drills, Drills (Drill Coordinator is an actual assigned position)


Procedures

●Only with permission of CO can procedures be violated – this mentality is religion

●All procedures are always conducted with a checklist in front of you, even if you have memorized them


Procedures - Emergency

●Emergency Procedures●To quickly bring plant to a stable condition●Immediate Actions (these actions for EVERY Emergency Procedure must be flawlessly memorized by every watch stander)

●Supplementary Actions


Procedures - Operating

●Operating Procedures●To carry out the day to day functions of the boat●I.e., Reactor Startup, Shifting the Electric Plant, Warming up the Main Engines


Maintenance Procedures

●The PMS (Planned Maintenance System)●Daily, weekly, bi-weekly, monthly, bi-monthly, quarterly, semi-

annually, annually, bi-annually, every 3, 5, 10 years, etc.●Approvals and Notifications●Status Boards●Complete “re-commissioning” required for some equipment after

every maintenance procedure●Audits

●Some General Maintenance Practices●The Tag-Out System●The Valve Positioning Checklist●Electrical Safety Procedures


The Tag-Out System

●Tag-Out Form always filled out first●Filled out and signed by 2 individuals●Presented to Officer-In-Charge with schematic of system affected for 3rd individual verification

●Yellow and Red Tags●“Hang Tags, Position Valves”●Some rules apply, i.e., “high energy” systems require 2 valve isolation and an electrical system must be completely deenergized whenever feasible


The Valve Positioning Checklist (some examples)

●Check for Tags●Check the VPI (Valve Positioning Indicator)●Always check in the shut direction first●Partially position the valve and check for appropriate

system response●Fully position the valve and check for appropriate system

response●When opening valve, go to fully open, then back off ¼ to ½

turn●When shutting (not “closing!”) valve, torque as appropriate

using calibrated wrench●And the list goes on


Electrical Safety Procedures (some examples)

●Completely deenergize and tag-out as much as possible

●Electrical safety area with rubber mats●Completely remove all metal●Minimize skin exposure and don electrical safety gloves

●Use only tools made “electrically safe”●Test for voltage first●Rope around waist and CPR training


Policies

●Use of Common Language●I.e., port and starboard, shut vs. close, etc.●Avoid acronyms (even though Navy is full of them)

●Repeat Backs●Mandatory – no exceptions


Policies (continued)

●Status Boards●All major systems status manually updated via grease pencils and status boards

●Use of Logs and Trending

●Meticulous Record Keeping

●Tool and Meter Calibration Programs



●Watches●Pre-watch plant tours●Turnover procedures

●Checks and Balances inserted into everything

●Self-Audits●Internal and External Inspections

●ORSE



●Lessons Learned and Incident Reports●Formalized program

●Shared Knowledge●Mandatory

●Manuals Update System●Very meticulous system for ensuring manual revisions and changes were always made correctly and on time


System Design●First and foremost, the entire sub is “pre-engineered” so that system compatibility is ensured

●Alarms and Indicators●Interlocks and Redundancy everywhere●The Importance of Standardization

●Every class (I.e., Los Angeles, Ohio, Seawolf, etc.) of sub is virtually identical

●Personnel from one sub can transfer to another sub with little to no additional training

●Labeling, color coding, general procedures all identical regardless of type of sub


The Irony of Human Error●Goal is to remove the “error” in “human error,” not the “human”

●Reliance on People●This one is important●Although much needs to be put in place to take out

human error, having the human in the process is critical to success (“can’t live with ‘em, can’t live without ‘em”)

●No machine can reason like a human●Saving lives means making hard decisions


War Stories (some relevant, some funny, some interesting)

●Hurried Maintenance and the Blackout

●The Bubbler Boy

●The USS Greenville

●A Friendly Encounter

●Fast Water Pick-Up

●The Pipe and the COB


The Result

●The US Nuclear Navy has never had a nuclear incident in over 50 years of operating nuclear ships and submarines at sea


The Nuclear Propulsion Plant Operator

Tough Minded, Skeptical, Sometimes Even Cantankerous, But Always Technically Competent,

ALWAYS THINKING – What If?

A SPECIAL BREED. He Makes the Difference Between Safe and Effective Operation and

Unacceptable Risk.


Now for the Big Question:

What, if any, of these concepts can or should be applied to

commercial data center facilities?


Commercial Reality

●Cannot train for too long without contributing to bottom line

●Cannot force individuals to “follow orders”

●Cannot discipline in the same way

●Some HR issues are handled differently


But, we can…●Provide Leadership

● Instill desire to perform at one’s top potential● Hire the best

●Utilize Non-Deviation Procedural Compliance● SOP’s for normal operating procedures● EOP’s for emergency procedure (easily accessible and memorized)● MOP’s for maintenance and other procedures● Change control process for all procedures● Maintain status boards● Methodical documentation of maintenance● Etc.

●Train When Possible● Schools● OJT● Perform drills

●Share Knowledge●These are just some examples, challenge yourself as to what else

might apply● Idea is to apply risk mitigation whenever possible


Cool Pictures…


QuestionsAnd

Thank You

schneider electric - nuclear navy and reducing human error

News & Politics

electric plant

electrical system

maintenance procedures

right people procedures

maintenance system

system tag

meticulous system

schematic of system