eareckson air force station, alaska, power plant generator protection project narrated by ted...
TRANSCRIPT
Eareckson Air Force Station, Alaska, Power Plant
Generator Protection ProjectNarrated by Ted Creedon, P.E
Chief Engineer
Creedon Engineering
04/22/23 © Creedon Engineering Rev 1.3 2
You will learn:
• Why a fiber LAN based protective relay architecture was selected
• The (new) skills required to design and commission LAN based generator relays
• The Cyber Security strategy used• What equipment is needed to program and test the
relays• Why Zero Defect Contracting was used• Why training was done prior to design• Project Deliverables• Surprising positive impact on other engine/generator
control systems
04/22/23 © Creedon Engineering Rev 1.3 3
04/22/23 © Creedon Engineering Rev 1.3 4
04/22/23 © Creedon Engineering Rev 1.3 5
04/22/23 © Creedon Engineering Rev 1.3 6
04/22/23 © Creedon Engineering Rev 1.3 7
• “Cobra Dane collects exoatmospheric, multiple-object intelligence data on Russian reentry vehicles and can extract and reduce the data on site. The system also provides early-warning and attack assessment for missiles that would impact the continental United States and detects and catalogs satellites for the U.S. Air Force’s Spacetrack systems. “
Cobra Dane Radar Mission
From USAF website
04/22/23 © Creedon Engineering Rev 1.3 8
04/22/23 © Creedon Engineering Rev 1.3 9
04/22/23 © Creedon Engineering Rev 1.3 10
04/22/23 © Creedon Engineering Rev 1.3 11
04/22/23 © Creedon Engineering Rev 1.3 12
04/22/23 © Creedon Engineering Rev 1.3 13
04/22/23 © Creedon Engineering Rev 1.3 14
04/22/23 © Creedon Engineering Rev 1.3 15
Bus Power Plant Existing Condition
• Unscheduled outages damage Cobra Dane components ($100K-$1.5M/outage)
• Plant can’t hold frequency
• 4 hours to sync generators
• Uncalibrated metering and relays
• Disconnected bus differential wiring
04/22/23 © Creedon Engineering Rev 1.3 16
Power Plant Existing Condition
• Governor actuators hunt causing excessive EGB-50 wear
• 2301 speed control’s frequency response detuned (see Running Hz on following slide)
04/22/23 © Creedon Engineering Rev 1.3 17
04/22/23 © Creedon Engineering Rev 1.3 18
Site Environment
• Winter wind 180 knots
• Summer ground fog
• Bad weather delays travel and spare part shipment for weeks at a time
• Temp +19 to +56F
• Seismic 1 G
04/22/23 © Creedon Engineering Rev 1.3 19
Other Factors
• Plant personnel want new equipment
• “We have to move ahead with new technology”
• Plant on PACAF and MDA commanding general’s daily reports
• 45 year old equipment maintained by very creative personnel
04/22/23 © Creedon Engineering Rev 1.3 20
Project Scope
• Hard dollar USAF Contract
• Design/build
• Contractor decides what to do w/ USAF approval
• Protective relay replacement is in budget and doable
• New engine controls are not in budget
04/22/23 © Creedon Engineering Rev 1.3 21
Relay Replacement Options
• Opt #1: Recalibrate and repair existing induction disk relays
• $2k / relay just for service estimate
• Flash hazard
• Unacceptable to all
04/22/23 © Creedon Engineering Rev 1.3 22
Relay Replacement Options
• Opt # 2: replace with electronic relays 1:1
• Reuse existing wiring and doors
• Recommended by protection engineer
• Risk of incorrect existing wiring causing startup problems
• Risk of excessive on site labor
• Flash hazard
04/22/23 © Creedon Engineering Rev 1.3 23
Relay Replacement Options
• Opt # 3: Rack and stack relays• New dutch doors required• No capability to add general purpose 4-20
ma I/O, rtd’s etc• RS 232 communication is unacceptable• Copper 300v network wiring doesn’t meet
Code• Architecture does not support future
expansion
04/22/23 © Creedon Engineering Rev 1.3 24
Relay Replacement Options
• Opt #4: Modular relays• New dutch doors to cover 15KV breakers while
troubleshooting• Removes maximum existing wiring• Function/card vs function/box supports future
expansion• Dual fiber LAN – will survive switchboard fire• Oscillography will be useful when
troubleshooting multiple generator problems
04/22/23 © Creedon Engineering Rev 1.3 25
Relay Replacement Options
• Opt #4: • Hard wire problems now “soft wire”• Can be prototyped and precommissioned in
Anchorage• Minimum parts count• Can remotely troubleshoot over modem/LAN• Weren’t sure it would be acceptable to the site.
This is why O&M training was held first.
04/22/23 © Creedon Engineering Rev 1.3 26
Relay Replacement Options
• Opt #4 con’t:
• Cyber security is a problem
• Keep 86 lockout relays hardwired
• Passive backplane with plug in CPU and I/O option cards is a well proven architecture
• 32 bit DSP on VT/CT cards offloads CPU
• The relay is a small Cray
04/22/23 © Creedon Engineering Rev 1.3 27
Relay Replacement Options
• Option 4 is the only option that enables Zero Defect Contracting:– Zero injuries– Zero defects– Zero disputes– On time– On budget
• In that order
04/22/23 © Creedon Engineering Rev 1.3 28
Anticipated Problems
• Option # 4 gave the Contractor the best chance to avoid:
• Safety issues (medivac could take days)
• Can reduce flash hazard risk with software
• The power plant having start up problems at the point of no return (Island goes dark)
• Warranty issues ($10-$50K per trip)
04/22/23 © Creedon Engineering Rev 1.3 29
Feature Copper LAN Copper RS 232/485 Fiber LAN
terminations/link 16RS232 2 - 7 (Depends on
TD,RD,CTS,RTS,DSR, GND,DTR)
2
Redundant links no no yes
Incorrect terminations w orks at reduced data rate w orks at reduced data rate doesn't w ork at all
Meets code in 120 VDC compartments No
No - if 600 volt cable is used need termination barriers and
line protectorsYes
Snif fable/tappable Yes Yes With great dif f iculty
Works during faultsNo - according to mfgr's tests,
TCP/IP packets dropped and need to be retransmitted
CMMR problem causd by touch potential shif ting ground levels Immune to faults
Ground Loops Slightly Susceptable very susceptable Immune to ground loops
Distance 100 meters >100 metersSingle mode multiple KM or multi
mode 1,000M
Works if broken Degraded performance Degraded performance No
Muxable Yes- LAN sw itch With dif f iculty - adds ports Yes - Fiber sw itch
Addressable Yes No Yes
Speed 1GB 345K max 1GB
Future speed enhancements (required for 20+ year life)
Replace cable and sw itches Replace cable and muxes Replace sw itches, reuse cable
04/22/23 © Creedon Engineering Rev 1.3 30
TYPICAL MODULAR RELAY ARCHITECTURE, OPTIONS AS
INSTALLED
Note: Governor speed control card was actually prototyped by vendor but never manufactured.
CPU + DUAL FIBER LAN CARD
Standard 4CT/4VT + DSP
4 Form-A (voltage with optional current) outputs, 8 digital inputs
Standard 4CT/4VT +DSP
2 Form-A (voltage with optional current) and 4 Form-C outputs, 4 digital inputs
FUTURE 4-20MA VOLTAGE REGULATOR PID LOOP OR SETPOINT CONTROL
FUTURE 4-400MA ENGINE GOVENOR ACTUATOR PID LOOP OR SETPOINT CONTROL
04/22/23 © Creedon Engineering Rev 1.3 31
Gen 110.1.1.1
Gen 210.1.1.2
Gen 310.1.1.3
Gen 410.1.1.4
Gen 510.1.1.5
Gen 610.1.1.6
Fiber Switch#1
10.1.1.16116 PORT
Fiber Switch#2
10.1.1.5116 PORT
Workstation10.1.1.51
Dual Fiber LAN
Laptop10.1.1.152
Copper LAN
Laptop10.1.1.153
Copper LAN
FUTURE FOUNDATION
FIELDBUS SENSORS & ACTUATORS
FUTURE PID LOOP KW SHARING
FUTURE PID LOOP VAR SHARING
FUTURE ENGINE PID LOOP SPEED
CONTROL
04/22/23 © Creedon Engineering Rev 1.3 32
• The next 3 slides show all the other engine and generator controls that could be replaced with 6 additional modular relays (if VAR and KW load sharing were available)
• However, 4-20ma I/O could be used for set point engine/generator control
• Feeder relays can also be added
04/22/23 © Creedon Engineering Rev 1.3 33
04/22/23 © Creedon Engineering Rev 1.3 34
04/22/23 © Creedon Engineering Rev 1.3 35
04/22/23 © Creedon Engineering Rev 1.3 36
Unified Plant Architecture
• It is now possible to manually control an entire power plant with set point control without having to interface different manufacturer’s equipment
• Complete automatic control would be feasible if PID was available on the DSP I/O cards
• Use LAN for KW and VAR sharing signals
04/22/23 © Creedon Engineering Rev 1.3 37
Project Team
• Chief Engineer & Electrical Contractor– Runs project, provides network & data engineering– Builds prototype system in Anchorage– Builds test bench on site– Writes O&M manuals
• Chief Protection Engineer– Short circuit & coordination study– Drawings – Programs relay and tester (w/ chief engineer)– Commissions relays on site
• Lifts each wire and verifies connectivity• Functional checkout &Start Up
04/22/23 © Creedon Engineering Rev 1.3 38
Project Team
• Switchboard wireman– Reviews design and consults– Builds & pre wires new doors– On site demolition and installation
• Fiber person– Installs and tests fiber on site
• CAD drafter
• Vendor trainer
04/22/23 © Creedon Engineering Rev 1.3 39
Project Team
• The minimum experience level required was 25 years per person
04/22/23 © Creedon Engineering Rev 1.3 40
Deliverables
• Short circuit and coordination study• Relay settings & relay program• Automated 6 voltage, 6 current relay tester
program• On site test reports• Hyperlinked O&M manuals (800 pages)
– 300 typeset custom pages + 500 pages relay manual (vendor manual had to be republished)
– LaTex was used to typeset the custom pages– MS word broke at 30 pages due to color photos
• Project data required 3 CD’s (~1.5 GB)
04/22/23 © Creedon Engineering Rev 1.3 41
Deliverables
• Drawings – @ 30 drawings per– Existing as builts– Demolition– New work– New as builts
• Specs– On drawings– Relays and fiber switches specified prior to
bid
04/22/23 © Creedon Engineering Rev 1.3 42
Submittals
• Vendor does no shop drawings
• Vendor builds exactly what is shown
• Engineering firm required to produce “zero defect, shop drawing quality work” by contract
• O&M manuals– In color– Hyperlinked, with index and table of contents
04/22/23 © Creedon Engineering Rev 1.3 43
Project Schedule
• Sign contract
• 1 week Training
• 1 week site visit
• Prepare drawings
• Fabricate and pre wire doors
• Short circuit and coordination study
• Write relay and tester programs
04/22/23 © Creedon Engineering Rev 1.3 44
Project Schedule
• Ship equipment
• Demolition, installation and commissioning on site
• Write O&M manuals
• As builts
• This all had to be done under a hard dollar, hard completion date, US Government Contract
04/22/23 © Creedon Engineering Rev 1.3 45
Training
• 1 week custom training with vendor, gear fabricator, owner, engineers
• 12 people
• 6 laptops and 6 relays
• 2 fiber switches
• Owner buys into architecture and orders SCADA packages
04/22/23 © Creedon Engineering Rev 1.3 46
Training
• At the end of a week:
• We’re exhausted
• Owner is happy
• Vendors are happy
• The engineers knew they had their work cut out for them
• (During negotiations the remark “After all its “just another relay” ” was made….)
04/22/23 © Creedon Engineering Rev 1.3 47
Training
• If we hadn’t done the training first, the project may have failed
• Training cost $16.5K in Anchorage
04/22/23 © Creedon Engineering Rev 1.3 48
Test Bench
• 3 computers– Relay software– Tester software– Linux box for network packet sniffing
• 2 modular relays• 2 fiber switches• 1 6 Voltage, 6 Current programmable tester -
GFE• Everything on fiber/copper LAN, (RS-232 was
too slow to use)
04/22/23 © Creedon Engineering Rev 1.3 49
Test Bench
• The test bench allowed complete simulation of all faults and trip points
• Pick up, drop out and trip were observed and compared with calculated values.
• 9 different protection groups (subsets of protection elements) were needed to isolate interaction between settings.
• Splitting the protection elements into test groups was unexpected and not covered by the training course.
04/22/23 © Creedon Engineering Rev 1.3 50
Test Bench
• We had a great deal of confidence in the design because of the testing in Anchorage
• Plant personnel comfortable with SCADA GUI, not with the relay or tester programming and GUI’s
• Additional plant personnel were trained in Anchorage using the test bench.
04/22/23 © Creedon Engineering Rev 1.3 51
04/22/23 © Creedon Engineering Rev 1.3 52
04/22/23 © Creedon Engineering Rev 1.3 53
Programming
• Training Gave us a heads up on the relay programming
• Had we known, we would have had relay tester training. The generator test program didn’t exist
• The test program required as much time and thought as the relay program
• About 6 weeks of 7/10’s for 2 engineers• 2 engineers, a protection engineer and a data
engineer/protection engineer are required.
04/22/23 © Creedon Engineering Rev 1.3 54
A Neat Tool
• We used a spreadsheet to organize the protection elements, their settings and test block connections
04/22/23 © Creedon Engineering Rev 1.3 55
04/22/23 © Creedon Engineering Rev 1.3 56
On Site Surprises
• There was one unexplained zit in the protection circles..– It was resolved on island– We had excellent 24/7 relay vendor support
during start up – yes, we woke engineers up
• The tester died during test out of generator #1, on Friday at 9 AM– The tester vendor shipped a replacement that
was ON SITE at 8 PM the next day – from 11,000 miles away
04/22/23 © Creedon Engineering Rev 1.3 57
On Site Surprises
• Oh, by the way, the relay program didn’t work on island– The protection engineer accidentally hit the
automatic firmware update feature in Anchorage
– It was incompatible with the download software
– We had a new CD in 24 hours
• Which brings up another concern:
04/22/23 © Creedon Engineering Rev 1.3 58
Cyber Security• It not a protective relay, it’s a computer on a LAN• TFTP is a problem with the fiber switch – it automatically tries to
load new code on startup. Disable tftp.• Web server can’t be disabled• Passwords use reversible encryption!• Fiber LANs are difficult to compromise• Use unroutable IP addresses (10.x.x.x)• Physical security – 5 rings
– Shemya is surrounded by 28 F water– Air Police with M-16’s patrol the power plant– Trusted employees– Isolation and disconnection from any network– Fiber
• Yearly cyber security audit required• Bit by bit firmware verification (Linux diff) do not rely on hash
schemes (MD5, SHA1, etc)
04/22/23 © Creedon Engineering Rev 1.3 59
Cyber Security
• If you’re not convinced that Cyber Security is a problem, read page 268 of ISBN 0-89141-821-0, “At The Abyss” by Thomas C Reed, Former Secretary of the Air Force. He details the sabotage of a Soviet gas line with firmware trojans planted in the Urgenoi gas field pump stations in 1983.
04/22/23 © Creedon Engineering Rev 1.3 60
Example of what happens if on a intranet or internet
• Hack the http server• [Sat Aug 19 21:33:36 2006] [error] [client 24.20.123.11] request failed: URI too long (longer than 8190)• [Sat Aug 19 21:34:08 2006] [error] [client 24.20.123.11] File does not exist: /srv/www/htdocs/_vti_bin• [Tue Aug 22 18:08:23 2006] [error] [client 24.236.230.63] File does not exist: /srv/www/htdocs/scripts• [Tue Aug 22 18:08:23 2006] [error] [client 24.236.230.63] File does not exist: /srv/www/htdocs/MSADC• [Tue Aug 22 18:08:24 2006] [error] [client 24.236.230.63] File does not exist: /srv/www/htdocs/c• [Tue Aug 22 18:08:24 2006] [error] [client 24.236.230.63] File does not exist: /srv/www/htdocs/d• [Tue Aug 22 18:08:24 2006] [error] [client 24.236.230.63] File does not exist: /srv/www/htdocs/scripts• [Tue Aug 22 18:08:24 2006] [error] [client 24.236.230.63] File does not exist: /srv/www/htdocs/_vti_bin• Hack the ftp server• Aug 31 07:39:47 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [user] [Administrator]• Aug 31 07:39:47 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [pass] [<*>]• Aug 31 07:39:47 redcloud pure-ftpd: ([email protected]) [WARNING] Authentication failed for user [Administrator]• Aug 31 07:39:51 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [user] [Administrator]• Aug 31 07:39:52 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [pass] [<*>]• Aug 31 07:39:52 redcloud pure-ftpd: ([email protected]) [WARNING] Authentication failed for user [Administrator]• Aug 31 07:40:00 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [user] [Administrator]• Aug 31 07:40:00 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [pass] [<*>]• Aug 31 07:40:00 redcloud pure-ftpd: ([email protected]) [WARNING] Authentication failed for user [Administrator]• Aug 31 07:40:11 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [user] [Administrator]• Aug 31 07:40:11 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [pass] [<*>]• Aug 31 07:40:11 redcloud pure-ftpd: ([email protected]) [WARNING] Authentication failed for user [Administrator]• Aug 31 07:40:24 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [user] [Administrator]• Aug 31 07:40:25 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [pass] [<*>]
• > 202.40.190.42• Server: 10.1.1.xxx• Address: 10.1.1.xxx#53
• Non-authoritative answer:• 42.190.40.202.in-addr.arpa name = ritt-190-42.ranksitt.net.• www.ranksitt.net is in Bangladesh• www.sovam.com is a Russian Army site in Krasnoyarsk• Protective relays and fiber switches should have internal firewalls as a Cyber Security option
04/22/23 © Creedon Engineering Rev 1.3 61
04/22/23 © Creedon Engineering Rev 1.3 62
04/22/23 © Creedon Engineering Rev 1.3 63
04/22/23 © Creedon Engineering Rev 1.3 64
04/22/23 © Creedon Engineering Rev 1.3 65
04/22/23 © Creedon Engineering Rev 1.3 66
04/22/23 © Creedon Engineering Rev 1.3 67
04/22/23 © Creedon Engineering Rev 1.3 68
04/22/23 © Creedon Engineering Rev 1.3 69
04/22/23 © Creedon Engineering Rev 1.3 70
Data from the Relay
• Data from the relay displayed on a laptop in the control room:– Enabled accurate setting of generator voltage
and frequency• Synchronization problem completely eliminated
– Enabled tuning of the 2301’s providing good frequency response
• Transient load sharing improved
– Eliminated frequency oscillations
04/22/23 © Creedon Engineering Rev 1.3 71
Data from the Relay
• Enabled rebuilding the engine fuel rack linkages, eliminating dead band.
• Allows 2 generators to accept 3500 KW each, when 1 of 3 on line goes down the other 2 now accept 100% load
• Eliminated complaints from Cobra Dane
• Electrical faults in Cobra Dane no longer trip generator breakers
04/22/23 © Creedon Engineering Rev 1.3 72
One problem
• While re-roofing the power plant one relay power supply was shorted out due to rain infiltration
• The relay power supply repair cost $2K• The relay program was loaded from its
own CD (6 CD’s each differing only by IP address) and installed
• Yes, the relay should have been recommissioned ($50K). But it wasn’t.
04/22/23 © Creedon Engineering Rev 1.3 73
One problem
• And it worked just fine!
04/22/23 © Creedon Engineering Rev 1.3 74
Follow Up
• As of 8/25/2006 (2 years later)
• Plant personnel receive periodic factory training
• Are planning relay improvements and additions designed and installed themselves
• They are totally self reliant
04/22/23 © Creedon Engineering Rev 1.3 75
The Future• SCADA/PMCS software can be user written in .NET 2 or .NET 3
– Event driven– Free AES encryption– The relay as a data source– Better graphical user interfaces
• More add in cards for– Speed + KW share– Voltage + VAR share– DSP for engine/generator controller apps
• Interface cards to other plant protocols – – Fieldbus – use redundant sensors that are NIST traceable– I/O interfaces to other protocols: CAN, etc.
• User programmable PID, etc..• Cyber Secure Certs by mfgrs
04/22/23 © Creedon Engineering Rev 1.3 76
Address
tcreedon /at/ easystreet /dot/ com
Ted Creedon, P.E.Chief EngineerCreedon Engineering5740 SW Childs RdLake Oswego Or 97035503 620 0492