come see what’s cooking in my lab
DESCRIPTION
Presented by: Chris Sistrunk, Entergy Abstract: IT folks have been doing it for years – building labs to test new products before rolling them out – but the concept is still rather revolutionary to most practitioners of SCADA security. Yet the benefits of a lab are many, including training staff and solving real-world problems by replicating and attacking them in the relatively low-risk lab environment. But how do you pitch this (not inexpensive) idea in a way that gets organizational buy-in? And if your organization is just too small, what are the factors to considering when using a third-party lab? Hear ideas and ask questions of someone who evolved his organization’s capabilities from one small lab to five complete labs.TRANSCRIPT
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Come see what’s cooking in my lab: Why you need a lab and how to get one
Chris Sistrunk, PE Sr. Engineer Entergy – Jackson, MS
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Why do we need a lab, Chris?
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
What happens when you use nmap on an Industrial Control System
http://securityreactions.tumblr.com
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Why do we need a lab?
With a lab, you can • Test relay and RTU seAngs on a replica of producDon systems
• Test new firmware before issuing to field • Perform root-‐cause analysis
– Why is this device locking up once a month?
• Try out new equipment from a vendor
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Why do we need a lab?
Save Dme & money by • CreaDng standard seAngs templates • Find problems before they are widespread (Not having to recall units with firmware issues) • Develop and test equipment pilots in-‐house rather than hiring a company to do it
• Use lab equipment as emergency spare
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Why security tesDng?
• Not all SCADA/relay vendors do negaDve or security tesDng at their factories
• Even if they did, they can’t test equipment the EXACT way that you use it
• Test your own equipment before hackers or some drive-‐by malware does it for you
• Use the results to miDgate vulnerabiliDes
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
What kinds of tesDng?
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
• Factory/Site Acceptance TesDng (RTU system) • Firmware/SoTware TesDng (new or patches) • Protocol TesDng (DNP3, Modbus, etc) • Protocol Fuzzing (custom or off-‐the-‐shelf) • PenetraDon TesDng (Metasploit, etc) • Physical security tesDng (cabinet locks etc) • DOCUMENT! DOCUMENT! DOCUMENT!
What kinds of tesDng?
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
What would be your stuxnet?
• Be a hardhat hacker • Think like an a\acker who has your prints! • Build your systems with layers of defense • If you find a vulnerability, let your vendor know (they might even have a patch)
“To make things work well, you must break them!”
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
How I Audit SCADA Systems
http://securityreactions.tumblr.com
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
OK, how do I get a lab?
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
OK, how do I get a lab?
• Ask your boss! Ask the CIO! Ask Ask Ask! • If you are the boss, ask your best people what they want in their lab and go buy it!
• Put together a plan or a business case! – Add it to NERC/CIP compliance budget (big driver)
• Go get spare equipment and make a rack! • Start small and add to it.
– Mine started as 2 relay racks in my cubicle
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Some ideas
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
SDll can’t afford it?
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Can’t afford one, don’t have the manpower, don’t have the experDse?
• 3rd party tesDng such as Enernex, Digital Bond, Kinectrics, CimaDon to name a few
• The US Gov’t has the Idaho NL NaDonal SCADA Test Bed, Pacific NW NL, & Sandia NL
• Colleges such as Louisiana Tech, Mississippi State, Jackson State have power, SCADA, and security equipment in their labs
• Farm out the tesDng and work with them to get the results you want & capitalize the test costs
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
To be the best, you need the best tools!
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Entergy THQ Virtual Lab Tour
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Transmission HQ Labs
• Transmission HQ moved from NOLA to Jackson • Business conDnuity aTer Hurricane Katrina • Brand new building in Fall of 2009 • 5 large rooms designated for lab space
– Relay & SCADA Lab – CommunicaDons & Security Lab – Real-‐Dme Power System Simulator Lab – Mississippi Grid Lab – High Voltage Lab
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Relay & SCADA Lab
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Relay & SCADA Lab NO
LAB RATS OR
CYBERATTACK SQUIRRELS ALLOWED
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Relay & SCADA Lab
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Relay & SCADA Lab
• Cubicle: 2 racks >> Old Break Room: 7 racks • New THQ: 15 bolted racks, 10 rolling racks
– 40+ ProtecDve Relays (7 different standard panels) – Digital Fault Recorder – 8+ RTUs, 3 CommunicaDon Processors – SubstaDon Grade LAN & Corp Network – GPS Clock (IRIG-‐B), HMI Screen & Keyboard – Toolbox, O-‐Scope, MulDmeter, Cables, WorkstaDons, Chip Burner, Relay & RTU Test Sets, etc
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Relay & SCADA Lab
• THE LAB OF MY DREAMS! • We can replicate almost any substaDon • Test new configuraDons • Test problemaDc field configuraDons • Test new firmware & soTware • Test drive new equipment • Train relay & RTU technicians and engineers
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
CommunicaDons & Security Lab
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
CommunicaDons & Security Lab
• SubstaDon Hardened Router & Switch • Radios of different bands and technologies • Six-‐sided PSP for simulaDng CCA sites • Several field firewalls • Wurldtech Achilles Fuzzer
– Test network robustness of devices – Fuzzing DNP3, Modbus, & IEC 61850 – Test new RTU & Relay firmware patches – Will network storm affect control outputs?
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
CommunicaDons & Security Lab
• Custom DNP3 Fuzzer – Created by Adam Crain to test openDNP3 – Closed source for now – Tests DNP3 *Client* and Server – Project Robus – h\p://Automatak.com/robus – Plan to release as open source next year …stay tuned
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Power Real-‐Time Simulator Lab
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Power Real-‐Time Simulator Lab
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Power Real-‐Time Simulator Lab
“Hypersim is the only real-‐Dme digital simulator with the power to simulate and analyze very large-‐scale power systems with more than 2000 three-‐phase buses.” -‐ h\p://www.opal-‐rt.com • Simulate different fault scenarios
– Will the Relay A, B, C have a misoperaDon? – Will relay fault acDvity affect comm (vice versa)?
• R&D & commissioning tests
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Mississippi Grid Lab
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Mississippi Grid Lab
• MulDpurpose type lab used by Entergy Mississippi T&D Grid Engineers
• InspecDng/repairing equipment • Pre-‐test new panels before field installaDon • Spare parts inventory
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
High Voltage Lab
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
High Voltage Lab
• The Hi-‐VARC (High Voltage AC ResisDve Current) test set provides rapid, automaDc evaluaDon of MOV arresters and polymer insulators using AC voltages up to 132kV.” h\p://www.jmxservices.com
• InspecDon & root cause of failed insulators, HV circuit breaker components, etc
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Last but not least…
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Go make stuff…Go break stuff
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
A Few Thoughts SCADA Security isn’t easy
• Doing the best we can with what we have SCADA, Relay, & Security Labs
• Having a lab is so valuable for tesDng, troubleshooDng, breaking & fixing stuff
• Yes I have a fuzzer and I’m not afraid to use it
DNP3/IP Secure AuthenDcaDon v5 • Please tell your vendors you want NEED it
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Dream BIG!
8th Security Summit Portland, Oregon 9th Security Summit Denver, Colorado
Follow @chrissistrunk
QuesDons?