ww

w.in

l.g

ov

AGA Webinar - II

Presented by

Jonathan Gray – Training and R&D Lead at INL

Integrating Cyber Security into

an Operations Structure

Who are you?

• Understand your business

• All Networks:

– Orders / Schedules

– Delivery / Operations

– Billing / Reclamation

– Marketing / Sales

– Organizational Management (HR, email, etc.)

– Databases

– Who are your customers & stakeholders?

– Customer Service (emergency and non-emergency)

SCADA LAN

CORPORATE LAN

MODEMPool

Communications

VENDORS

Substations

Internet

ICCPPEERS

SERIE

S

MetersBreakersSwitches

TransformersRelays

RTU/PLC

Corporate

PBX

Corporate

Firewall

DNSServer

WebApplications

Servers

`

BusinessWorkstations

BusinessServers

CommunicationsServers

Vendor

Modem

ConfigurationServer

`

HMIComputers

EngineeringWorkstation

DatabaseServer

HistorianApplications

ServerFront End Processor

SCADA

Firewall

DMZ LAN

WWWServer

ICCPServer Historian

Host Intrusion & Application Log Detection Systems

Network Intrusion Detection & Prevention Systems

Virus, Spyware, & Bot Detection Systems

?

Drivers of Automation

• Regulatory Guidelines and Requirements

– PHMSA, DOT, and NTSB recommendations and requirements

– 2011 TSA Pipeline Security Guidelines

– 2013 Presidential Executive Order & Policy Directive (PPD-21)

• Business requirements for cost efficient operations

– Intelligent maintenance systems

– Integration of business systems and operations

– Improved operations through better situational awareness

– Reduce company costs

Elements of a Security Plan per TSA

• System Description

• Security Administration and management structure

• Risk analysis and assessments

• Physical security and access control measures

• Equipment maintenance and testing

• Personnel screening

• Communications

• Personnel training

• Drills and exercises

• Security incident procedures

• NTAS response procedures

• Plan reviews

• Record keeping

• Cyber/SCADA security measures

• Essential contact list

• Security testing and audits

• Defense in depth approach

Challenges

Protecting Tangible Assets

Technology

• Barriers

• Alarms

• Locks

• Doors

Policies

• Access restrictions

• Methods and

practices

People

• Guards

• Operations

• Maintenance

• Inadequately

managed technology

• Poor management of

Change

• Operational

Inflexibility

• Insiders

• External Parties

Protecting Digital Assets

• Defense in depth approach

Technology

• Firewalls

• Logs

• Network Monitoring

tools (IDS, IPS, etc.)

• Antivirus

Policies

• Access restrictions

• Methods and

practices

• Security control

• Technology use

People

• Users

• Generic Accounts

• Administrators

Challenges

• Inadequately

managed technology

• Legacy systems

• Inadequate Change

Control Policies

• Operational

Inflexibility

• Insiders

• External Parties

Technology Challenges

• Managing technology

– Lifecycle (design, installation, production, patches, and disposal)

– Design documents and technology information needs to be safe guarded

– Chasing technology will not solve your problems

• Managing existing technology

• Identifying gaps in existing methods/technology and available technology to fill those gaps

• Implement the technology in collaboration with integrators

• Understand what the technology will and will not do

• Understand the technology challenges and interoperability difficulties that must be managed

• Manage your assets

• Continuous training

Manage Technology

• Perform technical audits of SCADA devices and networks, and any other connected networks, to identify security concerns

• Conduct physical security surveys and assess all remote sites connected to the SCADA network to evaluate their security

• Establish SCADA “Red Teams” to identify and evaluate possible attack scenarios for technology similar to a hazardous operations study

• Harden SCADA networks by removing or disabling unnecessary services

Managing Technology (contd)

• Clear operational boundaries

– If you shutdown x, restart within 5 min or everything goes down

– If you shutdown z, please restart within an hour

– Operations staff should acquaint themselves with asking permission to scan things and convince operations managers something is a good idea – biggest case for SAND BOX or TEST BED

• Newly procured/installed systems should not cause operations to fail if they themselves are turned off for a short period of time (e.g. rebooted)

– They should also come back as easily as possible, failing gracefully (e.g. hard boot)

Ancillary networks

• Back to basics, utilities are a business

• Orders, Deliveries, and Billing data, at some point, go through operations networks

• Business related operations networks such as those that host the support equipment for scheduling or billing, should be isolated from other business networks, just like the SCADA network

Vulnerabilities & Exploits

• Vulnerabilities are not exclusive intellectual property

– Any number of people can discover a vulnerability

• Not all vulnerabilities are exploitable

• Exploited vulnerabilities are typically prioritized first by vendors for mitigations and patches

Exploitable Vulnerability Discovered

Vendor Acquires Vulnerability

Vendor Releases Patch

End User Patches Vulnerability

Exploitable Vulnerability Discovered

Malware is noticed

Antivirus updates signatures

Vendor makes patch

Antivirus is tested/patched

System is patched

System is vulnerable

Responsible Disclosure Timeline

Malware Infection Timeline

Software is vulnerable but system is protected by AntiVirus

Exploitable Vulnerability Discovered

Malware is noticed

Antivirus updates signatures

Vendor makes patch

Antivirus is tested/patched

System is patched

System is vulnerable

Malware Infection Timeline

Software is vulnerable but system is protected by AntiVirus

What is the order of magnitude for the red line? 10 months

• On average, Symantec found 300 days of exploit before a patch was available.

• Additional research conducted by Red Tiger on contract with INL under the guidance

of The Department of Homeland Security also supports this order of magnitude

(330 days average for control systems to be patched once a patch was available).

This does not include containment and cleanup time.

Network Connectivity

1. Identify all network communications

2. Disconnect unnecessary connections

3. Strengthen remaining connections

– Firewall rules tailored for inbound and outbound traffic

– Communication monitoring

• Emergency response

– How are guest/visitors connections managed during a natural disaster?

• Vendor and contractor support

– How are vendor & contractor connections managed?

Policy Challenges

• Management of Change

– Automated systems can log who/when/what –

• Some control system vendors were driven to this by the pharmaceutical/food industries FDA - CFR 21 part 11

• NERC is requiring management of change per CIP 11

• Regulation (DOT, PHMSA, NTSB, etc.)

– Technology with some training is more effective than technology applied with no training

– Automated systems typically link into a database that will itself require some administration and security controls

– Operational flexibility often conflicts with management of change

– Implemented properly, management of change can assist in audits with hard evidence of tracking the change management process (Demonstrates application of NTSB recommendations and requirements)

Policy Challenges

• Operational Flexibility

– Equipment fails, fluke circumstances occur, configuration problems are spotted, etc.

– Self documenting software helps however you will typically be authorizing changes (logins) and clicking more per change

– The more mature the control algorithms, the less changes to the configuration are required

– Build configuration with variables that can be modified for altered behavior rather than having to rebuild the configuration

Temporary Connections

• Establish strong controls over any medium that is used as a temporary connection into the SCADA network

– USB devices

– Vendor VPN and modem connections

– Integrator VPN and modem connections

Monitoring Networks

• Monitor natural gas transmission and distribution pipeline operations 24/7 with available personnel for response

– Gas Control Center

– Customer Service call center

• Monitor networks and communication systems that manage the gas transmission and distribution pipeline operations 24/7

– Automated monitoring versus manual monitoring

– The world never sleeps

Operational Cybersecurity

• Logistical difficulties (management of change)

– Patching antivirus

– Patching software (operating system & all third party applications)

– Monitoring logs (use SIEMS, NBA, NBAD, etc.)

• Baselines without a compromised system

• Use a clean install to build a baseline configuration

• Apply data to a production system, check out all anomalies (can be time consuming)

Operational Difficulties

• People

– Current staff is busy

– Internships can augment staff

– You keep employees by competing with where they could go

• Logs

– Correlating Log files takes time

– Taking action based on the logs requires some measure of authority

– Data mine the logs

– Logs will include – workstations, network traffic, security devices, servers

Ownership

• “There’s Nothing I Can Do”

• “I Don’t Even Know What It Is”

• “That’s For The Networking People To Worry About”

• The three legged stool has grown another leg:

– Operators

– Engineers

– Technicians

– Information Technologists

• There may be wobble until you adjust each leg to fit the circumstance

23

Ensuring Effectiveness

• Ensure consistent commitment and leadership support

• Clearly define cyber security roles, responsibilities, and authorities for managers, system administrators, and users

• Establish a rigorous, ongoing risk management process

• Clearly identify cyber security requirements (formalize policies and procedures)

• Identify systems that serve critical functions and implement additional levels of protection

• Establish security controls over any medium used as a backdoor into SCADA network

• Establish effective configuration management processes (hardware and software)

Ensuring Effectiveness (contd.)

• Conduct routine training and information awareness campaigns

• Conduct routine self-assessments

– Audit versus assessment

• Cyber incident response plans

– Exercise incident response plans annually or as appropriate

• Emergency response plans (evacuation, notification, response, etc.)

– Exercise response plans annually

• Disaster recovery plans (backup SCADA control centers, communication, etc.)

– Routinely exercise recovery plans

– Adjust plans from lessons learned

Incident Response

• Two hypothetical scenarios:

– Vandal or thief

– Pipeline Leak

• Two hypothetical scenarios:

– Malware Infection

– Cyber Intrusion

Response – What to Do

• Turn off gas appliances.

• Leave the area by foot immediately. Do not try to locate the source of the odor or leak. Try to direct other individuals to leave the area. Attempt to stay upwind.

• Call 911 from a safe location; then, notify the pipeline company and or your local emergency response number if known. Provide the emergency operator your name, phone number, a brief summary of the incident, and the location.

These items were copied from several natural gas utility public service documents

Response – (continued)

• DO NOT come into direct contact with any escaping liquids or gas.

• DO NOT attempt to operate any pipeline valves yourself. You may inadvertently route more product to the leak or cause a secondary incident.

• DO NOT cause any open flame or other potential source of ignition such as an electrical switch, vehicle ignition, light a match, etc. Do not start motor vehicles or electrical equipment.

• DO NOT use telephone or cell phone. If inside a home or business, do not pull plugs from electrical outlet or open an automatic garage if the vehicle is parked inside.

• DO NOT ring doorbells to notify others of the leak. Knock with your hand to avoid a potential spark from metal knockers.

• DO NOT drive into a leak or vapor cloud while leaving the area.

• DO NOT attempt to extinguish a natural gas fire. Wait for local firemen and other emergency professionals trained to deal with such emergencies.

These items were copied from several natural gas utility public service documents

Response – What is Expected

• Upon notification of an incident or leak:

– Dispatch trained personnel to isolate the pipeline emergency

– Minimize the amount of product that leaks out

– Assist public and safety officials in their response to the emergency

These items were copied from several natural gas utility public service documents

Response – What to Do

• Do you reboot? Rebooting the system will destroy in-memory data relating to the malware or intruder. Rebooting will also disconnect remote network connections. This may be operationally ideal however it does not improve the understanding of who or what is intruding on the network. Understanding the malware is important to stopping the malware. Cover this operational policy in your cyber security plan.

• Note in the log file

– Date and time of the incident

– Applications in use

– Description of the observation

Response – What is Expected

• Upon notification of a cyber incident or leak:

– Dispatch trained personnel to isolate the cyber emergency

– Assess the amount of data loss / compromise

– Notify appropriate parties based on existing regulations

Questions?

33

• http://energy.gov/oe/national-scada-test-bed

• Jonathan Gray (Jonathan.Gray@INL.gov)

• Shabbir Shamsuddin (Shabbir.Shamsuddin@ANL.gov)

• 21 Steps to Improve Cyber Security of SCADA Networks

– http://energy.gov/oe/downloads/21-steps-improve-cyber-security-scada-networks

• Pipeline Security Guidelines

– http://www.tsa.gov/sites/default/files/assets/pdf/Intermodal/tsa_pipeline_sec_guideline_april2011.pdf

Contacts and Documents

34

U.S. Department of Energy

Office of Electricity Delivery and Energy Reliability

Cybersecurity for Energy Delivery Systems