WP-028-2020

Market survey of OT security sensors for substations

Version 0.2

7 August 2020

2

The European Network for Cyber Security (ENCS) is a non-profit member organization

that brings together critical infrastructure stake owners and security experts to deploy

secure European critical energy grids and infrastructure. Founded in 2012, ENCS has

dedicated researchers and test specialists who work with members and partners on

applied research, defining technical security requirements, component, and end-to-end

testing, as well as education & training.

3

Version History

Date Version Description

7 August 2020 0.1 Initial draft

7 August 2020 0.2 ENCS internal review

4

Table of Contents

Version History .................................................................................................................... 3

1 Introduction .................................................................................................................. 5

1.1 Objective ............................................................................................................... 5

2 Use Cases .................................................................................................................... 6

2.1 Usability ................................................................................................................ 6

2.2 Detection Capabilities ........................................................................................... 6

2.3 Deployment and Integration .................................................................................. 7

3 Analysis ........................................................................................................................ 8

3.1 Usability ................................................................................................................ 8

3.1.1 Asset Inventory Capability ............................................................................. 8

3.1.2 Network Communications Visualization ........................................................ 9

3.1.3 Learning Stage ............................................................................................ 10

3.1.4 Protocols Parsed ......................................................................................... 11

3.2 Detection Capabilities ......................................................................................... 11

3.2.1 Passive Scanning ........................................................................................ 11

3.2.2 Weak Ciphers, Algorithms and Software Versions...................................... 11

3.3 Deployment and Integration ................................................................................ 12

3.3.1 Bandwidth .................................................................................................... 12

3.3.2 Ports in the Sensors .................................................................................... 12

3.3.3 SIEM Integration .......................................................................................... 12

3.3.4 VLANs and Tagged Traffic .......................................................................... 12

3.3.5 Maintenance and Management ................................................................... 12

3.3.6 Security ........................................................................................................ 12

Annex ................................................................................................................................ 14

5

1 Introduction

In 2017, the European Network for Cyber Security (ENCS) developed an Operational

Technology (OT) Security Monitoring project for its members in the critical infrastructure

sector. The same year, work on the first part of the project with the goal to evaluate

various Intrusion Detection Systems (IDS) available on the market was completed. These

existing solutions were assessed against pre-defined use cases to understand if they

worked according to their claims.

In 2020, work on the second part of the project had begun, aiming to understand which

IDS solutions are most suitable for ENCS members’ needs. This report is the result of an

initial market research, documentation review and meetings with relevant vendors. The

solutions will be assessed further in the next testing stage.

The analysis of the results shows that most of the evaluated solutions work with flow

monitoring, whitelisting, Deep Packet Inspection (DPI) and signature-based detection

engines. Examples of these types of solutions are Nozomi, Forescout, Rhebo and

CyberX. Other solutions have detection engines based on machine learning, such as

Darktrace and statistic-based, such as Omicron.

The majority of the IDS vendors have a high degree of flexibility regarding the placement

of the sensors and their architecture. Therefore, even though a centralized approach is

considered the most common, the vendors do not discard the possibility of a

decentralized one. A decentralized approach can provide better benefits for securing the

substations. Yet, this solution can be expensive as more equipment is needed. For

achieving this, most of the vendors offer different types of sensors according to their

number of ports. Others, such as CyberX offer an approach based on mirrored traffic

which reduces hardware costs.

1.1 Objective The principal objective of the OT Security Monitoring Project 2020 is to assess which

Intrusion Detection System solutions for OT environments can best comply with

members’ needs for a substation environment. The results from this report can be used to

choose an IDS solution or a combination of solutions that can best fit their infrastructure

and processes. Moreover, this report will allow an informed decision to be made

regarding which solutions are worth testing in a substation environment.

6

2 Use Cases Based on ENCS members’ needs, four use cases have been developed to evaluate the

available IDS solutions.

2.1 Usability A solution with a high degree of usability can easily achieve efficiency and effectiveness

in the day-to-day operations. When the usability degree of a determined solution is high,

the user can easily navigate through the solution and the information is enough for the

user to work with. The main elements assessed are:

• the solution’s capability to build an asset inventory,

• the solution’s capability to visualize the communications network,

• the solution’s capability to manage and display the alerts.

• the expected learning curve for the users,

• the protocols (IT, OT, proprietary OT) parsed by the solution.

2.2 Detection Capabilities Intrusion Detection Systems utilize a combination of detection methods (also called

detection engines) to capture anomalies in the systems or networks. The three main

detection methods are:

1. Signature-based detection: This engine monitors the packets in the Network and

compares them with previously known malicious instructions sequences used by

malware. The limitation of this engine is that it cannot detect new threats or zero-

day malware.

2. Anomaly-based detection: This engine creates a baseline of the “Normal

Behavior” of the system in order to create a model of trustworthy activity. The

new behavior is compared against this model and when the behavior deviates

from the baseline an alert is raised. The baseline can be created with machine

learning or with statistical models. The limitation of this engine is that if unwanted

behavior is part of the baseline, the system classifies it as normal behavior.

3. Deep Packet Inspection: This engine inspects in detail the data being sent over a

computer network and ensures that the data is in correct format and analyses the

presence of malicious code or eavesdropping. Instead of using DPI, some

intrusion detection systems also use stateful packet inspection which a shallow

packet inspection that only considers the second header.

Nonetheless, some state-of-art Intrusion Detection Systems are now making use of

Machine Learning techniques or concepts such as functional security as part of their

detection suite.

The elements considered for the evaluation are:

• Passive Scanning Capabilities,

7

• Detection of weak ciphers, outdated cryptographic algorithms, and software

versions,

• Detection engines used.

2.3 Deployment and Integration The deployment of a solution covers all activities needed to prepare the software and

make it available for usage. For this, interaction and cooperation between the user and

vendor are required. The integration use case evaluates which are the requirements for

its installation, the installation procedure, the integration of the solution to hardware

technologies such as firewalls or routers, and the recommended architecture to position

the sensors. The ease of deployment and integration is evaluated based on:

• bandwidth,

• sensor management,

• SIEM integration,

• VLANs and tagged traffic,

• maintenance and management,

• security features.

8

3 Analysis

ENCS contacted a total of 16 IDS vendors for OT environments based on:

• an initial market research by ENCS,

• the interests expressed by members,

• previous consideration in the OT Security Monitoring Project 2017.

Out of the contacted, the following six fully completed the questionnaire and their

answers are thus considered in the analysis1:

1. Claroty (Claroty),

2. CyberX (CyberX),

3. Forescout (Silent Defense),

4. Nozomi (Guardian),

5. Omicron (StationGuard),

6. Rhebo (Industrial Protector).

The following three vendors did not complete the questionnaire, but have expressed

interest in participating in the Proof-of-Concept (PoC) testing stage of the project.

1. Darktrace (Industrial Immune System),

2. Sentryo (ICS CyberVision),

3. Tenable (Industrial Security).

The use cases analysis of the IDS solutions is carried out based on information gathered

from the vendors through marketing data, questionnaires, and meetings.

3.1 Usability

3.1.1 Asset Inventory Capability

All sensors evaluated can automatically create an asset inventory of devices in the

substation. They will all store network related information, such as the IP address, MAC

address, and host name. Most will also use information from deep-packet inspection to

gather information on the type of equipment, the equipment manufacturer, and software

or firmware versions. It is not clear how well the deep-packet inspection will work for

substation equipment. This will be evaluated in the PoC phase.

1 Enigmedia (https://enigmedia.es/) also completed the questionnaire, but their solution was deemed out of

scope for the project following an initial review. This is because their solution does not follow the traditional IDS scheme using a security-in-obscurity approach, where traffic is encrypted and a basic alert system created. This would still need to be combined with another traditional IDS system. Consequently, their responses are not considered in the analysis.

9

The asset inventory can be imported and exported in both manually (most common

format is .csv) and automatically (most common format is API). All the solutions

evaluated offer both, except for Claroty and CyberX which only offer automatic.

3.1.2 Network Communications Visualization

Network communications visualization is the capability of the solution to provide a

graphical interpretation of the communication between assets in a network. A network

map should properly demonstrate:

• network segments,

• assets and their characteristics per network segment,

• communication details between two different assets.

Additionally, to the functional network maps, most of the Intrusion Detection Systems will

also alert in the network map any type of anomalous activities (unusual connections,

unknown new assets, etc.). User interface and user experience play an important role

selecting a suitable vendor. Color coded status per asset, characteristics per asset and

characteristics of the communication between assets should be elements to consider.

All evaluated vendors provide a network communication visualization map. The difference

is the type of visualization technique used: Purdue model or network relationship map. In

general, a Purdue model should be preferred as it is the easiest to read.

Table 1. Network Diagram per vendor

Vendor Format

Claroty Purdue Model & Network-Based Diagrams

CyberX Purdue Model

Forescout Purdue Model, Network & Asset Role, Role

Nozomi Network Diagram

Omicron ZeroLine Diagram

Rhebo [TBD]

10

Image 1. Example of Purdue model by CyberX

Image 2. ZeroLine Diagram used by Omicron

3.1.3 Learning Stage

Most security sensors evaluated require a learning phase to create a baseline of normal

network traffic. The average expected learning time is two to three weeks.

Only StationGuard (Omicron) does not require a learning phase. Instead, it loads an SCL

file, created during substation configuration, which describes most of the devices and

their communication patterns. It then detects deviations from the SCL files.

11

3.1.4 Protocols Parsed

IDS solutions should be able to parse the most protocols used in substation automation,

specifically IEC 61850 MMS, IEC 61850 GOOSE and IEC 104. Only Omicron and Rhebo

stat the cannot to parse IEC 104.

3.2 Detection Capabilities

All the assessed IDS solutions can support passive scanning approaches for their

implementation and detection capabilities. Most solutions can identify old or weak

cryptographic protocols. The main difference lays in the library that they use for

identifying these protocols or algorithms. Most of all the evaluated solutions can detect

old software implementations, only Forescout and Nozomi offer a partial approach.

3.2.1 Passive Scanning

Passive Scanning is a technique where the IDS silently analyses the network traffic to

identify endpoints and traffic patterns. IT does not generate additional traffic and does not

disrupt critical processes as it is not in contact with the endpoints. All the tested solutions

can support passive scanning approaches. This implies the identification of the assets

and threats without communicating with the asset or the environment.

3.2.2 Weak Ciphers, Algorithms and Software Versions

A weak cipher is defined as an encryption algorithm that uses a key of insufficient length,

specifically less than 128 bits. As this increases the likelihood of an encryption being

compromised, the IDS solution should have the capability to detect key sizes that could

pose a risk to the system. Similarly, outdated cryptographic algorithms and old software

versions are also considered a vulnerability because of the existence of exploitation tools.

The IDS solution should be able to detect when a cryptographic algorithm has been

corrupted or when a software version is outdated.

Most evaluated solutions are able to identify weak encryptions and standards as well as

old or vulnerable protocols. Each solution works with a threat library serving as a data

base that can deliver an alert when a vulnerable protocol or algorithm is identified. Only

CyberX requires a third-party solution to identify weak ciphers and outdated algorithms.

Forescout and Nozomi are also exceptions as they can only partially identify old software.

Forescout can do this by extracting information about the installed ICS application or

firmware from network traffic. Custom SD Scripts can be written to alert if a minimum

version software is detected. Nozomi uses deep packet inspection to detect software

versions if the information comes from banners (SMTP).

12

3.3 Deployment and Integration

3.3.1 Bandwidth

The bandwidth is the amount of data that a solution can handle at a given time and is

expressed in bits per second. The bandwidth of the solutions ranges from 1Gbps to 8

Gbps. Rhebo has by far the lowest value of 100 Mbps and Omicron comes out on top

with 8Gbps. Several vendors indicate that the actual bandwidth is highly dependent on

the specifications of the sensor hardware.

3.3.2 Ports in the Sensors

The number of ports and the speed that each port can handle in each one of the IDS

sensors were evaluated. It is generally recommended to choose a solution with 4 or more

ports. Among the evaluated solutions the number of ports ranges from 3 to 20, with two

vendors not having provided any details at this stage. Forescout offers to most with 12

monitoring and 8 fiber connections.

3.3.3 SIEM Integration

The SIEM provides real-time analysis of the security alerts generated by the IDS. It is

important to consider which IDS solution integrates with which SIEM solution and how

that integration is achieved. All the vendors support integration with the major SIEM

platforms and most support open standard syslog formats CEF and LEEF. In general,

SIEM integration is considered such a basic feature that it is not expected to be a

problem with any IDS solution.

3.3.4 VLANs and Tagged Traffic

All the evaluated solutions have the capacity to handle different VLANs and tagged traffic.

This is important, as VLAN tags are use in GOOSE traffic for some IEC 61850

configurations.

3.3.5 Maintenance and Management

All solutions work with a central console or command center to centrally manage the

sensors. All have a monitoring system that alerts the user when health or performance

issues arise. This is done and displayed by the central management console.

Update management is done via the customer portal in all solutions, except Rhebo which

sends updates to the customer. All evaluated IDS solutions require manual installation of

updates.

3.3.6 Security

Security as part of the software development process and during usage is one of the

most important elements. This implies the process that involves people and practices to

13

make sure that the application or solution complies with the CIA triad: Confidentiality,

Integrity, Availability and Non-Repudiation as well.

All the evaluated solutions offer a central console for (whitelisting) rules and firmware

updates. The most common security measures applied include hardening, third party

penetration testing and some security certifications.

14

Annex Table 2. Usability

Usability/Vendor Claroty CyberX Forescout Nozomi Omicron Rhebo

Asset Inventory Yes Yes Yes Yes Yes Yes

Import Data Automatic Automatic Manual/

Automatic

Manual/

Automatic N/A

Manual/

Automatic

Network Communication

Visualization

Purdue Model/

Network Diagram Purdue Model

Purdue Model/

Network Diagram

Network

Diagram

ZeroLine

Diagram Spiral Diagram

Alert Management Yes Yes Yes Yes Yes Yes

Requires

Learning Time Yes 2-3 weeks Yes TBD Yes

2-3

weeks Yes TBD No N/A Yes 2 weeks

OT protocols

IEC 61850

MMS Yes Yes Yes Yes Yes Yes

IEC GOOSE Yes Yes Yes Yes Yes Yes

IEC 104 Yes Yes Yes Yes No No

15

Table 3. Detection Capabilities

Characteristics/Vendor Claroty CyberX Forescout Nozomi Omicron Rhebo

Passive Scanning Yes Yes Yes Yes Yes Yes

Weak Ciphers/Outdated cryptographic

algorithms Yes

No (Requires 3rd

party

integration)

Yes Yes Yes Yes

Outdated Software versions Yes – From

Banners

Yes – From

Banners

Partially –

Custom SD

Script can be

written to alert

Yes – rom

DPI and

Banners

Yes Yes

Detection Engines

Signature Based Yes Yes Yes Yes Yes Yes

Anomaly Based Yes No Yes No Yes Yes

DPI for OT

protocols Yes Yes Yes Yes No Yes

Whitelisting Yes Yes Yes Yes Yes Yes

Frequency Based Yes Yes Yes Yes Yes Yes

Others No No No No Yes Yes

16

Table 4. Integration

Integration

Characteristic/Vendor Claroty CyberX Forescout Nozomi Omicron Rhebo

Bandwidth

Throughput 1/Gbps 3/Gbps 3/Gbps 6/Gbps 8/Gbps 100/Mpbs

Management Console TBD

Not required.

Sensor

processes data

10% of the total

throughput

Limited. It

only uses

metadata

Not required.

For Client

software 1

Mbit/s

TBD

Ports

Number TBD Max. 8

Max. 12 monitoring

& 8 fiber

connections

3-18 ports TBD 4

Speed TBD 1 Gb/port 10, 100, 1000 MB 100 Mb to 10

Gb

9-8 Gb for

mirror ports, 1

Gb

management

interface, SOC

and HMI.

Depends on

the hardware

17

Table 5. Deployment

Deployment

Characteristic/Vendor Claroty CyberX Forescout Nozomi Omicron Rhebo

Central Console Yes Yes Yes Yes Yes Yes

SIEM Integration

FireEye, IBM

QRadar, Splunk,

ArcSight,

LogRhythm, RSA

NetWitness.

IBM QRadar,

Splunk,

FortiSiem, RSA

NetWitness,

ArcSight,

LogRhythm

TBD

IBM Qradar,

ServiceNow and

Splunk. By using

CEF with

LogRhythm,

Elastic,

FortiSIEM,

Solarwinds.

Commercial and

open-source

SIEM systems

using Syslog

UDP

Splunk, IBM

QRadar

Handle VLAN Yes Yes Yes Yes Yes Yes

Handle Tagged Traffic Yes Yes Yes Yes Yes Yes

18

Table 6. Deployment

Deployment

Characteristic/Vendor Claroty CyberX Forescout Nozomi Omicron Rhebo

Maintenance

Via HTTPs

and SSH

Remote

Support

Remote Support via

VPN

Collect

data from

Guardian

and

through

Nozomi

Support

By Email or secure upload TBD

Update

Management TBD Via

Customer

portal

Via Customer portal Via

Customer

portal

Via Customer portal Sent to the

customer

Customer

Notification

TBD TBD By email TBD Ny email TBD

Update

Installation

TBD Manual

Update

Manual Update Manual

Update

Manual Update Manual

Update

Security Hardened,

ISO 27001

compliant, 3rd

party pen

testing

SSDLC., 3rd

party pen

testing

FIPS, communication

encrypted (also for

SIEM and LDAP),

hardened, host-based

Firewalls, security

packages.

Hardened,

3rd party

pen testing

Crypto processor chip, keys stored on chip.

Secure boot, Private keys are stored in HSM.

Disk encryption key unique per device.

Hardened OS. Least privileges for the

processes, no maintenance access by

default. TLS encrypted com.

19