1

Asi@Connect DCNDSBotnet Mitigation & Systems Evaluation Workshop

BUET, Dhaka 18-19 February 2020

Network and Security MonitoringFlorian Wilkens & Nurefsan Sertbas

2

Agenda

Network Monitoring

– Overview

– Monitoring network by NetFlow

Intrusion Detection

– Intrusion Detection System Zeek

– Honeypots

Summary

3

The big picture

Detection

ReactionRecovery

Prevention

Secure System design /

Classic IT-Security Intrusion Detection

Systems (IDS)

Firewalls

Firewalls

Honeypots

PDRR

Process

4

Network Monitoring

What do we want to know?

– who accessed

– when, where and how

– whose and which resource?

– Performance, quality

– which protocols and services?

Why do we want to know?

1. Intrusion Detection/Fault Management

2. Analysis

• System & Service monitoring for reachability, availability,..

• Resource Management (e.g., capacity planning)

• Performance Monitoring & Optimization (e.g., RTT, throughput,..)

3. Statistics for accounting & metering

• Bill customers according to usage

This Photo by Unknown Author is licensed under CC BY-SA-NC

5

Overview

Configure device to generate flow accounting records

Export the flows from the device to a collector

Receive the flows, write them to disk

Analyze the flows

Goal:

– Collection of useful information from various parts of the network

– The network can be managed and controlled using the collected information

6

Overview

Passive Collector

e.g., Unix host

Receives a copy of every packet

Requires a mirror port

Resource intensive

All

packetsFlow

records

7

Overview

On a network device

e.g., routerDevice support is needed

Some impact on performance

Flow

records

queries

A flow is defined as a set of packets

having common properties such as

common packet header fields.

8

Flow Exporting Protocols

NetFlow by CISCO

Jflow by Juniper

NetStream by Huawei

IPFIX by IETF

s-Flow by Netgear, Dell and HP

Rflow by Ericsson

9

NetFlow flow definition

Key Fields

SrcIP 10.,3.3.3

DstIP 10.2.7.2

srcPort 23

dstPort 220178

Layer 3 prot TCP-6

TOS Byte 0

Interface Ethernet

NetFlow enabled device

10

NetFlow

Can be used for several applications:

– Answers questions regarding IP traffic: who, what, where, when, and how

• Which user / department has been uploading / downloading the most?

• Which are the most commonly-used protocols on my network?

– Identification of anomalies and attacks

– More fine-grained visualization

• Presentation of collected and processed data in a user-friendly format

What we cannot achieve by NetFlow monitoring?

11

NetFlow – DoS Attack Example - I

Potential DoS Attack on Router

Estimated: 660 pkt/s 0.2112 Mbps

Typical DoS Attacks Have the Same NetFlow Flow Entries

(Same Input Interface, Destination IP, 1 Packet per flow and Bytes per packet)

12

NetFlow – DoS Attack Example - II

To show all flows to one destination leverage:

• “ router#sh ip cache flow | include <destination> “

To look for known attack signatures

– e.g. if we know of an attack using UDP port 666 (Hex 029A)

• router#show ip cache flow | inc 029A

13

Top NetFlow Analyzers & Collectors

Features

– receive flow records from exporters

– analyze these records to produce sensible information

– present results in a user-friendly way

Some examples for flow collectors

– Solarwinds NetFlow Traffic Analyzer

– PRTG Network Monitor

– Scrutinizer

– ManageEngine NetFlow Analyzer

– nProbe and ntopng

14

Tool: Scrutinizer

15

Tool: ManageEngine

16

Tool: Solarwinds NetFlow Traffic Analyzer

17

What comes next after Monitoring ?

Detection

ReactionRecovery

Prevention

Secure System

design / classic

IT-Security

Intrusion Detection

Systems (IDS)

Firewalls

Firewalls

Honeypots

PDRR

Process

Intrusion Prevention

Systems (IPS)

Monitoring alone is not sufficient!

18

Overall goal:

– Supervision of computer systems and communication infrastructures in order to detect intrusions and misuse

Why detection of attackers?

– Full protection is usually not possible!

– Security measures too expensive or with too low flexibility, e.g., not possible to build every functionality in ASICs

– Unpatched systems for compliance reasons (medical systems etc.)

– …

What can be attained with intrusion detection?

– Detection of attacks and attackers

– Detection of system misuse (includes misuse by legitimate users)

Goal of Intrusion Detection Systems – I

19

Goal of Intrusion Detection Systems – II

Using detection system only makes sense if there are consequences!

Possible goals

– Limitation of damage if (automated) response mechanisms exist

– Gain of experience to recover from attack, improve preventive measures

– Deterrence of other potential attackers (if and only if police is able to arrest them!)

Detection

ReactionRecovery

Prevention

PDRR

Process

IDS is a

fraction of

this step!

20

Operation of Intrusion Prevention/Detection Systems

Events Logging

Automatic

reaction

Terminal

Monitoring/Audit

Central IDS / SIEM

Detection Reaction?

External

alert

Records all security relevant

events of a supervised systemAutomatic analysis

of audit data

Reporting of

detected attacks

Potentially also initiating

countermeasures

21

Audit Data

Recording security relevant events

From a computer system:

Opening of files

Execution of programs

Detected access violation

Failed password verification

From a network:

Connection establishment

and release

Packets transferred from /

to specific systems/ ports

App specific events:

Events are application

specific and indicate

security relevant

activities

Input for the intrusion detection and response mechanisms

• requires integrity protection !

22

Requirements of IDSs – I

High accuracy ( ↓ 𝐹𝑃 and ↓ 𝐹𝑁 )

TP FP

TNFN

Dete

ction

Event Nature

AN

A

False alarm:

Corresponds to an

anomalous event

that is inoffensive

Harmless event that

was successfully

labeled as normal

Successful

detection

of attacks

Attacks that

were not

detected

Confusion Matrix

23

Requirements of IDSs – II

Easy to integrate into a system / network

Easy to configure & maintain

Autonomous and fault tolerant operation

Low resource requirements

Self protection, so that an IDS itself can not easily be deactivated by a

deliberate attack (to conceal subsequent attacks)

24

IDS Taxonomy

Scope

Host based

NW based

Hybrid

Architecture

Centralized

Hierarchical

Distributed

Detection Mechanism

Signature based

Policy based

Anomaly based

25

IDS Taxonomy – Scope

Scope

Host based

NW based

Hybrid

analysis of system events

analysis of communication data (IP packets)

combined analysis

26

IDS Taxonomy – Host-based

Works on information available on a system,e.g., OS-Logs, application-logs, timestamps

Can easily detect

– attacks by insiders,

• as modification of files,

• illegal access to files,

• installation of Trojans or rootkits

Problems:

– has to be installed on every system

– produces lots of information

– often no real-time-analysis but predefined time intervals

– hard to manage a large number of systems

27

Example of a Host-Monitor – Osquery (1)

High-performance database view on operating system events

– SQL tables representing abstract concepts

Fast & battle-tested

Open-source

https://osquery.io

• running processes• logged in users• password changes• USB devices• firewall exceptions• listening ports• ….

28

Example of a Host-Monitor – Osquery (2)

High-performance and low-footprint distributed host monitoring

– Query the system in an abstract way

– It runs everywhere : Independent of OS, software or hardware configuration

Host monitoring daemon

– allows to schedule queries to be executed across entire infrastructure

– takes care of aggregating query results over time and generates logs which indicate state changes in the infrastructure

Cross platform operating system instrumentation framework for

– intrusion detection,

– infrastructure reliability

– or compliance monitoringg

Only monitoring, no intrusion detection capabilities on its own !

https://osquery.io

29

Works on information provided by the network, mainly packets sniffed from NW layer

Existing systems use combination of

– signature detection,

– protocol decoding,

– statistical anomaly analysis

Can detect

– DoS with buffer overflow attacks,

– invalid packets,

– attacks on application layer,

– DDoS,

– spoofing attacks,

– port scans

Often used on network hubs to monitor a segment of the network

IDS Taxonomy – Network IDS I

30

IDS Taxonomy – Network IDS II

LAN

DMZ

Internet

Probe monitors all

incoming traffic

• High load

• High rate of false

alarms

• Measurement of any

attack attempts

Probe monitors all traffic to and from

systems in the DMZ

• Reduced amount of data (less

unsuccessful attempts)

• Can only detect attacks on these

devices, but potentially revealing

compromised LAN devices

Probe monitors LAN traffic

• Low load

• Detection of inside

attacks (e.g.,

compromised devices) Switch forwarding all

data to a monitoring

port

Central IDS/SIEM

Monitoring

Network

SIEM = Security information and event management

31

IDS Taxonomy – Architecture

Architecture

Centralized Hierarchical Distributed

Analysis Unit

Analysis Unit

IDS IDS

Analysis Unit

IDS

Central

Analysis

Unit

IDS

IDS

IDS

IDS

IDS

IDS

IDS

IDS

IDS

IDS

IDS

IDS

Collaboration [VaKa+15] How to correlate alerts in collaborative scenario ?

32

IDS Taxonomy – Detection Mechanism

Detection Mechanism

Signature-based

Policy-based

Anomaly-based

Knows what is “malicious”

Knows what is “allowed”

Knowns what is “normal”

33

Signature-based Detection

Basic idea:

– Some attack patterns can be described with sufficient detail (“attack signatures”)

– The event audit analyzed if it contains known attack signatures

Identifying attack signatures:

– Analyzing vulnerabilities

– Analyzing past attacks that have been recorded in the audit

Specifying attack signatures:

– Based on identified knowledge so-called rules describing attacks are specified

– Most IDS offer specification techniques for describing rules

Drawbacks of signature-based detection:

– Requires prior knowledge of potential attacks

– Signature database requires continuous updating

– High rate of false negatives if signature database is not up to date

34

Signature-based Detection – Example: Snort

Network IDS and intrusion prevention system

Analysis of IP packets in real time

Mainly signature based, each intrusion needs a predefined rule

Preprocessing Detection Engine Action

35

Policy-based Detection

Basic Idea

– Specify what is allowed in a network and/or what is forbidden

– Violations create alerts

– Similar to a Firewall

Drawbacks

– You can only detect what you configured / what deviates from what you have configured

– Needs expert knowledge of the system to be protected

36

Policy-based Detection – Example: Zeek (aka Bro)

Real-time network analysis framework

– Primary a network monitoring tool

– Can be used for pure traffic analysis

– Powerful IDS

Focus on

– Application-level semantic analysis

– Policy-based detection in protocols

– Tracking information over time

Intrusion prevention

– Zeek can act as dynamic and intelligent firewall

[Pa99]

Network

libpcap

Event Engine

Policy Script Interpreter

Packet stream

Filtered packet stream

Event stream

Real-time notification

Record to diskPolicy script

Event control

Tcpdump filter

37

Policy-based Detection – Example: Zeek (aka Bro)

Network

Programming

Language

Packet Processing

Standard Library

Pla

tform

Vulnerabilit

.Mgmt

Intrusion

Detection

File

Analysis

Compliance

Monitoring

Traffic

Measure-

ment

Traffic

ControlApps

Ta

p

38

text/plain

application/octet-stream

text/html

application/xml

application/x-shockwave-flash

image/jpeg

image/pngimage/gif

application/pdf

Using Zeek to understand your network

cat files.log | zeek-cut mime_type | sort | uniq -c | sort -rn

Top File Types

39

Chrome

Microsoft-CryptoAPI

Windows-Update-Agent

GoogleUpdate

Safari

FirefoxMSIE

CaptiveNetworkSupport

DropboxDesktopClient

ocspd

Using Zeek to understand your network

cat software.log | zeek-cut host name | sort | uniq |

awk -F '\t' '{print $2}' | sort | uniq -c | sort -rn

Top Software by Number of Hosts

40

Anomaly based Detection

Basic idea: Detect behavior that differs significantly from normal use

Users have certain habits in their system usage:

– Duration of usage, Login times, Amount of file system usage

– Executed programs, accessed files, ..

Assumption: “normal user behavior” can be described statistically

– Requires a learning phase / specification of normal behavior

– Most approaches require labeled data -> hard to obtain !!

Analysis:

– compares recorded events with reference profile of normal behavior

Advantage:

– An attack scenario needs not to be defined a priori

– This approach can, in principle, detect unknown attacks

𝑁1

𝑁2

03

𝑂1𝑂2

x

y

41

Automatic Anomaly Detection – System model

Generic anomaly detection system

[ET04]

Sensor

subsystem

Probe

Probe

Probe

Centr

al

pre

pro

cessin

g

Modeling

subsystem

Model

derivation

Analysis subsystem

DetectionEvents

with attacks

Events

(no attacks)

Model

Reaction?

Network

42

Anomaly based Detection – Challenges

Defining normal behavior

– Including every possible normal behavior is difficult

– Boundaries between normal and anomalous behavior is often not precise

– Anomalous observation that lies close to boundary can actually be normal or vice versa (false positive and false negatives)

Adaption of the attacker

– Dynamic behavior of anomalous behavior

– Attackers often adapt themselves to make anomalous observation appear normal (false negatives)

– Renders task of defining normal behavior more difficult

Normal behavior is not static and evolves over time

[Chan09]

43

Anomaly based Detection – Challenges II

Notion of an Anomaly

– … different for different application domains

– Applying a technique developed in one domain to another, is not straightforward

Availability of labeled data

– Obtaining accurate and representative labeled data is expensive (labeling usually done via human experts)

Data Noise

– Data often contains noise that tends to be similar to the actual anomalies and hence is difficult to distinguish and remove

Privacy

– Collecting user specific usage patterns

– Work-related or personal habits

[Chan09]

44

Anomaly based Detection – Methods I

Nearest Neighbor-based Anomaly Detection

– Assumption: Normal data instancesoccur in dense neighborhoods, while

– anomalies occur far from their closest neighbors

– Distance or similarity measure for data instances required

– Ex: kNN

[Chan09]

+ Unsupervised+ Adapting to different data type only requires defining appropriate distance measure- Performance relies on distance measure- High FPR depending on nature of data

45

Anomaly based Detection – Methods II

Classification-based Anomaly Detection

– Training + Testing phases

– Ex: Neural Networks, Bayesian Networks, Support Vector Machines

More Methods ..

– Clustering-based Anomaly Detection

– Statistical Anomaly Detection

– Information Theoretic Anomaly Detection

– Spectral Anomaly Detection

[Chan09]

+ Fast testing phase due to precompiled model- Rely on availability of accurate labels for various normal classes,which is often not possible

46

Problems of IDS – Audit Data

Amount of log data

– Significant storage capacities are required

– Automated processing of audit data

Location of audit data storage

– If stored on log server, data must be transferred to this server

– If stored on system to be supervised, the log uses significant amounts of resources of the system

Protection of audit data

– If a system gets compromised, audit data stored on it might get compromised either

Expressiveness of audit data

– Which information is relevant?

– Audits often contain a rather low percentage of useful information

47

Problems of IDS – Privacy (data protection)

User identifying data elements are logged, e.g.,

– Directly identifying elements: user IDs

– Indirectly / partly identifying elements: names of directories and subdirectories (home directory), file names, program names

– Minimally identifying elements: host type + time + action, access rights + time + action

– IDS audits may violate the privacy of users!

Collected information might be abused if not secured properly

Pseudonymous auditLog activities with user pseudonyms and ensure, that they can only be mapped to user IDs upon incident detection

48

Problems of IDS – Analysis

Limited efficiency of analysis

– Most IDS follow a centralist approach for analysis:

“Agents/sensors collect audit data and one central evaluation unit analyzes this data”

– No (partial) evaluation in agents

– Performance bottleneck

– Insufficient efficiency, especially concerning sophisticated attacks and attack variants with parallel actions

High Number of false positives

– Some publications report up to 10.000 a month

– Potential countermeasure: Alert correlation

49

Further Problems of IDS

Self-protection

– Against insider attacks

– Strategies to cope with high load

High number of false positives

– In practice, many IDS report too many false alarms (some publications report up to 10.000 per month)

– Potential countermeasure: alert correlation ( hierarchical approach)

Cooperation between multiple IDS

Distributed attacks produce many alerts that belong to same attack

– Distributed port scan

– Worm spreading

– DDoS IDS

IDS

IDS

IDS

IDSIDS

IDSIDS

50

Summary on IDS

Classification of IDS

– Host sensors vs. network sensors

– Signature-based IDS vs. Policy-based vs. Anomaly-based IDS

– Passive vs. active (e.g., honeypots) monitoring

Collaborative IDS

– Detection of large-scale and coordinated attacks

– Centralized, hierarchical, and distributed CIDS

– Alert correlation as major challenge

Evasion Attacks to bypass IDS

– Signature Evasion

– Anomaly Evasion, e.g., via covert channel attacks

51

Return to the big picture again…

Detection

ReactionRecovery

Prevention

Intrusion Detection

Systems (IDS)

Firewalls

Honeypots

PDRR

Process

“A security resource whose value lies in

being probed, attacked or compromised”

We want to get compromised!

Not a standalone security mechanism

Intrusion Prevention

Systems (IPS)

52

Honeypots

Why?

– FUN!

– No false-positives!

– Research: Malware analysis/reverse engineering

– Early warning system for new threats

53

Honeypots – Advantages & Drawbacks

Advantages

– Small(er) data sets

– Highly flexible

– No false-positives

– Potential to catch false-negatives

– Low overhead (usually)

Drawbacks

– Limited visibility

– Additional attack surface

54

Summary

IDS and Honeypots as reactive security tools to detect attacks

IDS

– Signature-based vs. policy-based vs. anomaly-based IDS

– In combination with Firewalls: IPS

– Classification according to kind of sensors deployed, level of distribution

IDS problems

– Huge amounts of data to process

– Limited accuracy and large number of false positives

– Privacy

– IDS evasion techniques

Honeypots as purely passive monitoring component

– Nearly no false positives!

55

Thanks!

Any questions?

56

Additional References

[Chan09] Chandola, Varun, Arindam Banerjee, and Vipin Kumar. 2009. “Anomaly Detection: A Survey.” ACM Computing Surveys 41 (3) (July 1): 1–58. doi:10.1145/1541880.1541882.

[Pa99] V. Paxson, “Bro: a system for detecting network intruders in real-time,” in USENIX Security Symposium, 1999, vol. 7, pp. 1–22.

[VaKa+15] E. Vasilomanolakis, S. Karuppayah, M. Muehlhaeuser, and M. Fischer, “Taxonomy and Survey of Collaborative Intrusion Detection,” ACM Computing Surveys, pp. 1–35, 2015.