Multi-Layer Intrusion Detection and Prevention System:A New Approach

A Thesis Submitted

in Partial Fulfillment of the Requirements

for the Degree of

Master of Sciencein

Cyber Security

by

Vikash Kumar Saini14/MS/029

Under the Supervision of

Dr. B.M. MehtreAssociate Professor

Center for Cyber Security

Institute for Development and Research in Banking Technology, Hyderabad

(Established by Reserve Bank of India)

COMPUTER SCIENCE AND ENGINEERING DEPARTMENTSARDAR PATEL UNIVERSITY OF POLICE, SECURITY AND CRIMINAL

JUSTICE

JODHPUR – 342304, INDIAMay, 2016

UNDERTAKING

I declare that the work presented in this thesis titled “Multi-

Layer Intrusion Detection and Prevention System: A New Ap-

proach”, submitted to the Computer Science and Engineering De-

partment, Sardar Patel University of Police, Security and Criminal

Justice, Jodhpur, for the award of the Master of Science degree in

Cyber Security, is my original work. I have not plagiarized or sub-

mitted the same work for the award of any other degree. In case

this undertaking is found incorrect, I accept that my degree may be

unconditionally withdrawn.

May, 2016

Jodhpur

(Vikash Kumar Saini)

ii

CERTIFICATE

Certified that the work contained in the thesis titled “Multi-Layer

Intrusion Detection and Prevention System: A New Approach”, by

Vikash Kumar Saini, Registration Number 14/MS/029 has been car-

ried out under my supervision and that this work has not been sub-

mitted elsewhere for a degree.

May, 2016

Dr. B.M. MehtreAssociate Professor

Center for Cyber Security,

Institute for Development and Research in

Banking Technology, Hyderabad

(Established by Reserve Bank of India)

iii

Acknowledgment

I would like to take this opportunity to express my deep sense of gratitude to all who

helped me directly or indirectly during this thesis work.

First, I would like to thank my supervisor, Associate Professor Dr. B.M. Mehtre , for

being a great mentor and the best adviser I could ever have. His advice, encouragement

and critics are source of innovative ideas, inspiration and causes behind the successful

completion of this dissertation. The confidence shown on me by him was the biggest

source of inspiration for me. It has been a privilege working with him from last five

months.

I wish to express my sincere gratitude to Dr. Bhupendra Singh , Vice Chancellorand Sh. M.L. Kumawat, (Former) Vice Chancellor, for providing me all the facilities

required for the completion of this thesis work.

I would like to express my sincere appreciation and gratitude towards faculty members

at S.P.U.P., Jodhpur, especially Mr. Arjun Choudhary & Mr. Vikas Sihag for their en-

couragement, consistent support and invaluable suggestions. I thanks to Mr. Ghanshyam

Bopche PhD. Scholar, who helped me, guided me at the time I needed the most. When-

ever I get nervous, I used to talk with my colleagues. They always tried to encourage me,

without all mentioned above, this work could not have achieved its goal.

iv

Finally, I am grateful to my father Mr. Pratap Saini , my mother Mrs. Vidya Devifor their support. It was impossible for me to complete this thesis work without their love,

blessing and encouragement.

-Vikash Kumar Saini

v

Biographical Sketch

Vikash Kumar Saini

Gurjar Wala Kuwan, Narnaul Road, Singhana.(jhunjhunu)-Raj. PIN-333516

E-Mail: sainivikash128@gmail.com, Mobile. No. +91- 998216 6843

Father’s Name : Mr. Pratap Saini

Mother’s Name : Mrs. Vidya Devi

Education

• Pursuing Master of Science in Cyber Security, Computer Science & Engineering

from S.P.U.P., Jodhpur, 2016.

• B.Tech. in Information & Technology from Gyan Vihar University, Jaipur, with

69% in 2013.

• Intermediate from Jhunjhunu Academy, Jhunjhunu, with 78% in 2006.

• High School from B.S.S.S, Pilani, with 59% in 2009.

vi

Devoted to My Loving Family for their kind affection and backing,To my companions for indicating trust in me.

vii

}Only Two Things are Infinite, The Universe and Human Stupidity,And I’m not Sure about the Former~

-Albert Einstein

}Security is Not a Product,It’s a Process~

-Bruce Schneier

viii

Synopsis

A vulnerability in a single system constitutes a hole in the entire network. Exploiting

the new vulnerability by an intruder comes under zero-day threat. Just deploying net-

work intrusion detection and prevention system does not detect zero-day threats because

ids/ips work on the signature based detection mechanism. Signatures are based on the

known attacks. There is a possibility of bypassing single-level (ids/ips) security by using

intrusions. So, an attack which enters in the system is a big setback to any network.

In this thesis, A multi-layer architecture is proposed for intrusions detection and pre-

vention. Two devices (Snort, Suricata) are used in this scheme. Multilayer architecture is

based on the signatures as well as anomaly based detection and prevention mechanisms

and also a decision-making process is implemented in the architecture. Intrusions and

data flood attacks are detected and blocked by the proposed design. On the basis of test

results, it is clear that proposed architecture gives better performance compared to indi-

vidual (single) unit of ids/ips. It also collects dropped packets for analysis. This can be

used for prediction and prevention of new attacks. Thus, the proposed architecture gives

enhanced security to the network.

ix

Contents

Acknowledgment iv

Biographical Sketch vi

Synopsis ix

1 Introduction 11.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2.1 Network Protocols . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2.2 Different Forms of Network . . . . . . . . . . . . . . . . . . . . 2

1.2.3 The ISO/OSI Reference Model . . . . . . . . . . . . . . . . . . . 3

1.3 Network Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.3.1 Types of Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.4 Network Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.4.1 Various security threats . . . . . . . . . . . . . . . . . . . . . . . 5

1.4.2 Network Security Components . . . . . . . . . . . . . . . . . . . 6

1.4.3 Network Security Level’s . . . . . . . . . . . . . . . . . . . . . . 6

1.5 Organization of Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

x

2 Existing Tools and Techniques for Securing Network 102.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.2 Literature Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.3 Intrusion detection system(IDS) . . . . . . . . . . . . . . . . . . . . . . 11

2.3.1 Types of IDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.4 Intrusion Prevention System(IPS) . . . . . . . . . . . . . . . . . . . . . 13

2.4.1 Types of IPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.5 Working Mechanism of Network Intrusion Detection and Prevention System 14

2.5.1 Signature Based IDS/IPS . . . . . . . . . . . . . . . . . . . . . . 15

2.5.2 Anomaly Base IDS/IPS . . . . . . . . . . . . . . . . . . . . . . 15

2.6 Technology used in IDS/IPS . . . . . . . . . . . . . . . . . . . . . . . . 15

2.6.1 Working Mechanism of IDS/IPS . . . . . . . . . . . . . . . . . . 15

2.7 Tools in IDS/IPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.7.1 SNORT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.7.2 SURICATA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.8 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3 Multi-Layer Intrusion Detection and Prevention System : A New Approach 273.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.3 Proposed Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.3.1 What is Multilevel IDS/IPS? . . . . . . . . . . . . . . . . . . . . 29

3.3.2 Meaning of Proactive Approach in field of Network Security . . . 30

3.3.3 Working of MultiLevel IDS/IPS System . . . . . . . . . . . . . . 30

3.3.4 Analysis of Dropped Packet . . . . . . . . . . . . . . . . . . . . 34

3.3.5 Collecting Valuable Information from Dropped Packet . . . . . . 35

3.3.6 Comparison Between Snort and Suricata Results . . . . . . . . . 35

3.3.7 Various Cases of Decision-Making Machine . . . . . . . . . . . . 37

4 Experimental Setup and Results 394.1 Configurations for Experimental Performance . . . . . . . . . . . . . . . 39

4.1.1 Tools Configurations . . . . . . . . . . . . . . . . . . . . . . . . 39

xi

4.1.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.2 Results of Single-Layer Intrusion Detection System . . . . . . . . . . . . 44

4.2.1 Processing in Snort Machine . . . . . . . . . . . . . . . . . . . . 45

5 Conclusion and Future Work 51

6 Author’s Publications 52

References 53

xii

List of Figures

1 Architecture of Router . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2 Firewall Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3 Deployment of NIDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4 Deployment of HIDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

5 Process of Detection Mechanism . . . . . . . . . . . . . . . . . . . . . 16

6 Alert Generated Console . . . . . . . . . . . . . . . . . . . . . . . . . 17

7 Detailed Structure of Signature . . . . . . . . . . . . . . . . . . . . . . 21

8 MultiLevel IDS/IPS Architecture. . . . . . . . . . . . . . . . . . . . . 29

9 Replication Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

10 Normal Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

11 Data Flood Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

12 Installed Snort on Ubuntu Machine . . . . . . . . . . . . . . . . . . . 42

13 IP address of protecting machine . . . . . . . . . . . . . . . . . . . . . 45

14 Different types of Rules Saved in Machine . . . . . . . . . . . . . . . . 46

15 User Define Rule For Demo Purpose . . . . . . . . . . . . . . . . . . . 47

16 SYN Flood Record in Attacker Machine . . . . . . . . . . . . . . . . . 47

17 Alert Generating in Live Mode . . . . . . . . . . . . . . . . . . . . . . 49

xiv

18 File Generated when Malicious Activity Detected . . . . . . . . . . . . 49

19 Data in Dropped File . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

xv

List of Tables

1 ISO/OSI Reference Model . . . . . . . . . . . . . . . . . . . . . . . . 3

2 NIDS Output Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3 Structure of Snort Rule . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4 Distinctive Category of Signatures . . . . . . . . . . . . . . . . . . . . 22

5 Decision Making Table . . . . . . . . . . . . . . . . . . . . . . . . . . 34

6 Attacks on Various types of Packet . . . . . . . . . . . . . . . . . . . . 35

7 Valuable Information Relate to Packet . . . . . . . . . . . . . . . . . . 36

8 Snort vs Suricata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

9 Configuration of Host Machine . . . . . . . . . . . . . . . . . . . . . . 40

10 Snort Pre-Requisites File . . . . . . . . . . . . . . . . . . . . . . . . . 41

xvi

Chapter 1

Introduction

1.1 Overview

“Multi-Level Intrusion Detection and Prevention System”. Scheme have analyzed and all

the desired result collected from the work. The new architecture is proposed for giving

enhanced security to the network. The work is explained in this thesis work in coming

parts.

Securing a network is a big and important task. Because the network is the best source

for an intruder to make the cyber crime. Most of the time we fail to prevent the zero-day

threat because of no prior information for that particular attack. The proposed architecture

improve the graph level zero-day threat prevention and also detect and prevent malicious

activity on the network. In the proposed design a proactive approach is dependent on the

analysis of dropped packet.

The single-level intrusion detection and prevention is experimentally performed for

verification the performance of the single tool. Malicious activity is detected and pre-

vented by single level ids/ips but not for all intrusions. A single level device not capable

of preventing all types of attacks. So for giving strength to the network, the proposed

1

design works on two tools (Snort, Suricata). Multilevel security inspects the data and

prevents intrusions.

The intelligence of a machine is used for making intrusion on another intelligence/non-

intelligence machine. The purpose of doing this is either intentionally, or it may be the

business purpose, the scenario comes under cyber crime. The mechanism of Cyber Secu-

rity is designed in such a way that protection of Network, Program, and data from attacks

and an intruder, with the help of various technologies and processes. Various elements of

cyber security are given below

• Securing Network

• Securing Information

• Securing Application

1.2 Networking

Communication between two or more devices for sharing data or using various services

is either with the internet or without internet is called networking. For making communi-

cation between devices there are so many factors play’s role, explaining below

1.2.1 Network Protocols

when communication takes place this is the very basic step, for making the connection

there must a language required and in the cyber world the communication language is

known as Network Protocol.

e.g. TCP/IP mostly found on the internet because this protocol is widely used for com-

munication.

1.2.2 Different Forms of Network

The network deployment depends on the requirement; sometimes there will be a broad

requirement of the system or sometimes small, so according to that network can be deploy.

2

The networking shown in these forms which are given below 1.

• LAN: Services uses for small number of people located.

• WAN: Across a large geographical area.

• MAN: Over radio transreceivers2.

1.2.3 The ISO/OSI Reference Model

Table 1: ISO/OSI Reference Model

Top Most Layer: ApplicationPresentation

SessionTransport

Internet Layer: NetworkData Link

Bottom Layer: Physical

In the Table [1] the seven layer of OSI3 model start their role from the originating of

data from source to destintion till the data received by end user. Open Systems Intercon-

nect(OSI) model has seven layers, control is gone starting with one layer then onto the

next, beginning at the application layer in one station, and continuing to the base layer i.e.

physical layer. The purpose of the OSI reference model is the digital communication.

1.3 Network Attacks

An intrusion on the network infrstructure is called network attack[3]. A system assault

can be characterized as any strategy, process, or means used to malignantly endeavor to1Within the network, there are so many options available for using any topology. According to the

requirement or security point of view, the topologies are uses. e.g. Star, Bus, Ring, Hybrid, etc.2These network services are deployed or used according to the requirement, sometime bigger or some-

time less requirement.3OSI was officially adopted as an international standard by the International Organization of Standards

(ISO).

3

trade off system security. Intrusion collects the information by analyzing the network and

exploit the vulnerability or existing open ports[11].

1.3.1 Types of Attacks

• Passive Attack: Scanned for open ports, vulnerabilities, looks for clear text pass-

word, monitors unencrypted traffic and sensitive information. Passive attacks in-

clude monitoring of unprotected communication.

• Active Attack: Active attack is network exploit intruder attempts to committ changes

to data.

– DDoS4

– DoS

– Message modification attack

– Session reply attack

• Distributed Attack: Trojan horse or back-door program are the components of

distributed attack.

• Insider Attack: Attack is commenting by someone from the inside the organiza-

tion, such as employee.

• Close-in Attack: Intruder trying to get physically access to network components

or data.

• Phishing Attack: Phishing attack intruder creates a fake web site looks like as

original. In Phishing attack intruder try to read information about login credentials,

account information, email identity or other communication channels.

• Hijack Attack: Intruder get access between me and another’s session and discon-

nects the another from the communication.4There are a few sorts of DDoS assaults; These system assaults are developing all the more capable

consistently, and some send more than 100 Gbps at the crest.

4

• Spoof Attack: Source address is spoofed by intruder.5

• Buffer overflow: Intruder sends large data to an system than is expected.

• Exploit Attack: Intruder know the hole in system so by developing code for hole

exploiting data can be thieft.

• Password Attack: Crack the passwords stored in a network account database.

1.4 Network Security

For securing the network there are many techniques available which may be in the form

of hardware or the form of software. But most of the time all the existing techniques are

unable to detect the new attack and network get hacked. The are so many technologies

are uses for detecting or preventing the unwanted or malicious activity on the network

or block the sophisticated threats.6 But most of the time new attacks are not detected or

prevented by available security mechanism.

1.4.1 Various security threats

• Viruses, Worms and Trojan Horses

• Identity Theft

• DDoS

• DoS

• Zero-Day Threats

• Data Interception & Theft.

5The spoofing attack can be possible in these forms, IP Address Spoofing Attacks, ARP Spoofing At-tacks, DNS Server Spoofing Attacks

6The threat is a piece of program which is developed for making the cyber crime. The malicious activitycaused by an intruder and this activity are detected by the software/hardware which is deployed in thenetwork. The security mechanism can be different for every network system.

5

1.4.2 Network Security Components

The network security is provided in the different way and by the following components

which are given below.

• Anti Viruses

• Firewall

• IDS/IPS

• Virtual Private Network(VPN’s)7

1.4.3 Network Security Level’s

Securities at any network is the very first task. Now a days intruders are always trying to

make intrusions on the network. So intrusion detection can be possible in different levels

from basic to higher levels. The levels are explained below.

Basic Level Securities at Network

If any data wants to enter in the network there is identity verification on every data is

processed by security mechanism. The security structure is defined according to the se-

curity policies of the network. The security policies are the various rules which are set

in the machine according to the required data. And also the combination of equipment,

means which equipment is put at the initial level of network or which one afterward. e.g.

Gateway Router has putted at the initial level then after firewall. Some basic securities

are given, and type of securities they do is also explain.

Security by Router

• What is Router?

Device which connects multiple devices together by a network, either wireless or

wired andor it is layer 3 network gateway. Router contains CPU (processor), mem-

ory and I/O interfaces.7VPN creates encrypted connection between private networks. VPN relies on IPsec or SSL connection.

6

• Architecture of Router

In the Figure [1] the Architecture of router 8 and various components of router and

Figure 1: Architecture of Router

their processing is given.

Putting router at initial stage of the network entry have advantages and the func-

tionality are given below.

• Functionality and Security by Router

– Authentication

– SSID

– Mac Address filtering

– Disable SSID Broadcast9

– Assigns static IP Address to devices

All the above securities are provided by router and these are the basic thing for

security point of view. If any malicious data comes under any of above category

then it is blocked by router and unwanted traffic will remove.

8Router can be of wireless or wired and the functionality of both the router is same in both the cases. Indifferent types of security some more feature can be vary, its depend on the company of router.

9At the point when a remote gadget looks the region for remote systems it will recognize the SSID.Disabling the SSID Broadcast is one method for securing the remote system.

7

Security by Firewall

• What is Firewall?

A designed software or hardware which work on the basis of defined policy ac-

cording to the requirement of data. The policies are defined for the protection from

unwanted or untrusted data 10.

• Deployment of Firewall

Figure 2: Firewall Deployment

• Functionality and Security by firewall

– Excellent auditor

– Use to restrict the specific services

– Excellent at alerting.

Security by Antiviruses A designed software which detect/prevent and remove soft-

ware viruses, and other malicious software like Trojan’s, Worms, AdWare etc.

• Remove or prevent spyware and adware

• Uses as safeguard a computer from malware, trojan horses and computer worms.10Firewall can be of two types hardware and software firewall. Firewall cannot stop network attacks if

any relate policy not define

8

Higher Level Securities at Network

Higher level security at any network can provided by deep packet inspection of all incom-

ing and outgoing data[5]. IDS/IPS works as deep packet inspection for every data. Means

IDS/IPS monitor every packet which wants to enter in the network. IDS/IPS support

in-depth security policy and can be used to detect a wide range of threats.

• Software Vulnerability Exploits

• DoS or DDoS

• Malicious Activity

• Buffer Overflow

• Password Cracking

• Protocols Attacks

Every IDS/IPS system works on the principle of signature, which are defined in the ma-

chine. The signatures are based on the previously happened attacks or we can also add

more signature according the requirement11. For increasing the intelligence of machine,

we have added anomaly based detection mechanism. The anomaly-based is explained by

graphical view in the proposed architecture figures [10],[11].

1.5 Organization of Thesis

The remaining part of the thesis is unified as follows:

Chapter 1 Highlighted the introduction, summary of cyber security, Network security and

work carried out by various researchers in the field of Network Security.

Chapter 2 Describes about problem statement, literature survey and existing tools and

technique.

Chapter 3 Proposed a new Architecture in the network security field.

Chapter 4 Configuration and results discussing in this section.

Chapter 5 Conclusion and the Future Work is discussing in this section.11Some time the user defined signature are not able to give good result. So the machine gives false

positive and false negative generating at high rate by the ids/ips machine.

9

Chapter 2

Existing Tools and Techniques for

Securing Network

2.1 Overview

For securing the network, there are many tools and techniques available; they are in the

form of hardware or the kind of software. But most of the time all the existing methods are

unable to detect the new attacks and system get hacked. The network security architecture

deployment is vary from organization to organization. The security policies are different

for every company. There are various techniques which are followed by a company for

detect and prevent the intruders activity.

There are different types of ids/ips tools are available in the security field, and the

deployment of these tools help to prevent malicious activity.

10

2.2 Literature Survey

The idea of deep packet analysis had given by “D. E. Denning” in “1986”[5]. The idea for

inspecting data deeply is used now for analysing all the incoming and outgoing data on

the network [2]. The purpose of doing this for securing the network from new attacks and

preventing malicious activity on the network [11]. The intrusion detection and prevention

system is defined in two ways; Signature based detection and prevention mechanism and

Anomaly based detection and prevention mechanism. Both the mechanisms have used

for network level ids(NIDS) and host level ids(HIDS). The signature defined by previous

attacks. So the ids/ips work by past attacks, new attacks are not detectable by current

ids/ips system. Because of the signature-based detection mechanism, ids/ips generate a

large number of alert for false positive cases, which is overhead for the analyst. Some-

times malicious data successed to enter into the network. Therefore, the case comes under

false negative. The false negative is the big setback for any network. So for providing

more securities and reducing the false negative rate the new design has proposed in this

paper.

Network attacks are the best medium for intruder to committing a large crime [7]. Be-

cause if the entire network is compromised by the intruder then it can be used as zombie

network and another crime can be succeed. In the paper different security mechanism

had discussed for preventing network attacks [19]. The network attacks and their secu-

rity mechanism have discussed, how a network can be secured and how intruders can be

prevented to enter in the network [10], [11].

2.3 Intrusion detection system(IDS)

Intrusion is an unauthorized attempt it may or may not successful [9]. Intrusion detection

system has used for detecting malicious activity. A system is used to detect unauthorized

intrusions onto the network. IDS system work on the principle of Signature and Anomaly

based intrusion detection mechanisms by these two mechanisms, the malicious activity

are blocked.

11

2.3.1 Types of IDS

IDS can be deployed at two places for securing the system. Which are given below.

Netwok level IDS(NIDS)

The entire network as the monitoring scope. The network traffic is monitored to detect

intrusions [6]. NIDS detect data that may be harmful to the system. NIDS capture and

inspect every packet that is destined to the network whether it is permitted or not. NIDS

can run in many ways. It can be run on a separate machine and inspect the whole network,

or it can be run on a computer and investigate itself. e.g. It can watch itself or entire

network, if somebody is attempting an SYN Flood or a TCP Port scanning.

• Architecture of NIDS

The architecture of NIDS is given in Figure [3].

Figure 3: Deployment of NIDS

12

• Network Load Balancer

The network data is capture by NIDS and distribute to the whole network. It is a

software that runs from the sensor.

• Alert Notifier1

It contacts to the network security team responsible for handling if any incidents

occur on the network on the behalf of organization security policy.

• Database

The database maintained for capturing dropped packets, and these packets are uses

for prevention of future attacks on the network.

Host Level IDS(HIDS)

Inspection of the system within the organization network is called host-based intrusion

detection system. HIDS2 is detecting malicious activity within the single machine.

• Architecture of HIDS

The architecture of HIDS is given in Figure [4].

2.4 Intrusion Prevention System(IPS)

IDS are passive components they only detect and reports blindly. IPS has prevented the

attacks; IPS is also working like IDS in two forms.

2.4.1 Types of IPS

Network level Intrusion Prevention System(NIPS)

The bundle work on the network which detects and prevents is preferred for security

perspective, this prevention of attacks on the network level comes under NIPS. The entire

1The network analyst defines alert, and action specified in the machine so according to that machine willperform action.

2Host level ids work for a single system only, monitoring the data which arrive on the host machine.Within a single network, more than one HIDS can not be deployed

13

Figure 4: Deployment of HIDS

networks data are monitoring and intrusions prevented by NIPS.

Host level Intrusion Prevention System(HIPS)

Most HIPS prevents occurs by agent resides on the host machine. If any malicious code

is running on host machine, then it is detected and prevent takes place by HIPS.

2.5 Working Mechanism of Network Intrusion Detection

and Prevention System

Network Intrusion/Prevention System work by two mechanisms, either the malicious ac-

tivity made by the intruder is unknown by IDS/IPS machine, or device already knows it.

The principle of IDS/IPS is explained below.

14

2.5.1 Signature Based IDS/IPS

The various signature category is given on Table [4]. The signature type is defined by dif-

ferent types of attacks which have already exploited by the intruder and for that particular

attacks a signature is defined for preventing. The signature list updated mostly if any new

malicious activity is happening on the network. It is because an intruder does not repeat

the same activity.

2.5.2 Anomaly Base IDS/IPS

The anomaly-based detection cover the sensing capability of the machine. The machine

is not like human thinking, but we can define code in the machine. The graph has shown

in this thesis for detecting and preventing anomaly-based attacks [10][11]. e.g. Any DoS

or DDoS is happening on the network then bandwidth is field more than the threshold

value then IDS sense this activity because it is not the part of daily network activity and a

machine will block the attacks.

2.6 Technology used in IDS/IPS

IDS/IPS work’s only on two major principle’s one is signature based detection/prevention

and another is anomaly based detection /prevention system. There are so many IDS/IPS

tools available; some are open source, and some of them are licensed version. The func-

tionality of devices is varied it depends on the signature defined and sensing capability of

the machine.

e.g. Tools which are open sourced.

• SNORT

• SURICATA

2.6.1 Working Mechanism of IDS/IPS

Working of ids/ips is on the basis of two mechanisms are explained below.

15

Signature based IDS/IPS Detection Mechanism

This is the simple tasks for IDS to detect malicious activity if this is the defined in the

signatures. For detecting the malicious activity, there is a program stored in the machine.

The program has developed in such a way that if any intrusion made on the network by

the intruder, then it will detect and generate an alert. Various signature is defined by past

attacks. For a single type of attack, a single rule is developed. If the intruder changes the

malicious code, then it comes under a new kind of attacks. Update of the signature has

done in two phases either it is update by the developer team or it can be manually update.

We can update the signatures according to the requirement, any rules can be add or can

be removed.

An example of rules are shown below.

• Alert TCP $EXTERNAL NET any Ô $HOME NET 139 (msg:“NETBIOS NT

NULL session”: flags:A++:

content: “x 00 00 00 00 57 00 69 00 6E 00 64 00 6F 00 77 00 73 00 20 00 4E 00 54

00 20 31 00 33 00 38 00 31 x” ; reference: bugtraq, 1163 ; refernce: cve, cve-2000-

0347; reference : arachnids, 204 ; classtype: attempted-recon; sid:530 ; rev : 5;)

How a signature is devleoped and how port number and ip address are stored this

detailed explaination is on “Detailed structure of signature”

Process of Detection Mechanism The process of intrusion detection mechanism is

given in Figure [5]. And the alert is generated if intrusions has arrived. The alert is then

Figure 5: Process of Detection Mechanism

16

Figure 6: Alert Generated Console

process to the network security engineer, by trigger or by alert. Some form of secondary

processing on the information.

Pattern Matching In this process the matching of individual signature against of indi-

vidual packets. Let suppose a web server attack is commenting on the network then the

pattern is matched only under the category of web server area, which will reduce the time

complexity of the machine.

StatefulPattern Matching In this process the matching of a pattern on the entire ses-

sion not on a single packet. Stateful pattern matching to reassemble communication ses-

sion like TCP (Transmission Control Protocol) session and IP (Internet Protocol) frag-

mentation. State of each communication transaction is tracking and reassembling of

packets stream is on the receiving in the same manner.

Listening Alert When a malicious activity is happening on the network, then ids/ips

will inspect the activity and according to rule, machine will take action.

Challenges to Signature Based IDS Signature based ids is a very useful tool, as also

face some problems. Primary activity detects very well while sometimes not serving well.

Factors such as speed, network architecture, and signature are added regularly. Some ex-

amples have given below which are the problem in signature based IDS.

False Positive Data is okay, generate the alert. Some companies deployed IDS/IPS for

17

detecting malicious activity. But after few weeks, they removed because it generates hun-

dreds of alert per day, which is the unnecessary trigger. The reason of false positive case.

TCP/IP has designed very open and very flexible suit of protocol. Modifications are made

to the protocols daily basis, and Companies regularly find new users for existing protocol.

So for detecting harmful activity is a tough task.

Solution: False positive is overhead for the network analyst, so the architecture and their

functionality has proposed in such a way that it has decreased false positive case. The

functionality has discussed in this thesis by using the different location for saving dropped

packets. The complete discussion is given in topic [3.3.4]

False Negative Data is not okay(bad data) entered in the network. IDS/IPS is not capable

of detecting all kind of attacks. Therefore, false negative(attack successful) data entered

into the network which is the big setback for any network. When ids fail to generate alert

when malicious activity occurs. Because attacker knows the IDS/IPS has deployed and

various rules are defined, so they develop code apart to specified standards.

Solution: Analysis is necessary: The database has maintained by machine for collecting

dropped packets. These dropped packets were received from the single and multi-layer

ids/ips machine. The analysis has made for getting all the valuable information against

intruder and new attacks and data flood attacks prevented.

Anomaly Based IDS/IPS

Anomaly based ids work on the statistics of data on the daily basis record. Let suppose in

any company there is uploading, and downloading working is going on, and daily records

have counted by network monitoring tool (Wire-Shark). According to daily records, the

value is assigned in the graph. If any data flood is happening on the network, the machine

will treat the attack as anomaly detection. Therefore, an alert will generate. The graph

has discussed in figure [10], [11].

18

2.7 Tools in IDS/IPS

security on the network by ids/ips is comes under higher level security. Using term higher

level security has an important meaning because ids/ips inspect deep packet analysis of

coming data in the network. Other than ids/ips security component provides security

according the defined policy. Different types of security tools are available in ids/ips

some gives better result and some are not. Different types of tools are explained here

which are used in the proposed architecture Figure [8].

2.7.1 SNORT

• Snort is Network Intrusion Detection and Prevention System (NIDS)3

• Packet sniffer

• Inspect the traffic in real time

• Work on basis of signatures, can also work on the basis of anomaly detection.

Configuration modes of Snort

Configuration of Snort can be possibile in the three modes. Which are given below

• Sniffer mode

Reads the packet off network and displays on the screen (console). In sniffer mode

various types of process can be done which are discuss below

– We can print TCP/IP packet to the screen (./snort -v)

– Can show IP and TCP/UDP/ICMP header (./snort -vd)

– Displays packets and the headers (./snort -vde) or (./snort -d -v -e)

• Packets Logger Mode

Logs are generate and stored in disk. the path of disk is automatically saved in snort

or it can be change according to the requirement.3Network level intrusion detection inspect the entire network’s data. Deep inspection detection mecha-

nism provides better security to the network.

19

– ./snort -dev -1 ./log

• Network level ids(NIDS)

Detect, prevent and analysis on network traffis.

– ./snort -d -h “ipaddress” -1 ./log -c snort.conf

– NIDS mode outputs

Table 2: NIDS Output Mode

Choice Explaination-A Fast Fast Alert Mode-A Full Full Alert Mode

-A None Alerts are turn off-A Console Send alerts to screen(console) mode-A Unsock Alerts are sends to UNIX socket

– High performance configuration

We need to use barnyard2 it allows snort to log alerts in a binary form. If we

want tcpdump then logging with the fast Output mechanism. e.g. ./snort -b -A

fast -c snort.conf

– Alert order:

We can change default Pass rules; Drop rules, Alert, Log rules.

– Working principle of snort

Snort detect and prevent intrusions on the basis of signatures. Signatures are

used to verify all the parts of a data packets. Snort 1.x version can analyze

transport and network layer but in Snort 2.x version it is also added that it can

analyze application layer protocols.

Structure and Development of a Signature

Signatures are the collection of instruction which written for the detection and prevention

of malicious activity.

Snort signature defined in two logical parts, which are given in table [3].

20

Table 3: Structure of Snort Rule

Rule Header Rule Option

What action rule takes this information stored in rule header. Additionally contains crite-

ria for coordinating a standard against packets. Alternative part include an instant message

and tell which part of the packets ought to be utilized to create the instant message.

E.g. Alert ICMP any any Ô any any ( msg: “ping with TTL=100”;)

Figure 7: Detailed Structure of Signature

• Protocols:

IP, ICMP, TCP, UDP

• Directions:

Ô ,←, <>

21

Categories of Snort Rules

Table 4: Distinctive Category of Signatures

S.NO Category Various forms Action

1 Appdetect.

rules

Network activity Control parts of an applica-

tion carry on

2 Blacklist.

rules

Public list of malicious

URLs, Talos virus

sandboxes

Contains DNS, URI, USER-

AGENT and IP address

chooses that have been

set out to be pointers of

intruder’s development

3 Browser.

rules

Chrome, Firefox, ie,

webkit, plug-in

Detection for vulnerabilities

in browsers

4 Content Replace, Deleted Check for content replace and

detection

5 Exploit kit Searches for vulnerability

against programming in a

software

6 File Executable, flash, im-

age, identify, multime-

dia, office, pdf

Vulnerabilities are checked

that are found or conveyed

through executable records

7 Indicator Compromise, obfusca-

tion, shell code

Contains rules for positively,

obfuscated and identification

markers of shell code in traf-

fic

8 Malware Backdoor, cnc, tools Contains rules for the iden-

tification of activity bound

to known listening command

channels

Continued on next page

22

Table 4 – Continued from previous page

S.NO Category Various forms Action9 OS Linux, Solaris, win-

dows

Looking for vulnerabilities in

different OSes

10 Policy Multimedia, social,

spam

Detect potential violations of

different policies

11 Protocol ftp, icmp, imap, pop,

services, VoIP

Show the nearness of given

conventions activity or vul-

nerabilities in the system

12 Pua adwarep2p toolbars Potentially Unwanted Appli-

cations (pua) that deal with

unused application which in-

stalled as background process

13 Server Apache, iis, mssql,

mysql, oracle, webapp,

mail

Decides that identify vulner-

abilities in or assaults against

servers

2.7.2 SURICATA

• Network Intrusion Detection and Prevention security Monitoring Engine [12].

• Open Source

Engine

• NIDS & NIPS engine

• Network security monitoring(NSM) engine

• Traffic recording using PCAP logger

• Offline analysis of PCAP files

23

• Processing of PCAP files in UNIX socket mode.

O.S. Support

• Windows, FreeBSD, OpenBSD, MacOSX, Linux

Protocol passer

Support for packet decoding given below

• UDP, ICMP, IPV4, IPV6, TCP

If any of the above given protocols have arrived in encoding formate, then protocols

are decode by Suricata machine.

Detection Engine

• Protocol keywords

• Rule Profiling

• File Matching

– File Magic & Name and Extension & Size

Outputs

• All JSON alert and event output

• Request logging for Http

• Alert fast log

• SysLogÝ alert to sys log

• Flow logging

Output from Suricata machine can be captured in various forms, which have given

above. The capturing of the outputs from Suricata machine can be visualized at live

mode or can be kept in the database.

24

Packet WorkShop

• High Performance capture

– AF PACKET : Read Raw Socket & Read Load Balancing

– PF RING : Fast packet capturing, analysis and investigation

– NETMAP : Netmap is an extremely productive structure for line-rate for raw

packets input/output from client space,

• Standard capture

– PCAP : Is instrument in libpcap library.

– NFLOG (Net Filter Integration)

• IPS mode

– Netfilter based on linux

– NETMAP

• Multi Threading:-

– Fully Configurable threading from single thread to dozens of threads

– Optional CPU Affinity setting

– Optional lock profilling.

Working mechanisms, all the configurations and requirements of Snort and Suricata

has discussed in this chapter. The working process and outcomes is discussing in the next

chapter.

2.8 Problem Statement

Traditional security mechanisms use single IDS/IPS device. As a result, many intruders

succeed to bypass the ids/ips security and also new attacks are not detectable by used

methodology because ids/ips work on the signature based detection mechanism. So in

25

some cases, False Negative (attack successful) data has entered into the network. Also, a

drawback of ids/ips is generating more number of unnecessary alert for some data these

cases comes under the false positive case. The available tools for securing the network

are Snort & Suricata. Individual deployment of any tool has some problem but the combi-

nation of both the tools gives strength to the network. The explanation is given incoming

parts.

26

Chapter 3

Multi-Layer Intrusion Detection and

Prevention System : A New

Approach

3.1 Abstract

Traditional security mechanisms use the single unit of IDS/IPS device. In some cases,

intrusions are succeeded to enter into the network. These successful attacks are a big set-

back for any network. In this paper, We propose a Multi-Layer Architecture for intrusions

detection and prevention. The design is based on the two ids/ips tools Snort and Suricata.

It is found that the parallel combination of Snort and Suricata tools gives better results

compared to individual performance. Our test results show that the false negative (attack

Successful) cases are blocked and failed to enter into the network, and this scheme also

reduces the false positive cases. This is achieved by using dropped packets analysis in

the proposed design. Our results show that deployment of multilayer ids/ips system gives

enhanced security.

27

• Keywords: Snort, Suricata, Wire-shark, MySQL Router, Firewall.

3.2 Introduction

Here in this paper, the intelligence of the machine is increased by using more number

of tools in the ids/ips security mechanism. Because the combination of two or more

tools gives power to the detection mechanism. So intruder unable to bypass the security

system. The proactive approach of proposed architecture is based on the dropped packet;

these dropped packets are the collected from multi-level architecture. So by making the

analysis of these dropped packets will help to prevent the new attack. And also remove

the false positive rate which are dropped unnecessarily . When the data is flooded on the

network, the security mechanism can’t handle that flood and threshold value goes beyond

the defined value.

The motive of proposed architecture Figure [8] is preventing false negative cases 1 and

new attack [16]. The multilevel security will inspect the downloading and uploading data

on two different machines at the same point in time. The multilevel inspection of a single

data increases the threat detection and prevention probability.

There are so many types of attacks are possible on the network. The most useable

protocol is TCP/IP protocol for every service over the internet, so attackers try to find

vulnerability over TCP/IP protocol [14], [7].

3.3 Proposed Architecture

The given architecture Figure [8] is for increasing the security strength of any network.

That will help to detect the various types of attacks and prevent them. The proposed

architecture has defined in two forms to give the strength to a network. The first part is

Multilevel Intrusion Detection and Prevention system, and the another part is the Analysis

of dropped packet. Multilevel IDS/IPS system has a unique functionality that the data

wants to enter the network will go through multi-tools inspection, the tools are monitoring

1At the time of DoS/DDoS attack is happening on the network, the data are flooded on the networkwith a high rate, and the rate may be of 100-300GbPS or more. The whole system goes down, and all theservices of network failed. So by this, the intention of an attacker will be completed.

28

Figure 8: MultiLevel IDS/IPS Architecture.

same data at the different machine at a single point in time. Therefore, the system is

known as multilevel IDS/IPS system.

The another part is an analysis of dropped packet; these dropped packets are generated

from multilevel IDS/IPS because any intrusion made by an intruder comes under dropped

packets. The analysis of these packets falls within the proactive approach in the field of

network security. Because these dropped packet will tell the next step of an attacker. The

proposed architecture is deployed on the network level intrusion detection and prevention

system (NIDS). 2

3.3.1 What is Multilevel IDS/IPS?

Using two or more IDS/IPS tools (Snort, Suricata) for network security and parallel in-

spect the same data at a same point of time will increase the chances of detecting malicious

data, because if using a single tool it may bypass the security but using two or more tools

its difficult to bypass both the tool’s security. The default mechanism of detection/pre-

vention of both the tools are signature based but in the proposed architecture one more

functionality has added anomaly based detection and prevention mechanism. So both the

mechanism are used in the multilevel IDS/IPS system. The signature is predefined by the

developer team Table [4] and the regular update is generated, according to the requirement

2The multi-layer intrusion detection and prevention system has deployed on network level intrusiondetection/prevention(nids).

29

at the user level.

3.3.2 Meaning of Proactive Approach in field of Network Security

By the multilevel ids/ips system, the malicious activity and flood attack is on the network

is prevented. If any malicious activity is blocked by ids/ips, then it generates a file which

comes under dropped packet. These dropped packet used for analyzing them. Because

these packets are attempted intrusion by intruder, and used tools block them. All the

valuable information from these files are collected for securing the network. The valuable

information is used for preventing the zero-day threat/vulnerability. And by this, we can

protect the network from the zero-day threat. What information collected from dropped

packet is shown in Table [7].

3.3.3 Working of MultiLevel IDS/IPS System

The multilevel IDS/IPS is explained in the three phases.

Phase 1 : Replication Phase

Replication machine Figure [9] works only for duplication of packets there is no packet

inspection at this phase. Blindly replication processes and sends them to the next phase

on two different machines. The replication of packet can be possible by any available

tools, or this replication process can be defined manually by any script.

Figure 9: Replication Machine

30

Phase 2 :Inspection phase

The inspection of data is processed on phase 2 of given architecture. The examination of

data happens and also on the anomaly based detection mechanism. The anomaly detection

graphs have shown in figures [10],[11] in this thesis work.

Signature Based Detection and Prevention The signature [13] based detection is de-

fined by previously happened attacks. If any attack is repeating by the attackers, then it

will detect by the machine. Thats why a particular signature is used for particular types

of attacks. And the machine will block that attack and generate an alert.

Anomaly based detection and prevention This technique is based on the intelligence

of machine, but the machine is not like human thinking. Therefore here in this proposal,

some intelligence has given to the machine in the form of the graphs. It was found that

according to given technique machine work like human mind in the form of sensing ca-

pability of data only and prevent the DoS and DDoS attack if exploiting on the network.

The graph has developed by daily downloading and uploading data; the average has taken,

and here it is generalize based on an example. The machine is work on the behalf of the

average graph and senses the data which have defined in the algorithm. If other than

the algorithm is given data are downloading in the network, the machine will take action

according to the given instruction.

• Normal Graph

The value in the graph has assumed of a company for explaining the concept; The

expected data is a software company and according to the work of company the up-

loading data and downloading data are defined here. In the Figure [10] the average

is found that per second 0.01Gb data uploading and 0.005Gb data is downloading.

The average has found by network monitoring tool (Wire-Shark3). In wire-shark

every packet has captured and also count the number of packets arrived in the sys-

tem. Some more functionality wire-shark having but here we use only the packet

counting features.3Wire-Shark is network monitoring tool in which every incoming and outgoing data has captured and

relative information is shown in the system.

31

Figure 10: Normal Graph

• Data Flood Graph

In Figure [11] DoS or DDoS attack is happening and the data is flooding 4 on the

network and defined graph and equation show there is something anomaly action

on the network. Then IDS/IPS will generate alert and takes action. At the time of

attack, the average data is tiny as compared to attacked data means defined aver-

age graph has crossed the limit and graph go beyond aspect by IDS/IPS machine.

Therefore, immediate action will take place by machine and more than threshold

value data will not allow. This is the feature of sensing capability of the machine

and comes under Proactive approach in the field of network security. The data flood

graph is shown in Figure [11]. The above graphs are given on the basis of average

data of any company and the maximum data of a day is given. The maximum

data is calculated on the basis of working hours of the company. Here the working

hours are assumed 12 hours per day. And in working hours the downloading and

uploading data are capture and used in graph.

The average data have used by 12 hours per day, which are the working hours in the

company.

Threshold values are:

• Uploading = 432 Gb

4The flooding of data is at very high rate in DoS or DDoS attack, the data is flooded by attacker becauseintention is jamming the bandwidth and achieving goal.

32

Figure 11: Data Flood Graph

• Downloading = 216 Gb

In flood graph when DoS or DDoS attack is happening then, the downloading data

cross the threshold value, and IDS/IPS will take action. So more than threshold value no

data will allow entering in the network, according to defined action.

The above two algorithms are given for anomaly detection and prevention mecha-

nisms. Traditional ids/ips tools work on the signature-based detection system so for in-

creasing the intelligence of machine anomaly detection and prevention is included. Which

will help to protect the network from data flood attacks. If any data flood attack (DoS or

DDoS) is happening on the network then defined graph will cross the threshold point of

normal data. Therefore, according to defined action machine will work. And prevention

of DoS or DDoS attack is possible. So the used algorithm is a proactive approach in

network security field.

Phase 3 : Decision Making Phase

The action taking part is processed on the phase 3 machine which is shown in Table [5],

and all the input of this machine are dependent on the phase 2 outputs and according to

them, relevant action will take place. The decision-making condition will calculate by

AND GATE properties. In the table [5] showing that the data is allowed or not allowed to

enter in the network.

The cases give high-end security to the network because the data is allowed to enter

the system only in a single case, the given table[5]. The allow case of data is dependent on

33

both the IDS/IPS tools which mean if both cases allow condition then only data will enter

the network. If any of one give not allow state, then data will not come into the system,

and the alert will generate, and data keep in the database. The database has maintained

for collecting dropped packets because these dropped packets are used for making the

analysis.

Table 5: Decision Making Table

Case SNORT SURICATA SNORT ANDSURICATA

Action Taken by Ma-chine

Case1

Allow Allow Allow Data will enter in the net-work

Case2

Allow Not Allow Not Allow Blocked, Alert Generate,Keep in database 1

Case3

Not Allow Allow Not Allow Blocked, Alert Generate,Keep in database 2

Case4

Not Allow Not Allow Not Allow Blocked, Alert Generate,Keep in database 3

3.3.4 Analysis of Dropped Packet

Taking the Advantage from hackers Activity for securing the network.

The dropped packets are the advantages for us. The meaning of saying Advantage from

hackers Activity is that hacker is giving the hint to us something is vulnerable in your

network that’s why they are attracting. These dropped packets are the malicious activity

which was made by the attacker, but blocked by used scheme. The probability of success

of zero-day threat is not 1 at the first attempt. So many intrusions had made by the attacker,

and IDS/IPS blocks these attacks. So by these dropped attacks we are able to know the

target of an attacker.

Activity made by an attacker on Network The intruder tries to gain all the valuable in-

formation about the network. Without knowing the relevant information about the system,

the intruder is unable to exploit the network [10]. All kind of information like which type

34

network security is uses or various rules using for the network security, all the security

levels, etc.

Table 6: Attacks on Various types of Packet

Name Of Packet Name of Protocols Attacks based on ProtocolsTCP/IP ARP, IP, RARP, HTTP,

HTTPSPassword sniffing, Denial ofservice, TCP sequence num-ber, TCP session hijacking,Ping O Death, etc.

UDP FTP, TELNET,TFTP, SMTP,TIME,POP3, FINGER, NTP,IMAP2, SNMAP

DNS, SSDP, CharGEN,QOTD, BitTorrent, Kad,Steam Protocol, NTP, Net-BIOS, Quake NetworkProtocol, RIPv1, MulticastDNS(mDNS), Portmap

Various types of given packets and protocols on which attacker made attacks [11]. Some

of the attack may repeat and according to attack; the signature has defined for detecting

the attack.

3.3.5 Collecting Valuable Information from Dropped Packet

Every packet has some unique feature, unique information, and structure. The dropped

packet are of different types and valuable information from each dropped packet. Because

what the information will be access from dropped packet will only tell what action should

take for securing the network from new threat, Various types of packets and their valuable

information is given in Table [7]

3.3.6 Comparison Between Snort and Suricata Results

Using two types of machine give strength to the network because at least one of the two

machines gives the better result for all malicious activities. Because both the machine

35

Table 7: Valuable Information Relate to Packet

Name of Packet Valuable InformationIP Version, Type Of services(ToS), Flags, Destination Address, protocol

TCP Destination port, Control Bit Flags(URG, PSH), Urgent PointerUDP Destination port, Destination, Length

have different detection mechanism in some cases[1]. Combined both the device will

play the good role instead of single machine. The difference given in table [8] .

Table 8: Snort vs Suricata

Parameter Snort SuricataThread Single-Thread Multi-ThreadRules Talos(VRT) : Snort

rules, Shared Objectrules and EmergingThreat rules

Talos(VRT) : Snort rulesand Emerging Threat rules

Shellcodes Less Detectable More DetectableEvasion Technique More Detectable Less DetectableMalware and Viruses Less Detectable More DetectableMultiple failed logins More Detectable More DetectableFragmented packets More Detectable Less DetectableDenial of Service (DoS) More Detectable More Detectable

Both the tools are different in some cases of detection mechanism like in some cases both

are equally detectable or in some cases anyone is highly detectable and other may be less

detectable. We are using two tools and at least one has highly detectable in all the cases.

Cases are shown in table [8].

Deploying multilevel ids/ips on the network gives multifunctionality processes. The

initial stage is under detection and prevention of malicious activity on the network. The

intruder does not bypass the multilayer security system because using two or more tools

gives strength to the network. The another part of proposed architecture is preventing

new attacks on the network. Another advantage of multilayer ids/ips is decreasing the

false negative rate in the network. Because the intruders unable to bypass the security

36

area, it is possible by using two or more tools in the proposed architecture. The reason is

at least one of all the different tools give a true positive result for a particular malicious

activity.

3.3.7 Various Cases of Decision-Making Machine

In the decision-making machine, there are four cases arises by the AND Gate property.

These four cases play different role for every output from the machines. Because the

output is dependent on phase 2 of architecture. If decision-making machine drops any

data, then there may be the possibility of the false positive case. So for removing false

positive cases, there are three types locations have used for storing dropped packets. The

data location and various cases have discussed below which have discussed in table [5].

• Case 1:

Both the devices of Snort, Suricata on phase 2 will say the data is not malicious

therefore the incoming data have entered into the network.

• Case 2:

Snort machine said the data is not malicious, and Suricata machine said the data

is malicious. So there are chances of this case of false positive or it can be possi-

bility of malicious data. Therefore these categorie’s data are stored in the database

location 1.

• Case 3:

Snort machine said the data is malicious and suricata machine said the data is not

malicious. So there is chances of this case of false positive or it can be possibility of

malicious data. Therefore these categorie’s data are stored in the database location

2.

• Case 4:

Both the machines of phase 2 said the data is malicious, so it is the possibility of

data is malicious and dropped by ids/ips. These categories data are uses for making

analysis for preventing new attacks because the intruders attempts for intrusions

37

goes failed5.

5 Case 4 gives the high probability of intrusions because both the used tools Snort and Suricata sayingthe data is malicious. This type of dropped packets are analyzed regarding new attacks and false negativecases can be prevented.

38

Chapter 4Experimental Setup and Results

4.1 Configurations for Experimental Performance

In this area I tried to improve the graph level ”Intrusion Detection System” as well as re-

ducing “False Negative” cases and also tried to prevent the new threat by making analysis

of dropped packet. The work performed on the following terms which are given below.

• Detection of Malicious Activity

• Prevention of Malicious Activity

• Alert Generating

• Dropped Packet File Generating

• Analysis of Dropped Packet.

4.1.1 Tools Configurations

• Host Machine Configuration

The experimental performance has done on a single machine. Some more virtual

39

machines have installed on the single host machine, and the network has created

between all of these. The configuration has discussed below.

Table 9: Configuration of Host Machine

PROCESSOR Intel Core i7 @ 3.40GHzRAM 32.0 GB

OS Windows 8.1HDD 1TB

• Tools Used

– Vmware Workstation 12 player

– WireShark

– Nmap

– Zenmap

– Snort

– Suricata

– MySQL

– Software Firewall

All the above tools are used in the host machine and Snort configured machine

for network monitoring.

• Machines Installed in Vmware

– Snort Installed Machine(Ubuntu 14.04)

– Protection Machine(Windows 7)

– Attacker Machine(Kali 2.0)

– Traffic Generator Machine(Ubuntu 14.04)

40

4.1.2 Performance

The work is performed on the single machine and the configuration of machine have given

in Table [9]. Four VMware machine are installed on the host machine, and every machine

have different functionality and uses.

• Snort Installed Machine(Ubuntu 14.04)

The Snort tool gives better result in the Linux OSes because the functionality of

snort is more compatible with Linux. Here the reason for using Ubuntu is only

this. Snort has installed as Network Intrusion Detection System. Network Intru-

sion Detection and Prevention System has monitored the whole network which has

discussed in Figure [3]. NIDS inspect with the functionality of “Deep Packet In-

spection” [8]. Each and every packet is monitor by nids machine and check for

malicious activity.

– Snort Pre-Requisites Snort have four main Pre-Requisites 1

Table 10: Snort Pre-Requisites File

Library/FileName

Name to Search Available Source

libdnet libdumbnet-dev Accessible by Ubuntuvault(repository)

pcre libpcre3-dev Accessible by Ubuntuvault(repository)

pcap libpcap-dev Accessible by Ubuntuvault(repository)

daq www.snort.org compiled from source

– For installing snort some basic commands are as follows:

1Before installing Snort pre-requisites configuration is must, without configuring these libraries Snortwill not give accurate result.

41

Listing 4.1: Installing Snort in Ubuntu by Following commands

cd / s n o r t s r c <<<Line 1>>>

wget ‘ ‘ l i n k o f s n o r t f i l e ” <<<Line 2>>>

t a r −xvz f s n o r t 2 . 9 . 8 . 0 . t a r . gz % Unta r t h e

downloaded f o l d e r . <<<Line 3>>>

cd s n o r t 2 . 9 . 8 . 0 % E n t e r i n t h e s n o r t f o l d e r

where a l l t h e f i l e c o n s i s t i n g . <<<Line 4>>>

c o n f i g u r e − s o u r c e f i l e <<<Line 5>>>

make <<<Line 6>>>

sudo make i n s t a l l <<<Line 7>>>

Figure 12: Installed Snort on Ubuntu Machine

In Figure [12] the version of snort is shown. And also showing the pre-requisites

libraries.

– Two Libraries give strength to snort which has configured in snort machine.

∗ Barnyard

Barnyard2 is an open source mediator for Snort unified2 parallel yield

records. Snort write output to the hard drive and by using Barnyard han-

dle database inserts into binary form. For installing barnyard in snort

machine the following commands are uses which are given below.

Listing 4.2: Configuring Barnyard with Snort by Following commands

42

sudo wget ‘ ‘ l i n k o f b a r n y a r d 2 f i l e ” <<<Line

1>>>

sudo t a r −z x f m a s t e r % Unta r t h e downloaded

f i l e <<<Line 2>>>

cd f i r n s y−b a r n y a r d 2 ∗ % Wil l go i n t h e

b a r n y a r d f o l d e r <<<Line 3>>>

sudo a u t o r e c o n f − f v i −I . / m4 <<<Line 4>>>

sudo . / c o n f i g u r e w i t h −mysql w i t h −mysql−l i b r a r i e s = / u s r / l i b / i386−l i n u x−gnu %

C o n f i g u r i n g b a r n y a r d wi th s n o r t c o n f i g u r i n g

f i l e <<<Line 5>>>

sudo make <<<Line 6>>>

sudo make i n s t a l l <<<Line 7>>>

sudo cp / ‘ ‘ l o c a t i o n o f b a r n y a r d 2 . con f f i l e ” /

b a r n y a r d 2 . con f / ‘ ‘ l o c a t i o n o f s n o r t f o l d e r ” /

% Coping t h e b a r n y a r d 2 . con f f i l e a t t h e

l o c a t i o n o f s n o r t f i l e s <<<Line 8>>>

sudo cp schemas / c r e a t e mysql / u s r / s r c <<<Line

9>>>

sudo mkdir / v a r / l o g / b a r n y a r d 2 <<<Line 10>>>

∗ Oinkmaster

Oinkmaster is a tool to update snort rules file. Written in perl, installing

perl is must. Some rules are own created and if not to modified then

defined in oinkmaster not to update. For updating the snort signature the

oinkmaster is uses. The configuration process of oinkmaster is shown

below.

Listing 4.3: Configuring Oinkmaster with Snort by Following commands

cd / ‘ ‘ a c c e s s o i n k m a s t e r d i r ” <<<Line 1>>>

43

sudo bash −c ‘ ‘ sudo . / c r e a t e−s idmap . p l / e t c /

s n o r t / r u l e s >/ e t c / s n o r t / s i d−msg . map” <<<

Line 2>>>

sudo / ‘ ‘ l o c a t i o n o f b a r n y a r d 2 d i r ” −c / ‘ ‘

l o c a t i o n o f s n o r t f o l d e r / b a r n y a r d 2 . con f −d

/ ‘ ‘ l o c a t i o n o f s n o r t l o g s ” −f s n o r t . l o g −w /

v a r / l o g / b a r n y a r d 2 / by log . waldo −C / ‘ ‘ l o c a t i o n

o f s n o r t d i r ” / c l a s s i f i c a t i o n . c o n f i g <<<

Line 3>>>

CRTL+C t o e x i t b a r n y a r d 2 <<<Line 4>>>

• Protection Machine(Windows 7)

In my project the windows 7 machine configured as protected machine. Because

working only on a single host machine, the protected machine is defined only one

machine in snort machine. This protected machine assumed as a complete network,

and a range of IP addresses is given in snort machine.

• Attacker Machine(Kali 2.0)

Kali Linux is used as attacker machine because in the Kali Linux so many tools are

available. The various tools are uses for committing various network attacks. The

attacks have done on the Windows 7 machine because snort machine protects this

machine and that is my task for protecting the network by using Snort tool.

• Traffic Generator Machine(Ubuntu 14.04)

Traffic generator machine has used for generating unwanted traffic on the network

because of testing snort working. The data is massive which has sent on the win-

dows machine.

4.2 Results of Single-Layer Intrusion Detection System

The Snort has configured for detecting and preventing malicious activity on the network.

The Snort has installed on the Ubuntu machine, and three other machines are also in-

stalled. The detailed explanation is given below.

44

4.2.1 Processing in Snort Machine

The Network Intrusion Detection and Prevention system have configured on the Ubuntu

machine. The malicious activity made by the intruder on the whole network is detected

and prevented by snort machine. The detection/prevention mechanism is based on the

signature which are given on Table [4]. The snort mechanism has explained below.

1. Snort Running Commands

• snort -c /“snort installed location”/snort.confCommand will give information regarding correctness of snort configuration.

• sudo snort -i eth0 -vNormally we get a live packet sniffing.

• sudo snort -A console -u snort -g snort -c /“snort installed location”/snort.conf-i eth0 -TDoing a config loading test.

2. IP address of Protecting system

Figure 13: IP address of protecting machine

In Figure [13] the IP address is shown, this IP address is of Windows 7 machine

because I want to protect Windows 7 machine from intrusions. The network which

is to be protected, the IP addresses of the network is defined in the snort machine.

We can also define any particular IP address within the same network.

45

Figure 14: Different types of Rules Saved in Machine

3. Pre-defined Signatures List

In the Figure [14] the list of defined rules have shown. These signature are de-

termined by previous attacks which had already detected. And some rules can be

user specified by the requirement. The update of snort signature is possible by the

Oinkmaster tool. Oinkmaster updates the snort signature regularly, if any new sig-

nature is added to the developer team, then it will be maintained by oinkmaster. The

oinkmaster has configured in the snort machine after configuring the Snort tool.

4. User defined Signature

In Figure [15] the rule is given for demo purpose, and it is the user defined signa-

ture. The location of user defined rule is /etc/snort/rules/zzalert.rules and shown

in Figure [14] in the circle. The signature is developed for ICMP packet testing;

46

Figure 15: User Define Rule For Demo Purpose

any ICMP attack is made on protected machine by intruder then the alert will be

shown in console mode. Which is shown in Figure [6]

5. MySQL Server

MySQL server is configured in snort machine for keeping the file in the database.

Keeping the dropped files in the database for analysis purpose. And this analysis

of dropped file will help to prevent the zero-day threat as well as reduce the false

negative rate. And in the network security, it is the proactive approach. Because the

intention of the intruder has known.

6. SYN Flood on Protected Machine

The attacker machine does SYN flood on the protected computer. The protection

has given by snort machine which has discussed in Figure [13]. The packets have

sent at the slight time of interval at targeted device is shown in Figure [16]. And

also, IP address is shown, this is the target machine’s IP address.

Figure 16: SYN Flood Record in Attacker Machine

47

Features of Snort Machine

Snort have the capability of detection and prevention of malicious activity. And also,

the packet dropping facility is there. When a malicious activity made by intruder then

according to defined signature the action will be taken by the machine. All the features of

Snort have explained below.

Detection of Malicious Activity Snort detection mechanism is based on signature and

anomaly based. The list of signatures in Figure [14] defined in the snort machine accord-

ing to the past malicious activity. The anomaly-based detection mechanism is based on

the sensing capability of the machine. The machine will sense the network and according

to the network behavior, action will take place. For testing the machine, User define rule

is generated which is shown in Figure [15]. And from attacker machine, the SYN flood

attack is happening on protected machine, and detection of that attack is successful which

is shown in Figure [17].

Prevetion of Malicious Activity Prevention of malicious activity by snort is next phase

of detection mechanism. The basic step of snort against malicious activity is detection

and after that prevention activity is happening. The prevention of malicious activity is

also based on two mechanisms, signature and anomaly based detection and prevention

mechanisms. The prevention of malicious activity generated dropped file which is shown

in Figure [18]. At this location only the malicious activity is located. So we can say this

is the malicious activity and prevention of malicious activity is the successful.

Alert Generating Snort machine generates the alert according to the defined signature

in the computer, in Figure [17] Generating alert means if malicious activity is happening

on the network then detection/prevention is completed by the snort machine. Then after

the process is informed by the snort machine to the network analyst by alarm or by a

trigger. Or it is also possible of seeing this process in the console mode of snort machine.

For running snort in live console mode, the command is used “Root access” snort -Aconsole -q -u snort -g snort -c /“snort installed location”/snort.conf -i eth0 and the

output of snort machine is shown in Figure [17].

48

Figure 17: Alert Generating in Live Mode

Dropped Packet File Generating Snort machine blocks the malicious activity, and

this action comes under dropped packets. Snort generate the dump packages, and these

packets are uses for analysis purpose. Snort machine gives the file name according to the

date and time of attack this is the inbuilt features of Snort machine. The dropped file is

shown in Figure [18]. And the default location of file generating is /var/log/snort.

Figure 18: File Generated when Malicious Activity Detected

Analysis of Dropped Packet Analysis of dropped packet is the source for preventing

zero-day threat because these are the malicious activity which is made by the intruder.

And intruder trying to exploit the vulnerability on the network by malicious activity means

something wrong with the network. The detailed study of dropped packet has explained

in the Topic [3.3.4]. The file has generated once the malicious activity has detected, the

file has shown in Figure [18]. What this file contains? Is the dropped data can give any

information against new attacks? All the valuable information which will help for pre-

venting zero-day threat are collected. The valuable information collection has discussed

49

in Topic [3.3.5]. In the Figure [19]. Seeing all the packets of SYN flood is possible but

here in the Figure, only a few of them is possible to see. With the help of “x less” only one

page has seen here.

Figure 19: Data in Dropped File

The destination IP addresses and other related information which has discussed in table 7

are collected and kept in the database for analyzing. Because of using different locations

for dropped packets storage, it is helpful for making analysis[3.3.7].

50

Chapter 5Conclusion and Future Work

We have implemented and tested a scheme of multi-layer scheme using the combination

of Snort and Suricata. It is observed that Snort detects and prevents more intrusions

compared to Suricata, and some more results are discussed below.

• Suricata detected more intrusions on Shell Codes.

• Snort detected more intrusions on Multifailed logins.

• Both snort and suricata equally detected intrusions on Denial of Service attack.

A drawback of our proposed design is generated more alert unnecessary in some cases.

Future works include reducing unnecessary alert by machine learning concepts.

51

Author’s Publications

• Vikash Kumar Saini, B.M. Mehtre. “Multi-Layer Intrusion Detection and Pre-

vention System: A New Approach”, Submitted to Journal of Computers & Secu-rity, ELSEVIER, in May, 2016.

52

References

[1] Aldeid, . Digital forensics, computer-forensics, malware-network-detection,

snort-suricata. https://www.aldeid.com/wiki/Suricata-vs-snort;

2013.

[2] Aldwairi, M., Conte, T., Franzon, P.. Configurable string matching hardware for

speeding up intrusion detection. SIGARCH Comput Archit News 2005;33:99–107.

[3] Ballmann, B.. Understanding Network Hacks Attack and Defense with Python.

springer, 2015.

[4] CTDP, . The computer technology documentation project. http:

//www.comptechdoc.org/independent/networking/protocol/

protnet.html; 2010.

[5] Denning, D.E.. An intrusion-detection model. In: Security and Privacy, 1986 IEEE

Symposium on. 1986. p. 118–118.

[6] Guimaraes, M., Murray, M.. Overview of intrusion detection and intrusion pre-

vention. In: Proceedings of the 5th Annual Conference on Information Security

Curriculum Development. ACM; 2008. p. 44–46.

[7] Harris, B., Hunt, R.. Review: Tcp/ip security threats and attack methods. Comput

Commun 1999;22:885–897.

53

[8] He, X.D., Ling, T.C.. Enhanced automated intrusion prevention in network secu-

rity. In: Computer Engineering and Technology (ICCET), 2010 2nd International

Conference on. volume 2; 2010. p. 286–291.

[9] Kizza, J.M.. GUIDE TO COMPUTER NETWORK SECURITY. volume 2nd

Edition. Springer, 2013.

[10] Lan, F., Chunlei, W., Guoqing, M.. A framework for network security situation

awareness based on knowledge discovery. In: Computer Engineering and Technol-

ogy (ICCET), 2010 2nd International Conference on. volume 1; 2010. p. 226–231.

[11] NEEDHAM, R., LAMPSON, B.. Network attack and defense. 2010.

[12] OISF, . Open source ids and ips and nsm engine. 2015.

[13] Pearson, . Working with snort rules. https://www.pearsonhighered.

com/samplechapter/0131407333.pdf; 2015.

[14] Prowell, S., Kraus, R., Borkin, M.. Seven Deadliest Network Attacks. volume 1st.

Syngress Publishing, 2010.

[15] Roesch, M.. Description of snort. https://www.snort.org.

[16] Sahay, R., Blanc, G., Zhang, Z., Debar, H.. Towards autonomic ddos mitigation

using software defined networking. In: SENT 2015 : NDSS Workshop on Security

of Emerging Networking Technologies. .

[17] Shin, S., Wang, H., Gu, G.. A first step toward network security virtualization:

From concept to prototype. IEEE Transactions on Information Forensics and Secu-

rity 2015;10(10).

[18] Umesh Hodeghatta Rao, U.N.. THE INFOSEC HANDBOOK, An Introduction to

Information Security, 2014.

[19] Yan, F., Jian-Wen, Y., Lin, C.. Computer network security and technology re-

search. In: 2015 Seventh International Conference on Measuring Technology and

Mechatronics Automation. 2015. p. 293–296.

54

[20] Zhou, H., Wu, C., Jiang, M., Zhou, B., Gao, W., Pan, T., Huang, M.. Evolving

defense mechanism for future network security. IEEE Communications Magazine

2015;53(4):45–51.

55