using event correlation technologies h njemanze
TRANSCRIPT
-
8/6/2019 Using Event Correlation Technologies h Njemanze
1/42
Stop The Insanity: Using EventCorrelation Technologies, Tools, andTechniques to Extract MeaningfulInformation from Data Overload
Hugh NjemanzeCTO and Founder
May 2006
2005 ArcSi ht Confidential
-
8/6/2019 Using Event Correlation Technologies h Njemanze
2/42
2005 ArcSight Confidential 2
Agenda
What is the Problem?
What is Correlation?
How to Think about the Process
Correlation Technologies, Tools and Techniques
Benefits of Visual Representations
-
8/6/2019 Using Event Correlation Technologies h Njemanze
3/42
2005 ArcSight Confidential 3
What is the Problem?
-
8/6/2019 Using Event Correlation Technologies h Njemanze
4/42
2005 ArcSight Confidential 4
What is the Problem?Complexity of the Security Infrastructure
Flood of unread data/logs
Islands of defense
Massive false positives
Heterogeneous devices
Inefficient and Ineffective
AntiVirusAntiVirusDatabases
FirewallsFirewallsFirewallsFirewallsFirewallsFirewalls
IntrusionDetectionSystems
IntrusionDetectionSystems
IntrusionDetectionSystems
IntrusionDetectionSystems
IntrusionDetectionSystems
IntrusionDetectionSystems
Intrusion
DetectionSystems
IntrusionDetectionSystemsIntrusionDetectionSystems
Hosts
IntrusionDetectionSystems
IntrusionDetectionSystems
IntrusionDetectionSystems
IntrusionDetectionSystemsNetwork
Equipment
Applications
ApplicationsApplicationsApplicationsApplicationsApplicationsApplicationsApplicationsApplicationsApplications
Sign-OnSign-OnSign-On
DirectoryServices
-
8/6/2019 Using Event Correlation Technologies h Njemanze
5/42
2005 ArcSight Confidential 5
Deal with a Flood of Diverse Data
Events from many sensors
NIDS, HIDS, firewalls, anti-virus, more
Application logs, phone logs, moreUnderstanding the protected network
Vulnerability assessment scanners
Configuration management databases
Understanding ofvulnerabilities
CVEOASIS
-
8/6/2019 Using Event Correlation Technologies h Njemanze
6/42 2005 ArcSight Confidential 6
The Needle in the Haystack
Raised Alerts
Case Workflow
Raw events
NormalAudit trail
Failed attacks
False alarms
Pre-attacksAttack
formation
Verified
breachesPolicy
violations
Identifiedvulnerabilities
Misuse
Potentialbreaches
Tens of millionsper day
Millionsper day Less than
1 millionper month
A few thousandper month
-
8/6/2019 Using Event Correlation Technologies h Njemanze
7/42 2005 ArcSight Confidential 7
A Single Integrated Solutionis Required for ESM
AntiVirusAntiVirusDatabases
ArcSight Monitoring, Visualization, and Reporting
ArcSight Real-time Analysis, Correlation, and Workflow
ArcSight Event Collectors
FirewallsFirewallsFirewallsFirewallsFirewallsFirewalls
IntrusionDetectionSystems
IntrusionDetectionSystems
IntrusionDetectionSystems
IntrusionDetectionSystems
IntrusionDetectionSystems
IntrusionDetectionSystems
Intrusion
DetectionSystems
IntrusionDetectionSystemsIntrusionDetectionSystems
Hosts
IntrusionDetectionSystems
IntrusionDetectionSystems
IntrusionDetectionSystems
IntrusionDetectionSystemsNetwork
Equipment
Sign-OnSign-OnSign-On
DirectoryServices
Applications
ApplicationsApplicationsApplicationsApplicationsApplicationsApplicationsApplicationsApplicationsApplications
ArcSightConsoleTM
-
8/6/2019 Using Event Correlation Technologies h Njemanze
8/42 2005 ArcSight Confidential 8
What is Correlation?
-
8/6/2019 Using Event Correlation Technologies h Njemanze
9/42 2005 ArcSight Confidential 9
What is Correlation?
A relation existing betweenphenomena or things which tendto vary, be associated, or occurtogether in a way not expectedon the basis of chance alone.
Merriam-Webster Dictionary
-
8/6/2019 Using Event Correlation Technologies h Njemanze
10/42 2005 ArcSight Confidential 10
Also, Perhaps, Inference
The reasoning involved in drawing aconclusion or making a logical judgmenton the basis of circumstantial evidence
and prior conclusions rather than on thebasis of direct observation.
Princeton Universitys WordNet
-
8/6/2019 Using Event Correlation Technologies h Njemanze
11/42 2005 ArcSight Confidential 11
Highlight Changes in Behavior
Changes in the typical event flow may indicate
An ongoing attack
Denial of service: the source is deadCompromise: the source is behaving atypically
New patterns of behavior may indicate
The presence of malware
An insider threat
Introduction of new software or devices
-
8/6/2019 Using Event Correlation Technologies h Njemanze
12/42 2005 ArcSight Confidential 12
Escalation: Sounding the Alarm
Generate notifications
Email, page, pop-up
Open a caseTrouble tickets, incident tracking
Create alarms
Tracking events
-
8/6/2019 Using Event Correlation Technologies h Njemanze
13/42 2005 ArcSight Confidential 13
How to Think About theProcess
-
8/6/2019 Using Event Correlation Technologies h Njemanze
14/42 2005 ArcSight Confidential 14
Process
Intelligence
Collection, normalization and aggregation
Risk-based prioritization with vulnerability and asset information
Correlation across event sources
Rule-based correlation
Statistical Correlation
Advanced analysis
-
8/6/2019 Using Event Correlation Technologies h Njemanze
15/42 2005 ArcSight Confidential 15
Event Normalization and Categorization
Jun 01 2005 00:00:12: %PIX-3-106011: Deny inbound (No xlate) udp src
outside:10.50.215.97/6346 dst outside:204.110.228.254/6346
Jun 01 2005 00:00:12: %PIX-6-305011: Built dynamic TCP translation from
isp:10.50.107.51/1967 to outside:204.110.228.254/62013
Jun 01 2005 00:00:12: %PIX-6-302013: Built outbound TCP connection
2044303174 for outside:213.189.13.17/80 (213.189.13.17/80) to
isp:10.50.107.51/1967 (204.110.228.254/62013)
Jun 02 2005 12:16:03: %PIX-6-106015: Deny TCP (no connection) from
10.50.215.102/15605 to 204.110.227.16/443 flags FIN ACK on interface
outside
Sample Raw Pix Events:
Jun 02 2005 12:16:03: %PIX-6-106015:
Deny TCP (no connection) from
10.50.215.102/15605 to 204.110.227.16/443
flags FIN ACK on interface outside
Categorization:Normalization:
-
8/6/2019 Using Event Correlation Technologies h Njemanze
16/42
2005 ArcSight Confidential 16
Diverse Data Sources: Event Normalization
Comparing apples to apples
Many vendors
Many types of sensorsLots of overlap
Normalization
Common schema: info in the same place
Categorization: describing the event
Values: single domain
-
8/6/2019 Using Event Correlation Technologies h Njemanze
17/42
2005 ArcSight Confidential 17
Diverse Data Sources: Event Normalization
Aggregation: easier to establish equivalence
Rules can be written once and applied to all sensorsof a given type
Simplifies log review when multiple brands of sensorare present
Speeds training of new personnelEasier to understand events
-
8/6/2019 Using Event Correlation Technologies h Njemanze
18/42
2005 ArcSight Confidential 18
Risk-based Prioritization
WindowsSystems
Unix/Linux/
AIX/Solaris
SecurityDevice
SecurityDevice
Mainframe& Apps
SecurityDevice
Agents
Event
Manager
PrioritizedEvent
VulnerabilityScanner
Agents
AssetInformation
Model ConfidenceHas asset been
scanned for open portsand vulnerabilities?
RelevanceAre ports open on asset?
Is it vulnerable?
SeverityIs there a history withthis attacker or target
(active lists)?
Asset CriticalityHow important is this
asset to the business?
Agent SeverityMapping of reporting
device severity toArcSight severity
-
8/6/2019 Using Event Correlation Technologies h Njemanze
19/42
2005 ArcSight Confidential 19
Event Correlation
Most overused and least well-defined concept in ESM.
Combine multiple events through predefined rules
Or analyze statisticalproperties of event streams
Across devices
Heavily utilizing event categorization
Helps eliminate false positives
Correlation is not prioritization! Can use priorities of individual events
-
8/6/2019 Using Event Correlation Technologies h Njemanze
20/42
2005 ArcSight Confidential 20
Rule-based Correlation
Combine multiple events through predefined rules
Multiple failed loginson Windows systems
Multiple failed loginson UNIX systems
5 or more failed
loginsin a minute
from same source
Attempted Brute
Force Attack
Attempted Brute
Force Attack
-
8/6/2019 Using Event Correlation Technologies h Njemanze
21/42
2005 ArcSight Confidential 21
Rule-based Correlation
Combine multiple events through predefined rules
Attempted BruteForce Attack + Successful
Login
Successful
login to Windows system
Attempted Brute
Force Attack
-
8/6/2019 Using Event Correlation Technologies h Njemanze
22/42
2005 ArcSight Confidential 22
Statistical Correlation
Analyze statisticalproperties of event streams
?
50% increase
in traffic per port
and machine
Traffic per port going to 10.0.0.2
False positives reduction:
Correlate against other event streams
Restrict to only monitor specific systems and specific type of traffic
8
-
8/6/2019 Using Event Correlation Technologies h Njemanze
23/42
2005 ArcSight Confidential 23
Many Correlation/Inference Techniques
Model-based
Assets
ThreatsHeuristic
Pattern
Formula
Mathematical
AnomalyCovariant
Slide 23
-
8/6/2019 Using Event Correlation Technologies h Njemanze
24/42
j8 I think the following slides can be used as some of the voice over from the previous 4-6 slides?jkyte, 10/11/2005
-
8/6/2019 Using Event Correlation Technologies h Njemanze
25/42
2005 ArcSight Confidential 24
Model-Based Reasoning
Checking the protected network
Does the device exist?
Applications presentOperating systems
Vulnerabilities exposed
Business significance
Extensible via active lists
Attackers: suspicious, recon, hostileDevices: scanned, attacked, compromised
-
8/6/2019 Using Event Correlation Technologies h Njemanze
26/42
2005 ArcSight Confidential 25
Heuristic: Formula-Based
SeverityWhat is the attack potential?
Model Confidence and RelevanceCould it work?
Asset Criticality
How valuable is the target?
Priority
Which incident should be worked first?
-
8/6/2019 Using Event Correlation Technologies h Njemanze
27/42
2005 ArcSight Confidential 26
Mathematical
Statistical data monitors
Moving average
Statistics
Correlation
Pattern discovery
Covariant occurrence ofindividual events
Statistics data monitors spot gross changes in theevent flow
More attacks against certain ports, networks
Sudden drop in events from a service
Discovery spots behaviors on the protected network
New exploits
Returning exploits: that virus is back!
-
8/6/2019 Using Event Correlation Technologies h Njemanze
28/42
2005 ArcSight Confidential 27
How: Correlation
Technologies, Tools andTechniques
Traditional Approach Log Files and Events
-
8/6/2019 Using Event Correlation Technologies h Njemanze
29/42
2005 ArcSight Confidential 28
Traditional Approach Log Files and Events
A Visual Approach
-
8/6/2019 Using Event Correlation Technologies h Njemanze
30/42
2005 ArcSight Confidential 29
ppSituational Awareness - Instant Awareness
-
8/6/2019 Using Event Correlation Technologies h Njemanze
31/42
2005 ArcSight Confidential 30
Why a Visual Approach Helps
Reduce analysis and response times Quickly visualize thousands of events
Make better decisions
Situational awareness
Visualize status of business posture
Visual display of most important propertiesBe more efficient
Facilitate communication
Use graphs to communicate with other teams Graphs are easier to understand than textual events
A picture tells more than a thousandlog lines
-
8/6/2019 Using Event Correlation Technologies h Njemanze
32/42
2005 ArcSight Confidential 31
Three Aspects of Visual Security Analysis
Situational Awareness
What is happening in a specific business area(e.g., compliance monitoring)
ific network What is happening on a spec What are certain servers doing
Real-Time Monitoring and Incident Response
Capture important activities and take action Event Workflow
Collaboration
Forensic Investigation Selecting arbitrary set of events for investigation
Understanding big picture
Analyzing relationships
-
8/6/2019 Using Event Correlation Technologies h Njemanze
33/42
2005 ArcSight Confidential 32
Responding: Monitoring and Reporting
Live monitoring
Channels
DashboardsReporting
-
8/6/2019 Using Event Correlation Technologies h Njemanze
34/42
2005 ArcSight Confidential 33
Situational Awareness Event Graph Dashboard
-
8/6/2019 Using Event Correlation Technologies h Njemanze
35/42
2005 ArcSight Confidential 34
Real-time Monitoring Detect Activity
-
8/6/2019 Using Event Correlation Technologies h Njemanze
36/42
2005 ArcSight Confidential 35
Visual Detection
Scan Events
Firewall Blocks
Scanning activity is displayed
Vi l I i i
-
8/6/2019 Using Event Correlation Technologies h Njemanze
37/42
2005 ArcSight Confidential 36
Visual Investigation
D fi N C l ti R l d Filt
-
8/6/2019 Using Event Correlation Technologies h Njemanze
38/42
2005 ArcSight Confidential 37
Define New Correlation Rules and Filters
Assign for further analysis if More than 20 firewall drops
froman external machine
toan internal machine
1. Rule
Internal machines on white-list
connecting toactive directory servers
2. Filter
3. Open a ticketfor Operations to
quarantine and clean infected machines
F i A l i
-
8/6/2019 Using Event Correlation Technologies h Njemanze
39/42
2005 ArcSight Confidential 38
Forensic Analysis
Failed Logins High ratio of failed logins
Forensic Analysis
-
8/6/2019 Using Event Correlation Technologies h Njemanze
40/42
2005 ArcSight Confidential 39
Forensic Analysis
Attacks targeting internal systemsAttacks
Revenue Generating Systems
Summing Up
-
8/6/2019 Using Event Correlation Technologies h Njemanze
41/42
2005 ArcSight Confidential 40
Summing Up
Effective correlation enables codifying and leveragingdomain expertise to automate finding the needles in thehaystack of security logs, alerts and events
Visualization techniques provide a very intuitive way forhuman analysts to quickly spot patterns and activity thatwould otherwise be buried in logged data
Gathering all the data in one place to start with providesa vantage point from which to apply the tools andtechniques described above
-
8/6/2019 Using Event Correlation Technologies h Njemanze
42/42
Q & A
Email to: [email protected] 2005 ArcSi ht Confidential