This article was downloaded by: [Carnegie Mellon University]On: 09 November 2014, At: 00:50Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: MortimerHouse, 37-41 Mortimer Street, London W1T 3JH, UK

Information Systems SecurityPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/uiss19

Security Event Analysis through CorrelationAnton Chuvakin Ph.D., GCIA, GCIH aa A Senior Security Analyst at a major security company. His areas of infosec expertiseinclude intrusion detection, UNIX security, forensics, honeypots, etc. In his spare time hemaintains his security portal at http://www.info-secure.orgPublished online: 21 Dec 2006.

To cite this article: Anton Chuvakin Ph.D., GCIA, GCIH (2004) Security Event Analysis through Correlation, InformationSystems Security, 13:2, 13-18, DOI: 10.1201/1086/44312.13.2.20040501/81648.3

To link to this article: http://dx.doi.org/10.1201/1086/44312.13.2.20040501/81648.3

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) containedin the publications on our platform. However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose ofthe Content. Any opinions and views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be reliedupon and should be independently verified with primary sources of information. Taylor and Francis shallnot be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and otherliabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

A C C E S S C O N T R O L S Y S T E M S A N D M E T H O D O L O G Y

M A Y / J U N E 2 0 0 4

13

Security Event Analysis through Correlation

Anton Chuvakin, Ph.D., GCIA, GCIH

he security spending survey byInformation Security (http://www.infosecuritymag.com/2003/may/

coverstory.pdf) and recent research by For-rester indicate that deployment rates ofmany security technologies will soar in thenext three years. According to some esti-mates, security budgets (and thus technol-ogy purchases) will double by 2006.

INTRODUCTION TO SECURITY DATA ANALYSIS

Almost every Internet-connected organiza-tion now has a firewall included as part of itsnetwork infrastructure; most Windows net-works have an anti-virus solution. Intrusiondetection systems (IDS) are slowly butsurely gaining wider acceptance, and intru-sion prevention is starting to show morepromise despite the obvious hurdles. Newtypes of application security products suchas Web application firewalls are starting tobe deployed by security-conscious organi-zations. This buying trend is furtherenhanced by the growing popularity of so-called “appliance” security systems, whichare very easy to install and manage. Appli-ances combine software and hardware inone package and usually have much lowerinstallation and maintenance costs, thusfacilitating their adoption.

All the above devices, whether aimed atprevention or detection of attacks, usuallygenerate huge volumes of audit data. Fire-walls, routers, switched, and other devicesrecording network connection informationare especially guilty of producing vastoceans of data.

There are other problems induced by thislog deluge, turning its analysis into a pursuitfew dare to undertake. Many diverse dataformats and representations, some binary,1

obscure, and undocumented, are used forthose log files and audit trails. Also, a per-centage of events generated by networkIDSs and intrusion prevention systems(IPSs) are false alarms and do not map toreal threats or map to threats that have nochance of causing loss.

To further confuse the issue, differentdevices might report on the same things hap-pening on the network, but in a different way,with no apparent way of figuring the truth oftheir relationship. For example, a UNIX logfile might contain an FTP connection mes-sage. The same will also be recorded by thefirewall as “connection allowed to TCP port21.” A network IDS might also generate analert, warning that FTP with no password hasoccurred. All three messages refer to the sameevent, and a human analyst will recognizethem as such.

T

A C C E S S C O N T R O L S Y S T E M S A N D M E T H O D O L O G Y

ANTON CHUVAKIN, Ph.D., GCIA, GCIH, is a Senior Security Analyst at a major security company. Hisareas of infosec expertise include intrusion detection, UNIX security, forensics, honeypots, etc. In hisspare time he maintains his security portal at http://www.info-secure.org

Dow

nloa

ded

by [

Car

negi

e M

ello

n U

nive

rsity

] at

00:

50 0

9 N

ovem

ber

2014

14 I N F O R M A T I O N S Y S T E M S S E C U R I T Y

M A Y / J U N E 2 0 0 4

However, programming a system to dothat is much more challenging, especiallyfor a broad spectrum of messages. Thus,there is a definite need for a consistent anal-ysis framework to identify various networkthreats, prioritize them, and learn theirimpact on the target organization. This mustbe done as fast as possible (preferably inreal-time) for attack identification and alsoover the long term for threat trending andrisk analysis.

To understand the meaning of the pilinglogs, the data in them can be categorized inseveral ways. It should be noted that beforethe data can be intelligently categorized, itshould be normalized to a common schema.The normalization process involves extract-ing the parts of the log records serving thecommon purpose and assigning them to spe-cific fields in the common schema. Forexample, both firewall and network IDS logrecords will usually contain the source anddestination IP addresses. If you see bothfirewall and IDS logs referring to the samesource and destination at about the sametime, they are likely related.

Log categorization helps make the simi-larity between different log records standout. For example, the generated log dataacross many security devices, hosts, andapplications might be related to:

■ Device performance data■ Network traffic■ Known attacks■ Known network/system problems■ Anomalous/suspicious network/host

activity■ Access control decisions■ Software failures■ Hardware errors■ System changes■ Evidence of malicious agents■ Site-specific AUP2 violations

Each of the above types of events pre-sents unique analysis challenges. For exam-ple, some are produced in much highernumbers (network access control, wormevents) while others are often not what theyseem at first (such as network IDS “false

positives”). Moreover, sometimes the threatcan only be identified and rated by cross-device and cross-category analysis of theabove events.

Many questions arise upon seeing theabove data. How do you turn that flood ofdata into useful and actionable information?How do you find what is really relevant forthe organization at the moment and for thenear future? How do you tell normal logrecords, produced in the course of business,from the anomalous and malicious, pro-duced by attackers or misbehaving soft-ware?

Correlation performed by SIM (SecurityInformation Management) software isbelieved to be the solution to those chal-lenges. Correlation is defined in the dictio-nary as establishing or finding relationshipsbetween entities. However, a good security-specific definition is lacking. In security,“event correlation” can be defined asimproving the threat identification andassessment process by looking not only atindividual events, but also at their sets,bound by some common parameter(“related”).

TYPES OF CORRELATIONSecurity-specific correlation can be looselycategorized into rule-based and statistical(or algorithmic). Rule-based correlationneeds some preexisting knowledge of theattack (“the rule”) and is able to define whatit actually detected in precise terms (“Suc-cessful Shopping Cart Web ApplicationAttack”). Such attack knowledge is used torelate events and analyze them together inbroader context.

On the other hand, statistical correlationdoes not employ any preexisting knowledgeof the “bad” activity (at least not as a pri-mary detection vehicle), but instead relieson the knowledge of normal activities, accu-mulated over time. Ongoing events are thenrated by the built-in algorithm and are addi-tionally compared to the accumulated activ-ity patterns.

This distinction is somewhat similar tosignature versus anomaly IDS and makes

Statistical correlation does not employ any preexisting knowledge of the “bad” activity, but instead relies on the knowledge of normal activities, accumulated over time.

Dow

nloa

ded

by [

Car

negi

e M

ello

n U

nive

rsity

] at

00:

50 0

9 N

ovem

ber

2014

A C C E S S C O N T R O L S Y S T E M S A N D M E T H O D O L O G Y

M A Y / J U N E 2 0 0 4

15

the SIM solution a kind of meta-IDS, oper-ating on higher-level data (not packets, butlog records). Both correlation methods com-bined can help to sift through the large vol-ume of diverse data and identify high-severity threats.

Rule-Based CorrelationRule-based correlation uses some preexist-ing knowledge of an attack (a rule), which isessentially a scenario that an attack mustfollow to be detected. Such a scenario mightbe encoded in the form of “if this, then that,therefore some action is needed.”

Rule-based correlation deals with states,conditions, timeouts, and actions. Let usdefine these important terms. A state is astationary occurrence that the correlationrule might be in. A state might contain vari-ous conditions, such as matching incomingevents by the source IP address, protocol,port, event type, producing security devicetype, username, and other components ofthe event. It should be noted that althoughsuch data components vary with the device,the SIM solution normalizes them using thecross-device event schema without incur-ring the information loss. Timeout defineshow long the rule will be in a certain state.If the correlation engine has to maintain alot of rules in waiting state in memory, thisresource might be exhausted. Thus, ruletimeouts play an important role in correla-tion performance. A transition is an eventwhen one rule state is switched to anotherone. For a complicated rule, many transi-tions are possible. Action is what happenswhen all the rule conditions are met. Vari-ous actions can result from rules, such asuser notification, alarm escalation, configu-ration changes, or automatic incident caseinvestigation.

The correlation is usually performed bythe correlation engine, which is able to trackvarious states and switch from state to state,depending on conditions and incomingevents. It does all the above for multiplerules at the same time. The correlationengine gets a real-time event feed from thealarm-generating security devices and

applies the relevant correlation rules asneeded. The correlation engine also lever-ages other types of available data (such asvulnerability, open port, or asset businessvalue information) for a higher level of cor-relation.

Correlation rules can be applied to theincoming events as they arrive in real-timeor to the historical events stored in the data-base. In the latter case, the rules are used asa form of data mining or analytics, whichallows for uncovering hidden threats such asslow port scans or low-level Trojan orexploitation activity. Such rules can be runperiodically for incident identification or inthe course of the investigation of suspiciousactivity for seeking out the prior occur-rences of similar (and thus possibly related)activity. Unlike the real-time rules, whichbecome useless if prone to false alarms (justas signature-based IDSs sometimes are),database rules can tolerate a certain level offalse alarms for the purpose of drasticallyreducing false negatives. This is due to thefact that real-time rules usually feed thealarm notification system, while databaserule correlation will be launched by the ana-lyst during the security incident investiga-tion. As long as the rule-based analytics willuncover a hidden threat, which is impossi-ble to discover otherwise, an analyst mightbe able to tolerate a certain level of falsealarms not acceptable for the real-time cor-relation.

Statistical CorrelationStatistical correlation uses special numericalgorithms to calculate threat levelsincurred by the security-relevant events onvarious IT assets. Such correlation looks fordeviations from normal event levels andother routine activities. Risk levels can becomputed from the incoming events andthen tracked in real-time or historically sothat deviations are apparent. The algorith-mic correlation can leverage the event cate-gorization in order to compute the threatlevels specific to various attack types, suchas a threat of denial-of-service, a threat ofviruses, etc., and track them over time.

Dow

nloa

ded

by [

Car

negi

e M

ello

n U

nive

rsity

] at

00:

50 0

9 N

ovem

ber

2014

16 I N F O R M A T I O N S Y S T E M S S E C U R I T Y

M A Y / J U N E 2 0 0 4

Detecting threats using statistical correla-tion does not require any preexisting knowl-edge of the attack to be detected. Statisticalmethods can, however, be used to detectthreats on predefined activity thresholds.Such thresholds can be configured based onthe experiences monitoring the environ-ment. For example, if a normal level of spe-cific reconnaissance activity is exceeded fora prolonged period of time, the alarm mightbe generated by the system.

Correlation can also use various parame-ters for enterprise assets to skew the statisti-cal algorithm for higher accuracy detection.Some of them are defined by system users(such as the affected asset value to the orga-nization) or are automatically computedfrom other available event context data(such as vulnerability scanning results ormeasure of normal user activity on theasset). That allows one to define a broadercontext for transpiring security events andthus helps one understand how they contrib-ute to the organization’s risk posture.

If rule-based correlation is more helpfulduring threat identification, then algorith-mic correlation is conducive to impactassessment. In the case of higher threat lev-els detected by the algorithms, one canassume that there is a higher chance of cata-strophic system compromise or failure. Var-ious statistical algorithms can be used totrend such threat levels over long periods oftime to gain awareness of the normal net-work and host activities. The accumulatedthreat data is then used to compare the cur-rent patterns of activity with the baseline.This allows the system to make accurate(and possibly automated) decisions aboutevent flows and their possible impact.

Challenges with CorrelationBoth of the above types of correlation haveinherent challenges, which can fortunatelybe mitigated by combining both methods tocreate coherent correlation coverage, leadingto quality threat identification and ranking.

First, can we assume that the attacker willfollow a scenario that can be caught by therule-based correlation system? Unlike the

network IDS that needs a specific signaturewith detailed knowledge of the attack, a cor-relation system rule might cover the broadrange of malicious activities, especially ifintelligent security event categorization isutilized. This can be done without goinginto the specifics of a particular IDS signa-ture. For example, rules can be written tolook for certain activities that usuallyaccompany the system compromise, such asbackdoor communication or hacker toolsdownload. Doing those things is more diffi-cult for the attacker to avoid if he intends touse the compromised machine for his ownpurposes. Extensive research using decep-tion networks (also called honeynets)allows one to learn more and more about theattacker’s patterns of behavior and toencode them as correlation rules, availableout of the box.

Second, can multiple rules cause thenumber of false positives to actuallyincrease instead of decrease? Indeed,deploying many rules without any regardfor the environment might generate falsealarms. However, it is much easier to under-stand and tune the SIM correlation rulesthan intricate binary matching patterns. Thelatter requires an in-depth understanding ofthe attack network packets, memory corrup-tion issues, and the specifics of the exploita-tion techniques. On the other hand, tuningthe correlation rule involves changing thetimeouts and adding or removing condi-tions. Overall, in the case of correlationrules, one can also define response actionswith higher confidence because one canbind the rules to a specific asset or group ofassets.

Third, rule-based correlation is relativelyintensive computationally. However, usinghighly optimized correlation engines andintelligently applying filters to limit theflow of events allows one to gain maximumadvantage of the rule-based correlation.Additionally, many rules can be combinedso that the correlation engine does not haveto keep many similar events in memory. Italso makes sense to apply more specific cor-relation rules to a large number of assets,

Detecting threats using statistical correlation does not require any preexisting knowledge of the attack to be detected.

Dow

nloa

ded

by [

Car

negi

e M

ello

n U

nive

rsity

] at

00:

50 0

9 N

ovem

ber

2014

A C C E S S C O N T R O L S Y S T E M S A N D M E T H O D O L O G Y

M A Y / J U N E 2 0 0 4

17

where a false positive flood might endangerthe security, and to apply wider and moregeneric rules to critical assets, where anoccasional false alarm is better than missinga single important alert. In this way, all thesuspicious activities directed against a smallgroup of critical assets will be detected, and

Fourth, statistical correlation might notpick up anomalous activity if it is performedat low enough levels, essentially mergingwith the normal. Hiding attack patternsunder volumes and volumes of similar nor-mal activity might deceive the statisticalcorrelation system. Similarly, a singleoccurrence of an attack might not impact thestatistical profile enough to be noticed.However, careful “baselining” of the envi-ronment and then using statistical methodsto track the deviations from such a baselinemight allow one to detect some of the low-volume threats. Also, rule-based correlationefficiency compensates for those rare eventsand enables their detection, even if algorith-mic correlation misses them.

MAXIMIZING THE BENEFITS OF CORRELATIONCorrelation enables system users to take theaudit data analysis to the next level. Rule-based and statistical correlation allows theuser to:

■ Dramatically decrease the response times for routine attacks and incidents using the centralized and correlated evidence stor-age

■ Completely automate the response to cer-tain threats that can be detected reliably by correlation rules

■ Identify malicious and suspicious activi-ties on the network even without having any preexisting knowledge of what to look for

■ Increase awareness of the network via baselining and trending and effectively “take back your network”

■ Fuse data from various information sources to gain a cross-device business risk view of the organization

■ Use the statistical correlation to learn the threats and then deploy new rules for site-specific and newly discovered violations

Overall, combining rules and algorithmsprovides the best value for managing anorganization’s IT security risks.

CORRELATION RULE EXAMPLESProbes Followed by an AttackThe rule watches for the general attack pat-tern consisting of a reconnaissance activity,followed by the exploit attempt. Attackersoften use activities such as port scanning orapplication querying to scope the environ-ment and find targets for exploitation andget an initial picture of system vulnerabili-ties. After performing the initial informationgathering, the attacker returns with exploitcode or automated attack tools to obtainactual system penetration. The correlationenriches the information reported by theIDS and serves to validate the attack andsuppress false alarms. By watching forexploit attempts that follow the reconnais-sance activity from the same source IPaddress against the same destinationmachine, the SIM solution can increase boththe confidence and accuracy of reporting.

After the reconnaissance event isdetected by the system, the rule activatesand waits for the actual exploit to bereported. If it arrives within a specifiedinterval, the correlated event is generated.The notification functionality can then beused to relay the event to security adminis-trators by email, pager, and cell phone or toinvoke appropriate actions.

Login GuessingThe rule watches for multiple attempts offailed authentication to network and hostservices followed by a successful log-inattempt. While some intrusion detectionsystems are able to alert on failed log-inattempts, the correlation system is able toanalyze such activity across all authenti-cated services, both networked (such as Tel-net, SSH, FTP, Windows access, etc.) andlocal (such as UNIX and Windows console

Dow

nloa

ded

by [

Car

negi

e M

ello

n U

nive

rsity

] at

00:

50 0

9 N

ovem

ber

2014

18 I N F O R M A T I O N S Y S T E M S S E C U R I T Y

M A Y / J U N E 2 0 0 4

log-ins). This rule is designed to track suc-cessful completion of such an attack. Trig-gering of this rule indicates that an attackermanaged to log in to one of your servers.

It is well-known that system users wouldoften use passwords that are easy to guessfrom just several tries. Intelligent automatedguessing tools, available to hackers, allowthem to cut the guessing time to a minimum.The tools use various tricks such as trying toderive a password from a user’s log-inname, last name, etc. In the case that thosesimple guessing attempts fail, hackers mightresort to “brute-forcing” the password. Thistechnique uses all possible combinations ofcharacters (such as letters and numbers) totry as a password. After the non-root (non-administrator) user password is successfullyobtained, the attacker will likely attempt toescalate privileges on the machine toachieve higher system privileges.

The rule activates after the first failedattempt is detected. The event counter isthen incremented until the threshold level isreached. At that point, the rule engine will beexpecting a successful log-in message. Incase such message is received, the correlatedevent is sent. It is highly suggested to tune

the count and the interval for the environ-ment. Up to three failed attempts within sev-eral minutes is usually associated with userstrying to remember the forgotten password,while higher counts within a shorter periodof time might be more suspicious and indi-cate a malicious attempt or a script-basedattack.

CONCLUSIONSIM products leveraging advanced correla-tion techniques and intelligent alert catego-rization are becoming indispensable asenterprises deploy more and more securitypoint solutions, appliances, and devices.Those solutions alone only address smallparts of a company’s security requirementsand need to be integrated under the umbrellaof a Security Information Managementsolution, which will enable the users tocombat modern-day technology threats suchas hackers, hybrid worms, and even internalabuse.

Notes1. Binary = here, not containing human-readable

text, but binary data.2. AUP = Acceptable Use Policy.

Intelligent automated guessing tools, available to hackers, allow them to cut the guessing time to a minimum.

Dow

nloa

ded

by [

Car

negi

e M

ello

n U

nive

rsity

] at

00:

50 0

9 N

ovem

ber

2014