evidential alert correlation for network intrusion...
TRANSCRIPT
Evidential Alert Correlation for Network Intrusion Analysis
DSCS Workshop – 27 September 2017
Outline
• Motivating Problems
• Proposed Solution
• Conclusion
Intrusion Detection
• Intrusion detection together with other system defences, e.g. firewalls, provides the primary means of misuse identification and response
Motivating Problems
• Issues in Intrusion Detection – Tons of alerts, possibly up to 20,000 per day– Many false alarms– Most alerts are not isolated, but related to
different stages of attacks– Hard to make sense out of a large pile of
alerts
Security Event Analytics
• Challenges – Low level detections are not always reliable – uncertain evidence– There are many ways to perform an attack – heuristic attack structures – An attack may be successful through actions over several connected stages –
progressing process
• Our solution– Evidential network reasoning, based on Dempster-Shafer theory of evidence– Numerically model sensor detections and relationships between sensor detection and
security state– Provide operations of combination, extension and marginalisation for reasoning– Answer to the questions such as
• What does an alert instance mean to system security state, exploited or compromised?• With a bundle of alerts at hand, has the system been targeted by DDoS attack?• How sure about analysed security state?
Overview of the evidential alert correlation system
Evidential Alert Correlation
67762
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetTerminaltype
67763
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetXdisplay
67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67766
10/11/2
00106:054
926219
4.007.248
.1532
5172.016
.112.207E
mail_
Ehlo67767
10/11/2
00106:053
379917
2.016.115
.0209
325255
.255.255.25
5Mstrea
m_Zo
mbie
67768
10/11/2
00106:051
593717
2.016.113
.1052
5195.073
.151.050E
mail_
Ehlo67769
10/11/2
00106:051
599617
2.016.113
.2072
5197.182
.091.233E
mail_
Ehlo67770
10/11/2
00106:051
599817
2.016.112
.1942
3172.016
.112.050T
elnetTerminaltype
67771
10/11/2
00106:054
926313
5.008.060
.1822
5172.016
.113.105E
mail_
Ehlo67772
10/11/2
00106:051
640017
2.016.113
.1052
3172.016
.113.050T
elnetTerminaltype
67773
10/11/2
00106:066
452078
.111.082.04
125471
31.084.00
1.031Stream
_DoS
67774
10/11/2
00106:061
640417
2.016.113
.2042
5194.027
.251.021E
mail_
Ehlo67775
10/11/2
00106:061
641217
2.016.112
.2072
3194.027
.251.021T
elnetTerminaltype
67776
10/11/2
00106:063
380017
2.016.115
.0207
983172
.016.112.05
0Mstrea
m_Zo
mbie
67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67766
10/11/2
00106:054
926219
4.007.248
.1532
5172.016
.112.207E
mail_
Ehlo67762
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetTerminaltype
67763
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetXdisplay
67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67766
10/11/2
00106:054
926219
4.007.248
.1532
5172.016
.112.207E
mail_
Ehlo67767
10/11/2
00106:053
379917
2.016.115
.0209
325255
.255.255.25
5Mstrea
m_Zo
mbie
67768
10/11/2
00106:051
593717
2.016.113
.1052
5195.073
.151.050E
mail_
Ehlo67769
10/11/2
00106:051
599617
2.016.113
.2072
5197.182
.091.233E
mail_
Ehlo67770
10/11/2
00106:051
599817
2.016.112
.1942
3172.016
.112.050T
elnetTerminaltype
67771
10/11/2
00106:054
926313
5.008.060
.1822
5172.016
.113.105E
mail_
Ehlo67772
10/11/2
00106:051
640017
2.016.113
.1052
3172.016
.113.050T
elnetTerminaltype
67773
10/11/2
00106:066
452078
.111.082.04
125471
31.084.00
1.031Stream
_DoS
67774
10/11/2
00106:061
640417
2.016.113
.2042
5194.027
.251.021E
mail_
Ehlo67775
10/11/2
00106:061
641217
2.016.112
.2072
3194.027
.251.021T
elnetTerminaltype
67776
10/11/2
00106:063
380017
2.016.115
.0207
983172
.016.112.05
0Mstrea
m_Zo
mbie
67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67766
10/11/2
00106:054
926219
4.007.248
.1532
5172.016
.112.207E
mail_
Ehlo67762
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetTerminaltype
67763
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetXdisplay
67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67766
10/11/2
00106:054
926219
4.007.248
.1532
5172.016
.112.207E
mail_
Ehlo67767
10/11/2
00106:053
379917
2.016.115
.0209
325255
.255.255.25
5Mstrea
m_Zo
mbie
67768
10/11/2
00106:051
593717
2.016.113
.1052
5195.073
.151.050E
mail_
Ehlo67769
10/11/2
00106:051
599617
2.016.113
.2072
5197.182
.091.233E
mail_
Ehlo67770
10/11/2
00106:051
599817
2.016.112
.1942
3172.016
.112.050T
elnetTerminaltype
67771
10/11/2
00106:054
926313
5.008.060
.1822
5172.016
.113.105E
mail_
Ehlo67772
10/11/2
00106:051
640017
2.016.113
.1052
3172.016
.113.050T
elnetTerminaltype
67773
10/11/2
00106:066
452078
.111.082.04
125471
31.084.00
1.031Stream
_DoS
67774
10/11/2
00106:061
640417
2.016.113
.2042
5194.027
.251.021E
mail_
Ehlo67775
10/11/2
00106:061
641217
2.016.112
.2072
3194.027
.251.021T
elnetTerminaltype
67776
10/11/2
00106:063
380017
2.016.115
.0207
983172
.016.112.05
0Mstrea
m_Zo
mbie
67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67766
10/11/2
00106:054
926219
4.007.248
.1532
5172.016
.112.207E
mail_
Ehlo67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67768
10/11/2
00106:051
593717
2.016.113
.1052
5195.073
.151.050E
mail_
Ehlo67769
10/11/2
00106:051
599617
2.016.113
.2072
5197.182
.091.233E
mail_
Ehlo
67762
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetTerminaltype
67763
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetXdisplay
67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67766
10/11/2
00106:054
926219
4.007.248
.1532
5172.016
.112.207E
mail_
Ehlo67767
10/11/2
00106:053
379917
2.016.115
.0209
325255
.255.255.25
5Mstrea
m_Zo
mbie
67768
10/11/2
00106:051
593717
2.016.113
.1052
5195.073
.151.050E
mail_
Ehlo67769
10/11/2
00106:051
599617
2.016.113
.2072
5197.182
.091.233E
mail_
Ehlo67770
10/11/2
00106:051
599817
2.016.112
.1942
3172.016
.112.050T
elnetTerminaltype
67771
10/11/2
00106:054
926313
5.008.060
.1822
5172.016
.113.105E
mail_
Ehlo67772
10/11/2
00106:051
640017
2.016.113
.1052
3172.016
.113.050T
elnetTerminaltype
67773
10/11/2
00106:066
452078
.111.082.04
125471
31.084.00
1.031Stream
_DoS
67774
10/11/2
00106:061
640417
2.016.113
.2042
5194.027
.251.021E
mail_
Ehlo67775
10/11/2
00106:061
641217
2.016.112
.2072
3194.027
.251.021T
elnetTerminaltype
67776
10/11/2
00106:063
380017
2.016.115
.0207
983172
.016.112.05
0Mstrea
m_Zo
mbie
67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67766
10/11/2
00106:054
926219
4.007.248
.1532
5172.016
.112.207E
mail_
Ehlo67762
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetTerminaltype
67763
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetXdisplay
67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67766
10/11/2
00106:054
926219
4.007.248
.1532
5172.016
.112.207E
mail_
Ehlo67767
10/11/2
00106:053
379917
2.016.115
.0209
325255
.255.255.25
5Mstrea
m_Zo
mbie
67768
10/11/2
00106:051
593717
2.016.113
.1052
5195.073
.151.050E
mail_
Ehlo67769
10/11/2
00106:051
599617
2.016.113
.2072
5197.182
.091.233E
mail_
Ehlo67770
10/11/2
00106:051
599817
2.016.112
.1942
3172.016
.112.050T
elnetTerminaltype
67771
10/11/2
00106:054
926313
5.008.060
.1822
5172.016
.113.105E
mail_
Ehlo67772
10/11/2
00106:051
640017
2.016.113
.1052
3172.016
.113.050T
elnetTerminaltype
67773
10/11/2
00106:066
452078
.111.082.04
125471
31.084.00
1.031Stream
_DoS
67774
10/11/2
00106:061
640417
2.016.113
.2042
5194.027
.251.021E
mail_
Ehlo67775
10/11/2
00106:061
641217
2.016.112
.2072
3194.027
.251.021T
elnetTerminaltype
67776
10/11/2
00106:063
380017
2.016.115
.0207
983172
.016.112.05
0Mstrea
m_Zo
mbie
67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67766
10/11/2
00106:054
926219
4.007.248
.1532
5172.016
.112.207E
mail_
Ehlo67762
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetTerminaltype
67763
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetXdisplay
67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67766
10/11/2
00106:054
926219
4.007.248
.1532
5172.016
.112.207E
mail_
Ehlo67767
10/11/2
00106:053
379917
2.016.115
.0209
325255
.255.255.25
5Mstrea
m_Zo
mbie
67768
10/11/2
00106:051
593717
2.016.113
.1052
5195.073
.151.050E
mail_
Ehlo67769
10/11/2
00106:051
599617
2.016.113
.2072
5197.182
.091.233E
mail_
Ehlo67770
10/11/2
00106:051
599817
2.016.112
.1942
3172.016
.112.050T
elnetTerminaltype
67771
10/11/2
00106:054
926313
5.008.060
.1822
5172.016
.113.105E
mail_
Ehlo67772
10/11/2
00106:051
640017
2.016.113
.1052
3172.016
.113.050T
elnetTerminaltype
67773
10/11/2
00106:066
452078
.111.082.04
125471
31.084.00
1.031Stream
_DoS
67774
10/11/2
00106:061
640417
2.016.113
.2042
5194.027
.251.021E
mail_
Ehlo67775
10/11/2
00106:061
641217
2.016.112
.2072
3194.027
.251.021T
elnetTerminaltype
67776
10/11/2
00106:063
380017
2.016.115
.0207
983172
.016.112.05
0Mstrea
m_Zo
mbie
67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67766
10/11/2
00106:054
926219
4.007.248
.1532
5172.016
.112.207E
mail_
Ehlo67764
10/11/2
00106:044
921220
2.077.162
.2132
3172.016
.115.020T
elnetEnvAll
67765
10/11/2
00106:054
926119
4.007.248
.1532
5172.016
.113.084E
mail_
Ehlo67768
10/11/2
00106:051
593717
2.016.113
.1052
5195.073
.151.050E
mail_
Ehlo67769
10/11/2
00106:051
599617
2.016.113
.2072
5197.182
.091.233E
mail_
Ehlo
Intrusionalerts
Alertvalidation
EvidentialNetworkInferenceAlertDuplicationAlertFusion
HyperalertExtraction
Alertcorrelationengine Attackscenario
LOCAL GLOBAL
Local Correlation
• Alert validation– Keep alerts of relevant signature types
• Alert duplication– Remove repeated alerts
• Alert fusion– Aggregate alerts of same signature within time
window, satisfying certain conditions
• Hyper alert extraction– Merge alerts of different signature types
corresponding to same attack
Global Correlation
• Correlate hyper alerts corresponding to different stages of a complex attack
• Based on evidential network reasoning
• Knowledge base contains evidential network model of the attack
• Dempster-Shafer theory of evidence provides the foundation for attack modelling, uncertainty representation, and information inference
DS Theory of Evidence
• Represents system with a set of variables V = {v1, …, vn}
• Domain D = {x}
• Frame of discernment Q
• wi is a value of x
• Mass function defined on the power set of Q
Θ = {𝑤%, … ,𝑤) }
𝑚: 2. → [0,1]
𝑚 ∅ = 0,5 𝑚(𝑤7)�
:;⊆.= 1
Evidential Networks
• V is the set of variables
• QV is the set of frames
• MV is the set of mass
functions
• Combination
• Marginalisation• Extension
Relation Implication Rule
• Domain knowledge represented by IF-THEN rule
• Degree of confidence to measure uncertainty
• If A then B with degree of confidence 𝜌 ∈ [𝛼, 𝛽]where 0 ≤ 𝛼 ≤ 𝛽 ≤ 1
DS Frame Representation
• Domain knowledge represented by IF-THEN rule
• Degree of confidence to measure uncertainty
• If A then B with degree of confidence 𝜌 ∈ [𝛼, 𝛽] where 0 ≤ 𝛼 ≤ 𝛽 ≤ 1
with𝜌 ∈ 𝛼, 𝛽 , 0 ≤ 𝛼 ≤ 𝛽 ≤ 1.
𝐴 ⊆ ΘHI ⟹ 𝐵 ⊆ ΘHL
𝑚HM = N𝛼𝑖𝑓𝐶 = (𝐵×𝐴) ∪ (ΘHL×𝐴
T)1 − 𝛽𝑖𝑓𝐶 = (𝐵T×𝐴) ∪ (ΘHL×𝐴
T)𝛽 − 𝛼𝑖𝑓𝐶 = ΘHL×ΘHI
From Attack Tree
Sadmind_Ping
SadmindBOFVulnerableSadmind
Rsh
Mstream_Zombie
StreamDOS
AccessControl
SystemCompromised
ReadyToLaunchDDOS
LaunchDDoS
Intrusion action
System state
IPSweep
From Attack Tree
To Evidential Network Model
Evidential Network Model
𝑑%W = {𝐼𝑆𝑠}Domain of variablesΘ%W = {1,0}Frame of discernment
𝑚%W 1 = 0.9;𝑚%W 1,0 = 0.1
Mass function
evidence
Evidential Network Model
𝑑] = {𝐼𝑆𝑎, 𝐼𝑆𝑠}Domain of variables
Θ] = { 1,1 , 1,0 , 0,1 , (0,0)}
Frame of discernment
𝑚] 1,1 , (1,0) = 0.245;
Mass function
knowledge
𝑚] 1,1 , 1,0 , (0,0) = 0.325𝑚] 1,1 , 1,0 , (0,1) =0.185
𝑚] Θ] =0.245
Implication rulesIss à ISa [0.57, 1]~Iss à ISa [0.43, 1]
Evidential Inference
𝑚%W 1 = 0.9;𝑚%W 1,0 = 0.1
Evidence
𝑚′%c = 𝑚%Wdef↑dehExtension 𝑚′′%c = 𝑚]
di↑deh
Evidence Propagation
𝑚%c = 𝑚′%c ⊕𝑚′′%cCombination
𝑚′%k = 𝑚%cdeh↑delExtension 𝑚′′%k = 𝑚m
dn↑del
𝑚%k = 𝑚′%k ⊕𝑚′′%kCombination
𝑚′W% = 𝑚%kdel↓dfeMarginalisation
Forward Propagation
Experiments
• DARPA 2000 dataset
• Two DDOS attack scenarios– LLDoS 1.0: inside and dmz– LLDoS 2.0.2: inside and dmz
• RealSecure alert files
Results
DatasetObservableattacks#
RealSecure Ourmethod
Alerts#Detectedattacks#
Alerts#Detectedattacks#
Attackdetection%
LLDOS1.0
Inside 60 922 37 61 37 100DMZ 89 886 51 92 51 100
LLDOS2.0.2
Inside 15 489 12 23 12 100DMZ 7 425 4 8 4 100
LLDOS1.0 LLDOS2.0.2Inside DMZ Inside DMZ
Relatedalerts 61 96 25 8Correlatedalerts 61 95 23 8Correctlycorrelatedalerts 61 95 23 8Completeness%(correctlycorrelated/related)
100 98.96 92.00 100
Soundness%(correctlycorrelated/correlated)
100 100 100 100
Conclusions and Future Work
• Proposed an alert correlation technique– Evidential network reasoning– Models uncertain sensor detections and relationship
knowledge– Numerically infers security state changes to draw a
semantic view of attack
• Future work– Learning evidential network model of attack from
domain experts and data– Recognising the variations of attack – Real-time correlation
Thanks
Questions?