fix what matters: a data driven approach to vulnerability management
DESCRIPTION
Data driven approach to vulnerability management in information security using live breach and vulnerability data.TRANSCRIPT
Fix What Matters Michael Roytman
SIRAcon October 21, 2013
Why You Should(n’t) Listen
• Naive Grad Student Not Too Long Ago• Still Plays With Legos• Barely Passed Regression Analysis
• MS Operations Research, Georgia Tech
Michael Roytman• Data Scientist, Risk I/O
• Fraud Detection, Large Bank
Roadmap
• The Struggle
• What’s Good?
• Data Driven Insights• Framework
• Decision-Making
• What’s Bad?
Starting From Scratch
“It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories,
instead of theories to suit facts.”
-Sir Arthur Conan Doyle, 1887
Starting From Scratch
Starting From Scratch
Academia!• GScholar!• JSTOR!• IEEE!• ProQuest!
InfoSec Blogs!• CSIOs!• Pen Testers!• Threat Reports!• SOTI/DBIR!!
Twitter!• Thought Leaders (you
know who you are)!• BlackHats!• Vuln Researchers!
Primary Sources!• MITRE!• OSVDB!• NIST CVSS
Committee(s)!• Internal Message
Boards for ^!Text
CISOs
Data Fundamentalism
Don’t Ignore What a Vulnerability Is: Creation Bias !
(http://blog.risk.io/2013/04/data-fundamentalism/) !
Jerico/Sushidude @ BlackHat !
(https://www.blackhat.com/us-13/briefings.html#Martin)!
Luca Allodi - CVSS DDOS !
(http://disi.unitn.it/~allodi/allodi-12-badgers.pdf):!
Data Fundamentalism - What’s The Big Deal?
!
”Since 2006 Vulnerabilities have declined by 26 percent.” !(http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf)!
!
!
“The total number of vulnerabilities in 2013 is up 16 percent so far when compared to what we saw in the same time period in 2012. ”!
(http://www.symantec.com/content/en/us/enterprise/other_resources/b-intelligence_report_06-2013.en-us.pdf)!!
!
What’s Good?
Bad For Vulnerability Statistics:!!
NVD, OSVDB, ExploitDB, CVSS, Patches, Microsoft Reports, etc, et al, and so on. !
Good For Vulnerability Statistics:!!
Vulnerabilities. !
Data Is Everything And Everything Is Data.
What’s Good?
What’s Good?
What’s Good?
What’s Good?
What’s Good?
What’s Good?
Counterterrorism
Known Groups
Surveillance
Threat Intel, Analysts
Targets, Layouts
Past Incidents, Close Calls
What’s Good?
Uh, Sports?
Opposing Teams, Specific Players
Gameplay
Scouting Reports, Gametape
Roster, Player Skills
Learning from Losing
InfoSec?
Defend Like You’ve Done It Before
Groups, Motivations
Exploits
Vulnerability Definitions
Asset Topology, Actual Vulns on System
Learning from Breaches
Work With What You’ve Got:
Akamai, Safenet
ExploitDB, Metasploit
NVD, MITRE
Add Some Spice
Show Me The Money
23,000,000 Vulnerabilities!
Across 1,000,000 Assets!
Representing 9,500 Companies!
Using 22 Unique Scanners!
Whatchu Know About Dat?(a)
!
Duplication
Vulnerability Density
Remediation
Duplication
0
225,000
450,000
675,000
900,000
1,125,000
1,350,000
1,575,000
1,800,000
2,025,000
2,250,000
2 or more scanners 3 or more 4 or more 5 or more 6 or more
Duplication
We Have: F(Number of Scanners) => Number of Duplicate Vulnerabilities
We Want: F(Number of Scanners) => Vulnerability Coverage
Make Decisions At The Margins!
<---------Good Luck!
0.0
25.0
50.0
75.0
100.0
0 1 2 3 4 5 6
Density
Type of Asset ~Count
Hostname 20,000
Netbios 1000
IP Address 200,000
File 10,000
Url 5,000
Hostname
Netbios
IP
File
Url
0.0 22.5 45.0 67.5 90.0
CVSS And Remediation Metrics
0.0
350.0
700.0
1050.0
1400.0
1 2 3 4 5 6 7 8 9 10
Average Time To Close By Severity Oldest Vulnerability By Severity
CVSS And Remediation - Lessons From A CISORemediation/Lack Thereof, by CVSS
1 2 3 4 5 6 7 8 9 10
NVD Distribution by CVSS
The Kicker - Live Breach Data
1,500,000 !Vulnerabilities Related to Live Breaches Recorded!
June, July 2013 !
CVSS And Remediation - Nope
0.0
1750.0
3500.0
5250.0
7000.0
1 2 3 4 5 6 7 8 9 10
Oldest Breached Vulnerability By Severity
CVSS - A VERY General Guide For Remediation - Yep
0.0
40000.0
80000.0
120000.0
160000.0
1 2 3 4 5 6 7 8 9 10
Open Vulns With Breaches Occuring By Severity
The One Billion Dollar Question
Probability(You Will Be Breached On A Particular Open Vulnerability)?
1.98%=(Open Vulnerabilities | Breaches Occurred On Their CVE)/(Total Open Vulnerabilities)
I Love It When You Call Me Big Data
Probability A Vulnerability Having Property X Has Observed Breaches
RANDOM VULN
CVSS 10
CVSS 9
CVSS 8
CVSS 6
CVSS 7
CVSS 5
CVSS 4
Has Patch
0.00000 0.01000 0.02000 0.03000 0.04000
What’s the Alternative?
I Love It When You Call Me Big Data
Probability A Vulnerability Having Property X Has Observed Breaches
Random Vuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.0 0.1 0.2 0.2 0.3
Data Is Everything And Everything Is Data.
Be Better Than The Gap
I Love It When You Call Me Big Data
Spray and Pray => 2% !
CVSS 10 => 4% !
Metasploit + ExploitDB => 30% !
A Good Model That’s Not Built By One Kid Without Hadoop => ???!
