Download - Money$ec Evolved
Money$ec EvolvedWherein not everything has a tidy baseball
analogyJared PfostChief Executive OfficerThird Defense
Brian KeeferSecurity ArchitectLeading SaaS Security Company
Recap•Last year we applied baseball
“SABRmetrics” to InfoSec•We spent some time in the real
world•Oh yeah, some guy named Brad
was in a movie
In case you missed it
How Analytics Changed Baseball
Oakland A’s•Teams bid for players in Free Agent
market•Start of 2002 A’s had payroll
~$40M*•NY Yankees payroll ~$126M*•So poor teams have no shot at
winning, right?*From “Moneyball”
1999-2001Team Wins Losses Est
Payroll*
NYY 280 203 $257M
OAK 280 205 $70M
*Estimate from baseball-reference.com
Billy Beane•GM Billy Beane
defied convention
• i.e. he didn’t follow “best practices”
•made data-drive decisions
•Hired Paul DePodesta
Traditional baseball•Talent is evaluated by scouts•Scouts are usually washed-up
players•i.e. “Industry veterans” or
“experts”•Value statements are largely
subjective
Next-gen Baseball
•Started in 1977•Bill James wanted to see what
influenced game outcome•Realized stats created in 1859
didn’t properly attribute events
Key lessons•Don’t make emotional decisions•At least recognize your bias
•Collect the “right” data•Look for correlations
•Set reasonable criteria for success•Don’t overspend
This Applies to InfoSec
Problem statement
•Every organization is competing with attackers
•Most don’t have Fortune 50 budget•How can you be effective?
Conventional “wisdom”
•“Everyone knows” that you need•Firewall•Anti-virus•Change passwords frequently•Prohibit social networking•Etc.
Do they work?
•Port 80 goes through the firewall•Anti-virus misses custom malware•Stolen passwords used quickly•Social networking key to marketing
and employee satisfaction
Clearly this is not working
•Do we actually want a new strategy?
•What does winning look like?•How do we get started?
Cheap & Easy
Spend to Comply
Fix Gaps Now!
Ok, how much do we really need...?
Are You Ready To Win?
Motivating Event
•Winning is not losing...•No unacceptable risks realized •Cheap as possible
What Does Winning Look Like?
So, about that...• Started collecting info• Realized it was far from
complete• Historical incident
rates were meaningless
• Minimal ability to measure what helps
• 12 metricsMoney$ec 1.0
EvolutionMoney$ec 2.0
• Measure what’s easy
• Set Targets• Justify More• Optimize
Cost vs. Target
Start With “Easy”• Incidents - # of High, Moderate, Annoying• Application- # of Post-production application bugs
• Passwords- % passwords easily guessed• Scanned Vulnerabilities- # Patch & config vulns not mitigated per Severity Service
Level
Real Metrics Have Outcomes
• Stats are trendy, Metrics have Winners|Losers–Measure actual performance against target–Benefits
• Drives “acceptable risk” conversation with Management• Simplifies reporting e.g. are we above|below?
Back To “Easy”• Scanned Vulnerabilities
- # Patch & config vulns not mitigated per Severity Service Level- Sev 1 Server Vulns Mitigated within 30 days- Sev 2 within 60 days
You really can do this
Ooooh, shiny!
24
Expand Measurement• Access Management
- % Employee termination within policy- % Role/Access verification• Network- % critical systems monitored- Moving to % of full packet capture
• Vendors- % assessed per policy- # overdue findings• Employee- # of duplicate incidents• Change Management- # emergency or unplanned changes- % of changes with a regression
Every Metric Must Have A
Target
Optimize Cost - Target•Is target too high?
67
75
84
92
100
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb
Proposed Target
Cost - Benefit - AccountabilityRate Hrs Per
Test/Deploy# Personnel Cost Per
Server Update
$100/HR 40 10 $40,000
Evidence: Incidents, response performance, attack attempts
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 8 9 10
DoS PostMalware Post
Worm Post
Or
http://code.google.com/p/openpert/
Current Target
Proposed Target
Improve IR•Move IR out of IT?•Infections are incidents•Data is needed to evaluate
controls•Knowing root-cause guides future
controls and Targets
Integrate Metrics Into Root Cause Analysis
Find Leading Indicators
Parting Thought
•People implicitly decide not to measure.
•Money$ec says explicitly decide when you don’t.
Security Reformation?
http://www.liquidmatrix.org/blog/2012/02/21/we-are-losing/
http://lifecypha.wordpress.com/
Time to Share
•Data you find useful to collect?•Spotted any correlations?•Proved any controls too expensive?•What communities do you
participate in?
Thanks!
Brian Keeferb: http://rants.effu.se
e: [email protected]: @chort0
Jared Pfostb: http://thirddefense.wordpress.com
e: [email protected]: @JaredPfost
appendix
Task InfoSec Control Owner Business Owner
Define Metric A,R R C
Define Target R R A,R
Report Metric A,R R I
Review Target A,R R R
R – ResponsibleA – AccountableC – Contribute
I - Informed(There can be only one “A”)
RACI in action
2011 VZ DBIR vs. Money$ec
Device Patch & Config Monitoring
