Money$ec EvolvedWherein not everything has a tidy baseball

analogyJared PfostChief Executive OfficerThird Defense

Brian KeeferSecurity ArchitectLeading SaaS Security Company

Recap•Last year we applied baseball

“SABRmetrics” to InfoSec•We spent some time in the real

world•Oh yeah, some guy named Brad

was in a movie

In case you missed it

How Analytics Changed Baseball

Oakland A’s•Teams bid for players in Free Agent

market•Start of 2002 A’s had payroll

~$40M*•NY Yankees payroll ~$126M*•So poor teams have no shot at

winning, right?*From “Moneyball”

1999-2001Team Wins Losses Est

Payroll*

NYY 280 203 $257M

OAK 280 205 $70M

*Estimate from baseball-reference.com

Billy Beane•GM Billy Beane

defied convention

• i.e. he didn’t follow “best practices”

•made data-drive decisions

•Hired Paul DePodesta

Traditional baseball•Talent is evaluated by scouts•Scouts are usually washed-up

players•i.e. “Industry veterans” or

“experts”•Value statements are largely

subjective

Next-gen Baseball

•Started in 1977•Bill James wanted to see what

influenced game outcome•Realized stats created in 1859

didn’t properly attribute events

Key lessons•Don’t make emotional decisions•At least recognize your bias

•Collect the “right” data•Look for correlations

•Set reasonable criteria for success•Don’t overspend

This Applies to InfoSec

Problem statement

•Every organization is competing with attackers

•Most don’t have Fortune 50 budget•How can you be effective?

Conventional “wisdom”

•“Everyone knows” that you need•Firewall•Anti-virus•Change passwords frequently•Prohibit social networking•Etc.

Do they work?

•Port 80 goes through the firewall•Anti-virus misses custom malware•Stolen passwords used quickly•Social networking key to marketing

and employee satisfaction

Clearly this is not working

•Do we actually want a new strategy?

•What does winning look like?•How do we get started?

Cheap & Easy

Spend to Comply

Fix Gaps Now!

Ok, how much do we really need...?

Are You Ready To Win?

Motivating Event

•Winning is not losing...•No unacceptable risks realized •Cheap as possible

What Does Winning Look Like?

So, about that...• Started collecting info• Realized it was far from

complete• Historical incident

rates were meaningless

• Minimal ability to measure what helps

• 12 metricsMoney$ec 1.0

EvolutionMoney$ec 2.0

• Measure what’s easy

• Set Targets• Justify More• Optimize

Cost vs. Target

Start With “Easy”• Incidents - # of High, Moderate, Annoying• Application- # of Post-production application bugs

• Passwords- % passwords easily guessed• Scanned Vulnerabilities- # Patch & config vulns not mitigated per Severity Service

Level

Real Metrics Have Outcomes

• Stats are trendy, Metrics have Winners|Losers–Measure actual performance against target–Benefits

• Drives “acceptable risk” conversation with Management• Simplifies reporting e.g. are we above|below?

Back To “Easy”• Scanned Vulnerabilities

- # Patch & config vulns not mitigated per Severity Service Level- Sev 1 Server Vulns Mitigated within 30 days- Sev 2 within 60 days

You really can do this

Ooooh, shiny!

24

Expand Measurement• Access Management

- % Employee termination within policy- % Role/Access verification• Network- % critical systems monitored- Moving to % of full packet capture

• Vendors- % assessed per policy- # overdue findings• Employee- # of duplicate incidents• Change Management- # emergency or unplanned changes- % of changes with a regression

Every Metric Must Have A

Target

Optimize Cost - Target•Is target too high?

67

75

84

92

100

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb

Proposed Target

Cost - Benefit - AccountabilityRate Hrs Per

Test/Deploy# Personnel Cost Per

Server Update

$100/HR 40 10 $40,000

Evidence: Incidents, response performance, attack attempts

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10

DoS PostMalware Post

Worm Post

Or

http://code.google.com/p/openpert/

Current Target

Proposed Target

Improve IR•Move IR out of IT?•Infections are incidents•Data is needed to evaluate

controls•Knowing root-cause guides future

controls and Targets

Integrate Metrics Into Root Cause Analysis

Find Leading Indicators

Parting Thought

•People implicitly decide not to measure.

•Money$ec says explicitly decide when you don’t.

Security Reformation?

http://www.liquidmatrix.org/blog/2012/02/21/we-are-losing/

http://lifecypha.wordpress.com/

Time to Share

•Data you find useful to collect?•Spotted any correlations?•Proved any controls too expensive?•What communities do you

participate in?

Thanks!

Brian Keeferb: http://rants.effu.se

e: chort@effu.set: @chort0

Jared Pfostb: http://thirddefense.wordpress.com

e: jared@thirddefense.comt: @JaredPfost

appendix

Task InfoSec Control Owner Business Owner

Define Metric A,R R C

Define Target R R A,R

Report Metric A,R R I

Review Target A,R R R

R – ResponsibleA – AccountableC – Contribute

I - Informed(There can be only one “A”)

RACI in action

2011 VZ DBIR vs. Money$ec

Device Patch & Config Monitoring