Applied Quantitative Cyber Risk AnalysisMichael Rich, OSCP, CISSPDirector of IT Security, Infrastructure & OperationsMotion Picture Industries Pension & Health Care Plan

| 2 |

Disclaimer for those reading from the ISACA link

My talks are image and slide-build heavy. So they don’t “print” well. Sorry about that.

| 3 |

Agenda

Seek Beyond Your Interest– “Risk Analysis: Don’t Just Trust Your Gut” – Evan Wheeler @

BSidesLA 2016 The Idea:

– What is a Risk?– The Calibration of the Experts– Monte Carlo Risk simulation– A Cyber Risk Model Example The Application:

– Risk Decomposition– Gedanken Experiments– “The SHOCKING truth about probability they don’t want you to

know!!!”– Snowflakes and Monte Carlo– Equivalent Life Event Probabilities Now What?

| 4 |

The Idea

| 5 |

What is a Risk?

An event that has some chance of happening and causes effects we don’t want.

Qualitative Analysis

Source credit: https://www.risklens.com/blog/4-steps-to-a-smarter-risk-heat-map

Quantitative Analysis

| 6 |

What is a Risk?

Probability of Occurrence– Numerically-expressed probability– Can be a range to express uncertainty i.e.: 9-14% chance

Impact (Loss)– Numerically expressed range: Upper bound Lower bound 90% confidence

– Used with a log-normal distribution 5% values are < Lower bound 5% of values are > Upper bound Black Swans!

Log-normal distribution example

| 7 |

Log Normal – In Real Life

Image from Blackline.com

| 8 |

What is a Risk?

Estimated over given time period A basic risk:

– Tomorrow, if a traffic incident occurs during my morning commute, I will be delayed– Probability of occurrence: 30%– Impact (90% confidence): 5 – 60 minute delay from normal commute time

| 9 |

Subjective Range EstimationAKA The Calibration of the Experts

The Equivalent Bet: for 1000 Imperial Credits would you rather– See if the answer is in your interval– Spin the dial?

Win it all

Win nothing

What is the stated capacity of Wembley Stadium in London?

Capacity: 90,000

This slide covered on purpose so we don’t ruin the fun at the event!!

| 10 |

Monte Carlo Simulation

Iterate over probability of occurrence and generate random impactsMany times (100K+)

Probability: 30%Impact, Upper bound: 60Impact, Lower bound: 5Number Trials 10001Trial Delay

1 02 14.552443 17.377024 16.649685 06 07 08 09 0

10 49.68741

Example:

| 11 |

Sim Results and the Loss Exceedance Curve

| 12 |

Reducing Loss Exceedance Curves

Curves are pretty, but I need a number!– Ranking– Comparison– Mitigation effectiveness

In insurance world:– Average Annual Loss = Premium– “Area under the curve”

For Commute:– Average Event Impact– 6.8 minutes…. But…

241 Minute MAX impact

| 13 |

Methodology Demonstration – The Shared Home ComputerCost chosen as impact only for purposes of this example

Banking Trojan

Probability 5%

Max Impact $25,000 ($35,000)

Min Impact $500

Ransomware

Probability 10%

Max Impact $3000

Min Impact $200

Creepy Spyware

Probability 2%

Max Impact $2000 ($5000)

Min Impact $300

Clumsy Cat

Probability 5%

Max Impact $3000

Min Impact $750

Amazon Spree

Probability 30%

Max Impact $750

Min Impact $150

Risks over next 6 months

| 15 |

Simulation Results (100K iterations)Use Case: Ranking Risks

Total Expected Average Loss

$638

Banking Trojan $317

Amazon Spree $112

Ransomware $110

Clumsy Cat $80

Creepy Spyware $19

| 16 |

The Application

| 17 |

Risk Decomposition

Break your risk effects down into chunks– Measureable and observable– Company dependent

Manpower Costs– Business Departments– Leadership

Remediation Costs– IR Retainer– Legal– Hardware– Software

| 18 |

Risk Decomposition

LBUBCapLBUBCap

LBUBCapLBUBCap

LBUBCapLBUBCap

$/Hr

TimeSecurity

Active?

Time

$/Hr

IT Leadership

Active?

Time

$/Hr

IT Ops

Active?

LBUBCapLBUBCap

LBUBCapLBUBCap

LBUBCapLBUBCap

Time

$/Hr

Retirements

Active?

Time

$/Hr

PSC

Active?

Accounting

Active?

Time

$/Hr

LBUBCap

LBUBCap

LBUBCap

LBUBCap

IR Retainer

Active?

Cost

Legal

Active?

Cost

Active?

CostHardware

Software

Active?

Cost

| 19 |

Gedanken Experiments

| 20 |

The ONE SHOCKING Truth About Probability

Aggregate probability is a bitch… 2 times in 120 days, I escalated a security event to the CIOWhat are the odds I have to escalate an issue any given day:

– Odds: 2/120– Probability [Odds/(1+Odds)]: 1.64%

What is the probability (p) I’ll have an event in the next 6 months I have to escalate?Well:

– Probability (p-not) of it not happening [1-p]: 98.4%– Probability of it not happening for 120 consecutive days [(p-not)^120]: 14.4%– Probability of an escalated event in 120 days [1-(not happening)]: 85.6%

| 21 |

Is Monte Carlo a Precious Snowflake?(Sensitivity Analysis)

3 independent variables. How sensitive is the Average Event Loss?Probability Lower Bound Upper Bound

| 22 |

Monte Carlo IS a Precious Snowflake.. Probably

| 23 |

Ooof.. It’s Even Worse Than I Thought

| 24 |

Handling the Snowflake

Must include uncertainty in your probability estimate (i.e. a range) Instead of 1% probability, let’s make it 0.5% - 1.5% (50% error bar)

Test AEL($)

1% Fixed $72

1% +/- .5% $70

| 25 |

Beta Distribution

Single: $71.79Uniform: $71.15Beta: $71.63

Test EAL ($)

1% fixed $71.79

1% +/- 0.5% $71.15

1% Beta $71.63

| 26 |

Some More Experiments

Test EAL ($)

5% fixed $367

5% +/- 4% $355

5% Beta $356

| 27 |

Some More Experiments

Test EAL ($)

5% fixed $350

5% +/- 4% $349

4% +/- 3% $293

4% fixed $277

| 29 |

Statistically Equivalent Probabilities

100% - 50% 50% - 10% 10% 3% 1.5% 1% 0.8% 0.02%

| 30 |

Beta Distribution: Establish Probability from Test Cases

If you have a set of cases, you can get a probability distribution

| 32 |

Using Probability for Complicated Scenarios

Calibrate expertAsk expert to assess probability of the event given no other data

– “What is the probability of an adversary managing to inject code on the target system in the next 6 months?”

Ask expert to re-assess given various conditions– “What if the firewalls are discovered to be misconfigured?”– “What if a Cooperative Vulnerability Inspection team demonstrates code injection?”– “What if a black-box adversarial assessment team demonstrates it?”

Use Log-Odds-Ratio– Statistically valid method for combining the effects of multiple conditions on a final

probability

| 33 |

Log Odds Ratio ExampleUse Case: Using expert knowledge

Initial Prob: P(E) 1.0%

Firewall CVI Finding Adversarial Software Adversary Intel Rogue Network Device Rogue USBP(E|X1) Correct Config Code Injection Code Injection Updated 1 Hop away Detected DetectedP(E|X2) Incorrect Config No Injection No Injection Not Updated 2 Hops away Not detected Not DetectedP(E|X3) 3+ Hops awayP(E|X4)

P(E|X1) 0.5% 5.0% 10.0% 1.0% 10.0% 1.0% 1.0%P(E|X2) 2.0% 0.5% 0.5% 5.0% 3.0% 15.0% 15.0%P(E|X3) 1.0%P(E|X4)

Condition State Which Applies Correct Config No Injection Code Injection Updated 1 Hop away Detected Detected

Conditional Probability 23.2%

Conditions

| 34 |

Now What?

For Me– Solidify my risk decompositions– Identify my events to analyze– Calibrate my team– Model and Simulate– Submit Blackhat ‘18 paper For You

– Go read Hubbard’s book– Go get my code: https://github.com/richmr/QuantitativeRiskSim– Think about your decompositions– Identify your events– Model and Simulate– Come watch my Blackhat ‘18 presentation

| 35 |

Summary

Quantitative risk modeling can be a reality in Cybersecurity– Use Case: Risk ranking and prioritization– Use Case: Assessing control audit results– Use Case: Mitigation comparison– Use Case: Quantifying expert knowledge on complex systems– Use Case: Test planning

Networks can improve its cybersecurity… Measurably! Python Simulation Code available at:

– https://github.com/richmr/QuantitativeRiskSim

| 36 |