smu seminar 2014_03_26 v3

Fundamentals Matter – A Brief Introduction to Risk Analysis for

Information Security Southern Methodist University, March 26, 2014

Heather Goodnight, President

Patrick Florer, CTO Risk Centric Security, Inc.

www.riskcentricsecurity.com

Authorized reseller of ModelRisk from Vose Software

Risk Centric Security, Inc. Confidential and Proprietary . Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Risk Analysis for the 21st Century®

• Introductions • What we are going to talk about

o Why Fundamentals Matter / Current State o Definitions

• Risk and the Risk Landscape • Possibility and Probability • Measurement • Variability and Uncertainty • Precision vs. Accuracy • Scales of Measurement: Qualitative vs. Quantitative • Not Enough Data • Monte Carlo Simulation • Modeling Expert Opinion and PERT distributions

Agenda

Risk Centric Security, Inc. Confidential and Proprietary . Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Heather Goodnight is an accomplished Global Sales and Business Development Consultant. Over the years, her unique, practical insight into problems of risk and opportunity have provided important guidance for organizations both large and small. She is a cofounder of Risk Centric Security and currently serves as President of the Corporation. In 2010, she was appointed to the RIM Council (Responsible Information Council) of the Ponemon Institute. In addition to her role at Risk Centric Security, she serves as Business Development Manager at Triumfant, Inc., a vendor of advanced anti-malware products. Patrick Florer has worked in information technology for almost 35 years. For 17 years, he worked a parallel track in medical outcomes research, analysis, and the creation of evidence-based guidelines for medical treatment. His roles have included IT operations, programming, and systems analysis. From 1986 until now, he has worked as an independent consultant, helping customers with strategic development, analytics, risk analysis, and decision analysis. He is a cofounder of Risk Centric Security and currently serves as Chief Technology Officer. He is a member of the Ponemon Institute RIM council. In 2012, he was appointed Distinguished Fellow of the Ponemon Institute.

Introductions

Risk Centric Security, Inc. Confidential and Proprietary . Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

The Current State of Confusion … .

Risk Centric Security, Inc. Confidential and Proprietary . Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

ROI IRR EPS

EMV EBITDA

≠

Often leads to this …

Risk Centric Security, Inc. Confidential and Proprietary . Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

What is Risk?

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

What Risk Isn’t!

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Vulnerability Threat

Risk = Frequency x Impact

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Frequency

Impact

Risk

Risk and Opportunity

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Possibility and Probability: Possibility

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Possibility and Probability: Probability

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

What is a Measurement?

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Properties of Measurement

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Validity Reproducibility

Detail

Sources of Error in Measurement?

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Random Error

Errors from Bias

Variability and Uncertainty

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Variability

Uncertainty

Precision and Accuracy

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Scales of Measurement

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Qualitative Quantitative

Qualitative Scales

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Nominal/Categorical

Interval Ordinal

HIGH - Red MEDIUM - Orange LOW - Green

First, Second, Third … On a scale of …

Quantitative/Ratio Scales

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

1, 2, 3, 4, 5, 6, … n

Problems with Qualitative Scales

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

My Scale High Medium Low Red Orange Green

Your Scale High Medium Low Red Orange Yellow Green

(RED – GREEN + MEDIUM) / Somewhat Likely = ???

Mismatched Scales

Meaningless Calculations

Assessor Disagreements

Problems with Qualitative Scales

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Boundary Problems

$2.5M Loss Exposure = Moderate = Yellow $2.5M Loss Exposure = Moderate = Yellow $2.5M Loss Exposure = Moderate = Yellow $7.5M Aggregate Loss Exposure = not so Moderate !

Issues with Loss of Information

Quantitative Scales

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

2 + 2 - 1 = 3 360 * 10 = 3,600 Sqrt(25) = 5 f(x) = y etc.

Qualitative Methods - Problems Difficulty with arithmetic and statistical operations

From ISO 17999

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Qualitative Methods - Problems

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Qualitative Methods - Problems

On a scale of 1 to 5, where 1 = least and 5 = most,

please rate … Likert scale (From Wikipedia, the free encyclopedia)

When responding to a Likert questionnaire item, respondents specify their level of agreement or disagreement … In so doing, Likert scaling assumes that distances on each item are equal …

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Data

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Good Data Bad Data

Big Data

Little Data

How much data is enough data?

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

How do I get to the mall?

How do we build this?

vs.

Data from Calibrated Estimates

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

More often than you might think, the data we have to work with comes from Subject Matter Experts (SME’s). How can we improve the accuracy of these SME’s – to a 90% confidence level? With calibration. Example: How much does an iPhone 5s weigh?

Monte Carlo Simulation

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

The average = $12,500

$2,500 $12,500 $32,000

The range is:

The distributions are:

Monte Carlo Simulation

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

The Beta Pert Calculator

Minimum: What is the least or lowest (best or worst) numerical estimate that you believe to be reasonable? This will be the smallest number you come up with.

Most Likely:

What is the most likely or most probable numerical estimate in your opinion? This number must fall between the minimum and maximum. It may equal either the minimum or the maximum, but should not equal both

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

The Beta Pert Calculator

Maximum:

What is the greatest or highest (best or worst) numerical estimate that you believe to be reasonable? Note that “best” or “worst” case estimates could be either minimum or maximum values, depending upon the scenario. In a risk / loss exposure scenario, lower is better, so the minimum represents the lowest loss, or best outcome. The maximum represents the highest loss, or worst outcome. In a sales or opportunity scenario, it’s the reverse: lower is not better, so the minimum represents the worst case. Higher is better, so the maximum represents the best case.

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

The Beta Pert Calculator

Confidence: On a scale that includes “Very Low”, “Low”, “Average”, “High”, and “Very High”, how confident are you in the accuracy of your estimates? This parameter controls the sampling around the most likely value, and thereby also controls the height of the histogram or slope of the cumulative plot.

For most analyses, using “Average” for the confidence parameter works well. In this instance, “Average” really means having no strong feeling about the matter – being evenly divided between under-confidence and over-confidence.

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

The Beta Pert Calculator Percentile Tables

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

The Beta Pert Calculator Percentile Tables

1% of values are <= 10,044 and 99% are > 10,044 10% of values are <= 11,120 and 90% are > 11,120 20% of values are <= 11,658 and 80% are > 11,658 50% of values are <= 13,025 and 50% are > 13,025

The 50th percentile has another name - it’s called the Median.

The Median is the mid-point in a list of values - half of the values in the list are less and half are greater than the Median.

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

The Beta Pert Calculator Histogram

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

The Beta Pert Calculator Cumulative Plot

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Thank you !

Heather Goodnight President and Cofounder

Patrick Florer CTO and Co-founder

Risk Centric Security, Inc patrick@riskcentricsecurity.com

214.828.1172

Authorized reseller of ModelRisk from Vose Software

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Risk Analysis for the 21st Century ®

”We don’t have enough data!” - Sources

Open Security Foundation: datalossdb and osvdb http://www.opensecurityfoundation.org/

Office of Inadequate Security: http://www.databreaches.net/

Identity Theft Resource Center: http://www.idtheftcenter.org/

ISACA: www.isaca.org

ISSA: www.issa.org

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

”We don’t have enough data!” - Sources

Mitre Corporation: www.mitre.org

OWASP: http://owasp.com/index.php/Main_Page

Privacy Rights Clearing House: http://www.privacyrights.org/ SANS: www.sans.org The Ponemon Institute: www.ponemon.org

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

”We don’t have enough data!” - Sources Conference procedings: Black Hat, RSA, Source Conferences, BSides

Internet tools:

Search engines: Google, Bing, Yahoo, Ask.com

Trend Analyzers:

Google trends: http://www.google.com/trends

Twitter Trends: www.trendistic.com

Amazon: http://www.metricjunkie.com/

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

”We don’t have enough data!” - Sources

Securitymetrics.org – mailing list

Society of Information Risk Analysts (SIRA)

Books: How to Measure Anything – Hubbard The Failure of Risk Management – Hubbard Risk Analysis: A Quantitative Guide – Vose Clinical Epidemiology and Biostatistics – Kramer Data-Driven Security: Analysis, Visualization and Dashboards – Jacobs and Rudis

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.