information security effectiveness metrics: what … · “relating to measurement; involving, or...

IBM Global ServicesSecurity & Privacy Services

Information Security EffectivenessInformation Security EffectivenessMetrics: What Metrics? What Role for Metrics?Metrics: What Metrics? What Role for Metrics?

Matunda Nyanchama, PhD, CISSPNational Leader, Security & Privacy Delivery Services

IBM Global Services, CanadaE-mail: [email protected]

Website: www.ca.ibm.com

mailto:[email protected]

http://www.ca.ibm.com/

Security & PrivacyIBM Global Services

November 21, 2004November 21, 2004 22Copyright IBM Global ServicesCopyright IBM Global Services

AgendaAgenda• Background

– Some Definitions– Why Metrics?

• IS Metrics - Background– Value Information Security Metrics– Metrics Development Process– Scope of Measurement – ISO 17799

• Scoping out IS Metrics– Information Security Program – Example– Scope of Considerations for Measurement– Examples of Measures

• Metrics & Reporting– Data Sources for IS Metrics– IS Metrics Process & Reporting– Metrics – Breadth, Depth & Purpose– Incident Management Example– Sample IS Dashboard

• State of IS Metrics & Caveats & Some Suggestions• Summary



Some DefinitionsSome Definitions• Metric: “relating to measurement; involving, or proceeding by, measurement”

(Webster’s Revised Unabridged Dictionary) • “Information Security” pertains to integrity, confidentiality & availability;

auditability and accountability• Security Metric: “A measurable attribute of the result of a security engineering process

that could [be] evidence its effectiveness.” (see references) • Effectiveness: Having an intended/expected effect; operative; in effect; efficacy,

force, punch, power, strength, success, validity, vigor, weight (The American Heritage Dictionary)

• Efficiency: Production of desired effect/results with minimum waste of time, effort, or skill ; a measure of effectiveness; specifically, the useful output divided by input into a system; proficiency, capability, adeptness, adequacy, suitability (The American Heritage Dictionary)

• Benchmark: Reference, a standard by which something is measured; criterion, gauge, goal, measure, standard, touchstone, yardstick

• Return on Investment (ROI):A measure of profitability; it measures how effective a company uses its capital to generate profit; income that an investment provides in a specified time (e.g. one year)



Why Metrics?Why Metrics?Metrics are intended to:• Focus on measurable attributes … that could serve as evidence

of effectiveness/efficiency of a given program or process• Facilitate decision making: what are the shortcomings? How

closely are objectives met? Gaps/shortcomings if any? Need change of direction?

• Help improve performance and accountability: where are gaps? How can things be done better? Who is responsible?

• Can be objective or subjective, and quantitative or qualitative. • To be relevant, metrics should be SMART, i.e. Specific,

Measurable, Attainable, Repeatable and Time-independent

Remember: “If you cannot measure it, you cannot manage it.” - anonQuestion? Where are we with Information Security Metrics?



Value Information Security Metrics Value Information Security Metrics -- II• IS performance against defined IS goals, e.g.

– Efficacy of information security – Accountability to stakeholders

• Assess IS plans, programs, processes, etc. for– Efficiency – how well information security resources are utilized– Effectiveness of Information security program + existing security

controls• Identify IS risks

– What assets need protection? What is their value?– What threats and vulnerabilities exist to the assets? – What chances for exploitation exist?

• IS Risk Management– Risk assessment - extent of exposure to threats + potential business

impacts should attacks happen– Controls - What countermeasures/controls to identified risks– Controls assessment - How effective are those controls

• Assess IS posture



Value Information Security Metrics Value Information Security Metrics -- IIII• Security posture trends – is the “state of security” improving,

staying the same or getting worse?• Help identify priorities for resource deployment in based on

risk levels to assets• Facilitate corrective action where controls are weak, e.g. where

incident response times are unacceptable• Demonstrate the value of information security to executives • Benchmark against industry, where possible - how do we

compare with our peers in industry?• Can be used for compliance-related assessments – e.g. SOX for

internal controls assessment



Information Security Metrics Benefits SummaryInformation Security Metrics Benefits Summary• Productivity indicators:

– Effectiveness & efficiency of a security program– Security return on investment (ROI) (where possible to measure) – Information security program maturity

• Information Security posture:– Collected data can be used as baseline for measurements & trending– Risks are identified and a business case made to address the risks

• Help define a baseline and hence deviations:– Apply risk management methodology for deviations from baseline – Quantify risk and hence plan for better risk management strategy

Used appropriately:• Metrics can engender process improvement.• Demonstrate value of Information Security investment, e.g. ROI • Facilitate risk management• Allow benchmarking with industry peers



Metrics Development ProcessMetrics Development ProcessFollow ISO17799’s “plan-do-check-act” cycle• Plan

• Establish key objectives for the metrics required• Identify the required metrics and hence required data• Design & implement strategy for data collection & metrics generation• Establish targets/benchmarks; where possible compare with industry• Determine the process for collecting and analyzing data, and reporting • Establish metrics review program, and the refinement process/cycle

• DO• Communicate with stakeholders and ensure buy-in• Implement the metrics program – people, process and technology

• CHECK/Monitor• Continuously review metrics report against objectives and benchmarks• Monitor program performance against objectives and benchmarks• Identify gaps, if any, in the program

• ACT• Address gaps in program• Refine specific metrics, where necessary• Refine metrics program, where necessary



Scope of Measurement Scope of Measurement –– ISO 17799ISO 17799Sample MeasurementsISO Area

Sample MeasurementsGaps in policies; Potential impacts of policy gaps; # security violations per period of time.Security Policy

% staff with certification; formal roles and responsibilities; staff turnover; security spending/employee; IS spending as % IT budgetSecurity Organization

% assets in inventory; % assets with classification; % assets with valuation; % assets with protection plan

Asset Classification & Control

# security training sessions; level of security awareness; # of personnel security-related incidentsPersonnel Security

Frequency of review of physical access; # access anomalies or violationsPhysical & Environmental Security

# incidents; incident impacts; frequency of assessment; % systems with exposures; incident response metrics; how quickly threats are communicated; frequency of

awareness activities; change control issues

Communications & Operations Management

Access activation/termination turnaround; % of expired accounts; % accounts with expired pwds; % of accounts with weak passwordsAccess Control

% projects that use IS; # policy exceptions/risk acceptances; % projects that perform code reviews; freq. of VAs; % systems with vulnerabilities

Systems Development & Maintenance

% systems with BCP/DRP; frequency of BCP/DRP testing; % systems that pass BCP/DRP testing; System availability

Business Continuity Management

# & trend of exemptions; Compliance



Elements of an IS Program Elements of an IS Program –– The IS Management Life CycleThe IS Management Life Cycle

Maintain & Improve Security Management

Program

Monitor & Continuously

Review Program Performance

Establish Information Security Management

Program

Implement Information

Security Management

Program

Development, Maintenance

& Improvement of the ISM Program

ACT

CHECK

DO

PLAN

Key Security Program Elements

Strategic

Governance, Policies & Business Strategy

Strategy, Policy, Procedures, Standards, Awareness Plan

Tactical

Risk Assessment, Design Reviews, Due Care, New Technology Insertion, Risk Acceptance, Policy Exceptions

Operational

• Active Security: Intrusion Detection & Alerts, Incident Management, Vulnerability Assessments, Data Aggregation & Analysis, Trending, Root Cause Analysis; what takes place daily captures the robustness or weakness of controls, e.g. incidents, external events



Information Security Information Security -- Another ViewAnother View

TimeNow

Bus. Req . Design Development OperationsImplementationRisk/MoneyMoney

- (6-12) months

Operational -•Active Security Posture &

Analysis•Vulnerability Assessments•Intrusion Detection & Alerts•Incident Response•Anti-virus Management•Data Analysis & Trending

Reporting•Awareness

StrategicGovernance &

Policies•Business Strategy•Policies•Standards•Procedures•Guidelines•Awareness Strategy•Research

TacticalApplications & Systems

Development•Risk Assessment•Design & Code Reviews•IS Solutions•Due Care •Risk Acceptance•Policy Exceptions•Technology Insertion•Awareness

ACT

CHECK

DO

PLAN

Development, Development, Maintenance & Maintenance & ImprovementImprovement

- (1-3) years

Reference to Industry Standards: ISF, ISO17799, ITIL, COBIT



Scope of Considerations for MeasurementScope of Considerations for Measurement

Possible MeasuresOrganizational Level

Security OperationsIncidents, Vulnerability

Assessment, Patch Management, threat advisories

Applications & System DevelopmentProject Assessments

Risk Acceptances, Code Reviews

Strategy & Governance

Info Sec Program + Framework; Information Security Budget

Spending/employee% of IT budget in Info Sec;

Policy Gaps in existenceBench marking against industry;

Industry standards adoptedAwareness plan

% projects going assessment process# Outstanding policy exceptions & Risk

acceptances% projects performing code reviews

Frequency of vulnerability assessments# outstanding vulnerabilitiesRate of fixing vulnerabilities

Rate of response to incidents & $ImpactsTrend of incident response losses

# & frequency of awareness sessions



Examples of MeasuresExamples of Measures

Business StrategyPoliciesStandardsProceduresGuidelinesAwareness StrategyResearch

Vulnerability AssessmentsIntrusion Detection & AlertsIncident ResponseAnti-virus ManagementData Analysis & Trending ReportingAwareness

Risk AssessmentDesign & Code ReviewsIS SolutionsDue Care Risk AcceptancePolicy ExceptionsTechnology InsertionAwareness

Span of Measurement Across the ISLCStrategic: Governance & Policies Operations – Active SecurityApplications & Systems

Development

Security Spending/employeeStrength of the security

organizationSoundness of a security framework and security

program% of IT budget given to Info

SecBench marking against

industry; Industry standards adoptedExistence or otherwise of an

Examples of Measures Examples of Measures Examples of measures

• % projects going through security assessment process•Outstanding policy exceptions & risk acceptances•% projects performing code reviews

Frequency of vulnerability assessments; # outstanding vulnerabilities; Rate of fixing

vulnerabilities; Rate of response to incidents &

$Impacts; Trend of incident response losses; # & frequency

of awareness sessions; Existing policy gaps; IS program “fit” with other processes; Feedback

integration to security life cycle

Reference to Industry Standards ISO17799awareness plan management



Sources of Data for MetricsSources of Data for Metrics

Information SecurityVulnerability Assessments

Incident dataIntrusion detection statistics

Antivirus statisticsProject assessment reports

Policy exceptions & risk acceptancesEducation & Awareness dataRisk control self-assessmentAccess management reports

Risk Management GroupsAudit – external & internalConfiguration management

Organization UnitsLog analysis exceptions

Corporate security reportsRisk control self-assessments

IS ReportingRisk Assessment Reports

IS Posture



IS Metrics Process & ReportingIS Metrics Process & Reporting

Audience

Management

Operations Team

Planning

Divisions

Analysis outputsSecurity Posture

IS Posture Report

Benchmarks

Value @ Risk

Other

Other

Other

Process

Risk Management Methodology/

Process

Information Sources• Assessments – projects,

systems, infrastructure• Policy reviews• Vulnerability

Assessments• Intrusion detection

statistics• Incident Response Data• Anti-virus statistics• Access Management• Systems (physical &

logical) Logs• Audit reports – ext/Int.• Security Investigations• Self Assessments• Corporate Security

ReportsAdopted from Marc Stefaniu – see references



Metrics Metrics –– Breadth, Depth & MeaningBreadth, Depth & Meaning• "What you measure is what you get." R. S. Kaplan & D. P. Norton in "Putting

the Balanced Scorecard to Work,“ the results of measurement are as good as the data collected

• “Not everything that can be counted counts, and not everything that counts, can be counted.” -- Albert Einstein

• You can have too many or too few measures?• Selected measures can be too specific or too general• Usefulness of information depends of meaning derived from the metrics• Can be performed top-down or bottom up• Metrics useful at one level of in the organization may not mean much at

another level; ensure that generated reports make sense for the purpose for which they were meant

• Metrics selected should serve a purpose; this should lead to required data.• Measurements in any specific area of IS can be onerous – see example on

incident data



Example Example –– Incident Data collectedIncident Data collected• Incidents that took place within a reporting period? E.g. total number of incidents;

number of incidents of high, medium & low impact; • Percentage of total incidents with high material (high, medium) impact;• Associated business impacts (monetary and otherwise);• Losses (tangible & intangible) were incurred as a result of the incidents;• Incident losses comparison with industry for similar types of businesses and size;• Failures in security controls that led to the incidents:• Whether or not the failures have been fixed; outstanding gaps;• Improvement plans/processes are underway to prevent future recurrence of

similar incidents?• The trend to date; is the situation getting better or worse?• The incident reporting dashboard would have the following:

– Current incident posture (#incidents, monetary impacts, etc.)– Trend from last reporting period (are things getting better or worse?)– Overall trend to date– Comparison with industry benchmarks – Impact of past improvement plans– Existing gaps between desirable risk levels and current posture



Sample Incident Management DashboardSample Incident Management Dashboard

Trend to date

Existing (known) Gap Trends

Net Impact of Past Improvements

Comparison with Bench Marks

Trend from last report

$100K46212Incident Posture

Monetary Costs (S)

Low ImpactMedium Impact

High ImpactTotal #

Cumulative Incident costs to date = $500KCumulative Incident costs to date = $500K



Sample Charts Sample Charts –– Risk & Policy ExemptionsRisk & Policy Exemptions

Risk Acceptance and Policy Exemptions

0

20

4060

80

100

120

Q4 Year-2 Q1 Year - 1 Q2 Year - 1 Q3 Year-1 Q4 Year - 1



Visualizing results of VA scansVisualizing results of VA scans



Global Self AssessmentGlobal Self AssessmentGlobal Self Assessment Scorecard

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

IS P

olicy

Comp

lianc

eSe

curit

y Org

aniza

tion

Data

Clas

sifica

tion

Pers

onne

l & B

usine

ss R

elati

onsh

ips S

ecu

Phys

ical &

Env

ironm

enta

l Sec

ur

Comm

unica

tions

& O

pera

tions

Man

agem

Acce

ss C

ontro

l

Syste

ms D

evelo

pmen

t & M

ainte

na

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

Q1-2003Q2-2003Q3-2003Q4-20032003 ISF Benchmark

Legend:

0 = not implemented at all

5 = fully implemented

ISF Benchmark

• Based on ISO standard

• LOBs self-assessment reported to IS on a quarterly basis

• Gap between self reported level and ISF benchmark used to prioritize ongoing work and fed into awareness program



Sample IS DashboardSample IS Dashboard

Overall11

Compliance10

Business Continuity Management9

Systems Development and Maintenance8

Access Control 7

Communications & Operations Management6

Physical and Environmental Security5

Personnel Security4

Asset Classification and Control3

Security Organization (Roles & Responsibilities)2

Security Policy (P&Ps, Standards, Guidelines)1

Year 0Year – 1Year – 1ISO Category



The State of Security MetricsThe State of Security Metrics• There exists intense interest in IS metrics – just search

google to see # of hits• Most literature talks about how to define Info Sec

metrics i.e. qualities/properties of good metrics; few specifics are suggested

• IS metrics remain ill-defined; industry practices may in future lead to specific IS metrics

• Most suggested measurements tend to be qualitative; quantitative measures may yet emerge;

• Quality & effectiveness of IS program is dependent on individual opinion and judgment

• Debate on Return on Security Investment (ROSI) will continue for a while to come



Metrics Metrics –– Some CaveatsSome Caveats• Ultimately IS metrics are intended to improve understanding

or support decision making wrt IS posture. However, – They are often ill-defined and require context and process for their

generation;– There is a risk that IS measurement can become an end in itself, i.e. the

consumer of the metric may lost in the definition of the metric.• Context is key – ensure that metrics are used with intended

purpose.• Metrics should be performance indicators, assess the value of

IS and offer pointers to performance improvement.• The heterogeneous nature of infrastructures make

measurements difficult• Issues pertaining to IS change rapidly and hence measures

should evolve with the changes• The nature of threat can change with circumstance & time



Metrics Development Metrics Development –– Some SuggestionsSome Suggestions• Identify the key areas of risk to your business and

ensure appropriate focus• Select a few IS metrics that make sense to your

organization based on above risk assessment; think of the 80:20 rule

• Start small collecting required data & refine with time• Implement a program for continuous improvement;

seek feedback on the value of the measures selected• Focus on outcomes, i.e. what the analysis points to;

metrics should not be ends in themselves.• Keep abreast with industry practices and incorporate

best practices



ReferencesReferences• Ron Knode. Security Value Metrics – 2002. CSC Global Information Security Services.• http://www.csc.com/aboutus/lef/mds69_off/uploads/Enterprise_Info_Risk_Management.pdf• Paul W. Lowans Implementing a Network Security Metrics Program. GIAC Administrivia.

www.giac.org/practical/Paul_Lowans_GSEC.doc. • Dr. Stuart Katzke. Security Metrics. Computer System Security & Privacy Advisory Board. June 13-14, 2000.• James P. Craft. Metrics and the USAID Model Information Systems Security Program (MISSP).• Christina Kormos, Natalie Givens, Lisa A. Gallagher and Nadya Bartol. Using Security Metrics to Assess Risk Management

Capabilities.• Proceedings - Workshop on Information Security System Scoring and Ranking• Marianne Swanson, Nadya Bartol, John Sabato, Joan Hash, and Laurie Graffo Security Metrics Guide for Information Technology

Systems. NIST Special Publication 800-55. July 2003.• Shirley C. Payne. A Guide to Security Metrics, SANS Security Essentials GSEC Practical Assignment Version 1.2e. July 11, 2001• Workshop on Information-Security-System Rating and Ranking (WISSRR) - http://www.acsac.org/measurement/• Information Security Metrics. Using Foundstone’s FoundScoreTM to Assign Metrics and Measure Enterprise Risk.

www.foundstone.com. April 2003.• Proceedings. Workshop on Information Security System Scoring and Ranking Information System Security Attribute

Quantification or Ordering. May 21-23, 2001• C. Kormos, L. A. Gallagher, N. Givans & N. Bartol. Using Security Metrics to Assess Risk Management Capabilities• George Jelen. “SSE-CMM Security Metrics.” NIST & CSSPAB Workshop, Washington, D.C., 13-14 June

2000;http://csrc.nist.gov/csspab/june13-15/jelen.pdf July 2001.• Shirley Payne. A Guide to Security Metrics. Shirley C. Payne. SANS Security Essentials GSEC Practical Assignment. July 11,

2001• Eddie Schwartz, NetForensics Inc. Measuring Security. In Computerworld July 15, 2004.

http://www.computerworld.com/securitytopics/security/story/0,10801,94524,00.html• Steve Foster and Bob Pacl. Analysis of Return on Investment for Information Security. www.getronics.com• R. S. Kaplan and D. P. Norton in "Putting the Balanced Scorecard to Work“• Marc Stefaniu – Metrics & Executive Reporting. CFI-CIRT Presentation; March 2004

http://www.giac.org/practical/Paul_Lowans_GSEC.doc

http://www.giac.org/practical/Paul_Lowans_GSEC.doc

http://www.acsac.org/measurement/

http://www.foundstone.com/

http://www.computerworld.com/securitytopics/security/story/0,10801,94524,00.html

http://www.getronics.com/

information security effectiveness metrics: what … · “relating to measurement; involving, or...

Documents