measuring and maturing an appsec program · 2015-04-21 · presenter . bruce c jenkins cissp,...

Measuring and Maturing an AppSec Program

Presenter Bruce C Jenkins CISSP, CSSLP, CISM Fortify Security Lead & AppSec Program Strategist Hewlett-Packard Company Contact: bcj at hp dot com

2 (ISC)2 e-Symposium

Agenda

• Why Measure

• Preparing to Measure

• What to Measure

3 (ISC)2 e-Symposium

Why Measure

4 (ISC)2 e-Symposium

Why Measure Humans have a natural tendency to want to measure

5 (ISC)2 e-Symposium

Why Measure We have been measuring (and comparing) since ancient times

6 (ISC)2 e-Symposium

Source: wikipedia.org/wiki/Cubit

Source: www.theguardian.com

Why Measure Numerous guides, standards, and frameworks speak to measurement

7 (ISC)2 e-Symposium

2005 2006 2008* 2010 2010 2013

*OpenSAMM update scheduled for CY2015

Why Measure Bottom line: Decision Support

8 (ISC)2 e-Symposium

Bruce

Source: HP Fortify on Demand

Why Measure Views about the priority of security in custom software development

9 (ISC)2 e-Symposium

Source: Osterman Research White Paper, Jan 2015

Why Measure Views about the priority of security in custom software development

10 (ISC)2 e-Symposium

Source: Osterman Research White Paper, Jan 2015

Preparing to Measure

11 (ISC)2 e-Symposium

Preparing to Measure First some basic definitions

12 (ISC)2 e-Symposium

Preparing to Measure First some basic definitions

13 (ISC)2 e-Symposium

goal long-term aims that you want to accomplish

Preparing to Measure Goals often are broad or lofty and long term; Example Personal Goal: Be taller

14 (ISC)2 e-Symposium

Preparing to Measure First some basic definitions

15 (ISC)2 e-Symposium

objective concrete attainments achieved by following a certain number of steps; time-constrained, measurable goals

goal long-term aims that you want to accomplish

Preparing to Measure Objectives are concrete, measurable and time-constrained achievements on the path to reaching a particular goal

16 (ISC)2 e-Symposium

Obtain medieval-certified rack by 2015-05-31

Complete medieval rack Train-the-Trainer program by 2015-09-18

Train and certify four rack operators by 2015-12-31

Complete Phase I Stretching Program by 2016-04-01

Preparing to Measure First some basic definitions

17 (ISC)2 e-Symposium

objective

metric

concrete attainments achieved by following a certain number of steps; time-constrained, measurable goals

a quantitative measure

goal long-term aims that you want to accomplish

Preparing to Measure First some basic definitions

18 (ISC)2 e-Symposium

objective

metric

concrete attainments achieved by following a certain number of steps; time-constrained, measurable goals

a quantitative measure

KPI Key Performance Indicator (KPI) is used to evaluate the success of an organization or of a particular activity

goal long-term aims that you want to accomplish

Preparing to Measure First some basic definitions

19 (ISC)2 e-Symposium

objective

metric

concrete attainments achieved by following a certain number of steps; time-constrained, measurable goals

a quantitative measure

KPI Key Performance Indicator (KPI) is used to evaluate the success of an organization or of a particular activity

goal long-term aims that you want to accomplish

Preparing to Measure Sidebar: Top challenges in achieving software security goals*

20 (ISC)2 e-Symposium

Source: Gatepoint Research Pulse Report, Oct 2014 n = 300 executives

*Read as: software security assurance (SSA) program goals

Preparing to Measure Sidebar: Top challenges in achieving software security goals*

21 (ISC)2 e-Symposium

Source: Gatepoint Research Pulse Report, Oct 2014 n = 300 executives

*Read as: software security assurance (SSA) program goals

Preparing to Measure

“It is necessary that people work together in unison toward common objectives and avoid working at cross purposes at all levels if the ultimate in efficiency and achievement is to be obtained.”

22 (ISC)2 e-Symposium

Dave Packard Co-founder, Hewlett-Packard

Preparing to Measure Sidebar: Sound software security assurance (SSA) programs are based on business needs

23 (ISC)2 e-Symposium

Mission

Goals

Objectives

Strategy

m m m KPI

Policy

Standards

Training

Preparing to Measure Use security goals to establish SSA program direction, achieve stakeholder unity

24 (ISC)2 e-Symposium

Mission

Goals

Objectives

Strategy

m m m KPI

Policy

Standards

Training

• Establish security-related goals that are directly tied to the firm’s mission

Preparing to Measure Use security goals to establish SSA program direction, achieve stakeholder unity Example: Hewlett-Packard Co.

25 (ISC)2 e-Symposium

Profit

Customer Loyalty

Growth

Market Leadership

Commitment to Employees

Leadership Capability

Global Citizenship

Hewlett-Packard

See HP’s Corporate Objectives at http://www8.hp.com/us/en/hp-information/about-hp/corporate-objectives.html

Preparing to Measure Use security goals to establish SSA program direction, achieve stakeholder unity Example: Hewlett-Packard Co.

26 (ISC)2 e-Symposium

Profit

Customer Loyalty

Growth

Market Leadership

Commitment to Employees

Leadership Capability

Global Citizenship

Hewlett-Packard

...

Goal 1

... ...

Goal n

HP Software ...

Goal 1 ...

Goal n

Fortify

Goal 1

... ...

...

Ent. Security

...

Goal n

…

Security Goal 1

…

…

…

Security Group

Security Goal n

See HP’s Corporate Objectives at http://www8.hp.com/us/en/hp-information/about-hp/corporate-objectives.html

Preparing to Measure Use security goals to establish SSA program direction, achieve stakeholder unity Example: Hewlett-Packard Co.

27 (ISC)2 e-Symposium

Profit

Customer Loyalty

Growth

Market Leadership

Commitment to Employees

Leadership Capability

Global Citizenship

Hewlett-Packard

...

Goal 1

... ...

Goal n

HP Software

Goal 1

... ...

...

Ent. Security ...

Goal 1 ...

Goal n

Fortify

…

Security Goal 1

…

…

…

Security Group

...

Goal n Security Goal n

See HP’s Corporate Objectives at http://www8.hp.com/us/en/hp-information/about-hp/corporate-objectives.html

Preparing to Measure Use security goals to establish SSA program direction, achieve stakeholder unity Example: Hewlett-Packard Co.

28 (ISC)2 e-Symposium

Profit

Customer Loyalty

Growth

Market Leadership

Commitment to Employees

Leadership Capability

Global Citizenship

Hewlett-Packard

...

Goal 1

... ...

Goal n

HP Software

Goal 1

... ...

...

Ent. Security ...

Goal 1 ...

Goal n

Fortify

…

Security Goal 1

…

…

…

Security Group

...

Goal n Security Goal n

See HP’s Corporate Objectives at http://www8.hp.com/us/en/hp-information/about-hp/corporate-objectives.html

Preparing to Measure Use security goals to establish SSA program direction, achieve stakeholder unity Example: Hewlett-Packard Co.

29 (ISC)2 e-Symposium

Profit

Customer Loyalty

Growth

Market Leadership

Commitment to Employees

Leadership Capability

Global Citizenship

Hewlett-Packard

...

Goal 1

... ...

Goal n

HP Software

Goal 1

... ...

...

Ent. Security ...

Goal 1 ...

Goal n

Fortify

…

Security Goal 1

…

…

…

Security Group

...

Goal n Security Goal n

See HP’s Corporate Objectives at http://www8.hp.com/us/en/hp-information/about-hp/corporate-objectives.html

Preparing to Measure Example goal for anchoring security program (real-world Financial)

30 (ISC)2 e-Symposium

Corp Mission Statement

Goal 1

Goal 2

Goal 3

Protect our customers’ data

Goal n

Preparing to Measure Example goal for anchoring security program (real-world Financial)

31 (ISC)2 e-Symposium

Corp Mission Statement

Goal 1

Goal 2

Goal 3

Protect our customers’ data

Goal n

Preparing to Measure Example goal for anchoring security program (real-world Financial)

32 (ISC)2 e-Symposium

Corp Mission Statement

Goal 1

Goal 2

Goal 3

Protect our customers’ data

Goal n

Corp Security Group

Security Goal 1

Security Goal 2

Security Goal 3

Security Goal n

Preparing to Measure Example goal for anchoring security program (real-world Financial)

33 (ISC)2 e-Symposium

Corp Mission Statement

Goal 1

Goal 2

Goal 3

Protect our customers’ data

Goal n

Corp Security Group

Security Goal 1

Security Goal 2

Security Goal 3

Security Goal n

Proactively identify security risk in

Business Critical applications

Preparing to Measure Use security goals to establish SSA program direction, achieve stakeholder unity

34 (ISC)2 e-Symposium

Mission

Goals

Objectives

Strategy

m m m KPI

Policy

Standards

Training

• Establish security-related goals that are directly tied to the firm’s mission

Preparing to Measure Use security goals to establish SSA program direction, achieve stakeholder unity

35 (ISC)2 e-Symposium

Mission

Goals

Objectives

Strategy

m m m KPI

Policy

Standards

Training

• Establish security-related goals that are directly tied to the firm’s mission

• Develop a security strategy that is designed to support achievement of the security goal(s)

Preparing to Measure Use security goals to establish SSA program direction, achieve stakeholder unity

36 (ISC)2 e-Symposium

Mission

Goals

Objectives

Strategy

m m m KPI

Policy

Standards

Training

• Establish security-related goals that are directly tied to the firm’s mission

• Develop a security strategy that is designed to support achievement of the security goal(s)

• Based upon business priorities and portfolio risk*, design time-constrained, measurable objectives

*portfolio is known, classified and risk-ranked

Preparing to Measure Use security goals to establish SSA program direction, achieve stakeholder unity

37 (ISC)2 e-Symposium

Mission

Goals

Objectives

Strategy

m m m KPI

Policy

Standards

Training

• Establish security-related goals that are directly tied to the firm’s mission

• Develop a security strategy that is designed to support achievement of the security goal(s)

• Based upon business priorities and portfolio risk*, design time-constrained, measurable objectives

• Only choose metrics and construct KPI’s that show progress toward meeting the objectives; nothing else

*portfolio is known, classified and risk-ranked

What to Measure

38 (ISC)2 e-Symposium

What to Measure Revisited: Example goal for anchoring security program Focus: Security Objectives

39 (ISC)2 e-Symposium

Corp Mission Statement

Goal 1

Goal 2

Goal 3

Protect our customers’ data

Goal n

Corp Security Group

Security Goal 1

Security Goal 2

Security Goal 3

Security Goal n

Proactively identify security risk in

Business Critical applications

What to Measure Revisited: Example goal for anchoring security program Focus: Security Objectives

40 (ISC)2 e-Symposium

Corp Mission Statement

Goal 1

Goal 2

Goal 3

Protect our customers’ data

Goal n

Corp Security Group

Security Goal 1

Security Goal 2

Security Goal 3

Security Goal n

Proactively identify security risk in

Business Critical applications

What to Measure

(ISC)2 e-Symposium 41

Security Goals Security Objectives & Tasks* Metrics [measurement frequency]

SG-3 Proactively identify security risk in Business Critical applications

SO-3.1 Reveal baseline risk exposure of all Mission Critical applications no later than FY15Q2 ST-3.1.1 Conduct baseline static scans of all Mission Critical apps no later than FY15Q2 ST-3.1.2 Conduct baseline dynamic scans of all Mission Critical apps no later than FY15Q2

SM-3.1.1 Number of static scans remaining [w] SM-3.1.2 Number of dynamic scans remaining [w]

*implemented in accordance with the SSA program strategy

What to Measure

(ISC)2 e-Symposium 42

Security Goals Security Objectives & Tasks* Metrics [measurement frequency]

SG-3 Proactively identify security risk in Business Critical applications

SO-3.1 Reveal baseline risk exposure of all Business Critical applications no later than FY15Q2 ST-3.1.1 Conduct baseline static scans of all Mission Critical apps no later than FY15Q2 ST-3.1.2 Conduct baseline dynamic scans of all Mission Critical apps no later than FY15Q2

SM-3.1.1 Number of static scans remaining [w] SM-3.1.2 Number of dynamic scans remaining [w]

*implemented in accordance with the SSA program strategy

What to Measure

(ISC)2 e-Symposium 43

Security Goals Security Objectives & Tasks* Metrics [measurement frequency]

SG-3 Proactively identify security risk in Business Critical applications

SO-3.1 Reveal baseline risk exposure of all Business Critical applications no later than FY15Q2 ST-3.1.1 Conduct baseline static scans of all Mission Critical apps no later than FY15Q2 ST-3.1.2 Conduct baseline dynamic scans of all Mission Critical apps no later than FY15Q2

SM-3.1.1 Number of static scans remaining [w] SM-3.1.2 Number of dynamic scans remaining [w]

*implemented in accordance with the SSA program strategy

What to Measure

(ISC)2 e-Symposium 44

Security Goals Security Objectives & Tasks* Metrics [measurement frequency]

SG-3 Proactively identify security risk in Business Critical applications

SO-3.1 Reveal baseline risk exposure of all Business Critical applications no later than FY15Q2 ST-3.1.1 Conduct baseline static scans of all Mission Critical apps no later than FY15Q2 ST-3.1.2 Conduct baseline dynamic scans of all Mission Critical apps no later than FY15Q2

SM-3.1.1 Number of static scans remaining [w] SM-3.1.2 Number of dynamic scans remaining [w]

*implemented in accordance with the SSA program strategy

# of Scans Remaining by Week Ref. SM-3.1.1, SM-3.1.2

(ISC)2 e-Symposium 45

What to Measure

(ISC)2 e-Symposium 46

Security Goals Security Objectives & Tasks* Metrics [measurement frequency]

SG-3 Proactively identify security risk in Business Critical applications

SO-3.1 Reveal baseline risk exposure of all Business Critical applications no later than FY15Q2 SO-3.2 Identify security skills amongst application Architects, Developers and Testers no later than FY15Q2 SO-3.3 Qualify gap between application risk exposure and security skills no later than FY15Q3 SO-3.4 Implement training program to address security skills gap no later than FY15Q3 SO-3.5 Evaluate risk exposure of all Mission Critical applications on a quarterly basis

SM-3.5.1 # of Critical findings [q] SM-3.5.2 # of High findings [q]

*implemented in accordance with the SSA program strategy

What to Measure

(ISC)2 e-Symposium 47

Security Goals Security Objectives & Tasks* Metrics [measurement frequency]

SG-3 Proactively identify security risk in Business Critical applications

SO-3.1 Reveal baseline risk exposure of all Business Critical applications no later than FY15Q2 SO-3.2 Identify security skills among application Architects, Developers and Testers no later than FY15Q2 SO-3.3 Qualify gap between application risk exposure and security skills no later than FY15Q3 SO-3.4 Implement training program to address security skills gap no later than FY15Q3 SO-3.5 Evaluate risk exposure of all Mission Critical applications on a quarterly basis

SM-3.5.1 # of Critical findings [q] SM-3.5.2 # of High findings [q]

*implemented in accordance with the SSA program strategy

What to Measure

(ISC)2 e-Symposium 48

Security Goals Security Objectives & Tasks* Metrics [measurement frequency]

SG-3 Proactively identify security risk in Business Critical applications

SO-3.1 Reveal baseline risk exposure of all Business Critical applications no later than FY15Q2 SO-3.2 Identify security skills among application Architects, Developers and Testers no later than FY15Q2 SO-3.3 Qualify gap between application risk exposure and security skills no later than FY15Q3 SO-3.4 Implement training program to address security skills gap no later than FY15Q3 SO-3.5 Evaluate risk exposure of all Mission Critical applications on a quarterly basis

SM-3.5.1 # of Critical findings [q] SM-3.5.2 # of High findings [q]

*implemented in accordance with the SSA program strategy

What to Measure

(ISC)2 e-Symposium 49

Security Goals Security Objectives & Tasks* Metrics [measurement frequency]

SG-3 Proactively identify security risk in Business Critical applications

SO-3.1 Reveal baseline risk exposure of all Business Critical applications no later than FY15Q2 SO-3.2 Identify security skills among application Architects, Developers and Testers no later than FY15Q2 SO-3.3 Qualify gap between application risk exposure and security skills no later than FY15Q3 SO-3.4 Implement training program to address security skills gap no later than FY15Q4 SO-3.5 Evaluate risk exposure of all Mission Critical applications on a quarterly basis

SM-3.5.1 # of Critical findings [q] SM-3.5.2 # of High findings [q]

*implemented in accordance with the SSA program strategy

What to Measure

(ISC)2 e-Symposium 50

Security Goals Security Objectives & Tasks* Metrics [measurement frequency]

SG-3 Proactively identify security risk in Business Critical applications

SO-3.1 Reveal baseline risk exposure of all Business Critical applications no later than FY15Q2 SO-3.2 Identify security skills among application Architects, Developers and Testers no later than FY15Q2 SO-3.3 Qualify gap between application risk exposure and security skills no later than FY15Q3 SO-3.4 Implement training program to address security skills gap no later than FY15Q4 SO-3.5 Evaluate risk exposure of all Business Critical applications on a quarterly basis

SM-3.5.1 # of Critical findings [q] SM-3.5.2 # of High findings [q]

*implemented in accordance with the SSA program strategy

What to Measure

(ISC)2 e-Symposium 51

Security Goals Security Objectives & Tasks* Metrics [measurement frequency]

SG-3 Proactively identify security risk in Business Critical applications

SO-3.1 Reveal baseline risk exposure of all Business Critical applications no later than FY15Q2 SO-3.2 Identify security skills among application Architects, Developers and Testers no later than FY15Q2 SO-3.3 Qualify gap between application risk exposure and security skills no later than FY15Q3 SO-3.4 Implement training program to address security skills gap no later than FY15Q4 SO-3.5 Evaluate risk exposure of all Business Critical applications on a quarterly basis

SM-3.5.1 # of Critical findings [q] SM-3.5.2 # of High findings [q]

*implemented in accordance with the SSA program strategy

What to Measure

(ISC)2 e-Symposium 52

Security Goals Security Objectives & Tasks* Metrics [measurement frequency]

SG-3 Proactively identify security risk in Business Critical applications

SO-3.1 Reveal baseline risk exposure of all Business Critical applications no later than FY15Q2 SO-3.2 Identify security skills among application Architects, Developers and Testers no later than FY15Q2 SO-3.3 Qualify gap between application risk exposure and security skills no later than FY15Q3 SO-3.4 Implement training program to address security skills gap no later than FY15Q4 SO-3.5 Evaluate risk exposure of all Business Critical applications on a quarterly basis

SM-3.5.1 # of Critical findings [q] SM-3.5.2 # of High findings [q]

*implemented in accordance with the SSA program strategy

What to Measure Consolidated dashboards are effective at providing the status of key metrics (Key Performance Indicators)

53 (ISC)2 e-Symposium

Bruce

Source: HP Fortify on Demand

What to Measure Use security goals to establish SSA program direction, achieve stakeholder unity

54 (ISC)2 e-Symposium

Mission

Goals

Objectives

Strategy

m m m KPI

Policy

Standards

Training

• Establish security-related goals that are directly tied to the firm’s mission

• Develop a security strategy that is designed to support achievement of the security goal(s)

• Based upon business priorities and portfolio risk*, design time-constrained, measurable objectives

• Only choose metrics and construct KPI’s that show progress toward meeting the objectives; nothing else

*portfolio is known, classified and risk-ranked

Summary • Anchor your security program to the business

55 (ISC)2 e-Symposium

Summary • Anchor your security program to the business • Develop security goals that directly support the business goals

56 (ISC)2 e-Symposium

Summary • Anchor your security program to the business • Develop security goals that directly support the business goals • Strategies are used to achieve goals; develop a strategy

57 (ISC)2 e-Symposium

Summary • Anchor your security program to the business • Develop security goals that directly support the business goals • Strategies are used to achieve goals; develop a strategy • Goals must have supporting objectives; some may be long-lived

58 (ISC)2 e-Symposium

Summary • Anchor your security program to the business • Develop security goals that directly support the business goals • Strategies are used to achieve goals; develop a strategy • Goals must have supporting objectives; some may be long-lived • Never use a metric that does not support an objective; it will be

interesting at best, but will not add value to your program

59 (ISC)2 e-Symposium

Summary • Anchor your security program to the business • Develop security goals that directly support the business goals • Strategies are used to achieve goals; develop a strategy • Goals must have supporting objectives; some may be long-lived • Never use a metric that does not support an objective; it will be

interesting at best, but will not add value to your program • Report your progress to maintain program justification & budget

60 (ISC)2 e-Symposium

Summary • Anchor your security program to the business • Develop security goals that directly support the business goals • Strategies are used to achieve goals; develop a strategy • Goals must have supporting objectives; some may be long-lived • Never use a metric that does not support an objective; it will be

interesting at best, but will not add value to your program • Report your progress to maintain program justification & budget • Adjust the strategy as business goals, threats and risks change

61 (ISC)2 e-Symposium

hp.com/go/fortifyssa

Q&A