appsec survey 2.0 fine-tuning an appsec training program based on data
DESCRIPTION
Measuring the effectiveness of any security activity is widely discussed – security leaders debate the topic with a religious fervor rivaling that of any other hot button issue. Virtually every organization has some sort of application security training effort, but data on training effectiveness remains scarce. Last year our research team delivered the first-ever survey that captured developer awareness of secure coding concepts and the impact of formal application security training on a developer’s ability to write secure code. We learned that most software developer were aware of certain application security concepts, yet when asked how to write more secure code, they faired poorly. This year’s 600-developer survey provides more quantitative data on what software developers understand about application security, both concepts and practices. It dives most deeply into awareness of defensive coding practices, which most developers largely did not grasp in the 2013 survey. It also is separates respondents by roles, so we can better understand how architects, developers, and QA staff grasp key application security concepts and put them to work. It better captures how software developers learn in general, so one can tailor any security training effort to how software developers, in practice, actually learn. This information will provide data to application security managers responsible for corporate security training that should allow them them to make more fact-based decisions about security training.TRANSCRIPT
![Page 1: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/1.jpg)
AppSec USA 2014 Denver, Colorado
AppSec Survey 2.0: Fine-Tuning an AppSec Training Program Based on
Data
John B. Dickson, CISSP @johnbdickson
September 18, 2014
![Page 2: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/2.jpg)
John B. Dickson, CISSP
• Application Security Enthusiast • Ex-AF Guy & ISSA Distinguished Fellow
• Serial Entrepreneur & MBA Type
• Dad
Introduction
![Page 3: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/3.jpg)
When Not Thinking about AppSec…
I am Snake Hunting on a Ranch in South Texas
![Page 4: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/4.jpg)
Snake Hunting Essentials
Cooler Hat Cool Hat
Snake Guards Common Gardening Tools Machete
Guy who has a machete and who is actually good at “catching” snakes
OWASP AppSec 2011 t-‐shirt
© Copyright 2014 Denim Group - All Rights Reserved
![Page 5: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/5.jpg)
• Background • Premise • AppSec Study 1.0 Results – What We Learned • Approach and Survey ParKcipants • Key Results • What We Can Put To Work • Conclusions and QuesKons & Answers
Overview
![Page 6: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/6.jpg)
• Things we Knew Last Year
• Key Findings of Last Year’s Study
• AddiKonal Stuff We Learned Along the Way
• Development training is hard
• Results are rarely measured for ROI
• Training is typically part of any AppSec program
AppSec Study 1.0 Results
![Page 7: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/7.jpg)
• Things we Knew Last Year
• Key Findings of Last Year’s Study
• AddiKonal Stuff We Learned Long the Way
• 25% retenKon aXer training
• QA did worse than architects and soXware developers
• Respondents answered basic awareness quesKons but not coding pracKces
AppSec Study 1.0 Results
![Page 8: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/8.jpg)
• Things we Knew Last Year
• Key Findings of Last Year’s Study
• AddiConal Stuff We Learned Long the Way
• SoXware developers learn differently than companies teach
• IncenKves ma[er • Surveys are hard!
AppSec Study 1.0 Results
![Page 9: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/9.jpg)
Overview of 2014 “2.0” Study
• 600 respondents • Represents mulKple industries • Asked the same applicaKon security quesKons as
2013 survey • Expanded to include training method quesKons • No “before” and “aXer” analysis • No classroom training opportuniKes • Used more social media • Data collecKon ongoing
![Page 10: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/10.jpg)
Approach and Survey Participants
Sample QuesCons
QuesKons that tested basic knowledge of applicaKon security:
• ApplicaKon security is best defined as… • Threat Modeling is… • Input ValidaKon is…
![Page 11: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/11.jpg)
Approach and Survey Participants
Sample QuesCons
QuesKons that tested understanding of defensive coding:
• Marking a cookie as “secure” will… • Which of the following will help protect against XSS…
• Which of the following is NOT an example of good session policy…
![Page 12: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/12.jpg)
Delivery Means • Direct Delivery of Customized Links via E-‐mail
• Survey Monkey paid • Social Media
– Facebook – Linkedin
Targets • SoXware Developers • Architects • Quality Assurance
Approach and Survey Participants
![Page 13: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/13.jpg)
Demographic Questions Asked
• What is your primary job funcKon?
• What is your company's size?
• How many years of soXware development experience do you have?
• How much previous applicaKon security training have you received?
![Page 14: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/14.jpg)
2014 Study Demographics
Less than a Year 18%
1-‐2 Years 9%
2-‐4 Years 10%
4-‐7 Years 13%
7-‐12 Years 16%
More than 12 Years 34%
How many years of soMware development experience do you
have?
![Page 15: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/15.jpg)
2014 Study Demographics
Other 35%
SoXware Developer
53%
Quality Assurance
6%
Architect 6%
What is your primary job funcCon?
![Page 16: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/16.jpg)
2014 Study Demographics
8% 8%
29%
8% 10%
37%
What is your company size?
1-‐24 Employees
25-‐99 Employees
100-‐499 Employees
500-‐2499 Employees
2500-‐9999 Employees
10,000 or more Employees
![Page 17: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/17.jpg)
2014 Study Demographics
None 31%
Less than a Day 19%
At least 1 day, but less than 2 days
17%
At least 2 days, but less than 3 days
8%
More than 3 days 25%
How much previous applicaCon security training experience have
you received?
![Page 18: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/18.jpg)
Key Survey Results
• Data shows soXware developers posiKvely answer quesKons about applicaKon security 56% of the Kme
• 2013 Denim Group study results: 58% • 2014 Aspect Study: 60%
![Page 19: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/19.jpg)
Change Implementation
Yes 33%
No 25%
I don't know 42%
Did your organizaCon implement any SDLC or process improvement steps to formalize concepts learned in training?
![Page 20: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/20.jpg)
Types of Training Received
0 50 100 150 200 250
Instructor-‐Led PresentaKons
e-‐Learning, CBT
Social Media
Social Learning Plaqorms
Developer E-‐mail Lists or RSS feeds
Crowdsourcing Sites
Websites
Webinars or Videos
1-‐on-‐1 Coaching
Wri[en Materials
Other
Types of Training Received
![Page 21: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/21.jpg)
E-Learning & Instructor-Led Training
0 100 200 300
Instructor-‐Led PresentaKons
e-‐Learning, CBT
Social Media
Social Learning Plaqorms
Developer E-‐mail Lists or RSS feeds
Crowdsourcing Sites
Websites
Webinars or Videos
1-‐on-‐1 Coaching
Wri[en Materials
Other
Types of Training Received
E-‐Learning & Instructor-‐led Training are SKll the Primary ApplicaKon Security Training Approach
![Page 22: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/22.jpg)
Perceived Effectiveness of Training
0 50 100 150 200 250 300 350 400 450 500
Instructor-‐Led PresentaKons
e-‐Learning, CBT
Social Media
Social Learning Plaqorms
Developer E-‐mail Lists or RSS feeds
Crowdsourcing Sites
Websites
Webinars or Videos
1-‐on-‐1 Coaching
Wri[en Materials
1: Not EffecKve
2: Somewhat EffecKve
3: Very EffecKve
![Page 23: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/23.jpg)
Question Types
41%
59%
0% 10% 20% 30% 40% 50% 60% 70%
PrescripKve QuesKons
Awareness QuesKons
% of QuesKons Answered Correctly
Respondents Fared Far Worse on QuesKons Involving Secure Coding PracKces versus ApplicaKon Security Awareness QuesKons
![Page 24: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/24.jpg)
Pass Rate by Job Function
Average Pass Rate
0%
5%
10%
15%
20%
25%
Other SoXware Developer Quality Assurance Architect
70% or more quesKons answered correctly
Quality Assurance respondents Fared 50% worse than soXware developers and architects
![Page 25: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/25.jpg)
Pass Rate by Previous Training
Average Pass Rate
0%
5%
10%
15%
20%
25%
30%
Less than a Day or None At least 1 day, but less than 3 days More than 3 days
70% or more correct
The Pass Rate More Than Doubled for Respondents Who Had More Than Three Days ApplicaKon Security Training
![Page 26: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/26.jpg)
Pass Rate by Job Function: Security
Average Pass Rate
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Security-‐Related Everyone Else
70% or more quesKons answered correctly
Respondents that worked for security organizaKons or vendors DID fare well compared to other respondents
![Page 27: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/27.jpg)
What we Can Put to Work
• Refresher training is criCcal • Even with 3+ days of appsec training, most
respondents did not have a “passing” grade of 70%
• Like any other training topic, leX unreinforced, what learned will be forgo[en over Kme • ParKcularly given the lack of SDLC changes
• Likely an area for addiKonal study for 2015 appsec training study
![Page 28: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/28.jpg)
What we Can Put to Work
• Training without SDLC changes likely will produce the same results • 33% of the respondents said their organizaKon
implemented some security SDLC improvements • 67% either answered “no” or “don’t know” • OrganizaKons cannot rely exclusively on
developers retenKon and iniKaKve to produce long-‐term decline in applicaKon vulnerabiliKes
![Page 29: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/29.jpg)
What we Can Put to Work
• Augment QA with Focused AppSec Training • QA has consistently responded poorly relaKve to
developers and architects • Many organizaKon put their most junior
developers in QA to start • QA is where appsec “lives” in many
organizaKons • OrganizaKons might considering “doubling
down” on appsec training for QA staff to compensate for this fact
![Page 30: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/30.jpg)
What we Can Put to Work
• IncenCves Ma`er When Working with Developers
• We used incenKves throughout the study to collect responses -‐ #Success!
• SoXware developers have infinite reasons to ignore engagement by the AppSec team
• Rewards help nudge soXware developers
![Page 31: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/31.jpg)
What we Can Put to Work
• Training programs must be tailored to be effecCve
• Formal programs like classroom training and e-‐Learning are sKll the bread and bu[er of appsec training programs
• ConsumpKon rates of e-‐Learning sKll abysmal without incenKves or internal markeKng
• Add newer ways of learning to reinforce certain key points and to serve AppSec corner cases
• Leverage current events to reinforce other key points
![Page 32: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/32.jpg)
Conclusions
• Data shows soXware developers posiKvely answer quesKons about applicaKon security 56% of the Kme
• Data-‐driven applicaKon security programs will likely be more successful and chart improvement
• SophisKcated security managers use incenKves and tailor programs to improve appsec IQ
![Page 33: AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data](https://reader033.vdocuments.mx/reader033/viewer/2022042623/54708993b4af9fc30a8b47a8/html5/thumbnails/33.jpg)
White Paper? MenCon it on Twi`er
John B. Dickson, CISSP @johnbdickson #appsecstudy
Questions and Answers