secure application development in the age of continuous delivery
TRANSCRIPT
Secure Application Development in the
Age of Continuous Delivery
LinuxCon/ContainerCon Europe 2016 – Berlin Germany
#whoami – Tim Mackey
Current roles: Senior Technical Evangelist; Occasional coder• Former XenServer Community Manager in Citrix Open Source
Business OfficeCool things I’ve done
• Designed laser communication systems• Early designer of retail self-checkout machines• Embedded special relativity algorithms into industrial control system
Find me• Twitter: @TimInTech ( https://twitter.com/TimInTech )• SlideShare: slideshare.net/TimMackey• LinkedIn: www.linkedin.com/in/mackeytim
Standard Development Workflow
DEVELOP BUILD PACKAGE DEPLOY PRODUCTION
BUG TRACKING
TESTAUTOMATION
Understanding the Attacker
Model
Vulnerability Management Implies Data Breach Management
89% of data breaches had a financial or espionage motive
Legal costs and forensics dominate remediation expenses
Source: Verizon 2016 Data Breach Report
Attackers Decide What’s Valuable …
But security investment is often not aligned with actual risks
Anatomy of a New Attack
Potential Attack
Iterate
Test against platforms
Document
Don’t forget PR department!
Deploy
Exploiting a Vulnerability
CLOSED SOURCE COMMERCIAL CODE• DEDICATED SECURITY RESEARCHERS• ALERTING AND NOTIFICATION INFRASTRUCTURE• REGULAR PATCH UPDATES• DEDICATED SUPPORT TEAM WITH SLA
OPEN SOURCE CODE• “COMMUNITY”-BASED CODE ANALYSIS• MONITOR NEWSFEEDS YOURSELF• NO STANDARD PATCHING MECHANISM• ULTIMATELY, YOU ARE RESPONSIBLE
Who is Responsible for Code and Security?
Knowledge is Key. Can You Keep Up?
glibc
BugReported
July 2015
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
Knowledge is Key. Can You Keep Up?
glibc
VulnIntroduce
d
May 2008
glibc
BugReported
July 2015
CVE-2015-7547
CVE Assigned
Feb 16-2016
Low Security RiskVuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
Knowledge is Key. Can You Keep Up?
glibc
VulnIntroduce
d
May 2008
CVE-2015-7547
CVE Assigned
Feb 16-2016
glibc
BugReported
July 2015
NationalVulnerabilityDatabase
VulnPublished
Feb 18-2016
Moderate Security RiskLow Security RiskVuln: CVE-2015-7547: glibc getaddrinfo stack-
based buffer overflow
Knowledge is Key. Can You Keep Up?
glibc
VulnIntroduce
d
NationalVulnerabilityDatabase
VulnPublished
YouFind It
May 2008
CVE-2015-7547
CVE Assigned
Feb 16-2016 Feb 18-2016
glibc
BugReported
July 2015
Patches Available
YouFix It
Highest Security RiskModerate Security
RiskLow Security RiskVuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
Understanding Vulnerability Impact
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 20150
500
1000
1500
2000
2500
3000
3500
Open Source Vulnerabilities Reported Per Year BDS-Enhanced Vulnerability Data nvd
Reference: Black Duck Software KnowledgeBase, NVD
Vulnerability Disclosures Trending Upward
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 20150
500
1000
1500
2000
2500
3000
3500
283 383679 531
1099
17992293
19892422
2141 2087 21512521 2663 2637
2186
0 0
00
0
0
0
0
1
4 8 11
155305 490
918
Open Source Vulnerabilities Reported Per Yearnvd BDS-Enhanced Vulnerability Data
Reference: Black Duck Software KnowledgeBase, NVD
Vulnerability Disclosures Trending Upward
Vulnerabilities Enter Product and Become Deployed
DEVELOP BUILD PACKAGE DEPLOY PRODUCTION
BUG TRACKING
TESTAUTOMATION
Workflow Integrations Matter
Understanding the
Deployment Environment
Primary goals• Protect against BIOS and firmware
attacks• Protect cryptographic host state• Ensure valid hypervisor kernel • Validate launch of critical VMs• Attest to hosts’ trust state
Implemented by• Intel Haswell and newer• Cryptographic hashes stored in TPM
Intel TXT – Trusted Execution Protection - Foundational
Intel EPT – Extended Page Tables
Page 0…
Page 13553Page
13554…
…Page 126Page 127
…Page
64589Page
64590Page
64591
Page 0…
Page 217…
Page 31289……
Page 78924…
Page 97586…
0→64589 13553→12713554→6459
1
App Memory
OS MemoryTLB CR3
Virtual Machine
126→31289127→0
64589→97586
64590→21764591→7892
4
Host Memory
EPT
Hypervisor
Hypervisor Memory Introspection – Enabled by EPT
Implementation Overview• Critical memory pages are
assigned permissions in EPT• Exception handler defined in
hypervisor• Shadow EPT defined with
elevated privsProtects Against Attack Techniques
• Rootkit injection• Buffer overflow• API hooking
VM Kernel Memory Layout…
Kernel Code (R/X)Driver Code (R/X)
…
Driver Data (R/W)Kernel Code (R/X)Kernel Data (R/W)
…
126→31289 (R/X)127→0 (R/X)
64589→97586 (R/W)64590→217 (R/X)
64591→78924 (R/W)
EPT#1
126→31289 (+W)127→0 (+W)
64589→97586 (+X)64590→217 (+W)
64591→78924 (+X)
EPT#2 (Shadow)
Exception Handler
Guest Guest Guest Guest Guest
Critical Memor
y Access
Critical Memor
y Access
Critical Memor
y Access
Critical Memor
y Access
Critical Memor
y Access
Networking StorageCompute
Simplified Hypervisor Introspection Architecture Diagram
Xen Project Hypervisor
Control Domain (dom0)
Security Appliance(domU)
Memory Introspectio
n Engine
Direct Inspect APIs
Virtual Switches as Local Edge Protection – Silent Block
Guest VM
SSL access
Attack silently blocked
Virtual Switch RulesIngress:
HTTPS publicEgress:Dynamic port to originMySQL internalPrivate CIDR internal
Port 22 access
Virtual Switches as Local Edge Protection – Traffic Monitor
Guest VM
SSL access
Attack blocked with traffic log
Virtual Switch RulesIngress:
HTTPS publicEgress:Dynamic port to originMySQL internalPrivate CIDR internal
Port 22 accessovs Controller
Log SSH Port 22 accessCreate port mirror for attacker
Traffic Monitor
Virtual Switch RulesIngress:
HTTPS publicEgress:Dynamic port to originMySQL internalPrivate CIDR internal
Mirror: Port 22 to Traffic MonitorAll attacker traffic to monitor
Guest VM
Virtual Switches as Local Edge Protection – Quarantine
Guest VM
SSL access
Attack quarantined with full log
Virtual Switch RulesIngress:
HTTPS publicEgress:Dynamic port to originMySQL internalPrivate CIDR internal
Port 22 accessovs Controller
Log SSH Port 22 accessCreate port mirror for attackerQuarantine VM for attacker useTrigger replacement VM for farm
Traffic Monitor
Virtual Switch RulesIngress:
HTTPS attackerEgress:Dynamic port to origin
Mirror: Port 22 to Traffic MonitorAll attacker traffic to monitor
Production Container
Usage
Are Containers Production Ready?
Container Use Cases
Application containers• Hold a single application• Can follow micro-services, cloud native design
pattern• Starting point for most container usage• Short lifespan, many per host
System containers• Proxy for a VM• Insulate against core operating system• Perfect for legacy apps• Long lifespan, few per host
MyS
QL
Tom
cat
ngin
x
Kernel
MySQLTomcatnginx
KernelDon’t forget about patch validation!!
Securing the Container
Contents and Environment
Baseline to Limit the Scope of Compromise
• Enable Linux Security Modules• SELinux
• --selinux-enabled on Docker engine, --security-opt=“label:profile”• AppArmor
• -- security-opt=“apparmor:profile”
• Apply Linux kernel security profiles• grsecurity, PaX and seccomp protections for ALSR and RBAC
• Adjust privileged kernel capabilities• Reduce capabilities with --cap-drop• Beware –cap-add and –privileged=false, and CAP_SYS_ADMIN
• Use a minimal Linux host OS• Red Hat Enterprise Linux Atomic Host 7, CoreOS, RancherOS, Intel Clear Linux, Alpine, etc
• Reduce impact of noisy neighbors• Use cgroups to set CPU shares and memory
Red Hat Enterprise Linux Atomic Host 7
• What is Atomic Host?• Optimized RHEL7 variation designed for use with Docker• Uses SELinux for safeguards• Provides atomic upgrade and rollback capabilities via rpm-ostree• Pre-installed with Docker and Kubernetes
• Atomic App and Atomic Nulecule• Provides a model for multi-container application definition• Supports Docker, Kubernetes, OpenShift and Mesos• OpenShift artifacts run natively or via atomic provider
• Provides security compliance scan capabilities
Container Source Trust
Red Hat Atomic Host
Atom
ic Ap
pAt
omic
App
Atom
ic Ap
p
Red Hat Registry
MyS
QL
Redi
s
Jenk
ins
Docker Hub
Dock
er C
onta
iner
Dock
er C
onta
iner
Dock
er C
onta
iner
Dock
er C
onta
iner
Dock
er C
onta
iner
Third Party and Custom Problem: Who to trust, and why?
• Trusted source?• Unexpected image
contents• Locked application layer
versions (e.g. no yum update)
• Layer dependencies (monolithic vs micro-services)
• Validated when?
OpenSCAP vs. Black Duck Hub
OpenSCAP• Profile driven compliance policy engine• Vendor vulnerability data is but one component of policy• Integrated directly with Red Hat Atomic• Usage: atomic scan --scanner openscap {container id}
Black Duck Hub integration with Red Hat Atomic• Broad vulnerability data for most open source components• Covers vulnerability, license compliance and operational risk• Integrated with Red Hat Atomic• Rich tooling integration for development teams• Installed via: atomic install blackducksoftware/atomic• Usage: atomic scan --scanner blackduck {container id}
A DONATION HAPPILY GIVEN TO THE DEMO GODS
Gating Builds for Vulnerabilities
DEVELOP BUILD PACKAGE
BUG TRACKING
VULNERABILITY ASSESMENT
Control
Domain
NetworkingCompute Storage
Hypervisor
Container VM
Minimal OS
Understanding Scope of Compromise – Protect From the Inside
Cont
aine
rCo
ntai
ner
Cont
aine
r
Container VM
Minimal OS
Cont
aine
rCo
ntai
ner
Cont
aine
r
Secu
rity
Serv
ice
Cont
aine
r
Understanding Scope of Compromise – Protect From the Inside
Container VM
Minimal OS
Cont
aine
r
Cont
aine
r
Unpa
tche
d Co
ntai
ner
Com
prom
ised
Cont
aine
r
Com
prom
ised
Cont
aine
r
Vulnerability Assessment Post Deployment
DEVELOP BUILD PACKAGE DEPLOY PRODUCTION
BUG TRACKING
TESTAUTOMATION
VULNERABILITY ASSESMENT
Risk Mitigation Shrinks Scope of Compromise
Open source license compliance• Ensure project dependencies are understood
Use of vulnerable open source components• Is component a fork or dependency?• How is component linked?
Operational risk• Can you differentiate between “stable” and “dead”?• Is there a significant change set in your future?• API versioning• Security response process for project
Vulnerability Analysis Compliments SAST/DAST
Vulnerability Analysis- Identifies vulnerable dependencies- 3000+ disclosures in 2015- Most vulnerabilities found by
researchersAll possible security
vulnerabilities
Static and Dynamic Analysis - Discover common security patterns- Challenged by nuanced bugs- Focuses on your code; not upstream
7 of the top 10 Software Companies (44 of the top 100)
6 of the top 8Mobile Handset Vendors
6 of the top 10 Investment Banks
24Countries
250+Employees
1,800Customers
Who is Black Duck Software?
27
Founded
2002
8,500WEBSITES
350BILLION LINES OF CODE
2,400LICENSE TYPES
1.5MILLION PROJECTS
76,000VULNERABILITIES
• Largest database of open source project information in the world.
• Vulnerabilities coverage extended through partnership with Risk Based Security.
• The KnowledgeBase is essential for identifying and solving open source issues.
Comprehensive KnowledgeBase
Black Duck Hub Security Architecture
Hub Scan1 File and Directory Signatures
2 Open Source Component Identified
3
Hub Web Application
Black Duck KnowledgeBase
On Premise Black Duck Data Center
We Need Your Help
Knowledge is power• Know what’s running and why• Define proactive vulnerability response process• Don’t let technology hype cycle dictate security
Invest in defense in depth models• Don’t rely on perimeter security to do heavy lifting• Do look at hypervisor & container trends in security• Make developers and ops teams part of the solution• Focus attention on vulnerability remediation
Together we can build a more secure data center
Free Tools to Help
Free Docker Container Security Scanner• https://info.blackducksoftware.com/Security-Scan.html
14 Day Free Trial to Black Duck Hub• https://info.blackducksoftware.com/Demo.html