creating a serverless ami certification pipeline
TRANSCRIPT
Creating a Serverless AMI Certification PipelineSEATTLE AWS ARCHITECTS & ENGINEERS MEETUPDAVID F. SEVERSKI - JANUARY 31, 2017
2
Background I’m in information security
Most of my time is in the risk space
Started trying to figure out this cloud thing 5 years ago
AWS Full 5 Certified
Drinker of the DevOps Kool-Aid
3
GenesisJim Fink’s November AWS Inspector Talk
4
Lambda Already in Use AMI S3 Cleaner
Controlling AWS ML Endpoints
SNS-Slack Bridge
Periodic RDS Extractions
5
Enter Step Functions and Lambda Environment VariablesAnnounced at re:invent 2016
Step Functions and Environment Variables?!
6
Real World Architecture“Sometimes, it’s messy”
Image © Valve Software
7
Materials Will Be ProvidedGitHub, Slideshare, etc.
8
Problem StatementHow can I ensure that my AMIs are built without known vulnerabilities?
No persistent hosts
Rapidly changing images (Packer built AMIs)
Building a known good image verification system without statically running hosts (Jenkins, etc.)
9
Quick Lambda RefresherFunctions deployed without servers (that you manage anyways)
Maximum runtime of five minutes
Auto-scaled by AWS
Super cheap to run
Choice of languagesUsing Python 2.6 and Boto for this example
Can use IAM roles to access other AWS services
Wide support for triggering via other AWS Services
10
Basic Concepts of Step Functions
Collection of tasks◦ Worker resources – can be on premise!◦ Simple deciders
Steps◦ Pass State◦ Task State◦ Choice State◦ Wait State◦ Succeed State◦ Fail State
Multiple independent executions can run at once
JSON-based language - https://states-language.net/spec.html
11
Services Involved (Partial)CloudTrail CloudWatch Lambda
Inspector Step Functions EC2
12
Caution!Cutting, and sometimes bleeding, edge stuff here
13
Basic Flow
Approve/Reject New Image
Test the Image
Launch Test Instance
Detect New Image
14
Detect1. Track API calls - CloudTrail setup on account
2. CloudWatch Log event filter subscription tracks any image being built◦ RegisterImage (instance images)◦ CreateImage (EBS images)◦ CopyImage (copying public instances into this account)
3. CloudWatch triggers Lambda function to start Step Function◦ Input: AMI ID
4. Check Image State
Detect Launch Test Approve
15
Launch1. Launch Instance of candidate AMI
◦ Launch Instance◦ Uses an EC2 Spot Instance
2. Install Inspector Agent◦ UserData with simple “curl | bash”
3. Check Instance State
4. Tag Instance for Scanning◦ Assigns a batch ID to identify the instance(s) to scan
Detect Launch Test Approve
16
Test1. Start Inspector Run
2. Wait 15 Minutes
3. Poll Inspector Status
4. Try and Repeat with Timeout
Detect Launch Test Approve
17
Approve/Reject1. Terminate Instance
2. Parse results◦ Arbitrary criteria, are there any CVEs rated with a CVSS of 6 or above?
◦ If so, FAIL◦ Else PASS
3. Tag AMI
Detect Launch Test Approve
18
What We’ve Built So Far
19
DeploymentBASIC STEP FUNCTION WORKS…NOW WHAT?
20
Enter Terraform
21
Terraform – AMI-Security-Validator
Creates the IAM roles for Lambda functions◦ Launch EC2 Instance◦ Check Instance Ready◦ Tag EC2 Resources◦ Start Inspector Assessment Run◦ Check Inspector Assessment Run Complete◦ Parse Inspector Assessment Run Findings◦ Terminate Instances◦ Start Step Functions
Create Lambda Functions from S3 Artifacts
CloudTrail event trigger
22
Continuous Deployment with Travis
GitHub Commit Travis CI Build and Push to S3
23
Using Terraform Set up infrastructure VPC
VPC flow logs (use the VPC module)
Subnets in all three AZs
Private subnets
Internet gateways
Security group which allows access from home IP
24
Future Directions•Present a user interface that provides only tagged AMIs for launch by users
•Service Catalog?
25
Lessons Learned
26
Gripes and Lessons Learned – Step Functions
Step Functions cannot be edited once created
Eventual consistency is a very annoying problem
“Programing” in JSON
Triggering state machines…how?
Result processing is limited
Execution ID is not available to the state machine
Clearly defining the APIs of your functions is sanity saving
statelint tool is ruby based
AWS Config not supported
27
More Gripes – Lambda and Inspector
Lambda◦ Python environment only recently received updates to work
with Step Functions – No release notes!◦ AWS Config not supported◦ Python 3.5 & Tagging
Inspector◦ API is clunky – best suited for static target list◦ ResourceGroup entity required by TargetGroup, but no ability
to describe, modify, or delete them after creation…are they gone when the TargetGroup is deleted? Don't know.
◦ Ubuntu 16.04 LTS support launched only recently (Jan 5) available
◦ Preview-agents API takes a preview-agents-arn, but that's really an assessment-target-arn
◦ Strange convergence issues between resource groups, targets, and runs
28
Would You Like Some More Gripes?
Management◦ No Terraform or CloudFormation support for
Step Functions◦ Terraform support coming soon!
◦ No CloudFormation support for AWS Inspector◦ Terraform now supports AWS Inspector as of
v0.8.5
IAM Roles◦ Pathing strategy for IAM roles is really helpful◦ Placing roles under a /service-roles path makes
filtering and sorting a dream
30
ReferencesLambda Code
https://github.com/davidski/infrastructure-lambdas
Image Scanner Step Function Definition & Terraform Setuphttps://gist.github.com/davidski/788afdb2a79d77edfb5fea65b50369e7
Terraform Support for Step Functionshttps://github.com/hashicorp/terraform/pull/11420
AWS Step Language Documentationhttps://states-language.net/spec.html
Packerhttps://packer.io
Terraformhttps://terraform.io