testing on aws - aws il meetup

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Boaz Ziniman, Technical Evangelist, AWS

@ziniman

Testing on AWSIsrael AWS User Group – January 2018


Make your pipeline safer

1. Automating tests to catch regressions early

2. Identifying production issues quickly

3. Deploying changes safely

4. Automatically deciding when to release changes


Techniques

1. Automated testing

2. Continuous production testing

3. Manage deployment health

4. Segment production

5. Halt promotions


AWS Tools used for Deployment and Testing

MonitoringAmazon CloudWatch

Software DevelopmentAmazon SNS

AWS Lambda

Build & TestAWS CodeBuild

DeploymentAWS CodeDeploy

AWS CodePipeline


MyAppCodeCommit

Source

BuildCodeBuild

Build

DeployToIntegCodeDeploy

Integration

IntegTestEnd2EndTester

DeployToProdCodeDeploy

Production

Source

Build

Deploy to Integration Stack

Integration Tests

Deploy to Production

Model the release process in CodePipeline

Pipeline RunAction

Stage

Pipeline

Source change • starts a run and• creates an artifact to be used by

other actions.

Change 1


Release and deploy process: Starting point

MyAppCodeCommit

Source

BuildCodeBuild

Build

DeployToIntegCodeDeploy

Integration


DeployToProdCodeDeploy

Production

CodeDeploy


1. Automated testing2. Continuous production testing3. Manage deployment health 4. Segment production5. Halt promotions

Techniques


Be aware when a code change fails tests

Problem: A code change can fail automated tests without developer knowledge.

Consequence:Your pipeline becomes block and changes stop flowing to production.


Integrate testing into your pipeline

1. Run unit tests during build process

2. Deploy to integration environment

3. Run integration and UI tests

4. Notify developer chat room if the build or tests fail

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Configure CodeBuild


1. Trigger pipeline on source change2. Build and unit tests3. Deploy to integration environment4. Execute UI tests5. Execute integration tests

Tests

Step 1: Build and unit testsSource

MyAppSourceCodeCommit

Build

BuildAndUnitTestsCodeBuild

IntegrationDeployCodeDeploy

TestOnChromeCodeBuild




Configure CloudWatch Event notifications


Step 2: Notify on failed build and test

Lambda FunctionNotifySlackOnPipelineActionFailure()

Change 1

CloudWatchEvents

(Failed Action)

Tests

Source


Build

BuildAndUnitTestsCodeBuild

IntegrationDeployCodeDeploy




Change 2


Techniques





5. Halt promotions


Use synthetic traffic to simulate real users

• Test all business critical functionality (UI and APIs)

• Tests must run quickly

• Measure client latencies

• Check for reachability

• Alerts and Notifications


Techniques





5. Halt promotions


Add safety to rolling deployments

1. Validate each host’s health

2. Ensure a minimum percentage of the fleet is healthy

3. Rollback if the deployment failed


Techniques





5. Halt promotions


Lower deployment risk by segmenting

1. Break production into multiple segments• Single Host• AZ• Regions

2. Deploy to a segment

3. Test a segment after a deployment

4. Repeat 2 & 3 until done


Techniques





5. Halt promotions


Don’t compound problems during an outage

Pipeline Problem: The pipeline is unaware of the health of the infrastructure it is deploying to

Consequence: Production changes, usually deployments, can make it difficult for an operator to resolve a production event


Source


Build

MyAppBuildCodeBuild

DeployToProd

MyAppCodeDeploy

Change 1Change 2

Automatically stop deploying to production during an event

CloudWatch SyntheticTraffic

deployschecks

CloudWatchEvents (1m)

triggers

emitsdisables

disableTransition() CW alarm

EC2 instance

SNS


Code is available online

• github.com/awslabs/aws-codepipeline-synthetic-tests

• github.com/awslabs/aws-codepipeline-block-production


Related re:Invent Sessions

• DEV322-R - Continuous Integration Best Practices for

Software Development Teams

• DEV343 - How to Reduce Deployment Failures through

Code Reviews and Continuous Integration

• DEV325-R - Application Deployment Techniques for

Amazon EC2 Workloads with AWS CodeDeploy


Thanks!Boaz Ziniman, Technical Evangelist, AWS

@ziniman

testing on aws - aws il meetup

Technology