using devops to improve software quality in the cloud

T21

Test Automation

5/8/2014 3:00:00 PM

Using DevOps to Improve

Software Quality in the Cloud

Presented by:

Glenn Buckholz

Coveros

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Glenn Buckholz

Coveros

Currently, Glenn Buckholz leads CI and deployment automations efforts at Coveros. Using his fifteen years of industry experience, Glenn brings success to his customers. Beginning his career as a consultant implementing automated test frameworks, he introduced the concept of change management to many, many projects. Glenn then decided to become a part of honest society and settled down at the Public Company Accounting Oversight Board as their full time enterprise change manager. After several years, he ventured back into the real world at Coveros, specializing in implementing agile practices, implementing CI, and engineering configuration management instead of simply documenting it.

4/26/2014

1

© Copyright 2014 Coveros, Inc. All rights reserved.

Using DevOps to Improve Software

Quality in the Cloud

STAR EAST 2014 – 08 May 2014

Glenn Buckholz

[email protected]


Agenda

�What did I do?

�What is it good for?

�Demo

�Architecture

�Workflow

�Orchestration

�Cost vs Velocity

�Data Analysis

�What is Coveros doing with it for customers?

4/26/2014

2


What did I do?

Combined several technologies that Coveros is

familiar with plus Puppet to demonstrate a full

implementation of CI (Continuous Integration)

This could be titled “Everything I wanted to do on

Forge.mil but am not allowed.”

Specifically, I cobbled together:

Jenkins

Puppet

Puppet master

EC2

Linux

CURL

ANT


What is it good for?

�Fail Faster

Used with permission from the people at Extra Credits.

http://www.youtube.com/extracredits

4/26/2014

3


Demo


Major Points

How is this DevOps?

Releases are engineered, the systems engineer

is simply pushing the button.

If the list of manual installation instructions is

more than one paragraph there is more

engineering work to do. Code is the

documentation.

Operations people now govern the access,

developers govern the how.

New testing concerns.

Deployment code.

OS templates.

Developers now have the job of convincing

operations to run their deployment code.

4/26/2014

4


Architecture


Workflow

Developer commits code.

Jenkins detects code change.

Proper RPMs are built and distributed to

repositories.

Jenkins contacts EC2 and spins up blank

instances.

Puppet is delivered to the blank instances.

Jenkins works with the Puppet Master for

orchestration.

Jenkins confirms that the application is

running.

Jenkins fires off automated tests.

4/26/2014

5


Code Commit – Different

ApproachesSomething has to trigger the testing. A code

commit is typically a significant enough

change to require some measure of retesting.

Unit test are the minimum criteria to the

publishing a change.

Even if the code is bad the system must

still run.

By convention, code commits should be

complete and significant.

Each commit now carries a cost.

This should be enforced by training.

For larger projects with large volumes of


RPMs A Packaging

To move quickly code must be delivered in a

consistent format.

Copying files is too unpredictable and error-

prone.

Permissions

Location

Clean up of old file

Dependency checking

Any of the following are good choices

RPM – Centos, RHEL

Emerge/Portage - Gentoo

Deb – Debian, Ubuntu

4/26/2014

6


Cloud Provider

The main issue is elasticity.

Number of servers will vary.

Tests will be running in parallel.

Need a way to catalog all the machines

Assign metadata to VMs

Ability to group VMs and test results

by deployment

Automatically mapping code commits

to cloud deployments is necessary for

proper bookkeeping.


OS templates

Each OS must originate with a standard template.

All change must be managed by Puppet or some

other deployment language

If the OS is not standard too much time will be spent

engineering around OS peculiarities

CM must be mastered

Every change to a standard OS must be recorded.

Every change must be programmed, no hand-

jamming

All deployments are now hands-off

You can almost get Continuous Delivery for free.

OS Customizations for only two reasons

Integrate into the Jenkins/Puppet

Integrate into the cloud provider from a standard

4/26/2014

7


OS Template Testing

Must Be Automated

RHEL, CENTOS, Windows change every

month, Manual checks must be limited to

keep up.

Security

STIG - Manual

OpenSCAP – STIG Automated

OWASP top 10 - Automated

Organizational Standards

Centralized Login

Standard directories

etc


Deployment Testing

This is a new area since deployments are now

code.

No traditional tools to test this type of

functionality

Currently methods are mostly manual.

Small automated tests must be constructed

to help with orchestration.

It either worked or it didn't

Manually sifting through logs to

troubleshoot.

Who does this?

Unfortunately, either a sysadmin or a

4/26/2014

8


Automated Testing

Automated testing must be an established

practice.

You cannot perform the volume of tests

required for CI efficiently without some sort

of automated check to separate the good

changes from the bad changes.

The cost and practice must be understood

within the development and testing

organization.

Testing resources must be able to handle max

capacity.

Tests must be able to run in parallel against

s


Role of Manual Testing

Manual Functional Testing DOES NOT GO AWAY!

Automated tests are not as effective as a person

because they are so specific.

Manual Testing is now a precious resource that

can only be applied in a limited fashion.

Time is what limits Manual testing

1-2 weeks of testing while code is written.

2-3 weeks of total testing effort.

Testing effort is divided.

Automation of new functionality testing.

Spot manual testing.

Manual free form regression.

Targeted regression based on automated results.

4/26/2014

9


Disposition of the Systems

Test reporting and the cloud provider must be

linked through a dashboard.

Manual testers must be able to verify failures.

Testers must have access to the broken

running system

Access will allow reproducibility and

verification

Developers must be able to identify broken

VMs

Access to the app to see the unexpected

behavior


Orchestration

This is the combination of architectural

components and services in a

meaningful fashion to produce a running

system.

For this example:

Database

App server

Database initialization

App server initialization

Coordinated starting of all the services

4/26/2014

10


Costs vs Velocity

Velocity comes with parallelism.

Tests running in parallel

Code being tested in a pipelined fashion.

Cost

Number of testing machines and infrastructure.

Number of parallel instances x machines per

instances

Storage – keeping failed instances around

Budget and Team size

Even with infinite money if you have a team of 5, the

400th VM is likely only giving very little marginal utility.

When money is the constraint, developer and tester

habits must best utilize the limited VMs wisely.


Data Analysis

In order to move quickly the right people must get

the right data.

Test Results

Quick access to the failed test

Quick access a screen shot of the failure

Ability to assign errors to the proper developers

Integrated bug tracking.

Server Analysis

Developers need to zero in on failed servers

quickly

Developers need to link failures to their code

quickly

Developers need to separate system/OS failures

4/26/2014

11


Dashboard (Data Analysis

Visualization)Centralization of results.

Sonar Cube

Custom pages

Views for Management, Testers, and

Developers.

Traceability

Features all have GUIDs

GUIDs linked to code commits

Unit tests linked to GUIDs

Automated tests are linked to feature

GUIDs

VMs linked to list of new GUIDs


What is Coveros Doing with it for

Customers?Training and Process improvement

Being the exemplar on a project to

move this practice to other projects.

Showing the change in the cost to test

a feature within an organization

Implementing pieces of the CI to show

marginal improvements.

Using the gains marginal gains to sell

the whole process.

4/26/2014

12


Level of Effort

The demonstration needs to be customized

for each environment.

Very few projects that I have dealt with are

“the same”. While this demo can be a guide,

I don't believe it can be generalized to a

solution, each environment must be

customized.

This implementation required research into

the APIs and inner workings of several

complicated software suites and services.

The engineering effort is commensurate with

the complexity of the architecture.


Why Link All These Technologies

Together?The industry is moving this way in various forms.

AWS templates

Service Mesh

JuJu

Hadoop

Flexiant

Even though the products are varied, they all share

general themes.

Inventory

Relational Mapping

OS Templates

Virtual Network Capabilities

If you know one you can have a feel for them all.

4/26/2014

13


Thoughts? Questions?

Thank you for your time.

using devops to improve software quality in the cloud

Technology

glennbuckholz coveros

code change

glenn buckholz coveros

deployment code

architecture copyright

demo copyright

code commits

standard os