devops-as-a-service: towards automating the automation

Accenture Architecture Services

DevOps-as-a-Service:Towards Automating the AutomationDevOps Enterprise SummitNovember 8, 2016

Keith PleasDevOps Architecture Sr. [email protected]

2

Accenture’s Global DevOps PracticeWe support a large variety of industry standard tools – both open source and vendor/proprietary tools.

250 Clients on Accenture DevOps Platform

+ 100s more Clients with DevOps services

4400 DevOps Trained

1700+ DevOps experts

350 DevOps Certified

250+ DevOps Factory FTE

7 countries

15+ Years heritage of high-

performance CI/CD & DevOps

2000+ Members of DevOps community on Yammer sharing strategy, knowledge & collaboration

759 People deployed to AT&A

Operations Domain alone

10%+Cost reduction

across IT delivery

New features in hours not days

DevOps Leader• Accenture developed ADOP DevOps platform (now open-sourced)• DevOps Managed Services(DCSC)• Development Lifecycle Management• Our in-house developed Cloud Platform• DevOps maturity assessment• IP/Frameworks: Methods and Tools

Services• Assessment and Strategy• Application Architecture • DevOps Implementation

– Continuous Integration– Continuous Delivery– Continuous Testing

• Organizational Change• IT operating model transformation• Managed Service

Components• Build & deployment automation• Software configuration management• Test automation• Environment management & monitoring• Cloud environment provisioning• Development & deployment architecture

Key principles• Test early and often• Improve continuously• Automate everything• Cohesive teams• Strong source control• Deliver small increments• Experiment frequently without regret

Enabling DevOps Professionals

Global Expert Network

Toronto

Chicago

Wilmington

Sao Paulo

London

Málaga

Mumbai

Bangalore

Mauritius

Dalian

Manila

Prague

Atlanta

Note: Other geographic and industry focused delivery centres exist

Courses/TrainingsDevOps Introduction

• Introduction to various DevOps concepts and how to apply them

• Illustrative examples/case studies

DevOps Academy • Intensive 2 day classroom course• Continuous Integration and Delivery• Infrastructure as Code

Multiple technology specific trainings

Copyright © 2016 Accenture All rights reserved.

DevOps and Accenture

2009 2015201420132012201120102007 200820062005

Agile ConferencePatrick Dubois & Andrew Clay ShaferAgile Infrastructure

Qcon: Daniel SchauenbergEtsy 50 Deploys per day

Velocity ConferenceJohn Alspaw & Paul Hammond: Flikr Deploy10 time per day

DevOps Days Belgium #DevOps

Accenture DCSC Shared Service Created

2016

Copyright © 2016 Accenture All rights reserved. 3


The Joel Test (August 9, 2000)A friend from my distant past

• Do you use source control?• Can you make a build in one step?• Do you make daily builds?• Do you have a bug database?• Do you fix bugs before writing new code?• Do you have an up-to-date schedule?• Do you have a spec?• Do programmers have quiet working conditions?• Do you use the best tools money can buy?• Do you have testers?• Do new candidates write code during their interview?• Do you do hallway usability testing?

www.joelonsoftware.com/articles/fog0000000043.html

http://www.joelonsoftware.com/articles/fog0000000043.html

What is DevOps?Is it culture?



Our Challenges: Two Lenses• Technology– Automation – Software-defined Infrastructure– Automate the Automation – Software-defined DevOps– Templates, tools and stacks– Enterprise-specific concerns: licensing, security, support– Technical debt

• People – Do what you do best - leverage others– Open Source culture– New paradigm for engaging Dev and Ops


Focus on Technology

7


Roger’s Curve with Moore’s “Chasm”How innovative are you?


Modern IT methodologies emphasize collaborationDevOps in context

IDEA PLAN DESIGN BUILD DEPLOY TEST RELEASE OPERATE

DevOps

Continuous Delivery

Continuous Integration

Execution Architecture Development Architecture Testing Strategy Ops Arch

Design Thinking

Lean Manufacturing

Agile Development


What about those “non-functional” requirements?Auditability, consistency, maintainability, modifiability, operability, recoverability, repeatability, resilience, reusability, robustness, scalability, security, supportability, scalability, testability, usability

• How are you managing your own continuous delivery artifacts?• Are all of your build pipeline settings stored in source code control?• Can you rebuild your pipeline via automation?• Can you easily scale your pipeline?• Can you quickly onboard resources and make them productive?• Could you port your pipeline to a new build environment?• How are you managing your DevOps technical debt?


Our first “enabler”

11

12

DevOps Maturity Model – Framework


DELIVERY ORGANISATION AND METHODOLOGY

• Release Frequency• Delivery Organisation• Delivery Process & Methodology• Metrics & Governance

LEAN DELIVERY GOVERNANCE & PROCESSES

• Governance Approach• Continuous Improvement• Product Management• Delivery & Change Control

AUTOMATED RELEASE OF SOFTWARE • Source Code Version Control• Deployment Process• Deployment Artefact Management

CONTINUOUS INTEGRATION • Traceability• Continuous Integration• Code Quality & Unit Test

CONTINUOUS DELIVERY (INCLUDING AUTOMATED QA)

• Test Strategy• Test Automation• Test Data Management• Deployment Pipelines

AUTOMATED OPERATIONS • Monitoring• Organisational Considerations• Resilience

SOFTWARE DEFINED INFRASTRUCTURE & CLOUD

• Environments• Infrastructure Suitability• Responsiveness

PLATFORM/APPLICATION ARCHITECTURE • Architecture• Impact of Component Failure• Business Enablement

Architecture

Methodology

Lean Governance & Process

Continuous Delivery Pipelines

SW Defined Infrastructure

Automated Release of SW

Automated Operations


13

DevOps Maturity Model - Overview


REPEATABLE CONSISTENT OPTIMISEDAD-HOC LEADING2 3 41 5

DELIVERY ORGANISATION AND METHODOLOGY

Release scope poorly defined; subject to catastrophic and ad hoc change requests

Releases duration exceeds business need; releases face disruptive change

Release cadences well defined but exceeds business need; Requirements are stable;

Release on demand, multi-speed releases; time-box meets to business need (eg: monthly)

Small changes pushed through the pipeline ; Continuous deployment enables innovation

LEAN DELIVERY GOVERNANCE & PROCESSES

No defined or consistent applied delivery approach

Delivery approach is well defined, but largely manually governed

Industrialised project delivery; governance integrated into delivery and product management established

Blame free culture embedded in governance that uses real-time data rather than manual inputs

Continuous Delivery / Deployment supporting agile business change with automated governance

AUTOMATED RELEASE OF SOFTWARE

Mostly manual deployments Some CIs automated, environment tailoring required, no enterprise tools

Fully automated Single-touch deployments into environments.

Functioning environments can be build from nothing programmatically.

Zero-touch zero- downtime deployments

CONTINUOUS INTEGRATION

No consistent use of version control – builds cannot be traced back to source code

Source code consistently managed in VCS; releases traceable to source

Developers integrate changes by checking into trunk on regular basis (daily)

Build is typically green – if build breaks developers do not make other changes until resolved

Build is typically green – if build breaks the CI tooling automatically reverses the failed change

CONTINUOUS DELIVERY (INCL AUTOMATED QA)

Fully manual test scripts Testers run a harness / suite Test harness / suite run automatically for some envs

Automated test suites enforce a quality gate

Tests run as functional monitoring

AUTOMATED OPERATIONS

No monitoring tools Tools in place, but not configured beyond basic OS checks

Functional monitoring in place and DR is available

Environment and application health monitored and proactively managed

Service level monitoring (perf, usage) integrated with infrastructure and self-healing available

SW DEFINED INFRASTRUCTURE & CLOUD

Environments are managed ad-hoc without consistent blueprint

Environments follow standards but are manually created

Environments can be created automatically and follow common blueprint

SW defined infrastructure that closely resembles Production throughout SDLC

Utility based computing that leverages auto-scaling for several SDLC aspects

PLATFORM/APPLICATION ARCHITECTURE

Big monolithic application architecture

Architecture based on platforms, business requires IT for every change

Resilient Architecture minimising single point of failures

Modular and scalable architecture, that allows for some business self service configuration

Microservice based architecture supported by end-to-end ownership in teams

14

DevOps Maturity Model - Overview


REPEATABLE CONSISTENT OPTIMISEDAD-HOC LEADING2 3 41 5


Our second “enabler”

15


Structure, Standardize, AutomateThe complete pipeline to operations with ADOP

IDE• Eclipse• X-code• …

Git / Gerrit• Source code

repository• Code branch

management• Code review

Jenkins• Continuous

Integration• Multi-

platform builds

Code Analysis• Code quality• Coding

standards rules

• Compliance

Deploy• Dev app store• Release

repository• Environments

Test• Component• Behavioural• Functional• Security• Performance

Development Source Control

Code Review Build Code

Quality Deploy Testing A/B Test in Production

Please visit github.com/Accenture for more information.

Operations• Search &

Analytics• Log Processing

& Analysis• Visualization

https://github.com/Accenture

17Copyright © 2016 Accenture All rights reserved.

Fully automated solution for implementing ADOP

https://hub.docker.com/

Availability Zone

AWS region EU (Ireland)AWS

AWS cloud

ChefServer

Public security group

Public subnet

Internet

AWSAWS Cloud

AWS CloudFormation

CF template json

VPC

Single ADOP VM

ADOP Provisioning

Gerrit ELK

Nexus

Sonar

Nginx Jenkins

Platform Extensions

Load Cartridge TomcatEnvironment

CloudInit

https://docker.accenture.com/

18

Fully distributed, highly available, auto-scaling, and self-healingADOP platform


19

ADOP CartridgesAdditional enablers


Dedicated versus Managed ADOPDedicated Instance

Single tenancy

Supported by project or by DCSC

Installed on-cloud or on client infrastructure

Supports cartridge/blueprint configuration for various technologies

Managed Service

Multitenancy

Provided as a managed service by DCSC

Embeds ADLM and Innersource

Costs recovered by service-charge

Hosted primarily in AWS

Configurable to support project requirements

Clients Internal Projects

Users

Jira/Confluence 148 7 10,800

Innersource 69 2,820 10,600

Usage Jan 2016


Dedicated versus Managed ADOP Tools and ServicesDedicated Instance

On Premise

(Roadmap)

(Beta)

Selenium

AOWP

Digital/01

(Beta)

(Roadmap)

+Others

AVS

Hybrid

Managed Service

(Roadmap) (Beta)

Compatibility(Roadmap)



DCSC’s model• Service Catalog of pre-configured PMO tools

and reporting, enabling rapid mobilization and low cost run

• A number of processes benefit from a Business Process Management tool (Activiti), enabling an automated chasing and reporting

• Services under continuous improvement, rolling out the latest technology to projects

How to do shared services?

Founding principle for myPMO Services: Structure....Standardize…. Automate

To reduce local team effort


• The spectrums: standardization, governance, processes• Consolidating and simplifying the tools portfolio & licensing• Maturity assessments and coaching clients• Experiment with ways to obtain better feedback from clients e.g. periodic surveys, feedback channels, user groups, social media

• Geo-location• Metrics• Security, including VPN tunnels across trust boundaries– Detection (e.g. Security Monkey) rather than imposing restriction

What are the tradeoffs with shared-services?


• Proof of concepts and MVPs• Onboarding and training• Starting point for your own DevOps stack

• Starting point for your own shared services

How might you use ADOP?


Focus on People

25


Just one example from Jenkins World…


Benefits of a shared service model

• Re-use of assets between teams• Sharing of knowledge between teams• Flexible workloads e.g. person X from project A spends and afternoon helping person Y on project B when needed

• Creating strong culture that fosters shared values and creates a great and meaningful place to work

• Being able to have our own culture and identity that we can control

Focus on the People


Lessons learned from a decade of DCSC

• Avoiding the HiPPO (Highest paid person’s opinion)• Specialization, cross-training, silos, and scalability• Trust versus Empowerment– johnclapham.wordpress.com/2016/04/22/inflicting-trust/

• Relationships and staying connected with a lack of face to face• Ensure internal product teams don’t end up in a silo and lose touch• Flexibility for mobility of individuals between roles and meet their needs balanced with staying on engagement through realization

Focus on the People

https://johnclapham.wordpress.com/2016/04/22/inflicting-trust/


Where do we go from here?

29


• Ever-increasing complexity• Work harder, work smarter, or get help• Structure, standardize, automate

Evolution: Fact or Theory?


Large media clientComplex DevOps tools landscape

Log Aggregation and Op. Dashboards

Programme and Project Management

ConfigurationManagement

Test Automation

RequirementsManagement

Functional Testing


Continuous Delivery

ArtefactManagement

SourceControl

Change Management

Defect Management

Continuous Integration/Build Build Orchestration

JenkinsJenkins

Risks and Issues

Continuous Integration Build

DeploymentOrchestration

Jenkins

Code Validation Functional and API Tests Load Injection

Design DocumentationTeam Dashboards MI and Reporting

Security Tests

Travis CI

AWSS3

Github

Jenkins

Slack and Lita

ZapCharles

Box

Composer

BlazemeterPostmanSelenium

Team Communications

Slack

Netsparker

Development and Prototyping


Let’s look at those NFRs again…

• Versioning, packaging, dependencies• Chef and Puppet had a head start and showed us the way• Cloud Formation templates• Security settings• Certificates• ….

Where are our DevOps artifacts?

Auditability, consistency, maintainability, modifiability, operability, recoverability, repeatability, resilience, reusability, robustness, scalability, security, supportability, scalability, testability, usability


In general the same way

• Repositories• DSLs (Domain-Specific-Languages)– Example: Gradle’s Groovy-based DSL for builds

• Jenkins “Pipeline as Code”- jenkins.io/doc/book/pipeline-as-code/• A new opportunity for automated testing

How do we manage these?

https://jenkins.io/doc/book/pipeline-as-code/


Ah yes, software factories…did that end well?

“Why I Hate Frameworks” – Benji Smith, September 30, 2005

• A hammer• A universal hammer• A hammer factory• A hammer-factory-building factory• A general-purpose tool-building factory factory factory

Modification - the fatal flaw of MDA and software factories and…

Automating DevOps

discuss.joelonsoftware.com/?joel.3.219431.12 34

http://discuss.joelonsoftware.com/?joel.3.219431.12


Putting it all together

35


• Technology– Do you have standards for your DevOps practices?– Do you rationalize your DevOps tools?– Are you considering your DevOps technical debt?

• People– Are you attracting or building DevOps talent?– Are you making the best use of the talent you have?

Does DevOps-as-a-Service make sense to you?

Questions to consider


• www.Accenture.com/DevOps

• ADOP Wiki and quickstart: accenture.github.io/adop-docker-compose/docs/quickstart/• Mark Rendell DOES15 - www.youtube.com/watch?v=8WRRi6oui34• “The Joel Test” - www.joelonsoftware.com/articles/fog0000000043.html• “Why I hate frameworks” - discuss.joelonsoftware.com/?joel.3.219431.12• DevOps Maturity Model- [email protected]• John Clapham “Inflicting Trust”: johnclapham.wordpress.com/2016/04/22/inflicting-trust/• Accenture DevOps blog: www.accenture.com/us-en/blogs/blogs-accenture-devops• Mark Rendell (DCSC) blog: markosrendell.wordpress.com• Mirco Hering (Maturity Model) blog: notafactoryanymore.com

Resources

37

http://www.accenture.com/DevOps

http://accenture.github.io/adop-docker-compose/docs/quickstart/







http://www.youtube.com/watch?v=8WRRi6oui34

http://www.joelonsoftware.com/articles/fog0000000043.html

http://discuss.joelonsoftware.com/?joel.3.219431.12

mailto:[email protected]

https://johnclapham.wordpress.com/2016/04/22/inflicting-trust/

https://www.accenture.com/us-en/blogs/blogs-accenture-devops

http://markosrendell.wordpress.com/

https://notafactoryanymore.com/