software architecture for devops and continuous delivery

Software Architektur für DevOps

und Continuous Delivery

Eberhard Wolff Freelance Consultant / Trainer

Head Technology Advisory Board adesso AG http://ewolff.com

Eberhard Wolff - @ewolff

Overview •  DevOps

•  Continuous Delivery

•  Architecture

•  7 Rules


DevOps Development + Operations

=DevOps

Organizational change

No more silos




•  Architecture

•  7 Rules


Continuous Delivery: Build Pipeline

Commit Stage

Automated Acceptance

Testing

Automated Capacity Testing

Manual Explorative

Testing

Release


Continuous Delivery Needs Deployment

Automation •  But it is not the same

•  Many test systems required

•  Automation less error prone


Continuous Delivery Releases

•  Blue / Green Deployments •  New version on separate production

environment

•  Canary Releasing •  Release to a few servers in the

cluster




•  Architecture

•  7 Rules


Architecture: Wikipedia Set of structures

needed to reason about the software system

Comprises software elements

relations between them properties of both


Why Architecture? •  System too large to understand

•  Understand components in isolation

•  Understand how components interact


Architecture & Non-functional Requirements

•  Architecture defines technical foundation

•  Technical foundation influences •  Performance •  Security •  Scalability •  Etc




•  Architecture

•  7 Rules


Example

Order

Warehouse Customer

Functional decomposition Technology independent


Order

Warehouse Customer

Database

Libraries e.g. jar, dll Method calls

One deployment unit


Architecture Simple

High performance


Change •  Modify order process slightly •  Customer & Warehouse unchanged

Order

Warehouse Customer

Database


Order

Warehouse Customer

Database



Commit Stage


Testing


Manual Explorative

Testing

Release Order

Warehouse Customer

Database


Build Pipeline •  Just one component changed!

•  Lots of unneeded work

•  Takes much too long


Order

Warehouse Customer

Database

Call Center Payment Provider

No retesting for new versions


Order

Warehouse Customer

Database

Libraries e.g. jar, dll Method calls

One deployment unit


Order

Warehouse Customer

Database

Services Service calls Three

deployment units


Components = Services •  Independent deployment

•  REST/JSON •  ProtoBuf •  MessageBus

•  Micro Services: even smaller?


Order

Warehouse Customer

Database



Commit Stage


Testing


Manual Explorative

Testing

Release Order


Release •  Just a single component

•  Much easier to do Blue/Green •  or Canary Releases

•  Rollback of single service also easier


Other Benefits •  Might use different technologies per

component

•  For specific requirements

•  For specific developer skills


#1 Small Deployment Units Less testing

Faster deployment Less waste

Easier rollback


Change •  Optimized algorithm for warehouses

•  Additional information must be passed in from order


Order

Warehouse Customer

Database

Interface changes Order must also be deployed


Problems •  Order and Warehouse must be

deployed together

•  Hard to coordinate

•  More complex pipeline •  Harder rollback


Solution •  New warehouse version supports

old interface

•  Warehouse can be deployed first •  …and then Order

•  Old interface must be phased out


Robustness Principle •  aka Postel’s Law •  Be conservative in what you send •  be liberal in what you accept •  e.g. accept calls to the old version of

the interface

•  Better interoperability


#2 Interface Backwards Compatibility

Faster deployment Deploy components independently

Less waste


Order

Warehouse Customer

Database

Bug Fix Warehouse must be deployed again


Problems •  Warehouse stores current state in

RAM

•  Information lost during update

Stateless


Stateless •  Keep components stateless

•  Might store state in database etc.


#3 Stateless Easier to update

State e.g. in database


Order

Warehouse Customer

Database

Component failure Domino effect


Component Interfaces •  Design for Failure •  I.e. do not fail if other component

fail

•  Default values •  Simpler algorithm •  Graceful Degradation

Architecture deals with

Failure

Resilience


Resilience & Small Deployment Units

•  Downtime during deployment might be acceptable

•  More flexibility for deployments


Circuit Breaker •  Do not call a broken system •  Instead throw exception •  Retry after some time

•  Protect broke system •  Do not call it in vain


#4 Design for Failure More stable

Better for Ops Better for new releases


Change •  SEPA

•  Additional customer information


Order

Warehouse Customer

Database

Column must be added to database


Database Changes Are Hard •  Schema changes

•  Converting existing data •  Potential lots of data

•  Hard or even impossible to rollback


1 - Keep database schema stable

2 - Database = component


Keep Database Schema Stable

•  Change database before components use it

•  Change database seldom •  i.e. schema changes not part of

deployment •  No rollback


#5a Keep database schema stable

No rollback Less agile


Databases = Component •  Support old “interface” •  i.e. schema


Order

Warehouse Customer

Database

Database must support different data structures

V42

V43


Dealing withDatabase Changes

•  Add an abstraction layer

•  Stored procedures •  No direct access to tables •  Persistence layer in the database •  Database is another component

with an API


Dealing withDatabase Changes

•  Support previous version schema

•  Default value for new columns •  Remove column only if not used

any more •  Use views •  …


#5b Databases = Component

Need to be backwards compatible Probably even more important than

for components


NoSQL •  Schema changes much easier

•  Can support different schemas in parallel

•  Still need to convert data •  Same rules for compatibility apply


Change •  Support for new logistic partner

•  Won’t start until next month


Feature Branch •  Additional instance of the delivery

pipeline

•  Deployment must be synced to start of new partner


Feature Toggle

•  Implement the feature

•  Can be activated by feature toggle •  i.e. part of the configuration


Feature Toggle •  Implementation & deployment

independent •  Can deploy early – less risk •  Can evaluate in production •  A/B testing •  Slow ramp up possible •  Probably one toggle: Old / new

version


Feature Toggle •  Add another dimension to the

architecture

•  Must eventually be cleaned up


DevOps •  Support for new logistics partner

crashes system

•  Feature toggle allows Ops to disable feature

•  One toggle per feature •  Resilience


#6 Feature Toggle Decouple release / new features

Resilience: Provide fallback


After the Change •  New logistics partner

•  Revenue declined

•  Some orders were lost


Business Monitoring •  Provide metrics with business

meaning •  i.e. revenue •  #orders •  #registrations •  etc •  Feed into monitoring / metrics


#7 Monitoring Dev provides (business) metrics

Ops provides monitoring infrastructure


•  #1 Small Deployment Units •  #2 Interface Backwards

Compatibility •  #3 Stateless •  #4 Design For Failure •  #5a Keep database schema stable •  #5b Databases = Component •  #6 Feature Toggle •  #7 Monitoring


Thank You!

software architecture for devops and continuous delivery

Technology

ewolff components

ewolff release

ewolff change

ewolff stateless

ewolff problems warehouse

ewolff problems order

ewolff devops development

ewolff build pipeline