deployments in one click!

D E P L O Y M E N T S I N P R O D U C T I O N W I T H O N E C L I C K

M A N U E L D E L A P E Ñ A P E Ñ A S O F T WA R E C R A F T S M A N S H I P T O L E D O 2 0 1 5 . 0 4 . 3 0

http://www.improve-foundations.com/wp-content/uploads/2011/11/improve_solutions1.png

M A N U E L D E L A P E Ñ A

S O F T W A R E E N G I N E E R A T L I F E R A Y, I N C .

@ M D E L A P E N YA

M O S T A C T I V E G I T H U B U S E R O N # C L M

M E E T U P D R E S S - C O D E

• Ask me whenever you need

• Stop me if I’m speaking too fast

• Let’s have a conversation about it, but if it’s too long, let’s have a beer after the meetup!

A B O U T T H I S M E E T U P

– S O F T W A R E C R A F T S M A N S H I P T O L E D O

“Fomentar y compartir las metodologías ágiles y las buenas prácticas en Ingeniería del Software en los procesos de desarrollo en Castilla-La Mancha.”

– J AV I E R G A R Z A S

“BUILDING SOFTWARE IS NOT LIKE BUILDING HOUSES OR CARS”

http://www.javiergarzas.com/2011/02/diferencias-software-fabricacion-tradicional-1.html http://www.javiergarzas.com/2011/02/diferencias-software-fabricacion-tradicional-2.html

http://static.flickr.com/2177/2155970334_f3b8da87d1.jpg

0 . W H A T D O E S C I M E A N ( F O R M E ) ?

• Merge code/commits more frequently

• Deploy the code on different environments (for testing purpose)

• Strongly supported by (any kind of) testing

• CI is NOT a tool, is a practice

• Significant degree of discipline for all the team

• All team members are involved and collaborating to improve software quality

0 . W H AT D O E S C I M E A N ( F O R M E ) ?

• Merge code/commits more frequently

• Deploy the code on different environments (for testing purpose)

• Strongly supported by (any kind of) testing

• CI is NOT a tool, is a practice

• Significant degree of discipline for all the team

• All team members are involved and collaborating to improve software quality

0 . W H AT D O E S C I M E A N ( F O R M E ) ?

MANTRA: Software works all of the time, specially just BEFORE and just AFTER your changes!

W H AT A R E W E G O I N G T O S E E ?

1. Environments

2. Deployments types

3. Manual Deployment: a short example

4. Automate it!

5. CI Best practices

1 . E N V I R O N M E N T S

http://www.igst.com/img/tech/servervirtualization.jpg

http://www.nopuedocreer.com/quelohayaninventado/wp-content/images/2008/08/cardboard.jpg

http://www.toonpool.com/user/20546/files/quality_control_1529395.jpg

http://www.afocor.com/images/formacion.jpg

http://www.starwarswallpaper.org/wp-content/uploads/wallpapers/hd-star-wars-death-star-wallpaper-800x600.jpg

W E A S D E V E L O P E R S …1 . E N V I R O N M E N T S

• …want to move our code from our local computer to production in the least possible time

Local > Dev > Test > Pre > Prod

• …want all of them be very very similar

• …want to collaborate more with Ops team, as we don’t have many times access to several of those envs

2 . D E P L O Y M E N T S

http://sd.keepcalm-o-matic.co.uk/i/keep-calm-and-deploy.png

• Frequency

• Annually

• Semiannually

• Quarterly

• Monthly

• Weekly

D E P L O Y M E N T T Y P E S2 . D E P L O Y M E N T S

4 4 4 4 4 4 4 4 4 4 4 4 4

• Completeness

• Full

• Partial

D E P L O Y M E N T T Y P E S2 . D E P L O Y M E N T S

Monolith 100%

( M I C R O - ) S E R V I C E S D E P L O Y E D

0

7,5

15

22,5

30

0 4 8 12 16

20% 34% 12%50%

70%35% 68%22%

35%

M O T I VAT I O N T O D E P L O Y2 . D E P L O Y M E N T S

• Who?


• Who?

• Customer (usually)


• Who?


• Why?


• Who?


• Why?

• Bug fixing


• Who?


• Why?

• Bug fixing

• New functionalities


• Who?


• Why?

• Bug fixing


• When?


• Who?


• Why?

• Bug fixing


• When?

• Deployment policies


3 . M A N U A L D E P L O Y M E N T

http://ep01.epimg.net/cultura/imagenes/2012/08/23/actualidad/1345709139_149007_1345712998_noticia_normal.jpg

• Web Application

• Java WAR (War ARchive)

• Tomcat

• MySQL DB

• FileSystem

E X A M P L E3 . M A N U A L D E P L O Y M E N T S

PA C K A G E G E N E R AT I O N3 . M A N U A L D E P L O Y M E N T S

• Fetch code from the repository

• Generate the package using…

• IDE ???

• command line tools

• Name the package

• v1.0.1, v3.3.7, v5.0.0 …

D E P L O Y T O …3 . M A N U A L D E P L O Y M E N T S

• Send the package to Dev. environment

• Assert that it works as expected

• Manual or automated testing?

• Are there DB updates?

• Are there O.S. changes?

• Firewalls, WebServers, FileSystem…

• Which one is the next environment?

C O N F I G U R E E N V I R O N M E N T3 . M A N U A L D E P L O Y M E N T S

• Configure Tomcat datasources…

• Configure Apache sites…

• Execute SQL upgrades on DB…

• Create the file system structure…

T E A M S I N V O LV E D3 . M A N U A L D E P L O Y M E N T S


Developers


Developers

DBA’s


Developers

DBA’s

SysAdmins


Developers

DBA’s

SysAdmins

QA

I just want to deploy the package to the environment!!

http://www.city-data.com/forum/attachments/jacksonville/104205d1354214951-case-you-were-wondering-landing-still-sheldon_cooper.jpg

4 . A U T O M A T E I T !

http://homeinteriorsdesigns.info/wp-content/uploads/2014/07/automation-robot.jpg

N AT U R A L E V O L U T I O N4 . A U T O M A T E I T !

• No server: compile by hand, fewer commits, painful integration

• Nightly builds: automated compile, maybe tests executions, daily commits

• Continuous Integration: compile with every commit, test execution, notifications on breakages. Possible deployment to dev/pre environment

• Continuous Deployment: deployment to production, with one click

• Continuous Delivery: automatic deployment to production, with every commit

W H AT T O A U T O M AT E4 . A U T O M A T E I T !

• Configure the agnostic environment

• Build the software

• Quality metrics

• Tests execution

• Unit + Integration + Functional

• Deployment

• Everything?

T E A M S I N V O LV E D4 . A U T O M A T E I T !

Developers

DBA’s

SysAdmins

QA

P I P E L I N E S4 . A U T O M A T E I T !

http://www.jsg.utexas.edu/lacp/files/PGC_Pipeline_Data.jpg

P I P E L I N E S4 . A U T O M A T E I T !

• Define the minimum valid steps

• I.e.: Build + Metrics + Test + Deploy

• Fail fast -> take actions ASAP

• Display to the whole team each step status

• Choose your automation tool, or build it

C O D E M A N A G E M E N T4 . A U T O M A T E I T ! - P I P E L I N E S

• SCM

• NICE TO HAVE => distributed SCM (Git)

• SCM Server

– A N O N Y M O U S

“The code must be independent from the environment”

https://katiegrusz.files.wordpress.com/2013/04/independent-woman-copy1.jpg

H A R D C O D E D E N V I R O N M E N T4 . A U T O M A T E I T ! - P I P E L I N E S

H A R D C O D E D E N V I R O N M E N T4 . A U T O M A T E I T ! - P I P E L I N E S

FORBID

DEN!!!

B U I L D P H A S E4 . A U T O M A T E I T ! - P I P E L I N E S

• Fetch a specific commit HASH, i.e. last one

• Reproduces the state of software at that point

• Compile

• Use command line tools

B U I L D P H A S E4 . A U T O M A T E I T ! - P I P E L I N E S

• Fetch a specific commit HASH, i.e. last one

• Reproduces the state of software at that point

• Compile

• Use command line tools

Why not being able to reproduce the state of each HASH commit?

– @ T P R U M B S

“A programmer that doesn't explicit dependencies, or automate his/her build, has not

suffered enough”

M A N A G E D E P E N D E N C I E S4 . A U T O M A T E I T ! - P I P E L I N E S

M E T R I C S P H A S E4 . A U T O M A T E I T ! - P I P E L I N E S

• Static Analysis

• Integrate common-use tools

• Findbugs, PDM, CPD, Code Coverage, …

• Measure everywhere: on every peer-review

• Shall we create issues on our ticket system on each detection?

T E S T P H A S E4 . A U T O M A T E I T ! - P I P E L I N E S

• Write tests!!!

• TDD is a choice, but not a requirement

• Make them fast, to execute it locally

• Test everywhere: on every peer-review

• Share tests results!

http://www.seguetech.com/sites/default/files/article/726/image-2447.png

• Unit tests

• Isolated code

• Integration tests

• Relations between components

• Functional tests

• GUI tests + Exploratory Testing

• Non-Functional tests

• Performance, Security, User Experience, Usability

T E S T P H A S E4 . A U T O M A T E I T ! - P I P E L I N E S

PA C K A G E P H A S E4 . A U T O M A T E I T ! - P I P E L I N E S

• Your package represents the software in a specific HASH commit

• Use command line/automated tools

• The package must be environment-agnostic!!!

• Promote the package between environments

D E P L O Y P H A S E4 . A U T O M A T E I T ! - P I P E L I N E S

• Deploy the package to your runtime

• App server, host, virtual machine, etc.

• Something went wrong?

• Deploy previous commit, as you positively know is safe

T O O L I N G4 . A U T O M A T E I T !

• Own scripts

• Automation servers

• Jenkins, Go, Hudson…

• O.S. configurations

• Ansible, Chef, Puppet, Docker (Linux-kernel based)

http://cdn2.hubspot.net/hubfs/381387/Blog-image-tracking/ElasticBox-DevOps_Open_Source_Tools.png

D I S P L AY R E S U LT S4 . A U T O M A T E I T !

• Each team member must know about pipeline status

• Create dashboards with aggregated test results:

• Tests failures

• Build statuses

• Committers that caused the breakage

5 . B E S T P R A C T I C E S

http://www.sqlsoldier.com/wp/wp-content/uploads/2012/02/BestPractices.jpg

C O M M I T S T R AT E G Y5 . B E S T P R A C T I C E S

• A commit = a point in the project’s History

• A commit should be…

• Small

• Significative

• Descriptive commit message

• Issue referenced in the commit message

• Will you remember its purpose 6 months later?

C O M M I T S E X A M P L E5 . B E S T P R A C T I C E S


#45 Awesome functionality



#45 Create tests



#45 Create tests

#45 Refactor Service Layer



#45 Create tests


#46 Enable email sending on backend



#45 Create tests


#46 Enable email sending on backend#46 Add ‘Send email’ button on desktop



#45 Create tests



#46 Add Javadocs



#45 Create tests


#47 Fix bug on View layer (caused by #45-a)


#46 Add Javadocs



#45 Create tests


#47 Fix bug on View layer (caused by #45-a)#47 Update tests introduced by #45-b


#46 Add Javadocs



#45 Create tests




#46 Add Javadocs

#47 Add tests to prevent regressions



#45 Create tests




#46 Add Javadocs


Revert #46 Add Javadocs



#45 Create tests




#46 Add Javadocs


Revert #46 Add ‘Send email’ button …




#45 Create tests




#46 Add Javadocs


Revert #46 Enable email sending…

Revert #46 Add ‘Send email’ button …


R E L E A S A B L E C O M M I T S5 . B E S T P R A C T I C E S

• If a single commit passes ALL the safety checks on the pipeline… why not releasing in that state?

• Apply that sentence to commit history:

P E E R R E V I E W I N G5 . B E S T P R A C T I C E S

• The more pairs of eyes over the code the better

• Learn by watching

• Experienced team members teach beginners

• Leverage the tools your SCM provides (pull requests)

• Pair programming if time-zone allows it

B R A N C H I N G5 . B E S T P R A C T I C E S

• Never never never code on master branch

• Use ‘master’ branch as release, which IS ALWAYS stable!

• Have a ‘develop’ branch, which COULD be unstable

• Create feature branches for new commits

• Merge features into develop.

• Once ready to release, merge to master and tag

B R A N C H I N G : G I T F L O W5 . B E S T P R A C T I C E S

B R A N C H I N G : G I T F L O W5 . B E S T P R A C T I C E S

Git-Flow: http://danielkummer.github.io/git-flow-cheatsheet/index.html

D O N ’ T C H E C K - I N O N A B R O K E N B U I L D


• Any day, 8:00. Your build is broken

• Got an email from the CI server?

• Solve errors as soon as possible, do not start anything new

• Identify the cause very quickly

• Best position to work out

D O N ’ T C H E C K - I N O N A B R O K E N B U I L D


Current breakages + your changes

Adding complexity for the build to be fixed

Different problems

More time to fix the build

Still want to check-in?

You’ll get used to seeing the build broken all the time

http://www.pqmonthly.com/wp-content/uploads/2013/12/ani-1.png

A LW AY S R U N T H E T E S T S5 . B E S T P R A C T I C E S

• Do you remember that every single commit should be releasable and HAVE TO be small, significant, isolated?

• It has to pass the tests locally

• It has to pass the tests on the pipeline on the CI server too…

• with each commit or code submission!!!

W R I T E W E L L - W R I T T E N T E S T S5 . B E S T P R A C T I C E S

• They will verify that the commit works as expected

• Test functionality…

• And loops, conditional, input values…

• Not only the Happy Path

• Did I say that you have to execute the tests? :-)

C O N T R O L L E D E N V I R O N M E N T5 . B E S T P R A C T I C E S

• Developer machine: OS configurations, OS tuning, third party libraries/drivers…

• CI Server: Same conditions. In every build. For everyone.

• Repeatable results!!

• Centralized Information Repository

• Always display what is happening (CI logs, notifications…)

W A I T F O R T H E T E S T S T O PA S S5 . B E S T P R A C T I C E S

• Developers loop: start > work > finish??? > start

• Change of context very quickly

• Are you sure that your code works as expected?

• Are the test finding potencial bugs on it?

• If you don’t monitor the build, you cannot answer those questions until it’s very late and expensive.

• Work for the build execution to be fast

A S S U M E E R R O R S


“Everyone can commit errors”

“Errors are an expected part of the process”



N E V E R G O H O M E O N B R O K E N B U I L D S5 . B E S T P R A C T I C E S

• Friday, 14:58. Your build is broken

• What to do?

• Leave the build broken and go



• What to do?


• Leave late, trying to fix it

• Revert changes and try next week



• What to do?


• Leave late, trying to fix it

• Revert changes and try next week

If you keep the build green, other developers will be happy to pull safe code from SCM

D I S T R I B U T E D E X A M P L E5 . B E S T P R A C T I C E S

B U I L D M A S T E R5 . B E S T P R A C T I C E S

• Maintains the build

• Police it

• Write access to SCM

• Revert/Prioritize commits

• No personal offenses

• Critique the code!

• Accept rollbacks

https://s-media-cache-ak0.pinimg.com/236x/92/03/2a/92032a9e9ee898ac6106867421d81534.jpg

B U I L D M A S T E R D R A W B A C K S5 . B E S T P R A C T I C E S

• Feeling of culprit if the build is broken

• Team starts seeing him/her as a punisher

• Role gets tired of being responsible for others mistakes

• It’s not worth it pursuing people

• Team gets used to have a house-cleaner

https://s-media-cache-ak0.pinimg.com/236x/92/03/2a/92032a9e9ee898ac6106867421d81534.jpg

http://homeinteriorsdesigns.info/wp-content/uploads/2014/07/automation-robot.jpg

A U T O M AT E D B U I L D M A S T E R5 . B E S T P R A C T I C E S

• Test each pull request!!

• Create a reduced pipeline for them

• Notify developers on failures

• Never accept potential failures

• Be aware of false positives!

B E P R E PA R E D T O R E V E R T5 . B E S T P R A C T I C E S

• Developers are selfish about their code

• Developers don’t like others criticizing their code

• Important! Not to blame the developer

• We are members of the same team

• For the sake of quality, revert to previous change set, as it is safe. Why?

“Don’t be afraid, my friend… and revert”

- B R U C E L E E ? ? -

http://i2.asntown.net/9/bruce-lee-the-lost-interview.jpg

B E P R E PA R E D T O R E V E R T5 . B E S T P R A C T I C E S

http://szokblog.pl/ShareImages/1345223838846.jpg

T I M E - B O X F I X I N G5 . B E S T P R A C T I C E S

• Before reverting, try to fix it in a small amount of time

• 15 minutes?

• If you aren’t finished, revert to previous version

• Don’t be the sweeper or the last bullet and notify users who broke the build, avoiding magical fixes.

• Put mechanisms the sooner: on each code review!

T U N E T H E N O T I F I C AT I O N S5 . B E S T P R A C T I C E S

• No notifications are bad

• Bad notifications are worse

• Send the right email to the right people, otherwise they will start ignoring CI notifications

• Manual email when reverting explaining why?

• Wall of shame? list of committers who wrote the build

D O N ’ T C O M M E N T FA I L U R E S5 . B E S T P R A C T I C E S

D O N ’ T C O M M E N T FA I L U R E S5 . B E S T P R A C T I C E S

• Always last resort, as it will hide real problems

• Apply these sample rules ASAP:

• Has a regression been found by the test? -> Fix the code!

• Is one of the assumptions of the test no longer valid? -> Delete it!

• Has the application really changed the functionality under test for a valid reason? -> Update the test!

R E S P O N S I B I L I T Y5 . B E S T P R A C T I C E S

• Take responsibility for all breakages resulting from your changes.

• If you commit a change and all the tests you wrote pass, but others break, the build is still broken.

• It’s usually a regression

• Fix all the tests not passing, asking component leader for collaboration

T D D5 . B E S T P R A C T I C E S

• It’s not a requirement!!

• Tests are essential to CI

• Create tests when developing a new piece of functionality or fixing a bug:

• It’s an executable specification of the expected behavior of the code to be written:

• Application design, Regression tests, Documentation of the code

• Creates a safety net for future refactors

C O N C L U S I O N S6 . S U M M A R Y

• Discipline

• Luke, use the tools!

• Everyone can commit errors

• Wait for tests to pass

• Display what is happening

• Don’t be afraid of reverting

• Responsibility

• Tests/Pipeline stages as a safety net