continuous integration: a bittersweet love story

continuous integration:a bittersweet love story

DevOpsDays Baltimore 2017@pm_suzieHead of Product for ThoughtWorks Products

10+ years experience with agile, continuous integration and continuous delivery practices

Tweeting @pm_suzie

DevOpsDays Baltimore 2017@pm_suzie

Hello My name is Suzie Prince.

I am head of Products for Thoughtworks in the product division. We build tools for software development teams.

I have been working in software development for over 10 years. All of my professional experience has been working with agile, continuous integration and continuous delivery.

Im a Brit but I live and work in the San Francisco Bay Area.

If you want to catch me when Im not in Baltimore you can tweet me at @pm_suzie

over 20 years4000 people40 offices14 countries

@pm_suzie


For those of you not familiar with ThoughtWorks we are a group of passionate technologists.

We provide software delivery, pioneering tools and consulting services across the globe.

We are very diverse group with 4000 people, in 40 offices and across 14 countries.

In addition to helping our customers deliver better software, we also like to push the industry to be better.

We write books, publish tech roadmaps and support open source tools. Our commitment to software excellence and improving the IT industry is at the heart of what we do.

We created the first Continuous Integration server, Cruise Control. The first Continuous Delivery server with pipelines GoCD. We are the people behind Selenium. We also developed the concepts and practices that would become Continuous Delivery and the authors of the CD book were TWers at the time they wrote it. In fact Jez Humble worked with us in our product division when we were first building GoCD.

Ok enough about me and ThoughtWorks. If anyone wants to chat more about what we do or our products. Ill be in the here for most of the two days.

ci forever

@pm_suzie


Today Im going to tell you a love story about how we (the software industry) fell in love with Continuous integration.

Ill start by define continuous integration (or CI) and sharing what I think is important about the definition. Then share the benefits CI or what I like to call Why we fell in love with CIThen we will take a look at our relationship status with CIFinally review what we have learned And Im happy to take questions at the end

hands up

@pm_suzie


To get started Id like to know abit more about you all

Whos has heard of CI?Who is doing CI?

What is CI?

@pm_suzie


Ok for those who didnt know CI or need a review

@pm_suzie


Ci is where software developers or engineers who write code integrate their changes in a source repository, build the combined software and test that it works. Generally we use a CI server to do this and the process requires that we have self testing code. Its a feedback loop that helps us get better.

Thats basically the process. Quickly reviewed.

But there are principles and practices that are associated with the process too.

Jez Humble, continuousdelivery.comCI developers integrate all their work into trunk (also known as mainline or master) on a regular basis (at least daily).

@pm_suzie


Here is my favorite definition of the CI by Jez Humble

CI developers integrate all their work into trunk (also known as mainline or master) on a regular basis (at least daily).

includes two of fundamental principles of the practice.

dailymaster

@pm_suzie


That is that we check into master on a daily basis

For me these are very key elements to what makes continuous integration, continuous aka aka and integrated into master.

But there are a bunch of other things that you need to do to be doing CI such as ..

Check everything in

@pm_suzie


Everything you need to build the project is in source control.

The basic rule of thumb is that you should be able to walk up to the project with a virgin machine, do a checkout, and be able to fully build the system.

Build every commit

@pm_suzie


Building every commit once. Yes. You cant skip commits and still be truly continuous.

Automate the build

@pm_suzie


Automating your build can definitely help you be better / quicker / more efficient at CI

Keep it fast!

@pm_suzie


The whole point of Continuous Integration is to provide rapid feedback.

And we want this to be fast because its a feedback loop and feedback that doesnt come quickly is usually hard to deal with and ineffective

Have self testing

@pm_suzie


And of course you're going to need code that tests itself each time something is committed.

For self-testing code you need a suite of automated tests that can check a large part of the code base for bugs.

The result of running the test suite should indicate if any tests failed.

For a build to be self-testing the failure of a test should cause the build to fail.

Show visible failures

@pm_suzie


You also want to make failure visible Red lightsAlertsNoise Buzzers You want to know when your build is broken so you can fix it

Fix failures immediately

@pm_suzie


And then you want to fix you failures immediately No checking in on broken buildsNo long red streaks Red should be followed by green

dailymaster

@pm_suzie


But remember I think master and daily are two most important principles

Why we fell in love with CI?

@pm_suzie


Ok so now we have reviewed what CI is and what it means to me.

Lets take a look at why we fell in love with it in the first place.

Avoid merge conflicts

@pm_suzie


An obvious one Im sure and most like the reason that originally was a great driver for CI was to avoid merge conflicts.

These are the issues you get when you try to integrate a lot of changes at once. Traditionally this would have taken a long time at the end of a project potentially delaying releases and causing lots of rework.

CI allows this conflict to happen often so the conflict is small and manageable.

Fast feedback

@pm_suzie


The second benefit is fast feedback. Small changes frequently you get faster feedback,

This is a key driver for CI, but also agile and our favorite CD.

By doing things continuously you reduce assumptions. In the case of CI you are to assuming your code is going to work as expected. Instead of assuming you will actually confirm, quickly, that it works as expected and if it doesnt you get to resolve that immediately.

Reduce risks

@pm_suzie


And by doing it immediately you reduce assumptions AND risks later on in the project.

There are less likely to be unknowns later on if you get the hard stuff out the way quickly and often.

Collective ownership

@pm_suzie


Finally an often forgotten benefit of CI but one I think is as important as the other and particularly relevant to those of you working towards a DevOps culture is that you get collective ownership of the code, of the build server and of the product.

There are no code silos. Everyone can frequently see and integrate with the code of others. They can all own the health of the build and the status of the product.

In a world where DevOps culture is hot and people want to release frequently and often the benefits of CI grow and grow.

For example with CI, you have an always the most recent features ready to deliver and your ability to refactor and have evolutionary architecture are enhanced.

So thats why we fell in love in CI.

Lets take a look at ur love story so far

1996Steve McConnell describes the "Daily Build and Smoke Test" technique

2000Martin Fowlers first published his comprehensive Continuous Integration article

2010Book Continuous Delivery by Humble and Farley published Our story so far..

1993Grady Booch used the phrase continuous integration in Object-Oriented Analysis and Design with Applications

2007Book Continuous Integration by Duvall published2017Wikipedia lists 15+ continuous integration Tools

2001The first continuous integration server Cruise Control is released

2015CI described as most essential technical practice in SAFe

1998Continuous integration is cited as a core practice of Extreme Programming

@pm_suzie


Our love story started back in the 90s when we first meet CI.

Then1998 CI is cited as a key practice in XP and 2001 the first CI servers are released.

The love continues to grow in the 2000s as articles and books are published first chronicling CI and then CD.

Finally up to today we see that all agile method including management and process heavy SAFe (Scaled Agile Framework) include CI as key dev practice. And there are many many CI tools in the market place.

So looking at all that you think that our love would still be as sweet as it was in the beginning right? It should be sweet right? Well heres come the bittersweet.

COMPLICATED

ITS

@pm_suzie


Honestly our relationship with CI its pretty complicated. Let me tell you why.

http://stackoverflow.com/research/developer-survey-2016#work-checking-in-codeHow often do you check in or commit code?

@pm_suzie


This is some data from a Stack Overflow dev survey where they asked about frequency of code commits or check ins.

We can see that nearly 70% of people are checking in at least once a day.

On the surface things look rosy. If we think about my favorite def we have the daily part down.

http://public.brighttalk.com/resource/core/88443/december_16_agile_in_the_enterprise_wholz_131607.pdfWhen did or will your organization begin to practice continuous integration?

@pm_suzie


But if we dig a bit deeper we can see that this is not necessarily the whole story.

This is data from Gartner.

They surveyed their customers - specifically those in Agile Enterprise organisations. These are the organizations that make the majority of software out there in the world.

In those organisations, the one specifically embracing change, who want to move faster and be better, we see only about 40% practicing CI and 40% are most likely never going to do CI.

Thats huge. So this love story is definitely different than the one that many of us have been telling ourselves.

In fact when we dig a little deeper we will see that it is quite different than what we might think.

Contextual research affinity wall

https://blog.snap-ci.com/blog/2016/07/26/continuous-delivery-integration-devops-research/@pm_suzie


The data I am about to share with you are the results my colleague and I learned when we scratched a bit deeper into the CI love story.

We conducted a research study last year using contextual inquiry. Contextual inquiry or research is a methodology that uses a combined interview and observation approach.

We set up calls, video chats, and in-person conversations with software engineers at varying organizations and with varying levels of expertise to understand how they worked, what their big pains were, and the nuances of their days.

What do people know about Continuous Integration?

Do they practice Continuous Integration?

How do they practice Continuous Integration?

What tools do people use to practice Continuous Integration?

How do people define Continuous Integration?



Goals of this work were to answer some questions we had about CI

Primarily we wanted to

* Identify what users know about continuous integration (CI) and continuous delivery practices and principles.Identify pain points around practicing continuous delivery on a daily basis.

First Id like to share some things about that last one - How do people define Continuous Integration?

You commit some code, something picks it up and builds and makes sure it still builds

I push the code to the master branch after it's been tested on the test branch. That in return triggers something and gets deployed

testing your code all the time. Executing the code that you write all the time, making sure the code doesn't go stale



Here are some responses when we asked people to define CI.

You commit some code, something picks it up and builds and makes sure it still builds

I push the code to the master branch after it's been tested on the test branch. That in return triggers something and gets deployed.

testing your code all the time. Executing the code that you write all the time, making sure the code doesn't go stale

As you can see these participants (and they represent the vast majority) were unable to comprehensively define CI in the way that I did at the beginning of the talk. Specifically they didnt mention the two key things that I believe are fundamental to CI - commit to master on a daily basis.

But come no you might say - they werent super precise and a bunch of people were definitely missing some of the key pieces but maybe its ok? Surely even if I hadnt had the textbook quote from Jez I wouldnt have defined it precisely.

And its true definitions mean nothing. Its the doing that is important so regardless of whether they could define CI they might still be doing CI.

Lets look at what they said they were doing in terms of CI.

We tend to do anywhere between 15 and 50 branches in a push

The most challenging thing is when you end up with a lot of features piled together in one commit

Anytime a feature is finally complete then it gets merged into master



Unfortunately this is no better. And in fact many often specifically said they were practicing something that is clearly not continuous integration the way I defined it.

The most challenging thing is when you end up with a lot of features piled together in one commit

Anytime a feature is finally complete then it gets merged into master

We tend to do anywhere between 15 and 50 branches in a push

Of those who said they were using long-lived branches and not regularly checking into a master branch, 90% also said they were practicing continuous integration.

We also heard a lot of we have a CI server and we use x tool when we asked are you practicing CI?

In fact with the exception of just more than 10%, everyone using a CI tool said they were practicing CI. It also really does appears that the use of a CI server or tool is what people mean when they say we are practicing CI rather than specifically practicing the fundamentals of CI as we discussed before.

COMPLICATED

ITS

@pm_suzie


Our love story really has taken a little turn for the worse. And its not quite as rosy as wed think.

We think everyone is doing CI. But lots of people still dont really practice it.

And those that do say they practice it Cant define itProbably arent really doing itYet probably are using a CI tool

CI Theatre

@pm_suzie


My colleagues I call this status CI theatre. Its like playing at being software programmers - its close but not quite.

Common examples of CI theatre include running CI against a shared mainline but with infrequent commits, so integration isnt really continuous, running a build with poor test coverage, allowing the build to stay red for long periods, or running CI against feature branches, leading to continuous isolation. The resulting CI theatre usually makes people feel good, but is not a credible CI practice.

@pm_suzie


So what? You might say. Who cares if its complicated? What if many people practicing CI theatre.

Id like to take this opportunity to convince you why its important to care about CI and doing it correctly.

Not getting the benefits of CI

@pm_suzie


If I am doing CI but not doing CI - I am not getting the benefits we started doing this all for in the first place.

If I am are not checking in, working locally or working on a long lived branch then Im not getting fast feedback, Im not communicating my changes to the team, Im not mitigating risks that Ive misunderstood the architecture, or the requirements. Im potentially wasting my time.

Im definitely not getting the full benefits from CI.

Painful

@pm_suzie


Apart from that Im likely to be experiencing pain that true CI would resolve.

In our study it was very clear that not doing the the master and daily aspects of CI was painful to people. People who said they were doing CI also said they experienced anxiety when merging all their code to master, when they had to merge all their changes at once or explain a huge commit to the team. One participant described the horror of having worked in a branch, changed 100 files and 7000 lines of code, entered a code review and just being overwhelmed trying to explain it all to his colleagues. He said I wish there was a nicer to decompose commits because it's hard to keep everything in your head.

The whole time I was screaming inside there is - its called CI.

Not ready for Continuous Deliveryor DevOps

@pm_suzie


But really this is a heart of why I am mentioning this here. Why I came to this conference to talk about this topic. CI is fundamental for CD and DevOps

CI is important to organizations who want to move more quickly. All these organizations who want to move quickly, be faster, disrupt, innovate, do CD and have a DevOps culture they need to embrace CI. And not just do CI or CI theatre but really understanding it and get back to the fundamentals.

SAVE YOURCI NOW!3 HOT TIPS

@pm_suzie


Ok so lets assume that Ive convinced you that CI is a fundamental practice that takes work and needs to be committed to.

Where do you start to get your relationship with CI back on track?

I have some tips!

Commit more often

@pm_suzie


Back to basics. So if you are not doing CI this where you can get started and even if you are doing CI there is always remove to improve. I love the saying Frequency reduces difficulty.

Why? Because in doing something more often, the task becomes smaller, making it easier to achieve. Continuous integration is a key example of something that becomes easier and easier the more often you do it.

Continuous deployment is a similar example. So you can see how these things build on one another.

So my advice: commit code to your repos more often and integrate branches to master more often.

Automate, automate, automate

@pm_suzie


Automate. This goes along side frequency reduces difficultly. People fear doing something often because it feels like more work so make your life easier and automate it.

Automation is a cornerstone of strong continuous integration so if you you havent automated anything then start. And if you have automated start asking yourself Why cant this be automated? every time anyone on the team does anything manually more than once. If it doesnt help you do CI better it will definitely help you do CD better.

Trunk based development

@pm_suzie


Finally - trunk based development.

This isnt a talk about a trunk based development vs branch based development. But I can definitely tell you when we talked to people who had pain with their CI process it was those that didnt practice trunk based development.

Thats not to say trunk based is not hard. It is. But it brings the pain forward. Faster pain. Better results. I challenge you to consider doing trunk based development if you want to do better CI. If you cant do that please please do short lived branches. No longer than day a day.

dailymaster

@pm_suzie


Because master and daily, remember?

ci forever

@pm_suzie


Thats it. Thats our love story.

Weve taken a look at CI, reviewed what I think is important about the definition, reviewed the current state of the CI and how to get back to the sweet sweet fundamentals.

To be continued

@pm_suzie


So what is the end of our story.

I think how this love story is it up to you!?

Heart BeatZarabadeunullBlues6336.0

continuous integration: a bittersweet love story

Technology