PowerPoint Presentation

A Road to Continuous Delivery: Evolution, not RevolutionKsenia Burlachenko Engineer, Perforce Software

* with frequent appearances from various cats

1

Perforce workflow followed the business model

Helix server historically has a single source of truth for us and many other companies so as a result our development/test/build workflow followed the existing model at the time which can be summarized in 3 words:2

SLOW AND STEADY

SLOW AND STEADY3

DEVELOPMENT

Slow and steady development with compiled languages like C++4

TEST

Slow and steady test in a standard waterfall model where QA test changes after development implemented them5

BUILD* and let other people do it

Slow and steady build, where product teams are completely isolated from Ops, and a separate centralized team which delivers the product for you.6

The world has changed with Git and Web development

The world has changed - Enter to the world of Git and web development.Git and focus on web development, not software, meant that production workflow has changed developers make fast incremental changes to the code, and they share their projects peer to peer with Git7

Perforces business model has changed

So, as a result, Perforces business model has changed: we needed to support both workflows: the fast and collaborative and slow and steady8

FAST and COLLABORATIVE+SLOW and STEADY=

we needed to support both worlds: the fast and collaborative and slow and steady9

GIT FUSION

Enter Git Fusion: the bridge for our customers between the two words10

GitGitFusionp4dPerforce ServiceGit FusionServiceUser Workstationgit pushp4 commandsgit clonegit fetchgit pull

#

Git Fusion acts as a Perforce backend for Git users: it's a client to the server it's a server to the end user client11

Git Fusion changed our workflow: we can be continuous

12

EVOLUTION, NOT REVOLUTION

The process of adopting Continuous delivery pipeline was an evolution, not revolution because we had to get it done incrementaly over time13

Git Fusion is not SAAS

Customers install Git Fusion on their own platforms, in infrastructure that works for them14

#

We deliver artifacts to customers.

Artifact is simply what we call deliverables for our customers. There are 4 main kinds of artifacts in Git Fusion Linux Packages, Source Archive, OVA (which is virtual machine with Git Fusion installed and pre-configured), and Documentation15

Delivering artifacts poses unique challenges

* The fact that we deliver artifacts poses many challenges:16

Linux multi platform support

#

Think of our platform support as of support of multi browsers. We have a core set of Linux platforms, but there are some significant differences between them. CentOS package manager yum rarely behaves the same way as Ubuntus apt-get. Not all dependencies are readily available upstream17

Cant force upgrades:must support smooth upgrades even from very old versions

#

Of course, we also Can't force upgrades since customers decide for themselves when they upgrade18

Complex configuration performed by customers* creating system user, creating Perforce users, setting up Apache, installing triggers, etc, etc, etc

#

Configuration has been always a major pain point for our customers. With CD, we were hoping to tackle that problem and alleviate some of that pain..19

Supporting different architectures* Edge/Commit services, Windows and Linux case-insensitive services, Replicas

#

Since Git Fusion is a client to Perforce service, it means we have to support a variety of different architectures, each one with different set of problems20

Make some decisions

Where do we start? First, we had to make some decisions21

What is the size of your team?

The size of the team matters the smaller it is the easier it is to keep the main stable. Git Fusion team historically was around 6 people, so it is If your team is large, consider implementing pre-commit review model in either Swarm or GitSwarm.22

Which branch model should you use?

Git Fusion team uses mainline (master) -> candidate model. One of the rules that we adopted is that main is always stable, which means that every developer is responsible for making sure his commit is good enough to pass through our CD pipeline. In case build fails, the developers have to fix the build right away or revert their change in case it is not a simple fix.

Maincontains changes that are in progress and may not necessarily be ready for production.

Candidaterepresents the most stable version of your project and you only ever deploy to production from this branch.

If you have a team of a larger size, consider using feature branches. Some say feature branches are evil, but it is not necessary true if your developers continuously merge from main, which can be not as painful with awesome tools like P4Merge.

23

Which artifacts are the most important?YOU:CUSTOMER:

VS.

The goal is to identify something you work on something that is easy + valuable24

Get to work

Now that we established the foundation, we got to work25

Build your artifacts

For building your artifacts you want to make sure that you have a clear picture of the structure of your product and how one component might depend on another. - Consider your requirements, major components and the relationship between the components

26

#

Git Fusion release notes have to be included in every artifact, so they can be the first component in our Continuous Delivery pipeline. Moreover, source archive is an artifact of the build and it also can be a component of packages, and Ubuntu 12.04 packages are used to build our Virtual machine appliance27

Test your artifacts

The goal is to identify something you work on something that is easy + valuable Fast Exact customer environment Repeatable Lightweight

28

#

Some of the requirements for test are: You want them to be Fast They have to be lightweight They must allow for multiple configuration scenarious to replicate your customer environment And of course they have to be repeatable, and reliable

For us, surprisingly enough the answer was a homegrown test infrastructure build on top of Docker. Docker is usually thought of as a solution to micro-services architecture, which Git Fusion is not at all. However, it fit all of our requirements and we are extremely happy with Docker as a part of a solution to artifact testing.29

Make your pipeline efficient

Make your pipeline efficient As I mentioned before, it is possible to identify the relationship between the components of your Continuous Delivery pipeline.When thinking of this relationship, consider if you still want to deliver your product if there is a failure in build or test of one of the components

30

#

* For Git Fusion, we only deliver artifacts to customers when they have passed the required tests - In this diagram you can see the relationship between every component of the build pipeline. For example a single git push or p4 submit to the Git Fusion source causes the build of our release notes, a successful build of which triggers unit test, a success of which starts build of source archive, which gets tested and published and triggers package builds which get tested in return... - It is a complex system but it allows us to be very efficient and very confident in the quality of the product we deliver31

Add more tests!

Installation is a major pain point, or more precisely it was a pain point, for Git Fusion customers.Test upgrades, different configuration environments, test different deployment scenarios for your artifacts. Of course it is not practical to have all of your tests to run on every git push/p4 submit, so get in the habit of scheduling them some should run every other hour, while others just need to be run once a day.32

Before release, stabilize

Even with the continuous delivery pipeline in place, we still very much benefit from a couple of days of stabilization. Let all of your tests pasts, try out your changes There probably still will be some last minute changes going in, but keep it to the minimum33

Results

So, finally here is what we were able to achieve with our continuous delivery pipeline34

Release time2 weeks1-2 daysBEFORENOW

Release used to take 2 weeks, and now it is down just to 1 or 2 days.35

1 person behind the release

In the old days, a team on 3 QA people, 2 build engineers and 1 project manager were driving the release. Now it is all done just by 1 person.36

Customers can try out fixes and features as they come in

Customers can try out fixes and features as they come in In fact a couple of days before the release, the code in SNAPSHOT builds is *extremely* similar to what customer gets as a release37

Deliver fixes and features quickly and with high confidence

Because of all the work that went into the continuous delivery pipeline, you will be able to push out releases much more quickly, and with high confidence in your product38

Thank you!

@ksenish

39