empowering developers to take flight - …€¦ · jenkins pipeline job ... ... empowering...

Empowering Developers to Take FlightContinuous Integration at Gogo

June 26, 2017 • Room #304

Darin Pantley - Gogo

Introduction

The Problem

The Dream

Implementation

Lessons Learned

Gogo is the inflight internet company

At a glance:

• 17 commercial airline partners• 3,000+ commercial aircraft in-service• 1,500+ 2Ku backlog• 120M+ connectivity sessions to date• 85,000+ sessions/day• 8,000+ flights/day• Gogo equipped flight every 11 seconds• 1,300+ employees

About Gogo

Introduction

Gogo’s open architecture is designed to power passenger experience like no other in our industry

Flexiblepassengerexperience Entertainment

Entertainment/Messaging

PassengerConnectivity

ConnectedAircraftServices

Gogo Products and Services

Introduction

June 27, 2017 • Chicago Coder Conference 5

Introduction

Infrastructure Requirements§ Hardware:

§ airplanes, devices, WAPs, satellites, servers, etc.

§ Software:§ In the air, on the ground§ Local, off-site, outsourced

Introduction

The Problem

June 27, 2017 • Chicago Coder Conference 9Disciconsfromwww.flaticon.comComputer iconmadebyPanierAvide

The Problem

Software Engineering: Before Improvements1. Drink coffee2. Write some code3. Compile your changes4. Run some “unit tests” (optional)5. Commit your changes6. Create a deployable artifact

June 27, 2017 10

The Problem

Software Engineering: Before Improvements7. Copy it onto a USB stick8. Use the elevator to get to the lab9. Prepare the lab machine

§ obtain a monitor, plug in cables10. Plug in your USB stick

June 27, 2017 11

The Problem

Software Engineering: Before Improvements11. Boot it up!12. Hold your breath13. Fix the environment so it looks pretty clean14. Start manually testing your changes

June 27, 2017 12

The Problem

1. Drink coffee – better coffee machines2. Write some code – send developers to conferences3. Compile your changes – faster laptops4. Run some “unit tests” (optional) – not optional5. Commit your changes – do this early in the day6. Create a deployable artifact – lunch time

Software Engineering: After Improvements1. Drink coffee2. Write some code3. Compile your changes4. Run some “unit tests” (optional)5. Commit your changes6. Create a deployable artifact

June 27, 2017 13

The Problem

7. Copy it onto a USB stick – USB hub8. Use the elevator to get to the lab – work from the lab9. Prepare the lab machine – guard it with your life

10. Plug in your USB stick – SCP onto a running system

Software Engineering: After Improvements7. Copy it onto a USB stick8. Use the elevator to get to the lab9. Prepare the lab machine

§ obtain a monitor, plug in cables10. Plug in your USB stick

June 27, 2017 14

The Problem

11. Boot it up! – keep it running12. Hold your breath – breathe deeply13. Fix the environment so it looks pretty clean – scripts14. Start manually testing your changes – hiring time

Software Engineering: After Improvements11. Boot it up!12. Hold your breath13. Fix the environment so it looks pretty clean14. Start manually testing your changes

June 27, 2017 15

The Problem

What’s Wrong?§ It takes too long to release new features§ Many bugs aren’t caught during code reviews§ It’s difficult to obtain a test environment§ Everything is a manual process§ Running automated tests is optional§ We discover bugs too late§ All of this is expensive


The Problem

June 27, 2017 17

Our Infrastructure• AWS• GitLab• Jenkins• Artifactory• Docker• Ansible• Vagrant• Redis• …

The Problem

The Dream

Replace Legacy Processes with Best Practices§ Distributed Version Control System§ Agile Development§ Robust Infrastructure

§ Immutable and scalable§ Monitoring, metrics, and alerts§ Version controlled

§ Continuous Integration


The Dream

June 27, 2017 20

The Dream

Continuous Integration§ Automatically triggered by merge requests§ Automated Building

§ Run static analysis§ Compile the code§ Run automated tests in the build environment

§ Automated Packaging§ Create a deliverable

§ Automated Deployments§ Run automated tests in the deployed environment

§ If everything worked, let the merge proceed

June 27, 2017 21

The Dream

Reality: Maybe we can do this later?


1. Our code is in SVN

2. We’ve never heard ofcoupling and cohesion

3. Our tests aren’t automated

4. Everyone is busy

The Dream

Reality: Maybe we can do this later?5. We just ordered more

lab equipment

6. Feature development cannot be interrupted

7. Our code must run on physical hardware


The Dream

We can make improvements1. Our code is in SVN

§ Hard cut-over to Git2. We’ve never heard of coupling and cohesion

§ Componentize what you can now, the rest can wait3. Our tests aren’t automated

§ Provide a way to run them, regardless of whether they exist4. Everyone is busy

§ Prioritize continuous integration


The Dream

We can make improvements5. We just ordered more lab equipment

§ Some tests will still require real hardware6. Feature development cannot be interrupted

§ Avoid major changes near releases7. Our code must run on physical hardware

§ Virtualize the hardware environment


The Dream

Sell the Dream§ Diagram out a continuous integration workflow


The Dream

Sell the Dream§ Diagram out a continuous integration workflow§ Present your idea to developers and upper management§ Postpone solution space discussions§ Agree on the goals§ Agree on responsibilities


Engineering|DevOps

The Dream

Implementation

Overview from a DevOps perspective

1. JobDSL ↴2. Jenkins Pipeline Job ↴3. Jenkins Pipeline Library

§ → pre-defined “targets”§ → DevOps tools


Implementation

• runway/• static-analysis• build• unit-tests• integration-tests

• dsl.groovy• version

• smoke-tests• fast-functional-tests• slow-functional-tests• end-to-end-system-tests

Overview from a developer perspective


Implementation

• ci• nightly

• iteration• release

Pipeline Types


Implementation

GitLab Merge Requests


Implementation

Jenkins Pipeline


Implementation

runway / build#!/bin/bash -exexport SHORT_SHA=$(git rev-parse --short HEAD)export VERSION="$(cat runway/version).${BUILD_NUMBER}~$SHORT_SHA"export FULL_SHA=$(git rev-parse HEAD)export GIT_COMMIT="$FULL_SHA $(git log -1 --pretty=%B | head -1)"

docker login gogo-docker.jfrog.io -u builder -p hunter2docker run --rm -w /build -v $(pwd):/build/ gogo-docker.jfrog.io/

buildmachine:2.3.0.19 ./build.sh /build/runway/FS_ROOT $VERSION$GIT_COMMIT


https://www.docker.com

Implementation

runway / dsl.groovyimport utilities.ContinuousIntegrationimport utilities.NightlyBuild

REPO = '[email protected]:acpu/hal.git'

def ci = new ContinuousIntegration(job: pipelineJob('${SRC_JOB}_ci'))ci.createJenkinsPipeline()

def nightly = new NightlyBuild(job: pipelineJob('${SRC_JOB}_nightly'))nightly.createJenkinsPipeline(REPO)


https://github.com/jenkinsci/job-dsl-plugin

Implementation

ContinuousIntegration.groovythis.job.with {

description("Continuous Integration Pipeline")…

url('[email protected]:common/jenkins-pipeline.git')

…scriptPath('ContinuousIntegration.Jenkinsfile')

…}


https://github.com/jenkinsci/job-dsl-plugin

Implementation

Jenkins Pipeline Shared Libraries(root)+- src # Groovy source files | +- org | +- foo | +- Bar.groovy # for org.foo.Bar class+- vars | +- foo.groovy # for global 'foo' variable| +- foo.txt # help for 'foo' variable +- resources # resource files (external libraries only)| +- org | +- foo | +- bar.json # static helper data for org.foo.Bar


https://jenkins.io/doc/book/pipeline/shared-libraries/

Implementation

ContinuousIntegration.Jenkinsfilenode('ci') {

withEnv(["PIPELINE=ci"]) {… staticAnalysis()

buildComponent() unitTests() uploadComponent()

…


https://jenkins.io/doc/pipeline/

Implementation

vars / buildComponent.groovy...dir(COMPONENT_DIR) {

def target = 'runway/build'if (fileExists(target)) {

sh target}

}...


Implementation

• Artist• Written in Python• CLI tool

• Artifactory

Organizing the code


• Graffiti• Written in Python• CLI tool

• GitLab• Hosted Graphite

Implementation

Lessons Learned

Working with Developers§ Meet frequently with everyone involved§ Deliver features incrementally§ Spread access to tooling

§ Extend internal solutions to cover developer use cases§ Document answers to questions§ Agree on solutions to technical debt in existing code


Lessons Learned

Working without Developers§ Create alternatives to technical debt in existing code§ Create proof of concept projects

§ Choose between competing technologies§ Demo solutions to controversial problems§ Prove that something is possible

§ Working out-of-band allows constant refactoring


Lessons Learned

Hindsight is 20/20§ Explicitly define requirements up-front

§ General functionality, documentation, metrics, alerts, …§ Plan your current features with the next features in mind§ Find users quickly§ Eat your own dog food§ Feed the hungry§ Avoid bus errors


Lessons Learned

Overheard on Slack§ “I can show you how to use Artist to upload. It’s easy.”

§ “Should I merge? The button is red.”


Lessons Learned

June 27, 2017 47

Empowering Developers to Take FlightContinuous Integration at Gogo

Darin [email protected]

3:45pm – 4:45pm • Room #304

Thank You

empowering developers to take flight - …€¦ · jenkins pipeline job ... ... empowering...

Documents