automation and developer infrastructure — empowering engineers to move from idea to production

Automation and Developer Infrastructure

Empowering Engineers to Move from Idea to Production

Alex ThurlowEngineering Manager

I helppeoplegetjobs.

Indeed is the #1 external source of hire

Unique Visitors (millions)Million unique visitors

2009 2011 2012 2013 2014 20150

20

4060

80

100120140

160180

2010

180M

180 million unique users80.2Munique US visitors per month

16Mjobs

60+countries

28languages

64% of US job searchers search on indeed each month

Alex Thurlow

Plus many more!

What should we build?

What’s best for the jobseeker?

How do we know?

Data-Driven Product Designgo.indeed.com/ddpd

Learn Measure

Build

Lean Startup Cycle

Deliver

Learn Measure

Data driven product design requires quick feedback

Engineering Velocitygo.indeed.com/evtt


Developer Productivit

y

Software Architectur

e

ReleaseProcess

Service-Oriented Architecture

Make small, independent applications

Deliver

Learn Measure

Quickly!

Deliver

Learn Measure


Developer Productivit

y

Software Architectur

e

ReleaseProcess

Jon HanksSoftware Engineer

I helppeoplegetjobs.

HoboAutomated developer provisioning

First goal for a new developerPush code to production in week 1

Deliver

Learn Measure

1:1 Mentorship

Setup = 2-3 Days

1-2 days left to get to production

Team Rotation

Helps new developer understand Indeed

Rotate every 4 weeks through 3 teams

Rotate every 4 weeks through 3 teams and face the same setup challenges in each team.

Frustration!!!

Experienced devs face a similar problem working on multiple projects

or switching projects

Service-Oriented Architecture

Job Database

Search Webapp

Job Service Resume service Location Service

Search Webapp

Job Search Index Resume Database GeoIP Database

Need to run several interconnected applications

Total Developers Producing Code By Month

2014 Apr Jul Oct 2015 Apr Jul Oct 2016

1:1 mentorship didn’t scale

Use technology to speed developer provisioning

Vagrant automates creating virtualized development environments

Job Service Resume service

Location Service

Search Webapp

Job Search Index

Resume Database

GeoIP Database

Vagrantfile

Virtual Machine

Success!(kind of)

Reproducible environments

Our Solution—The Good


Sharable between devs


<1 day setup time

Our Solution—The Bad

5 Only Linux support

4 Limited support for different application types

1 Slow to create environments. 15 minutes or more!

2 Hard initial setup especially for large application stacks

3 Virtual machine memory overhead

Download full machine image

Download and install required software

Start every application

Job Service Resume service

Location Service

Search Webapp

Job Search Index

Resume Database

GeoIP Database

Vagrantfile

Virtual Machine

Job Service

Resume service

Location Service

Employer Webapp

Job Search Index

Resume Database

GeoIP Database

Virtual Machine

Job Service

Resume service

Location Service

Search Webapp

Job Search Index

Resume Database

GeoIP Database

Virtual Machine

Shared RAM

Remaining

MemoryVirtual Machine

We’re all getting Macbooks!Hooray?

Hobo (hō-bō)synonyms: vagrant

Hobo goals

1

2

3

4

5

6

Support all types of applications

Easy to share

Simple for developers to use for development and debugging

Support complex application stacks simply

OS X support

Quick startup

What is Docker?

HypervisorHost OS

GuestOS

Bins & Libs

GuestOS

Bins & LibsVM VM

Server with Virtual Machines

Bins & LibsHost OS

Server with Docker Containers

Dock

er E

ngin

e

Cont

aine

r 1

Cont

aine

r 2

Cont

aine

r 3

Docker terminology

ImageAn OS filesystem

Docker terminology

ContainerA running instance of an image

Docker terminology

RegistryA server for storing images

Hobo goals

1

2

3

4

5

6


Easy to share



OS X support

Quick startup

Our solution

Create and publish a base image for each application type

Java

CentOS

Java

CentOShobo-java

Publish

Docker allows “Inheriting” Images

Java

CentOS

Tomcat

hobo-java

Tomcat hobo-tomcat

Publish

Ready to use Hobo Images

T O M C A T

Hobo goals

1

2

3

4

5

6


Easy to share



OS X support

Quick startup


Search Webapp


Virtual Machine

Docker is designed to have one process per container

Search Webapp



Search Webapp



Employer Webapp

Networking is easy with One machine

Many machines = we need service discovery

Service DiscoveryEnabling services to find other

services automatically

What is Consul?

A key/value store geared towards service discoveryConsul has both an HTTP interface and a DNS interface

PUT /v1/catalog/register{"Node": "user-service", "Address": "10.0.0.3"}

$ dig +short user-service.node.consul10.0.0.3

Now all config files can reference other containers by name!

How do we start a stack of multiple dependent applications?

[HOBO-SETTINGS]depends = user-service search-service location-service

[job-service]type = application

[job-service-mysql]type = resource

[job-service-mongo]type = resource

Hobo Config File

Hobo Config Fileuser-service

[HOBO-SETTINGS]depends =

[user-service]type = application

[user-service-mysql]type = resource

Hobo goals

1

2

3

4

5

6


Easy to share



OS X support

Quick startup

src

test

README

build.xml

hobo

src

test

README

build.xml

hobo

job-service

job-service-mysql

job-service-mongo

hobo_config

job-service

config

Dockerfile

hobo

run.sh

Dockerfile

FROM dockerreg.indeed.net/hobo/hobo-tomcat

job-service

hobo

configtomcat.xmlmysql-conn.propertiesjob-service.properties

job-service

hobo

config

myconfig

job-service.properties

tomcat.xmlmysql-conn.propertiesjob-service.properties

job-service

config

Dockerfile

hobo

run.sh

run.shStart/Stop commands

Environment variablesrun.sh

Volume mountsrun.sh

What are volume mounts?They mount parts of the host filesystem

into the container

What do we mount?

What do we mount?Built code

What do we mount?Configuration files

What do we mount?Log directories

What do we mount?Data directories

Hobo goals

1

2

3

4

5

6


Easy to share



OS X support

Quick startup

git clone repositoryant hobo-deploy

Using Hobo

Results

2014 Apr Jul Oct 2015 Apr Jul Oct 2016 Apr

Active Usage Increase Over Time

Projects Configured For Hobo330

“Sometimes I need to run apps from other teams and the first thing I look

for is ‘Is this hobo’d?’. If so I’m happy!”- Indeed Developer

15 minutes 30 seconds

10 times per week means120 hours per year for each developer

Lessons Learned and Takeaways


Volume mounts are useful


Make it easy to use for the default case but fully customizable


Docker is moving quickly


Developer Productivity

Software Architecture

ReleaseProcess

Control TowerAutomating Release Management

Develop Features

Push Git Branch

Create CodeReview

Mark Issue as Resolved

Take Code Review

Feedback

Creates Pull Request

Local developmen

t

Define Release

Check code review statuses

for issues

Check Git branch merge status

Check with

developer

Check dependent issues of each

feature

Check Issue Statuses

Add to list of features for

release

Determine which library upgrades are available or

needed

Merge desired features to

release

Upgrade library dependencies

Push to origin

Create release branch

Update issue statuses

Create fix version

Create deploy ticket

Add fix version to issues in release

Link other dependent issues to deploy ticket

Test Release

Run unit tests in Jenkins

Send bugs back to

developer

Test release in QA

environment

Build Release in

Jenkins

Deploy to production

Developer fixes bugs

Merge feature

branch to release branch

Jenkins deploys in

QA environment

Are there bugs?

Record Release

Delete release branch

Merge release to integration

branch

Tag Release

Close Issues

Delete feature

branches

Merge release to production

branch

Close Deploy ticket

Release fix version

Yes

Does the feature meet all release

criteria?

Yes

Complex and manual

1

2

3

4

Goals of Our Release Process

Understand which features are being releasedUnderstand cross-product and cross-team dependenciesQuickly address bugs in releasesRecord release details for tracking, analysis, and repeatability


1

2

3

4



4 Record release details for tracking, analysis, and repeatability

3 Quickly address bugs in releases

2 Understand cross-product and cross-team dependencies

1 Understand which features are being released

Develop Features

Push Git Branch

Create CodeReview


Take Code Review

Feedback


Local developmen

t

Define Release


for issues


Check with

developer


feature



release


needed


release


Push to origin



Create fix version




Test Release


Send bugs back to

developer

Test release in QA

environment

Build Release in

Jenkins



Merge feature


Jenkins deploys in

QA environment

Are there bugs?

Record Release



branch

Tag Release

Close Issues

Delete feature

branches


branch

Close Deploy ticket

Release fix version

Yes


criteria?

Yes

The life of a feature

Test Deploy& Record

Develop Feature

Define Release

We use standard development and project management tools

Git: with great power comes great complexity

Git flow

master

master

jira/IND-123

Develop Features

Push Git Branch

Create CodeReview


Take Code Review Feedback


Local development

Develop Feature

=

Feature code can also be in libraries

common-webapp common-database jobservice-api

Search Webapp

Develop Features

Push Git Branch

Create CodeReview


Take Code Review Feedback


Local development

Develop Feature

=

Release Managers Execute Releases

Develop Features

Push Git Branch

Create CodeReview


Take Code Review

Feedback


Local developmen

t

Define Release


for issues


Check with

developer


feature



release


needed


release


Push to origin



Create fix version




Test Release


Send bugs back to

developer

Test release in QA

environment

Build Release in

Jenkins



Merge feature


Jenkins deploys in

QA environment

Are there bugs?

Record Release



branch

Tag Release

Close Issues

Delete feature

branches


branch

Close Deploy ticket

Release fix version

Yes


criteria?

Yes

Develop Features

Push Git Branch

Create CodeReview


Take Code Review

Feedback


Local developmen

t

Define Release


for issues


Check with

developer


feature



release


needed


release


Push to origin



Create fix version




Test Release


Send bugs back to

developer

Test release in QA

environment

Build Release in

Jenkins



Merge feature


Jenkins deploys in

QA environment

Are there bugs?

Yes


criteria?

Yes

Record Release



branch

Tag Release

Close Issues

Delete feature

branches


branch

Close Deploy ticket

Release fix version

master

release

common-webapp common-database jobservice-api

Search Webapp

common-webapp 1.2.6

common-database

3.1.5jobservice-api

1.0.1

Search Webapp

Build Run Tests Deploy to QA


1

2

3

4


What’s in the box?

Look at a dashboard in Jira with everything that’s supposed to be in the release.

Look at Git log to see commits for features

Look at Git log for each new library version

Look at code reviews for each feature to make sure they’re complete


1

2

3

4


Use Jira for Release Tracking

Bug Bug Feature

Fix Version

Bug Bug Feature

Fix Version DeployTicket

Bug Bug Feature

Fix Version

Other App Feature

DeployTicket

Bug Bug Feature

Fix Version

Other App Feature

DeployTicket

Depends upon

Bug Bug Feature

Fix Version

Other App Feature

DeployTicket

Depends upon

Depends upon


1

2

3

4


Develop Features

Push Git Branch

Create CodeReview


Take Code Review

Feedback


Local developmen

t

Define Release


for issues


Check with

developer


feature



release


needed


release


Push to origin



Create fix version




Test Release


Send bugs back to

developer

Test release in QA

environment

Build Release in

Jenkins



Merge feature


Jenkins deploys in

QA environment

Are there bugs?

Yes


criteria?

Yes

Record Release



branch

Tag Release

Close Issues

Delete feature

branches


branch

Close Deploy ticket

Release fix version

Deploy!


1

2

3

4


Develop Features

Push Git Branch

Create CodeReview


Take Code Review

Feedback


Local developmen

t

Define Release


for issues


Check with

developer


feature



release


needed


release


Push to origin



Create fix version




Test Release


Send bugs back to

developer

Test release in QA

environment

Build Release in

Jenkins



Merge feature


Jenkins deploys in

QA environment

Are there bugs?

Yes


criteria?

Yes

Record Release



branch

Tag Release

Close Issues

Delete feature

branches


branch

Close Deploy ticket

Release fix version

master

release

jira/IND-123

master

deploy/2016-06-22

jira/IND-123

jira/IND-123

master

deploy/2016-06-22

master

deploy/2016-06-22

JIRA updatesClose released issues

JIRA updatesClose deploy ticket

JIRA updatesMark Fix Version as “Released”

Develop Features

Push Git Branch

Create CodeReview


Take Code Review

Feedback


Local developmen

t

Define Release


for issues


Check with

developer


feature



release


needed


release


Push to origin



Create fix version




Test Release


Send bugs back to

developer

Test release in QA

environment

Build Release in

Jenkins



Merge feature


Jenkins deploys in

QA environment

Are there bugs?

Yes


criteria?

Yes

Record Release



branch

Tag Release

Close Issues

Delete feature

branches


branch

Close Deploy ticket

Release fix version

Problems

1 Release management took a lot of time

2 It was hard to understand what exactly was in a release

3 Lots of manual steps = possibility for error

4 Only senior engineers knew enough to handle releases

Clicks and Git Commands>100

Release with 4 features

Extra Actions Per Feature~13


Total Developers Producing Code By Month


Unique projects deployed per month

We needed to automate

Why automate rather than simplify?

Data History Transparency

We needed to automate

Control Tower


1

2

3

4


What feature code is in libraries?

common-webapp 1.2.6

common-database

3.1.5jobservice-api

1.0.1

Indeed Webapp


1

2

3

4


Develop Features

Push Git Branch

Create CodeReview


Take Code Review

Feedback


Local developmen

t

Define Release


for issues


Check with

developer


feature



release


needed


release


Push to origin



Create fix version




Test Release


Send bugs back to

developer

Test release in QA

environment

Build Release in

Jenkins



Merge feature


Jenkins deploys in

QA environment

Are there bugs?

Yes


criteria?

Yes

Record Release



branch

Tag Release

Close Issues

Delete feature

branches


branch

Close Deploy ticket

Release fix version


1

2

3

4


Develop Features

Push Git Branch

Create CodeReview


Take Code Review

Feedback


Local developmen

t

Define Release


for issues


Check with

developer


feature



release


needed


release


Push to origin



Create fix version




Test Release


Send bugs back to

developer

Test release in QA

environment

Build Release in

Jenkins



Merge feature


Jenkins deploys in

QA environment

Are there bugs?

Yes


criteria?

Yes

Record Release



branch

Tag Release

Close Issues

Delete feature

branches


branch

Close Deploy ticket

Release fix version

master

release

jira/IND-123

master

release

jira/IND-123

IND-123 commits In release IND-123 commit Not in release

master

release

jira/IND-123


1

2

3

4


Develop Features

Push Git Branch

Create CodeReview


Take Code Review

Feedback


Local developmen

t

Define Release


for issues


Check with

developer


feature



release


needed


release


Push to origin



Create fix version




Test Release


Send bugs back to

developer

Test release in QA

environment

Build Release in

Jenkins



Merge feature


Jenkins deploys in

QA environment

Are there bugs?

Yes


criteria?

Yes

Record Release



branch

Tag Release

Close Issues

Delete feature

branches


branch

Close Deploy ticket

Release fix version

Test DeployDevelop Feature

Define Release

Results

Problems





Clicks and Git Commands>100

Release with 4 features

Control Tower Clicks<15Release with 4 features

Extra Actions Per Feature~13

Extra Actions Per Feature0

Jul Oct 2015 Apr Jul Oct 2016 Apr

Deploys per month

Deploys Per Project1.9X

Problems





More knowledge = Less risk

of developers are release managers57%

To start doing releases~3 months


Everything can be automated


Engineers don’t like mystery


Automate the process you have, not the process you want

Where do we go from here?

Too much human interaction

Train machines to handle releases

Engineering blog & talks - indeed.tech

Open Source - opensource.indeedeng.io

Careers - indeed.jobs

Twitter - @IndeedEng

Learn More

http://indeed.tech/

http://opensource.indeedeng.io/

http://indeed.jobs/

automation and developer infrastructure — empowering engineers to move from idea to production

Technology