Auto-scaled Concourse CIw/ AWS w/o BOSH

Yusuke KUOKAChatWork (http://www.chatwork.com/)

@mumoshu

First of all:

Concourse CI…

Thanks a lot for sharing us such a GREAT Open-

Source Pipeline-based CI!

Finally we have a CI supporting pipelines as

first-class citizens!!

That’s all. Good-bye!

…

The issue

Is Concourse a production-ready pipeline-based CI

for a startup/small team?

Production readiness for CI

Something that:

* Just works™

* Costs as low as possible

* Is maintainable

Just works!* `vagrant up concourse/lite` to bootstrap

* `vagrant box update && vagrant destroy -f && vagrant up` to update

* Deployment w/ standalone binaries just works. The deployment will be more manual than using BOSH

(Note: It works as far as you use Ubuntu trusty or later with updated kernel as described in Concourse’s and Docker’s official documentation)

* A lot of builtin/community resources_types

* Concourse CI’s concourse pipeline is a greatest example

Costs as low as possible?

* Wasting money for running idling CI workers when e.g. you and your colleagues drinking beer is unacceptable

* Scale out my CI workers only when someone in your team is working.

* Scalable/Elastic Concourse

* Can be done on my side

Maintainability for a small team?

* I am the only one available to introduce Concourse to my organization

* No one (including me) in my org has experiences in CloudFoundry, BOSH

* BOSH/CloudFoundry/Pivotal all looks GREAT

* Have no time learning it (I wish I got paid to learn it)

* Need Concourse today, but it comes with BOSH

Recap: Issues? > Last one-mile to production-readiness for us = Low cost and BOSH

How?

Production-ready pipeline-based CI for me=

AWS + Auto-scaling + Concourse=

concourse-aws

Usage

1. Build AMI (Based on Ubuntu trusty, Docker/Concourse baked)

2. Provision Concourse cluster providing configuration params interactively

git clone https://github.com/mumoshu/concourse-aws

cd concourse-aws

curl -L https://github.com/mumoshu/concourse-aws/releases/download/latest/concourse-aws -o concourse-aws && chmod +x concourse-aws && ./concourse-aws up

./build-amis.sh

./concourse-aws up

Features

Features

* (Almost) one-command to bootstrap Concourse CI on AWS

* Supports both time-based and utilization-based auto-scaling (e.g. zero workers in after-hours, 80+ CPU usage for several minutes to trigger a scale-out)

* Multi-AZ (Single AZ failure doesn’t bring down the whole cluster. Running jobs will fail though)

Implementation

Implementation

* Terraform for provisioning AWS resources

* Some bash scripts to wrap terraform calls

* Some golang apps to interactively generate configuration files and passes it to terraform

TODO

TODO

* CoreOS support(for faster, easier deployment, customization, maintainability, and for fun)?

* Automatic VPC creation?(Do you need a separate VPC for Concourse?)

* Separate terraform template for DB(Postgres)?

* CloudFormation?

Wishes

I wish Concourse had…

Any official(or documented) way to:

* gracefully stop workers when an EC2 instance is shutting-down or ASG is going to stop it

* Currently we have to utilize AutoScaling Lifecycle Hooks + SQS + `fly pause-pipeline <EVERY PIPELINE>` + `fly builds | grep started | wc -l` to gracefully stop Concourse without killing running jobs

I wish Concourse had…

Any official(or documented) way to:

* cache pulled docker images to reduce build times immediately after scaling out

* Docker’s “local registry as a pull-through cache” is good for DockerHub. Can’t be used with AWS ECR or Quay.io though

Summing up…

Use mumoshu/concourse-aws to bootstrap a production-ready Concourse cluster w/o learning

BOSH

You can try it today!

Thanks!

https://github.com/mumoshu/concourse-aws

Yusuke KUOKA / @mumoshu

An Infrastructure Engineer @ ChatWork

We’re hiring!