scala bay meetup 9.17.2015 - presentation 1

Download Scala bay meetup 9.17.2015 - Presentation 1

If you can't read please download the document

Post on 11-Jan-2017




0 download

Embed Size (px)


A few tips!

Are we there yet?

Continuous Delivery @ComcastBrendan OBra, Principal Engineer ( @brendanobra LinkedIn:

September 17th, 2015Scala Bay Meetup

Tell a couple of jokes ;) Val is val, and not var

Q & A will be panel style @end1

We are more than just cable TV

Almost everything at National Scale10s of Datacenters across US Traffic can be surge-ishCarrier Grade Customer ExpectationsSoftware Upgrades Millions of DevicesCloud heterogeneity is important1000s of VMs running at any time, across all clouds

Continuous Delivery @Comcast3

-Geo Redundancy/load distribution is important-Special events (Sports, etc.) cause spikes, thundering herds, etc.-Customers REALLY care about Phones and TV ;)-Phased , with varying scope (by market, entire country, A/B, etc)-Subscriber boxes range from 20 years to a few months old.-Dont get locked in Openstack, Vmware, EC2 all are supported equally-VMS represent carry workload


In the past Many humans, some machines MUCH timeContinouos Delivery @Comcast4


Now: One Human, many machines much LESS timeContinouos Delivery @Comcast5


Lets go FASTMarket was clearly changing, with competition delivering some cool stuff.Time to market became a stronger driverWe did some light reading:


And then drank gallons of Continuous Delivery7

We set out to create the most leveraged continuous delivery system we could, using as many off the shelf/ FOSS components that we couldThe goals (which are the same as everyone else)Deliver production ready software , quicklyRepeatabilityNo HumansTake deployment from months to minutesRisk minimization


And then Epiphany!


Immutable Servers!

Someone smart stumbled upon Immutable Servers! Immutable servers are setup/configured once, and never touched again. Immutable Servers then led to everything is 1st class citizen and version worthy. The Holy Trinity is Code, Config and Automation. All are versioned.Comcast was getting into Openstack business about the same time, and we became aware of this cloud thingSNAPSHOTS became less important for everyting beyond 1st phase of CI pipeline it your commit passed tests, it will generate a release version of your artifact, which will be tested further, and possibly releaseWhat if my config changes? How do I change my server? You dont , you make a new version, deploy a new machine with the new version, and kill the old oneFood Not Friends (Cattle Not Pets)


We arrived at a system that consisted of:

9Puppet App ConfigHieraApp AutomationOpenstackVMWareRoute53GitCloud-init

Which can also be viewed like this


Feedback from initial attempts at Continuous Delivery

It needs a GUI

Why is it so hard?

What are all those moving parts there? WTF?!

write the GUI with whatever you want!


So, I did Play seemed like a good choice11


Gumby!Your elastic friend in cloud

V1 : Play + Akka in ScalaWe made an app, in Play! GUI was in Angular, REST layer -> Biz Layer in Play/Akka (one dude want to learn Akka/Play/Scala, other dude wanted to learn AngularJS). Gumby was born!It was a great experiment, it worked in production, and actually solved a real problem(s)


Notes on Bullet #2. We used app to deploy Olympics Live Extras MANY times as application code evolved over the course of the Olympics13

Gumby: Spray + Akka to scale the Comcast CloudThen.. The folks that that make X1 heard about this gumby thing and told us they wanted to use it.. Thats great, but it was only a science experiment.Time to rewrite in Spray, and Go BigSpray allowed up to really clean things up, and because it was so fast, we could DOS any cloud api we encontered ;)


Gumby: Current Architecture (Monolithic)15Workflow (Akka)VSphereOpenstackEC2REST(Spray)Client

Fun Facts about GumbyCan deploy ~400 vms per dc / hourCurrently about 21000 lines of scalaTypically does 2-3 full national deploys/weekDeploys 60% of national X1 backend footprint (4000 VMs)Will deploy 100% of national X1 backend footprint by Q1 2016Is transitioning to microservicesCan deploy itselfDavid Bolene: One Big Side Effect