scala bay meetup 9.17.2015 - presentation 1
Post on 11-Jan-2017
Embed Size (px)
A few tips!
Are we there yet?
Continuous Delivery @ComcastBrendan OBra, Principal Engineer (firstname.lastname@example.org)Twitter: @brendanobra LinkedIn: https://www.linkedin.com/in/brendanobra
September 17th, 2015Scala Bay Meetup
Tell a couple of jokes ;) Val is val, and not var
Q & A will be panel style @end1
We are more than just cable TV
Almost everything at National Scale10s of Datacenters across US Traffic can be surge-ishCarrier Grade Customer ExpectationsSoftware Upgrades Millions of DevicesCloud heterogeneity is important1000s of VMs running at any time, across all clouds
Continuous Delivery @Comcast3
-Geo Redundancy/load distribution is important-Special events (Sports, etc.) cause spikes, thundering herds, etc.-Customers REALLY care about Phones and TV ;)-Phased , with varying scope (by market, entire country, A/B, etc)-Subscriber boxes range from 20 years to a few months old.-Dont get locked in Openstack, Vmware, EC2 all are supported equally-VMS represent carry workload
In the past Many humans, some machines MUCH timeContinouos Delivery @Comcast4
Now: One Human, many machines much LESS timeContinouos Delivery @Comcast5
Lets go FASTMarket was clearly changing, with competition delivering some cool stuff.Time to market became a stronger driverWe did some light reading:
And then drank gallons of Continuous Delivery7
We set out to create the most leveraged continuous delivery system we could, using as many off the shelf/ FOSS components that we couldThe goals (which are the same as everyone else)Deliver production ready software , quicklyRepeatabilityNo HumansTake deployment from months to minutesRisk minimization
And then Epiphany!
Someone smart stumbled upon Immutable Servers! Immutable servers are setup/configured once, and never touched again. Immutable Servers then led to everything is 1st class citizen and version worthy. The Holy Trinity is Code, Config and Automation. All are versioned.Comcast was getting into Openstack business about the same time, and we became aware of this cloud thingSNAPSHOTS became less important for everyting beyond 1st phase of CI pipeline it your commit passed tests, it will generate a release version of your artifact, which will be tested further, and possibly releaseWhat if my config changes? How do I change my server? You dont , you make a new version, deploy a new machine with the new version, and kill the old oneFood Not Friends (Cattle Not Pets)
We arrived at a system that consisted of:
9Puppet App ConfigHieraApp AutomationOpenstackVMWareRoute53GitCloud-init
Which can also be viewed like this
Feedback from initial attempts at Continuous Delivery
It needs a GUI
Why is it so hard?
What are all those moving parts there? WTF?!
write the GUI with whatever you want!
So, I did Play seemed like a good choice11
Gumby!Your elastic friend in cloud
V1 : Play + Akka in ScalaWe made an app, in Play! GUI was in Angular, REST layer -> Biz Layer in Play/Akka (one dude want to learn Akka/Play/Scala, other dude wanted to learn AngularJS). Gumby was born!It was a great experiment, it worked in production, and actually solved a real problem(s)
Notes on Bullet #2. We used app to deploy Olympics Live Extras MANY times as application code evolved over the course of the Olympics13
Gumby: Spray + Akka to scale the Comcast CloudThen.. The folks that that make X1 heard about this gumby thing and told us they wanted to use it.. Thats great, but it was only a science experiment.Time to rewrite in Spray, and Go BigSpray allowed up to really clean things up, and because it was so fast, we could DOS any cloud api we encontered ;)
Gumby: Current Architecture (Monolithic)15Workflow (Akka)VSphereOpenstackEC2REST(Spray)Client
Fun Facts about GumbyCan deploy ~400 vms per dc / hourCurrently about 21000 lines of scalaTypically does 2-3 full national deploys/weekDeploys 60% of national X1 backend footprint (4000 VMs)Will deploy 100% of national X1 backend footprint by Q1 2016Is transitioning to microservicesCan deploy itselfDavid Bolene: One Big Side Effect