devops at obama for america(2012) and the dnc (devops days nyc jan 2013)
DESCRIPTION
Leo Zhandovsky (@leozh, formerly DNC, now AWS Solutions Architect) and I gave a brief talk about DevOps at the DNC and OFA campaign for DevOps Days in New York City. Video (Starts ~1hr55min) - http://new.livestream.com/devopsdaysorg/nyc2013TRANSCRIPT
![Page 1: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/1.jpg)
DevOps @ DNC & OFADevOps @ DNC & OFA
![Page 2: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/2.jpg)
Leo Zhadanovsky@leozhDirector of Systems Engineering @Democratic National Committee
John Schneider@jdotpFormerly Obama for AmericaCurrently DevOps / Internet Jedi @ Mozilla Foundation
![Page 3: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/3.jpg)
DevOps @ DNC - 2009 - 2011
Year 1● Cleanup● Virtualization● Documentation● Version Control● Making everything HA● Replacing all our network gear● Automation
![Page 4: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/4.jpg)
Labs Development Team
● Created first versions of Call Tool, Voter Registration Site, Canvassing iPhone App
![Page 5: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/5.jpg)
Labs Development Team
Start of DevOps Practices at the DNC● Agile Development● Usage of EC2● Continuous Integration● Deploys using Capistrano● Scaling up using custom AMIs● Monitoring using Nagios, Cacti, NewRelic,
Graphite, Pingdom, StatsD
![Page 6: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/6.jpg)
![Page 7: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/7.jpg)
Analytics Team
● Lessons learned from 2010 used in 2012● Big Data● Hardware vs AWS● Limitations of AWS at the time● What would we do differently now?● Developed software for data analysts,
modelers and field staff
![Page 8: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/8.jpg)
Cloud (AWS)
● Unpredictable traffic patterns: No problem! (We had no clue those birther mugs would sell so well.)
● Makes entire org far more agile
● Apples to apples testing envs are a snap
● Cloud rules changes the way we manage systems
![Page 9: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/9.jpg)
Dev + Ops = <3The New Mentality of Devops
SOA / 12 Factor
● Allow for easy migration, deployment● All components independently scaled
without retooling or re-architecting● Critical for building scalable, high-availability
apps in the cloud.● Goes well with Route53.
![Page 10: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/10.jpg)
![Page 11: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/11.jpg)
Dev + Ops = <3The New Mentality of Devops
Agile and Continuous● Tool agnostic (we used Pivatol, Basecamp
Classic, Basecamp Next, spreadsheets, github issues, zendesk tickets, etc)
● Small and fast releases. (we pushed ~40 new versions per day in the last weekend)
● SHIPIT! Build 'good enough' tools and iterate.
![Page 12: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/12.jpg)
![Page 13: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/13.jpg)
Dev + Ops = <3The New Mentality of Devops
Instrument Everything (and trust the data)
● CPU, traffic, response time, etc
● User/System events
● Deployment-related change metrics
● Logs are data too!
![Page 14: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/14.jpg)
Dev + Ops = <3The New Mentality of Devops
Visible Ops (aka, how to turn PMs, Devs, Management into Ops partners)
● They all get Nagios, NewRelic, and Pagerduty accounts and training
● Activity feed of commits, deploys, alerts, changes.
● Cross-pollinate team stands, retros● Give everyone contextual data
![Page 15: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/15.jpg)
Who doesn't love color charts?
![Page 16: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/16.jpg)
![Page 17: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/17.jpg)
![Page 18: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/18.jpg)
![Page 19: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/19.jpg)
Load Testing with NewRelic Deploys
![Page 20: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/20.jpg)
Dev + Ops = <3The New Mentality of Devops
Fail Early, Fail Fast, Fail Well
● It is not if, but when. ● Learn how to detect & mitigate, and then
iterate.● Reduce the time to detect, the time to
resolve. ● Preventing all failure is not the DevOps role,
making failure hurt less or not hurt at all is DevOps role
![Page 21: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/21.jpg)
Yo, mind if we bring the donations db down during the VP debate?
![Page 22: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/22.jpg)
Oh Hai East Virginia Datacenter!
![Page 23: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/23.jpg)
Other AWS Tips-Use EBS-backed instances, AMI-based deploys, and provisioned IOPS. -Use SQS. -Multi-AZ, Multi-Region early-Autoscale from any metric, push your own metrics to Cloudwatch even-Setup ifttt or zapier alerts on AWS status pages-Check out billing broken out by tag, and TrustedAdvisor
![Page 24: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/24.jpg)
S3 microsites are awesome
![Page 25: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/25.jpg)
Other Things That Helped
New Relic (We love you New Relic)Asgard (Thanks Netflix)StatsD/Graphite and CubismLogglyBeesWithMachineGunsAkamai DSA and GTMBotoPalominoDB is awesome!Lots of caffeineLastly....culture, culture, culture.
![Page 26: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/26.jpg)
![Page 27: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/27.jpg)
![Page 28: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/28.jpg)
![Page 29: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/29.jpg)
Election Day
![Page 30: DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)](https://reader033.vdocuments.mx/reader033/viewer/2022042613/545385faaf795904308b53dc/html5/thumbnails/30.jpg)