drupalgov2014 heartbleed
TRANSCRIPT
Tim HilliardDrupalGov CanberraAugust 22nd 2014
at
This session
1. Hearbleed case study2. Q & A with:
• Tim Hilliard (Cloud Eng)• Adam Malone (Support)• Chris O’Neill (Support)• Phil Ingrim (Ops)
About Me
Cloud Platform Engineer
@big_bear84 or timhilliard on github, d.o., etc.
Act 1: Technology
How it all started8 April 9:30 AM AEST
ZZZzzz….!
Me == Sleeping
How it all started8 April 9:26 AM AEST
How it all started8 April 9:33 AM AEST
How it all started
Risk assessment
Lucid:[00:35:27] [email protected]:~# openssl versionOpenSSL 0.9.8k 25 Mar 2009!Precise:[00:34:37] [email protected]:~# openssl versionOpenSSL 1.0.1 14 Mar 2012
Where’s Wally OpenSSL8000 EC2 Machines:- 99.9% of them puppetized- Candidates:
- Balancers- SVN Servers- Appliances
- ELBs- 3rd party AMIs
- Unique little snowflakes(Jira, Crucible,…)
Stack
Web tier
Other services DB, shared filesystem, memcache
BalancersVarnish
Port 80 Port 443Nginx
ELBsInternet
Here!
Stack
Web tier
Other services DB, shared filesystem, memcache
BalancersVarnish
Port 80 Port 443Nginx
ELBsInternet
Here!
and here!
Support
11:52:32 Adam Malone: hi QQ opes, I here ther is a heartbloom security issue with ssh. Is this being treated with high urgencies (p1) we need to escalade this if possible
Let the patching begin
RolloutAustralia:!Con:- Spiders- Snakes
!Pro:- Ops is awake
Rollout
RolloutWe did not fail over EIPs to passive balancers when
upgrading Nginx. !
Failing over an EIP leaves the IP disassociated for up to about 3 minutes. Upgrading Nginx in place takes as long as it takes
to restart Nginx. So a matter of seconds. !
Linux package management ++
Rollout
• Rollout across the entire platform took ~4 hours • ~800 balancers to upgrade.
Waiting on ELBs…
Internal Certificates
Suddenly:“reverse” Heartbleed
Act 2: Communication
Internal
• Pre-determined chat rooms
• Dial-in conference bridges
• A communication plan
Thanks SSAE-16, PCI and FedRAMP… I guess :)
Statuspage + Twitter
* Powered by StatusPage.io
Documentationhttps://docs.acquia.com/articles/heartbleed-acquia-cloud
Proactive communication
Phone calls by Acquia support, TAMs, …
Since then:Post mortem
Since then:Incident Commander
(shamelessly stolen from Heroku)http://en.wikipedia.org/wiki/Incident_command_system
Since then:Dedicated resource to vet security threats
Since then:Clean up intranet docs
Since then:Additional tooling
Q & A
• Tim Hilliard (@big_bear84) (d.o./u/timhilliard) • Phil Ingrim (he’s in ops so doesn’t have ANY social media) • Chris O’Neill (@cjoneill) • Adam Malone (@adammalone) (d.o/u/typhonius)