coolblue - behind the scenes continuous integration & deployment
TRANSCRIPT
July 2016 | Behind the Scenes
Agenda.
● Welcome & introduction● From Code to Node● Short Break● A look to the future of deployment● Tour of our office● Drinks● Fin
July 2016 | Behind the Scenes Flynn Bundy <@bundyfx> & Matthew Hodgkins <@matthodge>
“I was trying to sleep last night but all I could think about was how to make our pipelines better.”
Matt
PLAN.
CODE.[sensu]Get(){...}
[void]Set(){...}
[boolean]Test(){...}
static [void] XMLConversion (){...}
static [void] InstallService (){...}
static [void] ConfigureAgentTransport([Transport]$Transport){...}
CODE.node $AllNodes.Where{$_.Role -eq "our_app"}.role{...}
node $AllNodes.Where{$_.Role -eq "our_app_1"}.role{...}
node $AllNodes.Where{$_.Role -eq "our_app_2"}.role{...}
node $AllNodes.Where{$_.Role -eq "our_app_3"}.role{...}
node $AllNodes.Where{$_.Role -eq "our_app_4"}.role{...}
BUILD.
BUILD.
TEST. Context Download Agent Method [+] Should download the Sensu client 7.58s [+] Should be the full file 101ms Context Install Agent Method [+] Should Install the Sensu client 38.49s Context ConfigureAgentClient Method [+] Should be correctly formatted into JSON 1.77s [+] Should contain the Correct Information in the JSON | Subscriptions 20ms [+] Should contain the Correct Information in the JSON | Name 19ms [+] Should contain the Correct Information in the JSON | Private IPv4 Address 15ms Context ConfigureAgentTransport Method [+] Should be formatted correctly into JSON 154ms [+] Should contain the Correct Information in the JSON | Name 16ms [+] Should contain the Correct Information in the JSON | Reconnect on Error 13ms Context InstallService Method [+] Should deploy and install the Sensu client service 256ms Context Conversion XML Method [+] Should correctly create an valid XML configuration file 119ms Context RemoveAgent Method [+] It should remove the agent 239.16sDescribing DSC Based methods Context Testing DSC based methods [+] Test() Method should return a Boolean Type 283ms [+] Get() Method should return a Sensu Type 31ms
TEST.
Move fast and break things. Unless you are breaking stuff, you are not moving fast enough.
Mark Zuckerberg
DEPLOY.
DEPLOY.
WHAT IS DSC?.
WHAT IS DSC?.
WHAT IS DSC?.
OPERATE.
PILLARS OF MONITORING.
WHY MONITORING IS HARD.
▪Automation is an after thought▪Forgetting to add new services▪Checks are hard to write▪Don’t want to do annoying things
ENTER SENSU.
IIS
/purchases
Sensu Agent
Sensu Server
RabbitMQ
STEP 1 - DEPLOY AGENT WITH DSC.Sensu Install{ Ensure = "Present" DownloadURL = 'https://sensu.global.net/msi/sensu-0.24.0-1.msi' RabbitMQServer = 'coolblue-rabbitmq.internal.fake' Transport = 'rabbitmq' vHost = 'sensu' User = 'sensu' Port = '5672' Password = 'some_password' #Encrypted of course Subscription = 'windows_base'}
IIS
/purchases
Sensu Agent
Sensu Server
RabbitMQ Deploy Agent
STEP 2 - CREATE SOME CHECKS.{ "checks": { "disk_usage": { "command": "check-windows-disk.rb.bat -w 85 -c 95", "interval": 600, "subscribers": [ "windows" ] }, "windows_update": { "command": "powershell.exe -file C:/coolblue_checks/win_last_updated.ps1 -w 30 -c 60", "interval": 86400, "subscribers": [ "windows" ] } }}
SENSU CHECK DEFINITIONS.
IIS
/purchases
Sensu Agent
Sensu Server
RabbitMQ
Add Check Definitions
STEP 3 - UPDATE THE CONFIG.Sensu Install{ Ensure = "Present" DownloadURL = 'https://sensu.global.net/msi/sensu-0.24.0-1.msi' RabbitMQServer = 'coolblue-rabbitmq.internal.fake' Transport = 'rabbitmq' vHost = 'sensu' User = 'sensu' Port = '5672' Password = 'some_password' #Encrypted of course Subscription = 'windows_base','my_iis_app'}
STEP 4 - ADD APP SPECIFIC CHECKS.{ "checks": { "iis_purchases": { "command": "check-http.rb http://localhost/purchases", "interval": 60, "playbook": "https://github.com/devTeam/readme.md", "subscribers": [ "my_iis_app" ] } }}
WHY MONITORING IS HARD.
▪Automation is an after thought▪Forgetting to add new services▪Checks are hard to write▪Don’t want to do annoying things
THE SENSU SOCKET INPUT.
IIS
/purchases
Sensu Agent
Sensu Server
RabbitMQ
3030TCP/UDP
My App
JUST SEND IN JSON.
{ "name": "ravendb_connectivity", "output": "Could not connect to RavenDB!", "status": 2, "team": "TeamName", "application": "My App", "playbook": "https://github.com/devTeam/readme.md"}
USING A DEAD-MAN SWITCH.
{ "name": "daily_sql_backup", "output": "Backup of SQL databases performed", "status": 0, "team": "OpsTeamName", "application": "sql_backups", "playbook": "https://github.com/opsTeam/readme.md", "ttl": 86400}
MONITORING CAN BE EASY.
▪Automation is an after thought▪Forgetting to add new services▪Checks are hard to write▪Don’t want to do annoying things
FIN.
Matthew Hodgkins
@matthodge
hodgkins.io
Flynn Bundy
@bundyfx
flynnbundy.com
BREAK TIME.
July 2016 | Behind the Scenes Flynn Bundy <@bundyfx> & Chris Thorp <@thorpgeek>
Challenges.
● OS Base Memory Usage
● Large HDD Usage
● Build Times for Servers
● Further increase the development speed of Coolblue
SAD (Server And a Desktop).
Install web server feature
Server Core.
WE
Nano Server.
Nano Server Features.
Results.
Server Core Nano Server
Build Time: 6 min ~40 Seconds
Start Time: 25 Seconds 2 Seconds
Restart Time: 15 Seconds 1 Seconds
Hard Drive Space used:
8 GB 600 MB
Basic Memory Usage 250 MB 160 MB
Nano Server Thoughts.
● No 32 Bit Support
● MSI is gone
● Not all server core features available
● No Full .Net Framework Support
Challenges.
● OS base Memory Usage
● Large HDD Usage
● Build and Deployment Times for Servers
● Further increase the development speed of Coolblue
Summing up challenges.
● OS Base Memory Usage
● Large HDD Usage
● Build Deployment Times for Servers
● Further increase the development speed of Coolblue
FIN.
Chris Thorp
@thorpgeek
Flynn Bundy
@bundyfx
flynnbundy.com
Events: coolblue.nl/behindthescenes DevBlog: devblog.coolblue.nl