rovio's infrastructure the new path forward

25
Rovio © 2016 Confidential

Upload: paul-villacorta

Post on 12-Apr-2017

87 views

Category:

Technology


2 download

TRANSCRIPT

Page 1: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

Page 2: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

Rovio’s Infrastructure The New Path ForwardPaul Winston VillacortaSenior Cloud Engineer, Cloud OperationsRovio Entertainment

Page 3: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

HybridCloud

Private

Cloud

PublicCloud

3rd Party ManagedPrivate Cloud

Page 4: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

Private Cloud Public Cloud

Page 5: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

ORCHESTRATION

TRAFFIC MANAGEMENTTRAFFIC MANAGEMENT

TRAFFIC MANAGEMENTSERVICES

TRAFFIC MANAGEMENTDATABASE

Amazon CloudServices

External ServiceProvider

CDN

ExternalMonitoring

Incident Management

System

CONTINUOUSINTEGRATION

CONTINUOUSDELIVERY

Page 6: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

EXTERNAL TRAFFIC MANAGEMENT

SERVICES

INTERNAL TRAFFIC MANAGEMENT

MICROSERVICES SERVICES MICROSERVICES

DATABASE TIER

ORCHESTRATION / AUTOMATION LOGIC

CACHING COMPONENTS STORAGE

MON

ITOR

ING

LOG

COLL

ECTI

ON

INTE

RNAL

DNS

OPERATIONAL JOB SCHEDULER

VIRTUALIZED

Page 7: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

IT IS HUGE!!

Page 8: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

200+DatabaseInstances

60+Services

700 - 2000Running

Instances

{API}

45000+API Requestsper second

3BAnalyticsProcessedevery day

Page 9: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

AUTOMATION

WE DON’T ALWAYS AUTOMATE STUFFBUT WHEN WE DO IT’S END-TO-END

Page 10: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

AUTOMATION• Scalability – Automatic node creation

• Availability – Automatic node replacement

• Efficiency – Automatic intelligence

Load Testing, Disaster Recovery, Rolling Upgrades, Compliance, Portability, etc…

Page 11: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

AUTOMATION TOOLS• Scripting / Programmable Utilities• Puppet, Fabric, Ansible• Docker, Packer• Vagrant

Page 12: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

AUTOMATION

• Builder Script is written in Python• Utilizes REST APIs• 1,100 lines of code• Week of development + testing

• 5 seconds to create new project• We have saved 70 steps and 140 clicks

Continuous Integration /Continuous

Delivery

Page 13: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

AUTOMATION

• It’s GUI based• It’s API driven• It supports standalone CLI• It’s cloud aware• It’s cluster aware• It’s inside the cloud• It works like a charm

Rovio’s Orchestration

Server

Page 14: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

AUTOMATION

OrchestrationServer

Cloud Provide

r

New Node

Cloud Properties

Page 15: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

AUTOMATION

OrchestrationServer

Internal DNS

Monitoring

Ready Node

PACKAGE

PUPPETMANIFEST

S

Page 16: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

AUTOMATION

OrchestrationServer

Internal DNS

Monitoring

Ready NodeTraffic

Management

Page 17: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

LESSONS LEARNED

METRICS AND VISIBILITY - SMALL FAULT MARGINS - LEARNING FROM MISTAKES

Page 18: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

Metrics and Visibility• Sensible Metrics• Actionable Items• Metrics that can be visualized and aggregated

Page 19: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

Metrics and Visibility

Page 20: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

Metrics and Visibility

Page 21: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

Small fault margins• Small mistakes build up quickly• Small mistakes are unpredictable• Small mistakes costs money!

Page 22: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

Learning from mistakes• NEVER leave critical components UNATTENDED!• Do it RIGHT from the BEGINNING!• Do a proper root-cause analysis and publish the findings

in your organization.

Page 23: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

CHALLENGES

Page 24: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

CHALLENGES• Modernizing legacy applications• Cloud Providers configuration changes• Security• Legacy nodes• AZ Distribution• Architectural Challenges

Page 25: Rovio's Infrastructure The New Path Forward

Rovio © 2016 Confidential

Thank You!