the evolving design patterns of puppet enterprise - puppetconf 2014
DESCRIPTION
The Evolving Design Patterns of Puppet Enterprise - Jonathan Spinks, Sourced Group & John Painter, Sourced GroupTRANSCRIPT
T H E E V O LV I N G D E S I G N PAT T E R N S O F P U P P E T E N T E R P R I S E
John Painter Jon SpinksSolutions Architect Solutions Architect
sourcedgroup.com
www.linkedin.com/in/painterj www.linkedin.com/in/jonspinks
Sourced Group architect and deploy a globally distributed multi-master puppet solution for a large investment bank
February 2011
CUSTOMER REQUIREMENTS
• Provide a single point of control for a globally distributed IT environment
• Reduce overall configuration drift in the existing fleet
• Provide a change control process to manage code updates
DESIGN PATTERN 1 Global Multi Master
C E N T R A L M G M T. SYDNEY
Puppeteer
Puppet Master
LDAP ENC
Puppet Master
Puppet Console
NEW YORK
Puppet Master
LDAP ENC
Puppet Master
Puppet Console
HONG KONG
Puppet Master
LDAP ENC
Puppet Master
Puppet Console
LDAP ENC Master
LONDON
Puppet Master
LDAP ENC
Puppet Master
Puppet Console
USA: 6pm - 8amEurope: 6pm - 8amASIA: 6pm - 8am
Global risk management, simplified releases
Follow-the-Sun Code Release
Client Client
Puppet Master
Client Client
Puppet Master
Client Client
Puppet Master
AUDITING GOVERNANCE SIMPLIFIED CONTROL
Control and report on the point of truth
K E Y TA K E A W AY S
• LDAP is a great distributed ENC • Puppet Reports can be large en masse • A Puppet Master can used to manage other
Puppet Masters • Version control is critical as a basis for a change
control process
Sourced Group architect and deploy a globally distributed multi-master puppet solution for a large investment bank
February 2011
Sourced are the first to onboard an Asia Pacific
telecommunications company onto AWS, powered by Puppet
Enterprise
February 2012
CUSTOMER REQUIREMENTS
• Provide a single point of control for instance provisioning
• Provide infrastructure orchestration • Maintain portability of applications to the
traditional data centre
E V E R Y T H I N G I S P R O G R A M AT I C
S TA N D A R D I S AT I O N O F I N F R A S T R U C T U R E
DESIGN PATTERN 2 Puppet Managing AWS Compute
Puppet CloudPack
EC2 only support circa 2012
Compute Compute
Compute Compute
Public IP Public IP
Public IP Public IP
EC2
Customer
IP Whitelisting
Virtual Private Cloud!Private IP Space
Compute Compute
Compute ComputeCustomer
VPN / Direct Connect
Public IP
VPC
Bootstrap Agent
Classify Node
Application DeployedCreate Instance
K E Y TA K E A W AY S
• Puppet Masters can be used for OS instance delivery
• CloudPack supports EC2 and VPC
Sourced are the first to onboard an Asia Pacific
telecommunications company onto AWS, powered by Puppet
Enterprise
February 2012
“Using Puppet Enterprise Edition as Heterogeneous Cloud Glue”
PuppetConf 2012
C L O U D PA C K U P L I F T
• Uplifted the existing Puppet CloudPack to support AWS VPC
• Left Puppet Enterprise in the middle of the provisioning and lifecycle management
• Supported Multi-cloud provisioning
• http://www.youtube.com/watch?v=mwiBjJZWraA
“Using Puppet Enterprise Edition as Heterogeneous Cloud Glue”
PuppetConf 2012
Sourced implement a masterless Puppet Enterprise Edition solution for a financial
services organisation
March 2013
CUSTOMER REQUIREMENTS
• Harness transient compute while enforcing compliance
• Provide OS and application state transparency
S TA N D A R D I S AT I O N A N D C O N S O L I D AT I O N
DESIGN PATTERN 3 Masterless Puppet with AWS Integration
C L O U D C O M P U T I N G I S F U N D A M E N TA L LY D I F F E R E N T
W E B A P P V. 1
T E S T
B A K E
L A U N C H
T E A R D O W N
W E B A P P V. 2
T E S T
B A K E
L A U N C H
T E A R D O W N
W E B A P P D E V - V. 1
T E S T
B A K E
L A U N C H
T E A R D O W N
UAT
Production
UAT
ProductionA P P V 1 . 1
A P P V 1 . 0
V 1 . 1
Features
V 2 . 0
V 1 . 1
V 2 . 0
Infrastructure, deployment, and configuration have standardised
!
It was not just servers that were transient, it was entire application stacks
!
Cloud computing has significantly changed the application lifecycle
– N I G E L K E R S T E N - P U P P E T C A M P S Y D N E Y 2 0 1 4
“18 months really sucks”
- E V E R Y S I N G L E P R O D U C T M A N A G E R . E V E R .
“18 months really sucks”
Weekly/Monthly -> Hourly
Load
Time
$
$
$ $
$
$
PROBLEM SPACE
• Puppet Masters were designed to manage long lived OS instances
• Transient compute doesn’t really need a dashboard • Puppet Console accuracy was impacted by stale
instance data • Requirement to simplify the Auto Scale process
• CloudFormation for orchestration • AWS EC2 Instance Tags provide node definition • Facter is extended to understand AWS Instance
Metadata • Manifests and modules are bundled and centrally
managed • Centralised logging to provide state data
MASTERLESS COMPONENTS
Version Control
Instance Instance
CI/CD
AW S S 3
Centralised Logging
InstanceInstance
Application Configuration
Configuration Managent
Governance and Compliance
Configuration Managent
Governance and ComplianceApplication Configuration
K E Y TA K E A W AY S
• Cloud Computing is fundamentally different • Harnessing transient compute is extremely
valuable • Losing the Puppet Master reduces overall
visibility when considering long lived and transient instances as a whole
DESIGN PATTERN 4 Puppet Master and Transient Compute
D ATA B U I L D S T R A N S PA R E N C Y
Getting the Puppet Master Back in the Loop
A W S C O M P U T E T Y P E S
Transient Compute
Stateful!Compute
P U P P E T A W S A U T O S C A L I N G B R O K E R
Instance Instance
Simple Notification Service (SNS)
Autoscaling Group
SNS Topic
Simple Queue Service (SQS)Autoscale BROKER
Instance Instance
K E Y TA K E A W AY S
• Using a Puppet Master as a point of control for all compute builds transparency
• The Puppet Master framework is very extensible • PuppetDB is very valuable • Leverage AWS Auto Scale orchestration features
A LOOK FORWARD Heterogeneous Puppet Governance
Hardware
Operating System
Runtime
Application
Hardware
Operating System
Runtime
Application
Rather than making the OS a good candidate for the application, we are now tailoring the
applications to be a good citizen of a standardised compute platform.
Docker
Pivotal CF
OpenShift
Azure Pack
Examples of higher-level application delivery frameworks that can run on AWS
T H E F R A M E W O R K S S T I L L N E E D T O B E G O V E R N E D
H Y B R I D C L O U D
On Premise Managed
Stateful Compute
Transient Compute
GOVERNANCE
GOVERNANCE
GOVERNANCE
GOVERNANCE
GOVERNANCE
GOVERNANCE
In an ideal world with true vendor arbitrage we would be able to deliver an application to AWS, OpenShift on Rackspace and Docker running on
managed VMware using only CI/CD tooling
APP APP APP APP
Puppet + Autoscale Broker + Global Distribution + Multi Cloud
AWS - Sydney
Puppet Master
APP DELIVERY FRAMEWORK
APP DELIVERY FRAMEWORK
APP DELIVERY FRAMEWORK
AWS - USA
Puppet Master
APP DELIVERY FRAMEWORK
APP DELIVERY FRAMEWORK
APP DELIVERY FRAMEWORK
<CLOUD> - Singapore
Puppet Master
APP DELIVERY FRAMEWORK
APP DELIVERY FRAMEWORK
APP DELIVERY FRAMEWORK
Runtime
Puppet Master
APP DELIVERY FRAMEWORK
APP DELIVERY FRAMEWORK
APP DELIVERY FRAMEWORK
Your Datacenter
K E Y TA K E A W AY S
• Through the adoption of PaaS the bar of commoditisation continues to rise
• Governance and compliance are still critical • Application delivery performance continues to
increase
S O U R C E D G R O U P. C O M