solving real world data problems with jerakia
Post on 15-Apr-2017
856 Views
Preview:
TRANSCRIPT
Solving real world data problems with JerakiaCraig Dunn, Config Management Camp, Ghent 2016
• Best practice
• Code base design
• Workflow mangement
• Scaling Puppet
• Installation and support
• Module writing
• Throughout Europe
www.enviatics.com
• Puppet user since 2008
• IT consultant for 15+ years
• Active community member
• The “Roles and Profiles” guy
• Problem solver
• Lives in Málaga, Spain.
• …. and hotels
• Daddy!
www.craigdunn.org
Craig Dunn
@crayfishx
A brief history of Puppet
In the beginning…
• Over complex code
• Unsharable modules
• Making simple changes required alot of skill.
The embedded data era
class ntp { if $env == ‘dev’ { $server = ‘dev.ntp.local’ } else { if $hostname == ‘gateway’ { $server = ‘pool.ntp.org’ } else { $server = ‘prod.ntp.local’ } } …}
And then…
HieraThe dawn of the data separation era
• Separation of data from code
• Module authors could write sharable re-usable code
• Code was less complex and more readable
• The Forge became useful
• Managing data became a lot easier
Hierarchical Search
Pluggable• Pluggable interchangable backends
• Data can be sourced from multiple formats
• hiera-eyaml
• hiera-mysql
• hiera-http
• hiera-redis
• hiera-consul
Managing our data is now a critical part of configuration management
Infrastructure grows and requirementsget more complex
• Different teams and customers require different hierarchies
• A particular application needs to source data from a different place
• Control access to sub-sets of data for teams within an organisation
• Dynamically generate the lookup hierarchy at runtime
• Group together application specific data into separate files
• Manage encrypted data from any data source
• Global hiera.yaml file creates restrictions
Jerakia• Data lookup tool
• Open source
• Extendable framework
• Solving the most complex edge cases
Jerakia• Can be used as a Hiera backend
• Can be wired directly into Puppet as a data binding terminus
• Drop in replacement for Hiera, or not.
Why Jerakia?
One design goal…
Flexibility
• Lookup behaviour written in Ruby DSL
• Almost everything is pluggable
• Inter-changable data sources
• Easy integration
• Hiera compatible*
$ gem install jerakia
$ puppet module install crayfishx/jerakia
• A request is received containing a key and a namespace
• A policy is chosen to perform the request
• One or more lookups are called to act on the request
• A response is sent back to the requestor
• Container for lookups
• Written in Ruby DSL
• Different policies for different apps
Policy File
An Example Jerakia Policy File
policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } endend
An Example Jerakia Policy File
policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } endend
An Example Jerakia Policy File
policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } endend
An Example Jerakia Policy File
policy :main do lookup :default do datasource :file, { :docroot => "/var/jerakia/data", :format => :yaml, :searchpath => [ "host/#{scope[:hostname]}", "env/#{scope[:env]}", "common", ] } endend
• Lookups are contained within policies
• A policy can contain multiple lookups
• A lookup always contains at least a data source
Lookups
Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookupRequest consists of a
lookup key, a namespace and some metadata
Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookupInformation to be
used in determining how data is looked up
Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
Lookup plugins can read and modify the scope and
request objects
Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
A pluggable data source is used to lookup data
Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
Data returned from the datasource is passed to a
pluggable output filter
Lookup methods
confine / exclude
Invalidates a lookup unless/if the criteria is met
confine request.namespsace[0], "apache"
confine request.namespsace[0], [ /website_.*/, "apache", "php" ]
Stop
Do not proceed to the next lookup if this lookup is valid
lookup :special do … confine request.namespsace[0], "apache" stopend
lookup :main do …
Datasources• Easily pluggable and extendable
• File and HTTP datasources shipped out-of-the-box
Datasources datasource :name, { :option => “value”… }
Datasource definitionlookup :main do
datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", "common", ] }
end
/var/lib/jerakia/env/dev/apache.yaml
lookup :main do
datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", "common", ] }
end
/var/lib/jerakia/env/dev/apache.yaml
Datasource definition
lookup :main do
datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "host/#{scope[:certname]}", "env/#{scope[:environment]}", "common", ] }
end
/var/lib/jerakia/env/dev/apache.yaml
Datasource definition
/var/lib/jerakia/env/dev/apache.yaml/var/lib/jerakia/env/dev/apache.d/www_corp_com.yaml/var/lib/jerakia/env/dev/apache.d/www_acme_net.yaml/var/lib/jerakia/env/dev/apache.d/www_fake_org.yaml
Fragments• Introduced in 0.4
• If a .d directory is found, files within are concatenated
• One document is returned
Data Layout :searchpath => [
"host/#{scope[:certname]}", "env/#{scope[:environment]}", ]
# cat /var/lib/jerakia/env/dev/apache.yaml—-port: 80
# cat /var/lib/jerakia/env/dev.yaml—-apache::port: 80
Hiera
Jerakia
Plugins• Access to request and scope
• Can read or modify on-the-fly
• Re-usable
• Cleaner code in policy files
class Jerakia::Lookup::Plugin module Mything def do_something … end endend
Writing plugins• Written as Ruby extensions
• Can be placed in the plugin dir
• Or shipped as rubygems
lookup :main, :use => :mything do plugin.mything.do_something …end
Using plugins• Plugins are loaded into the lookup
• Referenced as plugin.name.method
lookup :main, :use => [ :mything, :foo ] do …end
lookup :main, :use => :hiera do plugin.hiera.rewrite_lookup datasource :file, { :docroot => "/var/lib/jerakia", :format => :yaml, :searchpath => [ "env/#{scope[:environment]}", "common", ]end
The hiera plugin• Provides compatibility to hiera filesystem layouts
• Shipped with Jerakia
# cat /var/lib/jerakia/env/dev.yaml—-apache::port: 80
Output filters
• Pluggable
• Specified in the lookup
• Parses data returned from the datasource
Output filters
• Two are currently shipped
• Encryption (provided by eyaml*)
• Strsub
*https://github.com/TomPoulton/hiera-eyaml
Output filters
lookup :main do …
output_handler :encryptionend
Output filters
lookup :main do …
output_handler :encryptionend
Example User Story• Team in Ireland manage PHP/Apache
• Autonomous team that don’t manage infra
• Their optimal hierarchy is different from “ours”
• “We” need to service them from Puppet
• They must not modify infra services
• “We” also manage PHP/Apache for other clients
policy :default do lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end
Our main lookup is responsible for the entire
infrastructure
policy :default do lookup :ireland do datasource :file, { :format => :yaml, :docroot => "/var/external/data/ie", :searchpath => [ "project/#{scope[:project]}", "common", ] } end lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end
Lookup for the Ireland team added above the
main lookup with separate docroot and
searchpath
policy :default do lookup :ireland do datasource :file, { :format => :yaml, :docroot => "/var/external/data/ie", :searchpath => [ "project/#{scope[:project]}", "common", ] } confine scope[:location], "ie" confine request.namespace[0], [ "apache", "php", ] end lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end
Only use this lookup if the requestor location is IE and the namespace is
apache or php
policy :default do lookup :ireland do datasource :file, { :format => :yaml, :docroot => "/var/external/data/ie", :searchpath => [ "project/#{scope[:project]}", "common", ] } confine scope[:location], "ie" confine request.namespace[0], [ "apache", "php", ] stop end lookup :main, do datasource :file, { :format => :yaml, :docroot => "/var/lib/jerakia", :searchpath => [ "hostname/#{scope[:fqdn]}", "environment/#{scope[:environment]}", "common" ], } end end
If this lookup is valid then do not proceed to the
main lookup, even if data is not found.
Command line $ jerakia lookup port —namespace apache
$ jerakia help lookupUsage: jerakia lookup [KEY]
Options: c, [--config=CONFIG] # Configuration file p, [--policy=POLICY] # Lookup policy # Default: default n, [--namespace=NAMESPACE] # Lookup namespace t, [--type=TYPE] # Lookup type # Default: first s, [--scope=SCOPE] # Scope handler # Default: metadata [--scope-options=key:value] # Key/value pairs to be passed to the scope handler m, [--merge-type=MERGE_TYPE] # Merge type # Default: array l, [--log-level=LOG_LEVEL] # Log level v, [--verbose], [--no-verbose] # Print verbose information D, [--debug], [--no-debug] # Debug information to console, implies --log-level debug d, [--metadata=key:value] # Key/value pairs to be used as metadata for the lookup
Lookup [KEY] with Jerakia
Integration with Puppet—-:backends: - jerakia
[master] . . . data_binding_terminus = jerakia
Roadmap & Contributing
Upcoming in 0.5• Data Schemas
• Better REST client/server
• Deep merge behaviour
• Lookup plugin “load method”
Contributions wanted• Code maturity
• Caching
• Features
• Bugfixes
• Documentation
• #jerakia (freenode) Sponsored by
Jerakia 1.0
top related