performance tuning your puppet infrastructure - puppetconf 2014

Performance TuningYour (Puppet) Infrastructure

Nic Benders

OWNING

http://nicbenders.com/presentations/puppetconf-2014


I work at New Relic. !

This is a story about treating your infrastructure the same

way you treat your applications. !

This is also a story about how we use New Relic,

at New Relic.



Managing Our Applications

• Source Control

• Continuous Deployment

• Performance Monitoring

• Operational Analytics

• Log Collection

GitHub with Pull Requests

Jenkins + Capistrano

New Relic APM

New Relic Insights

Heka + ElasticSearch + Kibana

Managing Our Infrastructure

• Source Control




• Log Collection



New Relic APM

New Relic Insights


Source Control

Continuous Deployment

Performance Monitoring

DON’T PANIC• These example are for Puppet Enterprise 3.1, but this will work with Puppet

Open Source, or with newer versions. You’ll just need to make a few changes here and there.

• If you don’t have a New Relic account, you can sign-up for a free “Lite” account.

• All of the examples are up in a more useful format at the URL below. (The URL will also be shown at the end of the presentation.)



Instrumenting the Puppet Master

• Install newrelic_rpm using the Puppet’s Ruby interpreter

• Modify the config.ru in the Puppet Master root dir

• Create a newrelic.yml in the same directory

# /var/opt/lib/pe-‐puppetmaster/config.ru !# Setup the New Relic Ruby Agent before anything else require 'rubygems' require 'newrelic_rpm' # END New Relic Ruby Agent setup

# /var/opt/lib/pe-‐puppetmaster/newrelic.yml !common: &default_settings license_key: '000-‐your-‐license-‐key-‐here-‐000' app_name: "Puppet Master" log_level: info log_file_path: '/var/log/pe-‐puppet/' ssl: false capture_params: true transaction_tracer: enabled: true error_collector: enabled: true ignore_errors: "ActionController::RoutingError" production: <<: *default_settings monitor_mode: true

Getting good transaction names

• Use set_transaction_name in a Rack Middleware to tell the Ruby Agent how we are going to name the “web transactions” for the Puppet Master

• Use add_custom_parameters to pass the environment and other useful data along

# # We don't want every transaction to just say "call", so define a # Transaction "namer" for the Ruby Agent based on the URL structure # class CoolNamer def initialize(app, opts = {}) @app = app end def call(env) req = Rack::Request.new(env) _, environment, kind_of_thing, resource = req.path.split('/',4) NewRelic::Agent.set_transaction_name(kind_of_thing, :category => :rack) NewRelic::Agent.add_custom_parameters(:puppet_env => environment) NewRelic::Agent.add_custom_parameters(:puppet_resource => resource) @app.call(env) end end use CoolNamer # END Transaction "namer"

Getting more detail• The Ruby Agent’s MethodTracer

allows us to get details of what is going on inside a Transaction.

• By hooking into Puppet’s built-in Profiler calls, we can quickly instrument all the most important things.

• This example is for Puppet 3.3.1. In newer versions it is much easier.

require "puppet/util/profiler" module Puppet::Util::Profiler include ::NewRelic::Agent::MethodTracer def self.profile(message, &block) if /^Processed request / === message # Top level trace, skip for now yield else metric_name = self.message_to_metric(message) trace_execution_scoped([metric_name]) do yield end end end def self.message_to_metric(message) message.chomp! message.gsub!(/^(Compiled catalog) for .*/, '\1') message.gsub!(/^(Filtered result for catalog) .*/, '\1') message.gsub!(/$\d+$ /, '') message.gsub!(/^(Evaluated resource) /, '\1/') message.gsub!(/\[.*\]/,'') message.gsub!(/\:\s/,'/') message.gsub!(/^(Called) /, '\1/') "Custom/Puppet/#{message}" end end

Instrumenting the Puppet DB

• Download the Java Agent from your “Account Settings” page.

• Create an /opt/puppet/newrelic dir and place the newrelic.jar and newrelic.yaml file into it.

• Modify the newrelic.yaml

• Modify JAVA_ARGS

# /opt/puppet/newrelic/newrelic.yml !common: &default_settings license_key: '000-‐your-‐license-‐key-‐here-‐000' ! app_name: 'Puppet DB' log_level: info log_file_path: /var/log/pe-‐puppetdb/ ssl: false transaction_tracer: enabled: true transaction_threshold: apdex_f record_sql: obfuscated stack_trace_threshold: 0.2 explain_enabled: true explain_threshold: 0.2 browser_monitoring: auto_instrument: false production: <<: *default_settings

# /etc/syslog/pe-‐puppetdb !# Enable the New Relic Java Agent by adding this # to the bottom of the file JAVA_ARGS="-‐javaagent:/opt/puppet/newrelic/newrelic.jar ${JAVA_ARGS}"

Operational Analytics

Log Collection

Puppet Agent already goes into Syslog

• Agent syslogs have a ton of useful information. We just need to get it to a useful place.

• Our application and syslog data is collected by a Heka process on each server. It then makes its way to ElasticSearch, where it can be queried by Kibana.

Send run reports to the same place

• Add your log system as a Report processor!

• We use https://github.com/logstash/puppet-logstash-reporter

• To get the data into Heka, we had to write some Lua, but it was pretty straightforward.

https://github.com/logstash/puppet-logstash-reporter

# Logstash-‐style report processer through Heka class { 'logstash_reporter': logstash_port => '22253', logstash_host => hiera('logstash_reporter::logstash_host'), }

pe_puppet::master::config_hash: reports: 'http,puppetdb,logstash' !logstash_reporter::logstash_host: 'our-‐heka-‐server'

Managing Our Infrastructure

• Source Control




• Log Collection



New Relic APM

New Relic Insights


Site Reliability Engineers MySQL Experts Engineering Managers Data Pipeline Engineers newrelic.com/about/careers

We’re Hiring

http://newrelic.com/about/careers


[email protected]


performance tuning your puppet infrastructure - puppetconf 2014

Technology

new relic account

puppetopen source

pull requestsjenkins

treatingyour infrastructure

free lite account

thepuppets ruby interpreter

newer versions

puppet enterprise