hacking ansible

Hacking Ansiblemake it do more

ansiblefest 10/2014

Who am I?

programmerDBAsystem administratortinkererdevopstech-janitor

____________ < irc lurker > ------------ \ , , \ /( )` \ \ \___ / | /- _ `-/ ' (/\/ \ \ /\ / / | ` \ O O ) / | `-^--'`< ' (_.) _ ) / `.___/` / `-----' /<----. __ / __ \<----|====O)))==) \) /====<----' `--' `.__,' \ | | \ / ______( (_ / \______ ,' ,-----' | \ `--{__________) \/

________________/ It runs a TASK \ \ on a HOST / ---------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||

What is ansible?

● Configuration management?

● Automation platform?

● Release management?

● Orchestration system?

The Guts: ansible internal objects● inventory: defines my targets● runner: actually does the work● connection: how to get to my targets● playbook: all plays in current invocation

○ play■ host: where to do it■ task: “it” aka “what to do”

● callback: show me what was done

Example hack - the core--- a/lib/ansible/playbook/__init__.py+++ b/lib/ansible/playbook/__init__.py@@ -346,6 +346,8 @@ class PlayBook(object): run_hosts=hosts ) + runner.module_vars.update({'play_hosts': hosts})+ if task.async_seconds == 0: results = runner.run() else:

Example hack - the core

● 10 second patch to add play_hosts

● 3h to find correct spot to patch

● learned a lot about playbook/host iteration

● I did not take “serial:” into account

● runner rewrite in progress

The

plugin

system

PluginLoader...

lookup_loader = PluginLoader( 'LookupModule', 'ansible.runner.lookup_plugins', C.DEFAULT_LOOKUP_PLUGIN_PATH, 'lookup_plugins')vars_loader = PluginLoader( 'VarsModule', 'ansible.inventory.vars_plugins', C.DEFAULT_VARS_PLUGIN_PATH, 'vars_plugins')

lib/ansible/utils/plugins.py

__________________ / all plugins \\ except inventory / ------------------ \ ,__, \ (oo)____ (__) )\ ||--|| *

Plugins

● Library: host tasks/actions/modules

● Action: master side tasks/actions/modules

● Cache: fact caching

● Callback: play output

● Connection: host connections

Plugins

● Shell: what shell to use to execute tasks● Lookup: master side info lookup● Vars: variable imports● Inventory: aside from inventory scripts● Filter: jinja2 filters for data modification● Doc Fragment: shared docs for library

Library/Action - May 2013 (72)add_host,debug,get_url,mount,postgresql_user,slurp,apt,django_manage,git,mysql_db,rabbitmq_parameter,subversion,apt_key,easy_install,group,mysql_user,rabbitmq_plugin,supervisorctl,apt_repository,ec2,group_by,nagios,rabbitmq_user,svr4pkg,assemble,ec2_facts,hg,ohai,rabbitmq_vhost,sysctl,async_status,ec2_vol,ini_file,opkg,raw,template,async_wrapper,facter,libr,pacman,script,uri,authorized_key,fail,lineinfile,pause,seboolean,user,cloudformation,fetch,lvol,ping,selinux,virt,command,file,macports,pip,service,wait_for,copy,fireball,mail,pkgin,setup,yum,cron,gem,mongodb_user,postgresql_db,shell,zfs

Library/Action - October 2014 (175)a10_server,a10_service_group,a10_virtual_server,accelerate,acl,add_host,airbrake_deployment,alternatives,apache2_module,apt,apt_key,apt_repository,apt_rpm,assemble,assert,async_status,at,authorized_key,azure,bigip_facts,bigip_monitor_http,bigip_monitor_tcp,bigip_node,bigip_pool,bigip_pool_member,bigpanda,boundary_meter,bzr,campfire,capabilities,cloudformation,command,composer,copy,cpanm,cron,datadog_event,debconf,debug,digital_ocean,digital_ocean_domain,digital_ocean_sshkey,django_manage,dnsimple,dnsmadeeasy,docker,docker_image,easy_install,ec2,ec2_ami,ec2_ami_search,ec2_asg,ec2_eip,ec2_elb,ec2_elb_lb,ec2_facts,ec2_group,ec2_key,ec2_lc,ec2_metric_alarm,ec2_scaling_policy,ec2_snapshot,ec2_tag,ec2_vol,ec2_vpc,ejabberd_user,elasticache,facter,fail,fetch,file,filesystem,fireball,firewalld,flowdock,gc_storage,gce,gce_lb,gce_net,gce_pd,gem,get_url,getent,git,github_hooks,glance_image,group,group_by,grove,hg,hipchat,homebrew,homebrew_cask,homebrew_tap,hostname,htpasswd,include_vars,ini_file,irc,jabber,jboss,jira,kernel_blacklist,keystone_user,layman,librato_annotation,lineinfile,linode,lldp,locale_gen,logentries,lvg,lvol,macports,mail,modprobe,mongodb_user,monit,mount,mqtt,mysql_db,mysql_replication,mysql_user,mysql_variables,nagios,netscaler,newrelic_deployment,nexmo,nova_compute,nova_keypair,npm,ohai,open_iscsi,openbsd_pkg,openvswitch_bridge,openvswitch_port,opkg,osx_say,ovirt,pacman,pagerduty,pause,ping,pingdom,pip,pkgin,pkgng,pkgutil,portage,portinstall,postgresql_db,postgresql_privs,postgresql_user,quantum_floating_ip,quantum_floating_ip_associate,quantum_network,quantum_router,quantum_router_gateway,quantum_router_interface,quantum_subnet,rabbitmq_parameter,rabbitmq_plugin,rabbitmq_policy,rabbitmq_user,rabbitmq_vhost,raw,rax,rax_cbs,rax_cbs_attachments,rax_cdb,rax_cdb_database,rax_cdb_user,rax_clb,rax_clb_nodes,rax_dns,rax_dns_record,rax_facts,rax_files,rax_files_objects,rax_identity,rax_keypair,rax_meta,rax_network,rax_queue,rax_scaling_group,rax_scaling_policy,rds,rds_param_group,rds_subnet_group,redhat_subscription,redis,replace,rhn_channel,rhn_register,riak,rollbar_deployment,route53,rpm_key,s3,script,seboolean,selinux,service,set_fact,setup,shell,slack,slurp,sns,stackdriver,stat,subversion,supervisorctl,svr4pkg,swdepot,synchronize,sysctl,template,twilio,typetalk,ufw,unarchive,uri,urpmi,user,virt,vsphere_guest,wait_for,win_feature,win_get_url,win_group,win_msi,win_ping,win_service,win_stat,win_user,xattr,yum,zabbix_maintenance,zfs,zypper,zypper_repository

Filters: Jinja2’ism

● best way to change data

● chainable pipes

● simple to expand

● can hide complexity

__________________________< [hello, world]|join(‘ ‘) > -------------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||

Example hack - filter

import random

def randomize_list(mylist): random.shuffle(mylist) return mylist

...

def filters(self): # this maps filter names to functionsreturn {

‘shuffle’: randomize_list,

...

}

lib/ansible/runner/filter_plugins/core.py

Example hack - filter- hosts: all connection: local gather_facts: false vars: - mylist: [1,2,3,4,5] tasks: - debug: msg=”{{item}}” with_items: “{{mylist|shuffle}}”

Example hack - filter __________________________ < TASK: debug msg={{item}} > --------------------------

ok: [localhost] => (item=1) => { "item": 1, "msg": "1"}ok: [localhost] => (item=3) => { "item": 3, "msg": "3"}ok: [localhost] => (item=4) => { "item": 4, "msg": "4"}ok: [localhost] => (item=5) => { "item": 5, "msg": "5"}ok: [localhost] => (item=2) => { "item": 2, "msg": "2"}

__________________________ < TASK: debug msg={{item}} > --------------------------




#>ansible-playbook test.yml -i ‘localhost,’

Lookups

● you are already using them: with_<lookup>

● they execute on the “master”

● a way to access external files and/or data

● normally returns a list

Example hack - lookup

class LookupModule(object):

def __init__(self, basedir=None, **kwargs): self.basedir = basedir self.etcd = etcd() # initializes etcd class

def run(self, terms, inject=None, **kwargs): terms = utils.listify_lookup_plugin_terms(terms, self.basedir, inject) if isinstance(terms, basestring): terms = [ terms ]

ret = [] for term in terms: key = term.split()[0] value = self.etcd.get(key) # gets the data from etcd class ret.append(value) return ret

lib/ansible/runner/lookup_plugins/etcd.py


class etcd(): # simplified version

def __init__(self, url=ANSIBLE_ETCD_URL): self.url = url self.baseurl = '%s/v1/keys' % (self.url)

def get(self, key): url = "%s/%s" % (self.baseurl, key) r = urllib2.urlopen(url) data = r.read()

item = json.loads(data) value = item['value']

return value

lib/ansible/runner/lookup_plugins/etcd.py

_______________/ written by \\ Jan-Piet Mens / --------------- \ ,__, \ (oo)____ (__) )\ ||--|| *


#>ansible all -m debug -a ‘msg={{lookup(“etcd”,”key1”)}}’ -i ‘localhost,’

localhost | success >> { "msg": "value1"}

#>ansible all -m debug -a ‘msg={{lookup(“etcd”,”key2”)}}’ -i ‘localhost,’localhost | success >> { "msg": "value2"}

Example hack - callback

class CallbackModule(object): # simplified example

def playbook_on_task_start(self, name, is_conditional):

if self.current is not None: # Record the total time of the previous self.stats[self.current] = time.time() - self.stats[self.current]

self.current = name # Record the start time of the current task self.stats[self.current] = time.time()

def playbook_on_stats(self, stats): # Sort the tasks by their running time

... results = sorted(self.stats.items(), key=lambda value: value[1], reverse=True)

for name, elapsed in results: # Print the timings print "{0:-<70}{1:->9}".format('{0} '.format(name),'{0:.02f}s'.format(elapsed))

callback_plugins/profile_tasks.py

_______________/ written by \\ Jharrod LaFon / --------------- \ ,__, \ (oo)____ (__) )\ ||--|| *

Example hack - callback

# re-running the previous filter/shuffle playbook


ok: [localhost] => (item=2) => { "item": 2, "msg": "2"}

...

ok: [localhost] => (item=4) => { "item": 4, "msg": "4"}

PLAY RECAP ******************************************************************** debug msg={{item}} ------------------------------------------------------ 0.02slocalhost : ok=1 changed=0 unreachable=0 failed=0

#>ansible-playbook test.yml -i ‘localhost,’

Example hack - cacheimport exceptions

class BaseCacheModule(object):

def get(self, key): raise exceptions.NotImplementedError

def set(self, key, value): raise exceptions.NotImplementedError

def keys(self): raise exceptions.NotImplementedError

def contains(self, key): raise exceptions.NotImplementedError

def delete(self, key): raise exceptions.NotImplementedError

def flush(self): raise exceptions.NotImplementedError

def copy(self): raise exceptions.NotImplementedError

________________ / I abandoned it \| and Josh Drake |\ revived it / ---------------- \ /\ ___ /\ \ // \/ \/ \\ (( O O )) \\ / \ // \/ | | \/ | | | | | | | | | o | | | | | |m| |m|

lib/ansible/cache/base.py

Example hack - cache

from ansible.cache.base import BaseCacheModule

class CacheModule(BaseCacheModule):

def __init__(self, *args, **kwargs): self._cache = {}

def get(self, key): return self._cache.get(key)

def set(self, key, value): self._cache[key] = value

def keys(self): return self._cache.keys()

def contains(self, key): return key in self._cache

def delete(self, key): del self._cache[key]

def flush(self): self._cache = {}

def copy(self): return self._cache.copy()

lib/ansible/cache/memory.py

Example hack - cache

from redis import StrictRedis

class CacheModule(BaseCacheModule): # simplified version

self._cache = StrictRedis(*connection)...

def get(self, key): value = self._cache.get(self._make_key(key)) # make_key creates correct prefix # guard against the key not being removed from the zset if value is None: self.delete(key) raise KeyError return json.loads(value)

def set(self, key, value): value2 = json.dumps(value) if self._timeout > 0: # a timeout of 0 is handled as meaning 'never expire' self._cache.setex(self._make_key(key), int(self._timeout), value2) else: self._cache.set(self._make_key(key), value2)

self._cache.zadd(self._keys_set, time.time(), key)...

lib/ansible/cache/redis.py

Tests: test/

● sadly the presenter sucks at test coverage

● no really, tests are good, regressions!

● assert: the test module

● destructive, non destructive, integration

● ‘make tests’, runs unit tests

Before you submit

● update documentation (it is also in the repo)

● create tests, assert! (also in the repo)

● prepare a clear usage/example to post in the PR

● its a dialog, be ready to make your ‘use case’

● patience … … … …

● not everything belongs in core

hacking ansible

Technology