hacking ansible
Post on 29-Nov-2014
1.898 Views
Preview:
DESCRIPTION
TRANSCRIPT
Hacking Ansiblemake it do more
ansiblefest 10/2014
Who am I?
programmerDBAsystem administratortinkererdevopstech-janitor
____________ < irc lurker > ------------ \ , , \ /( )` \ \ \___ / | /- _ `-/ ' (/\/ \ \ /\ / / | ` \ O O ) / | `-^--'`< ' (_.) _ ) / `.___/` / `-----' /<----. __ / __ \<----|====O)))==) \) /====<----' `--' `.__,' \ | | \ / ______( (_ / \______ ,' ,-----' | \ `--{__________) \/
________________/ It runs a TASK \ \ on a HOST / ---------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||
What is ansible?
● Configuration management?
● Automation platform?
● Release management?
● Orchestration system?
The Guts: ansible internal objects● inventory: defines my targets● runner: actually does the work● connection: how to get to my targets● playbook: all plays in current invocation
○ play■ host: where to do it■ task: “it” aka “what to do”
● callback: show me what was done
Example hack - the core--- a/lib/ansible/playbook/__init__.py+++ b/lib/ansible/playbook/__init__.py@@ -346,6 +346,8 @@ class PlayBook(object): run_hosts=hosts ) + runner.module_vars.update({'play_hosts': hosts})+ if task.async_seconds == 0: results = runner.run() else:
Example hack - the core
● 10 second patch to add play_hosts
● 3h to find correct spot to patch
● learned a lot about playbook/host iteration
● I did not take “serial:” into account
● runner rewrite in progress
The
plugin
system
PluginLoader...
lookup_loader = PluginLoader( 'LookupModule', 'ansible.runner.lookup_plugins', C.DEFAULT_LOOKUP_PLUGIN_PATH, 'lookup_plugins')vars_loader = PluginLoader( 'VarsModule', 'ansible.inventory.vars_plugins', C.DEFAULT_VARS_PLUGIN_PATH, 'vars_plugins')
lib/ansible/utils/plugins.py
__________________ / all plugins \\ except inventory / ------------------ \ ,__, \ (oo)____ (__) )\ ||--|| *
Plugins
● Library: host tasks/actions/modules
● Action: master side tasks/actions/modules
● Cache: fact caching
● Callback: play output
● Connection: host connections
Plugins
● Shell: what shell to use to execute tasks● Lookup: master side info lookup● Vars: variable imports● Inventory: aside from inventory scripts● Filter: jinja2 filters for data modification● Doc Fragment: shared docs for library
Library/Action - May 2013 (72)add_host,debug,get_url,mount,postgresql_user,slurp,apt,django_manage,git,mysql_db,rabbitmq_parameter,subversion,apt_key,easy_install,group,mysql_user,rabbitmq_plugin,supervisorctl,apt_repository,ec2,group_by,nagios,rabbitmq_user,svr4pkg,assemble,ec2_facts,hg,ohai,rabbitmq_vhost,sysctl,async_status,ec2_vol,ini_file,opkg,raw,template,async_wrapper,facter,libr,pacman,script,uri,authorized_key,fail,lineinfile,pause,seboolean,user,cloudformation,fetch,lvol,ping,selinux,virt,command,file,macports,pip,service,wait_for,copy,fireball,mail,pkgin,setup,yum,cron,gem,mongodb_user,postgresql_db,shell,zfs
Library/Action - October 2014 (175)a10_server,a10_service_group,a10_virtual_server,accelerate,acl,add_host,airbrake_deployment,alternatives,apache2_module,apt,apt_key,apt_repository,apt_rpm,assemble,assert,async_status,at,authorized_key,azure,bigip_facts,bigip_monitor_http,bigip_monitor_tcp,bigip_node,bigip_pool,bigip_pool_member,bigpanda,boundary_meter,bzr,campfire,capabilities,cloudformation,command,composer,copy,cpanm,cron,datadog_event,debconf,debug,digital_ocean,digital_ocean_domain,digital_ocean_sshkey,django_manage,dnsimple,dnsmadeeasy,docker,docker_image,easy_install,ec2,ec2_ami,ec2_ami_search,ec2_asg,ec2_eip,ec2_elb,ec2_elb_lb,ec2_facts,ec2_group,ec2_key,ec2_lc,ec2_metric_alarm,ec2_scaling_policy,ec2_snapshot,ec2_tag,ec2_vol,ec2_vpc,ejabberd_user,elasticache,facter,fail,fetch,file,filesystem,fireball,firewalld,flowdock,gc_storage,gce,gce_lb,gce_net,gce_pd,gem,get_url,getent,git,github_hooks,glance_image,group,group_by,grove,hg,hipchat,homebrew,homebrew_cask,homebrew_tap,hostname,htpasswd,include_vars,ini_file,irc,jabber,jboss,jira,kernel_blacklist,keystone_user,layman,librato_annotation,lineinfile,linode,lldp,locale_gen,logentries,lvg,lvol,macports,mail,modprobe,mongodb_user,monit,mount,mqtt,mysql_db,mysql_replication,mysql_user,mysql_variables,nagios,netscaler,newrelic_deployment,nexmo,nova_compute,nova_keypair,npm,ohai,open_iscsi,openbsd_pkg,openvswitch_bridge,openvswitch_port,opkg,osx_say,ovirt,pacman,pagerduty,pause,ping,pingdom,pip,pkgin,pkgng,pkgutil,portage,portinstall,postgresql_db,postgresql_privs,postgresql_user,quantum_floating_ip,quantum_floating_ip_associate,quantum_network,quantum_router,quantum_router_gateway,quantum_router_interface,quantum_subnet,rabbitmq_parameter,rabbitmq_plugin,rabbitmq_policy,rabbitmq_user,rabbitmq_vhost,raw,rax,rax_cbs,rax_cbs_attachments,rax_cdb,rax_cdb_database,rax_cdb_user,rax_clb,rax_clb_nodes,rax_dns,rax_dns_record,rax_facts,rax_files,rax_files_objects,rax_identity,rax_keypair,rax_meta,rax_network,rax_queue,rax_scaling_group,rax_scaling_policy,rds,rds_param_group,rds_subnet_group,redhat_subscription,redis,replace,rhn_channel,rhn_register,riak,rollbar_deployment,route53,rpm_key,s3,script,seboolean,selinux,service,set_fact,setup,shell,slack,slurp,sns,stackdriver,stat,subversion,supervisorctl,svr4pkg,swdepot,synchronize,sysctl,template,twilio,typetalk,ufw,unarchive,uri,urpmi,user,virt,vsphere_guest,wait_for,win_feature,win_get_url,win_group,win_msi,win_ping,win_service,win_stat,win_user,xattr,yum,zabbix_maintenance,zfs,zypper,zypper_repository
Filters: Jinja2’ism
● best way to change data
● chainable pipes
● simple to expand
● can hide complexity
__________________________< [hello, world]|join(‘ ‘) > -------------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||
Example hack - filter
import random
def randomize_list(mylist): random.shuffle(mylist) return mylist
...
def filters(self): # this maps filter names to functionsreturn {
‘shuffle’: randomize_list,
...
}
lib/ansible/runner/filter_plugins/core.py
Example hack - filter- hosts: all connection: local gather_facts: false vars: - mylist: [1,2,3,4,5] tasks: - debug: msg=”{{item}}” with_items: “{{mylist|shuffle}}”
Example hack - filter __________________________ < TASK: debug msg={{item}} > --------------------------
ok: [localhost] => (item=1) => { "item": 1, "msg": "1"}ok: [localhost] => (item=3) => { "item": 3, "msg": "3"}ok: [localhost] => (item=4) => { "item": 4, "msg": "4"}ok: [localhost] => (item=5) => { "item": 5, "msg": "5"}ok: [localhost] => (item=2) => { "item": 2, "msg": "2"}
__________________________ < TASK: debug msg={{item}} > --------------------------
ok: [localhost] => (item=3) => { "item": 3, "msg": "3"}ok: [localhost] => (item=2) => { "item": 2, "msg": "2"}ok: [localhost] => (item=5) => { "item": 5, "msg": "5"}ok: [localhost] => (item=2) => { "item": 2, "msg": "2"}ok: [localhost] => (item=1) => { "item": 1, "msg": "1"}
__________________________ < TASK: debug msg={{item}} > --------------------------
ok: [localhost] => (item=2) => { "item": 2, "msg": "2"}ok: [localhost] => (item=4) => { "item": 4, "msg": "4"}ok: [localhost] => (item=1) => { "item": 1, "msg": "1"}ok: [localhost] => (item=3) => { "item": 3, "msg": "3"}ok: [localhost] => (item=5) => { "item": 5, "msg": "5"}
#>ansible-playbook test.yml -i ‘localhost,’
Lookups
● you are already using them: with_<lookup>
● they execute on the “master”
● a way to access external files and/or data
● normally returns a list
Example hack - lookup
class LookupModule(object):
def __init__(self, basedir=None, **kwargs): self.basedir = basedir self.etcd = etcd() # initializes etcd class
def run(self, terms, inject=None, **kwargs): terms = utils.listify_lookup_plugin_terms(terms, self.basedir, inject) if isinstance(terms, basestring): terms = [ terms ]
ret = [] for term in terms: key = term.split()[0] value = self.etcd.get(key) # gets the data from etcd class ret.append(value) return ret
lib/ansible/runner/lookup_plugins/etcd.py
Example hack - lookup
class etcd(): # simplified version
def __init__(self, url=ANSIBLE_ETCD_URL): self.url = url self.baseurl = '%s/v1/keys' % (self.url)
def get(self, key): url = "%s/%s" % (self.baseurl, key) r = urllib2.urlopen(url) data = r.read()
item = json.loads(data) value = item['value']
return value
lib/ansible/runner/lookup_plugins/etcd.py
_______________/ written by \\ Jan-Piet Mens / --------------- \ ,__, \ (oo)____ (__) )\ ||--|| *
Example hack - lookup
#>ansible all -m debug -a ‘msg={{lookup(“etcd”,”key1”)}}’ -i ‘localhost,’
localhost | success >> { "msg": "value1"}
#>ansible all -m debug -a ‘msg={{lookup(“etcd”,”key2”)}}’ -i ‘localhost,’localhost | success >> { "msg": "value2"}
Example hack - callback
class CallbackModule(object): # simplified example
def playbook_on_task_start(self, name, is_conditional):
if self.current is not None: # Record the total time of the previous self.stats[self.current] = time.time() - self.stats[self.current]
self.current = name # Record the start time of the current task self.stats[self.current] = time.time()
def playbook_on_stats(self, stats): # Sort the tasks by their running time
... results = sorted(self.stats.items(), key=lambda value: value[1], reverse=True)
for name, elapsed in results: # Print the timings print "{0:-<70}{1:->9}".format('{0} '.format(name),'{0:.02f}s'.format(elapsed))
callback_plugins/profile_tasks.py
_______________/ written by \\ Jharrod LaFon / --------------- \ ,__, \ (oo)____ (__) )\ ||--|| *
Example hack - callback
# re-running the previous filter/shuffle playbook
__________________________ < TASK: debug msg={{item}} > --------------------------
ok: [localhost] => (item=2) => { "item": 2, "msg": "2"}
...
ok: [localhost] => (item=4) => { "item": 4, "msg": "4"}
PLAY RECAP ******************************************************************** debug msg={{item}} ------------------------------------------------------ 0.02slocalhost : ok=1 changed=0 unreachable=0 failed=0
#>ansible-playbook test.yml -i ‘localhost,’
Example hack - cacheimport exceptions
class BaseCacheModule(object):
def get(self, key): raise exceptions.NotImplementedError
def set(self, key, value): raise exceptions.NotImplementedError
def keys(self): raise exceptions.NotImplementedError
def contains(self, key): raise exceptions.NotImplementedError
def delete(self, key): raise exceptions.NotImplementedError
def flush(self): raise exceptions.NotImplementedError
def copy(self): raise exceptions.NotImplementedError
________________ / I abandoned it \| and Josh Drake |\ revived it / ---------------- \ /\ ___ /\ \ // \/ \/ \\ (( O O )) \\ / \ // \/ | | \/ | | | | | | | | | o | | | | | |m| |m|
lib/ansible/cache/base.py
Example hack - cache
from ansible.cache.base import BaseCacheModule
class CacheModule(BaseCacheModule):
def __init__(self, *args, **kwargs): self._cache = {}
def get(self, key): return self._cache.get(key)
def set(self, key, value): self._cache[key] = value
def keys(self): return self._cache.keys()
def contains(self, key): return key in self._cache
def delete(self, key): del self._cache[key]
def flush(self): self._cache = {}
def copy(self): return self._cache.copy()
lib/ansible/cache/memory.py
Example hack - cache
from redis import StrictRedis
class CacheModule(BaseCacheModule): # simplified version
self._cache = StrictRedis(*connection)...
def get(self, key): value = self._cache.get(self._make_key(key)) # make_key creates correct prefix # guard against the key not being removed from the zset if value is None: self.delete(key) raise KeyError return json.loads(value)
def set(self, key, value): value2 = json.dumps(value) if self._timeout > 0: # a timeout of 0 is handled as meaning 'never expire' self._cache.setex(self._make_key(key), int(self._timeout), value2) else: self._cache.set(self._make_key(key), value2)
self._cache.zadd(self._keys_set, time.time(), key)...
lib/ansible/cache/redis.py
Tests: test/
● sadly the presenter sucks at test coverage
● no really, tests are good, regressions!
● assert: the test module
● destructive, non destructive, integration
● ‘make tests’, runs unit tests
Before you submit
● update documentation (it is also in the repo)
● create tests, assert! (also in the repo)
● prepare a clear usage/example to post in the PR
● its a dialog, be ready to make your ‘use case’
● patience … … … …
● not everything belongs in core
top related