Automating DB Ops withAnsible, Chef, and Puppet

Tyler Duzan, Product Manager

Percona

2

Who Am I?

• My name is Tyler Duzan

• Formerly an operations engineer for more than 12 years focused on security and automation

• Now a Product Manager at Percona

• Have used Puppet, Chef, Ansible, Saltstack, and Terraform professionally in the past

Basics of Automation

4

Why We Automate

• Provides a source of actionable and testable documentation of processes

• Reduces likelihood of human error

• Expands the scale and capability of human engineers

• Reduces time to deploy and time to recover

• Saves money in the long term

• Improves compliance with organizational policies

• Reproducibility of environments

5

When Should You Consider Automating a Task?

• Repetition• Task needs to be done many times exactly the same way or with minimal variation

• Any variation can be templated, or follows a pattern

• Good rule: If you’ve manually done something 3 times, after the 3rd time you should consider automating that task

• Compliance• Task needs to be done exactly a certain way to comply with organizational policies

• Part of a process pipeline where future tasks depend on the way this task is completed

• Error Reduction• Is critical to production environments

6

Some DB Tasks Worth Automating

• Initial Server / Cluster Install

• User / Role Management

• Backups

• Restore

• Compliance Tasks

• Server Configuration Changes

7

Security and Compliance with Automation

• Automation simplifies deploying centralization within your infrastructure• Centralized logging and log shipping

• Centralized user management and single-sign on for servers

• Can enforce approved configuration which is policy-compliant

• Provides a way to version configuration and provide controls

• Change Management processes can integrate easily with automation

• Handle complex/error-prone security configuration, such as custom SELinux policies

8

Types of Automation Tools

• Infrastructure Automation• Cloud-Focused

- Terraform

- Cloud Formation

- SaltCloud

• Physical-Focused

- Foreman

- Open Crowbar

- Cobbler

• Container-Focused

- Kubernetes Operators

- DC/OS Data Frameworks

9

Types of Automation Tools

• Configuration Management• Agent-Based

- Chef

- Puppet

- SaltStack

• Remote Execution

- Ansible

- Capistrano

- Fabric

10

Types of Automation Tools

• Automation Language Choice• Domain Specific Language (DSL)

- Terraform

- Cloud Formation

- Puppet

- Ansible

• General Purpose Programming Language

- Chef

- SaltStack

- Capistrano

- Fabric

11

What Tool is Right for You or Your Team?

• Importance of tooling• Linters, pipeline intregration, CI, how do you test your automation

• Testability

• Language Familiarity

• Developer Focused vs Operations Focused

• Team Experience with writing automation

Tool Overview and Examples

13

Ansible

• Agentless, remote execution, DSL-based automation

• Uses SSH to execute playbooks against nodes

• Actions are executed from a controller which maintains an inventory of playbooks, modules, and nodes

• Ansible playbooks are effectively like shell scripts. They run in sequential order, just like a shell script, and can take any action that could be otherwise run on the target node via ssh remote commands.

• Playbooks can be idempotent, but are not guaranteed to be idempotent

• Ansible and its modules are written in Python

14

Ansible: Setup MySQL Slave Example

---

- hosts: "{{ slave }}"

tasks:

- name: configure MySQL slave process

mysql_replication:

master_host: "{{ mysql_replication_master | default(master) }}"

master_user: "{{ mysql_replication_user }}"

master_password: "{{ mysql_replication_password }}"

master_log_file: "{{ binlog_file.stdout }}"

master_log_pos: "{{ binlog_position.stdout }}"

mode: changemaster

- name: start MySQL slave process

mysql_replication:

mode: startslave

Adapted from https://github.com/ensibel/setup-mysql-slave/blob/master/main.yml

To execute the playbook, assuming you have defined mysql_replication_user and mysql_replication_password in your host_vars file you can run just the following:

ansible-playbook main.yml -e

'master=master.example.com

slave=slave.example.com'

15

Ansible: MySQL User Example

---

- hosts: all

remote_user: root

tasks:

- name: Create MySQL Database User

mysql_user: user=example password=12345 priv=*.*:ALL state=present

- name: Ensure no user named "example2" exists and delete if it does

mysql_user: user=example2 state=absent

Adapted from official Ansible examples

To execute this playbook, you can run the following:

ansible-playbook -i example

user.yml

16

Chef

• Agent-based, local execution, pure Ruby automation

• Uses a local agent on each node to execute a collection of cookbookscontaining recipes

• Actions are defined in a series of cookbooks which are typically structured as Ruby gems which can be installed and managed using normal Ruby processes and tools, such as Berkshelf or Bundler

• Cookbooks contain a collection of recipes, attributes, resources, and templates, can include libraries, can depend upon Ruby gems, and also can optionally include tests

• Cookbooks are always idempotent and declarative, agents rerun cookbooks regularly to ensure that a given node matches its declared environment configuration

• Chef is written in Ruby and Erlang, Chef cookbooks are written in Ruby

17

Chef: How the Chef-Client Works

1. Register and authenticate the node with the Chef server

2. Build the node object

3. Synchronize cookbooks and the node run list

4. Compile the resources and execute the first pass (‘compile’)

5. Converge and execute the second pass (‘converge’)

6. Handle any exceptions and report the status to the Chef server

18

Chef: Setup MySQL Server Example

# Depends on `mysql` community cookbook

yum_repository 'mysql57-community' do

mirrorlist

'http://repo.mysql.com/yum/mysql-5.7-

community/el/$releasever/$basearch/'

description 'MySQL 5.7 Community Edition'

enabled true

gpgcheck true

end

Adapted from example in the documentation of the ‘mysql’ community cookbook

mysql_service 'default' do

version '5.7'

bind_address '0.0.0.0'

port '3306'

data_dir '/var/lib/mysql'

initial_root_password

'Ch4ng3me'

action [:create, :start]

end

19

Chef: MySQL User Example

chef_gem 'mysql-grantee'

# Retrieve mysql application

data bag items

mysql_apps =

search(node['mysql']['users']['d

ata_bag'])

# Apply permissions

ruby_block 'Configure MySQL

Users' do

block {

mysql_apply_grants(self,

mysql_apps) }

action :run

end

Adapted from old Chef cookbook

20

Puppet

• Agent-based, local execution, DSL-based automation

• Uses a local agent on each node to execute a collection of manifestscontaining declarative configuration

• Actions are defined in a series of manifest, which are structured as a collection of resource abstractions defined in Puppet’s DSL

• Manifests are idempotent and declarative and abstracted away from platform specifics. Agents apply manifests in a transactional way.

• Agents run as a daemon and regularly reapply manifests to ensure the node matches the declared environment configuration

• Puppet is written in C++, Clojure, and Ruby

21

Puppet: Agent Transaction Details

1. An agent send facts from Facter to the master.

2. Puppet builds a graph of the list of resources and their interdependencies, representing the order in which they need to be configured, for every client. The master sends the appropriate catalog to each agent node.

3. The actual state of the system is then configured according to the desired state described in manifest file. If the system is already in the desired state, Puppet will not make any changes, making transactions idempotent.

4. Finally, the agent sends a report to the master, detailing what changes were made and any errors that occurred.

22

Puppet: Setup Percona Server Example

include '::mysql::server'

yumrepo { 'percona':

descr => 'CentOS $releasever -

Percona',

baseurl =>

'http://repo.percona.com/centos/$relea

sever/os/$basearch/',

gpgkey =>

'http://www.percona.com/downloads/perc

ona-release/RPM-GPG-KEY-percona',

enabled => 1,

gpgcheck => 1,

}

$override_options = {

'mysqld' => {

'data_dir' => '/var/lib/mysql',

'bind_address' => '0.0.0.0',

'port' => '3306'

}

}

Adapted from official MySQL Puppet module

23

Puppet: Setup Percona Server Example

class { '::mysql::server':

package_name => 'Percona-Server-server-57',

package_ensure => '5.7.23-23.1.el7',

service_name => 'mysql',

config_file => '/etc/my.cnf',

root_password => 'Ch4ng3me',

remove_default_accounts => true,

override_options => $override_options

}

24

Terraform

• Has its own specific DSL called Terraform HCL, which is used to create an execution plan

• Interacts with provider APIs to define infrastructure as code

• Terraform configuration is idempotent and declarative

• Region abstracted to allow easily reproducible infrastructure

• Terraform configuration consists of providers, modules, resources, variables, and attributes.

• Terraform is written in Go

25

Terraform: Deploy AWS RDS and Aurora

# AWS provider

provider "aws" {

region = "us-west-2"

}

data "aws_db_instance" "database"

{

db_instance_identifier = "my-

test-database"

}

resource "aws_rds_cluster" "default" {

cluster_identifier = "my-test-

aurora"

engine = "aurora-mysql"

availability_zones = ["us-west-2a",

"us-west-2b", "us-west-2c"]

database_name = "testdb"

master_username = "testuser"

master_password = "Ch4ng3me"

backup_retention_period = 5

preferred_backup_window = "07:00-

09:00"

}

Testing Your Automation

27

Kitchen CI

• Provides an easy way to automate local or remote integration testing of your automation code

• Supports drivers for docker, vagrant, aws, and other methods for deploying test systems

• Uses chef-zero to compile and converge your cookbooks on the local VMs

• Provides a method to do TDD with your automation

• Uses ServerSpec + Ruby for running tests

• Kitchen configuration is declared as YAML

• Ships with ChefDK, can be extended to support Ansible and Puppet

28

Kitchen CI: Test PostgreSQL Example

---

driver:

name: vagrant

provisioner:

name: chef_zero

platforms:

- name: ubuntu-14.04

- name: windows-2012r2

suites:

- name: client

run_list:

- recipe[postgresql::client]

- name: server

run_list:

- recipe[postgresql::server]

Adapted from the Kitchen CI Website Example

29

ServerSpec & InSpec

• Extends RSpec test framework to support integration testing of infrastructure

• Provides resources which map to physical infrastructure attributes

• Tests are written in simplified English DSL which defines what the expected outcome of running some automation is, and ServerSpecchecks to see if it matches those expectations

• Chef has created an extended version of ServerSpec with many improvements called InSpec that is to migrate to if you choose.

• Both are included/integrated with Kitchen CI

30

ServerSpec: Example Test for MySQL

require 'spec_helper'

%w(user_deleted1 user_deleted2 app_one_rw1 app_one_su1).each do |user|

describe command("mysql -u root -e \"select * from mysql.user where User like '#{user}'\"") do

its(:stdout) { should eq('') }

its(:exit_status) { should eq(0) }

end

end

describe file('/etc/my.cnf.d') do

it { should be_directory }

it { should be_owned_by('root') }

it { should be_grouped_into('root') }

it { should be_mode('755') }

end

Questions?

32

Thank You Sponsors!!

33

Rate My Session