automating mysql operations with puppet - percona · automating mysql operations with puppet ......

51
Automating MySQL operations with Puppet Kris Buytaert & Julien Pivotto @krisbuytaert @roidelapluie

Upload: trinhnguyet

Post on 28-Jul-2018

230 views

Category:

Documents


0 download

TRANSCRIPT

Automating MySQL operations with Puppet

Kris Buytaert & Julien Pivotto

@krisbuytaert @roidelapluie

Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source

Consultant @Consultant @inuits.euinuits.eu● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Some books, some papers, some blogsSome books, some papers, some blogs● Evangelizing devopsEvangelizing devops● Organiser of #devopsdays, #cfgmgmtcamp, Organiser of #devopsdays, #cfgmgmtcamp,

#loadays, ….#loadays, ….● Part of the travelling geek circusPart of the travelling geek circus

Julien PivottoJulien Pivotto

● I am a sysadminI am a sysadmin● I develop sometimesI develop sometimes● Chief Traveling Officer and Open Source Chief Traveling Officer and Open Source

Consultant @Consultant @inuits.euinuits.eu● Born almost at cloud age :)Born almost at cloud age :)● Monitoring, automation, databases, all mixedMonitoring, automation, databases, all mixed

What's this devops What's this devops thing anyhow ? thing anyhow ?

C(L)AMSC(L)AMS● CultureCulture

● (Lean)(Lean)

● AutomationAutomation

● MeasurementMeasurement

● SharingSharing

Damon Edwards and John WillisDamon Edwards and John Willis

Gene KimGene Kim

A global movement to improve the quality of A global movement to improve the quality of software delivery leveraging Open Source software delivery leveraging Open Source

experience, started in Gent in 2009experience, started in Gent in 2009

Because the old days: Because the old days: ● ““Put this Code Live, here's a tarball” NOW! Put this Code Live, here's a tarball” NOW!

● What dependencies ?What dependencies ?

● No machines available ?No machines available ?

● What database ?What database ?

● Security ?Security ?

● High Availability ? High Availability ?

● Scalability ?Scalability ?

● My computer can't install this ?My computer can't install this ?

10 days into operations10 days into operations

DevSalesDBASecQAMarkDevSalesDBASecQAMarketingMonitoringBusOpsetingMonitoringBusOps

● Nope DevOps didn’t forget anyoneNope DevOps didn’t forget anyone

● Actually DevOps does not make sense without Actually DevOps does not make sense without everyoneeveryone

● Also devops != everyone is dba/tester/dev/op...Also devops != everyone is dba/tester/dev/op...

● devops != a job titledevops != a job title

● devops != a team namedevops != a team name

Why automate ?Why automate ?

Common ProblemsCommon Problems● Many manual changes to systems● Many undocumented changes ● Emergency Administration only● Disaster Recovery site is a Disaster● Time to deliver a box is to slow● All boxen are different● Computers don’t work hard enough for us

More ProblemsMore Problems● How long does it take to reinstall a machine from 0● To the exact same point as before ?● With different Hardware ? In a different cloud ?● What about your (customer/personal data )● Did I just redirect prod frontend to dev backend?

Automation is not just Automation is not just about scaleabout scale

● ““I don’t automate coz I just have X servers” is I don’t automate coz I just have X servers” is no excuseno excuse

● You don’t need zillions of apps/servers to You don’t need zillions of apps/servers to benefit from automationbenefit from automation

● It starts from one serverIt starts from one server

● Especially in the database world where we look Especially in the database world where we look for performance, fine tuning, and consistencyfor performance, fine tuning, and consistency

Security ?Security ?● Monitoring that your platform hasn't changed.Monitoring that your platform hasn't changed.

• Why is selinux disabled ?Why is selinux disabled ?

• Who added / dropped that firewall ?Who added / dropped that firewall ?

• What did this originally look like ?What did this originally look like ?

• Is this file really what Bernd meant it to be ?Is this file really what Bernd meant it to be ?

#monitoringsucks#monitoringsucks● Monitoring is out of sync with realityMonitoring is out of sync with reality

● Managed manuallyManaged manually

● Can't keep up.. Can't keep up..

ChallengesChallenges● ReproducabilityReproducability

● SpeedSpeed

● AuditingAuditing

● Keeping stuff in sync Keeping stuff in sync

• MonitoringMonitoring

• SecuritySecurity

• BackupBackup

Facts!Facts!● Data Backup is only a part● Sysadmin backup needs to be done also● Manual Installations = bad● Bad installations = unusable

infrastructure● Bad installations = unproductive users● Bad installations = manual efforts● Manual efforts = no time● No time = no updates no patches no

security● Manual work = high costs● No security + high costs = Bancrupcy

The 10The 10thth floor test floor test

● Grab a random machine (don’t take a backup before)

● Throw it out a 10th floor window● Can you recover it in 10 minutes ?

Imagesprawl AND Imagesprawl AND SnowflakesSnowflakes

● Image Sprawl :Image Sprawl :

• Copy vm 3xCopy vm 3x

• Modify 2x Modify 2x

• Copy 21x Copy 21x

• How the Heck did we get here ?How the Heck did we get here ?

● SnowFlakes :SnowFlakes :

• Don't touch this box it might breakDon't touch this box it might break

• Look how nice it is !Look how nice it is !

You never deploy You never deploy something “just” oncesomething “just” once

● Local test … experiment, Local test … experiment,

• Vagrant box / local containersVagrant box / local containers

● Integration PlatformIntegration Platform

• Same codebase,, different environmentSame codebase,, different environment

● Dev/ UAT/ Prod / DR … Dev/ UAT/ Prod / DR …

● Or your customer just forgot to renew the lease Or your customer just forgot to renew the lease on his VPS. #toldyousoon his VPS. #toldyouso

What's different in the cloud ?What's different in the cloud ?

● ScaleScale

● VelocityVelocity

● ChangeChange

Your machines as CattleYour machines as Cattle

Treat your people as petsTreat your people as pets

Configuration MgmtConfiguration Mgmt● Configure 1000 nodes,Configure 1000 nodes,

● Modify 15000 files, Modify 15000 files,

● Think : Think :

•Cfengine,Puppet, Chef, SaltCfengine,Puppet, Chef, Salt

● Put configs under version controlPut configs under version control

● Please don't roll your own ... Please don't roll your own ...

Infrastructure as CodeInfrastructure as Code● Treat configuration automation as code Treat configuration automation as code

● Development best practicesDevelopment best practices

• Model your infrastructureModel your infrastructure

• Version your cookbooks / manifestsVersion your cookbooks / manifests

• Test your cookbooks/ manifestsTest your cookbooks/ manifests

• Dev/ test /uat / prod for your infraDev/ test /uat / prod for your infra

● Model your infrastructureModel your infrastructure

● A working service = automated ( Application Code + A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )Infrastructure Code + Security + Monitoring )

● IAC -ne scripting (or translating bash to yaml)IAC -ne scripting (or translating bash to yaml)

AuditabilityAuditability● git loggit log

● git blamegit blame

● Review, Review,

● authorizationauthorization

File monitoringFile monitoring

Other benefits of codeOther benefits of code

● Code can be testedCode can be tested

● You can attach it to a bugtrackerYou can attach it to a bugtracker

● You can deploy, redeploy, …You can deploy, redeploy, …

● You can analyse it without getting access to You can analyse it without getting access to the infrathe infra

● You can revert changesYou can revert changes

PuppetPuppet● Open Source Config Management Open Source Config Management ToolTool Ecosystem Ecosystem

● 10y old ; very mature10y old ; very mature

● Ruby, Clojure, Jruby, C++, ...Ruby, Clojure, Jruby, C++, ...

● Large communityLarge community

● Lots of open source modules (cose shared by Lots of open source modules (cose shared by other people to automate things)other people to automate things)

● Puppet is also the name of the company behind itPuppet is also the name of the company behind it

Puppet in shortPuppet in short● You define the desired state of your infraYou define the desired state of your infra

● Puppet looks at the current state and takes Puppet looks at the current state and takes actions to get to the final stateactions to get to the final state

● You have the report of what has been You have the report of what has been changed/errored/…changed/errored/…

● It is declarativeIt is declarative

How Puppet WorksHow Puppet Works● Puppet works by default on a server/agent Puppet works by default on a server/agent

modelmodel

● Agents fetch their “catalog” (= final desired Agents fetch their “catalog” (= final desired state) from the master and apply that locallystate) from the master and apply that locally

● Then they report back to the serverThen they report back to the server

● The catalog is computed depending on “facts” The catalog is computed depending on “facts” the machine sends to the server, user input the machine sends to the server, user input (parameters) and modules (source code)(parameters) and modules (source code)

Puppet ModulesPuppet Modules● Puppet modules are reusable set of Puppet Puppet modules are reusable set of Puppet

manifestsmanifests

● Contains resources, templates, files, …Contains resources, templates, files, …

● e.g. Puppet-apache, puppet-postgresql, puppet-e.g. Puppet-apache, puppet-postgresql, puppet-corosynccorosync

● Thousands of them are published on github and Thousands of them are published on github and Puppet ForgePuppet Forge

Roles and ProfilesRoles and Profiles● A profile is a puppet module that explains how A profile is a puppet module that explains how

to install a serviceto install a service

● A role is a Puppet manifest that includes A role is a Puppet manifest that includes profilesprofiles

● A server should get one and only one roleA server should get one and only one role

class profile_mysql’ { class { '::mysql::server': } class { '::mysql::xtrabackup': }}

class role::mysql_server { include profile_mysql include profile_base}

Defining a ServiceDefining a Service● profile that :profile that :

• Configures service using a standard Configures service using a standard module call with hiera based parametersmodule call with hiera based parameters

• Configures BackupConfigures Backup

• Configures logrotation Configures logrotation

• Configures logshipping Configures logshipping

• Exports Monitoring NeedsExports Monitoring Needs

MySQL+PuppetMySQL+Puppet● There used to be:There used to be:

• Puppet-perconaPuppet-percona

• Puppet-mariadbPuppet-mariadb

• Puppetlabs-mysqlPuppetlabs-mysql

● Now community has been built around Now community has been built around puppetlabs-mysql (4 millions download on puppetlabs-mysql (4 millions download on puppet forge – 215 contributors)puppet forge – 215 contributors)

● Same story with other pieces of software Same story with other pieces of software (apache, ntp, postgresql, iptables...)(apache, ntp, postgresql, iptables...)

What is in the module?What is in the module?● Mysql server configMysql server config

● Mysql db creationMysql db creation

● Mysql users creationMysql users creation

● Root passwordRoot password

● Mysql bindings (c, java, perl, php…)Mysql bindings (c, java, perl, php…)

● BackupsBackups

Setting up mysqlSetting up mysql(or mariadb depending on distro default)(or mariadb depending on distro default)

class { '::mysql::server': }

You probably want thisYou probably want this

class { '::mysql::server': service_manage => false,}

And let your HA service taking care of that

OptionsOptionsclass { '::mysql::server': override_options => { 'mysqld' => { 'server-id' => $::mysql_server_id, 'bind-address' => '0.0.0.0', 'log-error' => '/var/log/mysqld.log', 'pid-file' => '/var/run/mysqld/mysqld.pid', 'innodb_file_per_table' => true, }, },}

UsersUsers mysql_user { "[email protected]": ensure => 'present', password_hash => mysql_password(‘secret’), } mysql_grant { "[email protected]/*.*": ensure => 'present', privileges => [ 'ALL' ], table => '*.*', user => "[email protected]", }

But its Puppet like 2011But its Puppet like 2011● Nowadays in infra as code you split data and Nowadays in infra as code you split data and

codecode

● Hiera : yaml as source of dataHiera : yaml as source of data

● Other backends than yaml availableOther backends than yaml available

• VaultVault

● Merges data from env-specific, os-specific, Merges data from env-specific, os-specific, datacenter-specific key value stores...datacenter-specific key value stores...

---mysql::server::db: "dev": user: "dev" password: "devpass" host: "127.0.0.1" grant: - "ALL"

create_resources(mysql::db, hiera('mysql::server::db', {}))

In Puppet:

In Hiera:

Exported resourcesExported resources

Exported resourcesExported resources● You can export resources from nodesYou can export resources from nodes

● And fetch them in another nodeAnd fetch them in another node

● Exporting:Exporting:

● Collecting:Collecting: Mysql::Db <<| tag == “$tag” |>>

@@::mysql::db { $name: user => $user, password => $password, dbname => $dbname, host => $host, grant => $grant, tag => $tag, }

Downsides & Downsides & WorkaroundsWorkarounds

● Exported resources are “slow”, puppet needs to Exported resources are “slow”, puppet needs to run on the app server to export resources to run on the app server to export resources to the mysql serverthe mysql server

● Requires a puppetserver (with puppetdb)Requires a puppetserver (with puppetdb)

● Workaround 1: Accept that and make your app Workaround 1: Accept that and make your app resilient to DB not readyresilient to DB not ready

● Workaround 2: Design your app in hiera as Workaround 2: Design your app in hiera as yaml and pick the info from there in both app yaml and pick the info from there in both app server and database serverserver and database server

Replication userReplication user

mysql_user { 'replica@%':ensure => 'present',password_hash => mysql_password('percona17'),

}

mysql_grant { 'replica@%/*.*':ensure => 'present',privileges => ['REPLICATION SLAVE'],table => '*.*',user => 'slave_user@%',

}

Replication OptionsReplication Optionsclass { '::mysql::server': override_options => { 'mysqld' => { 'server-id' => 1, 'binlog-ignore-db' => [‘mysql’, ‘performance_schema’], 'log-bin' => 'binary-log.', 'binlog-format' => 'ROW', }, },}

CHANGE MASTER TO master_host="$mysql_master", master_port=3306, master_user='replica', master_password='percona17', master_use_gtid=current_pos;START SLAVE;

Chose if you want to runin a Puppet exec, a more

Intelligent script, …(MariaDB based example:)

Configuring HAConfiguring HA● Example using corosync/pacemaker (puppet-Example using corosync/pacemaker (puppet-

corosync)corosync)cs_primitive { ‘p_mysql’: primitive_class => 'ocf', primitive_type => 'mysql', provided_by => 'heartbeat', Promotable => true, operations => [ { 'monitor' => { 'interval' => '10s', 'timeout' => '30s' } }, { 'monitor' => { 'interval' => '5s', 'timeout' => '30s', 'role' => 'Master', } }, ]}

cs_colocation { 'mysql_master': Primitives => [ 'mysql_vip', 'p_mysql:Master' ],}

cs_primitive { 'mysql_vip': primitive_class => 'ocf', primitive_type => 'IPaddr2', provided_by => 'heartbeat', Parameters => { 'ip' => '172.21.17.100', 'cidr_netmask' => '24' }, Operations => { 'monitor' => { 'interval' => '10s' } },}

ConclusionConclusion● IAC solves a lot of problems IAC solves a lot of problems

• Improves SecurityImproves Security

• Creates Monitoring LoveCreates Monitoring Love

• Creates SpeedCreates Speed

• Creates ConsistencyCreates Consistency

● But it still is code, and needs to be treated like But it still is code, and needs to be treated like code !code !

More of this More of this

@cfgmgmtcamp@cfgmgmtcamp

Gent , 5 & 6 february Gent , 5 & 6 february

Right after FosdemRight after Fosdem

ContactContactKris Buytaert [email protected] Buytaert [email protected] Pivotto [email protected] Pivotto [email protected]

Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/https://roidelapluie.behttps://roidelapluie.behttps://inuits.eu/https://inuits.eu/

InuitsInuits

Essensteenweg 31Essensteenweg 31BrasschaatBrasschaatBelgiumBelgium891.514.231891.514.231

+32 475 961221+32 475 961221