replication features, technologies and 3rd party extinction

Percona Live London MySQL Conference, 2012

Replication features, technologies and 3rd party Extinction.

Ben MildrenMySQL Team Technical [email protected]

Andrew MooreMySQL [email protected]

mailto:[email protected]

mailto:[email protected]

© 2012 – Pythian

Why Pythian?• Recognized Leader:

• Global industry leader in data infrastructure managed services and consulting with expertise in Oracle, Oracle Applications, Microsoft SQL Server, MySQL, big data and systems administration

• Work with over 200 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments

• Expertise:• One of the world’s largest concentrations of dedicated, full-time DBA

expertise. Employ 9 Oracle ACEs/ACE Directors

• Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC

• Global Reach & Scalability:• 24/7/365 global remote support for DBA and consulting, systems

administration, special projects or emergency response

© 2012 – Pythian

What do we cover in this presentation?

1) MySQL Replication Overview

2) MySQL Replication Topologies

3) MySQL Replication Ecosystem

4) MySQL Replication Tools

5) MySQL Replication in 5.6

6) MySQL Replication Tools in 5.6

MySQL ReplicationOverview

© 2012 – Pythian

What is replication?• MySQL Replication is a means of propagating database

changes from one server to another.• Master / Slave architecture• MySQL Replication is easy to setup and can be used as the

basis of a number of different functions:

Scale out

BackupsFailover

TestingReporting

© 2012 – Pythian

Native Replication Overview

© 2012 – Pythian

Binary Log• The binary log was introduced in version 3.23.14, and records

events to capture changes made to the database.• DDL, DCL, and DML – includes transaction grouping.• Not equivalent to a redo log.• Not just used for Replication, but also for Point in Time

Recovery. It's also a good source of information to use for auditing and troubleshooting.

• Can be examined using the mysqlbinlog tool or SHOW BINLOG EVENTS; MySQL statement.

© 2012 – Pythian

Binary Log

© 2012 – Pythian

Binary LogStatement based logging

• Captures just the SQL statements, means logging is lightweight and binlog storage requirements are minimized.

• Non-deterministic statements can mean it's prone to data drift.Row based logging

• Introduced in 5.1• More verbose, as a result can mean larger binary logs, but also

less data drift.Mixed format logging

• By default uses statement based logging, but switches to row based logging for unsafe statements.

© 2012 – Pythian

Binary LogBinary log – Format_desc

[ec2 mysql]$ sudo mysqlbinlog mysqld-bin.000001/*!40019 SET @@session.max_insert_delayed_threads=0*/;/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;DELIMITER /*!*/;# at 4#121112 3:05:24 server id 6 end_log_pos 106 Start: binlog v 4, server v 5.1.61-log created 121112 3:05:24BINLOG 'xK2gUA8GAAAAZgAAAGoAAAAAAAQANS4xLjYxLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEzgNAAgAEgAEBAQEEgAAUwAEGggAAAAICAgC'/*!*/;

• Binlog v4 introduced MySQL 4.1

(v1 – 3.23, v3 – 4.02 - 4.1, v2 – redundant)

• Server version

• end_log_pos: 98 (5.0), 106 (5.1), 107 (5.5), 120 (5.6)

© 2012 – Pythian

Binary LogBinary log - Statement based logging

[ec2 mysql]$ sudo mysqlbinlog mysqld-bin.000001...# at 106...# at 127924#121112 3:50:03 server id 6 end_log_pos 128160 Query thread_id=18112805 exec_time=0 error_code=0SET TIMESTAMP=1352710203/*!*/;UPDATE t1 SET c2 = 'ben' WHERE c1 = '1'/*!*/;...

© 2012 – Pythian

Binary LogBinary log - Row based logging

[ec2 mysql]$ sudo mysqlbinlog -vvv mysqld-bin.000001# at 2609282#121129 0:24:19 server id 6 end_log_pos 2609282 Table_map: `db1`.`t1` mapped to number 9057#121129 0:24:19 server id 6 end_log_pos 2609389 Write_rows: table id 9057 flags: STMT_END_F

BINLOG 'g/GUBMGAAAAQAAEjAAAAAAEABWAAPYTHIANROCKSAAAMVyX3Nlc3Npb24ABwMHB/4DDw8G/kD/ADIAdA==g/GUBcGAAEAB//kn2U5AIPxtlBAMmM0MTYyYWE2MmQxMTAMAMAMAI4OTdhMxMWMxNHciBwA='/*!*/;### INSERT INTO db1.t1### SET### @1=2 /* INT meta=0 nullable=0 is_null=0 */### @2='andy' /* VARCHAR(10) meta=0 nullable=0 is_null=0 */### @3=NULL /* TIMESTAMP meta=0 nullable=1 is_null=1 */### @4=1 /* INT meta=0 nullable=1 is_null=0 */

© 2012 – Pythian

Binary Log

© 2012 – Pythian

Relay Log• The relay log is a binary log • end_log_pos relates to master binlog not the event size

[ec2 mysql]$ sudo mysqlbinlog -vvv –base64-output=decode-rows relay-bin.000001# at 2609427#121129 0:24:19 server id 6 end_log_pos 2609282 Table_map: `db1`.`t1` mapped to number 9057#121129 0:24:19 server id 6 end_log_pos 2609389 Write_rows: table id 9057 flags: STMT_END_F### INSERT INTO db1.t1### SET### @1=2 /* INT meta=0 nullable=0 is_null=0 */### @2='andy' /* VARCHAR(10) meta=0 nullable=0 is_null=0 */### @3=NULL /* TIMESTAMP meta=0 nullable=1 is_null=1 */### @4=1 /* INT meta=0 nullable=1 is_null=0 */

© 2012 – Pythian

Relay Log

© 2012 – Pythian

Replication Filters• binlog-do-db and binlog-ignore-db - Filters on the master• Replicate-do-db, replicate-ignore-db, replicate-do-table, and

replicate-ignore-table - Filters on the slave.• Replicate-wild-do-table and replicate-wild-ignore-table - Filter

on the slave.• Can have unforeseen consequences, Row based logging tends

to have the best results. Additionally has implication on point in time recovery.

• Column based filtering – Slave has more/fewer columns• Row based filtering – Triggers

© 2012 – Pythian

Replication AdministrationSHOW SLAVE STATUS\G

*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: repl_host... Master_Log_File: mysql-bin.000118 (IO_thread) Read_Master_Log_Pos: 137492455 (IO_thread) Relay_Log_File: mysqld-relay-bin.000011 (Relay Log) Relay_Log_Pos: 131560187 (Relay Log Pos) Relay_Master_Log_File: mysql-bin.000118 (SQL_thread) Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table:... Exec_Master_Log_Pos: 137492455 (SQL_thread)... Seconds_Behind_Master: 0...

© 2012 – Pythian

Communication

© 2012 – Pythian

CommunicationAsynchronous replication

• Built in, default, replication method, doesn't require any additional configuration.

• Lightweight, doesn't wait for any remote confirmations.• Highest performance, lowest durability.

Semi-Syncronous replication• Introduced in 5.5, implemented using plug-ins on master and

slave.• Waits for confirmation that an event from the binary log on the

master has been written to the relay log of at least 1 slave.• Reverts to asynchronous replication if the confirmation times

out.

© 2012 – Pythian

Communication

© 2012 – Pythian

Native Replication Overview

MySQL Replication Topologies

© 2012 – Pythian

Master / Slave(s)

© 2012 – Pythian

Master / Relay

© 2012 – Pythian

Master / Master

© 2012 – Pythian

Circular

© 2012 – Pythian

Multiple Master (Fan-in)

Replication Ecosystem

© 2012 – Pythian


Replication EcosystemEcosystem within Ecosystem

© 2012 – Pythian


OverviewProducts

Tools

Tungsten Replicator

Galera

Master High Availability

Data Consistency

Prefetching

Maintainence

Monitoring

© 2012 – Pythian


Compliment ImproveReplace

Wait...replication doesn't rock?

© 2012 – Pythian

Continuent Tungsten Replicator

https://code.google.com/p/tungsten-replicator/

(Talking here at Percona Live)

© 2012 – Pythian

Tungsten Replicator

● External to MySQL Replication (but depends on binlog)

● Cross version (5.1, 5.5), cross fork (Percona, Oracle...)

● Global Transaction IDs

● Parallel Replication

● Heterogeneous Environments (MySQL → Oracle → Postgres, ...)

● Complex topologies, Multiple masters with single slave

● Time-Delay replication

● Auto Provisioning tools

● Enterprise functionality and support available



© 2012 – Pythian

Tungsten ReplicatorReplication Ecosystem


High level overview of Tungsten

© 2012 – Pythian

Replication Jackpot?



© 2012 – Pythian

● Relies on the binglog

● Another product to learn, configure, maintain

● New considerations for monitoring (Nagios plugins etc)

Further Reading:

http://code.google.com/p/tungsten-replicator/wiki/TungstenReplicatorCookbook

https://s3.amazonaws.com/releases.continuent.com/doc/replicator-

2.0.4/html/Tungsten-Replicator-Guide/content/index.html

http://datacharmer.blogspot.co.uk

https://launchpad.net/galera/

Nice to know


© 2012 – Pythian

Codership Galera Cluster


(also here at PLUK!)

© 2012 – Pythian

● Synchronous replication (no slave lag)

● No SPOF

● Simplified high availability using JDBC/PHP connectors

● Multi threaded slaves (row level)

● Consistency guaranteed

● Scalable reads & writes

● True Master-Master Active-Active (read/write on any node)

● XtraDB Cluster, MariaDB support

● Auto Provisioning tools

● Commercial support available

GaleraReplication Ecosystem


© 2012 – Pythian



High level overview of Galera

© 2012 – Pythian

● InnoDB only● Synchronicity adds overhead● As fast as the slowest node

Further Reading:

http://codership.com/content/using-galera-cluster

http://www.severalnines.com/clustercontrol-mysql-galera-tutorial

http://openlife.cc/search/node/galera



Nice to know

http://www.severalnines.com/clustercontrol-mysql-galera-tutorial

© 2012 – Pythian

Master High Availability

http://code.google.com/p/mysql-master-ha

© 2012 – Pythian

http://code.google.com/p/mysql-master-ha/

● Not a replication product

● Simplifies Failover

● Takes care of re-aligning slaves

● Elects most up-to-date slave to promote as new Master

● Automated monitoring and failover of Master

● Interactive failover

● Failover without the monitoring or the interactivity

● Online switching masters (think maintenance)

● Commercial support available from SkySQL

MHAReplication Ecosystem

© 2012 – Pythian



Simplified Overview of MHA slave promotion

1. 2. 3.

© 2012 – Pythian



● No automated failback (not a bad thing!)

● Manager node now a SPOF unless clustered (heartbeat?)● Author now working at Facebook so ? Around future

development of the tool.

Further Reading:

http://code.google.com/p/tungsten-replicator/wiki/TungstenReplicatorCookbook

https://s3.amazonaws.com/releases.continuent.com/doc/replicator-

2.0.4/html/Tungsten-Replicator-Guide/content/index.html

http://datacharmer.blogspot.co.uk

Nice to know

Eco-Tools

© 2012 – Pythian

Data ConsistencyPerformance

MonitoringFeatures

Primary

Secondary

Replication has got it's problems...(currently)

Eco-Tools

© 2012 – Pythian

Eco-Tools

Data Consistency

Performance

Monitoring

Features

● set global sql_slave_skip_counter=1;● set sql_log_bin=0;● DML statements executed directly on the

slave

● Writing to both sides of a master-master topology

Human Intervention

● sync_binlog=0 + crashTechnical Difficulties

© 2012 – Pythian

UPDATE t1 … LIMIT 1;… UUID()

… now()

Non-Deterministic Statements (SBR)

Data Consistency

Performance

Monitoring

Features

Eco-Tools

© 2012 – Pythian

[ERROR] Duplicate entry '1' for key "PRIMARY"

[ERROR] Slave SQL: Could not execute Delete_rows

!!!YOUR DATA IS* INCONSISTENT!!!

INSERT INTO t1 … VALUES (1, ...)

DELETE FROM t1 WHERE date = '2012-12-04 13:45:00'

*probably

Data Consistency

Performance

Monitoring

Features

Eco-Tools

© 2012 – Pythian

Rebuild your slave

“WTF??? It's 5tb of data!!!”

“Sure, it's a 100mb database”

Data Consistency

Performance

Monitoring

Features

Eco-Tools

© 2012 – Pythian

Find it, fix it

pt-table-checksum

pt-table-sync

Data Consistency

Performance

Monitoring

Features

find

fix

From percona toolkit

Eco-Tools

© 2012 – Pythian

pt-table-checksumData Consistency

Performance

Monitoring

Features

● Connects to tables and checksums chunks of rows.

● Works across one table at a time

● Self aware

● Slave aware

● Writes to table percona.checksums (by default)

Eco-Tools

© 2012 – Pythian

pt-table-syncData Consistency

Performance

Monitoring

Features

● Synchronizes data (only) between tables

● Changes data!

● Runs no-op queries on the master to replicate to slaves

● Foreign Key Funkiness

● Strongest with tables using PK or Unique key

Eco-Tools

© 2012 – Pythian

Problem:

● Data drift detected

Solution:

● Read Only Slaves

● Respect the binary logs

Workaround:

● Find and fix with pt-table tools

Summary #1Data Consistency

Performance

Monitoring

Features

Eco-Tools

© 2012 – Pythian

Master: MultithreadedData Consistency

Performance

Monitoring

Features

Eco-Tools

© 2012 – Pythian

mk-slave-prefetch by maatkit

Replication Booster by Yoshinori Matsunobu

faker by MySQLatFacebook

slave readahead by Anders Karlsson

Meet the fortune tellersData Consistency

Performance

Monitoring

Features

Eco-Tools

© 2012 – Pythian

mk-slave-prefetch by maatkit

Replication Booster by Yoshinori Matsunobu

faker by MySQLatFacebook

slavereadahead by Anders Karlsson

Meet the fortune tellersData Consistency

Performance

Monitoring

Features

Eco-Tools

© 2012 – Pythian

Problem:

● serialized application of relay log events can cause slave lag.

Solution:

● multithreaded slaves

Workaround:

● Hardware hack

● Prefetch

Summary #1Data Consistency

Performance

Monitoring

Features

Eco-Tools

© 2012 – Pythian

Data Consistency

Performance

Monitoring

Features

Eco-Tools

oak-get-slave-lag● Reports if slave is lag exceeds given threshold

● Works off SHOW SLAVE STATUS

● Bound to native replication

● Can be used in load balancer to determine if slave should be accessed

oak-show-replication-status● Reports replication status of slaves connected to a master

● Non-recursive (doesn't show if slave is a master and it's connected

slaves)

● Also depends on slaves being connected at runtime

● Can be used in load balancer to determine if slave should be

accessed

http://code.openark.org/forge/openark-kit

© 2012 – Pythian

Data Consistency

Performance

Monitoring

Features

Eco-Tools

--pt-heartbeat● Monitors replication lag on MySQL or Postgres

● Operates beyond native replication (Tungsten,

etc)

● Supports multi tiered replication topology

● Can Emulate GTID (Global Transaction ID)

© 2012 – Pythian

Data Consistency

Performance

Monitoring

Features

Eco-Tools

--pt-slave-restart● Monitors replica(s) for errors

● Brute force method of restarting replication

● Not for regular consumption

● Does not 'fix' replication

--pt-slave-delay● Intentionally causes slave to lag behind it's master

● Works by starting and stopping slave threads

● Helpful as a means to retrieve old/lost data

● Considered a good means to reproducing race

condition to test application

MySQL Replication in 5.6

© 2012 – Pythian

Multi-threaded slaves• Like Tungsten Replicator, operations are only parallel across

multiple distinct databases.• Requires:

--slave-parallel-workers(+relay_log_info_repository='table';)(+STOP SLAVE; START SLAVE;)

• Tunable:--slave_pending_jobs_size_max--slave_checkpoint_group--slave_checkpoint_period

© 2012 – Pythian

Binary log group commit• Improves performance by writing several events to the binary

log file as opposed to one.• Not “new”, previously seen in MariaDB 5.3.• Requires no extra config• Tunable

--binlog_order_commits--binlog_max_flush_queue_time

© 2012 – Pythian

Global Transaction Identifier• GTID is combination of the server uuid and the transaction_id

which is simply a sequential id on that server.• Requires:

–log-bin

–log-slave-updates

–gtid-mode=ON

–disable-gtid-unsafe-statements (becomes enforce-gtid-consistency)

• CHANGE MASTER TO ... MASTER_AUTO_POSITION=1• Required for some of the new MySQL Utilities

(mysqlrpladmin and mysqlfailover)• http://mysqlmusings.blogspot.co.uk/2012/10/round-robin-

replication-using-gtid.html

© 2012 – Pythian

Global Transaction Identifiermysql> show slave status\G*************************** 1. row ***************************... Master_Log_File: mysql_sandbox5608-bin.000003 Read_Master_Log_Pos: 1257... Relay_Master_Log_File: mysql_sandbox5608-bin.000003 Slave_IO_Running: Yes Slave_SQL_Running: Yes... Exec_Master_Log_Pos: 1257... Master_UUID: 2a499ad4-388b-11e2-ae3c-12313c035dd8... Retrieved_Gtid_Set: 2A499AD4-388B-11E2-AE3C-12313C035DD8:1-10 Executed_Gtid_Set: 2A499AD4-388B-11E2-AE3C-12313C035DD8:1-10,2AFB091E-388B-11E2-AE3C-12313C035DD8:1 (inconsistent case fixed in 5.6.10)

mysql> show global variables like 'server_uuid';+---------------+--------------------------------------+| Variable_name | Value |+---------------+--------------------------------------+| server_uuid | 2afb091e-388b-11e2-ae3c-12313c035dd8 |+---------------+--------------------------------------+

© 2012 – Pythian

Crash safe slaves• master.info and relay-log.info files are updated immediately

after a transaction commits, however there's still a chance of a crash between the commit and the files being updated.

• New variables: master_info_repository and relay_log_info_repository allow positional information to be stored in tables.

• By default the master_info_repository and relay_log_info_repository system tables use InnoDB storage engine and are located in the MySQL db.

© 2012 – Pythian

Replication Checksums• Not equivalent to pt-table-checksum, these checks are focused

on the events written in the binary log and replication. • Uses CRC-32 checksum to verify the events in the binary log

and relay logs.• binlog_checksum generates checksum in the binlog• master_verify_checksum verifies events when read from the

binary log• slave_sql_verify_checksum verifies events when read from the

relay log

© 2012 – Pythian

Slave Delay• Provides a means of enforcing slave lag, similar to pt-slave-delay.• Requires:

CHANGE MASTER TO MASTER_DELAY = N;

mysql> show slave status\G*************************** 1. row ***************************

...

SQL_Delay: 0

SQL_Remaining_Delay: NULL

...

© 2012 – Pythian

Remote Binlog Backup• Uses mysqlbinlog to provide functionality to backup binary logs

without the need of a slave.• Requires:

--read-from-remote-server (or -R)

--raw

• Optional:--stop-never--stop-never-slave-server-id=id:--result-file

© 2012 – Pythian

5.6 Replication Potpourri

• Optimized row-based replication• Informational log events – useful new feature enables the

original statement to be shipped in the binary log along with the corresponding row before / after image.

• Slave_last_heartbeat – pt-heartbeat it is not, heartbeat configured in CHANGE MASTER TO ... however only kicks in during periods of inactivity.

© 2012 – Pythian

Further Reading

http://larsthalmann.blogspot.co.uk/

http://d2-systems.blogspot.co.uk/

http://mysqlmusings.blogspot.fr/

http://drcharlesbell.blogspot.co.uk/

http://datacharmer.blogspot.co.uk/

http://dimitrik.free.fr/blog/

http://dev.mysql.com/doc/refman/5.6/en/replication.html

http://larsthalmann.blogspot.co.uk/

http://d2-systems.blogspot.co.uk/

http://mysqlmusings.blogspot.fr/

http://drcharlesbell.blogspot.co.uk/

http://datacharmer.blogspot.co.uk/

http://dimitrik.free.fr/blog/

http://dev.mysql.com/doc/refman/5.6/en/replication.html

MySQL Replication Tools in 5.6

© 2012 – Pythian

MySQL Utilities

• MySQL Utilities can be installed via MySQL Workbench, or as a standalone package.

• Set of scripts written in Python.

http://dev.mysql.com/doc/workbench/en/mysql-utilities.htmlhttps://launchpad.net/mysql-utilities

http://dev.mysql.com/doc/workbench/en/mysql-utilities.html

https://launchpad.net/mysql-utilities

© 2012 – Pythian

MySQL Utilities

• mysqldbcompare• mysqldbcopy• mysqldbexport• mysqldbimport• mysqldiff• mysqldiskusage• mysqlfailover• mysqlindexcheck• mysqlmetagrep• mysqlprocgrep• mysqlreplicate• mysqlrpladmin

• mysqlrplcheck• mysqlrplshow• mysqlserverclone• mysqlserverinfo• mysqluserclone• Mysqluc (cmd line)• mut (tests)

© 2012 – Pythian

MySQL Utilities• mysqlfailover

Performs replication health monitoring, provides automatic failover on replication topologies (GTID, MySQL Server 5.6.5+)

• mysqlreplicateAutomates the process of setting up replication

• mysqlrpladminAdministers the replication topology, allows recovery of the master, commands include elect, failover, gtid, health, start, stop, and switchover.

• mysqlrplcheckCheck replication configuration, tests binary logging on master.

• mysqlrplshowShow slaves attached to master, can search recursively, shows the replication topology as a graph or list.

© 2012 – Pythian

Thank you and Q&A

http://www.pythian.com/news/

http://www.facebook.com/pages/The-Pythian-Group/163902527671

@pythian

http://www.linkedin.com/company/pythian

1-877-PYTHIAN

[email protected]

To contact us…

To follow us…

@pythianjobs

replication features, technologies and 3rd party extinction

Technology