high availability with mysql

THAVA ALAGU

Staff Engineer, Database Group,

Sun Microsystems, Bangalore

thavamuni.alagu@sun.com

http://blogs.sun.com/thava/

Oct 2008

HIGH AVAILABILITY WITH MYSQL

Welcome !Welcome !

2

LET US DISCUSS ABOUT …

Aspects of High AvailabilityAspects of High AvailabilityMySQL ReplicationMySQL ReplicationMySQL ClusterMySQL ClusterApplication HAApplication HAResourcesResources

3

ARCHITECTURE OVERVIEW

4

HIGH AVAILABILITY

Data Must Not be LostData + Application Must be

Available Always

5

MYSQL SOLUTIONS …

MySQL Replication (Asynchronous)

MySQL Cluster (Synchronous)

Supporting Solutions: MySQL Proxy (Alpha) MySQL Load Balancer (Based on Proxy) 3rd Party Solutions from Partners and Others

6

MYSQL REPLICATION HIGHLIGHTS

Binary Log Updates From Master Applied at Slave

Usually One Master Multiple Slaves Asynchronous Copy Slave could be used as Read-Only Geographically Distributed, Slow Links, OK Can be chained –

Master Slave/Master Slave

7

MYSQL REPLICATION BENEFITS

Hot Standby Node for Fail-Over Can be Used for transparent Backups Load Balancing – Slaves Can be Active for Read Automated copy across geographically

distributed sites

8

REPLICATION ARCHITECTURE

9

REPLICATION TECHNIQUES AND TIPS

10

REPLICATION NOTES

Binary logs rotated every 1GB You should clean/purge old logs Master/Slave similar architecture

11

REPLICATION FORMATS

3 Types:Statement Based ReplicationRow-based ReplicationMixed Based Replication

Depends on MySQL Version, Storage Engine

Can override the defaults e.g. use row-format=row option

Usually row based replication is saferBut exceptions exist

12

REPLICATION FOR BACKUPS

Slave can be shutdown for Cold Backup Logical or File System backups Backup Logs too Logs can be used for Point-In-Time Recovery

13

REPLICATE ACROSS STORAGE ENGINES

Slave can be ARCHIVE Engine Insurance against SE specific failuresFaster Slave with No Transaction Engine

e.g. InnoDB to MyISAMHow:

Use storage_engine system variable Mysqldump; edit; reload Stop slave; Alter table …; Start Slave Disable some storage engines at Slave Create new tables with default engine

14

REPLICATE FOR SCALING

Use Multiple Slaves For Read Load Dedicate Master for Write Load Perfect for Load Balancing

15

REPLICATE SELECTIVELY

Filter at Slave – Database, Tables, specify in my.cnf replicate-wild-do-table=sales.% replicate-wild-do-table=finance.%

Master Filters: binlog-do-db, binlog-ignore-db

16

REPLICATE MULTILEVEL

Less Load at MasterCan Use Blackhole Engine for Master2Use log-slave-updates option at Master2

17

MULTIMASTER (AVOID)

2 Masters updating each otherRing of MastersUpdates on disjoint setsUsually, Stay Away

18

REPLICATION TOPOLOGIES SUMMARY

19

REPLICATE USING SSL

For High SecurityUses Certificate-Authority Certificate,

Server Public Key, Server Private Key

20

REPLICATION SETUP UPGRADE

Upgrade Slaves First, Then MasterShutdown, Upgrade, RestartWatch out Versions Upgradability

21

REPLICATION COMMANDS

On Master: Show master status Flush logs Show master logs Show binlog events Purge master logs to ‘mysql-bin.005’;

22

On Slave: Change Master To master_host=‘hostA’ master_user=‘slave’

master_password=‘slavepassword’ ;

Show slave status

Load Data From Master ; # MyISAM only. Locks Tables on master

REPLICATION FAILOVER

Does not happen automaticallyMany solutions exist like virtual IP, Sun

Cluster, Linux-HA Heartbeat, etcApplication should retry

23

MYSQL CLUSTER HIGHLIGHTS

Synchronous Ultra High AvailabilityThink Telecom, Banking Memory (mostly) and Disk basedNo Single Point Of FailureAutomatic Failover

24

MYSQL CLUSTER ARCHITECTURE

25

MYSQL CLUSTER NODEGROUP

26

MYSQL CLUSTER SIMPLE CONFIG

27

MYSQL CLUSTER NOTES

ACID transactions Supports READ COMMITTED only Synchronous replication uses 2 Phase Commit Local and Global Checkpoints to disk Data access through SQL or NDB API Online Backups supported All nodes same architecture – big endian or little endian

28

MYSQL CLUSTER NOTES (CONTD)

Max number of data nodes = 48; Max(data nodes + management nodes )=63 Max total number of all = 256 (including SQL nodes) Tables highly recommended to have primary keys for

cluster replication.

29

MYSQL CLUSTER CONFIGURATION

NDB config File = config.ini Number Of Replicas : 1 upto 4 [ usually 2 ] Memory Sizing

Per Node Memory = (DataSize * Replicas * 1.1) / Nodes

Default Port number for Mgmt Node = 2202 Default Port Number for Data Node = 1186

SQL Node Config File = my.cnf [mysqld] ndbcluster ndb-connectstring=mgmt-hostname

30

NODE FAILURE DETECTION

Uses Heartbeats Failed Node Excluded Node taken over by another in same node group Failed node tries to repair itself and then joins

cluster

31

NETWORK PARTITIONING

Also Called Split Brain condition 2 Sets of Nodes Diverges with writes Management node acts as arbitrator by default Nodes with maximum votes survive

32

CLUSTER CHECKPOINTS

Transactions Commits in memory Local checkpoint writes all node data and UNDO

log written to disk; old REDO logs can be deleted Global Checkpoint: Frequency controlled by

TimeBetweenGlobalCheckpoints (default 2000 millisecs)

33

CLUSTER COMMANDS

ndb_mgmd –f config.ini # start management daemon ndbd --connect-string=mgmt_hostname # start data node mysqld_safe & # Start SQL node ndb_mgm -c mgmt_hostname –e “show” mysql> show engine ndb status;

34

CLUSTER ROLLING UPGRADE/DOWNGRADE

Online Upgrae/Downgrade supported. Shutdown node, replace binary, bring it up One node at a time in this order:

Management Node(s) Data Nodes SQL Nodes

Watch out for upgradability/downgradbility of versions

35

REPLICATING CLUSTERS

36

Note: Row-based logging must be enabled for this to work

ANOTHER HA SOLUTION:DISTRIBUTED REPLICATED BLOCK DEVICE

Available on Linux. Provides synchronous data copy. Uses virtual block device Replicated from primary server to secondary Implemented with kernel level and user level software Often used with Heartbeat Linux Cluster manager Secondary server is passive – Not for load balancing Like RAID but software based

37

RESOURCES …

Developer Zone: Developer Articles, etc. http://dev.mysql.com

MySQL White Papers http://www.mysql.com/why-mysql/white-papers/

MySQL Forums: http://forums.mysql.com

Planet MySQL – Blog aggregator http://planetmysql.org

38

THAVA ALAGU

Blog: http://blogs.sun.com/thava/thavamuni.alagu@sun.comTwitter: thavamuniOct 2008