mysql cluster · title: mysql cluster author: vinay created date: 9/10/2007 10:13:00 am
TRANSCRIPT
![Page 1: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/1.jpg)
1Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
MySQL High-Availability andScale-Out architectures
Oli SennhauserSenior Consultant
![Page 2: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/2.jpg)
2Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Introduction
• Who we are?• What we want?
![Page 3: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/3.jpg)
3Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Table of Contents
• Scale-Up vs. Scale-Out• MySQL Replication• MySQL HA solution• MySQL Cluster• Application architectural solutions• Other architectural stuff• It's your turn...• Let us build a replication
![Page 4: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/4.jpg)
4Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
When do we think about architecture?
• Performance problems• HA requirements
![Page 5: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/5.jpg)
5Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Scale-up vs. Scale-Out
• Scale-up
• Scale-out
t
Relaxation of constraints
![Page 6: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/6.jpg)
6Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
MySQL Replication
• The MySQL answer: Master-Slave-Replication:
Even Oracle uses the expression “scale-out” now!
Master
Slave1
Slave2
Slave3
Application
LB
...
ro
rtw
2 types ofsessions!!!
![Page 7: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/7.jpg)
7Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
MySQL Replication in detail
M S
Appl
bin-logbin-log... relay-log
IO thread
SQL thread
asynchronous!
relay-log
rtw
ro
• wide distance possible!• thin line possible!
![Page 8: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/8.jpg)
8Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
MySQL Replication varieties
M
SI
SI
SI
• Cascaded replication:
for example Yahoo!
S1
S2
S3
S4
S6
S5
S7
S8
...
...
black hole SE!
![Page 9: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/9.jpg)
9Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
MySQL Replication varieties
M
S2
S3
• HA with “hot standby” Slave:
• active – active fail over!
S1
Sbackup
Sstandby
Appl
VIP HA solution
![Page 10: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/10.jpg)
10Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
MySQL HA solution
M
S2
S3
• HA with MySQL:
• 99.99% HA (four nine)• active – passive fail over!
S1
M'
Appl
VIP
DRBD
heartbeat
![Page 11: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/11.jpg)
11Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
MySQL HA solution
M
S2
S3
• HA with MySQL and SAN:
S1
M'
Appl
VIP
SAN
heartbeat
![Page 12: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/12.jpg)
12Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Multi-Master Replication
M1
S12
S13
• actually Master-Master Replication
• This architecture does NOT solve your write problems!!!
S11
Appl
M2
Appl
VIP/LB
S22
S23
S21
![Page 13: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/13.jpg)
13Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Multi-Master Replication
M1
S12
S13
• HA Multi-Master Replication
S11
M2
S22
S23
S21
M2'
Appl
VIP
DRBD
M1'
Appl
VIP
DRBD
VIP/LB
![Page 14: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/14.jpg)
14Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Shared disk cluster
DB1
• This is in 99.9% of the cases NOT your solution!!!• It looks like Oracle RAC but MySQL does NOT
(yet) have the instruments needed!• Can be useful in some special kinds of reporting.
DB2
shareddisk
DB2
ro!
![Page 15: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/15.jpg)
15Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Where are we now?
• The architectures above solved:– your read problems.– some HA requirements.
• But they did not solve:– your write problems!
• And what when we need:– higher HA?– synchronous replication?– more write speed?
![Page 16: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/16.jpg)
16Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
MySQL Cluster
• Shared-nothing architecture• Synchronous replication (2-Phase commit)• Fast automatic fail over• High performance (also writing)• High transactional throughput• No special components required• In-Memory database (in 5.1 also disk support)• Scalable, 1000's of transactions per second• 99.999% HA (five nine)• On-line upgrade path (at least on GA within same
version)
![Page 17: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/17.jpg)
17Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
MySQL Cluster architecture
Application Application Application
MySQLServer
NDB Cluster(Data nodes)
MySQLServer
MySQLServer
DB DB
DBDB
MGM Server
MGM client
ApplicationApplicationApplication
config.ini
my.cnfSQLNodes
DataNodes
MgmtNodes
![Page 18: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/18.jpg)
18Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
MySQL Cluster HA features
MySQLServerMySQLServer
DataNodeDataNode
DataNodeDataNode
NDBStorage Engine
ManagementServer
ManagementServer
NDB APINDB API
MySQL Cluster
DataNodeDataNode
DataNodeDataNode
MySQLServerMySQLServer
MySQLServerMySQLServer
ManagementServer
ManagementServer
MySQLServerMySQLServer
MySQLServerMySQLServer
X X X X
X XX
![Page 19: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/19.jpg)
19Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Cluster with Replication
S2
S3
• for read scale-out (Reporting):
S1
Appl
ndbd ndbd
ndbd ndbd
mysqld mysqld mysqld
Reporting
rw
![Page 20: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/20.jpg)
20Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Cluster with Replication
• Cluster-Cluster replication for disaster fail over (MySQL 5.0):
Appl
ndbd ndbd
ndbd ndbd
mysqld mysqld mysqld
rw
ndbd ndbd
ndbd ndbd
mysqld mysqld mysqld
fail over
![Page 21: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/21.jpg)
21Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Cluster with Replication
• Cluster-Cluster replication for disaster fail over (MySQL 5.1):
Appl
ndbd ndbd
ndbd ndbd
mysqld mysqld mysqld
rw
ndbd ndbd
ndbd ndbd
mysqld mysqld mysqld
fail over
![Page 22: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/22.jpg)
22Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Cluster with Replication
• circular Cluster-Cluster replication (>= MySQL 5.1.18?):
![Page 23: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/23.jpg)
23Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Cluster examples
• Session handling• Telecom (Mobile)• VoIP• RSS-Feed aggregation• Mail• On-line Games
• Use cluster where– you need HA– you have high write load– you do little Joins and Grouping
![Page 24: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/24.jpg)
24Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
How to go on?
• Read is a caching problem!➔ More RAM or scale-out.
● Write is a batching problem!➔ Batch your load,➔ Buy stronger I/O system.➔ Use MySQL Cluster.➔ SSD!
● What then?
![Page 25: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/25.jpg)
25Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
SSD disruption
• I/O system without any movable parts!• SSD = Solid State disk (Flash memory, NAND,
NOR chips, RAM-SAN).• During the last year(s)• Price from 15'000 USD / 160 Gbyte -> 100 USD /
32 Gbyte• 10-50 times faster than mechanical I/O systems• 1 Mio write cycles dead :-( (special FS!)
• RAM: Huge amount of memory is cheap!
➔ This will disrupt the database world!
![Page 26: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/26.jpg)
26Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
What then?
• With or without SSD we will reach a physical limit. What then?
• Application partitioning:– Split applications
• OLTP vs. OLAP• all in one
– Segment your application
![Page 27: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/27.jpg)
27Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Application partitioning
• OLTP vs. OLAP– hot business data (trx) vs. “old” reporting data
• “All in one”– Sessions, user tracking, ads, chat, booking
• Segment– split by for example 1 Mio users (split by user_id).
![Page 28: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/28.jpg)
28Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Architecture examples
![Page 29: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/29.jpg)
29Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Architecture examples
![Page 30: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/30.jpg)
30Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Architecture examples
![Page 31: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/31.jpg)
31Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Architecture examples
![Page 32: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/32.jpg)
32Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Some more architecture stuff?
• Databases are slow!!!➔ Whenever possible try to avoid databases! Use
memcached for example.
• The SP trap.➔ Stored Procedures are a lock in! Try to avoid SP!
• Use Materialized Views (MV) and/or shadow tables.
• VM/SAN is nice for consolidation but not for performance/scale-out!
• Backup and Staging• MySQL – Proxy
![Page 33: MySQL Cluster · Title: MySQL Cluster Author: vinay Created Date: 9/10/2007 10:13:00 AM](https://reader035.vdocuments.mx/reader035/viewer/2022063014/5fd150f95d54a418560a4fc5/html5/thumbnails/33.jpg)
33Copyright 2007 Oli Sennhauser The World’s Most Popular Open Source Database
Now it's your turn...
• Your problems?
• Let us build a replication...