running multiple mysql instances on a single server
DESCRIPTION
Running Multiple MySQL Instances On a Single Server. Ben Black – [email protected] Mark Filipi – [email protected]. About us. Ben (Kansas State University, 2001) SysAdmin / Oracle DBA Started using MySQL in 2006 Mark (University of Kansas, 2008) Hired to work on Oracle - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/1.jpg)
Running Multiple MySQL Instances On a Single Server
Ben Black – [email protected] Filipi – [email protected]
![Page 2: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/2.jpg)
About us
Ben (Kansas State University, 2001) SysAdmin / Oracle DBA Started using MySQL in 2006
Mark (University of Kansas, 2008) Hired to work on Oracle Started on SQL Server 90% MySQL
![Page 3: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/3.jpg)
About us
24/7 databases for websites, connected services, and manufacturing for Garmin.
Examples of data we process… ~5.5 million data points from connected units per day ~100,000 web orders per day Massive amounts of binary traffic data every 3
minutes Largest MySQL instance: 2.5 TB
![Page 4: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/4.jpg)
In the beginning…
1 instance per server Some masters had slaves Some servers were old… really old. Only some DBs were documented, protected, backed
up, etc.
![Page 5: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/5.jpg)
Reasons for consolidation
Underutilized hardware Long lead times to provision new hardware Licensing / hw / OS costs Smaller datacenter footprint Reduced overhead
![Page 6: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/6.jpg)
Reasons for consolidation
Standardize hardware and operating system Increased redundancy Separate DB from Application load balanced active-passive pool for planned
maintenance and failover Enterprise Monitor/query analyzer
![Page 7: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/7.jpg)
Reasons for consolidation
F5 db traffic routing Garmin’s MySQL environment no longer takes
MONTHS to master
Standard repeatable build Automated configurations with Puppet Able to build a new db instance in 15 minutes
![Page 8: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/8.jpg)
The old ways
![Page 9: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/9.jpg)
What we did
Added multiple alias network interfaces per server – one per instance plus extras for expansion
Added multiple MySQL unix accounts per server One init script per instance Placed shared binaries in /opt/mysql/ belonging to
mysql unix group
![Page 10: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/10.jpg)
What we did
[filipi@olaxpd-myz03 mysql]$ ls -lh
total 8.0K
drwxr-xr-x 3 mysql mysql 4.0K Jun 8 2009 5.0.72sp1
drwxr-xr-x 3 root root 4.0K Jul 31 2009 5.1.31sp1
lrwxrwxrwx 1 mysql mysql 21 Jun 8 2009 mysql -> /opt/mysql/5.0.72sp1/
lrwxrwxrwx 1 mysql mysql 26 Nov 25 2008 mysql5.0 -> 5.0.72sp1/mysql-5.0.72sp1/
lrwxrwxrwx 1 root root 26 Jul 30 2009 mysql5.1 -> 5.1.31sp1/mysql-5.1.31sp1/
![Page 11: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/11.jpg)
What we did
LVM on SAN LUNs mounted at /sqldata and /sqllogs Directory for each instance under /sqldata and
/sqllogs One my.cnf for each instance under
/sqldata/instance_name/
Route traffic through F5 VIP
![Page 12: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/12.jpg)
What we did
[filipi@olaxpd-myz03 sqldata]$ ls -lh
total 56K
drwxr-xr-x 4 mysql05 mysql05 4.0K Oct 8 15:24 eepl_s
drwxr-xr-x 4 mysql03 mysql03 4.0K Jun 9 2009 extensis_m
drwxr-xr-x 4 mysql07 mysql07 4.0K Sep 29 2009 forums_ger_s
drwxr-xr-x 4 mysql01 mysql01 4.0K Jul 6 2009 forums_m
drwxr-xr-x 4 mysql04 mysql04 4.0K Aug 31 2009 gcs_ps
drwxr-xr-x 4 mysql06 mysql06 4.0K Nov 3 08:45 gif_ps
drwxr-xr-x 4 mysql00 mysql00 4.0K Jun 9 2009 jahia_m
drwxr-xr-x 4 mysql02 mysql02 4.0K Jul 1 2009 jahiauk_m
drwxr-xr-x 4 mysql03 mysql03 4.0K Nov 16 13:58 phpmini_m
![Page 13: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/13.jpg)
What we did
![Page 14: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/14.jpg)
What we didn’t do
Why not mysqld_multi? Keep instances portable and self-contained MySQL processes are independent of each other
Server virtualization (VMWare)
![Page 15: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/15.jpg)
my.cnf
[mysqld_safe]ledir = /opt/mysql/mysql5.0/bin
[mysqld]
user = mysql01bind-address = 192.168.15.171
port = 3307socket = /sqldata/mg_m/db/mysql.sock
basedir = /opt/mysql/mysql5.0
datadir = /sqldata/mg_m/db/ innodb_data_home_dir = /sqldata/mg_m/db/
innodb_log_group_home_dir= /sqllogs/mg_m/
innodb_buffer_pool_size = 8G
![Page 16: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/16.jpg)
MySQL Proxy
Standard
With Proxy
![Page 17: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/17.jpg)
MySQL Proxy
Ours
![Page 18: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/18.jpg)
mysql-monitor-agent.ini
[mysql-proxy]
keepalive = true
plugins=proxy,agent
agent-mgmt-hostname = http://agent:xxxx@localhost:18080/heartbeat
mysqld-instance-dir= etc/instances
agent-item-files = share/mysql-proxy/items/quan.lua,share/mysql-proxy/items/items-mysql-monitor.xml,share/mysql-proxy/items/agent-allocation-stats.lua
proxy-address = 192.168.15.171:3306
proxy-backend-addresses = 192.168.15.171:3307
proxy-lua-script = share/mysql-proxy/quan.lua
max-open-files=15000
agent-uuid = 5479b948-b97d-40c9-a84f-7f26a1dd5d78
log-file = mysql-monitor-agent.log
pid-file=/sqldata/mg_t/opt/mysql/enterprise/agent/mysql-monitor-agent.pid
![Page 19: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/19.jpg)
MySQL Proxy
SSH tunneling
![Page 20: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/20.jpg)
Advantages
Run multiple versions of MySQL, and rapidly switch between them for testing
Able to upgrade a single instance without affecting the other instances on the same server
All production instances are paired with a slave for backups and redundancy
43 Prod instances on 14 hosts 65 non-prod on 28 hosts
![Page 21: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/21.jpg)
Problems/Issues
Proxy port issues – 3306/3307/4040
Proxy performance under load
Slave filling disk
Server locking up after running out of RAM
LVM Snapshots of separate LUNs
![Page 22: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/22.jpg)
“Fixes”
Only route through proxy if you have a good reason
Set ulimit in start script
Limit RAM if necessary
Be very careful when setting up load balancing and failover
QUAN using new JDBC connector, not proxy!
![Page 23: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/23.jpg)
Challenges
Each instance can adversely affect others especially disk I/O
Binary logs can chew up lots of disk
![Page 24: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/24.jpg)
Maintenance Planning
Embodiment of all advantages of our consolidation With a capable slave and sufficient planning,
downtime is minimized
![Page 25: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/25.jpg)
Maintenance Planning
![Page 26: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/26.jpg)
Maintenance Planning
![Page 27: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/27.jpg)
Maintenance Planning
![Page 28: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/28.jpg)
Maintenance Planning
![Page 29: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/29.jpg)
Maintenance Planning
![Page 30: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/30.jpg)
Maintenance Planning
![Page 31: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/31.jpg)
Maintenance Planning
![Page 32: Running Multiple MySQL Instances On a Single Server](https://reader036.vdocuments.mx/reader036/viewer/2022062323/56815a95550346895dc8118a/html5/thumbnails/32.jpg)
Ben Black – [email protected] Mark Filipi – [email protected]