high availability perl dbi + mysql
DESCRIPTION
A talk I gave @ YAPC::NA in 2005 on implementing High Availability for Perl applications that span multiple datacenters.TRANSCRIPT
![Page 1: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/1.jpg)
High Availability?Setting Up MySQL
Our ApproachDBI Framework
Questions
High Availability DBI & MySQLYAPC::NA 2005
High AvailabilityDBI & MySQL
Steve Purkis
![Page 2: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/2.jpg)
High Availability?Setting Up MySQL
Our ApproachDBI Framework
Questions
High Availability DBI & MySQL
![Page 3: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/3.jpg)
Slide 3
High Availability?
• What is High Availability?• Ever have an application fail because a
mission critical server crashed?
• High Availability is an attempt to provide zero application downtime.
![Page 4: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/4.jpg)
Slide 4
High Availability?
• How do I achieve HA?:• Analyze your apps• Most important components?
• Remove single points of failure (SPOFs):• Add redundant servers• Replicate data (backups, realtime)• Load balancing• Failover when things go pear-shaped
![Page 5: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/5.jpg)
Slide 5
High Availability?
• This talk…• Introduces one approach to building HA Perl
apps that rely on MySQL.
• We’ve used similar techniques with Oracle
• They work for us…• They may not be right for you!
![Page 6: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/6.jpg)
High Availability?Setting Up MySQL
Our ApproachDBI Framework
Questions
High Availability DBI & MySQL
![Page 7: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/7.jpg)
Slide 7
Setting Up MySQL
• Some typical architectures…
![Page 8: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/8.jpg)
Slide 8
Setting Up MySQL
• Ye olde basic setup• Not HA.
Application Server
MySQLServer
![Page 9: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/9.jpg)
Slide 9
Setting Up MySQL
• One master, one read-only slave• Good for load balancing• Still not really HA!
Application Server
MySQLMaster
MySQL Slave Data
Replication
reads writes
Load Balancer
![Page 10: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/10.jpg)
Slide 10
Setting Up MySQL
• Two masters• It’s HA, but…• Flaky, tricky to work with!
Application Server
MySQLMasterTwo-way
Replication
MySQLMaster
Load Balancer
![Page 11: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/11.jpg)
Slide 11
MySQL Slave
Setting Up MySQL
• Two masters, multiple slaves• Only one master active at a time• HA
Application Server
MySQL Slaves Data
Replication
reads writes
Primary Master
2nd aryMaster
Master Failover
Load Balancer
• HA (unless your hosting center goes down)
![Page 12: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/12.jpg)
Slide 12
Setting Up MySQL
• Your HA MySQL architecture will depend on:• SLAs• Application requirements• Load• Physical spread of servers• Budget
• Caveats:• Are your apps read or write heavy?• Beware of replication lag time
• Common in high-latency networks• Do your apps need the latest data?
• Transactions?
![Page 13: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/13.jpg)
Slide 13
Setting Up MySQL
• Load balancing• Hardware?• Software?
• Master Failover• On Slaves:
CHANGE MASTER TO …• Manual?• Automate?
• Cron? Application? SLB?
• On App servers:• Connect to Secondary
• Not required if using a hot spare w/same ip
![Page 14: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/14.jpg)
Slide 14
Setting Up MySQL
• Recommended reading:• High Performance MySQL
- Jeremey Zawodny & Derek Balling
• MySQL Replication docs• MySQL Cluster white paper
• For Linux:• Linux Virtual Server Project• Linux-HA
![Page 15: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/15.jpg)
Slide 15
Setting Up MySQL
• If you have any trouble:• MySQL mailing list
![Page 16: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/16.jpg)
High Availability?Setting Up MySQL
Our ApproachDBI Framework
Questions
High Availability DBI & MySQL
![Page 17: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/17.jpg)
Slide 17
Our Approach
• Our requirements:• Read-heavy apps• Over 750 clients, many with SLAs• Reliability:
• Read availability most important• Must work if a server farm goes down
• Speed is of essence:• Millions of requests a day
• Don’t need transactions• Minimize costs
![Page 18: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/18.jpg)
Slide 18
Our Approach
Application Server
Application Server
Application Server
MySQL Slave
MySQL Slaves
MySQL Slaves
Application Server
MySQL Master
MySQL2nd ary
Master F
ailover
Data Replication
95% 5%
readswrites
![Page 19: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/19.jpg)
Slide 19
Farm 2
Farm 3Farm 1
Our Approach
Application Server
Application Server
MySQL Slave
Application Server
MySQL Master
MySQL 2nd ary
Master F
ailoverIn-Farm Data Replication
Application Server
MySQL SlaveMySQL Slaves
MySQL SlaveMySQL Slaves
Query caching
![Page 20: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/20.jpg)
Slide 20
Our Approach
• DBI Framework:• Load balancing• Server selection• Read / Write query?
• Failover
Application
DBI Framework
----------------------------------- snip: how to tell reads from writes? ---------------------------------
my ($action, $dbh) = $query =~ /\Aselect|\A\(select|\Ashow|\Adesc/i ? ("read", $dbh_read) : ("write", $dbh_write);
----------------------------------------------------------------------------------------------------------
![Page 21: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/21.jpg)
Slide 21
Our Approach
• Load balancing• On connect• Weighted server selection
• Availability, number of processes, user weights, which farm
• MySQL idle timeout
• Failover• Connect to next slave• Automatic query retry
Application
DBI Framework
MySQL Slave
MySQL Slave
reads
![Page 22: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/22.jpg)
Slide 22
Our Approach
• Failover• Persistent connection to
all masters• Automatic query retry• Automatic fallback
Application Server
DBI Framework
Primary Master
2nd ary Master
Master F
ailover
writes
![Page 23: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/23.jpg)
High Availability?Setting Up MySQL
Our ApproachDBI Framework
Questions
High Availability DBI & MySQL
![Page 24: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/24.jpg)
Slide 24
DBI Framework
• Two wrapper functions:• dbconnect( 'read' | 'write' )• Read: Select slave to connect to• Write: Connect to all masters
• $sth = sql( $query )• Read / write dbh selection• Failover & fallback
• Pseudo code?
![Page 25: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/25.jpg)
Slide 25
DBI Framework
• dbconnect( 'read' | 'write' )• Read:
For each slaveweight = 0, next if can’t pingcheck number of processes with mysqladminweight = no. processes / user weighting
Connect to slave with lowest weightSanity check: run a simple queryTry next slave if that failed
• Write:For each master
ConnectSanity check: run a simple query
Set $write_dbh to primary master
![Page 26: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/26.jpg)
Slide 26
DBI Framework
• $sth = sql( $query )• Determine query type…• Read:
Execute query.Failover to next slave on error, and retry query.
• Write:Fallback?
If not using primary, and we failed over X seconds ago, try reconnecting to master.
Execute query.Failover to next master on error, and retry query.
![Page 27: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/27.jpg)
Slide 27
DBI Framework
• Major Drawbacks:• Using DBI-based CPAN modules is hard!
![Page 28: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/28.jpg)
Slide 28
DBI Framework
• Looking Ahead…
• Push HA logic into the DBI layer• Write DBD::MysqlHA ?• DBIx::HA ?• DBD::Multiplex ?• DBIx::DBCluster ?• SQL Relay ?
• MySQL Cluster?
![Page 29: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/29.jpg)
High Availability?Setting Up MySQL
Our ApproachDBI Framework
Questions
High Availability DBI & MySQL
![Page 30: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/30.jpg)
Slide 30
DBI Frameworks on CPAN
• If there’s time…
![Page 31: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/31.jpg)
Slide 31
DBI Frameworks on CPAN
• The ones I know a bit about:• DBIx::HA• DBD::Multiplex• DBIx::DBCluster
![Page 32: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/32.jpg)
Slide 32
DBI Frameworks on CPAN
• DBIx::HA• Generic HA solution
(written & tested for Sybase)• Configurable by db name (%DATABASE::conf)• Looks well thought out
![Page 33: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/33.jpg)
Slide 33
DBIx::HA
Pros• Does Failover
• On query failure & dbh disconnected
• Does timeouts:• connect, query execute• Safe signals
• Connect all dsns on init• Supports Apache::DBI
Cons• No read / write distinction• No way to choose which
dbh to use on failover• (want to use a mysql-
specific algorithm)
• Timeouts with SIGALRM• (but how else, really?)
• No ping checks for non-Apache::DBI• (uses $dbh->ping anyways -
we prefer ICMP ping)
![Page 34: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/34.jpg)
Slide 34
DBIx::HA
• Also: written & tested for Sybase!
• Potential DBD::Mysql problems:• auto_reconnect
we may reconnect to a db when we should be failing over
![Page 35: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/35.jpg)
Slide 35
DBI Frameworks on CPAN
• DBD::Multiplex• send requests to multiple dsn's• Configure servers to use in $dsn
(pipe-separated list)
![Page 36: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/36.jpg)
Slide 36
DBD::Multiplex
Pros• Supports master/slave
setup:• differentiates between
reads / writes• Connects to all dsns
(good for fast master failover)
• Does failover:• default behaviour
reads: first_successwrites: first_error
Cons• Can only specify one
master• (though if you specify none,
writes can go to all with 'first_error' exit mode)
• Connects to all dsns• (don't want to connect to all
slaves)
• No customizable slave failover algorithm
• No reconnects• No fallback!
![Page 37: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/37.jpg)
Slide 37
DBI Frameworks on CPAN
• How could we re-use CPAN modules?
• DBIx::HA• Sub-class to introduce MySQL specific functionality?• Introduce a callback for server selection on failover?• Use in conjunction with DBD::Multiplex for read/write dbh
selection?• There could be problems though…
• In general:• Backwards compat• sql() wrapper (for backwards compat)• Custom logging?• Our db wrappers do other things too…
![Page 38: High Availability Perl DBI + MySQL](https://reader034.vdocuments.mx/reader034/viewer/2022052410/5558c51cd8b42a235c8b4738/html5/thumbnails/38.jpg)
Slide 38
DBI Frameworks on CPAN
• Maybe the best way to reuse them is to nick their ideas?
• DBD::MysqlHA ?• Combination of DBIx::HA and DBD::Multiplex• MySQL specific:
• Customizeable server selection algorithm• Persistent connections