scaling mysql writes through partitioning
DESCRIPTION
MySQL is fairly fast for most applications, but every now and then you might come across an application that needs fast writes at a very large scale. The problem with this is that if your inserts are not ordered according to your primary key, then once you cross a limit known as the innodb_buffer_pool_size (for InnoDB tables), write performance starts to degrade because you're now hitting disk a lot. This limit is dependent on the amount of RAM you have, and that in turn is limited by how much money you can throw at the problem. At some point it makes sense to close your pocket and start thinking of a way to hack around the problem. Faced with a similar problem, we figured out a way to hack it with MySQL 5's partitioning to scale writes to a consistently high rate. In this talk, I'll cover all the steps we went through to get to this solution in the hopes that either the solution itself or the thought processes behind it will help others solve their own scaling problems.TRANSCRIPT
The ProblemThe Tests
Breakthroughs
Scaling MySQL writes through partitioning
Philip Tellis / [email protected]
ConFoo / 2010-03-10
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
$ finger philip
Philip [email protected]
bluesmoon.info@bluesmoonyahoogeek
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Web requests
Millions of beacons from a web pageNo response requiredCan be batch processedVery small amounts of data loss is acceptable
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Large volume
2000 requests/second on most daysup to 8000 requests/second on some days200MM requests/daySome data is fake or abusive
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Access patterns
Lots of writes throughout the dayOne huge read at the end of the daySummarise data and throw out the detailsMany reads of summary data over several months
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Why not use a data warehouse?
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
I like to get the most out of my hardware
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Hardware setup
MySQL 5.1Multi-master replication in two colos, 1 remote slave permasterOnly one master writable at any point of time4GB RAM (later 16GB), Big disk with RAID 10
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
DB config
innodb_buffer_pool_size=2078Minnodb_flush_log_at_trx_commit=1innodb_log_buffer_size=8Minnodb_max_dirty_pages_pct=90innodb_doublewrite=1, innodb_support_xa=1sync_binlog=0key_buffer_size=32M, myisam_sort_buffer_size=512ktransaction_isolation=REPEATABLE-READ
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Data setup
Each row 120bytes+ InnoDB overheadinnodb_file_per_table so we can see how the table growsNo Autoincrement fieldsPRIMARY KEY derived from data + one other index
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Test requirements
Insert records until the system breaks downFind out why it broke downFind out how to make it not break downFind out how fast we can insert records (must be >2000 i/s)
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Test requirements
Insert records until the system breaks downFind out why it broke downFind out how to make it not break downFind out how fast we can insert records (must be >2000 i/s)
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Test requirements
Insert records until the system breaks downFind out why it broke downFind out how to make it not break downFind out how fast we can insert records (must be >2000 i/s)
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Test requirements
Insert records until the system breaks downFind out why it broke downFind out how to make it not break downFind out how fast we can insert records (must be >2000 i/s)
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
How I tested
Insertion script measured insertion speed v/s number ofrecordsNumber of records roughly translates to table sizeOn DB box we measure disk performance and table size
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Test 1
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Test 2 - Drop the secondary index
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Test 3 - innodb_buffer_pool_size=1000
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Realisation
Max table size directly proportional toinnodb_buffer_pool_size
Extra index reduces insertion rateExtra index reduces max table sizePossible solution: increase RAM andinnodb_buffer_pool_size
But this only postpones the problem
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Realisation
Max table size directly proportional toinnodb_buffer_pool_size
Extra index reduces insertion rateExtra index reduces max table sizePossible solution: increase RAM andinnodb_buffer_pool_size
But this only postpones the problem
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Test 4 - innodb_flush_log_at_trx_commit=2
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Test 5 - innodb_max_dirty_pages_pct=60
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Test 6 - Let’s try MyISAM
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Test 7 - Inserts in a transaction
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Other stuff we tried
innodb_doublewrite=0 - no effectServer side prepared statements - no effecttransaction_isolation=READ-COMMITTED - no effectinnodb_support_xa=0 - 12% increase in insertion rateCombination of the best options - negligible effect
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
What we knew at this point
Sticking with InnoDBWe need a large buffer poolWe need to drop extra indicesflush_log_at_trx_commit=2 is good enoughTransactions are good
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Our big problem
Insert rate was barely reaching the rate of incoming data!Still breaks down before getting a day’s worth of data
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
Test 8 - Single bulk insert
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
Bulk insert specifications
40,000 records in one insert statementUse INSERT IGNORE4-6 seconds per statementPRIMARY KEY drops duplicatesWe still have a breakdown when we cross the buffer pool
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
Test 9 - bulk inserts + partitioning
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
What happened?
Split the table into partitionsEach partition < 0.5 x innodb_buffer_pool_sizecurrent and next partition fit in memory at any timePartition key is based on incoming data and not onSELECTs
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
Schema
CREATE TABLE (...
) PARTITION BY RANGE( ( time DIV 3600 ) MOD 24 ) (Partition p0 values less than (2),Partition p1 values less than (4),...Partition p10 values less than (22),Partition p11 values less than (24)
);
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
Test 10 - Ran for 7 days
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
Still running
Terabytes of dataaround 8500 inserts per secondPotentially 700+ MM inserts per day
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Summary
Bulk inserts push up your insert ratePartitioning lets you insert more recordsPartition based on incoming data key for fast insertshttp://tech.bluesmoon.info/2009/09/scaling-writes-in-mysql.html
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Thanks, Merci
ConFoo organisersExceptional Performance team @ Yahoo!Monitoring team @ Yahoo!MySQL Geeks at Yahoo!
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
contact me
Philip [email protected]
bluesmoon.info@bluesmoonyahoogeek
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Photo credits
Disused warehouse on Huddersfield Broad Canal / by TDR1http://www.flickr.com/photos/tdr1/3578203727/
Hardware store dog / by sstrudeauhttp://www.flickr.com/photos/sstrudeau/330379020/
North Dakota, Broken Down Van / by mattdentehttp://www.flickr.com/photos/mattdente/46944898/
One red tree / by EssjayNZhttp://www.flickr.com/photos/essjay/155223631/
The Leaning Tree / by stage88http://www.flickr.com/photos/stage88/3179612722/
ConFoo / 2010-03-10 Scaling MySQL writes through partitioning