best perf in products part2 arubin 15 sept 2010

8/8/2019 Best Perf In Products Part2 Arubin 15 Sept 2010

http://slidepdf.com/reader/full/best-perf-in-products-part2-arubin-15-sept-2010 1/55

<Insert Picture Here>

The World’s Most Popular Open Source DatabaseCopyright 2010 Oracle

Getting the Best MySQL Performance in Your Products:Part 2, Beyond the Basics

Alexander Rubin

Principle Consultant MySQL



About MySQL

• Founded, first release in 1995• Acquired by Sun in February 2008

• Acquired by Oracle in January 2010

• #1 Most Popular Open Source Database

• Market-leading customers



Oracle’s Plans for MySQL

• Fill-in Oracles database product line

• MySQL Global Business Unit• Managed by Edward Screven, Chief Corporate Architect

• Invest in MySQL!

• “Make MySQL a Better MySQL”

• Develop, Promote and Support MySQL

• Improve engineering, consulting, and support

• MySQL Sunday at Oracle Open World• Leverage World-Wide, 24x7 Oracle Support

• MySQL Community Edition

• Source and binary releases

• GPL license



MySQL’s Market Position

• Ubiquitous

• Over 100 million copies downloaded worldwide

• Over 12 million installations

• Over 70,000 MySQL downloads each day

• Popular

• MySQL is the number one most popular database• 31% use MySQL as primary DB for deployed applications

(Eclipse Community Survey, 2010)

• MySQL is the 3rd most deployed database worldwide

• Behind MSFT SQL Server and Oracle (Gartner 2008)

• MySQL is the most popular open source database

• 40% of OSDB market with 25% growth (Evans Data, 2007)

• Majority of Oracle database users also use MySQL



Industry-Leading Customers

Rely on MySQL

Enterprise 2.0TelecommunicationsOn Demand, SaaS, Hosting

Web / Web 2.0 OEMs / ISVs

http://images.google.com/imgres?imgurl=http://www.infinitytechnology.com/uploadedImages/Home/TechFocus_News/AlcatelLucent_Hor_2col_lrg.gif&imgrefurl=http://www.infinitytechnology.com/phonesystems.aspx&h=251&w=720&sz=11&hl=en&start=4&sig2=urYvX47BOQRqvADY_UMA0w&um=1&tbnid=pA91nzVKrUFImM:&tbnh=49&tbnw=140&ei=94SgSOaRCaqkpATZu-jgBw&prev=/images?q=alcatel+lucent&um=1&hl=en&suggon=0&rls=com.microsoft:en-us&sa=N



The World’s Most Popular Open Source DatabaseCopyright 2010 OracleThe World’s Most Popular Open Source Database

Agenda

Presentation OverviewMySQL versions

Configuration

Monitoring

Queries Tuning

+ Tips

Resources


http://slidepdf.com/reader/full/best-perf-in-products-part2-arubin-15-sept-2010 7/55The World’s Most Popular Open Source DatabaseCopyright 2010 Oracle

Overview Covers MySQL configuration

Monitoring Query tuning Based on what customers are asking This will be technical No new tools required; everything you need

comes with MySQL! You cannot become a performance tuning

wizard in 45 minutes - PT Class is 4 dayclass

– http://www.mysql.com/training/courses/performance_tuning.html

MySQL Performance Forum – http://forums.mysql.com/list.php?24

http://www.mysql.com/training/courses/performance_tuning.html


http://forums.mysql.com/list.php?24

http://forums.mysql.com/list.php?24





MySQL Versions and Performance

Latest GA release: MySQL 5.1.50, InnoDB plugin



MySQL Configuration



MySQL Configuration

• Single configuration file: my.cnf (Linux/Unix/Mac),

my.ini (Windows)• Good example (with comments): my-innodb-heavy-

4g.cnf• Treat it as an example only

• Variables with different scope:

• Global variables• Session variables• Engine Specific variables

• Configure buffers• Global, allocated per MySQL instance• Session, allocated per each connections• Make sense only for InnoDB or MyISAM or other engine



Table Cache

• table_cache – Defines the number of open tables for all threads that

the server should allow

• Open_table_definitions – Displays the number of cached .frm

files

• Open_tables - Displays the number of currently open tables• Opened_tables - Displays the number of tables that have been

opened




Temporary Tables

• tmp_table_size – maximum size for in Memorytemporary tables created by MySQL

• max_heap_table_size - Sets the maximum size towhich MEMORY tables are allowed to grow

• Watch:• Created_tmp_tables – number of temporary table MySQL

created in RAM

• Created_tmp_disk_tables - number of temporary table

MySQL created on DISK




Session Variables• Some session variables control space allocated by

each session (connection)

• Setting these to small can give bad performance• Setting these too large can cause the server to swap !• Can be set by connection

• SET SORT_BUFFER_SIZE=8*1024*1024; (8M)

• Set small be default, increase in connections that need it

• sort_buffer_size - Used for ORDER BY, GROUP

• BY, SELECT DISTINCT, UNION DISTINCT• Monitor Sort_merge_passes < 1-2 an hour optimal

• Usually a problem in a reporting or data warehouse database

• Other important session variables (better to keep small)• read_rnd_buffer_size - Set to 1/2 sort_buffer_size

• join_buffer_size - (BAD) Watch Select_full_join• read_buffer_size - Used for full table scans, watchSelect_scan




InnoDB Concepts: Buffer Pool

Disk StorageBufferPool

DatabaseServer

Read

Write

Read

Write

The In-Memorybuffer pool writesand reads datapages from O/S

The In-Memory

buffer pool writesand reads data

pages from the DBserver




innodb_flush_log_at_trx_commit

• This server variable tells when to execute atransaction commit to flush log buffer to disk

• 0 - The logs are flushed to disk during checkpoints which occursapproximately once per second

• 1 - The COMMIT statement initiates the flush

• 2 - The log_buffer is written to the file (an fsync occurs every second)




Log Files

• The size of each log file should be chosen to avoidexecuting checkpoint operations too often• The bigger log file size reduces disk I/O in checkpointing

• innodb_log_file_size – Defines the size of each log file in a log

group in megabytes

• The larger the value, the less checkpoint flush activity isneeded in the buffer pool, saving disk I/O

• Larger log files mean that recovery will be slower in case ofa crash

• The combined size of log files must be less than 4 GB on32-bit computers




Other InnoDB settings

• innodb_file_per_table• Create tablespace (.ibd) per table

• innodb_flush_method=O_DIRECT

• Prevent unnecessary OS buffering

• For MySQL 5.1.46+

• ignore-builtin-innodb

• plugin-load=innodb=ha_innodb_plugin.so• Enable new InnoDB plugin with better performance




Storage MediumMultiple Tablespaces

Country.frm(Metadata)

MySQLMySQL

DataData

worldworld

testtest

mysqlmysql

City.ibdCity.ibd

Country.ibdCountry.ibd

Countrylanguage.ibd

Multiple tablespacesrequires theinnodb_file_per_table

clause

Countrylanguage.ibd

Multiple tablespacesrequires theinnodb_file_per_table

clause

InnoDB multiple tablespaces

Internal datadictionary- - - - - - - - - - - - - - - - - - - -

Insert undologs

Updateundo logs

Internal datadictionary- - - - - - - - - - - - - - - - - - - -

Insert undologs

Updateundo logs

ibdata files

Transaction

records,redo logrecords, etc.

Transactionrecords,

redo logrecords, etc.

Ib_logfile files



Countrylanguage.frm(Metadata)

Countrylanguage.frm(Metadata)




InnoDB configuration example6-8G RAM, dedicated box, v.5.1.50

ignore-builtin-innodbplugin-load=innodb=ha_innodb_plugin.so

innodb_buffer_pool_size = 4G # OR 6G

innodb_flush_log_at_trx_commit = 1

innodb_log_buffer_size = 8M

innodb_log_file_size = 512M

innodb_log_files_in_group = 2

innodb_file_per_table=1

innodb_flush_method=O_DIRECT




MySQL Monitoring




MySQL monitoring

•SHOW PROCESSLIST

– This command displays a real-timelist of all the connections to the server and the actions that eachone is performing

• INFORMATION_SCHEMA.PROCESSLIST -

• SHOW [GLOBAL|SESSION] STATUS – This command

provides server status information

• SHOW [GLOBAL|SESSION] VARIABLES – Thiscommand provides the values of MySQL system variables

• SHOW ENGINE INNODB STATUS – shows innodb specific

information




MySQL monitoring: mysqladmin

• SHOW GLOBAL STATUS – provides global information

since uptime• Real-time information:

• > mysqladmin .. -uroot -i 10 -r ex

• Will show interactive statistics per 10 seconds, example:

mysqladmin … -uroot -i 10 -r ex| grep tmp

| Created_tmp_disk_tables | 0 || Created_tmp_files | 5 || Created_tmp_tables | 543352 || Created_tmp_disk_tables | 0 |

| Created_tmp_files | 0 || Created_tmp_tables | 20 |

• 2 tmp tables per second




Slow Query Log

• Contains text of long running queries

• log queries executing longer than long_query_time server variable (in seconds, but

supports microseconds resolution when logging to file)

• Helps identify candidates for query optimization

• Enabling log•--log-slow-queries

or --log-slow-queries=file_name

• Can log non-indexed queries• --log-queries-not-using-indexes

• Written to log file or table

• slow_log table in mysql database

• Use --log-short-format option for less verbose logging




Show InnoDB Status: Buffer Pool

----------------------

BUFFER POOL AND MEMORY

----------------------

Total memory allocated 4391436288; in additional pool allocated 0

Buffer pool size 262143

Free buffers 262124

Database pages 19

Modified db pages 0




Show InnoDB Status: Row Operations

--------------

ROW OPERATIONS

--------------

0 queries inside InnoDB, 0 queries in queue

1 read views open inside InnoDB

Main thread process no. 31744, id 1189128544, state: waiting for server activity

Number of rows inserted 0, updated 0, deleted 0, read 0

0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s




Trick with InnoDB : “are we done yet?”• Delete from a where t1 > 2000000

• How long it will take? How much rows already deleted? What isthe speed?

• 4M rows in table, deleting half of rows

• Show innodb status:---TRANSACTION 0 97281, ACTIVE 21 sec …

1179 lock struct(s), .. undo log entries 650910

… delete from a where t1 > 2000000

• 1 undo log entries = 1 row

• Deleted 650K rows for 21 sec, avg. speed: 30K rows/sec

• 2M rows will be deleted for 66.6 sec• 1 min 6 sec – raw estimate

• mysql> delete from a where t1 > 2000000 ;

• Query OK, 2000000 rows affected (1 min 13.64 sec)




Monitoring GUI tools

• MySQL Enterprise Monitor

• Apply for trial: http://www.mysql.com/trials/

• MySQL Workbench

• http://www.mysql.com/products/workbench/ • Plugins for Nagios

• Lots of other tools

http://www.mysql.com/trials/

http://www.mysql.com/products/workbench/

http://www.mysql.com/products/workbench/

http://www.mysql.com/trials/




Query Optimization



MySQL Optimizer Overview

Query Performance 17.1 Query Optimization

MySQL DBMS

Query execution engine

Query Compiler

Optimizergenerate optimal Query Execution Plans

(QEP

’s)

ParserAnalyze

Syntax

Preprocessor

–Semantic checking, name

resolution

Query Transformations

Search for optimal execution plan

Plan refinement

Query Compiler

OptimizerGenerate Optimal Query Execution Plans

(

QEP’s)

ParserAnalyze Syntax

Preprocessor –Semantic checking, name resolution

Query Transformations

Search for optimal execution plan

Plan refinement

QEP Plan

Parse

Tree

Query

Execution




Using Indexes for Optimization

• Benefits of indexes• Contain sorted values

• Result in less disk I/O

• MySQL supports three general types of indexes

• Primary key

• Unique index

• Non-unique index

• Specialized indexes• FULLTEXT

• SPATIAL




Principles for Index Creation

• Use NOT NULL if possible

• Avoid “over-indexing”

• Over-indexing may slow down writes

• Avoid creating redundant indexes

• Estimate whether indexing is efficient• Indexes that are not selective enough will not be used (efficiently)

• Choose unique and non-unique indexes appropriately

• Make indexes as small as possible

• Index column prefix rather than entire column

• Use a single ALTER TABLE to perform creation/alteration of multiple

indexes (rebuild the table only once)




Composite Indexes

• Composite indexes facilitate quick lookup

• MySQL can use left-most part of any index

• Leftmost prefix of an index is not the same thingas an index on a column prefix




Leftmost Index Prefixes (2/2)

• SHOW INDEX displays composite index information mysql> SHOW INDEX FROM CountryLanguage\G*************************** 1. row ***************************

Table: CountryLanguage Non_unique: 0

Key_name: PRIMARYSeq_in_index: 1Column_name: CountryCodeCollation: A

Cardinality: NULLSub_part: NULLPacked: NULL Null:

Index_type: BTREEComment:

*************************** 2. row ***************************Table: CountryLanguage

Non_unique: 0Key_name: PRIMARY

Seq_in_index: 2

Column_name: LanguageCollation: A

Cardinality: 984Sub_part: NULLPacked: NULL Null:

Index_type: BTREEComment:




Duplicate Indexes

• Example 1CREATE TABLE `dupl_index` (

ì` int(11) NOT NULL,

à` int(11) DEFAULT NULL,

PRIMARY KEY (ì`),

UNIQUE KEY ì` (ì`), DUPLICATE KEY

KEY ì_2` (ì`)

DUPLICATE KEY• Example 2CREATE TABLE `dupl_index1` (

à` int(11) DEFAULT NULL,

`b` int(11) DEFAULT NULL,

`c` int(11) DEFAULT NULL,

KEY ì1` (à`,`b`,`c`),

KEY ì2` (à`,`b`), DUPLICATE KEY

KEY ì3` (à`) DUPLICATE KEY




Using EXPLAIN to Analyze Queries

• Find out how the query optimizer would improve a

SELECT query

• Useful optimizer information

• Points out need to index

• Finds out if optimizer is using existing indexes

• Can help qualify query rewrites

• Can also be used with UPDATE and DELETE




How EXPLAIN Works (2/3) • Shows the efficiency of SELECT statements mysql> EXPLAIN SELECT * FROM Country WHERE Name = 'France'\G

*************************** 1. row ***************************

id: 1select_type: SIMPLE

table: Countrytype: ALL

possible_keys: NULLkey: NULL

key_len: NULLref: NULLrows: 239Extra: Using where

mysql> EXPLAIN SELECT * FROM Country WHERE Code = 'FRA'\G*************************** 1. row ***************************

id: 1

select_type: SIMPLEtable: Countrytype: const

possible_keys: PRIMARYkey: PRIMARY

key_len: 3ref: constrows: 1Extra:




How EXPLAIN Works (3/3)

• Performs a “dry run” of the query

• Indicates number of rows which need to be

examined during processing the query

• EXPLAIN is especially important for join analysis




Analyzing a Query (1/9)

• Example of using EXPLAIN to analyze and optimize

a query

• Starting out with no indexes ...

mysql> DESC CountryList;+-------+----------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+-------+----------+------+-----+---------+-------+| Code | char(3) | NO | | | || Name | char(52) | NO | | | |+-------+----------+------+-----+---------+-------+

mysql> DESC CityList;+-------------+----------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |

+-------------+----------+------+-----+---------+-------+| CountryCode | char(3) | NO | | | || Name | char(35) | NO | | | || Population | int(11) | NO | | 0 | |+-------------+----------+------+-----+---------+-------+...





• Query used:

mysql> SELECT Co. Name, Ci.Name, Ci.Population-> FROM CountryList Co, CityList Ci

-> WHERE Co.Code = Ci.CountryCode-> AND Ci.Population > 8000000;

• Obtain the query execution plan: mysql> EXPLAIN

-> SELECT Co. Name, Ci.Name, Ci.Population

-> FROM CountryList Co, CityList Ci-> WHERE Co.Code = Ci.CountryCode-> AND Ci.Population > 8000000\G





• EXPLAIN output:*************************** 1. row ***************************


table: Cotype: ALL


key_len: NULL

ref: NULLrows: 209Extra:

*************************** 2. row ***************************id: 1

select_type: SIMPLEtable: Citype: ALL


key_len: NULLref: NULLrows: 4018Extra: Using where; Using join buffer





• Example (continued)

• EXPLAIN shows the need for indexing:

• type: ALL – all rows are retrieved

• possible_keys: NULL – there are no indexes whatsoever

• Extra: Using where – criteria tested on data of each

scanned row

• Extra: Using join buffer – A row cache is used to

scan the table for each row of previous table

• Good columns to index are those used for searching, grouping

and sorting

• Use Co.Code and Ci.CountryCode to match rows

• Uses CityList.Population to cull rows





• Adding indexes:• ALTER TABLE CountryList ADD PRIMARY KEY (Code)

+-------+----------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+-------+----------+------+-----+---------+-------+

| Code | char(3) | NO | PRI | | || Name | char(52) | NO | | | |+-------+----------+------+-----+---------+-------+

• ALTER TABLE CityList ADD INDEX (CountryCode)

+-------------+----------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |

+-------------+----------+------+-----+---------+-------+| CountryCode | char(3) | NO | MUL | | || Name | char(35) | NO | | | || Population | int(11) | NO | | 0 | |+-------------+----------+------+-----+---------+-------+


A l i Q (6/9)




• EXPLAIN after adding indexes:*************************** 1. row ***************************

id: 1select_type: SIMPLEtable: Cotype: ALL

possible_keys: PRIMARYkey: NULL

key_len: NULLref: NULL

rows: 225Extra:

*************************** 2. row ***************************id: 1

select_type: SIMPLEtable: Citype: ref

possible_keys: CountryCodekey: CountryCodekey_len: 3

ref: world.Co.Coderows: 7Extra: Using where





• Adding an index for Population too changes the order:*************************** 1. row ***************************


table: Citype: range

possible_keys: CountryCode,Populationkey: Population

key_len: 4

ref: NULLrows: 10Extra: Using where

*************************** 2. row ***************************id: 1

select_type: SIMPLEtable: Cotype: eq_ref

possible_keys: PRIMARYkey: PRIMARY

key_len: 3ref: world.Ci.CountryCoderows: 1Extra:





• Example (continued)• For CityList:

• possible_keys: CountryCode, Population – two

indexes are considered

• key: Population – the index called Population will be used• type: range – only a partial index scan is required

• rows: 10 – the scan likely yields only 10 rows

• For CountryList

• type: eq_ref – values from the previous table are used toperform single row lookups in this table

• In this case the range scan drastically reduced number of rows,

reversing the join order as compared to the previous queries




EXPLAIN for Table Processing (1/2)

• The Extra column provides process information

• Can indicate an efficient or inefficient query

• Efficient query Extra output

• Using index (index sufficient to obtain all data without als looking up

table data)

• Using where (rows tested for criteria before joining with next table)

• Distinct (only unique value combinations are required instead of

scanning all rows)

• Not exists (scan for an outer joined table that is required not to exist

can be skipped once the first row is found to match)




EXPLAIN for Table Processing (2/2)

• Indicators for inefficient query Extra output• Using filesort (extra pass to sort rows required to retrieve

actual rows in sorted order)

• Using temporary (result copied to temporary table, indicatesORDER BY and GROUP BY with different implied row order)

• Using join buffer (repeated scans buffered, might indicate amissing join condition or index)

• Indicators of poor performance

• Range checked for each record (a series of range scans

are performed preventing efficient caching of scanned rows)

• Just indicators – it’s not a black and white thing


H A id “U i Fil ”?



How to Avoid “Using Filesort”?

• Covered (combined) index can help. Before:CREATE TABLE `City` (`ID` int(11) NOT NULL AUTO_INCREMENT,`Name` char(35) NOT NULL DEFAULT '',`CountryCode` char(3) NOT NULL DEFAULT '',`District` char(20) NOT NULL DEFAULT '',`Population` int(11) NOT NULL DEFAULT '0',PRIMARY KEY (`ID`),

KEY `CountryCode` (`CountryCode`)

mysql> explain select name, District from City whereCountryCode ='USA' order by Population desc limit 10\G

table: Citytype: ref

possible_keys: CountryCode

key: CountryCodekey_len: 3

ref: constrows: 267

Extra: Using where; Using filesort


H t A id “U i Fil t”?



How to Avoid “Using Filesort”?

• After:

mysql> alter table Cityadd key comb_ind(CountryCode, Population);

mysql> explain select name, District from City

where CountryCode ='USA'order by Population desc limit 10\G...


possible_keys: CountryCode,comb_ind

key: comb_ind key_len: 3ref: constrows: 267Extra: Using where


F ll d i d



Full covered index

mysql> alter table City add key

covered_ind (CountryCode, Population, name, District);

mysql> explain select name, District from Citywhere CountryCode ='USA'order by Population desc limit 10\G...


possible_keys: CountryCode,comb_ind,covered_ind key: covered_ind

key_len: 3ref: const

rows: 457

Extra: Using where; Using index


E t R d Q&A



Event, Resources and Q&A

• MySQL Sunday at Oracle Open World, 19 Sept 2010 –

http://www.oracle.com/us/openworld/mysql-sunday-078000.html• “Getting the Best MySQL Performance in Your Products:

Part 1, The Fundamentals” replay – http://mysql.com/news-and-events/on-demand-webinars/display-od-

552.html

•

mysql.com/performance – http://mysql.com/why-mysql/performance/index.html

• Webinar replay – See: “Embedded Server for ISVs and OEMs” section on

http://mysql.com/news-and-events/on-demand-webinars/

•

Questions? – http://www.mysql.com/about/contact/sales.html?s=oem

– Phone: USA=+1-866-221-0634; Outside USA = +1-208-327-6494




The presentation is intended to outline our general

product direction. It is intended for information

purposes only, and may not be incorporated into any

contract. It is not a commitment to deliver any

material, code, or functionality, and should not berelied upon in making purchasing decisions.

The development, release, and timing of any

features or functionality described for Oracle’s

products remains at the sole discretion of Oracle.

best perf in products part2 arubin 15 sept 2010

Documents