mysql performance tuning: the perfect scalability (hol3025)

Copyright © 2019 Oracle and/or its affiliates.

MySQL Performance Tuning:

The Perfect Scalability

(HOL3025)

MySQL Principal Technical Support Engineer

Oracle MySQL SupportSeptember 17, 2019

Mirko Ortensi

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation.

Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events.

Safe Harbor


http://www.oracle.com/investor

Our Goal

When dataset and concurrency increase, horizontal and vertical scaling can address service growth. But in order to achieve perfect scalability, I/O, CPU and Network bound workloads can meet additional improvements at no expense to get the most out of MySQL Server.


Why This Matters

Scaling up and out a database implies more performing storage, more processing capacity, additional memory and higher bandwidth. Keeping adjustements to infrastructure as little as possible helps keeping operational expenditure limited.


“Query is slow


Anonymous


Mirko Ortensi


Agenda

• Horizontal vs Vertical scaling

• I/O Bound Workload

• CPU Bound Workload

• Network Bound Workload

• OS-wise

• Conclusions


Data and Concurrency grow

Service is growing: more data and more active connections will grow the database «old»


THREAD

THREAD

THREAD

THREADCLIENT

CLIENT

CLIENT

CLIENT

THREAD

THREAD

I/O, CPU and NETWORK bound

More locks + scans Buffer pool should be increased

More connections threads

I/O throughput


Agenda





• OS-wise

• Conclusions



Scaling out READSWith additional slaves

MySQL Router

WRITEREADREPL


Scaling out READSWith additional slaves

• Create slave server (clone(), MEB, mysqldump)

• Distribute read and write operations accordingly

• Slaves can be delayed


Scaling out READSWith InnoDB Cluster

MySQL Router

WRITEREADREPL


Scaling out READSWith InnoDB Cluster

• Highly available with automatic failover

• Integrated solution to scale and balance reads

• Up to 9 replicas

• Writes do not scale (single or multi master)

• Read your own writes


Scaling out WRITESSharding

• MySQL Server does not support sharding

• Only auto-sharding solution is NDB Cluster

• Custom / third partyID SERVICE DATA

1773467253 chat xxx

6257346892 location xxx

1773467253 chat xxx

7234782739 photos xxx

8437829249 location xxx


Scaling Up?

• More sockets / cores

• Faster disks / SSD

• More memory

• More bandwidth

MySQL Multiplies!

Take advantage of:• Multiple storages• Thread pooling• Parallel slave

And tune as usual:• Configuration• Queries


CLIENT

CLIENT

CLIENT

CLIENT

THREAD

THREAD

THREAD

CLIENT

CLIENT

WORKER

WORKER

MASTER

1 2 3


If Data and Concurrency grows, "MySQL multiplies"

IO bound• Single storage• Temporary tables• Set to prevent

data loss• Insufficient

Buffer pool• Write workload

CPU bound• Active

connections• Redundant

indexes• sort/ joins• Scans• Background tasks

Network bound• Uncompressed

• Huge recordsets

• Linux (native/VM)

• MySQL 8

• sysstat

• world.sql https://bit.ly/2kO70nY

HOL3025


https://bit.ly/2kO70nY

Agenda





• OS-wise

• Conclusions



I/O BoundMonitoring

• iostat

• SHOW ENGINE INNODB STATUS

• Performance schema (MEM)

• SYS schema

select * from sys.io_by_thread_by_latency;

select * from sys.io_global_by_file_by_bytes;

select * from sys.io_global_by_file_by_latency;

select * from sys.waits_global_by_latency;

select * from sys.statements_with_temp_tables;

Reconfigure Move ib_logfile files Permissions START MySQL


I/O BoundREDO Log

Stop the instance, edit Custom innodb_log_group_home_dir and save changes.

From data directory to the new directory, The files are usually ib_logfile0 and ib_logfile1, but they could be more, depending on the value of innodb_log_files_in_group

Make sure that the MySQL server has the proper access rights to read/create files in the log directory.

It will read the configuration file with the new location.

https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_log_group_home_dir

https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_log_files_in_group

On storage devices with cache, data loss is possible if data filesand redo log files reside on different storage devices, and a crash occurs before data file writes are flushed from the device cache.

If you use or intend to use different storage devices for redo logs and data files, use O_DIRECT instead.

I/O BoundREDO Log


I/O BoundBinary Logs


Custom log-bin

1. Stop: systemctl stop mysqld

2. Create: /opt/mysql/binlog

3. Edit my.cnf: log_bin=/opt/mysql/binlog/binlog

4. Move binlog.* to /opt/mysql/binlog

5. Edit: binlog.index to adopt absolute paths (e.g. /opt/mysql/binlog/binlog.000011)

6. Start: systemctl start mysqld

https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#option_mysqld_log-bin

I/O BoundBinary Logs


TEST

1. FLUSH BINARY LOGS;

2. SELECT @@log_bin_basename;

3. RESET MASTER;

I/O BoundUNDO Logs


Custom innodb_undo_directory

1. Stop: systemctl stop mysqld

2. Create: /opt/mysql/undo

3. Relocate undo_00*

4. Start: systemctl start mysqld

5. Check:

1. select @@innodb_undo_directory

https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_undo_directory

I/O BoundEXTERNAL tablespace


• Creating a file-per-table tablespace outside of the data directory.

• The DATA DIRECTORY clause cannot be used with ALTER TABLE to change the location later.

• CREATE TABLE..DATA DIRECTORY

TESTmysql> USE test;Database changed

mysql> CREATE TABLE t1 (c1 INT PRIMARY KEY) DATA DIRECTORY = '/opt/mysql/tablespaces';

https://dev.mysql.com/doc/refman/8.0/en/glossary.html#glos_file_per_table

https://dev.mysql.com/doc/refman/8.0/en/tablespace-placing.html

I/O BoundGENERAL tablespace


• Acts as a container of tablespaces, same as system tablespace

• Can be located outside of the data directory

• From file-per-table or system to general tablespace (ALTER)

• From system or general (shared) to file-per-table (ALTER)

• Directory must be known: set innodb_directories (read-only)

TEST

CREATE TABLESPACE `ts1` ADD DATAFILE '/opt/mysql/tablespaces/ts1.ibd' Engine=InnoDB;

CREATE TABLE t1 (c1 INT PRIMARY KEY) TABLESPACE ts1;

I/O BoundPartitioning


• Horizontal partitioning—that is, different rows of a table may be assigned to different physical partitions.

• Partitioning makes it possible to store more data in one table than can be held on a single disk or file system partition.

TEST

CREATE TABLE th (id INT, name VARCHAR(30), adate DATE)PARTITION BY LIST(YEAR(adate))(PARTITION p1999 VALUES IN (1995, 1999, 2003)DATA DIRECTORY = '/opt/mysql/tablespaces/',PARTITION p2000 VALUES IN (1996, 2000, 2004)DATA DIRECTORY = '/opt/mysql/tablespaces/');

I/O BoundSession and Global Temporary Tablespace


• If no value is specified for innodb_temp_data_file_path, the default behavior is to create an auto-extending data file named ibtmp1 in the innodb_data_home_dir (system tablespace location)

• Not a dynamic setting

• At database start a pool of session 10 temporary tables is created

• At database start, ibtmp1 is created to

• Temp tables rollback segments in ibtmp1

• Example:--innodb-temp-data-file-path=../../../tmp/ibtmp1:12M:autoextend

https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_temp_data_file_path

https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_data_home_dir

I/O BoundSession and Global Temporary Tablespace


TEST

• mysql> select @@innodb_temp_data_file_path;

• mysql> SELECT * FROM INFORMATION_SCHEMA.FILES WHERE TABLESPACE_NAME='innodb_temporary'\G

• shell> ls –l <location>

• Stop Server and list directory again

I/O BoundLogging

General loggeneral_log_file

Slow Query Logslow_query_log_file

Audit Logaudit_log_file


https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_general_log_file

https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_slow_query_log_file

https://dev.mysql.com/doc/refman/8.0/en/audit-log-reference.html#sysvar_audit_log_file

I/O BoundFlush method


Data can travel through different layer before reaching the durable storage.

I/O BoundFlush method


Flush method: innodb_flush_method (not dynamic)

• fsync is default: fsync is used for both data and log file

• O_DIRECT can help to avoid double buffering between the InnoDB buffer pool and the operating system file system cache. But is buffer pool is not enough, buffer cache won’t come to help. fsync is used for both data and log file

• O_DIRECT_NO_FSYNC same as before, but skips fsyncs

• O_DSYNC: InnoDB uses O_SYNC to open and flush the log files, and fsync() to flush the data files, only. Based on O_SYNC: nfsync(), write() calls go to buffer cache, blocks until writes are into the physical storage.

https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_flush_method

I/O BoundFlush method monitoring


TEST 1

1. sync; echo 1 > /proc/sys/vm/drop_caches

2. vmstat 1 30

3. mysqlslap -uroot -h127.0.0.1 -uroot -p --auto-generate-sql --concurrency=20 --iterations=100 —number-of-queries=10

TEST 2

At different innodb_flush_method:

• mysql> show status like "Innodb_data_fsyncs";

• mysql> show status like "Innodb_data_pending_fsyncs";

• bash> free -m

https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_flush_method

I/O BoundMiscellaneous


An all-comprehensive discussion in our official online documentation.

• Disable logging of compressed pages

• Act on I/O capacity for background tasks: innodb_io_capacity and innodb_flush_sync

• Threshold on write buffer before fsync(): innodb_fsync_threshold

• Data size

• ...

https://dev.mysql.com/doc/refman/8.0/en/optimizing-innodb-diskio.html

https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_io_capacity

https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_flush_sync

https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_fsync_threshold

https://dev.mysql.com/doc/refman/8.0/en/data-size.html

Agenda





• OS-wise

• Conclusions



CPU BoundMonitoring

• vmstat / mpstat

• Status

SHOW STATUS LIKE "Handler%";

SHOW STATUS LIKE "select_full_join";

• sys schema

select * from sys.schema_redundant_indexes;

select * from sys.schema_unused_indexes;

select * from sys.statements_with_sorting;

CALL sys.diagnostics(60, 60, 'current');

https://dev.mysql.com/doc/refman/8.0/en/server-status-variables.html#statvar_Handler_commit


CPU BoundThread Pool

High CPU may also be a good sign, no long IO wait or heavy locking. But can also be for context switches by CPU scheduler.

1. Too many thread stacks make CPU caches almost useless in highly parallel execution workloads

2. With too many threads executing in parallel, context switching overhead is high

Thread pool promotes thread stack reuse

The thread pool controls when transactions start to ensure that not too many execute in parallel.


CPU BoundQuery Tuning

TEST1. Import world.sql

2. Monitor: mpstat 1 30

3. mysqlslap -uroot -p --no-drop --create-schema=world --query="select * from world.city where Name='Kabul';" --concurrency=10 --iterations=1000

4. EXPLAIN SELECT * FROM world.city WHERE Name='Kabul';

5. SELECT * FROM sys.schema_tables_with_full_table_scans;

6. CREATE INDEX city_idx ON world.city(Name);

7. Repeat the test


CPU BoundReplication

• In 5.6 we introduced parallel slave, that could run load for different databases in parallel. With operations split on several DBs, this could be a simple option.

• In 5.7 we introduced LOGICAL_CLOCK algorithm, that allows running many transactions in parallel if they were run in parallel on master.

• In 8.0 version we have introduced write sets, that allows to run some statements in parallel, even if they were executed by the same thread on master.


CPU BoundMiscellaneous

• Tablespace compression

• Protocol compression

• Thread cache

• SSL

• Background

CheckpointPurgeAutomatic Statistics Calculation

Agenda





• OS-wise

• Conclusions



NETWORK BoundCompression

--compress: Compress all information sent between the client and the server if possible

TEST• mysql –uroot –compress –p

• mysqlslap -uroot -h127.0.0.1 -uroot -p --auto-generate-sql --concurrency=20 --iterations=100 —number-of-queries=10 –compress

• sar -n DEV 3 30

• mpstat 1 30

Look at averages

https://dev.mysql.com/doc/mysql-shell/8.0/en/mysqlsh.html#option_mysqlsh_compress

Agenda





• OS-wise

• Conclusions



OS-wiseSWAP, SWAPPINESS and AFFINITY

SWAP & SWAPPINESScat /proc/sys/vm/swappiness

AFFINITY

At the linux OS level, you can limit the number of processors that the kernel will utilize for a user space process using numactl, or more commonly taskset.


OS-wiseSWAP, SWAPPINESS and AFFINITY

TEST• Identify systemd configuration file: /etc/systemd/system/multi-user.target.wants/mysqld.service

• Modify:

[Service]

CPUAffinity=1

• bash> systemctl daemon-reload

https://dev.mysql.com/doc/refman/8.0/en/using-systemd.html

CPU monitor Press f Last CPU used Esc

OS-wiseAFFINITY

top -H -p <pid> To get the list of options

Scroll down and select "P = Last Used Cpu(SMP)", press space bar

Column on the right side will show what CPU has just been used


Agenda





• OS-wise

• Conclusions


Conclusion

Setup the most reliable production-like traffic generator

Test and monitor at different concurrencies

Experiment!


Thank You



Oracle MySQL SupportSeptember 17, 2019

Mirko Ortensi

Session Survey

Help us make the content even better. Please complete the session survey in the Mobile App.


What’s Ahead

Tuesday

5pm-5.45pm

Thursday


NDB, Simply the World’s Highest-

Throughput Transactional Datastore[DEV2554]Moscone South - Room 205

10am-10:45am What’s New in MySQL Optimizer and

Executor? [DEV2077]Moscone South - Room 205

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation.

Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019and Oracle undertakes no duty to update any statement in light of new information or future events.

Safe Harbor


http://www.oracle.com/investor

mysql performance tuning: the perfect scalability (hol3025)

Documents