upcoming changes in mysql 5.7

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |

MySQL 5.7: Upcoming Changes

Morgan Tocker MySQL Community Manager February 2015


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

3

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |4

Today’s Agenda

Introduction

New Features

Changes to Prepare For

1

2

3


MySQL 5.7 -‐ The Team• 2x Engineering Staff • 3x QA Staff • 2x Support Staff

5

Jan 2010 Sept 2014


Development Milestone Releases• Features are merged into main branch only when stable. • High level review complete. • Low level review complete. • 80%+ code coverage testing goal.

• DMRs are intended to be of “Release Candidate” Quality.

6


Labs Releases• Features have not passed qualification. • We are releasing an experiment because we want to insert an extra feedback cycle.

• Current Labs Releases: • New Data Dictionary • Optimizer/InnoDB/Replication • HTTP Plugin • Group Replication

7


MySQL 5.7 DMR Schedule

8

April 2013 DMR1 221 Bugs Fixed 29 Changes 21 Worklogs

Sep 2013 DMR2 384 Bugs Fixed 41 Changes 52 Worklogs

Dec 2013 DMR3 155 Bugs Fixed 23 Changes 27 Worklogs

April 2014 DMR4 259 Bugs Fixed 38 Changes 56 Worklogs

5.6 GA Feb 2013

Sept 2014 DMR5 300 Bugs Fixed 50 Changes 79 Worklogs


Model Successful in 5.6

9

https://www.flamingspork.com/blog/2013/03/05/mysql-code-size/ Also See: http://www.mysqlperformanceblog.com/2013/01/27/mysql-5-6-improvements-in-the-nutshell/

https://www.flamingspork.com/blog/2013/03/05/mysql-code-size/

http://www.mysqlperformanceblog.com/2013/01/27/mysql-5-6-improvements-in-the-nutshell/


Today’s Agenda

Introduction

New Features


1

2

3

2

1


Common Themes

11

From DMR0 to DMR5

▪ Improved support for high connection creation rate

▪ Lock Thread count impact reduced ▪Multiple Metadata Locking (MDL) improvements ▪ Parallel Replication (Intra-schema) ▪ Additional query optimization for

Fulltext Search ▪ Make use of condition filtering in the

optimizer

▪ InnoDB Improvements for: ▪ Index lock contention ▪ Read Only Transactions ▪ Temporary Tables ▪ Redo Logs ▪ Multiple page cleaner threads. ▪ Scalability ▪ Bulk Loading Indexes

Performance & Scalability


Common Themes

12

From DMR0 to DMR5

▪ Overhead has been reduced. ▪ Improved to do lazy initialization on

new connections. ▪ Now instruments:

▪ Memory usage ▪ Stored programs ▪ Replication ▪ Metadata Locking (MDL) ▪ Transactions ▪ Prepared Statements ▪ User Variables ▪ Stage Progress ▪ SX-lock

Performance Schema▪ Password expiration support +

timestamp of password last changed. ▪ mysql_secure_installation changed

to C++ program. ▪ Improvements to SSL Support. ▪ Support to change AES key length

and block encryption mode

Security


Common Themes (cont.)

13

From DMR0 to DMR5

▪Internal logging API ▪Timestamps are now written with log entries in YYYY-MM-DDThh:mm:ss.uuuuuu format. ▪Configurable Verbosity

▪InnoDB Refactored: ▪Locking Code ▪Redo Log Code (for performance) ▪Tablespace Class ▪Atomics refactored

▪Optimizer refactoring (multiple improvements) ▪Optimizer cost model ▪mysql_install_db changed to C++ program.

▪Strict and Ignore defined and reimplemented. ▪Remove custom rwlock windows implementation ▪Refactor strings: use of (const char *) in THD ▪Using Boot.Geometry ▪Better ONLY_FULL_GROUP_BY mode ▪Refactor mysql_upgrade ▪Move system tables from MyISAM to transactional storage ▪Make sure that when 32-bit connection-IDs roll over, still in-use IDs are not re-issued ▪Memroot allocator for C++ STL

Refactoring


Benchmarks

14

http://dimitrik.free.fr/blog/archives/2014/10/indeed-‐mysql-‐57-‐rocks-‐oltp_rorw-‐1table-‐benchmarks.html

http://dimitrik.free.fr/blog/archives/2014/10/indeed-mysql-57-rocks-oltp_rorw-1table-benchmarks.html


Benchmarks (cont.)• 645K Queries/Second is the new peak.

15

http://dimitrik.free.fr/blog/archives/2014/09/indeed-‐mysql-‐57-‐rocks-‐oltp_ro-‐pointselects-‐8tables-‐benchmark.html

http://dimitrik.free.fr/blog/archives/2014/09/indeed-mysql-57-rocks-oltp_ro-pointselects-8tables-benchmark.html


DMR0

• Error issued when creating duplicate index. • Warning when strict SQL Mode removed. • Ctl-‐C fixed in MySQL CLI! • Non-‐blocking SHOW SLAVE STATUS • Idempotent mysqlbinlog mode. • Stacked Diagnostics Area

16

Internal Release Only


DMR1

• Online DDL Now supports RENAME INDEX and VARCHAR extensions.

• InnoDB support for Spatial Data Types. • Rewrite-‐db option for mysqlbinlog with RBR.

17

Released April 2013


DMR2

• Semi sync replication improvements. • Multiple Triggers per event per table. • EXPLAIN FOR CONNECTION • EXPLAIN showing cost information.

18

Released September 2013


Example: Cost Information in EXPLAIN

19

EXPLAIN FORMAT=JSON FOR CONNECTION 2;*************************** 1. row *************************** EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "4786.00" }, "ordering_operation": {

"using_temporary_table": true, "using_filesort": true, "cost_info": { "sort_cost": "2151.00" }, "nested_loop": [ { "table": { .. ], "key_length": "3", "rows_examined_per_scan": 239, "rows_produced_per_join": 239, "filtered": 100, "using_index": true, "cost_info": { "read_cost": "6.00", "eval_cost": "47.80", "prefix_cost": "53.80", "data_read_per_join": "61K" },

..


Example: EXPLAIN in MySQL Workbench

20

Visual EXPLAIN Including Cost Information!


DMR3

• UNION ALL no longer requires temporary table. • Filesort now packs VARCHAR and CHAR keys in temporary files.

• Optimizer Support for row value constructors. • i.e. WHERE (a,b) in ((‘a’, ‘b’), (‘c’, ‘d’));

21

Released December 2013


DMR3 (cont.)• InnoDB Fulltext now supports external parsers. • Dynamic replication filters. • Support ICP optimization for partitioned tables.

22


UNION ALL Optimization

23

***************** 1. row ***************** id: 1 select_type: PRIMARY table: table_a type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 3 Extra: NULL***************** 2. row ***************** id: 2 select_type: UNION table: table_b type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 3 Extra: NULL***************** 3. row ***************** id: NULL select_type: UNION RESULT table: <union1,2> type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: NULL Extra: Using temporary

***************** 1. row ***************** id: 1 select_type: PRIMARY table: table_a partitions: NULL type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 3 filtered: 100.00 Extra: NULL***************** 2. row ***************** id: 2 select_type: UNION table: table_b partitions: NULL type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 3 filtered: 100.00 Extra: NULL

EXPLAIN SELECT * FROM a UNION ALL SELECT * FROM bMySQL 5.6: MySQL 5.7:


Row-‐value Constructors

24

EXPLAIN SELECT * FROM City WHERE (CountryCode,Name) IN (('USA', 'San Francisco'), ('USA', 'Chicago'), ('CAN', ‘Toronto'))

**************** 1. row **************** id: 1 select_type: SIMPLE table: City type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 4188 Extra: Using where1 row in set (0.00 sec)

**************** 1. row **************** id: 1 select_type: SIMPLE table: City partitions: NULL type: rangepossible_keys: CountryCode,CountryCode$Name key: CountryCode$Name key_len: 38 ref: NULL rows: 3 filtered: 100.00 Extra: Using where

MySQL 5.6: MySQL 5.7:


DMR4

• Statement Timeout. • Change master without stopping SQL thread. • Transportable tablespaces for partitioned InnoDB tables.

25

Released April 2014


Server-‐Side Statement Timeout

• SELECT MAX_STATEMENT_TIME = 109 * FROM my_table;

• SET SESSION MAX_STATEMENT_TIME = 100; • SET GLOBAL MAX_STATEMENT_TIME = 100;

26

Incorporates Contributions from Davi Arnaut, the MySQL Community


DMR5

• Online UNDO log truncation • Online Buffer Pool Resize • Partitioning Enhancements • InnoDB GIS support • Support InnoDB as additional storage engine for tmp table

• Method to bring servers offline

27



DMR5 (cont.)

• Native support for syslog • Allow multiple locks in GET_LOCK() • Improve dtrace support on Oracle Linux 6 • New Optimizer Cost Model

28



Offline Mode

29

mysql> select * from a;+----+| id |+----+| 2 || 2 || 3 |+----+3 rows in set (0.00 sec)

mysql> select * from a;No connection. Trying to reconnect...ERROR 1917 (HY000): The server is currently in offline modeERROR:Can't connect to the server

mysql> set global offline_mode = 1;Query OK, 0 rows affected (0.01 sec)

Session #1 Session #2


Online Buffer Pool Resize

30

mysql> SET GLOBAL innodb_buffer_pool_size = 256 * 1024 * 1024;Query OK, 0 rows affected (0.00 sec)

mysql> SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_resize_status'\G*************************** 1. row ***************************Variable_name: Innodb_buffer_pool_resize_status Value: Completed resizing buffer pool at 141107 16:33:12.1 row in set (0.00 sec)

mysql> SHOW GLOBAL VARIABLES LIKE 'innodb_buffer_pool_size';+-------------------------+-----------+| Variable_name | Value |+-------------------------+-----------+| innodb_buffer_pool_size | 268435456 |+-------------------------+-----------+1 row in set (0.00 sec)


DMR6

• Release notes starting to show features • Fulltext support for CJK • MySQL Install DB folded into server • More improvements in optimizer, performance schema, InnoDB.

31

Not Yet Released


Today’s Agenda

Introduction

New Features


1

22

1

2

3


• Change of Defaults

• Command Line Client

• Replication

• InnoDB

• Optimizer

• Errors, Warning and Logging

• Performance Schema

• SQL Modes

• Other

• Deprecation

• ENCODE/DECODE functions

• SHOW PROFILES

33

Functionality Changed• Behaviour Change

• STRICT

• ONLY_FULL_GROUP_BY

• IGNORE

• DELAYED

• EXPLAIN

• Removal

• SHOW ENGINE INNODB MUTEX

• InnoDB Monitor Tables

• ALTER IGNORE TABLE

• Unused CLI Utilities

• YEAR(2) support

• Old Passwords


Behaviour is different, but previous behaviour can be restored.

Change of Defaults

34


Command Line Client

• Two small changes to improve default experience:

35

[mysql] prompt="\u@\h [\d] > “ show_warnings


Replication

• Safer by default. • Reduces timeouts, which are now also used for server heartbeats.

36

[mysqld] sync_binlog=1binlog_format=ROWslave_net_timeout=60binlog_error_action=ABORT_SERVERbinlog_gtid_recovery_simplified=1


InnoDB

37

[mysqld] innodb_autoinc_lock_mode=2innodb_checksum_algorithm=crc32innodb_page_cleaners=4innodb_purge_threads=4innodb_strict_mode=1innodb_log_file_size=128Minnodb_buffer_pool_dump_at_shutdown=1innodb_buffer_pool_load_at_startup=1innodb_buffer_pool_dump_pct=25innodb_file_format = Barracudainnodb_large_prefix=1


Optimizer

• New Optimizations Enabled by Default • Setting for ranged queries to use statistics over sampling changed in response to feedback

38

[mysqld] eq_range_index_dive_limit=200


Errors, Warnings & Logging

39

[mysqld] log_slow_admin_statements=1log_slow_slave_statements=1long-query-time=2log-queries-not-using-indexes=1min-examined-row-limit=1000log_warnings=2


Performance Schema

• performance-‐schema-‐consumer-‐events_statements_history=ON

• performance-‐schema-‐consumer-‐events_transactions_history=ON

40


SQL Modes

• Fewer surprises caused by how edge cases are handled: • Very good for new applications • May need to be disabled for those upgrading*

41

[mysqld] sql-mode=“ONLY_FULL_GROUP_BY, STRICT_TRANS_TABLES, NO_ENGINE_SUBSTITUTION, NO_AUTO_CREATE_USER"

* http://www.tocker.ca/2014/09/01/suggestions-‐for-‐transitioning-‐into-‐strict-‐sql-‐mode.html

http://www.tocker.ca/2014/09/01/suggestions-for-transitioning-into-strict-sql-mode.html


SQL Modes (cont.)

42

mysql> SELECT id, invoice_id, description FROM invoice_line_items GROUP BY invoice_id;+----+------------+-------------+| id | invoice_id | description |+----+------------+-------------+| 1 | 1 | New socks || 3 | 2 | Shoes || 5 | 3 | Tie |+----+------------+-------------+3 rows in set (0.00 sec)

ERROR 1055 (42000): Expression #3 of SELECT list is not in GROUP BY clause and contains nonaggregated column ‘invoice_line_items.description’ which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by

Only full group by


SQL Modes (cont.)

43

STRICT for transactional tables

CREATE TABLE test (a int unsigned);INSERT INTO test VALUES (-1);ERROR 1264 (22003): Out of range value for column 'a' at row 1insert into test values (0/0);ERROR 1365 (22012): Division by 0

CREATE TABLE test2 (a varchar(10));INSERT INTO test2 VALUES ('abcdefghijklmnopqrstuvwxyz');ERROR 1406 (22001): Data too long for column 'a' at row 1

CREATE TABLE test3 (a datetime);INSERT INTO test3 VALUES ('0000-00-00 00:00:00');ERROR 1292 (22007): Incorrect datetime value: '0000-00-00 00:00:00' for column 'a' at row 1


Other

44

[mysqld] # default buffer sizesgroup_concat_max_len=1Mmax_allowed_packet=64M# securitysymbolic-links=0


Still present, but usage is now discouraged.

Deprecated

45


ENCODE/DECODE Functions

46

mysql> SELECT HEX(ENCODE('MyPassword', 'Super Secret!')); +--------------------------------------------+ | HEX(ENCODE('MyPassword', 'Super Secret!')) | +--------------------------------------------+ | A0E844F97A48FE6E8C1F | +--------------------------------------------+ mysql> SELECT DECODE(UNHEX('A0E844F97A48FE6E8C1F'), 'Super Secret!'); +--------------------------------------------------------+ | DECODE(UNHEX('A0E844F97A48FE6E8C1F'), 'Super Secret!') | +--------------------------------------------------------+ | MyPassword | +--------------------------------------------------------+ mysql> show warnings\G

*************************** 1. row ***************************

Level: Warning

Code: 1287

Message: 'ENCODE' is deprecated and will be removed in a future release. Please use AES_ENCRYPT instead

1 row in set (0.01 sec)


ENCODE/DECODE Functions (cont.)• Functions are cryptographically weak. • For crypto to be upgraded, storage format would become incompatible.

• Overlaps with similar AES functionality already present.

47


SHOW PROFILES

48

mysql> set profiling=1; Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> show profiles; +----------+------------+-------------------------------------------------------+ | Query_ID | Duration | Query | +----------+------------+-------------------------------------------------------+ | 1 | 0.00039200 | SELECT * FROM table_a UNION ALL SELECT * FROM table_b | +----------+------------+-------------------------------------------------------+ 2 rows in set, 1 warning (0.01 sec)

Including Information_schema.profiling table.


SHOW PROFILES (cont.)

49

mysql> show profile for query 1; +----------------------+----------+ | Status | Duration | +----------------------+----------+ | starting | 0.000059 | | checking permissions | 0.000005 | | checking permissions | 0.000005 | | Opening tables | 0.000091 | | System lock | 0.000011 | | optimizing | 0.000004 | | statistics | 0.000012 | | preparing | 0.000009 | | optimizing | 0.000002 | | statistics | 0.000005 | | preparing | 0.000004 | | executing | 0.000003 | | Sending data | 0.000053 | | executing | 0.000002 | ..

..

| Sending data | 0.000029 | | optimizing | 0.000006 | | statistics | 0.000006 | | preparing | 0.000004 | | executing | 0.000002 | | Sending data | 0.000019 | | removing tmp table | 0.000007 | | Sending data | 0.000002 | | query end | 0.000007 | | closing tables | 0.000010 | | freeing items | 0.000019 | | cleaning up | 0.000016 | +----------------------+----------+ 26 rows in set, 1 warning (0.00 sec)


SHOW PROFILES (cont.)• Overlaps with Performance Schema interface. • Performance Schema has more flexible configuration • Instrumentation level can be increased/lowered to reduce overhead.

• Performance Schema has more data available.

50


Existing syntax works, server may respond differently.

Change of Behaviour

51


STRICT SQL_MODE• Some other SQL Modes have been merged into the definition of STRICT_TRANS_TABLES and STRICT_ALL_TABLES: • These options are ERROR_FOR_DIVISION_BY_ZERO, NO_ZERO_DATE and NO_ZERO_IN_DATE.

• The feature has also been reimplemented internally. Previous behaviour was not always consistent with all syntax.

52


SQL Mode ONLY_FULL_GROUP_BY• Has been reimplemented. • Now more standards compliant, and allows “functional dependencies”

53


IGNORE• Reimplemented, motivated by code cleanup. • Similar to strict:

• Some behaviour changed to be more consistent between statements.

54


DELAYED• Usage was deprecated in 5.6. • Syntax has been kept but functionality has now been removed. i.e. identical behaviour: • INSERT DELAYED INTO my_table (..) VALUES (..); • INSERT INTO my_table (..) VALUES (..);

55


DELAYED (cont.)• How do you handle delayed statements that violate UNIQUE constraints?

• Offers little measurable benefit to a row-‐level locking store with MVCC. • Act of delaying may actually incur more overhead.

56


EXPLAIN• Previously could have optional modifiers:

• EXPLAIN EXTENDED • EXPLAIN PARTITIONS

• These modifiers are now always enabled. • Motivated by simplification and code-‐cleanup (meta data only adds value).

57


EXPLAIN (cont.)

58

mysql> EXPLAIN SELECT * FROM table_a\G********************** 1. row ********************** id: 1 select_type: PRIMARY table: table_a partitions: NULL type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 3 filtered: 100.00 Extra: NULL2 rows in set, 1 warning (0.00 sec)


Functionality that will no longer be supported

Removed

59


SHOW ENGINE INNODB MUTEX (cont.)• Overlaps with PS. • Can not be disabled. • Code refactored so that InnoDB can use different locks internally.

61


InnoDB Monitor Tables• Existed in a time when information_schema did not. • Usage was:

62

CREATE TABLE innodb_monitor (a INT) ENGINE=INNODB;

/* view to the error log */

DROP TABLE innodb_monitor;


InnoDB Monitor Tables (cont.)• To be removed:

• innodb_tablespace_monitor • innodb_table_monitor • innodb_mem_validate

• To be replaced by config setting: • innodb_monitor • innodb_lock_monitor

63


ALTER IGNORE TABLE

64

mysql> CREATE TABLE a (id INT NOT NULL);Query OK, 0 rows affected (0.02 sec)

mysql> INSERT INTO a VALUES (2), (2), (3);Query OK, 3 rows affected (0.01 sec)Records: 3 Duplicates: 0 Warnings: 0

mysql> ALTER TABLE a ADD PRIMARY KEY (id);ERROR 1062 (23000): Duplicate entry '2' for key 'PRIMARY'


ALTER IGNORE TABLE (cont.)

65

mysql> ALTER IGNORE TABLE a ADD PRIMARY KEY (id);Query OK, 3 rows affected (0.01 sec)Records: 3 Duplicates: 1 Warnings: 0

mysql> SELECT * FROM a;+----+| id |+----+| 2 || 3 |+----+2 rows in set (0.00 sec)


ALTER IGNORE TABLE (cont.)• Will silently drop rows for you! • Semantically may even drop different rows on slaves. • We would prefer users to build a query to drop the rows themselves and not offer this functionality:

66

mysql> SELECT * FROM a GROUP BY id HAVING count(*) > 1;+----+| id |+----+| 2 |+----+1 row in set (0.01 sec)


Removal of CLI Programs• msql2mysql [Shell Script]

• mysqlaccess [Perl program]

• mysqlbug [Shell script]

• mysql_zap [Shell script]

• mysql_waitpid [C program]

• mysql_convert_table_format [Perl Program]

• mysql_fix_extensions [Perl Program]

• mysql_find_rows [Perl Program]

• mysql_setpermission [Perl Program]

• mysqlhotcopy [Perl Program]

67


Removal Example: mysqlhotcopy• Only works for MyISAM and ARCHIVE tables. • Is not a hot copy. • Unsafe with backing up views. • Removed to prevent users doing dangerous things.

68


Removal Example: msql2mysql• Allows you to import tables from the mysql database. • Useful 20 years ago. • No longer required.

69


YEAR(2)• Ambiguous!

70

mysql> CREATE TABLE a (a year(2));mysql> INSERT INTO a VALUES (20), (90);

mysql> SELECT * FROM a;+------+| a |+------+| 2020 || 1990 |+------+2 rows in set (0.00 sec)


Removal of Old Password Support• MySQL password hash prior to 4.1 uses weak crypto.

• 16-‐byte hash • New format has been available for 10 years (2004). • Will require client libraries to be upgraded as well.

• PHP’s mysql_* extension

71


How to prepare?• Try MySQL 5.7 defaults out now:

• https://github.com/morgo/mysql-‐compatibility-‐config

72

https://github.com/morgo/mysql-compatibility-config


We want your feedback!• We blog about our change proposals • Feedback has been great:

• SET GLOBAL sql_log_bin -‐ 32 Comments

• System tables in InnoDB -‐ 15 comments

• Changes to user management -‐ 10 comments

• Deprecate the old password format -‐ 6 comments

• Still opportunities for more feedback • Subscribe your RSS reader to:http://planet.mysql.com/

73

http://planet.mysql.com/

upcoming changes in mysql 5.7

Software

optimizer innodb improvements

locking code redo log

ssl support

c program

c stl refactoring

fromdmr0todmr5 overhead

transactions temporary

index lock contention