mysql 5.7 + java

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

What’s New in MySQL 5.7 + Even Better MySQL & Java Applications

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.


Safe Harbor Statement The following is intended to outline our general product direcOon. It is intended for informaOon purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcOonality, and should not be relied upon in making purchasing decisions. The development, release, and Oming of any features or funcOonality described for Oracle’s products remains at the sole discreOon of Oracle.

2


MySQL Connector/J •  Supports Java 5/6/7/8 •  Supports MySQL 5.0/5.1/5.5/5.6/5.7

•  Stable & mature 5.1 branch – Maintenance updates released approximately quarterly

•  Supports MySQL Fabric

•  Supports high-availability configurations –  Load-balancing, failover, (multi-)master/slave replication –  JMX-administration

•  Extensible

•  Much more!


<Insert Picture Here>

Program • Newest Connection Properties • Security with SSL and Pluggable Auth • Understanding Prepared Statements • Performance Considerations



New Connection Properties


SOCKS Proxy Support • Per-connection support of SOCKS proxies

– Fine-grained configuration – Previously only via JVM-wide system properties

• New in Connector/J 5.1.34 • socksProxyHost• socksProxyPort


Configure Server-side Read-only • MySQL 5.6 and InnoDB support

– SET SESSION TRANSACTION READ ONLY• Enabled by calling Connection.setReadOnly()• Queried when calling Connection.isReadOnly()• Generally very efficient when useLocalSessionState=true• As of C/J 5.1.35 can now be avoided altogether with readOnlyPropagatesToServer=false

•  Important for connection pool users – Pools like to reset to a consistent state when connections are released


SSL Cipher Suites • New in Connector/J 5.1.35 • Allows fine-grained choice of SSL ciphers • Required for MySQL 5.7 and Java 7

– Details to be discussed later


Avoid ODKU Checks • ODKU = ON DUPLICATE KEY UPDATE • Check required to properly support generated keys • C/J 5.1.32 allows bypassing this check by setting dontCheckOnDuplicateKeyUpdateInSQL=true

• Code contribution from Andrej Golovnin – Thanks Adrej!


Connection Properties • Connection Properties list in MySQL documentation • Many configurable options

– Some very specialized

• Get familiar with them so you know what to use when it’s appropriate for your application

• Don’t depend on forum / StackOverflow posts without understand the implications



Security with SSL and Pluggable Authentication


SSL in MySQL •  MySQL Community built with yaSSL •  MySQL Enterprise built with OpenSSL

•  MySQL 5.7 includes SSL improvements –  Automatic SSL configuration with OpenSSL

•  Previous versions require manual configuration –  Increased requirements for Diffie-Hellman key exchange

•  Key size minimum increased from 512 to 2048 – Command line client requires SSL when --ssl is given – New mysql_ssl_rsa_setup utility

•  Requires OpenSSL to be installed

•  TLS 1.0


SSL Access Control •  Any user can connect with SSL •  MySQL 5.7 moves REQUIRE SSL et al to CREATE USER and ALTER USER statements

–  Previously included with GRANT•  Users created with REQUIRE SSL will be denied access when connecting without SSL

•  Additional constraints available –  REQUIRE X509–  AND ISSUER ‘issuer’–  AND SUBJECT ‘subject’–  AND CIPHER ‘cipher’


SSL in Connector/J • Encrypted communications • Establish identity of server • Allow server to establish identity of client • Required for regulatory and corporate policy compliance


SSL Options in Connector/J •  useSSL•  requireSSL•  verifyServerCertificate•  clientCertificateKeyStoreUrl•  clientCertificateKeyStoreType•  clientCertificateKeyStorePassword•  trustCertificateKeyStoreUrl•  trustCertificateKeyStoreType•  trustCertificateKeyStorePassword•  enabledSSLCipherSuites


Keys Used in SSL • Server public key in truststore

– Can be specified as connection properties or Java system properties javax.net.ssl.trustStore and javax.net.ssl.trustStorePassword

– Used to verify the server identity – Not needed if verifyServerCertificate=false

• Client keypair in keystore – Can be specified as connection properties or Java system properties javax.net.ssl.keyStore and javax.net.ssl.keyStorePassword

– Used to prove client identity to the server – Client identity is not verified by default, use REQUIRE X509


Identities and Verification • Server identity

– Allows a client to verify that it is talking to a trusted server – Prevents man-in-the-middle attacks when establishing a connection

• Client identity – Allows the server to verify the identity of the client – Prevents unauthorized clients from connecting even if the correct credentials

are provided


Making SSL Connections • useSSL & requireSSL

– useSSL enables SSL connections – requireSSL causes connections to abort if SSL is not supported

•  Importing keys with Java keytool – Check manual for tutorial:

•  Connector/J Reference •  “Connecting Securely Using SSL”

– keytool -import -alias mysqlServerCACert -file cacert.pem -keystore keystore


Java 7 & MySQL 5.7 Community • MySQL 5.7 community uses Diffie-Hellman key exchange by default •  Java 7 doesn’t support MySQL 5.7’s required 2048 bit key size for

DH key exchange • Exception will be thrown giving instructions • Force RSA key exchange with:

– enabledSSLCipherSuites= TLS_RSA_WITH_AES_128_CBC_SHA


Pluggable Authentication • New feature in MySQL 5.5 • Supports traditional MySQL authentication and improved SHA-256

password hashing • Allows additional methods to be added with server plugins • Fully supported including extensible interfaces in Connector/J


SHA-256 Authentication

•  Plugin bundled with MySQL server •  Can be enabled by default in the server with

–  default_authentication_plugin=sha256•  Can be enabled on with CREATE USER (MySQL 5.7 syntax)

–  IDENTIFIED WITH sha256_password BY ‘…’•  Requires SSL connection or RSA encryption

– RSA encryption only available with OpenSSL built servers – RSA encryption supports pre-shared key via serverRSAPublicKeyFile property – RSA encryption can obtain key from server via allowPublicKeyRetrieval=true


Security Summary • Be diligent about auditing your configuration • Require SSL if appropriate • Use newer authentication methods where appropriate



Performance Considerations


Understanding Performance • Several aspects of performance

– Time (CPU) – Time (network latency) – Space (memory) – Space (network bandwidth)

• Know your performance characteristics first – Profiling

•  Impossible to optimize what you don’t understand – Measure first to obtain baseline


Latency • Round trips to the server are relatively expensive • The further the server is, the more round trips will cost

– Local vs geo-distribution

• E.g. server prepared statements require three round trips to execute one statement – Only one for a direct or client prepared statement – Adds up quickly


Measure & Reduce Latency • General query log

– Shows every command sent to the server

• Benchmarking – Tools like OpenJDK JMH

• Re-use PreparedStatement instances where possible – Can be cached on connection and in connection pool

• Connection pool validation & pings – Read-only status

• Use connection pools – Connection initialization is not free


Bandwidth Considerations • Use fine-grained queries and avoid fetching unnecessary fields and

rows • Run tests with useUsageAdvisor=true to warn for inefficiencies

– Reports on unused columns – Reports on result sets which were closed without reading all rows

• Cache server configuration by setting cacheServerConfiguration=true– Avoids additional querying during connection initialization


Buffered vs Streaming Results •  Buffered results read entire result set into memory

–  Ala mysql_store_result()–  Faster local access –  Additional memory required –  Best for OLTP applications –  Fully scrollable cursors

•  Streaming results read individual rows as used –  Ala mysql_use_result()–  Access may be slowed while waiting for network reads – Reduced memory requirements –  Best for very large results –  Forward only scrollability


Enable Streaming Results • com.mysql.jdbc.Statement

– Vender extension interface

• Cast statement instances • Call enableStreamingResults()• Call disableStreamingResults()• Optionally set clobberStreamingResults=true

– Result sets automatically closed when new statements are executed on the same connections


Low-level C/J Performance Options • useDirectRowUnpack

– Enabled by default – Avoids reading entire row before splitting it into per-field byte buffers – Doesn’t work with very large result rows (multi-packets are 16MB+)

• largeRowSizeThreshold– Enabled for larger rows (default = 2k+) – Reads entire row into byte buffer – Avoids splitting row into per-field buffers


Performance Recommendations • Be methodical • Test different approaches • Use intuition • Consider performance while developing code


Wrap-up • Q&A • MySQL Forums :: Connector/JDBC and Java

– http://forums.mysql.com/list.php?39

• MySQL Connector/J Developer Guide – http://dev.mysql.com/doc/connector-j/en/

• Todd Farmer Blog – http://mysqlblog.fivefarmers.com/

• Connectors Team Blog – http://insidemysql.com/category/mysql-development/connectors/


0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

8 16 32 64 128 256 512 1,024

Que

ries p

er Secon

d

Connec9ons

MySQL 5.7: Sysbench Read Only (Point Select)

MySQL 5.7

MySQL 5.6

MySQL 5.5

MySQL 5.7: Sysbench: Read Only

Intel(R) Xeon(R) CPU E7-‐4860 x86_64 4 sockets x 10 cores-‐HT (80 CPU threads) 2.3 GHz, 512 GB RAM Oracle Linux 6.5

2x Faster than MySQL 5.6 3x Faster than MySQL 5.5

645,000 QPS


MySQL 5.7: InnoDB, NoSQL With Memcached 6x Faster than MySQL 5.6 Thank you, Facebook

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

8 16 32 64 128 256 512 1,024

Que

ries p

er Secon

d

Connec9ons

MySQL 5.7 vs 5.6 -‐ InnoDB & Memcached

MySQL 5.7

MySQL 5.6

1 Million QPS



MySQL 5.7: ConnecOons per Second 1.7x Faster than MySQL 5.6 2.5x Faster than MySQL 5.5

67,000 Connec9ons/Sec

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

MySQL 5.5 MySQL 5.6 MySQL 5.7

Conn

ec9o

ns/Secon

d

Connec9ons Per Second

MySQL 5.5

MySQL 5.6

MySQL 5.7



MySQL 5.7: Improvements in latest DMR 6 •  InnoDB for befer transacOonal throughput, availability, IO • Replica9on for befer scalability and availability • MySQL Fabric for high availability and sharding • Performance Schema for new and improved performance metrics • Op9mizer for befer EXPLAINing, parsing, query performance • GIS with naOve InnoDB spaOal indexes and Boost.Geometry integraOon •  Security with easier, safer instance iniOalizaOon, setup and management

Available Now! dev.mysql.com/downloads/mysql/

36


MySQL 5.7 Parser & OpOmizer Refactoring

OpOmizer

Logical transformaOons

Cost-‐based opOmizer: Join order and access methods

Plan refinement

Query execuOon plan

Query execuOon

Parser

Resolver: SemanOc check,name resoluOon

SQL DML query

Query result

Storage Engine InnoDB MyISAM

Improves readability, maintainability and stability

– Cleanly separate the parsing, opOmizing, and execuOon stages

– Allows for easier feature addiOons, with lessened risk


MySQL 5.7: Query Rewrite Plugin • New pre and post parse query rewrite APIs

– Users can write their own plug-‐ins • Provides a post-‐parse query plugin

– Rewrite problemaOc queries without the need to make applicaOon changes – Add hints – Modify join order – Many more …

•  Improve problemaOc queries from ORMs, third party apps, etc •  ~Zero performance overhead for queries not to be rewrifen


MySQL 5.7: OpOmizer -‐ Cost Info in JSON EXPLAIN •  Expanded JSON EXPLAIN

– Now includes all available cost info – Used for Visual Explain In MySQL Workbench

39

{ "query_block": { "select_id": 1, "cost_info": { "query_cost": "200.40" }, "table": { "table_name": "nicer_but_slower_film_list", "access_type": "ALL", "rows_examined_per_scan": 992, "rows_produced_per_join": 992, "filtered": 100, "cost_info": { "read_cost": "2.00", "eval_cost": "198.40", "prefix_cost": "200.40", "data_read_per_join": "852K" }, "used_columns": [ "FID", "title", "description", "category", "price", "length", "rating", "actors" ], ...


Memory Instrumenta9on •  Aggregates staOsOcs by

–  Type of memory used (caches, internal buffers, …)

–  Thread/account/user/host indirectly performing the memory operaOon

•  Afributes include – Memory used (bytes) –  OperaOon counts –  High/Low Water Marks

Statement Instrumenta9on •  Stored Procedures •  Stored FuncOons •  Prepared Statements

•  TransacOons

Addi9onal Informa9on •  ReplicaOon slave status •  MDL lock instrumentaOon

•  User variables per thread •  Server stage tracking •  Track long running SQL •  Improved configuraOon

•  All while reducing total footprint and overhead

MySQL 5.7: Performance Schema


MySQL 5.7: SYS Schema Helper objects for DBAs, Developers and Opera9ons staff • Helps simplify DBA / Ops tasks -‐ Monitor server health, user, host staOsOcs -‐ Spot, diagnose, and tune performance issues

•  Easy to understand views with insights into -‐ IO hot spots, Locking, Costly SQL statements -‐ Schema, table and index staOsOcs

•  SYS is similar to -‐ Oracle V$ catalog views -‐ Microsov SQL DMVs (Dynamic Mgmnt Views)


MySQL 5.7: InnoDB Improvements •  Scalability! RO, RW • Parallel “dirty page” flushing

– Improved adapOve flushing algorithm and mechanisms

• AutomaOc truncaOon of UNDO logs – Stored in separate tablespace

• Online, resize buffer pool •  Fast Create Index, Bulk load •  Improved Temporary Table performance • Compression, ParOOoning, Tablespaces, GIS

42


MySQL 5.7: InnoDB -‐ Online • Resize the InnoDB Buffer Pool online

– Allows DBAs to tune the buffer size without any downOme – Adapt in real-‐Ome to changes in database usage paferns

• Online ALTER TABLE – Enlarge VARCHAR, Rename Index

• Dynamic configuraOon – As a design principle for new features & sezngs – But also make exisOng variables dynamically sefable

43


MySQL 5.7: InnoDB -‐ Bulk Load for Create Index • Much faster INDEX creaOon and bulk loads •  Sorted index builds, done from the bofom-‐up

– Improves speed by increasing locality and decreasing node splizng

• Pages are compressed only when full • New innodb_fill_factor opOon controls free space lev in each page • Performance results show

– 2-‐3x performance improvement for ADD/CREATE INDEX operaOons – 2-‐5% improvement for standard INSERT operaOons

44


MySQL 5.7: InnoDB Temporary Tables • New separate tablespace for temporary tables

– Improved CREATE/DROP performance – DDL changes are transient, which eliminates some disk IO

• OpOmize DML operaOons – No REDO logging, no change buffering, less locking

• New intrinsic temporary tables – Specialized temporary tables with tailored ACID/MVCC semanOcs – Light weight and ultra-‐fast, great for intermediate query execuOon operaOons

•  InnoDB as default storage engine for disk based temp tables – OpOmizer switched from MyISAM to InnoDB (faster) for internal temp tables

45


MySQL 5.7: InnoDB •  InnoDB support for MySQL Group ReplicaOon

– High priority transacOons •  InnoDB NaOve ParOOoning

– Will allow us to eliminate previous limitaOons on parOOoned tables – Removes memory usage problems when many parOOons are used

•  Support for 32K and 64K pages • General TABLESPACE support

– Store mulOple tables in user defined shared tablespaces


EAF: InnoDB Compression Thank you, SanDisk Fusion-‐io

•  Transparent Page Level Compression – Happens transparently in background threads – Managed enOrely within the IO layer – Uses sparse file and "hole punching" support in OS kernels and File Systems

• Reduces IO – Improves MySQL performance – Improves storage efficiency – Reduces write cycles, thus increasing SSD lifespan

• Applies to all InnoDB data, including the system tablespace and UNDO logs

47

labs.mysql.com


MySQL 5.7: Syslog Support for Linux/Unix pla~orms

• NaOve support for syslog •  Simple opOon to (re)direct log output to naOve syslog facility •  Start-‐up server configuraOon opOon • Dynamically in the running server

–  System variable log_syslog (ON/OFF, defaults to OFF).

Thank you, Simon Mudd at booking.com


MySQL 5.7: Security -‐ EncrypOon, Passwords, InstallaOon • AES 256 EncrypOon

– Default in MySQL 5.7

• Password rotaOon policies – Can be set globally, and at the user level

• Deployment: enable secure unafended install by default – Random password set on install – Remove anonymous accounts – Deployment without test account, schema, demo files

•  Easier instance iniOalizaOon and setup: mysqld -‐-‐iniOalize

49


MySQL 5.7: Security -‐ SSL •  Enabled by default • Auto-‐detecOon of exisOng keys and certs • Auto generaOon of keys and certs when needed • New helper uOlity: mysql_ssl_rsa_setup

50


MySQL 5.7: Locking

MulOple User Level Locks per ConnecOon • User-‐level locks can be used to organize mutual exclusion

–  When accessing some resource –  When table or row-‐level locks are not appropriate

• Request mulOple locks by issuing a series of GET_LOCK statements • Replaces custom user-‐level lock implementaOon

– With one based on the MDL lock manager – Deadlocks between different connecOons acquiring user-‐level locks, metadata locks, and those waiOng for table flushes are properly detected and reported as errors.

Thank you, Konstan9n Osipov!


MySQL 5.7: Server-‐side Statement Timeouts Thank you Davi Arnaut!

•  Server side statement Omeouts – Global for server, per session, or for individual SELECT statements

•  Expanded to Windows and Solaris, restricted by removing USER opOon

SELECT MAX_STATEMENT_TIME = 109 * FROM my_table;

52


MySQL HA & Scaling SoluOons MySQL

Replica9on MySQL Fabric

Oracle VM Template

Oracle Clusterware

Solaris Cluster

Windows Cluster DRBD MySQL

Cluster

App Auto-‐Failover ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Data Layer Auto-‐Failover ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Zero Data Loss MySQL 5.7 MySQL 5.7 ✔ ✔ ✔ ✔ ✔ ✔

Pla~orm Support All All Linux Linux Solaris Windows Linux All

Clustering Mode Master + Slaves

Master + Slaves

Ac9ve/Passive

Ac9ve/Passive

Ac9ve/Passive

Ac9ve/Passive

Ac9ve/Passive

Mul9-‐Master

Failover Time N/A Secs Secs + Secs + Secs + Secs + Secs + < 1 Sec

Scale-‐out Reads ✔ ✖ ✖ ✖ ✖ ✖ ✔

Cross-‐shard operaOons N/A ✖ N/A N/A N/A N/A N/A ✔

Transparent rouOng ✖ For HA ✔ ✔ ✔ ✔ ✔ ✔

Shared Nothing ✔ ✔ ✖ ✖ ✖ ✖ ✔ ✔

Storage Engine InnoDB+ InnoDB+ InnoDB+ InnoDB+ InnoDB+ InnoDB+ InnoDB+ NDB

Single Vendor Support ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✔


MySQL 5.7: ReplicaOon Improvements •  On-‐line, phased deployment of GTIDs •  Use GTIDs without binary logging on slave •  Improved Master throughput

–  Befer synchronizaOon between replicaOon-‐user sessions; Ack receiver thread, group commit tuning…

•  Lossless ReplicaOon through enhanced Semi-‐sync

•  Configurable number of Semi-‐sync slave Acks

54


• MulO-‐Source ReplicaOon – Consolidate updates from mulOple Masters into one Slave •  Consolidated view of all shards • More flexible topologies •  Centralized point for backups

– CompaOble with Semi-‐Sync ReplicaOon & enhanced MTS

• Performance Schema tables for monitoring slave

• Online OperaOons: Dynamic ReplicaOon Filters, switch master

MySQL 5.7: ReplicaOon Improvements

Binlog

Master 1

Binlog

Master 2

…

…

Binlog

Master N

IO 1

Relay 1

Coordinator

W1 W2 … WX

IO 2

Relay 2

Coordinator

W1 W2 … WX

…

…

Coordinator

W1 W2 … WX

IO N

Relay N

Coordinator

W1 W2 … WX

Slave


• MulO-‐Threaded Slaves: – Performance -‐ Slave applies transacOons in parallel even within same database; group commit tuning

– OpOon to preserve Commit order – AutomaOc slave transacOon retries

• Up to 5X performance vs single threaded slave – No changes to the applicaOon

• GTIDs & Crash-‐Safe Slave enabled

MySQL 5.7: ReplicaOon Improvements

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

0 4 16 25 50 100 200 Worker Threads

Slave Transac9ons per Second

Baseline

50 clients

100 clients

150 clients

200 clients

•  Sysbench OLTP test –  10M rows –  SSD / 48 core HT / 512 GB RAM


•  Shared-‐nothing virtually synchronous database system

•  MulO-‐master update anywhere –  Conflict detecOon and resoluOon (transacOon rollback) –  OpOmisOc State Machine ReplicaOon

•  AutomaOc group membership management and failure detecOon –  No need for server fail-‐over –  ElasOc scale out/in –  No single point of failure –  AutomaOc reconfiguraOon

•  Well integrated –  InnoDB –  GTID-‐based replicaOon –  PERFORMANCE_SCHEMA

EAF: Group ReplicaOon

ApplicaOon

MySQL Masters ReplicaOon Plugin

API

MySQL Server

Group Comms (Corosync)

labs.mysql.com


EAF: HTTP Plugin for MySQL •  Server Plugin adds HTTP(S) endpoints to MySQL • Results are serialized to JSON format encoded as UTF8 • Provides 3 choices of User Endpoint Types

– SQL – CRUD -‐ Key-‐Value – JSON -‐ Document

labs.mysql.com


MySQL Fabric 1.5

•  High Availability –  Server monitoring with auto-‐promoOon and transparent applicaOon failover

•  OpOonally scale-‐out through sharding –  ApplicaOon provides shard key –  Range or Hash –  Tools for resharding –  Global updates & tables

•  Fabric-‐aware connectors rather than proxy: Python, Java, PHP, .NET, C (labs) –  Lower latency, bofleneck-‐free

•  Server provisioning using OpenStack etc.

High Availability + Sharding-‐Based Scale-‐out

MySQL Fabric

Connector

ApplicaOon

Read-‐slaves

mappings

SQL

HA group

Read-‐slaves

HA group

Connector

ApplicaOon

GA


MySQL Workbench 6.2 •  Fabric

– Add fabric node, browse, view, connect • Performance Dashboard

– Performance Schema Reports & Graphs

• Visual Explain • GIS Viewer • MigraOon

– New Microsov Access – Microsov SQL Server, Sybase, PostgreSQL

GA


MySQL on Windows •  MySQL Installer for Windows •  MySQL Workbench

•  MySQL MigraOon Wizard – Microsov SQL Server – Microsov Access

•  MySQL for Visual Studio

•  MySQL for Excel

•  MySQL NoOfier

•  MySQL Connector/.Net

•  MySQL Connector/ODBC


•  Simple and convenient way to install & update MySQL products

•  Supports the following distribuOons –  Oracle, Red Hat, CentOS –  Fedora –  Ubuntu, Debian –  SUSE

•  Coming Soon –  Preconfigured Containers –  Improved support for popular DevOps deployment tools

•  The latest releases of – MySQL Database – MySQL Workbench – MySQL Connector/ODBC – MySQL Connector/Python – MySQL Connector/NET – MySQL UOliOes

MySQL Community Repositories: Yum, APT, NuGET


MySQL Source Code Available on GitHub •  Git for MySQL Engineering

–  Fast, flexible and great for a distributed team –  Great tooling –  Large and vibrant community

•  GitHub for MySQL Community –  Easy and fast code availability to the community and to downstream projects

hfps://github.com/mysql


Even Better MySQL & Java Applications

mysql 5.7 + java

Technology

mysql mysql community

mysql connectorj

openssl mysql

mysql documentation

mysql fabric

new connection properties

yassl mysql enterprise

ssl cipher suites new