mariadb 10 tutorial - 13.11.11 - percona live london

Ivan Zoratti

MariaDB 10 TutorialNew features, engines and tools

V1311.01Tuesday, 26 November 13

Who is Ivan

?Tuesday, 26 November 13

SkySQL•Leading provider of open source databases, services and solutions

•Home for the founders and the original developers of the core of MySQL

•The creators of MariaDB, the drop-‐in, innovative replacement of MySQL

Tuesday, 26 November 13

Agenda

•What’s new in MariaDB 10

•MariaDB 10 Optimizer

•MariaDB 10 Administration

•The CONNECT storage engine

•The Spider storage engine

•The Cassandra storage engine

•MariaDB replication

•All the rest!

4

NOTE: We refer to

MariaDB 10.0.5


What’s New in MariaDB 10Tuesday, 26 November 13

First: What Is MariaDB 10?

•A fork of MySQL 5.5 with extra features, requested by the users

•It is backward compatible with MySQL 5.5 and MariaDB 5.5 for file formats, replication and configuration files

•It is application and data files compatible with MySQL 5.6

•Other Key points:•A project that enhances collaboration•Hundreds of thousands lines of code from community contributors!

•100% GPL•100% Free -‐ commercial add-‐ons and extensions are welcome, but they are clearly identified and separated

6Tuesday, 26 November 13

MariaDB Inheritance•MariaDB 5.1 (MySQL-‐5.1 base)•Table elimination•new storage engines•cleanup•better tests•pool of threads

•MariaDB 5.2 (MariaDB-‐5.1 base)•Virtual columns•extended user statistics•segmented MyISAM keycache

•MariaDB 5.3 (MariaDB-‐5.2 base)•Biggest changes to optimizer (faster subqueries, joins, etc.)

•Microsecond precision• faster HANDLER (HANDLER RED 50% faster w/530,000 qps)

•dynamic columns•Better replication (group commit, etc.)•HandlerSocket

•MariaDB 5.5 = MariaDB 5.3 + MySQL 5.5•Opensource, more efficient threadpool•Non-‐blocking client library•New LIMIT ROWS EXAMINED option•Extended keys for XtraDB/InnoDB•New SphinxSE•Dynamic replication settings•Lots of security fixes, new status variables, etc.


MySQL 5.6 Backported and Reimplemented Features

•InnoDB/XtraDB (5.6.5)•Optimised read only transaction, with support for TRANSACTION READ ONLY

•PERFORMANCE_SCHEMA (5.6.5)

•Online ALTER TABLE (in-‐progress)•Privileges on temporary tables•--plugin-load-add

•GET DIAGNOSTIC statement•Character set extensions

•Temporal literals•Filesort optimisation for ORDER BY...LIMIT (shows only few rows of a resultset)

•Reimplemented:•Errormessages(w/system error string) •CURRENT_TIMESTAMP as DEFAULT for DATETIME columns

•GlobalTransactionID•Parallel replication•EXISTS-‐TO-‐IN optimisation


MariaDB 10 only

•SHOW EXPLAIN for <thread_id>

•Faster ALTER TABLE with unique keys for Aria & MyISAM

•Per-‐thread memory usage•INFORMATION_SCHEMA.PROCESSLIST with MEMORY_USAGE & EXAMINED_ROWS now •SHOW STATUS with memory usage


Additions to MariaDB 10

•MariaDB Galera Cluster

•Connectors and Drivers•MariaDB C Client Library•MariaDB JDBC Driver

•MariaDB Enterprise•MariaDB Manager API

•MariaDB Audit Plugin


MariaDB 10 OptimizerTuesday, 26 November 13

PAGE

Features Comparison


PAGE

Features Comparison

14

h"ps://mariadb.com/kb/en/query-‐op7miza7ons/


https://mariadb.com/kb/en

https://mariadb.com/kb/en

PAGE

EXPLAIN and Query Plans

•-‐-‐log_slow_verbosity=query_plan•Query plan in the slow query log

•SHOW EXPLAIN FOR thread_id•Query plan for a running query

•EXPLAIN ANALYZE•Query plan and query execution

•EXPLAIN ... JSON•Extra information (WHERE and attached conditions) to the query plain

•EXPLAIN UPDATE/DELETE•Query plan for UPDATE and DELETE statements


PAGE

Engine-‐independent Statistics•Collected/used on SQL layer, not at storage engine layer

•No auto updates, only ANALYZE TABLE•100% accuracy

•More Statistics•Index statistics•Table statistics•Column statistics•MIN/MAX values•NULL/NON NULL values•Histograms

• use_stat_tables•never•complementary•preferably

•ANALYZE TABLE tbl PERSISTENT FOR COLUMNS (col1,col2,...) INDEXES (idx1,idx2,...);

• System tables•mysql.table_stat•mysql.index_stat•mysql.column_stat


PAGE

Index Condition Pushdown (ICP)•Without ICP:

•Read index•Read record•Check the WHERE condition

•With ICP:•Read index•Check the WHERE condition on index•Read record•Check the WHERE condition

• In the query plan• “Using index condition”

•Performance gain (example)•Cold Buffer: 1min vs. 5min (5x)•Hot Buffer: 0.07sec vs. 0.19 sec (2.7x)

17

5.5


PAGE

Index Merge/Union

•Used for OR in the WHERE clause•They cannot be resolved with a single index

•Can be turned off globally•optimizer_switch=‘index_merge=off’

18

5.5

SELECT * FROM ontime WHERE ( origin=‘LHR’ OR dest=‘LHR’ );


PAGE

Index Merge/Intersection

•Used for AND in the WHERE clause and no composite index

•Must be turned on•optimizer_switch=‘index_merge_sort_intersection=on’

19

5.5

EXPLAIN SELECT AVG(arrdelay) FROM ontime WHERE depdelay15=1 AND origin=‘LHR’;

+--+-----------+------+-----------+---------------+---------------+-------+----+-----|id|select_type|table |type |possible_keys |key |key_len|ref |rows+--+-----------+------+-----------+---------------+---------------+-------+----+-----| 1|SIMPLE |ontime|index_merge|Origin,DepDel15|Origin,DepDel15|3,5 |NULL|76952+--+-----------+------+-----------+---------------+---------------+-------+----+-----

+--------------------------------------------+|Extra |+--------------------------------------------+ |Using intersect(Origin,DepDel15);Using where|+--------------------------------------------+


PAGE

Index Merge/Sort Intersection

20

5.5

•Used for AND and IN in the WHERE clause

•Must be turned on•optimizer_switch=‘index_merge_sort_intersection=on’

SELECT AVG(arrdelay) FROM ontime WHERE depdel15=1 AND OriginState IN ( 'CA', 'GB' );


PAGE

SubqueryOptimization

21

• optimizer_switch=‘subquery_cache=on’

• Variables:subquery_cache_hitssubquery_cache_miss


PAGE

Subquery Optimization

•Cache ON/OFF: 1.01 sec / 1hr 31m 43.33 sec•Cache Hit/Miss: 149975 / 25

22

SELECT COUNT(*) FROM customer WHERE c_acctbal > 0.8 * ( SELECT MAX(c_acctbal) FROM customer C WHERE C.c_nationkey=customer.c_nationkey GROUP BY c_nationkey );

5.5


PAGE

Semi-‐join Materialization

23

SELECT * FROM Country WHERE Country.code IN ( SELECT City.Country FROM City WHERE City.Population > 7*1000*1000 ) AND Country.continent='Europe'

Materialization scanvs.

Materialization lookup

5.5


PAGE

Multi-‐range Read

•Faster disk access by sorting record read requests and then doing one ordered disk sweep.

•Particularly efficient for legacy spindles, but still reduces I/O for SSDs and flash storage

•optimizer_switch='mrr=on'


PAGE

Multi-‐range Read -‐ Range Access

25

EXPLAIN SELECT * FROM tbl WHERE tbl.key1 BETWEEN 1000 AND 2000;+----+-------------+-------+-------+---------------+------+---------+------+------| id | select_type | table | type | possible_keys | key | key_len | ref | rows+----+-------------+-------+-------+---------------+------+---------+------+------| 1 | SIMPLE | tbl | range | key1 | key1 | 5 | NULL | 960+----+-------------+-------+-------+---------------+------+---------+------+------+-------------------------------------------+| Extra |+-------------------------------------------+| Using index condition; Rowid-ordered scan |+-------------------------------------------+


PAGE

Multi-‐range Read -‐ Batched Key Access

•Similar to the sorting buffer in the range buffer, but applied to row IDs

•Same benefits as for Range access

26

EXPLAIN SELECT * FROM t1,t2 WHERE t2.key1=t1.col1;+----+-------------+-------+------+---------------+------+---------+--------------+------| id | select_type | table | type | possible_keys | key | key_len | ref | rows+----+-------------+-------+------+---------------+------+---------+--------------+------| 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 1000| 1 | SIMPLE | t2 | ref | key1 | key1 | 5 | test.t1.col1 | 1+----+-------------+-------+------+---------------+------+---------+--------------+------+--------------------------------------------------------+| Extra |+--------------------------------------------------------+| Using where || Using join buffer (flat, BKA join); Rowid-ordered scan |+--------------------------------------------------------+


MariaDB 10 AdministrationTuesday, 26 November 13

New and Improved Commands

•Progress reporting

•SHOW EXPLAIN

•SHOW PLUGIN SONAME

•SHUTDOWN

•Per-‐connection memory accounting

•Roles28


29

Progress reporting

•MariaDB 5.3 and later supports progress reporting for some long running commands.•INFORMATION_SCHEMA.PROCESSLIST has three new columns for progress reporting: STAGE, MAX_STAGE, and PROGRESS•There is a new column Progress in SHOW PROCESSLIST which shows the total progress (0-‐100 %)•The client receives progress messages which it can display to the user to indicate how long the command will take.

•Valid for:•ALTER TABLE, ADD/DROP INDEX, LOAD DATA INFILE•CHECK/REPAIR/ANALYZE/OPTIMIZE TABLE


https://mariadb.com/kb/en/show-processlist/

https://mariadb.com/kb/en/show-processlist/

30

SHOW EXPLAIN

•The SHOW EXPLAIN command allows to get an EXPLAIN (that is, a printout of a query plan) of a query running in a certain thread.•The syntax is: SHOW EXPLAIN FOR <thread_id>;


31

SHOW PLUGIN SONAME

•It displays information about compiled-‐in and all server plugins in the @@plugin_dir directory, including not installed ones.


32

SHUTDOWN

•SQL statement to shut the server down

•Same as mysqladmin shutdown

•Requires SHUTDOWN privilege


Per-‐connection memory usage

•Monitor memory usage per connection:


Roles

•Contribution from Vicentju Ciorbaru at Google Summer of Code

•SQL standard implementation of roles•CREATE ROLE, DROP ROLE•GRANT role TO user, GRANT role TO role•SET ROLE•DEFINER=role•CURRENT_ROLE•INFORMATION_SCHEMA tables


The CONNECT Storage EngineTuesday, 26 November 13

The CONNECT Storage Engine

•What is the CONNECT storage engine?•CONNECT is a storage engine that enables MariaDB to use external data as they were standard tables in the server•Data is not loaded into MariaDB

•History of the CONNECT storage engine•The engine has been mainly developed by Olivier Bertrand, an ex IBM database researcher, with the intent to have a more versatile way to access external data sources in various formats•The idea dates back in 2004 and Olivier has been in touch with MySQL and MariaDB since•Today CONNECT is a standard engine in MariaDB 10


CONNECT Engine Usage

•The CONNECT engine:•Integrates/access data directly in many non-‐MariaDB formats•Simplifies the ETL procedures in Business Intelligence and Business Analytics•Simplifies the export/import of data from/to MariaDB, to/from other data sources

•There are strong similarities with the CSV engine and some similarities with FederatedX and Merge•The CSV engine does not allow indexing and block reading•The FederateX engine imposes several restrictions and performance limitations

•FILE privilege is required


CONNECT Engine -‐ Advantages

•Multiple File Table (option multiple=[0|1|2]) processes sequentially files of the same type

•Indexing for most of the table types•Data compression•Block reading boosts read performance

•Condition push down•Valid for ODBC, MYSQL, TBL and WMI•set optimizer_switch='engine_condition_pushdown=on'

...or...--engine_condition_pushdown=on

•Even more possibilities with the OEM file type


CONNECT Table Types

39

Type Descrip*on

DOS

The table is contained in one or several files. The file format can be refined by some other ophons of the command or more oien using a specific type as many of those described below. Otherwise, it is a flat text file where columns are placed at a fixed offset within each record, the last column being of variable length.

FIX Text file arranged like DOS but with fixed length records.

BIN Binary file with numeric values in plalorm representahon, also with columns at fixed offset within records.

VEC Binary file organized in vectors, in which column values are grouped consecuhvely, either split in separate files or in a unique file.

DBF* File having the dBASE format.

CSV*"Comma Separated Values" file in which each variable length record contains column values separated by a specific character (defaulhng to the comma)

FMTFile in which each record contains the column values in a non-‐standard format (the same for each record) This format is specified in the column definihon.

INI File having the format of the inihalizahon or configurahon files used by many applicahons.

Type Descrip*on

XML File having the XML or HTML format

ODBC* Table extracted from an applicahon accessible via ODBC or unixODBC. For example from another DBMS or from an Excel spreadsheet.

MYSQL* Table accessed using the MySQL API like the FEDERATED engine.

PROXY* A table based on another table exishng on the current server.

TBL* Accessing a collechon of tables as one table (like the MERGE engine does for MyIsam tables)

XCOL* A table based on another table exishng on the current server with one of its column containing of comma separated values.

OCCURA table based on another table exishng on the current server, several columns of the object table containing values that can be grouped in only one column.

DIR Virtual table that returns a file list like the Unix ls or DOS dir command.

WMI*Windows Management Instrumentahon table displaying informahon coming from a WMI provider. This type enables to get in tabular format all sort of informahon about the machine hardware and operahng system (Windows only).

MAC Virtual table returning informahon about the machine and network cards (Windows only).

OEM Table of any other formats not directly handled by CONNECT but whose access is implemented by an external plugin module (DLL or Shared Library).

(*) Auto discovery of the table structure


CONNECT Table Options

40

Type Descrip*on

TABLE_TYPEThe type of the external table: DOS, FIX, BIN, CSV, FMT, XML, INI, DBF, VEC, ODBC, MYSQL, TBL, DIR, WMI, MAC and EOM. Defaults to DOS.

FILE_NAME The file (path) name for all table types based on files. Can be absolute or relahve to the current data directory.

XFILE_NAME The file (path) base name for a table index files. Can be absolute or relahve to the data directory. Defaults to the file name.

TABNAME The target table or node for ODBC, MYSQL, XML or catalog tables.

TABLE_LIST The comma separated list of a TBL table sub-‐tables.

DBNAME The target database for ODBC, MYSQL or catalog tables.

DATA_CHARSET The character set used in the external file or data source.

SEP_CHAR Specifies the field separator character of a CSV tables.

QCHAR Specifies the character used for quohng some fields of a CSV table or the idenhfiers of an ODBC tables.

MODULE The (path) name of the DLL or shared lib implemenhng the access of a non-‐standard (OEM) table type.

SUBTYPE The subtype of an OEM table type.

CATFUNC The catalog funchon used by a catalog table.

Type Descrip*on

OPTION_LIST Used to specify all other ophons not yet directly defined.

MAPPED Specifies whether “file mapping” is used to handle the table file.

HUGE To specify that a table file can be larger than 2GB.

COMPRESS True if the data file is compressed. Defaults to NO.

SPLIT True for a VEC table when each columns are in separate files.

READONLY True if the data file must not be modified or erased.

SEPINDEX When true, indexes are saved in separate files.

LRECL The file record size (oien calculated by default).

BLOCK_SIZE The number of rows each block of FIX, BIN, DBF OR VEC tables contains. For an ODBC table this is the RowSet size ophon.

MULTIPLE Used to specify mulhple file tables.

HEADER Applies to CSV, VEC and HTML files. Its meaning depends on the table type.

QUOTED The level of quohng used in CSV table files.

ENDING End of line length. Default to 1 for Unix/Linux and 2 for Windows.


CONNECT Engine -‐ Features•Table “auto-‐creation” when the the file does not exist or it is not specified

•Large tables support (>2GB)•Available for FIX, BIN and VEC•Use ‘option_list’=‘huge=1’

•Compression -‐ gzlib format•Available for DOS, FIX, BIN, CSV and FMT

•ODBC Format•Access to ODBC and UnixODBC data sources

•WHERE conditions are push to the ODBC source

•Multiple tables/files from data sources can be consolidated into a single table

•MYSQL Format -‐ to access local or remote MySQL tables•Allows to define a subset of the source columns and type conversion

•Condition LIMIT push down


CONNECT Engine -‐ Features•TBL -‐ Table List Table•Collection of tables seen as one•No limitation on the storage engine: tables can be from different storage engines (including CONNECT)

•The tables may have different column structure

•VEC -‐ Column Store•Data is stored in binary files as vectors•I/O optimization, as CONNECT reads only columns that are requested by the query

•The “split format” allows to split columns in multiple files

•XML -‐ for XML and HTML data•HTML attributes can be defined in the option_list


CONNECT Engine -‐ Hands On•Install the plugin

•Note: The library may not be in the rpm on RH/CentOS, you can find it in the standard tarball

•Create a HTML table and export data

43

INSTALL PLUGIN CONNECT SONAME 'ha_connect';

CREATE TABLE employees engine=connect table_type=XMLfile_name='/var/lib/mysql_connect/employees.html' header=yes option_list='name=TABLE,coltype=HTML,attribute=border=1;cellpadding=5,headattr=bgcolor=yellow'SELECT emp_no, birth_date, first_name, last_name, hire_date FROM employees.employees;Query OK, 300024 rows affected (20.99 sec)Records: 300024 Duplicates: 0 Warnings: 0


CONNECT Engine -‐ Hands On

44

MariaDB [connect_examples]> select * from employees limit 5;+--------+------------+------------+-----------+------------+| emp_no | birth_date | first_name | last_name | hire_date |+--------+------------+------------+-----------+------------+| 10001 | 1953-09-02 | Georgi | Facello | 1986-06-26 || 10002 | 1964-06-02 | Bezalel | Simmel | 1985-11-21 || 10003 | 1959-12-03 | Parto | Bamford | 1986-08-28 || 10004 | 1954-05-01 | Chirstian | Koblick | 1986-12-01 || 10005 | 1955-01-21 | Kyoichi | Maliniak | 1989-09-12 |+--------+------------+------------+-----------+------------+5 rows in set (44.32 sec)


45

MariaDB [connect_examples]> select * from employees limit 5;+--------+------------+------------+-----------+------------+| emp_no | birth_date | first_name | last_name | hire_date |+--------+------------+------------+-----------+------------+| 10001 | 1953-09-02 | Georgi | Facello | 1986-06-26 || 10002 | 1964-06-02 | Bezalel | Simmel | 1985-11-21 || 10003 | 1959-12-03 | Parto | Bamford | 1986-08-28 || 10004 | 1954-05-01 | Chirstian | Koblick | 1986-12-01 || 10005 | 1955-01-21 | Kyoichi | Maliniak | 1989-09-12 |+--------+------------+------------+-----------+------------+5 rows in set (44.32 sec)

[root@Sky1 mysql_connect]# head -17 employees.html<?xml version="1.0" encoding="UTF-8"?><TABLE border="1" cellpadding="5"> <TR> <TD>10001</TD> <TD>-515379600</TD> <TD>Georgi</TD> <TD>Facello</TD> <TD>520124400</TD> </TR> <TR> <TD>10002</TD> <TD>-176173200</TD> <TD>Bezalel</TD> <TD>Simmel</TD> <TD>501379200</TD> </TR>


46

CREATE TABLE employees_csv ENGINE=connect table_type=CSVfile_name='/var/lib/mysql_connect/employees.csv'SELECT emp_no, birth_date, first_name, last_name, hire_date FROM employees.employees;Query OK, 300024 rows affected (1.36 sec)Records: 300024 Duplicates: 0 Warnings: 0

CREATE UNIQUE INDEX employees_csv ON employees_csv(emp_no);Query OK, 0 rows affected (0.11 sec)Records: 0 Duplicates: 0 Warnings: 0

EXPLAIN SELECT * FROM employees_csv WHERE emp_no = 10005;+------+-------------+---------------+-------+---------------+---------------+---------+-------+------+-------+| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |+------+-------------+---------------+-------+---------------+---------------+---------+-------+------+-------+| 1 | SIMPLE | employees_csv | const | employees_csv | employees_csv | 4 | const | 1 | |+------+-------------+---------------+-------+---------------+---------------+---------+-------+------+-------+1 row in set (0.00 sec)



47

CREATE TABLE MyFileSys ( PATH varchar(256) NOT NULL flag=1, FNAME varchar(256) NOT NULL, FTYPE char(4) NOT NULL, SIZE double(12,0) NOT NULL flag=5)engine=CONNECT table_type=DIR file_name='/var/lib/mysql/*'option_list='subdir=1';

SELECT * FROM MyFileSys;+------------------------------------+----------------------------------------------------+-------+-----------+| PATH | FNAME | FTYPE | SIZE |+------------------------------------+----------------------------------------------------+-------+-----------+| /var/lib/mysql/mysql/ | plugin | .MYI | 2048 || /var/lib/mysql/mysql/ | spider_tables | .MYD | 0 || /var/lib/mysql/mysql/ | time_zone_transition | .MYD | 0 || /var/lib/mysql/mysql/ | time_zone_transition | .MYI | 1024 || /var/lib/mysql/mysql/ | slow_log | .CSM | 35 || /var/lib/mysql/mysql/ | plugin | .frm | 1518 || /var/lib/mysql/mysql/ | db | .MYI | 9216 || /var/lib/mysql/mysql/ | help_relation | .MYI | 19456 || /var/lib/mysql/mysql/ | proxies_priv | .MYI | 10240 |...| /var/lib/mysql/ | Sky1 | .pid | 5 || /var/lib/mysql/ | multi-master | .inf | 0 |+------------------------------------+----------------------------------------------------+-------+-----------+209 rows in set (0.00 sec)


The Spider Storage EngineTuesday, 26 November 13

The Spider Storage Engine

•What is the Spider storage engine?•Spider is a storage engine based on the MySQL partitioning features, with built-‐in sharding capabilities•Tables of different MariaDB instances are handled as if they are on the same instance•It supports XA transactions and multiple storage engines (InnoDB, MyISAM etc.)

•Developed by Kentoku Shiba, available on Launchpad, first introduced in 2008 and now available in MariaDB 10


Spider Engine -‐ Under the Bonnet

•Spider expands local partitioning linking each partition to remote MariaDB tables

•Links are stored into Spider systems tables (created during the installation)

•Batched Key Access Support

•Support for Handlersocket

•XA Transactions guarantee the atomicity within the cluster


Current Limitations

•Query cache must be disabled

•Efficiency is expected in the same way partition pruning is effective on partitioning•If partition pruning is not effective, Spider will access all the remote servers, with significant overhead•Data can be also partitioned in each shard

•Still experimental!


Spider Engine -‐ An Example

52

ServerA

ServerBServerC

ServerD

Table ColorsTable ColorsTable Colors

Parhhon 1 color = ‘red’ ServerA

Parhhon 2 color = ‘black’ ServerB

Parhhon 3 color =‘white’ ServerC

Parhhon 4 color = ‘yellow’ ServerD


Spider Engine -‐ An Example

53

ServerA

ServerBServerC

ServerD

Table ColorsTable ColorsTable ColorsPartton 1 color = ‘red’ ServerAPartton 2 color= ‘black’ ServerBPartton 3 color =‘white’ ServerCPartton 4 color = ‘yellow’ ServerD


Spider Engine -‐ Hands On•Install the plugin and create Spider objects:

54

/usr/local/mariadb-10.0.5-linux-x86_64/share/install_spider.sql...or.../usr/share/mysql/install_spider.sql

MariaDB [mysql]> show tables like 'spider%';+---------------------------+| Tables_in_mysql (spider%) |+---------------------------+| spider_link_failed_log || spider_link_mon_servers || spider_tables || spider_xa || spider_xa_failed_log || spider_xa_member |+---------------------------+6 rows in set (0.00 sec)

install_spider.sql creates the procedures:spider_fix_one_tablespider_fix_system_tablesspider_plugin_installerThe procedures are removed after the installation. If you see them in the DB instance, something went wrong...


Spider Engine -‐ Hands On

55Database Database

CREATE TABLE employees ( emp_no int(11) NOT NULL, birth_date date NOT NULL, first_name varchar(14) NOT NULL, last_name varchar(16) NOT NULL, gender char(1) NOT NULL, hire_date date NOT NULL, PRIMARY KEY ( emp_no ) ) ENGINE=MyISAM;

Sky1 Sky2

Sky4CREATE TABLE employees ( emp_no int(11) NOT NULL, birth_date date NOT NULL, first_name varchar(14) NOT NULL, last_name varchar(16) NOT NULL, gender char(1) NOT NULL, hire_date date NOT NULL, PRIMARY KEY ( emp_no ) ) ENGINE=MyISAM;

1

CREATE TABLE employees ( emp_no int(11) NOT NULL, birth_date date NOT NULL, first_name varchar(14) NOT NULL, last_name varchar(16) NOT NULL, gender char(1) NOT NULL, hire_date date NOT NULL, PRIMARY KEY ( emp_no ) ) ENGINE=spider COMMENT 'wrapper "mysql", user "spider", password "spider", database "spider_test", table "employees", port "3306"' PARTITION BY HASH( emp_no ) ( PARTITION p1 COMMENT='host "Sky1"', PARTITION p2 COMMENT='host "Sky2"' );

2

INSERT INTO employees SELECT * FROM employees.employees;

3


The Cassandra Storage EngineTuesday, 26 November 13

The Cassandra Storage Engine

•What is the Cassandra storage engine?•Cassandra is a storage engine that makes the Cassandra column family appear as a table in MariaDB•The engine acts as a “window” from MariaDB to look into Cassandra•Possible operations are:•INSERT, UPDATE, SELECT•Joins between Cassandra tables and other storage engines


A quick look at Cassandra

•Open Source distributed NoSQL database

•Initially developed at Facebook, with lots of influence from Amazon Dynamo•Written in Java, first release in 2008, now part of the Apache Foundation

•Key features:•Top 1 feature: fast inserts/writes with linear scalability•Large volume of data automatically distributed on a cluster (ring)•Asynchronous masterless distribution, customizable replication•Support for geographical distribution and replication


Cassandra Data Model•Distributed key/value store (limited range scan support)•Optionally flexible schema •Pre-‐defined “static” columns•Ad-‐hoc dynamic columns

•Automatic sharding/replication•Eventual consistency

•Column families are like “tables”

•Row key -‐> column mapping

•Supercolumns are not supported in the storage engine59


Cassandra Storage Engine Data Types


Cassandra Storage Engine vs.Cassandra Query Language (CQL)

•Cassandra Query Language (CQL) is the default and primary interface into the Cassandra DBMS•The concept of a table having rows and columns is almost the same in CQL and SQL

•CQL queries are tightly bound to the way Cassandra accesses its data internally•MariaDB SQL queries have the standard MySQL format

•Cassandra does not support joins or subqueries, except for batch analysis through Hive•The Cassandra storage engine supports column family as tables that can be joined with other tables

•No GROUP BY, ORDER BY must be able to use available indexes•The Cassandra storage engine support the GROUP BY clause

•WHERE clause must represent an index lookup•The Cassandra storage engine provides the standard flexibility of the other engines in the WHERE clause


Command Mapping•MariaDB SQL Commands:•SELECT -‐> GET/Scan•INSERT -‐> PUT (upsert)•UPDATE/DELETE -‐> read + write

•INSERT works as “INSERT or UPDATE”•If a row with the same PK exists, it overwrites the row

•INSERT...SELECT and multi-‐line INSERT write data in batches•The batch size is controlled by @@cassandra_insert_batch_size

•UPDATE works as a standard SQL UPDATE

•DELETE maps to the truncate(column_family) call•A DELETE...WHERE will do a per-‐row deletion

•SELECT works as a standard MySQL

•Batched Key Access is supported


Issues and limitations

•Cassandra 1.2 has slightly changed its data model•It is described here: http://www.datastax.com/dev/blog/thrift-‐to-‐cql3.•This has caused some of Thrift-‐based clients to no longer work•for example, here's a problem experienced by Pig: CASSANDRA-‐5234

•Currently, Cassandra SE is only able to access Cassandra 1.2's column families that were defined WITH COMPACT STORAGE attribute.


http://www.datastax.com/dev/blog/thrift-to-cql3

http://www.datastax.com/dev/blog/thrift-to-cql3

https://issues.apache.org/jira/browse/CASSANDRA-5234

https://issues.apache.org/jira/browse/CASSANDRA-5234

Cassandra Engine -‐ Hands On•Install the plugin

•Create a table, which is a view of a column family

64

INSTALL PLUGIN cassandra SONAME 'ha_cassandra.so';

SET GLOBAL cassandra_default_thrift_host=‘Cassandra’CREATE TABLE cas_table ( col1 VARCHAR(36) PRIMARY KEY, col2 VARCHAR(68), col3 bigint )ENGINE=cassandra keyspace=‘mariadbtest’ thrift_host=‘Cassandra’ column_family=‘cf1’;

By using the default thrift host and not the explicit host in the table definition, the table can be

remapped dynamically to a different Cassandra cluster

This command does not create a new table in Cassandra, only a view in the Cassandra Storage Engine

A primary key is mandatory

Columns map to cassandra static columns


Cassandra Engine -‐ Hands On

65

$ cqlsh cqlsh> CREATE KEYSPACE mariadbtest2 WITH strategy_class = 'org.apache.cassandra.locator.SimpleStrategy' AND strategy_options:replication_factor='1'; cqlsh> USE mariadbtest2; cqlsh:mariadbtest2> CREATE COLUMNFAMILY cf1 ( pk VARCHAR PRIMARY KEY, data1 VARCHAR, data2 BIGINT );

1

insert into cas_t1 values ('rowkey10', 'data1-value', 123456); insert into cas_t1 values ('rowkey11', 'data1-value2', 34543); insert into cas_t1 values ('rowkey12', 'data1-value3', 454); 3

CREATE TABLE cas_t1 ( pk VARCHAR(36) PRIMARY KEY, data1 VARCHAR(60), data2 BIGINT ) ENGINE=cassandra thrift_host='localhost' keyspace='mariadbtest2' column_family='cf1';

2


Cassandra Engine -‐ Hands On

66

cqlsh:mariadbtest2>select * from cf1; pk | data1 | data2----------+--------------+-------- rowkey12 | data1-value3 | 454 rowkey10 | data1-value | 123456 rowkey11 | data1-value2 | 34543

MariaDB [cassandra_test]> select * from cas_t1;+----------+--------------+--------+| pk | data1 | data2 |+----------+--------------+--------+| rowkey12 | data1-value3 | 454 || rowkey10 | data1-value | 123456 || rowkey11 | data1-value2 | 34543 |+----------+--------------+--------+3 rows in set (0.02 sec)


MariaDB ReplicationTuesday, 26 November 13

Replication in MariaDB 10

•Group Commit

•Global Transaction ID

•Parallel Slave

•Multi-‐source replication (contribution from Lixun Peng at TaoBao)

•MySQL/MariaDB 5.X compatibility


MariaDB Replication Benefits

•Allows safe failover or switchover•GTIDs are unique and replication can be safely resume when a master change from one node to another

•Allows more sophisticated topologies

•The slave is crash safe for single and multi-‐source replication•GTIDs are saved in a transactional system table

•Faster with group commit and parallel replication


Group Commit•binlog_commits•Total number of transactions commited to the binary log

•binlog_group_commits•Total number of groups of transactions committed to the binary log•When sync_binlog=1 it is the number of fsync()’s

70

COMMIT&


Global Transaction ID -‐ GTID

•Available since 10.0.2•Enabled by default

•Treated as a new event in an event group•i.e. 1 GTID for each event group

•Unique across a group of servers

•Make the life of the DBAs much easier•Make a transaction unique throughout the whole organisation•Easy to failover to a slave•Easy to identify transactions•Make multi-‐master replication safe


72

Global Transaction ID -‐ GTID

#131109 14:59:59 server id 1 end_log_pos 4151 GTID XXX-YYY-ZZZ/*!100001 SET @@session.gtid_seq_no=ZZZ*//*!*/;

Sequence Number64bit unsigned

It increases monotonically at each commitIt is applied for each Event Group, i.e., for each BEGIN/COMMIT

or for groups that have no BEGIN/COMMIT(for example DDL commands, TRUNCATE and others)

Server ID32bit unsigned

The server_id value for the MariaDB instance

Domain ID32bit unsigned

The domain_id value for the Replication stream


MariaDB Replication Commands

73

--On the Master:SELECT BINLOG_GTID_POS("master-bin.000001", 600);

--On the Slave:SET GLOBAL gtid_slave_pos = "0-1-2";CHANGE MASTER TO master_host="Node1", master_port=3306, master_user="root", master_use_gtid=slave_pos;START SLAVE;

Start replication from a given position

STOP SLAVE;CHANGE MASTER TO master_host="Node1", master_port=3306, master_user="root", master_use_gtid=current_pos;START SLAVE;

Switch from old-‐style replication to the new GTID

STOP SLAVE;CHANGE MASTER TO master_host='Node2', master_port=3306;START SLAVE;

Change Master

START SLAVE master_gtid_pos = "1-11-100,2-21-50";

The slave will stop at a given position


Parallel Slave•Sponsored by Google•Transactions are applied in parallel if they have been executed in parallel on the master.

•Works beyond the boundaries of MySQL 5.6 parallel slave

•Parallel threads apply to:•Queries that are run on the master in one group commit.

•Queries that are from different domains.•Queries from different masters (when using multi-‐source replication).

•slave_parallel_threads•Number of parallel threads on the slave node

•slave_parallel_max_queued•Number of parallel threads on the slave node


Multi-‐source Replication

75

Stream 1 Stream 2 Stream 3•Data partitioned over many masters can be pulled together onto one slave for analytical queries

•Many masters can replicate to the same slave and a complete backup can be done on the slave

•Newer hardware usually provides more performance. Usually all hardware isn’t upgraded at once and multi-‐source can be used for replicating many masters to a powerful new slave.

•Up to 64 masters


Multi-‐source Replication•New Syntax

•You specify which master connection you want to work with by either specifying the connection name in the command or setting default_master_connection to the connection you want to work with

CHANGE MASTER ["connection_name"] ...FLUSH RELAY LOGS ["connection_name"]MASTER_POS_WAIT(....,["connection_name"])RESET SLAVE ["connection_name"]

SHOW RELAYLOG ["connection_name"] EVENTSSHOW SLAVE ["connection_name"] STATUSSHOW ALL SLAVES STATUSSTART SLAVE ["connection_name"...]]START ALL SLAVES ...STOP SLAVE ["connection_name"] ...STOP ALL SLAVES ...


https://mariadb.com/kb/en/master_pos_wait/






https://mariadb.com/kb/en/reset-slave/

https://mariadb.com/kb/en/reset-slave/

https://mariadb.com/kb/en/show-relaylog-events/

https://mariadb.com/kb/en/show-relaylog-events/

https://mariadb.com/kb/en/show-slave-status/




https://mariadb.com/kb/en/start-slave/




https://mariadb.com/kb/en/stop-slave/




GTID New Files and Structures•gtid.info (Data Directory)•On a master, fsync() the current seq_no at shutdown, it will be read at startup

•mysql.gtid_slave_pos table•GTID of the last transaction applied on the slave

•gtid variables•gtid_binlog_pos

•GTID of the last event group written to the binlog•gtid_binlog_state

•Used to restore the state of the binlog after a RESET MASTER and PURGE BINARY LOGS

•gtid_current_pos•Used when an ex master server rejoin a cluster

•gtid_domain_id

•ID of the stream•gtid_slave_pos

•Used when a slave starts• Equal to gtid_current_pos when the binlog is disabled on the slave

•gtid_strict_mode•Errors may be generated if the GTID is manually changed

•BINLOG_GTID_POS(binlogFile, binlogPos)•Provides the GTID associated to a binlog file and a position within the file


Multi-‐source Replication•Set the servers in my.cnf as for standard replication

•Assuming we will use Node 1 and Node 2 as masters and Node 3 as slave:

78

[mariadb-10.0]gtid_domain_id=1server_id=1bind_address=192.168.56.21log_bin

SET @@default_master_connection='Node1'; CHANGE MASTER TO MASTER_HOST = '192.168.56.21', MASTER_USER = 'root'; CHANGE MASTER 'Node2' TO MASTER_HOST = '192.168.56.22', MASTER_USER = 'root';

START ALL SLAVES;SHOW ALL SLAVE STATUS;



79

******************** 1. row *********************** Connection_name: Node1 Slave_SQL_State: Slave has read all... Slave_IO_State: Waiting for master... Master_Host: 192.168.56.21 Master_User: root Master_Port: 3306 Connect_Retry: 60 Master_Log_File: Sky1-bin.000004 Read_Master_Log_Pos: 520 Relay_Log_File: Sky3-relay-bin-... Relay_Log_Pos: 729 Relay_Master_Log_File: Sky1-bin.000004 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: spider_test Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 520 Relay_Log_Space: 1394 Until_Condition: None

Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_SSL_Crl: Master_SSL_Crlpath: Using_Gtid: No Retried_transactions: 0 Max_relay_log_size: 1073741824 Executed_log_entries: 1749 Slave_received_heartbeats: 36 Slave_heartbeat_period: 1800.000 Gtid_Slave_Pos: 1-1-1,2-2-1,0-2-581



80

********************* 2. row ********************* Connection_name: Node2 Slave_SQL_State: Slave has read all... Slave_IO_State: Waiting for master... Master_Host: 192.168.56.22 Master_User: root Master_Port: 3306 Connect_Retry: 60 Master_Log_File: Sky2-bin.000004 Read_Master_Log_Pos: 520 Relay_Log_File: Sky3-relay-bin-... Relay_Log_Pos: 729 Relay_Master_Log_File: Sky2-bin.000004 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: spider_test Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 520 Relay_Log_Space: 1394 Until_Condition: None

Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 2 Master_SSL_Crl: Master_SSL_Crlpath: Using_Gtid: No Retried_transactions: 0 Max_relay_log_size: 1073741824 Executed_log_entries: 1721 Slave_received_heartbeats: 37 Slave_heartbeat_period: 1800.000 Gtid_Slave_Pos: 1-1-1,2-2-1,0-2-581


Multi-‐source Replication -‐ Hands On

81

MariaDB [spider_test]> CREATE SCHEMA server_1;Query OK, 1 row affected (0.00 sec)

MariaDB [spider_test]> USE server_1

CREATE TABLE èmployees` ( èmp_no` int(11) NOT NULL, `birth_date` date NOT NULL, `first_name` varchar(14) NOT NULL, `last_name` varchar(16) NOT NULL, `gender` enum('M','F') NOT NULL, `hire_date` date NOT NULL, PRIMARY KEY (èmp_no`)) ENGINE=InnoDB;

Node 1



82







Node 2



83







Node 2

MariaDB [spider_test]> SHOW SCHEMAS;+--------------------+| Database |+--------------------+| connect_examples || employees || information_schema || mysql || performance_schema || server_1 || server_2 || spider_test || test |+--------------------+9 rows in set (0.00 sec)

Node 3



86



Node 3



MariaDB [spider_test]> SELECT * FROM employees;+--------+------------+------------+-----------+--------+------------+| emp_no | birth_date | first_name | last_name | gender | hire_date |+--------+------------+------------+-----------+--------+------------+| 10001 | 1953-09-02 | Georgi | Facello | M | 1986-06-26 || 10002 | 1964-06-02 | Bezalel | Simmel | F | 1985-11-21 |+--------+------------+------------+-----------+--------+------------+2 rows in set (0.00 sec)



87





MariaDB [spider_test]> SELECT * FROM employees;+--------+------------+------------+-----------+--------+------------+| emp_no | birth_date | first_name | last_name | gender | hire_date |+--------+------------+------------+-----------+--------+------------+| 10001 | 1953-09-02 | Georgi | Facello | M | 1986-06-26 || 10002 | 1964-06-02 | Bezalel | Simmel | F | 1985-11-21 |+--------+------------+------------+-----------+--------+------------+2 rows in set (0.00 sec)

There is no conflict resolution -‐ conflicting operations will stop replication.

For example: CREATE TABLE executed on both masters


Multi-‐Master Replication

88

Stream 1 Stream 2

Stream 3


All the rest!Tuesday, 26 November 13

Virtual & Dynamic Columns

90

VIRTUAL COLUMNS•For InnoDB, MyISAM and Aria•PERSISTENT (stored) or VIRTUAL (generated)

CREATE TABLE t1 ( c1 INT NOT NULL, c2 VARCHAR(32), c3 INT AS ( c1 MOD 10 ) VIRTUAL, c4 VARCHAR(5) AS ( LEFT(B,5) ) PERSISTENT);

DYNAMIC COLUMNS•Implement a schemaless,

document store•COLUMN_ CREATE, ADD, GET, LIST,

JSON, EXISTS, CHECK, DELETE•Nested colums are allowed•Main datatypes are allowed•Max 1GB documents

CREATE TABLE assets ( item_name VARCHAR(32) PRIMARY KEY, dynamic_cols BLOB );

INSERT INTO assets VALUES ( 'MariaDB T-shirt', COLUMN_CREATE( 'color', 'blue', 'size', 'XL' ) );INSERT INTO assets VALUES ( 'Thinkpad Laptop', COLUMN_CREATE( 'color', 'black', 'price', 500 ) );


TokuDB Storage Engine

• Fast inserts/increased performance

• Increased Compression

• Online administration

• No Index rebuild


SSD capabilities andAtomic Writes with FusionIO

innodb_use_atomic_writesinnodb_doublewrite=0innodb_file_flush_method= O_DIRECT | ALL_O_DIRECT | O_DIRECT_NO_FSYNC

•Used with DirectFS

•Lower latency

•Increased Flash life

•Less write amplification


Audit Plugin

•Logs the server activity•Who connected to the server•What queries a user ran•What tables a user touched

•Stored to a rotating log file or sent to the local syslogd.

•https://mariadb.com/kb/en/server_audit-‐plugin/


94

MaxScaleCore

Data Stores

Client Applications

Backend Backend Backend

Frontend Frontend Frontend Frontend Frontend

Clie

nt

Clie

nt

Clie

nt

Clie

nt

Clie

nt

Router

Router

Router

Monitor

Monitor

Auth

Auth

Auth

Service Service Service

Listener Listener Listener Listener

Server Server Server ServerServer

SkySQLMaxScale


MariaDB Manager?

•MariaDB Manager is a set of servers used to provision, administer and monitor MariaDB servers

•The servers can be co-‐located with MariaDB, they can be installed and co-‐located on a separate machine or they can be installed on many separate machines.


MariaDBManagerArchitecture

96

RESTful(APICore

Monitor/ManagementInterface



Monitor(DataProvider

Agent Agent Agent

MariaDB(Galera(Cluster


Provision a new node

97

HTTP POST method -‐ Create a nodeURI:

• .../restfulapi/system/systemid/node• Parameters:• name• hostname• publicip, privateip, port• instanceid• dbusername, dbpassword• ...

1



98



1

HTTP POST method -‐ Run the commandConnect to the new nodeURI:

• .../restfulapi/command/connect• Parameters:• systemid• nodeid• rootpassword

2



99



1



2

HTTP POST method -‐ Run the commandProbe the state of the nodeURI:

• .../restfulapi/command/probe• Parameters:• systemid• nodeid

3



100



1



2

HTTP POST method -‐ Run the commandProbe the state of the nodeURI:

• .../restfulapi/command/probe• Parameters:• systemid• nodeid

3

HTTP POST method -‐ Run the commandProvision the nodeURI:

• .../restfulapi/command/provision• Parameters:• systemid• nodeid

4


Start a node

101

HTTP POST method -‐ Run the commandStart the nodeURI:

• .../restfulapi/command/start• Parameters:• systemid• nodeid


Retrieve the status of a cluster

102

• .../restfulapi/system/systemid


Monitor the # of connections of a node

103

• ../reslulapi/system/systemid/node/nodeid/monitor/{monitorid}/data


For More Information...

104

• Web, Doc & Knowledge Base:www.mariadb.orgwww.mariadb.com

• Launchpad:https://launchpad.net/mariahttps://code.launchpad.net/mariadb-‐native-‐client

• Jira (bugs and development):mariadb.atlassian.net

• IRC:irc.freenode.net #mariawebchat.freenode.net/?randomnick=1&channels=maria


http://www.mariadb.org

http://www.mariadb.org

http://www.mariadb.com

http://www.mariadb.com

https://launchpad.net/maria

https://launchpad.net/maria

https://code.launchpad.net/mariadb-native-client




http://bugs.skysql.com

http://bugs.skysql.com

http://webchat.freenode.net/?randomnick=1&channels=maria




Thank You!

[email protected]

www.slideshare.net/skysqlwww.slideshare.net/izorattiwww.skysql.com

Some pictures courtesy of:• h{p://www.worksmartmompreneurs.com• h{p://www.nydailynews.com


mailto:[email protected]










http://www.interflora.co.uk




mariadb 10 tutorial - 13.11.11 - percona live london

Technology