Preparse Query Rewrite PluginsNew SQL syntax for fun & performance

Sveta SmirnovaPrincipal Support Engineer

January, 30, 2016

Table of Contents

•Introducing new SQL syntax

•Working with results

•Variables

•Summary

2 www.percona.com

Introducing newSQL syntax

3 www.percona.com

MySQL often receives blames• From mascots and from humans

• It cannot make a toast• It does not support some syntax

4 www.percona.com

MySQL often receives blames

• From mascots and from humans

• It cannot make a toast

• It does not support some syntax

4 www.percona.com

MySQL often receives blames• From mascots and from humans• It cannot make a toast• It does not support some syntax

4 www.percona.com

Or does it?

• FILTER clause in MySQL on my homemachine

5 www.percona.com

Or does it?

• FILTER clause in MySQL on my homemachine

5 www.percona.com

Or does it?

• FILTER clause in MySQL on my homemachine

• But not in the user manual

5 www.percona.com

How is it done?

• With 181 lines of code

• Including comments!

• And new Query Rewrite Plugin interface

6 www.percona.com

A little bit of history

• First introduced in version 5.7.5

• Was available at MySQL Labs

• Two types of plugins

• Pre-parse

• Post-parse

7 www.percona.com

Today

• Part of Audit plugin interface• Step in at

• MYSQL AUDIT GENERAL ALL• MYSQL AUDIT CONNECTION ALL• MYSQL AUDIT PARSE ALL

• MYSQL AUDIT PARSE PREPARSE

• MYSQL AUDIT PARSE POSTPARSE

• MYSQL AUDIT AUTHORIZATION ALL• ...

8 www.percona.com

Plugin skeleton

#include <mysql/plugin.h>

#include <mysql/plugin_audit.h> - Audit plugin declaration

...

static MYSQL_PLUGIN plugin_info_ptr; - Pointer to the plugin

...

static int filter_plugin_init(MYSQL_PLUGIN plugin_ref); - Plugin initialization

...

static int filter(MYSQL_THD thd, mysql_event_class_t event_class,

const void *event); - Entry point for MYSQL_AUDIT_PARSE_PREPARSE

...

static st_mysql_audit filter_plugin_descriptor;

...

mysql_declare_plugin(filter_plugin);

9 www.percona.com

Plugin descriptor

static st_mysql_audit filter_plugin_descriptor= {

MYSQL_AUDIT_INTERFACE_VERSION, /* interface version */

NULL,

filter, /* implements FILTER */

// You can also use MYSQL_AUDIT_PARSE_ALL

{ 0, 0, (unsigned long) MYSQL_AUDIT_PARSE_PREPARSE,}

};

10 www.percona.com

Plugin declaration

mysql_declare_plugin(filter_plugin)

{

MYSQL_AUDIT_PLUGIN,

&filter_plugin_descriptor,

"filter_plugin",

"Sveta Smirnova",

"FILTER SQL:2003 support for MySQL",

PLUGIN_LICENSE_GPL,

filter_plugin_init,

NULL, /* filter_plugin_deinit - TODO */

0x0001, /* version 0.0.1 */

NULL, /* status variables */

NULL, /* system variables */

NULL, /* config options */

0, /* flags */

}

mysql_declare_plugin_end;

11 www.percona.com

Memory management for plugins#include <my_thread.h> // my_thread_handle needed by mysql_memory.h

#include <mysql/psi/mysql_memory.h>

...

static PSI_memory_key key_memory_filter;

static PSI_memory_info all_rewrite_memory[]=

{

{ &key_memory_filter, "filter", 0 }

};

static int filter_plugin_init(MYSQL_PLUGIN plugin_ref)

{

plugin_info_ptr= plugin_ref;

const char* category= "sql";

int count;

count= array_elements(all_rewrite_memory);

mysql_memory_register(category, all_rewrite_memory, count);

return 0; /* success */

}

12 www.percona.com

SQL:2003

<filter clause> ::=

FILTER <left paren> WHERE <search condition> <right paren>

(10.9 <aggregate function>, 5WD-02-Foundation-2003-09.pdf, p.505)

Only for aggregate functions:

<computational operation> ::=

AVG | MAX | MIN | SUM | EVERY | ANY

| SOME | COUNT | STDDEV_POP | STDDEV_SAMP

| VAR_SAMP | VAR_POP | COLLECT | FUSION | INTERSECTION

<set quantifier> ::=

DISTINCT

| ALL

MySQL only supports

COUNT | AVG | SUM | MAX | MIN

| STDDEV_POP | STDDEV_SAMP

| VAR_SAMP | VAR_POP

13 www.percona.com

Implementing FILTER clause

• FILTER is practically

CASE WHEN foo THEN bar ELSE NULL

• So we only need to catch

FUNCTION(var) FILTER(WHERE foo)

• And replace it with CASE

14 www.percona.com

Catching up the querystatic int filter(MYSQL_THD thd, // MySQL Thread object

mysql_event_class_t event_class, // Class of the event

const void *event // Event itself

)

{

const struct mysql_event_parse *event_parse=

static_cast<const struct mysql_event_parse *>(event);

if (event_parse->event_subclass != MYSQL_AUDIT_PARSE_PREPARSE)

return 0;

string subject= event_parse->query.str; // Original query

string rewritten_query;

//requires std::regex and GCC 4.9+

regex filter_clause_star("(COUNT)\((\s*\*\s*)\)\s+"

+ "FILTER\s*\(\s*WHERE\s+([^\)]+)\s*\)",

ECMAScript | icase);

rewritten_query= regex_replace(subject, filter_clause_star,

"$1(CASE WHEN $3 THEN 1 ELSE NULL END)");

...

15 www.percona.com

Rewritten query

void _rewrite_query(const void *event,

const struct mysql_event_parse *event_parse,

char const* new_query

)

{

char *rewritten_query= static_cast<char *>(my_malloc(

key_memory_filter, strlen(new_query) + 1, MYF(0)));

strncpy(rewritten_query, new_query, strlen(new_query));

rewritten_query[strlen(new_query)]= ’\0’;

event_parse->rewritten_query->str= rewritten_query; // Rewritten query

event_parse->rewritten_query->length=strlen(new_query);

// You must set this flag to inform MySQL Server what query was rewritten

*((int *)event_parse->flags)|=

(int)MYSQL_AUDIT_PARSE_REWRITE_PLUGIN_QUERY_REWRITTEN;

}

16 www.percona.com

Working withresults

17 www.percona.com

Can we do better?

• Playing with syntax is fun

• But can we introduce something moreMySQL-ish?

18 www.percona.com

Custom hint plugin

• MySQL 5.7 has Optimizer HintsSELECT /*+ NO_RANGE_OPTIMIZATION(t3 PRIMARY, f2_idx) */ f1

FROM t3 WHERE f1 > 30 AND f1 < 33;

SELECT /*+ BKA(t1) NO_BKA(t2) */ * FROM t1 INNER JOIN t2 WHERE ...;

SELECT /*+ NO_ICP(t1, t2) */ * FROM t1 INNER JOIN t2 WHERE ...;

SELECT /*+ SEMIJOIN(FIRSTMATCH, LOOSESCAN) */ * FROM t1 ...;

EXPLAIN SELECT /*+ NO_ICP(t1) */ * FROM t1 WHERE ...;

• But sometimes thread-specific buffersaffect query execution

• Workaround requires processing result setof each of these statements

• This is why I extended optimizer hintsyntax

19 www.percona.com

Custom hint plugin

• MySQL 5.7 has Optimizer Hints• But sometimes thread-specific buffers

affect query execution• Common workaround exists:

SET tmp_table_size=1073741824;

SELECT * FROM t1 INNER JOIN t2 WHERE ...;

SET tmp_table_size=DEFAULT;

• Workaround requires processing result setof each of these statements

• This is why I extended optimizer hintsyntax

19 www.percona.com

Custom hint plugin

• MySQL 5.7 has Optimizer Hints• But sometimes thread-specific buffers

affect query execution• Workaround requires processing result set

of each of these statements• This is why I extended optimizer hint

syntaxSELECT /*+ join_buffer_size=16384 */ f1 FROM t3 WHERE f1 > 30 AND f1 < 33;

SELECT /*+ tmp_table_size=1073741824 BKA(t1) NO_BKA(t2) */ *

FROM t1 INNER JOIN t2 WHERE ...;

19 www.percona.com

New features

• For Custom Hints we need to:

• Store previous values of threadvariables we are going to modify

• Modify variables

• Revert them back before sending result

20 www.percona.com

Store previous values

// map to store modified variables

static map <my_thread_id, map<supported_hints_t, ulonglong> > modified_variables;

...

/* The job */

static int custom_hint(MYSQL_THD thd, mysql_event_class_t event_class,

const void *event)

{

...

// If we have a match store create map of thread variables

std::map<supported_hints_t, ulonglong> current;

...

After processing variables store them in modified_variables map

modified_variables[thd->thread_id()]= current;

...

21 www.percona.com

Modify variables

• Since we have access to MYSQL THDthis is easy:

switch(get_hint_switch(ssm[1]))

{

case JOIN_BUFFER_SIZE:

current[JOIN_BUFFER_SIZE]= thd->variables.join_buff_size;

thd->variables.join_buff_size= stoull(ssm[2]);

break;

case TMP_TABLE_SIZE:

current[TMP_TABLE_SIZE]= thd->variables.tmp_table_size;

thd->variables.tmp_table_size= stoull(ssm[2]);

break;

...

22 www.percona.com

Revert variables back

• First we need to tell plugin descriptor whatwe needMYSQL AUDIT GENERAL RESULTstatic st_mysql_audit custom_hint_plugin_descriptor= {

MYSQL_AUDIT_INTERFACE_VERSION, /* interface version */

NULL,

custom_hint, /* implements custom hints */

{ (unsigned long) MYSQL_AUDIT_GENERAL_RESULT, 0,

(unsigned long) MYSQL_AUDIT_PARSE_PREPARSE,

}

};

• Then revert variables before sending result• And, finally, erase stored values for current

thread:

23 www.percona.com

Revert variables back• First we need to tell plugin descriptor what

we needMYSQL AUDIT GENERAL RESULT

• Then revert variables before sending resultif (event_general->event_subclass == MYSQL_AUDIT_GENERAL_RESULT)

{

map<my_thread_id, map<supported_hints_t, ulonglong> >::iterator

current= modified_variables.find(thd->thread_id());

if (current != modified_variables.end())

{

for (map<supported_hints_t, ulonglong>::iterator it=

current->second.begin(); it!= current->second.end(); ++it)

{

switch(it->first)

{

case JOIN_BUFFER_SIZE:

thd->variables.join_buff_size= it->second;

break;

• And, finally, erase stored values for currentthread:

23 www.percona.com

Revert variables back

• First we need to tell plugin descriptor whatwe needMYSQL AUDIT GENERAL RESULT

• Then revert variables before sending result

• And, finally, erase stored values for currentthread:

modified_variables.erase(current);

23 www.percona.com

Before Custom Hint Pluginmysql> flush status;

Query OK, 0 rows affected (0.00 sec)

mysql> select count(*), sum(c) from

-> (select s, count(s) c from joinit where i < 1000000 group by s) t;

+----------+--------+

| count(*) | sum(c) |

+----------+--------+

| 737882 | 737882 |

+----------+--------+

1 row in set (24.70 sec)

mysql> show status like ’Created_tmp_disk_tables’;

+-------------------------+-------+

| Variable_name | Value |

+-------------------------+-------+

| Created_tmp_disk_tables | 2 | -- 2 temporary tables on disk

+-------------------------+-------+

1 row in set (0.00 sec)

24 www.percona.com

Custom Hint Plugin at workmysql> flush status;

Query OK, 0 rows affected (0.00 sec)

mysql> select /*+ tmp_table_size=134217728 max_heap_table_size=134217728 */

-> count(*), sum(c) from

-> (select s, count(s) c from joinit where i < 1000000 group by s) t;

+----------+--------+

| count(*) | sum(c) |

+----------+--------+

| 737882 | 737882 |

+----------+--------+

1 row in set, 2 warnings (6.21 sec) -- 4 times speed gain!

mysql> show status like ’Created_tmp_disk_tables’;

+-------------------------+-------+

| Variable_name | Value |

+-------------------------+-------+

| Created_tmp_disk_tables | 0 | -- No disk-based temporary table!

+-------------------------+-------+

1 row in set (0.00 sec)

25 www.percona.com

Variables

26 www.percona.com

BACKUP DATABASE plugin

• Very simple syntax

• mysql> BACKUP SERVER;

+---------------------------------+

| Backup finished with status OK! |

+---------------------------------+

| Backup finished with status OK! |

+---------------------------------+

1 row in set, 1 warning (42.92 sec)

• Supports many tools

• Needs to pass options

27 www.percona.com

BACKUP DATABASE plugin

• Very simple syntax• Supports many tools

• mysqldump• mysqlpump• mysqlbackup• XtraBackup - Planned!

• Needs to pass options

27 www.percona.com

BACKUP DATABASE plugin

• Very simple syntax

• Supports many tools

• Needs to pass options

• Credentials

• Backup directory

• Custom

27 www.percona.com

Customization: credentials

• We have access to• MYSQL_THR->security_context

• thd->security_context()->user().str

• Password still has to be in the configuration file, under

[client]

or

[toolname]

section

• System variables• Since we are interested in backing up local server we will use

mysqld_unix_port

28 www.percona.com

Customization: variables

• Global variables - Example only!

static MYSQL_SYSVAR_STR(backup_dir, backup_dir_value, PLUGIN_VAR_MEMALLOC,

"Default directory...", NULL, NULL, NULL);

static MYSQL_SYSVAR_ENUM(backup_tool, backup_tool_name,

PLUGIN_VAR_RQCMDARG, "Backup tool. Possible values:

mysqldump|mysqlbackup", NULL, NULL,

MYSQLDUMP, &supported_tools_typelib);

• Thread variables

• Add to plugin declaration

29 www.percona.com

Customization: variables

• Global variables - Example only!

• Thread variables

static MYSQL_THDVAR_STR(backup_dir, PLUGIN_VAR_MEMALLOC,

"Default directory...", NULL, NULL, NULL);

static MYSQL_THDVAR_ENUM(backup_tool, PLUGIN_VAR_RQCMDARG,

"Backup tool. Possible values:

mysqldump|mysqlbackup|mysqlpump", NULL, NULL,

MYSQLDUMP, &supported_tools_typelib);

...

• Add to plugin declaration

29 www.percona.com

Customization: variables• Global variables - Example only!• Thread variables• Add to plugin declaration

static struct st_mysql_sys_var *mysqlbackup_plugin_sys_vars[] = {

MYSQL_SYSVAR(backup_dir),

MYSQL_SYSVAR(backup_tool),

MYSQL_SYSVAR(backup_tool_basedir),

MYSQL_SYSVAR(backup_tool_options),

NULL

};

mysql_declare_plugin(mysqlbackup_plugin)

{

MYSQL_AUDIT_PLUGIN,

&mysqlbackup_plugin_descriptor,

"mysqlbackup_plugin",

...

NULL, /* status variables */

mysqlbackup_plugin_sys_vars, /* system variables */

...

29 www.percona.com

Summary

30 www.percona.com

More possibilities

• Custom locks

• Access to thread- and server-specificvariables

• Fine control at multiple steps of queryexecution

• More

31 www.percona.com

Code

• https://github.com/svetasmirnova/

• filter plugin

• custom hint plugin

• mysqlbackup plugin

32 www.percona.com

More information

• MySQL source dir/plugin• rewriter• rewrite example

• Writing Audit Plugins manual• MySQL Services for Plugins manual

33 www.percona.com

Place for your questions

???

34 www.percona.com

Thank you!

http://www.slideshare.net/SvetaSmirnova

https://twitter.com/svetsmirnova

35 www.percona.com