Dave Stokes David.Stokes@oracle.com @stoker

MySQL's NoSQL -MySQL's NoSQL -Best of Both Worlds on One Set of DisksBest of Both Worlds on One Set of Disks

Slideshare.net/davestokesSlideshare.net/davestokes

SQL

● Structured Query Language

– Store data efficiently, no or minimal duplication

– Set theory & relational calculus

– Declarative Language● Made up of DML & DDL

– Data Manipulation Language– Data Description Language

● Does not always mesh with procedural or object oriented languages

– Not well taught

NoSQL

● Not relational, usually schema-less

● Don't know how data will look, or care

● Not 'new'

– Berkeley Data Base● Scaling problems

● Usually not a general purpose data store

SQL & NoSQL – different platforms

● Many companies end up with:

– Separate RDMS environment● Many nodes

– Sharding– Replication

– Separate NoSQL environment● Thousands of nodes

– Tools (lots of 'em) to move between the two environments

– Minimal cross functionality

But what if you could...

● Use the same server/disks?

● Access data as SQL and key/value pair

– The same data?

– The same time?● Easily convert RDMS schemas into NoSQL?

● Drink from the NoSQL 'fire hose', use SQL to query

● Keep ACID

● Not have to learn an entire ecosystems of new tools

Some history

● Yohinori Matsunobu, then DeNA now Facebook creates Handler Socket in ~2010

– 750,000 qps on commodity hardware

– Used memcached as front end cache but not for for storing rows

– http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html

MySQL's InnoDB Memcached Plugin

● It uses a standard memcached protocol, with drivers available for many programming languages.

● It does not automatically allow access to all tables in your database, but instead tables 'opt-in' via a mapping containers table.

● 1,100,000 qps – http://dimitrik.free.fr/blog/archives/2013/11/mysql-performance-over-1m-qps-with-innodb-memcached-plugin-in-mysql-57.html

Benefits of the InnoDB / memcached Combination

● The combination of InnoDB tables and memcached offers advantages over using either by themselves:

– Direct access to the InnoDB storage engine avoids the parsing and planning overhead of SQL.

– Running memcached in the same process space as the MySQL server avoids the network overhead of passing requests back and forth.

– Data that is written using the memcached protocol is transparently written to an InnoDB table, without going through the MySQL SQL layer. You can control the frequency of writes to achieve higher raw performance when updating non-critical data.

Benefits of the InnoDB / memcached Combination

● Subsequent requests for the same data is served from the InnoDB buffer pool. The buffer pool handles the in-memory caching. You can tune the performance of data-intensive operations using the familiar InnoDB configuration options.

● Data can be unstructured or structured, depending on the type of application. You can make an all-new table for the data, or map the NoSQL-style processing to one or more existing tables.

● InnoDB can handle composing and decomposing multiple column values into a single memcached item value, reducing the amount of string parsing and concatenation required in your application. For example, you might store a string value 2|4|6|8 in the memcached cache, and InnoDB splits that value based on a separator character, then stores the result into four numeric columns.

Benefits continued

● The transfer between memory and disk is handled automatically, simplifying application logic.

● Data is stored in a MySQL database to protect against crashes, outages, and corruption.

● You can still access the underlying table through SQL, for reporting, analysis, ad hoc queries, bulk loading, multi-step transactional computations, set operations such as union and intersection, and other operations well suited to the expressiveness and flexibility of SQL.

● You can ensure high availability of the NoSQL data by using this feature on a master server in combination with MySQL replication.

More benefits

● The integration of memcached with MySQL provides a painless way to make the in-memory data persistent, so you can use it for more significant kinds of data. You can put more add, incr, and similar write operations into your application, without worrying that the data could disappear at any moment. You can stop and start the memcached server without losing updates made to the cached data. To guard against unexpected outages, you can take advantage of InnoDB crash recovery, replication, and backup procedures.

● The way InnoDB does fast primary key lookups is a natural fit for memcached single-item queries. The direct, low-level database access path used by the memcached plugin is much more efficient for key-value lookups than equivalent SQL queries.

● The serialization features of memcached, which can turn complex data structures, binary files, or even code blocks into storeable strings, offer a simple way to get such objects into a database.

Even more benfits

● Because you can access the underlying data through SQL, you can produce reports, search or update across multiple keys, and call functions such as AVG() and MAX() on the memcached data. All of these operations are expensive or complicated with the standalone memcached.

● You do not need to manually load data into memcached at startup. As particular keys are requested by an application, the values are retrieved from the database automatically, and cached in memory using the InnoDB buffer pool.

● Because memcached consumes relatively little CPU, and its memory footprint is easy to control, it can run comfortably alongside a MySQL instance on the same system.

● Because data consistency is enforced by the mechanisms used for regular InnoDB tables, you do not have to worry about stale memcached data or fallback logic to query the database in the case of a missing key.

The Architecture

Installation:

● Because the memcached library comes with the MySQL server, installation and setup are straightforward. You run a SQL script to set up a table for memcached to use, issue a one-time install plugin statement to enable memcached, and add to the MySQL configuration file or startup script any desired memcached options, for example to use a different port. You might still install the regular memcached distribution to get the additional utilities such as memcp, memcat, and memcapable.

Programming interfaces:

● You can access the MySQL server through the InnoDB and memcached combination using the same language as always: C and C++, Java, Perl, Python, PHP, and Ruby.

Specify the server hostname and port as with any other memcached server. By default, the integrated memcached server listens on the same port as usual, 11211. You can use both the text and binary protocols.

Prerequisites for the InnoDB memcached Plugin

● Currently, the memcached Daemon Plugin is only supported on Linux, Solaris, and OS X platforms.

● You must have libevent installed, since it is required by memcached

Setup

● To configure the memcached plugin so it can interact with InnoDB tables, run the innodb_memcached_config.sql configuration script to install the necessary tables used behind the scenes:

mysql> source MYSQL_HOME/share/innodb_memcached_config.sql

● This is a one-time operation. The tables remain in place if you later disable and re-enable the memcached support.

Setup creates three tables

● In database innodb_memcache

– cache_policies

– config_options

– Containers● Plus test.demo_test

– (there is also one in innodb_memcache but unused)

–

containers

Key

Column used for K/V lookup

Data Store

cache_policies & config_options

Policies can be InnoDB, Memory, or both!

Load plugin

● To activate the daemon plugin, use the install plugin statement, just as when installing any other MySQL plugin:

– mysql> install plugin daemon_memcached soname "libmemcached.so";

– Once the plugin is installed this way, it is automatically activated each time the MySQL server is booted or restarted.

● Disabling the Daemon Plugin

– When making major changes to the plugin configuration, you might need to turn off the plugin. To do so, issue the following statement:

– mysql> uninstall plugin daemon_memcached;

Test from memcached

An example using telnet to send memcached commands and receive results through the ASCII protocol:

telnet 127.0.0.1 11211set a11 10 0 9 – set 'a11' 'flag' 'ttl' size123456789STOREDget a11VALUE a11 0 9123456789ENDquit

From SQL side

To prove that all the same data has been stored in MySQL, connect to the MySQL server and issue:

mysql> select * from test.demo_test;

So what else can it do?!?!?

● daemon_memcached_w_batch_size

– Use to commit batches, 1=every record● innodb_api_disable_rowlock

– SET and GET operations w/o locks● innodb_memcache.cache_policies

– Memory, disk, or both

– Set independently for SET, INCR, GET, DECR, DELETE, and FLUSH

– Use to have DELETE/FLUSH work on memory, not disk

Remember

● Memcached may be flushing slowly compared to InnoDB

– Use READ_UNCOMMITED SQL mode

● Memcached data can be sent to replication slaves if binary logging enabled

– innodb_api_enable_binlog

● More details in MySQL Manual

– 14.18 InnoDB Integration with memcachedhttp://dev.mysql.com/doc/refman/5.6/en/innodb-memcached.html

5.6 good, 5.7 better

Another Approaches with MySQL 5.7

● MySQL's Http Plugin

– Lets lightweight applications use JSON to use database without using intermediate server

– three APIs: ● key-document for nested JSON documents● CRUD for JSON mapped SQL tables● SQL with JSON replies

● JSON UDFs

– 0.3.3 January release

Q&A

● Slides: slideshare.net/davestokes

● Tweets: @stoker

● Email: David.Stokes@Oracle.com

● Blog: OpenSourceDBA.Wordpress.com