database cluster as a service in openstack

Database  Cluster  as  a  Service  in  OpenStackIntegrated,  Scalable,  Highly  Available  and  Secure

Ivan  Zora>,  CTO  -­‐  14.11.2013

Thursday, 14 November 13

Who  is  Ivan

?Thursday, 14 November 13

SkySQL•Leading  provider  of  open  source  databases,  services  and  solutions

•Home  for  the  founders  and  the  original  developers  of  the  core  of  MySQL

•The  creators  of  MariaDB,  the  drop-­‐in,  innovative  replacement  of  MySQL

Thursday, 14 November 13

MySQL  ,  MariaDB  and  Percona  Serverin  the  Cloud

Thursday, 14 November 13

The  Status  of  MySQLMySQL  

•From  Oracle,  acquired  with  Sun  in  2010

•Trademark,  code,  documentation,  bugs  are  available  at  *.mysql.com

•5.6  GA  now•5.7  DMR  available  for  testing

MariaDB

•5.5  GA:  branch  from  Oracle  with  add-­‐ons  from  Percona  and  other  key  contributors

•10.0  Alpha:  fork  from  Oracle  with  add-­‐ons  and  new  storage  engines  

Percona

•5.6  GA:  branch  from  Oracle  with  add-­‐ons  from  MariaDB  and  other  key  contributors

5Thursday, 14 November 13

More  in-­‐depth  featuresMySQL  

•5.6• Scales  up  to  48  CPU  threads• Performance  Schema• Memcached  plugin  for  InnoDB• InnoDB  API• InnoDB  Full  Text  Indexing• Replication  with  Global  Transaction  ID• HA  utilities• Improved  DDL  online  operations• Transportable  tablespaces• Subquery  optimization• Batched  key  access  and  multi-­‐range  read

• Thread  pool• PAM  Authentication• Audit  plugin

MariaDB•5.5

• Subquery  optimization• Batched  key  access  and  multi-­‐range  read

• Index  condition  pushdown• and  many,  many  other  improvements  in  the  optimizer...

• SphinxSE  and  ScaleDB  engines• Handler  Socket• Dynamic  and  virtual  columns• Group  Commit  for  binary  log• Microsecond  Support• GIS  operations• XtraDB  Engine• PAM  Authentication• Thread  pool• Audit  plugin

Percona•5.6

• Porting  of  the  MySQL  5.6  features• XtraDB  5.6• Per-­‐table/index/user/client  performance  counters

• FlashCache  support

6Thursday, 14 November 13

More  in-­‐depth  featuresMySQL  

•5.7• Improved  DDL  online  operations• EXPLAIN  improvements• Improved  triggers• MySQL  Client  >  syslog• Parallel  thread  replication• EXPLAIN  in  JSON  format• EXPLAIN  for  running  statements• Subquery  optimization• Loss-­‐less  MySQL  Replication• Multi-­‐source  replication

MariaDB

•10.0• Includes  XtraDB  5.6• Performance  schema• Fusion  I/O  atomic  writes• Further  optimizer  improvements• Improved  DDL  online  operations• Replication  with  Global  Transaction  ID• Parallel  thread  replication• Multi-­‐source  replication• EXPLAIN  for  running  threads  and  logged  queries

• Engine-­‐independent  table  statistics• Per-­‐engine  table  attributes• Per-­‐thread  memory  usage

• TokuDB,  Spider,  CASSANDRA  and  CONNECT  storage  engine

7Thursday, 14 November 13

High  AvailabilityMySQL  

•MySQL  Replication  with  HA  utilities

•DRBD  from  Linbit•Shared  storage  with  Oracle  Enterprise  Linux

MariaDB•MySQL  Replication  with  MHA  and  Pacemaker

•MariaDB  Galera  Cluster  Synchronous  Replication  with  Codership  Galera

•Support  from  SkySQL  for:•DRBD•Shared  storage

Percona•MySQL  Replication  with  Percona  Replication  Manager

•Percona  XtraDB  ClusterSynchronous  Replication  with  Codership  Galera

•Support  for:•DRBD•Shared  storage

8Thursday, 14 November 13

ScalabilityMySQL  

• Improved  InnoDB  scalability•Group  commit,  parallel  and  multi-­‐source  replication  in  5.7

•MySQL  Replication  for  read  scalability

•MySQL  Cluster  for  high  volume/small  transactions

MariaDB• In  MariaDB  5.5:

• XtraDB  5.5• Group  commit•MySQL  Replication  for  read  scalability

• TokuDB  and  ScaleDB  engine

• Plus,  in  MariaDB  10:• XtraDB  5.6• Parallel  and  multi-­‐source  replication• Spider  and  Cassandra  engine

• MariaDB  Galera  Cluster  for  synchronous  replication

• MySQL  Cluster  for  high  volume/small  transactions

Percona

•XtraDB  5.6•Group  commit•MySQL  Replication  for  read  scalability

•Percona  XtraDB  Cluster  for  synchronous  replication

9Thursday, 14 November 13

Ease  of  UseMySQL  

•MySQL  Replication  Utilities

•Enterprise  Monitoring

•Online  Backup•MySQL  Workbench

MariaDB

•MHA

•Enterprise  Monitor

•Xtrabackup•Query  Editor•MariaDB  Manager

Percona

•Percona  Replication  Manager

•Nagios  and  Cacti  plugins•Xtrabackup•Percona  Toolkit

10Thursday, 14 November 13

MySQL  in  Rackspace

•Rackspace  Cloud  Database•MySQL  Community  5.1•Up  to  16GB  RAM,  150GB  storage

•Standard  Servers•Up  to  8  vCPUs,  30GB  RAM,  1.2TB  storage•BYODBMS

11Thursday, 14 November 13

MySQL  in  HPCloud

•Manage  a  single  DB  instance  through  the  REST  API

•Standard  Servers•Up  to  8  vCPUs,  30GB  RAM,  960GB  storage•BYODBMS

•Marketplace•GenieDB•MariaDB  ScaleDB

12Thursday, 14 November 13

MySQL  in  AWS

•Amazon  RDS•MySQL  Community  5.1,  5.5  &  5.6•Reserved  instances,  provisioned  IO  and  Flash  storage•32  vCPUs,  244GB  Memory,  10GB  network

•Automatic  backups,  Storage  replication,  MySQL  Replication•Limited  tuning

•Standard  Servers  and  Marketplace•Similar  instances•MariaDB  on  CentOS,  Debian  and  Ubuntu•GenieDB

13Thursday, 14 November 13

MySQL  and  OpenStackThursday, 14 November 13

MySQL  [or  MariaDB,  or  Percona]  in  OpenStack

•Used  as  repository  -­‐  alternative  to  SQLite•Increased  portability  and  availability

•High  Availability  is  the  key  factor:•MySQL  Replication•DRBD•Galera

15Thursday, 14 November 13

MySQL  Replication

16

DatabaseDatabase

Database DatabaseDatabase

Database

Node  Manager

HA  Manager

Control Network Compute

Thursday, 14 November 13

DRBD

17

Control Network Compute

Database Database Database Database

Thursday, 14 November 13

Galera  Cluster

18

Control Network Compute

Database Database DatabaseDatabase Database Database

Thursday, 14 November 13

MySQL  as  a  ServiceThursday, 14 November 13

Automatic  Provisioning

•REST-­‐based  for  interoperation

•GUI-­‐assisted  for  end-­‐users

•From  server  to  cluster

20Thursday, 14 November 13

The  false  promise  of  the  elasticity•Distributed  MySQL  is  not  something  you  can  have  out  of  the  box•Or  not  for  all  your  applications

21

Read-Write

Read-Only Read-Only

binlog

relaylog relaylog relaylog relaylog

DatabaseDatabase

Database Database Database

LU

LU LU LU LU

SHARD0-99

SHARD 100-199

SHARD 200-299

SHARD 300-399

1 2 3 4M

Thursday, 14 November 13

DBaaS  vs  PaaS

22

Database Database

vs.

Thursday, 14 November 13

Storage

•Backups  on  Swift•Full  or  incremental

•Moving  tablespaces  in  Cinder•For  high  availability•For  backups  and  archiving•For  maintenance

23

in01

DatabaseDatabase

in02

Database

in03

Database

in04

Database Database

in05

Database

Thursday, 14 November 13

Networking

•Multiple  NICs  and  floating  IPs

•Public  IPs

•Local  proxies  vs  enhanced  connectors

24

Database Database Database Database Database Database

Thursday, 14 November 13

Security

•LDAP  &  Keystone

•Integration  with  MySQL•LDAP  /  OpenLDAP•MySQL  Security  Plugin  using  OpenLDAP

25Database

Keystone

LDAP

Thursday, 14 November 13

MariaDB  Cluster  in  OpenStackThursday, 14 November 13

What’s  new  with  MariaDB

27

MariaDB Manager MariaDB Cluster

MariaDBServer(s)

MHA Galera

GUI

API

Monitor

MaxScale

Configuration& Provisioning

Thursday, 14 November 13

SkySQLMaxScale

28

MaxScaleCore

Data Stores

Client Applications

Backend Backend Backend

Frontend Frontend Frontend Frontend Frontend

Clie

nt

Clie

nt

Clie

nt

Clie

nt

Clie

nt

Router

Router

Router

Monitor

Monitor

Auth

Auth

Auth

Service Service Service

Listener Listener Listener Listener

Server Server Server ServerServer

Thursday, 14 November 13

IntroducingMariaDBManager

29

RESTful(APICore

Monitor/ManagementInterface

Monitor/ManagementInterface

Monitor/ManagementInterface

Monitor(DataProvider

Agent Agent Agent

MariaDB(Galera(Cluster

Thursday, 14 November 13

MariaDB  Galera  Cluster  -­‐  State

30Thursday, 14 November 13

Provision  a  new  node

31

HTTP  POST  method  -­‐  Create  a  nodeURI:

• .../restfulapi/system/systemid/node• Parameters:• name• hostname• publicip,  privateip,  port• instanceid• dbusername,  dbpassword• ...

1

Thursday, 14 November 13

Provision  a  new  node

32

HTTP  POST  method  -­‐  Create  a  nodeURI:

• .../restfulapi/system/systemid/node• Parameters:• name• hostname• publicip,  privateip,  port• instanceid• dbusername,  dbpassword• ...

1

HTTP  POST  method  -­‐  Run  the  commandConnect  to  the  new  nodeURI:

• .../restfulapi/command/connect• Parameters:• systemid• nodeid• rootpassword

2

Thursday, 14 November 13

Provision  a  new  node

33

HTTP  POST  method  -­‐  Create  a  nodeURI:

• .../restfulapi/system/systemid/node• Parameters:• name• hostname• publicip,  privateip,  port• instanceid• dbusername,  dbpassword• ...

1

HTTP  POST  method  -­‐  Run  the  commandConnect  to  the  new  nodeURI:

• .../restfulapi/command/connect• Parameters:• systemid• nodeid• rootpassword

2

HTTP  POST  method  -­‐  Run  the  commandProbe  the  state  of  the  nodeURI:

• .../restfulapi/command/probe• Parameters:• systemid• nodeid

3

Thursday, 14 November 13

Provision  a  new  node

34

HTTP  POST  method  -­‐  Create  a  nodeURI:

• .../restfulapi/system/systemid/node• Parameters:• name• hostname• publicip,  privateip,  port• instanceid• dbusername,  dbpassword• ...

1

HTTP  POST  method  -­‐  Run  the  commandConnect  to  the  new  nodeURI:

• .../restfulapi/command/connect• Parameters:• systemid• nodeid• rootpassword

2

HTTP  POST  method  -­‐  Run  the  commandProbe  the  state  of  the  nodeURI:

• .../restfulapi/command/probe• Parameters:• systemid• nodeid

3

HTTP  POST  method  -­‐  Run  the  commandProvision  the  nodeURI:

• .../restfulapi/command/provision• Parameters:• systemid• nodeid

4

Thursday, 14 November 13

Start  a  node

35

HTTP  POST  method  -­‐  Run  the  commandStart  the  nodeURI:

• .../restfulapi/command/start• Parameters:• systemid• nodeid

Thursday, 14 November 13

Retrieve  the  status  of  a  cluster

36

• .../restfulapi/system/systemid

Thursday, 14 November 13

Monitor  the  #  of  connections  of  a  node

37

• ../res^ulapi/system/systemid/node/nodeid/monitor/{monitorid}/data

Thursday, 14 November 13

What’s  coming•MariaDB  Replication  and  MHA

•MariaDB  10

•MaxScale  integration

•Juju  charms  for  MariaDB  Cluster  and  MariaDB  Manager

•Puppet  &  Chef  manifests  and  recipes•MariaDB  and  Trove

•Trove  for  MariaDB  Manager38

Thursday, 14 November 13

For  More  Information...

39

• MariaDB  Enterprise:www.mariadb.orgwww.mariadb.com

• MariaDB  Manager  and  MaxScale:http://downloads.skysql.com/archives

• Bugs:  http://bugs.skysql.com

Thursday, 14 November 13

Thank  You!

ivan@skysql.comizoratti.blogspot.com

www.slideshare.net/izorattiwww.skysql.com

Pictures  courtesy  of:•www.pluralsight.com

Thursday, 14 November 13