oracle mysql tutorial -- mysql nosql cloud buenos aires nov, 13 2014

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |Oracle Confidential – Internal/Restricted/Highly Restricted

ORACLE MySQL Enterprise Tutorial

Manuel Contreras Rico [email protected]

Principal SW Consultant

ORACLE MySQL Noviembre, 2014

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Mayor inversión e innovación al ecosistema

Jan 2010 Sept 2014

2

2x Engineering Staff 3x QA Staff 2x Support Staff

5 años de innovación Oracle

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

MySQL ~ Big Data

3


Oracle Confidential – Internal/Restricted/Highly Restricted

Arquitectura MySQL

4

MySQL Server



MySQL Storage Engines Motores de Almacenamiento

5

MySQL Server


MySQL Storage Engines• Pluggable architecture

6

InnoDB MyISAM NDBClusterACID Propósito General

Not ACID Bulk operations / Catálogo consultas

ACID in-Memory DB real time OLTP

ArchiveData Compression

MemoryVolatile Data High Performance

3rd PartyCommunity

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 7

MySQL Replication sharding & high availability

App

Writes & Reads

MySQL Master Server

Writes

Index & Binary Log Rotation

MySQL Slave Server

Writes

Reads

ReplicationRelay Log


Oracle database & MySQL

** Oracle eBusiness Suite CRM / ERP * Custom Ad-Hoc ERP / CRM


Herramientas de administración

Caracterísiticas Avanzadas Soporte Técnico

• Scalability • High Availability • Security • Audit

• Monitoring • Backup • Development • Administration • Migration

• Technical Support

• Consultative Support

• Oracle Certifications

MySQL Enterprise Edition

12


Mejor escalabilidad en MySQL Enterprise MySQL 5.5.16 Oracle Linux 6.1, Unbreakable Kernel 2.6.322 sockets, 24 cores, 2 X 12-coreIntel(R) Xeon(R) X5670 2.93GHz CPUs72GB DDR3 RAM2 X LSI SCSI Disk (MR9261-8i) (597GB)

mysql.com/products/enterprise/scalability.html

MySQL Scalability Thread Pool ( Performance )

http://www.mysql.com/products/enterprise/scalability.html


• Visión consolidada en todo el entorno de MySQL, en las instalaciones y en la nube

• Automatizado, monitoring emite alertas basado en normas (SMTP, SNMP)

• Query Analyzer, seguimiento, análisis y puesta a punto, en correlación con los gráficos del monitor

• Identifica los servidores problemáticos • Monitor de replicación en tiempo real con

auto-descubrimiento de topologías maestro-esclavo

MySQL Enterprise Monitor 3.0 Ahorra tiempo. Soluciona

problemas de manera proactiva.

MySQL Enterprise Edition : Monitor


MySQL Enterprise Backup

• Online, non-locking backup & recovery – Complete MySQL instance backup (datos &

Config ) – Partial backup & restore

• Direct Cloud storage backups (S3, etc.) • Incremental backups • Point-in-time recovery • Compressed & encryption • Backup to tape (SBT) • Backup validation • Online backup MySQL (Windows, Linux,

Unix)

21


MySQL Workbench EE + Backup

• Programar & administrar backup jobs – Create new backup jobs – Schedule backup jobs para

ejecución automática – Visualiza backup jobs actuales – Visualiza actividad backup

• Restore backups – Full, partial, incremental …

13


MySQL Enterprise Monitor + Backup

14

• Monitor backup performance • Asegurar que los backups

programados han sido exitosos y están al día.


▪ Diagnostica problemas de desempeño código SQL problemático.

▪ Correlación gráfica de código SQL

▪ Encuentra y optimiza cuellos de botella

▪ Identifica la fuente de problemas en desempeño

▪ Estadísticas detallas de ejecución de queries SQL

▪ Query Response Time index (QRTi)

Con el analizador de consultas MySQL, hemos sido capaces de identificar y analizar el código SQL problemático y triplicar nuestro rendimiento de base de datos. Más importante aún, hemos sido capaces de lograr esto en tres días, en lugar de tomar semanas

Keith Souhrada Ingeniero de Desarrollo Software Big Fish Games

MySQL Query Analyzer

mysql.com/products/enterprise/query.html

http://www.mysql.com/products/enterprise/query.html


MySQL Enterprise Security

• SSL enabled • Access control

– Enterprise authentication (PAM, Windows, LDAP, etc.) – Proxy users

• Auditing & monitoring – MySQL security advisors – Oracle Audit Vault

• Oracle Database Firewall

16


MySQL Enterprise Encryption

• Librerías MySQL encryption – Symmetric encryption AES256 – Public-key / asymmetric cryptography

• Administración de Keys PKI – Generate public and private keys – Key exchange methods: RSA, DSA, DH

• Sign & verify data – Cryptographic hashing for digital signing, verification, & validation

17


MySQL Cluster CGE

18

In-Memory Database

SQL & NoSQL Flexibility


MySQL: Sakila & El Chavo del 8

19

playfulplay.com/lavecindaddeelchavo/index.php

Caso de estudio: MySQL Cluster CGE habilita un Sharding transparente para la aplicación ( NDBCluster )

Facebook Game #1 Categoría Familiar - Cloud Elástico bajo demanda - In-memory database - Arquitectura Non-Shared Disk

>10 MILLONES de suscriptores >80 MIL jugadores concurrentes


La vecindad de El Chavo del 8

20

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |21

Hosting IaaS, PaaS

SaaS

MySQL en la Nube The #1 RDBMS in the Cloud


MySQL Enterprise Replication Monitor

22

• Auto-discover de topología de replicación

• Monitor de replicación Master/Slave ( Replication advisor )

• Mejores prácticas en configuración de Grupos de Replicación ( Async, Semi-sync )

"I use the MySQL Enterprise Monitor every day to monitor and keep tabs on our MySQL databases. Quick one stop shopping for keeping tabs on them.”

-Wes Homer, Sr System and Network Administrator

mysql.com/products/enterprise/monitor.html

http://www.mysql.com/products/enterprise/monitor.html


ORACLE — MySQL Integration

Oracle Enterprise Manager 12c Oracle Linux Oracle VM Oracle Clusterware Oracle Secure Backup Oracle Fusion Middleware Oracle Golden Gate Oracle Audit Vault Oracle Database Firewall MyOracle Support

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |24

PerformanceSecurity

Availability

• Availability monitoring • Performance monitoring • Configuration monitoring • Permite crear custom

threshold basados en reportes de incidencia

• MySQL auto-detection

ORACLE Enterprise Manager 12c GA


Soporte MySQL Enterprise Technical support & Consultative Support

• El equipo más grande de ingenieros MySQL • Apoyado por MySQL developers • World-class support, en 29 languages ( Español ) • Hot fixes & maintenance releases • 24x7 x 365 • Incidentes ilimitados • Soporte Consultivo • Global scale and reach

Ayuda inmediata para cualquier problema de MySQL, además de

asesoría de expertos

41


Información Adicional

26

MySQL Products http://www.mysql.com/products/

MySQL Enterprise http://www.mysql.com/products/enterprise/

MySQL FAQ http://dev.mysql.com/doc/refman/5.6/en/faqs-general.html

http://www.mysql.com/products/

http://www.mysql.com/products/enterprise/

http://dev.mysql.com/doc/refman/5.6/en/faqs-general.html


MySQL Enterprise Server Arq. Referencia

27


Drools (Business Rule Management System)

Arquitectura MySQL Enterprise Edition

Red para sincronización de datos entre nodos MySQL

DAS LUN (1)

MySQL Enterprise Primario

VLAN2 ( Privada )

MySQL Enterprise Secundario

DAS LUN (2)

Load balancer: Fabric-Aware Connectors: Java, PHP, .NET, ( C Labs ); otros Load Balancers

Custom OLTP App

Red de servicio Application Server & MySQL

VLAN1

MySQL Fabric Controller MySQL Enterprise Monitor

OLAP BI Reporting Analítica


PRODUCCION — MySQL Enterprise (2 Servidores virtuales / físicos)CPU: 16 Cores CPU x86 64 bits; RAM: 16G ( o según tamaño de Base de datos / volumen de transacciones )

Storage: 1 Logic Unit asociada a un punto de montaje para datos $DATADIR, para cada unos de los nodos ( Primario & Secundario), como disco no compartido que permita un alto performance de lectura / escritura ( p.e. RAID 10 )OS: Linux varias distribuciones, Solaris, Mac OSX, MS Windows 2012 ServerRed: 2 puertos de Gigabit ethernet configurados con VLAN1 red de servicio & VLAN2 red privada de sincronización de datos entre nodos

MySQL Fabric Controller / MySQL Enterprise Monitor ( 1 servidor virtual / físico )CPU: 8 Cores CPU x86 64 bits; RAM: 8G; HDD Disco Interno

PRE-PRODUCCION ( QA ) — MySQL Enterprise 1 Servidores virtual / físico con:CPU: 16 Cores CPU x86 64 bits; RAM: 16G ( o según tamaño de Base de datos / volumen de transacciones )

Storage: 1 Logic Unit asociada a un punto de montaje para datos $DATADIR de MySQL Enterprise Monitor & Fabric ControllerOS: Linux varias distribuciones, Solaris, Mac OSX, MS Windows 2012 ServerRed: VLAN1: red de servicio

Total 4 Servidores MySQL Enterprise Edition ( ambiente QA & PROD )

Arquitectura MySQL Enterprise Edition — Detalle Técnico


MySQL & High Availability



Oracle MySQL HA & Scaling Solutions

MySQL Replication

MySQL Fabric

Oracle VM Template

Oracle Clusterware

Solaris Cluster

Windows Cluster DRBD MySQL

Cluster

App Auto-Failover ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Data Layer Auto-Failover ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Zero Data Loss MySQL 5.7 MySQL 5.7 ✔ ✔ ✔ ✔ ✔ ✔

Platform Support All All Linux Linux Solaris Windows Linux All

Clustering Mode Master + Slaves

Master + Slaves

Active/Passive

Active/Passive

Active/Passive

Active/Passive

Active/Passive

Multi-Master

Failover Time N/A Secs Secs + Secs + Secs + Secs + Secs + < 1 Sec

Scale-out Reads ✔ ✖ ✖ ✖ ✖ ✖ ✔

Cross-shard operations N/A ✖ N/A N/A N/A N/A N/A ✔

Transparent routing ✖ For HA ✔ ✔ ✔ ✔ ✔ ✔

Shared Nothing ✔ ✔ ✖ ✖ ✖ ✖ ✔ ✔

Storage Engine InnoDB+ InnoDB+ InnoDB+ InnoDB+ InnoDB+ InnoDB+ InnoDB+ NDB

Single Vendor Support ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✔

31


MySQL Enterprise HA : Shared Storage

32

• Stricter data durability, integrity constraints – Shared storage persists commits across

instances – Clustering software manages data access – Auto-failover of applications and database – Deploy with MySQL Fabric for scale-out

• MySQL certified & supported solutions – Oracle Clusterware – Windows Failover Clustering – Oracle Solaris Cluster

Virt

ual I

P

Clie

nts


MySQL Enterprise HA : Shared Nothing

33

• DRBD + Clustering – Based on distributed storage, not a SAN – Synchronous replication eliminates risk

of data loss

• Open source, mature, & proven • Certified and fully supported by

Oracle – DRBD integrated into Oracle Linux

Unbreakable Enterprise Kernel R2 – Pacemaker and Corosync for clustering /

failover – Updates to stack via ULN channel


MySQL Enterprise HA : Oracle VM Templates

34

• Oracle Linux • Oracle VM • Oracle VM Manager • Oracle Cluster File System 2 (OCFS2) • MySQL Database (Enterprise Edition)

• Pre-installed & pre-configured • Full integration & QA testing • Single point of support

Oracle VM Servers

Oracle VM Server Pool

ocfs2

Oracle VM Manager

SAN / iSCSI

Oracle VM

Secure Live Migration (SSL)

Oracle VMAutomatic Fault

Detection & Recovery


MySQL Cluster CGE: NDBCluster

35

• Automatic Sharding • In-memory engine • Sync replication • Real-time OLTP • 99.999 % • Requires MySQL Cluster CGE

binaries / suscription • NoSQL / SQL supported • Minimum 3 physical servers, 4

production environments

MySQL Cluster Data Nodes

ClientsApplication Layer

Management

Data Layer


MySQL Fabric 1.5.2

36


MySQL FabricMySQL Fabric es un framework extensible & fácil de utilizar, para administrar ambientes de clustering con MySQL. El framework implementa 3 funcionalidades: Tolerancia a Fallas, Distribución de carga Sharding, & aprovisionamiento de entornos de manera automática.

MySQL Fabric, gestiona también grupos de servidores con MySQL Replication y permite un failover automático gestionado por el nodo MySQL Fabric Controller en combinación a los Drivers Fabric-Aware: Java, PHP, Python & .NET y próximamente Lenguaje C.

El Fabric connector, puede en rutar transacciones en base a un criterio definido por el desarrollador, permitiendo también escalar de manera horizontal en activo - activo en múltiples grupos de data shards.

Por otro lado, la extensión Sharding de MySQL Fabric, permite que el desarrollador pueda distribuir bases de datos, sobre un conjunto de servidores físicos gestionados por MySQL Fabric, esto con el fin de poder escalar elásticamente agregando o reduciendo el número de nodos en el grupo de servidores.

37

http://www.mysql.com/products/enterprise/fabric.html



MySQL Fabric 1.5

• High Availability • Auto-promotion failover & transparente

para la application failover • Opcionalmente scale-out a través de

data sharding – Application provee shard key – Range ó Hash

• Conectores Fabric-aware: Python, Java, PHP, .NET, C (labs) – Lower latency, bottleneck-free

• Server provisioning a través de OpenStack

High Availability + Sharding-Based Scale-out

MySQL Fabric

Connector • Application

Read-slaves

mappings

SQL

HA group

Read-slaves

HA group

Connector • Application

38


MySQL Fabric Framework (HA)

All Data

Primary Secondary

Extra Read Replicas

MySQL Fabric Controller

SQL Queries

State & Routing Info

HA Group

Coordination and Control

39


MySQL Replication & MySQL Fabric HA

• MySQL Replication is the initial implementation used in HA Groups – PRIMARY = Replication Master & receives all writes – SECONDARY = Replication Slave & receives share of reads

• Failover – MySQL Fabric detects failure of PRIMARY/Master – Selects a SECONDARY/Slave and promotes it – Updates State Store – Updated state fetched by Fabric-aware connectors

How this effects failover

40


High-Availability Group Concept• Abstract Concept

– Set of servers – Server attributes

• Connector Attributes – Connection information – Mode: read-only, read-write, ... – Weight: distribute load

• Management Attributes – State: state/role of the server

State: PrimaryMode: Read-WriteHost: server-1.example.com

41


Create HA Groups and add Servers• Definiendo el grupo

mysqlfabric group create my_group

• Agregando servidores al grupo

mysqlfabric group add my_group server1.example.com

mysqlfabric group add my_group server2.example.com

42

http://server2.example.com


Create HA Groups and add Servers• Promoviendo un servidor a Primario Master

mysqlfabric group promote my_group

• Activación del FailOver detector

mysqlfabric group activate my_group

43


Demostración MySQL Fabric

44


MySQL Fabric ( Sharding )

45


• Write scalability – Can handle more writes

• Large data set – Database too large – Does not fit on single server

• Improved performance – Smaller index size – Smaller working set – Improve performance (reads and

writes)

Benefits of ShardingUID 10000-20000 UID 20001-40000

46


• Connector API Extensions – Support Transactions – Support full SQL

• Decision logic in connector – Reducing network load

• Shard Multiple Tables – Using same key

• Global Updates – Global tables – Schema updates

• Sharding Functions – Range ( Integer, String, datetime ) – Hash (Consistent)

• Shard Operations – Shard move – Shard split

• Server Provisioning – Integrated with OpenStack & other

frameworks

MySQL Fabric Sharding & Provisioning Features

47


Sharding Architecture

Shards

MySQL Fabric Node

Application

Global Group

Global Updates

ShardUpdates

Replication

48

Connector

Connector

Connector


MySQL Fabric (HA + Sharding)

Global Data

Primary Secondary

Shard 1

Primary Secondary

Shard 2

Primary Secondary

MySQL Fabric Controller

SQL Queries

Server/Shard State & Mapping

Global Group HA Group

Coordination and Control

HA Group

Extra Read Replicas Extra Read Replicas

49


Routing Transactions

Shard #2

Shard #1

Shard #3

Executor

App ServerConnector

Cache

State Store

App ServerConnector

Cache

50


MySQL Fabric: Sharding Setup• Set up some groups

– my_global – for global updates – my_group.N – for the shards – Add servers to the groups

• Create a shard mapping – A “distributed database” – Mapping keys to shards – Give information on what tables are sharded

• Add shards

51


MySQL Fabric:

• Moving a shard (id=5) from existing group to another (my_group.8) mysqlfabric sharding move 5 my_group.8

• Splitting a shard (id=5) into two parts with new half stored in group my_group.6 mysqlfabric sharding split 5 my_group.6

Moving and Splitting Shards

52


• Indicate tables to be used in query – Property: tables – Fabric will compute map

• Indicate read-only queries – Property: mode

• Provide sharding key – Property: key – Fabric will compute shard

• Joins within the shard (or with global tables) supported

Connector API: Shard Specific Query

conn.set_property(tables=["test.subscribers"], key=sub_no, mode=fabric.MODE_READONLY) cur = conn.cursor() cur.execute( "SELECT first_name, last_name FROM subscribers WHERE sub_no = %s", (sub_no) ) for row in cur: print row

53


• Set global scope – Property: scope – Query goes to global group

Connector API: Global Update

conn.set_property(tables=[], scope='GLOBAL') cur = conn.cursor() cur.execute("ALTER TABLE test.subscribers ADD nickname VARCHAR(64)")

54


> mysqlfabric provider register my_stack \ my_user my_password \ http://8.21.28.222:5000/v2.0/ \ --tenant=my_user_role \ --provider_type=OPENSTACK

> mysqlfabric machine create my_stack \ --image id=8c92f0d9-79f1-4d95-b398-86bda7342a2d \ --flavor name=m1.small

> mysqlfabric machine list my_stack

• Fabric creates new machines, & MySQL Servers – Initially using OpenStack Nova – Other frameworks on the way

(OpenStack Trove, AWS,…)

• Server setup – Clones slave – Sets up replication – Performs custom operations

55

Server Provisioning – OpenStack Nova Integration


• Event driven – Events will trigger execution of

procedures – Procedures can trigger events themselves – Each step of a procedure is called a job

• Procedures – Written in Python – Interacts with servers – Write state changes into backing store – Lock manager for conflict resolution

• Conservative two-phase locking strategy • Avoid deadlocks

MySQL Fabric executor

Backing Store

Queue

Events


• Register procedure for event – @on_event decorator – Accept event to register for

• Fetch the group the server belonged to

• Fetch a new server from the provider

• Add the server to the group

@on_event(SERVER_LOST) def _add_server(group_id, server_uuid): group = Group.fetch(group_id) machines = PROVIDER.create_machines( parameters ) server = MySQLServer( server_uuid, address ) MySQLServer.add(server) group.add(server) _configure_as_slave(server)

Example of User-Defined Executor Script: Automatically replacing a server in a group on failure


Extensible ArchitectureMySQL Fabric Node

Connector

Connector

Connector

MySQL FabricFramework

AMQP? MySQL XML-RPC

Executor State Store (persister)

Shard

HA

Prov

???

MySQL Backing Store

SQL

58


• Connector API Extensions – Support Transactions – Support full SQL

• Fabric-Aware Connectors at GA: – PHP + Doctrine, Python, Java + Hibernate,

.NET, C (pre-GA)

• Decision logic in connector – Reducing latency &network load

• Load Balancing – Read-Write Split – Distribute transactions

• Global Updates – Global data – Schema updates

• Sharding Functions – Range – (Consistent) Hash

• Shard Operations – Shard move – Shard split

• Server Provisioning – OpenStack Integration (& other

frameworks)

MySQL Fabric: Goals & Features

59


• Routing is dependent on Fabric-aware connectors – Currently Java (+ Hibernate), PHP (+

Doctrine), Python, .NET & C (labs)

• MySQL Fabric node is a single (non-redundant process) – HA Maintained as connectors continue to

route using local caches

• Establishes asynchronous replication – Manual steps to switch to semisynchronous

• Sharding not completely transparent to application (must provide shard key – column from application schema)

• No cross-shard joins or other queries • Management is through CLI, MySQL

protocol or XML/RPC API – No GUI

MySQL Fabric – Current Limitations

60


Oracle MySQL HA & Scaling SolutionsMySQL

Replication

MySQL Fabric

Oracle VM Template

Oracle Clusterwar

e

Solaris Cluster

Windows Cluster DRBD MySQL

Cluster

App Auto-Failover ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Data Layer Auto-Failover

✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Zero Data Loss MySQL 5.7 MySQL 5.7 ✔ ✔ ✔ ✔ ✔ ✔

Platform Support All All Linux Linux Solaris Windows Linux All

Clustering Mode Master + Slaves

Master + Slaves

Active/Passive

Active/Passive

Active/Passive

Active/Passive

Active/Passive

Multi-Master

Failover Time N/A Secs Secs + Secs + Secs + Secs + Secs + < 1 Sec

Scale-out Reads ✔ ✖ ✖ ✖ ✖ ✖ ✔

Cross-shard operations N/A ✖ N/A N/A N/A N/A N/A ✔

Transparent routing ✖ For HA ✔ ✔ ✔ ✔ ✔ ✔

Shared Nothing ✔ ✔ ✖ ✖ ✖ ✖ ✔ ✔

Storage Engine InnoDB+ InnoDB+ InnoDB+ InnoDB+ InnoDB+ InnoDB+ InnoDB+ NDB

Single Vendor Support ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✔

61


MySQL Fabric Resources• Download and try

http://dev.mysql.com/downloads/fabric/

• Documentation http://dev.mysql.com/doc/mysql-utilities/en/fabric.html

• MySQL Fabric on the webhttp://www.mysql.com/products/enterprise/fabric.html

• Forum (MySQL Fabric, Sharding, HA, Utilities)http://forums.mysql.com/list.php?144

• Tutorial: MySQL Fabric - adding High Availability and Scaling to MySQLhttp://www.clusterdb.com/mysql-fabric/mysql-fabric-adding-high-availability-and-scaling-to-mysql

• White Paper: MySQL Fabric - A Guide to Managing MySQL High Availability and Scaling Outhttp://www.mysql.com/why-mysql/white-papers/mysql-fabric-product-guide

• Webinar Replays http://www.mysql.com/news-and-events/on-demand-webinars/#en-20-41

62



http://dev.mysql.com/doc/mysql-utilities/en/fabric.html

http://dev.mysql.com/doc/mysql-utilities/en/fabric.html



http://forums.mysql.com/list.php?144

http://forums.mysql.com/list.php?144

http://www.clusterdb.com/mysql-fabric/mysql-fabric-adding-high-availability-and-scaling-to-mysql

http://www.clusterdb.com/mysql-fabric/mysql-fabric-adding-high-availability-and-scaling-to-mysql

http://www.mysql.com/why-mysql/white-papers/mysql-fabric-product-guide

http://www.mysql.com/why-mysql/white-papers/mysql-fabric-product-guide

http://www.mysql.com/news-and-events/on-demand-webinars/%23en-20-41

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |Oracle Confidential – Internal/Restricted/Highly Restricted

ORACLE MySQL Enterprise

Manuel Contreras Rico [email protected]

Principal Software Consultant Latinoamérica & El Caribe

ORACLE MySQL Noviembre, 2014

oracle mysql tutorial -- mysql nosql cloud buenos aires nov, 13 2014

Software

oracle andor

oracle database mysql

oracle linux

mysql enterprise backup

oracle ebusiness suite

rights reservoerda

mysql enterprise edition12

mysql workbench ee backup