high-availability using mysql fabric

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

High Availability using MySQL Fabric:Managing Farms of Servers

Mats Kindahl

Copyright © 2015, Oracle and/or its affiliates. All rights reserved.

Copyright © 2015, Oracle and/or its affiliates. All rights reserved.2

The following is intended to outline our general product direction. It is intended

for information purposes only, and may not be incorporated into any contract.

It is not a commitment to deliver any material, code, or functionality, and

should not be relied upon in making purchasing decision. The development,

release, and timing of any features or functionality described for Oracle’s

products remains at the sole discretion of Oracle.

Safe Harbor Statement


Program Agenda

Building reliable systems MySQL Fabric overview Managing redundancy Procedure automation and the Executor Failure detection and failure handling


Program Agenda

Using Fabric with existing high-availability setups Making Fabric highly available Thoughts for the future Closing remarks


Insert Picture Here

Building Reliable SystemsInsert Picture Here

Copyright © 20135 Oracle and/or its affiliates. All rights reserved.6

Insert Picture Here

High-availability is an integral part of designing a reliable system

Building for reliability


What causes downtime?

● System failures● Hardware faults

● Software bugs

● Disasters

● Maintenance

● User errors


High-availability concepts

● Redundancy● Duplicate critical components

● Monitoring● Detecting failing components● Monitor load

● Procedures● Activate replacements● Distribute load


High-availability solutions

● Primary-seconday approach

● MySQL Replication

● Shared-nothing clusters

● MySQL Cluster

● MySQL Group Replication (not GA)

● Tightly coupled clusters

● DRBD

● WSFC

● Solaris Clustering

● Oracle Clusterware

● Oracle VM High Availability


Insert Picture Here

MySQL Fabric Overview


What is MySQL Fabric?

An extensible and easy-to-use framework for managing a farm of MySQL servers supporting high-availability and sharding.


What does it mean?

● Management system● Manages a MySQL Farm● Distributed framework

● Framework● Procedure execution● State store● Transaction Routing

● Extensible● High-availability groups● Sharding● Cloud support

● Written in Python

● MySQL 5.6 (or later)

● Open Source● You can participate


Birds-eye view

MySQL Fabric Node

Application

Operator

High-Availability Groups (Shards)


MySQL Fabric Components

● Fabric-aware connectors● Enhanced Connector API● Python, PHP, Java, .NET, C

● MySQL Fabric controller● Manage farm meta-data● Provide status information● Execute Procedures

● MySQL servers● Organized in high-availability groups● Handle application data

MySQL Fabriccontroller node

High-availabilitygroup

Application withFabric-aware connectors


MySQL Fabric Controller Architecture

XML-RPC

MySQL-RPC

AMQP

Protocol Server

XML-RPC

MySQL-RPC

AMQP

Protocol Server

Sharding

Master-Slave

Providers

Extensions

StateStore

XML-RPC

MySQL-RPC

AMQP

Protocol Server

Executor

Model

Persistance

Fabric Core

Requests

Events

Results

Eample only!

Eample only!


Insert Picture Here

Managing Redudancy


High-Availability Group Concept

● Group of servers● Hardware redundancy● Data redundancy

● Generic Concept● Implementation-independent● Self-managed or externally managed

● Different Types● Primary-Backup (Master-Slave)● Shared or Replicated Storage● MySQL Cluster

DRBD

ndbdndbd

ndbd ndbdDefault

Eamples Only

Not Implemented


● Create a logical group for the servers

● Empty initially

mysqlfabric group create my_group --description='My Group'

Creating a high-availability groupCreate an empty group


● Add servers to group

● Group will have no master

● All servers are secondaries (!)

Creating a high-availability groupAdding servers to the group

mysqlfabric group add my_group server1.example.com




● Promote one secondary to primary

● Selects secondary at random

● Specific secondary can be selected

Creating a high-availability groupPromote a primary

mysqlfabric group promote my_group

mysqlfabric group promote my_group --slave_id='server1.example.com'


● Enable built-in failure detector

● Monitor servers in group

Creating a high-availability groupEnable failure detector

mysqlfabric group activate my_group


● On primary failure● Mark primary as faulty● Trigger fail-over

● On secondary failure● Mark secondary as faulty

Creating a high-availability group


Insert Picture Here

Procedure Automation and the Executor


● Management Procedures● Fail-over● Slave promotion● Shard split

● Triggered on events● Crashing server● Administrative decision● Increasing load

● Resilient execution● Controller node can crash● Recover partially executed procedure

Automating management of a farm

Find Candidate

Check Candidate

Disable Read-only

Process Backlog

Re-direct Slaves

SLAVE_PROMOTED

SERVER_LOST


MySQL Fabric executor

● Event driven executor● Events will trigger execution of procedures● Procedures can trigger events themselves● Each step of a procedure is called a job

● Procedures● Written in Python● Interacts with servers● Write state changes into backing store● Lock manager for conflict resolution

– Conservative 2PL– Avoid deadlocks

Queue

BackingStore

Events


Example: keep high-availability profile


● Register job for event● @on_event decorator● Register job with event

● Fetch group of lost server

● Fetch new server from provider

● Add server to group

@on_event(SERVER_LOST)def _add_server(group_id, server_uuid): group = Group.fetch(group_id) machines = PROV.create_machines( parameters ) server = MySQLServer( server_uuid, address ) MySQLServer.add(server) group.add(server) _configure_as_slave(server)

Automating adding a server


● Before starting a job:● Aquire the necessary locks● Checkpoint execution state in backing store● Start a transaction on the backing store

● When executing job:● Updates to backing store inside transaction● Interact with servers

● After executing a job:● Mark job completed in internal log● Commit transaction on backing store

What about crashes?

Queue

BackingStore

Events

MySQL Fabric execution flow


● Two types of jobs:● Idempotent: Restart the job● Not idempotent: Execute compensation

● Recovery procedure● Start the executor● Collect unfinished checkpoints● Execute compensation actions

… if there are any

● Re-schedule each job in checkpoint

Queue

BackingStore

Events

MySQL Fabric executor recovery


Insert Picture Here

Failure detection and failure handling


● Group level detection● Fabric node ping servers in group● Servers need to be managed by Fabric

● On primary failure● Mark primary as faulty● Trigger fail-over of connectors and slaves

● On secondary failure● Mark secondary as faulty

Built-in failure detector


Built-in failure detectorConfiguration

● Detections● Number of failed pings before

marked as faulty

● Detection Interval● Interval between server ping,

in seconds

● Detection Timeout● Timeout for ping, in seconds

[failure_tracking]detections = 3detection_interval = 6detection_timeout = 1


● External failure detectors● Connectors● Custom failure detectors

● Reporting API● Error: suspected server failure● Failure: server is known to have failed

● Reporting server error● Trigger fail-over if threshold is exceeded

● Reporting server failure● Trigger immediate fail-over

External failure detectors

?

!

MySQL Fabriccontroller node


External failure detectorsConfiguration

● Notifications● Error threshold

● Notification clients● Threshold for number of

unique clients

● Notification interval● Notification window

[failure_tracking]notifications = 300notification_clients = 50notification_interval = 60failover_interval = 0prune_time = 3600


External failure detectorsConfiguration

● Failover interval● Minimum interval between

failovers● Used to prevent flapping

● Prune time● Size of error log (in seconds)

to keep

[failure_tracking]notifications = 300notification_clients = 50notification_interval = 60failover_interval = 0prune_time = 3600


Connector as external failure detector

● Error reporting from connector● Depends on connector support● Report suspected failures

● Enabling error reporting● Error reporting off by default

● Avoid a thundering herd● Do not enable error reporting

for all connectors!● Failing server will cause all

connectors to report failure

fabric_config = { … 'report_errors': True, …}

cnx = connect( … fabric=fabric_config …)


Connector as external failure detectorError reporting

● Default errors reported

● Extra errors can be added● extra_failure_report

CR_SERVER_LOSTCR_SERVER_GONE_ERRORCR_CONN_HOST_ERRORCR_CONNECTION_ERRORCR_IPSOCK_ERROR

from mysql.connector.fabric import extra_failure_report

extra_failure_report([error1, error2, …, errorn])


Connector as external failure detectorCache invalidation

● Cache invalidation by default on● Server Lost (CR_SERVER_LOST)● Server read-only (ER_OPTION_PREVENTS_STATEMENT)

from mysql.connector.fabric import RESET_CACHE_ON_ERROR

RESET_CACHE_ON_ERROR.append(error)


Insert Picture Here

Using Fabric with Existing High-availability Setups


Using Fabric with Existing Solution

● Servers already managed● Group Based Solutions● Virtual IP-based solutions

● Fabric as lookup server● Connectors can route transactions● Application can retrieve information from Fabric

● Update state store only


Example: An existing setup

● DRBD for redundancy● Disk replicated

● Pacemaker for fail-over● Heartbeat detect failure● Resource agent handle

fail-over

● Fabric as lookup server

● Fabric for routing transactions

Secondary Node

Primary Node

DRBD Replication

Pacemaker Pacemaker


Example: An existing setupCreate a group

● Create a group

● Add server to group● Fabric should only update state store

● “Promote” the DRBD primary to be primary in group

mysqlfabric group create my_group

mysqlfabric group add my_group server1.example.com --update_only

mysqlfabric group promote my_group --update_only --slave_id=...


Example: An existing setupUpdate resource agent

● Change resource agent script● On Ubuntu: /usr/lib/ocf/resource.d/heartbeat/mysql

● Update resource agent actions to inform Fabric● Remove old server● Only update the state store

mysqlfabric group demote --update_only --slave_id=7bcb0804-...

mysqlfabric group remove --update_only 7bcb0804-...


Example: An existing setupUpdate resource agent

● Change resource agent script● On Ubuntu: /usr/lib/ocf/resource.d/heartbeat/mysql

● Update resource agent actions to inform Fabric● Add standby server● Only update the state store

mysqlfabric group add --update_only standby.example.com

mysqlfabric group promote --update_only 8308b0c4-...


Insert Picture Here

Making Fabric highly available


Making Fabric highly available

● Standard deployment● Fabric node and state store on

same machine● Need to use TCP

– Socket connection not available yet (Bug#71946)

● Three things can fail:● State store● Fabric node● Machine


Making Fabric highly availableHandling state store failure

● If state store connection is lost:● Fabric retry until state store

becomes available● Ongoing transactions fail● Fabric report error if

connection not recovered “quickly enough”

● Solution: restart state store● MySQL handle recovery● Fabric re-connect

automatically


Making Fabric highly availableHandling state store failure

● Connection timeout● Timeout (in seconds) for

connection attempt

● Connection attempts● Number of attempts before

reporting state store failed

● Connection interval● Delay (in seconds) between

connection attempts

[storage]connection_timeout = 6connection_attempts = 6connection_interval = 1


Making Fabric highly availableHandling Fabric controller node failure

● If Fabric node is lost:● Ongoing jobs fail● Execution state checkpointed

● On Fabric node restart:● Execution state recovered

● Solution: restart Fabric node● Detect failure

– Local ping script● Restart Fabric node

– init.d script● Neither distributed with Fabric


Making Fabric Highly AvailableHandling machine failures

● If the machine fails:● State store is lost● Fabric node is lost● Catastrophic failures can

prevent machine recovery

● Solution:● Replicate meta-data● Detect machine failure● Activate duplicate deployment


Making Fabric Highly AvailableReplicate meta-data

● Replicate state store● DRBD● MySQL Cluster● MySQL Replication

● Configure DRBD● Version 8.3 or later● Replicate block device

● Configure MySQL Servers● Data directory on replicated

device


Making Fabric Highly AvailableReplicate meta-data

● Active node● MySQL Fabric● MySQL Server● DRBD primary

● Passive node● DRBD secondary● Server and Fabric started on

fail-over


Making Fabric Highly AvailableDetect machine failure & activate replacement

● Detecting machine failure● Corosync● Version 2.0 or later

● Activate Replacement● Pacemaker● Version 1.1 or later



● Configure MySQL Fabric● State store in DRBD volume

● Configure Corosync● Set no-quorum-policy to

'ignore'– Prevent remaining node to

shut down● Turn off STONITH

– Node will commit suicide



● Configure Pacemaker● Add MySQL Fabric resource

agent● Colocate Fabric, DRBD, and

MySQL and order them

● Avoiding split-brain● Reliably detect network

partition● Ping reliable resource

– Example: Router


Insert Picture Here

Closing Remarks &Ideas for the Future


Multi-Node FabricReplicated State Machine

● Multiple Fabric Nodes● Built-in support● Fail-over● Local read instance● Distributed execution

● Replicated State Machine● Coordinate procedure execution● Automatic fail-over● Paxos or Raft-like implementation


More Flexibility

● Server Providers● Amazon AWS● Kubernetes?

● Built-in high-availability group types● DRBD● MySQL Cluster● Amazon RDS?


MySQL Fabric ResourcesUseful links

● Download and try

● http://dev.mysql.com/downloads/utilities/

● MySQL Fabric Documentation

● http://dev.mysql.com/doc/mysql-utilities/1.5/en/fabric.html

● Forum (MySQL Fabric, Sharding, HA, Utilities)

● http://forums.mysql.com/list.php?144


MySQL Fabric ResourcesBlogs

● MySQL High-Availability

● http://mysqlhighavailability.com

● Mats Kindahl

● http://mysqlmusings.blogspot.com

● Alfranio Correia

● http://alfranio-distributed.blogspot.com

● Narayanan Venkateswaran

● http://vnwrites.blogspot.com


Thank You!

high-availability using mysql fabric

Technology

oracle andor

sole discretion of oracle

python mysql

c mysql fabric controller

farm of mysql servers

picture herehighavailability

easytouse framework

failing components