tungsten university: setup and operate tungsten replicators
DESCRIPTION
Do you have the background necessary to take full advantage of Tungsten Replicator in your environments? Tungsten offers enterprise-quality replication features in an open source package hosted on Google Code. This virtual course teaches you how to set up innovative topologies that solve complex replication problems. We start with single MySQL servers running MySQL replication and show a simple path migration path to Tungsten. Course Topics - Checking host and MySQL prerequisites - Downloading code from http://code.google.com/p/tungsten-replicator/ - Installation using the tungsten-installer utility - Transaction filtering using standard filters as well as customized filters you write yourself - Enabling and managing parallel replication - Configuring multi-master and fan-in using multiple replication services - Backup and restore integration - Troubleshooting replication problems - Logging bugs and participating in the Tungsten Replicator community Replication is a powerful technology that takes knowledge and planning to use effectively. We give you the background that makes replication easier to set up, and allows you to take full advantage of the Tungsten Replicator benefits. Learn how to configure and use it more effectively for your projects in the cloud as well as on-premises hardware.TRANSCRIPT
©Continuent 2013
Tungsten University: Set Up and Operate
Tungsten Replication Robert Hodges, CEO
Giuseppe Maxia, Director of QA
©Continuent 2013
About Continuent
• The leading provider of clustering and replication for open source DBMS
• Tungsten Clustering - Commercial-grade HA, performance scaling and data management for MySQL
• Tungsten Replication - Real-time, "exible, high-performance replication
2
©Continuent 2013
Introducing Tungsten Replicator
3
©Continuent 2013
Tungsten Replicator Overview
4
Master
(Transactions + Metadata)
Slave
THL
DBMSLogs
Replicator
(Transactions + Metadata)
THLReplicator
Download transactions via network
Apply using JDBC
©Continuent 2013
Tungsten Replication Service
5
Extract Filter Apply
StageExtract Filter Apply
StageExtract Filter Apply
Stage
Pipeline
MasterDBMS
TransactionHistory Log
In-MemoryQueue
SlaveDBMS
©Continuent 2013
Overview of Installation Process
1. Set up hosts
2. Prepare MySQL replicas
3. Download software
4. Install using tungsten-installer
6
Amazon Setup:https://docs.continuent.com/wiki/display/TEDOC/
Preparing+EC2+Servers
©Continuent 2013
How tungsten-installer Works
7
db1
db2
db3
Staging copy of files
check prereqscopy codeconfigure
©Continuent 2013
What Is Tungsten Doing?
8
©Continuent 2013
Commands to Check Replication
• trepctl services -- All services
• trepctl status -- Overall pipe status
• trepctl status -name stages -- Stage summary
• trepctl status -name stores -- Store summary
• trepctl status -name tasks -- Task detail
• trepctl [-service name] heartbeat -- Is it Alive?
9
©Continuent 2013
Changing Replicator Con!guration
10
©Continuent 2013
Commands to Change Con!guration
• con#gure-service - Alter con#guration
• update - Upgrade the replicator to a new version
11
Hint: Don’t change replicator configuration files directly...
©Continuent 2013
Recon!guring a Replicator
• No Restart
trepctl offlinetools/configure-service ...trepctl configuretrepctl online
• With Restart
tools/configure-service ...replicator restart
12
©Continuent 2013
Enabling Parallel Replication
13
©Continuent 2013
Parallel Replication of Sharded Data
14
Extract Filter Apply
StageExtract Filter Apply
StageStage
Pipeline
MasterDBMS
TransactionHistory Log
ParallelQueue
SlaveDBMS
Extract Filter ApplyExtract Filter ApplyExtract Filter Apply
©Continuent 2013
Fixing Problems
15
©Continuent 2013
The Scene of the Crime
16
trepctl -host db2 statusProcessing status command...NAME VALUE---- -----...pendingError : Event application failed: seqno=1957469 fragno=0 message=java.sql.SQLException: Statement failed on slave but succeeded on masterpendingErrorCode : NONEpendingErrorEventId : mysql-bin.000008:0000000461736666;0pendingErrorSeqno : 1957469pendingExceptionMessage: java.sql.SQLException: Statement failed on slave but succeeded on master create table trouble(id int primary key) /* ___SERVICE___ = [university] */...Finished status command...
©Continuent 2013
Correcting problems
1. Look at the transaction (thl list -seqno NNN)
2. Look at the replicator message log
3. Fix the problem and go online
OR
4. Skip over the problem and go online
OR
5. Reprovision the slave
17
©Continuent 2013
Tungsten Logs
18
©Continuent 2013
Simple Facts about Logs
• Replicator log = THL (Transaction History Log)
• One THL per replication service
• 2x the size of MySQL binlog
• Default retention of 7 days
• Take service o$ine to move or delete logs safely
• The thl tool shows you log contents
19
©Continuent 2013
Resetting Replication
20
©Continuent 2013
Resetting and Reinstalling
Reset
1. Stop replicator
2. Remove THL #les
3. Remove tungsten_<svc> databases
Reinstall
1. All of the above, plus delete /opt/continuent/*
2. Rerun installation
21
©Continuent 2013
Backup and Restore
22
©Continuent 2013
What Backup Tools Does MySQL Have?
23
MySQL Server
Process
File System
DBMS Files
mysqldump(dataset < 10Gb)
xtrabackup(10Gb < dataset < 1Tbtar/rsync(dataset < 1Tb)
file system snapshot(1Tb < dataset)
©Continuent 2013
Advanced Installation using Cookbooks
24
©Continuent 2013
How Do I Install Fan-In Replication?
25
db1Replicator
db3
Service db1
db2Replicator
Service db2
Replicator
Service db1
Service db2
©Continuent 2013
How Do I install Multi-Master?
26
db1 Replicator
Service db1
Service db2
Service db3db3
Service db1
Service db2
Service db3db2
Replicator
Service db1
Service db2
Service db3
©Continuent 2013
Tungsten Cookbook Helps!
27
• Installs complex topologies automatically
• Directions in cookbook/README
• Parameters in cookbook/USER_VALUES.sh
• Node names in COMMON_NODES.sh
• Installation script for each supported topology
©Continuent 2013
Joining the Community
28
©Continuent 2013
Tungsten Replicator is Open Source
29
• Project home:
http://code.google.com/p/tungsten-replicator/
• Log bugs, #nd builds, post in replicator discussion group
• Documentation:
https://docs.continuent.com/wiki/display/TEDOC/Tungsten+Documentation+Home
©Continuent 2013
Wrapping Up
30
©Continuent 2013
Tungsten University Sessions
31
• Unleashing the Power of Tungsten Connectors(February 21 & 26)
• Managing Multi-Master and Fan-in Topologies(March)
• Zero-downtime DBMS Migration and Maintenance (March)
Send any feedback to: [email protected]
©Continuent 2012.
Continuent Web Page:http://www.continuent.com
Tungsten Replicator 2.0:http://code.google.com/p/tungsten-replicator
Our Blogs:http://scale-out-blog.blogspot.comhttp://datacharmer.org/bloghttp://www.continuent.com/news/blogs
560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009e-mail: [email protected]