spread the database love with heterogeneous replication · ©continuent ltd 2017 heterogeneous...

©Continuent Ltd 2017

Spread the Database Love with Heterogeneous Replication

MC Brown, VP, Products


Heterogeneous Replication is NOT

• Exporting and Importing Data

• Moving to a different database platform

• One Time Exports

• ETL


Heterogeneous Replication IS

• Live, constant, low-latency movement of data

• For analytics

• For migration

• For upgrades

• For Caching

• Data/format matching

• Effective target reproduction


Know Your Databases


Not all Databases are Created Equal

• Transactional over non transactional

• Object Reference

• Rows

• Columns

• Documents

• Free text

• Unstructured


Is that a record, a field, a row a column?

• Row of data?

• Collection of related tables?

• What does it look like as a document?

• What does a document look like as a row?

• Databases, tables, collections, objects, buckets…


Related Tables or Document?


Mapping DB Compatibility

RDBMS Columnar Store Document Database

Freetext/unstructured data store

RDBMS Vendor specific only

Vendor specific only

Field mappings only Application specific

Columnar Store Vendor specific only

Vendor specific only

Field mappings only Application specific

Document Database

Field mappings only

Field mappings only

Vendor specific only Application specific

Freetext/unstructured data store

Application specific Application specific Application specific Application specific

Vendor specific - i.e. unique data typesField mappings - how we map the data

App Specific - how the data is used


Know Your Data


Hetero Replication Challenges

• Effective data replication

• nothing lost or removed

• low latency

• Automatic mapping

• Data typing

• Indexing and native use


Challenges: Data Typing

• Data types are not supported everywhere

• For some, the type does not matter

• Even if the type does matter, the format, precision, structure might be different

• Numbers, Dates, Strings, Compound Data Types all cause problems


Extraction/Apply Rates

• Data extraction rates vary

• Data apply rates

• Different solution handle data loading at different rtes

• Rows-based extraction/bulk apply

• Bulk extraction/row apply

• Non-destructive


Whats the solution?


Replicator Needs

• Native, neutral format

• Ability to change, reformat, restructure information

• Standalone nature

• Two-way

• Handle impedance problem


Guess What?

• Tungsten Replicator does this

• High Performance

• Flexible storage interchange format

• Built-in filtering

• Operates standalone

• Stop and restart

• Transactionally consistent

• Open Source


Applying Data


Native is Best, Batch an Alternative

• Native:

• Applying to JDBC

• Adapt JDBC Applier to construct statement

• Or apply a record to target using API

• Batch

• Use CSV for data interchange

• Call scripts to import


How Batch Apply Works

Replicator

Service ora2vrTransactions from master

CSVFilesCSVFilesCSVFiles

StagingTablesStagingTablesStagingTables

Base Tables

Base Tables

Base Tables

Merge Script

(or)COPY

directly to base tables

COPY to stage tables SELECT to

base tables


How Batch Apply Works

• Works on one table at a time

• Five functions in JavaScript

• Prepare - Run when going online

• Begin - Start of transaction

• Apply - During transaction

• Commit - After transaction

• Release - When going offline


During a transaction

• Copy, import, load the CSV

• Have access to column, key and transaction information

• Merge the data

• Delete and Insert, or

• Delete, Update and Insert

• Done


Case Study: Cassandra/CQL

• Load table data:

• COPY staging_tablename (optype,seqno,uniqno,id,message) from ‘FILENAME’

• Delete:

• delete from sample where id in (#{deleteidlist})

• Insert:

• insert into sample ("+collist+") values ("+substlist+")


Filters


Filter Execution

Extract Filter Apply

StageExtract Filter Apply

Stage

MySQLMaster

TransactionHistory Log

In-MemoryQueue

Slave ReplicatorsBinlog

tcp/ip


Filter Operation

• Always get one transaction at a time

• Transaction must be processed inline

• Metadata

• Data blocks

• SQL or ROW Info

• Always returns the transaction


JS Filters

• prepare() - called when going online

• filter() - does the work

• release() - called when going offline

• Access to:

• Connection to DB

• Full Java class environment

• Bunch of utility functions


Data Structure

ReplDBMSEvent DBMSData StatementData

DBMSData StatementData

DBMSData RowChangeData OneRowChange

OneRowChange

...

StatementData

ReplDBMSEvent DBMSData RowChangeData OneRowChange

OneRowChange

...

http://docs.continuent.com/tungsten-replicator-5.1/filters-javascript-writing-data-repldbmsevent.html

http://docs.continuent.com/tungsten-replicator-5.1/filters-javascript-writing-data-dbmsdata.html

http://docs.continuent.com/tungsten-replicator-5.1/filters-javascript-writing-data-statementdata.html




http://docs.continuent.com/tungsten-replicator-5.1/filters-javascript-writing-data-rowchangedata.html

http://docs.continuent.com/tungsten-replicator-5.1/filters-javascript-writing-data-rowchangedata.html#filters-javascript-writing-data-onerowchange



http://docs.continuent.com/tungsten-replicator-5.1/filters-javascript-writing-data-repldbmsevent.html


http://docs.continuent.com/tungsten-replicator-5.1/filters-javascript-writing-data-rowchangedata.html




Get/Set Valuesfor(j = 0; j < rowChanges.size(); j++){ oneRowChange = rowChanges.get(j); columns = oneRowChange.getColumnSpec(); columnValues = oneRowChange.getColumnValues(); for (c = 0; c < columns.size(); c++) { columnSpec = columns.get(c); type = columnSpec.getType(); if (type == TypesDATE || type == TypesTIMESTAMP) { for (row = 0; row < columnValues.size(); row++) { values = columnValues.get(row); value = values.get(c);

if (value.getValue() == 0) { value.setValueNull() } } } }}


What you can do in a filter

• Anything


Case Study: Building a Kafka

Applier


Kafka?

• Message queue/bus

• Full publish/subscribe model

• Huge flexible

• Very practical

• High performance

• Not a database


Message Format for Data

• Embedded JSON

• CSV Row

• Encoded binary fields

• Message topic?

• Schema/table/primary key?


Impedance

• What happens with multi-row transactions?

• What happens when a multi-row transaction is not applied?

• Should we split data into chunks?


What we do already

Sources TargetsMySQL MySQLOracle Oracle

RedShiftVerticalHadoop

TextSQLite

RabbitMQS3

MongoDB


What are we adding?

Sources TargetsREST API Input Cassandra

MongoDB Amazon AthenaCouchbase CouchbaseCouchDB CouchDB

PostgreSQL ElasticSearchFlumeKafka

Native JDBC to HadoopPostgreSQL


Where Next

• github.com/continuent/tungsten-replicator

• mcb.guru

http://github.com/continuent/tungsten-replicator