Data Replication with Advanced Replication &

Oracle Streams

John AbrahamsTechnology Sales ConsultantOracle Nederland

What is Replication

Multiple copies of data at different sites Increased availability Manual data replication implementations

– Export/Import– CREATE TABLE AS SELECT FROM

REMOTETABLE– COPY

Oracle9i Features for Information Sharing Features Introduced In Prior Releases

– Data Guard – Physical Standby Database– Advanced Queuing -- Message Queuing– Advanced Replication -- Replication of

Data– Change Data Capture -- DW Loading

Features available in Oracle9iR2– Oracle Streams -- a comprehensive

information sharing solution– Data Guard -- Logical Standby Database

based on Oracle Streams

Oracle Advanced Replication

Oracle9i Replication Technology

Oracle9i provides built-in technology to create and manage replicated environments

– Integrated, no add-ons, no special commands– Managed with Oracle Enterprise Manager

Advanced data replication technology– Bi-directional, all copies potentially updatable– Automatic conflict detection and resolution– Tables and supporting objects– Full copies or subsets– DDL (schema changes) as well as DML (transactions)– Continuous or on demand replication

Architectural OverviewSingle master replication

Single, updatable master Multiple updatable or

read-only materialized views (snapshots)

– Full copies or subsets of master

All conflicts resolved at parent site of materialized view

Oracle9i standard edition

master

materialized view

materialized view

master materialized

view

Support for multiple, n-way connected, updatable masters

Improves scalability and availability Oracle9i Enterprise Edition

mastermaster

master

M View

M View

M View

Master M View

M View

Architectural OverviewMultiple master replication

Master vs. Materialized View Replication

Multimaster Materialized View

• Server-to-Server • Mass Deployment

• Large number of small remote sites

• HQ and regional offices

• Continuous, near real- time data propagation

• Periodic Bulk Transfer

• Updates per transaction

• Final Values of Changed Rows Only

• Full Copies • Subsets or Full Copies

Advanced Replication Key Features Near real-time replication (multimaster)

– Parallel data propagation– Multiple, pre-defined conflict resolution methods

Mass deployment (materialized views)– subquery materialized views– deployment templates– multitier materialized views

Oracle Enterprise Manager for configuration and administration

Specialized options: procedural and synchronous replication

Near Real-time Replication Benefits:

– availability, scalability, failover

Uses:– Telesales, support

Requirements:– Efficient data capture and storage– Efficient data propagation– Continuous data propagation– Automatic resolution of conflicting updates

Architectural OverviewMultimaster

Groups of related schema objects kept in synch at multiple locations

Sites communicate by broadcasting changes to all other sites

mastermaster

master

Replication Objects

Database object replication to multiple servers

The following database objects can be replicated– Tables, Indexes, Views, Synonyms,

Triggers– Packages, Procedures, Functions– Advanced Data Types

User-Defined Types, Indextypes Tables with column objects, object tables Nested Tables, Varrays

Efficient Data Capture and Storage

Committed changes are added to queue for later propagation to remote sites

– Enqueued using advanced queueing mechanism

– Captured and applied using internal C code

– Minimum data needed to apply change is captured

Updates

Internal Trigger

Advanced Queue

Source tables

Efficient Data Propagation

Queued changes are pushed to remote sites in parallel for improved performance– Single parallel stream– Maintains transactional consistency– Automatically detects transaction dependencies

If update…If delete…If insert...

If update…If delete…If insert...

internalprocedure

background process

Parallel Data Propagation

Dependencies– Transaction “B” is dependent on

transaction“A” if “B” accesses data “A” has updated

– Dependency detection is dynamic and light-weight

Ordering– Dependent transactions are propagated

in dependency order– All other transactions are propagated in

parallel

Continuous Propagation

Changes can be continuously propagated or at a fixed interval, fixed time, or on demand

Different intervals can be used for each location

Different intervals can be used for each group

Dynamic Views to monitor propagation activity and throughput

Automatic Conflict Resolution

Automatic conflict detection with user-selectable conflict resolution routines

– latest timestamp, earliest timestamp, maximum or minimum value, overwrite, priority group, discard, site priority, average, or additive

User-definable resolution routines Detection and resolution based on

column groups

Mass Deployment Replication Benefits:

– disconnected, updatable Uses:

– field sales, field service Requirements:

– Easily define unique subsets– Easily deploy to 100’s of sites– Support mobile users

refresh on demand

– Full transactional consistency

– Efficient, batch-oriented refresh Scheduled or on demand Refresh groups preserve

master-detail relationships

– Updatable materialized views use deferred transactions to push changes to master

Architectural OverviewMaterialized Views

SELECT…FROM...

network

Master table

log

Materialized View

Updatable or read-only copy of a table, or portion of a table, at a point in time

Managing Advanced Replication

Specialized Options

Synchronous Replication– always up to date, no conflicts– slower response, network dependent

Procedural Replication– faster for batch processing– must be serialized, best done in off hours– useful for purging

Replication Summary

Full and subset Near realtime or on demand Graphical administration tool Sophisticated Functionality

– Efficient data capture and storage– Parallel data propagation– Automatic conflict detection and resolution– Subquery subsetting– Deployment templates– Multitier materialized views– Specialized options

Oracle Streams

Oracle Streams – Unified Messaging and Data Movement

Oracle Advanced Queuing (AQ)

Oracle Advanced Replication

Update Standby

Feed DataWarehouse

Messaging

Replication

Standby

Data Warehousing

Oracle Streams

Intelligent, Unified, Time-OrderedInformation Stream

Oracle Streams

A new solution for information sharing Provides a unified architecture for all

information sharing solutions– uniquely flexible replication– message queuing– data warehouse loading– event management and notification

The foundation of Data Guard Logical Standby Database

Streams Basic Elements

Three basic elements in each database– Capture– Staging– Consumption (apply)

ConsumptionConsumptionStagingStagingCaptureCapture

Multi-Database Streams

A stream can contain multiple elements from multiple databases

Events flow between staging areas

ConsumptionConsumptionStagingStagingCaptureCapture

ConsumptionConsumptionStagingStagingCaptureCapture

Capture

Streams captures events– Implicitly: log-based capture of DML and

DDL– Explicitly: Direct enqueue of user

messages

Captured events are published in the staging area

CaptureCapture

Log-Based Change Capture

Low overhead, low latency change capture– Changes to the database are written to the

online redo log– Oracle Streams can extract changes from the

log as it is written (mining the active log)– Changes are formatted as a Logical Change

Record (LCR), a SQL like representation of the change

CaptureCapture

Direct Enqueue

User applications can explicitly enqueue user messages into the staging area

– Multiple open interfaces supported: JMS, C, PLSQL, SOAP (XML/HTTP), XML/SMTP

– Allows applications to communicate at a higher level

– Allows users to introduce events into the stream from non-Oracle systems

CaptureCapture

Staging

Streams publishes captured events into a staging area– Implemented as a queue– Supports for new self-describing type “any”

datatype allows a single staging area to hold any type of data

– All events, LCRs and user-messages, can be staged in the same queue

– Messages remain in staging area until consumed by all subscribers

StagingStaging

Staging Area Propagation

Other staging areas can subscribe to events

– in same database

– in a remote database

Events can be routed through a series of staging areas

Propagation

StagingStaging

StagingStaging

Transformations

Transformations can be performed– as events enter the staging area– as events leave the staging area– as events propagate between staging areas

Transformation examples– change format, data type, column name,

table name

StagingStaging

Consumption

Staged events are consumed by subscribers

– Implicitly: Apply Process Default Apply User-Defined Apply

– Explictly: Application dequeue via open interfaces JMS, C, PLSQL, SOAP (XML/HTTP), XML/SMTP

ConsumptionConsumption

Default Apply

The default apply engine will directly apply the DML or DDL represented in the LCR

– apply to local Oracle table– apply via DB Link to non-Oracle table

Automatic conflict detection with optional resolution

– unresolved conflicts placed in exception queue

Parallel apply maximizes concurrency

ConsumptionConsumption

User-defined Apply

User-written custom apply functions Written in PL/SQL, Java, C, C++ Uses:

– custom transformations– column subsetting– normalizing or denormalizing data– populating related fields or tables

ConsumptionConsumption

Rule-based Subscription

Consumers subscribe to published events Content-based subscriptions limit delivered

events to those meeting the subscription criteria

Rules govern capture, staging, and consumption

Staging Area

RulesEngine

PublishSubscribe

UPDATE EMP... UPDATE EMP...

WHERE OBJECT = ‘EMP’

NYNY(master)(master)

LondonLondon(subset)(subset)

MilanMilan(subset)(subset)

ParisParis(subset)(subset)

Directed Networks

Propagation independent of Apply

Rules-based subscription determine if event is locally applied

– London applies UK only

WAN Friendly– Send once, fan out– NY-->London,

London-->Milan,London-->Paris

INSERT …VALUES (‘EUROPE’,’ ITALY’)

ITALY

EUROPE

FRANCE

Heterogeneous Support

Oracle to non-Oracle apply via gateway

– Apply process on Oracle node applies change

Non-Oracle to Oracle change capture supported via explicit enqueue of LCRs

Message Gateways– MQ Series– Tibco

LCR orLCR oruser messageuser message

GatewayGateway MessageMessageGatewayGateway

MQ SeriesMQ SeriesSybaseSybase

Streams Deployments

Streams can be deployed to meet a variety of information sharing requirements

– Replication– Data Warehouse Loading– Event Notification– Message Queuing– Data Guard Logical Standby Database

Replication Streams asynchronously maintains multiple

copies of objects via automatic apply– Identical objects– Related via a transformation or function

Streams automatically captures, propagates, and applies DML and DDL changes

– Detects and optionally resolves conflicts Supports flexible data movement and subsetting Gateways and APIs for heterogeneous support Compatible with Materialized Views

Replication

Benefits: – No quiesce for

DDL– Lower

overhead on production system

– Reduced network traffic

– Flexible configurations

Log-Log-basedbased CaptureCapture

StageStage DefaultDefaultApplyApply

Log-Log-basedbased CaptureCapture

StageStage DefaultDefaultApplyApply

PropagationPropagation

Data Warehouse Loading

Streams can load data warehouse staging areas and Operational Data Stores

– Updates captured from a production system– Messages and business events from a

process flow

Supports continuous or batch loading Automatically transforms data to

appropriate format and schema during Operational Data Store load

Data Warehouse Loading

Benefits: – low

overhead– automatic

transformation

– near real-time loading of operation data stores

StageStageUser-definedUser-definedApplyApply

PropagationPropagation

StageStageLog-Log-basedbased CaptureCapture

Production Database

Staging Table or ODS

Event Notification

Streams can notify subscribers that events of interest have occurred

– Pager notification of flight delays (Orbitz)– Notification of price drops (CNET Shopper)– Notification to sales manager of Gold

Customer purchase (CRM App)

Streams can evaluate DML events and send notifications to applications that send emails, page users, etc

– Users get information they want

Event Notification

Benefits: – scalable – reduced custom

development

Log-Log-basedbased CaptureCapture

StageStage ExplicitExplicitDequeueDequeue

Message Queuing Streams can be deployed as an enhanced

database integrated message queuing solution

– Point-to-point messaging, publish and subscribe– Single data, security and transactional model for

database and message queuing operations– Centrally managed and multi-consumer queues to

simplify configuration– Content-based subscriptions, internet access– Automatic dequeue to server-run user function– Automatic transform DML/DDL into messages

Message Queuing

Benefits: – Reduced

development costs– Easy database

integration– Single development,

operational, security model

– Reliability and integrity of database StageStage ExplicitExplicit

DequeueDequeue

PropagationPropagation

StageStageExplicitExplicitEnqueuEnqueu

ee

Source Database

Destination Database

Data Guard Logical Standby Database Multiple copies of data protects from

human and data errors, and disasters Special case of replication

– Entire database (by default)– One direction only

Streams supports reporting from standby as updates are applied

Data Guard adds higher level interface, tailored GUIs and broker

Data Guard Logical Standby Database

Benefits: – Open while

protecting data – Support near real-

time reporting from standby

– Protects from physical corruptions

– Additional indexes and materialized views

DefaultDefaultApplyApply

Remote LoggingRemote Logging

Log-Log-basedbased CaptureCapture

Production Database

Logical Standby Database

Other Oracle9i Information Sharing Features Advanced Replication

– provides compatible replication with Oracle 8, 8i, and 9i databases

– Migration path to Oracle Streams in future release Advanced Queuing

– Compatible with Oracle Streams– Most functionality offered in Streams– API’s retained for compatibility– Migration path to Oracle Streams in future release

Other Oracle9i Information Sharing Features Data Guard Physical Standby Database

– Uses media recovery mechanism to apply changes to database

– Creates an exact copy of the production database block-for-block copy Same version of Oracle, same

hardware/software architecture– Supports very high transaction workloads– Will coexist with Data Guard Logical

Standby Database

Summary

Oracle Streams unifies all enterprise information into a single Stream

– Unifies database, messaging, replication, publish/subscribe APIs and capabilities

Allows deployment of a variety of solutions

Provides a single, unified solution to the problem of Information Sharing

ConsumptionConsumptionStagingStagingCaptureCapture