Oracle Streams--Simplifying Information Sharing in Oracle10g

Patricia McElroyProduct Manager

Oracle Corporation

Session id: 40208

Patricia McElroyProduct Manager,Distributed SystemsOracle Corporation

Oracle Streams--Simplifying Information Sharing in Oracle10g

Agenda

Oracle Streams Overview Streams Architecture Hints and Tips

Oracle Streams

Simple solution for information sharing Provides

– uniquely flexible replication– message queuing– data warehouse loading– event management and notification

Streams Basic Elements

Capture Staging Consumption (apply)

ConsumptionConsumptionStagingStagingCaptureCapture

Multi-Database Streams

A stream can contain multiple elements from multiple databases

Events flow between staging areas

ConsumptionConsumptionStagingStagingCaptureCapture

ConsumptionConsumptionStagingStagingCaptureCapture

Capture

Streams captures events– Implicitly: log-based capture of DML and DDL– Explicitly: Direct enqueue of user messages

Captured events are published in the staging area SQL and messaging APIs in multiple languages

– JDBC, JMS, PL/SQL, C, SOAP

CaptureCapture

Log-Based Change Capture

Low overhead, low latency change capture– Changes to the database are written to the online

redo log– Oracle9i Streams can extract changes from the

log as it is written (hot mining)– Changes are formatted as a Logical Change

Record (LCR), a representation of the change

CaptureCapture

Logical Change Record(LCR)

Database change = LCR– DML

Object name, owner, Type of DML, SCN Row change = LCR OLD, NEW values

– DDL Object name, owner, Type of DDL, SCN DDL text

– LOB Multiple LCRs per LOB Piecewise chunks

Staging

Streams publishes captured events into a staging area

– Implemented as a queue– Supports for new type “any” datatype allows a single

staging area to hold any type of data– All events, LCRs and user-messages, can be staged in the

same queue– Messages remain in staging area until consumed by all

subscribers

StagingStaging

Staging Area Propagation

Other staging areas can subscribe to events

– in same database

– in a remote database

Events can be routed through a series of staging areas

Propagation

StagingStaging

StagingStaging

Transformations

Transformations can be performed– as events enter the staging area– as events leave the staging area– as events propagate between staging areas

Transformation examples– change format, data type, column name, table

name

StagingStaging

Consumption

Staged events are consumed by subscribers– Implicitly: Apply Process

Default Apply User-Defined Apply

– Explicitly: Application dequeue via open interfaces JMS, C, C++, PLSQL, SOAP (XML/HTTP)

ConsumptionConsumption

Default Apply

The default apply engine will directly apply the DML or DDL represented in the LCR

– apply to local Oracle table– apply via DB Link to non-Oracle table

Automatic conflict detection with optional resolution

– unresolved conflicts placed in exception queue

Parallel apply maximizes concurrency

ConsumptionConsumption

User-defined Apply

User-written custom apply procedures Written in PL/SQL, Java, C, C++ Uses:

– full control over apply– normalizing or denormalizing data– populating related fields or tables

ConsumptionConsumption

Rule-based Configuration

Consumers subscribe to published events Content-based subscription Rule is expressed as SQL WHERE clause

dbms_rule_adm.create_rule(

rule_name=>‘scott.rule1',

condition=>':dml.get_object_owner() = ''SCOTT'' AND

:dml.get_object_name()=''EMP''');

Rule sets for simplicity Rule sets govern capture,staging, and apply

– Negative, Inclusion

Dynamic rule maintenance

Replication Rules

Rule-based – DML, DDL– content– True/False

Granularity– Table– Subset of Table– Schema– Database– Tablespace

Tailored Replication API

DBMS_STREAMS_ADM– ADD_TABLE_RULES– ADD_SCHEMA_RULES– ADD_GLOBAL_RULES– ADD_SUBSET_RULES– MAINTAIN_TABLESPACES– ADD_MESSAGE_RULE

NYNY(master)(master)

LondonLondon(subset)(subset)

MilanMilan(subset)(subset)

ParisParis(subset)(subset)

Directed Networks

Propagation independent of Apply

Rules-based subscription determine if event is locally applied

– London applies UK only WAN Friendly

– Send once, fan out– NY-->London,

London-->Milan,London-->Paris

INSERT …VALUES (‘EUROPE’,’ ITALY’)

ITALY

EUROPE

FRANCE

Automatic Conflict Detection

Automatic conflict detection with user-selectable conflict resolution routines

– latest timestamp, earliest timestamp, maximum or minimum value, overwrite, discard

– User-definable resolution routines

Conflict detection compares current row values at receiving site with “old” values of changed row from the originating site

– if match, “new” values are applied to row– if not, conflict resolution method is used, if supplied– if still unresolved, place transaction in exception queue

Ability to disable conflict detection by column or table

Streams Capture and Apply

EMP

Update EMP set job=‘coding’ where empid=510;

Redo Log

Capture

EMP

empid|job |..

100 | sales |…

510 | coding| ...

Apply

Queue-----

LCRs

Queue------LCRs

Propagation

ACK

Capture

• Background process, CPnn

• Invokes Logminer

• Staged in buffer queue (SGA)

• Restartable

•Automatic Flow ControlRedo Log

Queue-----

LCRs

Capture

Ack

Capture Restart

• Every database change is in redo• Ordered by SCN• Last enqueued SCN

• capture restart• Last acknowledged SCN

• instance restart

Redo Log

Queue-----

LCRs

Capture

Ack

Downstream Capture

Zero impact at source site Changes are captured by another database

– Same Platform– Archive logs

Log transport services/ RFS to move logs

IMPLICITAPPLY

Propagation

Propagation

• Job Queue processes

• Stream LCRs

• Maintain status information

• 9i:Re-propagate on instance crash

•10g: Reuse spilled messages during recovery

In memoryQueue/LCRs

IMPLICITCAPTURE

In memoryQueue/LCRsAcknowledgements

LCRs

Apply

Reader

Coordinator

Server Server Server …..

• Background processes• Assembles transactions• Exactly once apply• Dependency computation• Conflict detectionEMP

empid|job |..

100 | sales |…

510 | coding| ...

Apply

Queue------LCRs

Ack

Transformation Example

Redo Log

Queue-----

LCRs

EMP

Update EMP set job=‘coding’ where empid=510;

Capture

Queue------LCRs

Person

Apply

Propagation

empid|job |..

100 | sales |…

510 |coding| ...

Emp->PersonT•1-1 Mapping•LCR based

ACK

Customized Apply Example

Queue-----

LCRs

EMP

Update EMP set state=‘CA’ where empid=100;

Redo Log

Capture

Queue------LCRs

Emp

Apply

Propagation

empid|state|..

95 | GA |…

105 | LA | ...

Do not apply Changes to EMP for state=‘CA’; Maintain AUDIT table of each LCR

U

•Full Control•Transaction Based

Audit

100 | CA

|…

Bit BucketACK

Data Subsetting

Queue-----

LCRs

EMP

Update EMP set job=‘CA’ where empid=100;

Redo Log

Capture

Queue------LCRs

EMP

ApplyPropagation

empid|state|..

100 | CA |…

205 | CA | ...

Queue------LCRs

EMP

Apply

empid|state|..

100 | LA |…

105 | LA | ...LA

CA

Insert e

mpid 100

Delete empid 100

ACK

ACK

Streams and RAC

Oracle9i– No hot mining– All Streams activities done on a single instance

(owning instance)– Streams restart after failure not automatic

Oracle10i– Hot mining– Primary and secondary owning instance– Failover of Streams processes automatic

Messaging Enhancements

Streams encompasses AQ Simplified messaging APIs Batch enqueue/dequeue Criteria-based purge Gateway to Tibco

Heterogeneous Support

Oracle to non-Oracle Apply via gateway

– Apply process on Oracle node applies change

Non-Oracle to Oracle change capture supported via explicit enqueue of LCRs

Message Gateways– MQ Series

LCR orLCR oruser messageuser message

GatewayGateway MessageMessageGatewayGateway

MQ SeriesMQ SeriesSybaseSybase

Heterogeneous Example

Queue-----

LCRs

EMP

Update EMP set state=‘CA’ where empid=100;

Redo Log

Capture

Queue------LCRs

EMP

Apply

Propagation

empid|state|..

100 | CA |…

105 | LA | ...

ACK

Apply2

Sybase

EMPTG4Sybs

US

Gateway

SybaseDR

UK JP

Report

Customer Example

Maintain application tables in 3 regional centers

– autonomous– low overhead – low latency

Provide Special Services– Reporting Database– Disaster Recovery– Legacy Application hosted

on Sybase

Streams Replication Features

Log-based Change Capture Customizable Apply Engine Directed Networks Schema Evolution Transformations Heterogeneous Support Explicit Enqueue/Dequeue

Streams Benefits for Replication Flexible Configurations Reduced Network Traffic Low Overhead No Downtime Requirements

Software Updates

9.2.0.4 patch, minimum 9.2.0.2 Streams Wizard patch Watch OTN Streams website for custom

software– http://otn.oracle.com/products/dataint/content.html

9.2.0.4 Configuration Tips

Use separate queues– Capture – Apply

Increase shared_pool_size and max_sga_size init.ora parameters

Relocate Streams data dictionary tables from SYSTEM

– Set tablespace before configuring streams– Including apply-only sites

Rule Tips Spelling counts! Eliminate duplicate rules in a rule set. Make sure that the source_database_name is

correctly specified.

– At source site: SELECT global_name FROM GLOBAL_NAME;

Ensure that rules do not allow objects with invalid data types

– When using GLOBAL rules, modify existing rule to eliminate the Streams Administrator schema and other schemas with queues, ADTs or tables with invalid data types.

Rule Management Tips

Having no rule set defined is NOT the same as an empty rule set.

Use DBMS_STREAMS_ADM package to create replication rules

Use same package to create or remove rules

– DBMS_STREAMS_ADM

– DBMS_RULE_ADM

Troubleshooting Tips Message Number columns = source DML,DDL SCN

– CAPTURE_MESSAGE_NUMBER, ENQUEUE_MESSAGE_NUMBER– DEQUEUE_MESSAGE_NUMBER, APPLIED_MESSAGE_NUMBER

ORA-26687 no instantiation scn set– Set Instantiation SCN– Exp/Imp with appropriate clauses– DBMS_APPLY_ADM.SET_TABLE_INSTANTIATION_SCN

ORA-01031 insufficient privilege– Explicitly grant privileges to apply user

MISSING Streams data dictionary information!– DBMS_CAPTURE_ADM.PREPARE_TABLE_INSTANTIATION

Troubleshooting #2

Determine the scope of the problem:

– Apply not working for this site only Check DBA_APPLY_ERROR for errors

Procedure to display error details in doc Check trace files Check rules at this site Still not working? …

– Apply not working for any site Check rules for both propagation and capture Check propagation job is working

Job queue processes DB Link

Make sure COMMIT was performed for transaction

Operational Tips

Replicating DDL?– Ensure that privileges are explicitly granted to the apply

user.– Modify any manual hotbackup scripts to set an apply tag

before starting the backup. – Configure Instantiation SCN at next higher level

Removing Archive log files from disk?– Ensure that scripts do not automatically remove files that

may be needed for capture to restart.

Implement “heartbeat” table – Update periodically.

Next Steps….

Recommended sessions36637  – Data Integration with Oracle Streamsata

Wed 4:30 Room 130 Integration Using Oracle St

Recommended demos and/or hands-on labs– Hands-On Lab: Integrate your Information.

See Your Business in Our Software– Visit the Streams booth in the DEMOgrounds for more

information.

Relevant web sites to visit for more information– http://otn.oracle;com/products/dataint/content.html

AQ&Q U E S T I O N SQ U E S T I O N S

A N S W E R SA N S W E R S