oracle streams--simplifying information sharing in oracle10 g
DESCRIPTION
Session id: 40208. Oracle Streams--Simplifying Information Sharing in Oracle10 g. Patricia McElroy Product Manager Oracle Corporation. Patricia McElroy Product Manager, Distributed Systems Oracle Corporation. Oracle Streams--Simplifying Information Sharing in Oracle10 g. Agenda. - PowerPoint PPT PresentationTRANSCRIPT
Oracle Streams--Simplifying Information Sharing in Oracle10g
Patricia McElroyProduct Manager
Oracle Corporation
Session id: 40208
Patricia McElroyProduct Manager,Distributed SystemsOracle Corporation
Oracle Streams--Simplifying Information Sharing in Oracle10g
Agenda
Oracle Streams Overview Streams Architecture Hints and Tips
Oracle Streams
Simple solution for information sharing Provides
– uniquely flexible replication– message queuing– data warehouse loading– event management and notification
Streams Basic Elements
Capture Staging Consumption (apply)
ConsumptionConsumptionStagingStagingCaptureCapture
Multi-Database Streams
A stream can contain multiple elements from multiple databases
Events flow between staging areas
ConsumptionConsumptionStagingStagingCaptureCapture
ConsumptionConsumptionStagingStagingCaptureCapture
Capture
Streams captures events– Implicitly: log-based capture of DML and DDL– Explicitly: Direct enqueue of user messages
Captured events are published in the staging area SQL and messaging APIs in multiple languages
– JDBC, JMS, PL/SQL, C, SOAP
CaptureCapture
Log-Based Change Capture
Low overhead, low latency change capture– Changes to the database are written to the online
redo log– Oracle9i Streams can extract changes from the
log as it is written (hot mining)– Changes are formatted as a Logical Change
Record (LCR), a representation of the change
CaptureCapture
Logical Change Record(LCR)
Database change = LCR– DML
Object name, owner, Type of DML, SCN Row change = LCR OLD, NEW values
– DDL Object name, owner, Type of DDL, SCN DDL text
– LOB Multiple LCRs per LOB Piecewise chunks
Staging
Streams publishes captured events into a staging area
– Implemented as a queue– Supports for new type “any” datatype allows a single
staging area to hold any type of data– All events, LCRs and user-messages, can be staged in the
same queue– Messages remain in staging area until consumed by all
subscribers
StagingStaging
Staging Area Propagation
Other staging areas can subscribe to events
– in same database
– in a remote database
Events can be routed through a series of staging areas
Propagation
StagingStaging
StagingStaging
Transformations
Transformations can be performed– as events enter the staging area– as events leave the staging area– as events propagate between staging areas
Transformation examples– change format, data type, column name, table
name
StagingStaging
Consumption
Staged events are consumed by subscribers– Implicitly: Apply Process
Default Apply User-Defined Apply
– Explicitly: Application dequeue via open interfaces JMS, C, C++, PLSQL, SOAP (XML/HTTP)
ConsumptionConsumption
Default Apply
The default apply engine will directly apply the DML or DDL represented in the LCR
– apply to local Oracle table– apply via DB Link to non-Oracle table
Automatic conflict detection with optional resolution
– unresolved conflicts placed in exception queue
Parallel apply maximizes concurrency
ConsumptionConsumption
User-defined Apply
User-written custom apply procedures Written in PL/SQL, Java, C, C++ Uses:
– full control over apply– normalizing or denormalizing data– populating related fields or tables
ConsumptionConsumption
Rule-based Configuration
Consumers subscribe to published events Content-based subscription Rule is expressed as SQL WHERE clause
dbms_rule_adm.create_rule(
rule_name=>‘scott.rule1',
condition=>':dml.get_object_owner() = ''SCOTT'' AND
:dml.get_object_name()=''EMP''');
Rule sets for simplicity Rule sets govern capture,staging, and apply
– Negative, Inclusion
Dynamic rule maintenance
Replication Rules
Rule-based – DML, DDL– content– True/False
Granularity– Table– Subset of Table– Schema– Database– Tablespace
Tailored Replication API
DBMS_STREAMS_ADM– ADD_TABLE_RULES– ADD_SCHEMA_RULES– ADD_GLOBAL_RULES– ADD_SUBSET_RULES– MAINTAIN_TABLESPACES– ADD_MESSAGE_RULE
NYNY(master)(master)
LondonLondon(subset)(subset)
MilanMilan(subset)(subset)
ParisParis(subset)(subset)
Directed Networks
Propagation independent of Apply
Rules-based subscription determine if event is locally applied
– London applies UK only WAN Friendly
– Send once, fan out– NY-->London,
London-->Milan,London-->Paris
INSERT …VALUES (‘EUROPE’,’ ITALY’)
ITALY
EUROPE
FRANCE
Automatic Conflict Detection
Automatic conflict detection with user-selectable conflict resolution routines
– latest timestamp, earliest timestamp, maximum or minimum value, overwrite, discard
– User-definable resolution routines
Conflict detection compares current row values at receiving site with “old” values of changed row from the originating site
– if match, “new” values are applied to row– if not, conflict resolution method is used, if supplied– if still unresolved, place transaction in exception queue
Ability to disable conflict detection by column or table
Streams Capture and Apply
EMP
Update EMP set job=‘coding’ where empid=510;
Redo Log
Capture
EMP
empid|job |..
100 | sales |…
510 | coding| ...
Apply
Queue-----
LCRs
Queue------LCRs
Propagation
ACK
Capture
• Background process, CPnn
• Invokes Logminer
• Staged in buffer queue (SGA)
• Restartable
•Automatic Flow ControlRedo Log
Queue-----
LCRs
Capture
Ack
Capture Restart
• Every database change is in redo• Ordered by SCN• Last enqueued SCN
• capture restart• Last acknowledged SCN
• instance restart
Redo Log
Queue-----
LCRs
Capture
Ack
Downstream Capture
Zero impact at source site Changes are captured by another database
– Same Platform– Archive logs
Log transport services/ RFS to move logs
IMPLICITAPPLY
Propagation
Propagation
• Job Queue processes
• Stream LCRs
• Maintain status information
• 9i:Re-propagate on instance crash
•10g: Reuse spilled messages during recovery
In memoryQueue/LCRs
IMPLICITCAPTURE
In memoryQueue/LCRsAcknowledgements
LCRs
Apply
Reader
Coordinator
Server Server Server …..
• Background processes• Assembles transactions• Exactly once apply• Dependency computation• Conflict detectionEMP
empid|job |..
100 | sales |…
510 | coding| ...
Apply
Queue------LCRs
Ack
Transformation Example
Redo Log
Queue-----
LCRs
EMP
Update EMP set job=‘coding’ where empid=510;
Capture
Queue------LCRs
Person
Apply
Propagation
empid|job |..
100 | sales |…
510 |coding| ...
Emp->PersonT•1-1 Mapping•LCR based
ACK
Customized Apply Example
Queue-----
LCRs
EMP
Update EMP set state=‘CA’ where empid=100;
Redo Log
Capture
Queue------LCRs
Emp
Apply
Propagation
empid|state|..
95 | GA |…
105 | LA | ...
Do not apply Changes to EMP for state=‘CA’; Maintain AUDIT table of each LCR
U
•Full Control•Transaction Based
Audit
100 | CA
|…
Bit BucketACK
Data Subsetting
Queue-----
LCRs
EMP
Update EMP set job=‘CA’ where empid=100;
Redo Log
Capture
Queue------LCRs
EMP
ApplyPropagation
empid|state|..
100 | CA |…
205 | CA | ...
Queue------LCRs
EMP
Apply
empid|state|..
100 | LA |…
105 | LA | ...LA
CA
Insert e
mpid 100
Delete empid 100
ACK
ACK
Streams and RAC
Oracle9i– No hot mining– All Streams activities done on a single instance
(owning instance)– Streams restart after failure not automatic
Oracle10i– Hot mining– Primary and secondary owning instance– Failover of Streams processes automatic
Messaging Enhancements
Streams encompasses AQ Simplified messaging APIs Batch enqueue/dequeue Criteria-based purge Gateway to Tibco
Heterogeneous Support
Oracle to non-Oracle Apply via gateway
– Apply process on Oracle node applies change
Non-Oracle to Oracle change capture supported via explicit enqueue of LCRs
Message Gateways– MQ Series
LCR orLCR oruser messageuser message
GatewayGateway MessageMessageGatewayGateway
MQ SeriesMQ SeriesSybaseSybase
Heterogeneous Example
Queue-----
LCRs
EMP
Update EMP set state=‘CA’ where empid=100;
Redo Log
Capture
Queue------LCRs
EMP
Apply
Propagation
empid|state|..
100 | CA |…
105 | LA | ...
ACK
Apply2
Sybase
EMPTG4Sybs
US
Gateway
SybaseDR
UK JP
Report
Customer Example
Maintain application tables in 3 regional centers
– autonomous– low overhead – low latency
Provide Special Services– Reporting Database– Disaster Recovery– Legacy Application hosted
on Sybase
Streams Replication Features
Log-based Change Capture Customizable Apply Engine Directed Networks Schema Evolution Transformations Heterogeneous Support Explicit Enqueue/Dequeue
Streams Benefits for Replication Flexible Configurations Reduced Network Traffic Low Overhead No Downtime Requirements
Software Updates
9.2.0.4 patch, minimum 9.2.0.2 Streams Wizard patch Watch OTN Streams website for custom
software– http://otn.oracle.com/products/dataint/content.html
9.2.0.4 Configuration Tips
Use separate queues– Capture – Apply
Increase shared_pool_size and max_sga_size init.ora parameters
Relocate Streams data dictionary tables from SYSTEM
– Set tablespace before configuring streams– Including apply-only sites
Rule Tips Spelling counts! Eliminate duplicate rules in a rule set. Make sure that the source_database_name is
correctly specified.
– At source site: SELECT global_name FROM GLOBAL_NAME;
Ensure that rules do not allow objects with invalid data types
– When using GLOBAL rules, modify existing rule to eliminate the Streams Administrator schema and other schemas with queues, ADTs or tables with invalid data types.
Rule Management Tips
Having no rule set defined is NOT the same as an empty rule set.
Use DBMS_STREAMS_ADM package to create replication rules
Use same package to create or remove rules
– DBMS_STREAMS_ADM
– DBMS_RULE_ADM
Troubleshooting Tips Message Number columns = source DML,DDL SCN
– CAPTURE_MESSAGE_NUMBER, ENQUEUE_MESSAGE_NUMBER– DEQUEUE_MESSAGE_NUMBER, APPLIED_MESSAGE_NUMBER
ORA-26687 no instantiation scn set– Set Instantiation SCN– Exp/Imp with appropriate clauses– DBMS_APPLY_ADM.SET_TABLE_INSTANTIATION_SCN
ORA-01031 insufficient privilege– Explicitly grant privileges to apply user
MISSING Streams data dictionary information!– DBMS_CAPTURE_ADM.PREPARE_TABLE_INSTANTIATION
Troubleshooting #2
Determine the scope of the problem:
– Apply not working for this site only Check DBA_APPLY_ERROR for errors
Procedure to display error details in doc Check trace files Check rules at this site Still not working? …
– Apply not working for any site Check rules for both propagation and capture Check propagation job is working
Job queue processes DB Link
Make sure COMMIT was performed for transaction
Operational Tips
Replicating DDL?– Ensure that privileges are explicitly granted to the apply
user.– Modify any manual hotbackup scripts to set an apply tag
before starting the backup. – Configure Instantiation SCN at next higher level
Removing Archive log files from disk?– Ensure that scripts do not automatically remove files that
may be needed for capture to restart.
Implement “heartbeat” table – Update periodically.
Next Steps….
Recommended sessions36637 – Data Integration with Oracle Streamsata
Wed 4:30 Room 130 Integration Using Oracle St
Recommended demos and/or hands-on labs– Hands-On Lab: Integrate your Information.
See Your Business in Our Software– Visit the Streams booth in the DEMOgrounds for more
information.
Relevant web sites to visit for more information– http://otn.oracle;com/products/dataint/content.html
AQ&Q U E S T I O N SQ U E S T I O N S
A N S W E R SA N S W E R S