![Page 1: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/1.jpg)
I t d i th l l ti RMS d t Introducing the only real-time RMS data
replication solutionpUsing CDC and Data Replication
October 19, 2010
John Apps, OpenVMS Ambassador, HP
Hein van den Heuvel, HvdH Performance Consulting
Brian Schenkenberger, TMESIS
Richard Thomas, Director of Technical Services, Attunity
![Page 2: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/2.jpg)
Agenda
Connect
Offloading Data for BI – Enabling Technologies
RMS CDC – Attunity Stream for RMS
RMS Data Replication – Attunity RMS-CDC for SSIS
Demo - end to end!
Summary + Q&A
2
![Page 3: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/3.jpg)
Attunity - CONNECT
ODBC, JDBC, OLEdb… data access
DB2, Oracle, Enscribe, … RMS
Heterogeneous queries / optimizer
Bindings ( Tuning, Log level, Optimizer settings, Codepages )
Deamons
Workspaces ( Server Processes, OpenVMS Usernames )Workspaces ( Server Processes, OpenVMS Usernames )
Prestarted, re-usable, log files, tcpi
Datasources ( Dictionaries / Directories
TablesTables
Adapters
![Page 4: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/4.jpg)
Metadata Import
Automatically define relational schema based on existing
dmetadata
Automated imports available from:
COBOL (COPY books)
BASIC (MAP files)
FORTRAN
Oracle CDD
DIBOL
DATATRIEVE
…
![Page 5: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/5.jpg)
Agenda
Connect
Offloading Data for BI – Enabling Technologies
RMS CDC – Attunity Stream for RMS
RMS Data Replication – Attunity RMS-CDC for SSIS
Demo - end to end!
Summary + Q&A
5
![Page 6: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/6.jpg)
Offloading RMS Data for BI
Copy RMS data into a relational database
As frequently as needed
With little impact on the OpenVMS systemWith little impact on the OpenVMS system
SQL, JDBC, & Reporting toolsSQL, JDBC, & Reporting tools
RMS RelationalDatabaseReplication
/ETL ServerIncremental CDC
Initial full load
Load and apply
6
![Page 7: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/7.jpg)
Customer Example:Southeastern Freight Linesg
Privately-held trucking company, specializing in next day and second day
se ice The compan ’s dedication to service. The company’s dedication to process improvement is represented by
over 285 quality awards.
RMS1) R/O
RMS
RMS
Oracle RdbCOBOL R/W
2) R/W
CDC
.
.
.
![Page 8: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/8.jpg)
Customer Example:Express-Scripts – large pharmacyp p g p y
Connect
BigApplicationRefresh.
/ GJava / GUI
RMS
RMS
RMS
OracleCOBOL R/W
CDC + Replication
.
.
.
![Page 9: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/9.jpg)
Enabling Technologies
Ch C (C C) d h lChange Data Capture (CDC) is a data integration technology that captures and delivers only the changes made to enterprise data sources. It enables real-time and efficient data integration/replication.
CDC
Data Replication creates a copy of one data source in another, periodically or in real-time. Replication employs CDC to replicate changes from source to target efficiently and in
ReplicationCDC to replicate changes from source to target efficiently and in real-time.
![Page 10: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/10.jpg)
Drivers for CDC and Replication
1 Demand for real-time data1. Demand for real-time data
2 Increasing data volumes2. Increasing data volumes
3 Shrinking batch windows3. Shrinking batch windows
4 Pressure to reduce cost 4. Pressure to reduce cost
5 Reduce risks of failing long batch jobs5. Reduce risks of failing long batch jobs
10
![Page 11: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/11.jpg)
Use Cases for CDC and Replication
1. Offload Production Data for BI
Create a copy for reporting
Stage data for data warehousing
2. Maintain Consistency across Operational DBs
Keep data integrityKeep data integrity
For application integration and MDM (master data management)
3. Data Migration/Upgrade
Sync with production following initial set-up
Enable fail-back
11
![Page 12: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/12.jpg)
Challenges for CDC and Replication
1. Low Impact on existing applications
Achieving minimal/no changes to existing applications
Achieving low overhead on resources
2. Data mapping challenges
Defining a relational schema for the RMS structures
A l i RMS h i l h l i l blApplying RMS changes appropriately to the relational tables
Arrays (of records) in records; Variant records.
Funky data types (notably Dates!)Funky data types (notably Dates!)
3. Complexity/Cost
Making it simple automatedMaking it simple, automated
Low cost of ownership12
![Page 13: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/13.jpg)
Agenda
Connect
Offloading Data for BI – Enabling Technologies
RMS CDC – Attunity Stream for RMS
RMS Data Replication – Attunity RMS-CDC for SSIS
Demo - end to end!
Summary + Q&A
13
![Page 14: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/14.jpg)
CDC Approaches and Technologies
Traditional approaches – Intrusive/High Impact:
Use timestamps in extracts and file dumps
Extract and compare current data to its history Connx uses Checksums on RMS Indexed files buckets to detect changes.
Use custom-built triggers
Attunity approach Non Intrusive:Attunity approach – Non Intrusive:
Log-based CDC
Low impact Non-intrusiveLow impact. Non intrusive.
Efficient.
Reliable.
Scalable.
14
![Page 15: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/15.jpg)
Attunity Stream for RMS
Log-based CDC for RMS files
ETL/BI
OpenVMS (Alpha/Itanium) Windows, UNIX
RMS
BPEL, BPM
JMS
ChangeTables
CDC Capture CDC Router
ESB
CEP
JMSPublisher
15
![Page 16: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/16.jpg)
RMS – CDC - Key Capabilities
Log-based CDCg
Filter Changes (by file, operation, …)
Flexible Change Delivery (periodic continuous)Flexible Change Delivery (periodic, continuous)
Interfaces to ETL and EAI Tools
l bl l dReliable Delivery and Recovery
Easy to Use
16
![Page 17: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/17.jpg)
CDC for RMS
Unique service intercepts $RMS system servicesProvides system wide capture of RMS changes
Cobol, Basic, Datatrieve, DCL, Utilities... Any and all RMS record operation.
Record based (log : header + record data )H d S # Ti Fil ID P ID RFA O i SiHeader : Sequence#; Timestamp; FileID; ProcessID; RFA; Operation; Size.
No changes required to any RMS programsSupports all RMS file typesS O VMS lSupport OpenVMS clustersIndependent of RMS (AI/BI) Journaling. RU-J not supported.High performance, low impact
Per node Logger not noticeable (3%?)Shared Agent example usage: 1/4 CPU for 10 Million changes in 3 GB Logs.
Easy ConfigurationGUI b d ith i d i th Att it St di
17
GUI-based with wizards in the Attunity StudioMetadata import available from many sources
![Page 18: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/18.jpg)
RMS CDC Architecture - Overview
Change
OpenVMS (Alpha/Itanium) Windows, UNIX
RMS
ChangeTables
CDC Capture CDC Router
JMSPublisher
LegacyProcess 1 Process 2 Process 3
Attunity RMS Application
RMS Intercept RMS Logger CDC Agentp gg g
RMS
![Page 19: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/19.jpg)
RMS CDC Architecture – RMS Intercept
RMS Intercept – forms an envelope around the $RMS
system service. Captures changes only for specified list of
files. By file-id (from filename) or by name (“next version”)
LegacyProcess 1 Process 2 Process 3
Attunity RMS Application
RMS Intercept RMS Logger CDC Agentp gg g
RMS
![Page 20: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/20.jpg)
RMS CDC Architecture – RMS Logger
RMS Logger – a separate process (1 per node) which reads
the intercepted changes from the intercept and writes them
to the transient storage
LegacyProcess 1 Process 2 Process 3
Attunity RMS Application
RMS Intercept RMS Logger CDC Agentp gg g
RMS
![Page 21: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/21.jpg)
RMS CDC Architecture – Transient Storageg
Transient Storage – a sequence of typically-fixed size-
sequential files holding the captured changes. When one fills
up, a new one is created. (ALQ, DEQ, Max.). Utility to
LegacyProcess 1 Process 2 Process 3
control attributes and force new logs if so desired (Daily?)
Attunity RMS Application
RMS Intercept RMS Logger CDC Agentp gg g
RMS
![Page 22: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/22.jpg)
RMS CDC Architecture – CDC Agent
Attunity RMS CDC Agent – Async to the capture process, it
reads changes from the transient storage and propagates them
to the change processor off platform. Single active agent per
LegacyProcess 1 Process 2 Process 3
‘solution’ (router on linux/windows). Failover as needed.
Attunity RMS Application
RMS Intercept RMS Logger CDC Agentp gg g
RMS
![Page 23: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/23.jpg)
Agenda
Offloading Data for BI – Enabling Technologies
RMS CDC – Attunity Stream for RMS
RMS Data Replication – Attunity RMS-CDC for SSIS
Demo - end to end!
Summary + Q&A
24
![Page 24: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/24.jpg)
Attunity RMS-CDC for SSIS
Complete solution for data replicationInitial Load + Incremental LoadSupports any target (e.g. Oracle/Rdb, SQL Server,...)Monitoring and Control
Leverage CDC for incremental loadLog-based RMS CDCS nch oni ed ith Initial LoadSynchronized with Initial LoadGranular scheduling
Easy to UseDesign-time with wizards in BIDS/Visual StudioAutomatic generation of metadata and packagesAutomatic generation of metadata and packagesRuntime uses standard SSIS packages
25
![Page 25: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/25.jpg)
Attunity RMS-CDC for SSISProduct Components
RMS
26
RMS
![Page 26: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/26.jpg)
Attunity Stream for Multiple Consumers
Targets
RMS RMS
OpenVMS
MQ
ETL/Staging Server
CDC R t
ChangeTablesSQL Server
AttunityCDC Agent
RMS
MQ
Router
DWETL
SSIS
SQLT Logs
SQLTables
SQLTables
AttunityCDC Agent Informatica,
OracleClient
ODSg
OracleTables
OracleTables
Oracle Server
Agent ,DataStage
ETL
Audit
Oracle Server
OracleLog Miner
Audit
D/R
StreamFlow
27Oracle
Log Miner
OracleTables
OracleTables Attunity is compatible with any ETL Tool
![Page 27: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/27.jpg)
Agenda
Introduction
Connect
RMS CDC – Attunity Stream for RMS
RMS Data Replication – Attunity RMS-CDC for SSIS
Demo - end to end!
Summary + Q&A
28
![Page 28: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/28.jpg)
Agenda
Offloading Data for BI – Enabling Technologies
RMS CDC – Attunity Stream for RMS
RMS Data Replication – Attunity RMS-CDC for SSIS
Demo - end to end!
Summary + Q&A
29
![Page 29: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/29.jpg)
Summary
Offload RMS data for reporting and BI
Simplify data migration and modernization projects
Synchronize RMS with other data sources
Extract RMS data efficiently and in real-time
Eliminate batch windows required to move RMS data
30
![Page 30: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/30.jpg)
Summary – About Attunity
• Real-time data integration & event capture
Corporate • +15 years of expertise in data/legacy integration
• Worldwide operations with offices in US, Europe and Asia
Products
• CDC and Operational Data Replication
• Data Connectivity and FederationProducts • Data Connectivity and Federation
• Legacy Adapters and Web Services
Customers
• +400 direct customers worldwide
• Technology of choice for HP, Microsoft, Oracle, IBM, ...
• Thousands via OEM partners
31
![Page 31: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/31.jpg)
Next Steps…
1 A k ti1. Ask questions
2. Get started today with an evaluation
3. Check our introductory pricing for existing customers
32
![Page 32: Using CDC and Data Replication - vmsconsultancy.com Attunity real-ti… · Reduce risks of failing long batch jobs 10. ... Transient Storage – a sequence of typically-fixed size-](https://reader034.vdocuments.mx/reader034/viewer/2022051105/5ab44d177f8b9a86428ba07e/html5/thumbnails/32.jpg)
Thank You!