Migrating a Data Warehouse from Microsoft SQL Server to Oracle 11g

Dylan KuceraSenior Manager – Data ArchitectureOntario Teachers’ Pension Plan

Collaborate 2009 – Session ID 387Wednesday May 6, 20093:15pm-4:15pm

Agenda

• Gaining buy-in for a DW migration• Adoption of a new DW• Oracle Migration Workbench• Views over Transparent Gateway• Stored Procedure stubs over TG• Oracle Streams

Gaining buy-in for a DW migration

• Patience isn’t just a virtue, it is a

requirement

• Start thinking about metrics before

anything else

• Tie every message about the DW to

business requirements

Gaining buy-in for a DW migration

• Scalability• Don’t just focus on RAC• Locking Model

• Writers do not block Readers• No Lock Escalation

• Workload Balancing

• Availability• RAC• Extprocs and links to foreign DB’s run out of

process• Flashback

Gaining buy-in for a DW migration

• Environment Capability• PL/SQL capabilities may simplify your

deployments (Language constructs,

packages)• Advanced DW features eg. Materialized Views

• Maintainability• Oracle Enterprise Manager• Less replication = lower support costs

• Fit with strategic deployment• Best, if you are lucky enough!

A new Data Warehouse – If we build it, will they come?

• Promotion• Skills Development• Mentoring• Standards• Architectural Reviews• Controls

Oracle Migration (Workbench)

• OMW can help fast-track code

migration• Result may not conform to your standards

• Can be used for tables as well, but• Do you want to adjust model?• What about the legacy DW?

OMW – Capturing a SQL Server model

OMW – Captured T-SQL Procedure

OMW – Converted PL/SQL Procedure

OMW – But what about the legacy?

• Big bang – not possible or risky• Migrating consumers will take time

• Take a staged approach• Transparent Gateway• Oracle Streams

Views employing Transparent Gateway

CREATE OR REPLACE VIEW PLAY.VALUE_TABLE_SAMPLE ASSELECT "IDENTIFIER" AS ID_, "VALUE" AS VALUE_, FILE_DATE AS FILE_DATEFROM SampleLegacyTable@MSSQL

Pitfalls of Transparent Gateway - Views

DECLARE tDate DATE := '2008-12-31';BEGIN INSERT INTO PLAY.TEMP_SAMPLE_7445 (ID_, NAME_, PREV_VALUE, CURR_VALUE, VALUE_SUPPLIER, DATE_VALUE_CHANGED) SELECT ID_, '', '', '', 'SAMPLE', MAX(FILE_DATE) FROM PLAY.VALUE_TABLE_SAMPLE WHERE FILE_DATE <= tDate GROUP BY ID_;END;

Results in ORA-03113: end-of-file on communication channel

Alert log says ORA-07445: exception encountered: core dump [intel_fast_memcpy.A()+18] [ACCESS_VIOLATION] [ADDR:0x115354414B] [PC:0x52A9DFE] [UNABLE_TO_READ] []

Fixed in 11.1.0.6 Patch 10 and 11.1.0.7 Patch 7

CREATE PROCEDURE dbo.GetScheduleForRange

@inStartDate DATETIME, @inEndDate DATETIME

AS

SELECT DATE, DURATION, SESSION_ID, TITLEFROM NorthWind..COLLABSCHEDWHERE DATE BETWEEN @inStartDate AND @inEndDate

Procs employing Transparent Gateway

CREATE OR REPLACE PROCEDURE PLAY.RPT_COLLABORATE_SCHEDULE_RANGE ( inStart_Date DATE, inEnd_Date DATE, RC1 IN OUT SYS_REFCURSOR) IS

tRC1_MS SYS_REFCURSOR;tDate DATE;tDuration NUMBER;tSession_ID NUMBER;tTitle VARCHAR2(256);

BEGIN DELETE FROM PLAY.TEMP_COLLABORATE_SCHEDULE; dbo.GetScheduleForRange@MSSQL(inStart_Date, inEnd_Date, tRC1_MS);

LOOP FETCH tRC1_MS INTO tDate, tDuration, tSession_ID, tTitle; EXIT WHEN tRC1_MS%NOTFOUND; BEGIN INSERT INTO PLAY.TEMP_COLLABORATE_SCHEDULE (DATE_, DURATION, SESSION_ID, TITLE) VALUES(tDate, tDuration, tSession_ID, tTitle); END; END LOOP; CLOSE tRC1_MS; OPEN RC1 FOR SELECT DATE_, DURATION, SESSION_ID, TITLE FROM PLAY.TEMP_COLLABORATE_SCHEDULE ORDER BY SESSION_ID;

END RPT_COLLABORATE_SCHEDULE_RANGE;

Procs employing Transparent Gateway

RECIPE:

1) Declare Variables for all MSSQL Result set columns

2) Call MSSQL Procedure

3) Fetch Result one row at a time

4) Insert row to Oracle Temporary Table

5) Open Ref Cursor result set

{CALL PLAY.RPT_COLLABORATE_SCHEDULE_RANGE('2009-05-06 12:00:00', '2009-05-06 17:00:00')}

Results in ORA-06504: PL/SQL: Return types of Result Set variables or query do not match

Fixed in 11.1.0.7 Patch 7

Pitfalls of Transparent Gateway – Procs and Result Sets

Oracle Streams as an enabler of migration

Oracle Streams

ETL

Heterogeneous Streams – General Architecture

Diagram Adapted from “Oracle Database 11g: Oracle Streams Replication, An Oracle White Paper, July 2007”

Heterogeneous Streams – Example

Legacy Microsoft SQL Server Data Warehouse:

TABLE NorthWind.dbo.COLLABSCHED ( DATE DATETIME, DURATION INT, SESSION_ID INT, TITLE VARCHAR(256))

New Oracle Data Warehouse:

TABLE PLAY.COLLABORATE_SCHEDULE ( DATE_ DATE, DURATION NUMBER, SESSION_ID NUMBER, TITLE VARCHAR2(256));

BEGIN DBMS_APPLY_ADM.CREATE_APPLY( queue_name => 'SAMPLE_STREAM_Q', apply_name => 'SAMPLE_APPLY_NORTHWIND', apply_captured => TRUE, apply_database_link => 'MSSQL_STREAMS_NORTHWIND'); END; /BEGINDBMS_STREAMS_ADM.ADD_TABLE_RULES( table_name => 'PLAY.COLLABORATE_SCHEDULE', streams_type => 'APPLY', streams_name => 'SAMPLE_APPLY_NORTHWIND', queue_name => 'SAMPLE_STREAM_Q', include_dml => true, include_ddl => false);END;/

Heterogeneous Streams – Heterogeneous Apply

Heterogeneous Streams – Heterogeneous Apply Transforms

BEGINDBMS_STREAMS_ADM.RENAME_TABLE( rule_name => 'COLLABORATE_SCHEDULE554', from_table_name => 'PLAY.COLLABORATE_SCHEDULE', to_table_name => '"dbo".COLLABSCHED', step_number => 0, operation =>'ADD');END;/BEGIN DBMS_STREAMS_ADM.RENAME_COLUMN( rule_name => 'COLLABORATE_SCHEDULE554', table_name => 'PLAY.COLLABORATE_SCHEDULE', from_column_name => '"DATE_"', to_column_name => '"DATE"', value_type => '*', step_number => 0, operation => 'ADD');END;/

Heterogeneous Streams – Before inserts to Captured Oracle table

Heterogeneous Streams – Insert some rows into Captured Oracle table

SQL> INSERT INTO PLAY.COLLABORATE_SCHEDULE (DATE_,DURATION,SESSION_ID,TITLE) VALUES ('2009-05-06 08:30:00',60,359,'Oracle Critical Patch Updates: Insight and Understanding'); 1 row insertedSQL> INSERT INTO PLAY.COLLABORATE_SCHEDULE (DATE_,DURATION,SESSION_ID,TITLE) VALUES ('2009-05-06 11:00:00',60,237,'Best Practices for Managing Successful BI Implementations'); 1 row insertedSQL> INSERT INTO PLAY.COLLABORATE_SCHEDULE (DATE_,DURATION,SESSION_ID,TITLE) VALUES ('2009-05-06 12:15:00',60,257,'Best practices for deploying a Data Warehouse on Oracle Database

11g'); 1 row insertedSQL> INSERT INTO PLAY.COLLABORATE_SCHEDULE (DATE_,DURATION,SESSION_ID,TITLE) VALUES ('2009-05-06 13:30:00',60,744,'Business Intelligence Publisher Overview and Planned Features'); 1 row insertedSQL> INSERT INTO PLAY.COLLABORATE_SCHEDULE (DATE_,DURATION,SESSION_ID,TITLE) VALUES ('2009-05-06 15:15:00',60,387,'Migrating a Data Warehouse from Microsoft SQL Server to Oracle 11g'); 1 row insertedSQL> INSERT INTO PLAY.COLLABORATE_SCHEDULE (DATE_,DURATION,SESSION_ID,TITLE) VALUES ('2009-05-06 16:30:00',60,245,'Data Quality Heartburn? Get 11g Relief'); 1 row inserted SQL> COMMIT; Commit complete

Heterogeneous Streams – After inserts to Captured Oracle table

Heterogeneous Streams – NULL vs. Empty String (SQL Server)

Heterogeneous Streams – NULL vs. Empty String (SQL Server via Oracle TG)

Heterogeneous Streams – NULL vs. Empty String (Oracle)

Heterogeneous Streams – Syncing tables containing Floats

Heterogeneous Streams – Syncing tables containing Floats

Heterogeneous Streams – Syncing tables containing Floats

Wrap-up

• Gaining buy-in for a DW migration• Oracle Migration Workbench• Views over Transparent Gateway• Stored Procedure stubs over TG• Oracle Streams

• Standards, metadata, education,

mentoring

Questions?

Dylan KuceraSenior Manager – Data ArchitectureOntario Teachers’ Pension Plan

Collaborate 2009 – Session ID 387Wednesday May 6, 20093:15pm-4:15pm

Thank You! – Please complete the evaluation form – I value your feedback!