odi 12c workshop syd 20140820 slides

8/10/2019 ODI 12c Workshop Syd 20140820 Slides

1/78

Copyright 2013, Oracle and/or its affiliates. All rights reserved.1


2/78


Oracle Data Integrator Overview

Eddie Ho,Senior Sales Consulting Manager, DISPrashant Kompella, Senior Sales Consultant

20thAug, 2014


3/78


Agenda

8.45 Arrival, Registration & Housekeeping & Introductions

9.00 Overview of ODI, Whats New, Demo & introduction to the Lab environment

10.00 Labs session 1Basic mapping, Distinct, Expression, Sort

10.45

11.00 Labs session 2Join, Filter, Lookup, Unions, Reusable mappings

12.30 < Lunch >

1.00 ODI Concepts & Product Update

2.00 Labs session 3Aggregate, Split, Multiple targets and Parallel loads

3.00 < Break >

3.15 Labs session 4Debugging, Standalone Agent, Scheduling

4.30 Q&A, / wrap-up / next steps

4.45 Close


4/78


lear_Guest

Logistics


5/78Copyright 2013, Oracle and/or its affiliates. All rights reserved.5

Introductions

Your name

Your company

Your role Your objectives


6/78Copyright 2013, Oracle and/or its affiliates. All rights reserved.6

Agenda




10.45


12.30 < Lunch >



3.00 < Break >



4.45 Close


7/78


Oracle Data IntegrationOverview


8/78


Data Integration TrendsData Integration Tools TodayNeed to Do More

Big Datais the new normal

Real-timedata jumps to the

Application data now lives inin a multitude of disparate en

Conventional ETL tools are smeet performancedemand


9/78


New Requirements for Data Integration

Integrated andHigh Productivity

Tooling

High Performance,High Availability

Any Data Latency,Real-time

Any Data,Any Source


10/78


MDMApplications

SOPlatf

BusinessIntelligence

ActivityMonitoring

CustomApplications

Oracle GoldenGate

Log-based CDC

Data Replication

Real-time Data

SOA Abstraction LayerService BusProcess Manager Data Services

Oracle Data IntegratorELT/ETL

Data Transformation

Bulk Data Movement

OLTPSystem

Flat FilesData Warehouse/Data Mart

OLAP Cube Web 2.0 Web anServiceExadata

Data Verification

Oracle Data QualitData Profiling

Data Parsing

Data Cleansing

Data Federation

Data Lineage Match and Merge

Comprehensive Data Integration Solution

Oracle Data Integration SolutionsBest-in-class Heterogeneous Platform for Data Integration

Oracle

Applications


11/78


Oracle Data IntegratorOverview


12/78


Certifiedtechnolotime to v

High-perof ownerarchitect

Lightwe

Flexiblefunctiona

Oracle Data Integrator

High Performance E-LT

Declarative Design

CEP Data Services

Extensible Knowledge Modules

Capable of Integrating Structured &Unstructured Data

Big Data

Database

Apps

Cloud

Oracle Data Integrator 12cBulk Data Processing and Data Transformation


13/78


Conventional ETL Architecture

Extract LoadTransform

Next Generation Architecture

E-LTLoad

TransformTransform

Extract

Industry Leading Performance of Oracle ODIExtremely Fast Execution and Reduced Cost

ODIPeformance


14/78


Declarative DesignImproved Developer Productivity For Faster Time to Soluti

Easyand universalfor designingsimple to complex mappings

Robust andReusable: CreateMappings once for any physical design

Quickto define and refactor maps,high degree of automation

Extensiblemodel for any dataintegration mechanism

Benefits

Significantly reduces the learning curve

Shortens implementation times

Simpler maintenance

ODI Declarative Design

DefineHow: Knowledge Mo

Define

WhatYou Want

AutomGeDa

1


15/78


ReverseEngineerMetadata

Journalize(CDC)

Load fromSource toStaging

CheckConstraint

Integrate,Transform

Pluggable Knowledge Modules Architecture

Key ArchitectureBenefits:

Faster development and simpler maintenanc

Easy to extend and add new best practices

Enforces predictability and reduces cost of o

Knowledge ModulesSimpler Physical Design and Shorter Implementation Time

Sample Out-of-the-Box Knowledge Modules (100+)

Non-Oracle Hadoop IBM DB2 SQL ServerSAP ABAP JSybase NoSQL

Oracle GoldenGateOracleUtilities

Siebel CRMOracle

DBLINKOr

Web SE-Business

SuiteOracle Merge


16/78


Big Data: Hadoop support

Best for Oracle: Merge, Spatial, Multi-Table Insert, Optimizer Hints and more

Real-Time: CDC with GoldenGate

Applications: E-Business Suite, Siebel,PeopleSoft, JD Edwards EnterpriseOne, JD Edwards World, SAP ERP andSAP BW

Heterogeneous: Optimizations for allmajor RDBMS: IBM DB2, Microsoft SQLServer, Teradata, Netezza etc.

Extended ConnectivityLeverages Existing IT, Faster Implementation

Dat

App

Leg

Uns

JMS


17/78


Oracle Data Integratorin Data IntegrationSolutions


18/78


Oracle Real-time EDW Architecture

Load

ODS Data MartApplication Databases

ETL

Staging

Real-timeReplication oftransactional

Changesby GoldenGate

High frequencymicro-batchesof deltasby OracleData Integrator(ODI) + GoldenGateintegrated solution

Incrementalupdatesof Dimensions

andFacts by ODICDC framework


19/78


Sources

EWs

Oracle DataIntegrator

Data ProfilingAnalyze and understand datato build ODI mappings

Automated ProcessesDe-duplication, complexcleansing and parsinginvoked in ODI workflow

MeasureAssess qin target sis ETL wo

Enterprise DataQuality

EDQ and ODI: Overview

Data qualityfirewall


20/78


Sources

Oracle Enterprise Data Quality

Parsing Standardization Cleansing Matc

Oracle Data IntegratorE-LT/ETL Process

Create newData Quality Rules

2

- Add Data Qualityto E-LT/ETL Flow

3

Profile Data1

EDQ and ODI: Comprehensive Data Quality Pro


21/78


ODI Concepts & ProductUpdate


22/78


ODI EL-T Architecture Overview

Repository

(design-time+Run-time)

Flat

Files

ODI Agent

(Java process)

Flat

Files

ODI Agent(Java process)

ODIStudio

ODIStudio

Design/Admin/Monitoring Runtime (scheduling)

Runtime (scheduling)

ODI M d l


23/78


ODI Modules

JavaMost Platforms

DB - Most ISO-92

Security Ma

Manage userprivileges

Operator

Operate productionMonitor sessions

Topology Manager

Define the infrastructureof the IS

Designer

Reverse-EngineerDevelop ProjectsRelease Scenarios

Repository

ODI Designer

R Ti C t


24/78


Run-Time Components

Any ISO-92 RDBMS

Execute Jobs

Agent

Handles schedulesOrchestrate sessions

Java - Any Platform

Operator

Operate productionMonitor sessions

Designer

Reverse-EngineerDevelop ProjectsRelease Scenarios

Java - Any Platform

Information System

Return Code

Submit Jobs

Read sessionsWrite reports

Monitor sessions

View Reports

Lightweight

Distributed Architectur

Repository

ODI 12 M i


25/78


ODI 12c MappingDeclarative and Flow combined

Easy for moderate use cases

Shows relationships of sourcetables

Abstracts out physicalimplementation from logical map

Quick to define and refactormaps

Universal for big and complexuse cases

Extensible model for any dataintegration mechanism

M i C t


26/78


Mapping Components

ODI 12.1.3 Out-of-Box Components: Aggregate

Dataset Distinct

Expression

Filter

Join

Lookup

Pivot, Unpivot Set

Sort

Split

Subquery

Table Functions

M lti l T t


27/78


Multiple Targets

A mapping can contain multipletargets

Any component can map result tomultiple downstream components

Splitcomponent allows to split inputlogically based on expressions

R bl M i


28/78


Reusable Mappings

Capture common logic (sources, targets, transformations

Reuse logic in mappings

Step by Step Debugger


29/78


Provide control to the user on the

Session Execution Flow Pause, Resume, Run to next Step, etc.

Define breakpoints on Steps inPackages

Toggle breakpoints in Sessions

Query data through Agent (givesaccess to uncommitted data)

Step by Step DebuggerOverview

T pes of ODI Kno ledge Mod les


30/78


Types of ODI Knowledge Modules

KM Type Description

CKM Check Checks data in a datastore or during an integIKM Integration

Uses a given strategy to populate the target from the staging area.

LKM Loading Assembles data from source datastores to th

RKM Reverse-engineeringRetrieves the structure of a data model fromneeded for customized reverse-engineering.

JKM JournalizingSets up a system for Changed Data Captureamount of data that needs to be processed.

Interfac

es

Models

There are six different types of knowledge modules:

SKM Web ServicesDefines the code that will be generated to crServices (Exposing data as a web service)

E-LT Architecture with Data Integrator


31/78


Key Architecture Benefits: 100% Java, Open APIs, very fast E-LT

E-LT Architecture uses DBMS CPUs, not the Application CPUs

Open APIs, Open Metadata, Open XML Knowledge Modules

AB

FileC

C$_0

C$_1

LKM

LKM

IKM

I$

IKM

RKM

JKM

ChecTransformExtract-Load

ODI Agent

Apps Business Intelligence

ODI Agent may be deployedin any part of thearchitecture

ODI nativall major

c tectu e t ata teg atoComprehensive, Lightweight Data Integration


32/78


Todays hands-on labenvironment

Hands On Lab Scenario


33/78


Hands-On Lab Scenario

Orders Application:An application for trackingcustomer orders, hosted in Oracle and used as

a source.

Parameters (File): Flat files (ASCII) issued fromthe production system containing a list of salesrepresentatives and the segmentation of agesinto age ranges.

Sales Administration: The administration ortracking of sales, hosted in another supplieddatabase. This data warehouse is populatedwith our transformations.

OrdersApplication

Parameters(Sales Source)

Lookup

Join

1

2L

ogical

Physical

Hands On Lab Scenario


34/78


Hands-On Lab ScenarioWhat you are about to see

OrdersApplication

Parameters(Sales Source)

SalesAdministration

ODI Studio Easy to use Navigation

Centrally ManagedMetadata

Logically &Physically

Abstracted


35/78

Hands-On Lab Scenario


36/78


Hands-On Lab ScenarioWhat you are about to see

ODI Studio

Predefined Integrationvia Knowledge Modules

Session logging

Record tracking

Agenda


37/78


Agenda




10.45


12.30 < Lunch >



3.00 < Break >



4.45 Close


38/78


Hands-On LabSessions 1 & 2


39/78


ODI (More) AdvancedConcepts

ODI Model & Logical/Physical Schema


40/78


ODI Model & Logical/Physical Schema Environments

Project 1 Project 2

Model A Model B Model C

LS X LS Y LS Z

Testing Product

Project 1

Model A Model B

LS X LS Y

ODI Context

Logical/Physical Views for Productivity Impro


41/78


og ca / ys ca e s o oduct ty p o

ETL jobs aredeveloped onlogical views- Portable acrossenvironments

Logical switchingto differentenvironments


42/78


43/78


44/78


45/78


46/78


47/78


48/78

ODI Check Knowledge Module (CKM)


49/78


g ( )Static ControlCheck Data in Table against Constrain


50/78


51/78

Comparison of 2 ODI CDC Frameworks


52/78


CDC Journal TableJ$

Primary Keycolumns

window_id

123 FromSEQ+RBA oftrail file

GoldenGatereplicat

DML on table

(Classic)

CDC Journal TableJ$

Primary Keycolumns

window_id

123 Empty

Insert by OGG

Insert byper-rowtable trigger


53/78


54/78


55/78


56/78


57/78

EDQ Profiling Example


58/78



59/78

Key Feature: Pre-built processors


60/78


Comprehensive DQ Functionality with a Single User Interface and Rep


61/78


62/78


Latest ODI Updates (12.1.3)

ODI 12.1.3 New Features


63/78


FIPS Compliance

XML Driver Enhancements (e.g. Recursion)

JSON Support Hadoop Sqoop/Hbase Integration

Multi-Threaded Target Table Load in ODI Engine

Improved Control for Scenario/Load Plan Concurrent Ex

Create New Mode/Topology Objects (without building topphysical/logical to model)

More details can be found from viewlethttp://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/odi/odi_12c/odi_12.html
http://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/odi/odi_12c/odi_12.1.3_nf/presentation.htmlhttp://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/odi/odi_12c/odi_12.1.3_nf/presentation.htmlhttp://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/odi/odi_12c/odi_12.1.3_nf/presentation.htmlhttp://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/odi/odi_12c/odi_12.1.3_nf/presentation.html


64/78


65/78

Management Pack for Oracle Data IntegratorKey Features


66/78


Discovery of ODI components Agents: Standalone and JEE

Repositories Source and Target databases

SLA Management Alerts creation

Notifications: email, SNMP, etc.

Monitoring

Availability Monitoring ODI homepage to quickly assess

infrastructure status

Performance Monitoring

ODI components performance

y


67/78

ODI Big DataSQOOP Load


68/78


Fast Load betweendatabases and Hadoop

Database can be sourceor target

HDFS, Hive, or HBasesupported

Parallelized load using adata partitioning scheme

Oracle D

SQOOP

SQOOP

SQOOP

ParallelJDBC Queries

Source Database

OverviewIn-Session Parallelism


69/78


Goal

Allow the execution of tasks andsteps in parallel rather thansequentially

How

Introduction of a new Runtime

concept: Execution Units In 12.1.2 loading of C$ tables can

be executed in parallel

ODI 11g

ODI 12.1.2

SrcSet1

SrcSet2

SrcSet3

SrcSet1

SrcSet2

SrcSet3

Integration

Total Execution Time

Total Execution Time

Multi-Threaded Target Load (12.1.3)


70/78


Fast Load through parallelJDBC connections

Configurable per DataServer

Source ODI Agent

Para

Load

Thre

JD

Data Server settings:

Enterprise Role IntegrationIntroduction


71/78


Goal Tighter integration with LDAP security

How Introduction of ODI Roles ODI Roles can be mapped to Enterprise Roles

Enterprise Role inherits ODI Role privileges


72/78


Oracle Data Integration

Oracle WarehouseBuilder Product Update

OWB Support PoliciesCustomer Downloadable OWB Support Policies:
http://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdfhttp://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdf


73/78


Release GA DatePremier

Support EndsExtended

Support Ends Sus

OWB 9.2 Jul 2003 Oct 2007 Not Available OctOWB 10.1 Apr 2004 Jul 2007 Not Available Jul

OWB 10.2 Jul 2005 Jul 2010 Jul 2013 Ind

OWB 11.1 Aug 2007 Aug 2012 Aug 2015 Ind

OWB 11.2 Sep 2009 Current Dates:Jan 2015

Future Dates:

Same as DB12

Current Dates:Jan 2018

Future Dates:

Same as DB12

Ind

For more Information about Support Levels:http://www.oracle.com/us/support/lifetime-support/index.html

Customer Downloadable OWB Support Policies:http://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdf

Best Practices for OWB to ODI Migratio
http://www.oracle.com/us/support/lifetime-support/index.htmlhttp://www.oracle.com/us/support/lifetime-support/index.htmlhttp://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdfhttp://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdfhttp://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdfhttp://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdfhttp://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdfhttp://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdfhttp://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdfhttp://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdfhttp://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdfhttp://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdfhttp://www.oracle.com/us/support/lifetime-support/index.htmlhttp://www.oracle.com/us/support/lifetime-support/index.htmlhttp://www.oracle.com/us/support/lifetime-support/index.htmlhttp://www.oracle.com/us/support/lifetime-support/index.html


74/78


ODI 12c supports a phased migration approach

Execute and monitor existing OWB 11gR2 jobs from ODI 12

Use the migration utility to convert existing OWB 11gR2 objetheir ODI equivalents

Start new developments using ODI 12c

Understand that OWB and ODI are different ETL tools

There is no free lunch when migrating a technical migrat

between different tools will never be 100% perfect or predict

Invoking OWB 11.2 Processes from OD


75/78


New OWB Runtime Repositorytechnology available in Topology

OWB processes can be invokedusing the OdiStartOwbJob tool inPackages

OWB audit information is availablein Operator, ODI Console andEnterprise Manager

OWB to ODI Migration UtilityOverview


76/78


Versions supported

OWB 11.2.0.4 to ODI 12c

Design-time metadata migration only

Data Objects

Mappings

Separate command line utility

uses OWB SDK and ODI SDK


77/78


Hands-on Labs


78/78


odi 12c workshop syd 20140820 slides

Documents