1 copyright © 2012, oracle and/or its affiliates. all ... · optimized for oracle database 11g and...

52
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration 1

Upload: others

Post on 21-Mar-2020

14 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration1

Page 2: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

The following is intended to outline our general product direction. It is intended for information purposes only,

and may not be incorporated into any contract. It is not a

commitment to deliver any material, code, or

functionality, and should not be relied upon in making

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration2

functionality, and should not be relied upon in making

purchasing decisions.The development, release, and timing of any features or

functionality described for Oracle’s products remains at

the sole discretion of Oracle.

Page 3: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Magdalena BartnikUgo Pollio

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration3

Oracle Data Integration

Moving Data

to Transform

Business

Oracle Data Integration Platform

Moving Data

to Transform Business

Page 4: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler
Page 5: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Program Agenda

� Three New Disruptors in Data Integration

– Zero Downtime and Continuous Availability

– Real-time Analytics

– Trusted Business Insight

� GG, ODI & EDQ

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration5

� GG, ODI & EDQ

� Putting it all Together

� Architecture Principles and Best Practices

� Use cases

Page 6: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

New Requirements in Data Integration

Real-time AnalyticsReal-time Analytics

Any Data, Any SourceAny Data,

Any SourceZero Downtime, High AvailabilityZero Downtime, High Availability

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration6

Page 7: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Big Data Drives Big Benefits

Any Data, Any SourceAny Data,

Any Source

TransactionsYour Data:Decisions based on

your dataDocuments

Oracle SAP

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration7

Big Data:Decisions based on all

data relevant to you

Machine-Generated Data

Social Data

Page 8: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Oracle Strategy for Integrating Big DataOracle Data Integrator and Big Data Connectors

Any Data, Any SourceAny Data,

Any Source

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration8

Big Data Transformation

Bulk Data, E-LT

OLTP OLAP UnstructuredLegacy

Page 9: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Convergence of Real-time and Availability

Zero Downtime, High AvailabilityZero Downtime, High Availability

Real-Time Enterprise

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration9

Systems that require 24/7

availability contain critical data to be shared in real time

To supply real-time data

source systems have to be continuously available

Page 10: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Oracle GoldenGateReal-time Data Integration and Continuous Availability

Zero Downtime, High AvailabilityZero Downtime, High Availability

Real-time

� Log-based change data capture, distribution,

transformation, and delivery

� Supports heterogeneous databases and

operating systems

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration10

Real-time Replication � Bidirectional replication without distance

limitation

� Transactional integrity. Reliable data delivery

and fast recovery after interruptions

OLTP OLAP Legacy

Page 11: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Oracle GoldenGate 11gR2Real-time Data Integration and Continuous Availability

� Efficiently capture changed data in high-volume and high-throughput implementations

� Optimized for Oracle Database 11g

Integrated CaptureIntegrated Capture

GoldenGate 11gR2

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration11

� Optimized for Oracle Database 11g and Oracle Exadata environments

� Reduce the time required to detect and resolve data conflicts

� Increased support for business-critical and heterogeneous systems

Enhanced Conflict Detection & ResolutionEnhanced Conflict Detection & Resolution

Security & PerformanceSecurity & Performance

Monitoring, ManagementMonitoring, Management

Expanded Heterogeneity (iSeries)Expanded Heterogeneity (iSeries)

Page 12: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

GoldenGate 11gR2 Differentiator: HeterogeneityDatabases O/S and Platforms

Oracle GoldenGate Capture and Delivery:

� Oracle

� DB2

� Microsoft SQL Server

� Sybase ASE

� Teradata

� Enscribe

SQL/MP

Linux

Sun Solaris v11

Windows 2000, 2003, XP

HP NonStop

HP-UX

New

New

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration12

� SQL/MP

� SQL/MX

� MySQL v 5.5

� JMS message queues

Oracle GoldenGate Delivery:

� All listed above, plus:

�TimesTen, Postgres

� Netezza, Greenplum, HP Neoview

� ETL products

HP-UX

IBM AIX v7.1

IBM z Series

zLinux

IBM i Series New

New

New

New

Page 13: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Oracle Strategy for Real-time Analytics

Data Quality (Product & Customer Data)

Real-time AnalyticsReal-time Analytics

Business Analytics

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration13

Big Data Transformation

Real-time Replication

Bulk Data, E-LT

Data Services, Federation

OLTP OLAP UnstructuredLegacy

Page 14: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Real Time AnalyticsDelivering maximum performance at a lower cost

� Moving towards the real-time enterprise via real-time data delivery

� Optimized for Oracle Database 11g and Oracle Exadata environments

� Handles continuous data loading while

Decrease batch windows for data acquisitionDecrease batch windows for data acquisition

GoldenGate ODI 11G EDQ

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration14

� Handles continuous data loading while users run reports

� Efficiently capture changed data in high-volume and high-throughput implementations

� Transform data into valuable information

Enable efficiencyEnable efficiency

“Active” vs “Archive” data“Active” vs “Archive” data

Business Decision happen in Real-TimeBusiness Decision happen in Real-Time

Reliable data, trustable insightsReliable data, trustable insights

Page 15: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

� Super fast data refresh: GoldenGatefor real-time replication

� Super Fast data movement : Data Integrator for Movement &

Real Time AnalyticsDelivering maximum performance at a lower cost

Real-time AnalyticsReal-time Analytics

Real-time

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration15

Integrator for Movement & Transformation

� Fast Aggregates and view refresh: In-memory optimizations in Times Ten

OLTP Legacy

Real-time Replication

Page 16: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs.

Trail: stages and queues data for routing.

Pump: distributes data for routing to target(s).

Route: data is compressed, encrypted for routing to target(s).

Delivery: applies data with transaction integrity, transforming the data as required.

How Oracle GoldenGate Works

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration16

integrity, transforming the data as required.

Source

Oracle & Non-Oracle

Database(s)

Target

Oracle & Non-Oracle

Database(s)Bi-directional

Page 17: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

• A high-speed, low impact data comparison solution

• Identifies and reports data discrepancies

between heterogeneous databaseswithout interrupting their availability

• Benefits:

Oracle GoldenGate VeridataData Comparison without Downtime

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration17

• Benefits:

• Reduce financial/legal risk exposure

• Speed and simplify IT work in comparing data sources

• No disruption to business systems

• Improved failover to backup systems

• Confident decision-making and reporting

Page 18: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

� Advanced monitoring, alerting (SNMP support), lag graphs, and historical repository.

Management Pack for Oracle GoldenGate

Oracle GoldenGate Monitor

Monitoring

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration18

� Based on new agent infrastructure, which will be the foundation for all future integration projects. Supports GoldenGate 11.1.1.1.1+

Page 19: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

• Oracle Preferred monitoring tool, covers entire Oracle stack –Database, Storage, FMW etc

Management Pack for Oracle GoldenGate

Enterprise Manager Plug-in

Monitoring

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration19

• For customers, who prefers to drill into DB sessions, OS stats, etc, apart from monitoring GoldenGate process

• Included in Management Pack

Page 20: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Oracle Data Integrator 11gFastest ETL/E-LT, Simpler Time-to-Value, Lowest TCO

Best in class performance, scalability

Lowered TCO, lightweight deployment Any Data

Legacy Sources

Oracle Data Integrator Enterprise Edition

20 Copyright © 2012, Oracle and/or its affiliates. All rights

reserved.

Lowered TCO, lightweight deployment

Improved usability and manageability

Heterogeneous

Open, standards-based

Any Data Warehouse

Any Planning SystemOLTP DB

Sources

Application Sources

Page 21: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

ODI – Oracle’s Strategic ELT/ETL

ODI is Faster

• Fastest E-LT Bulk/Batch Performance

• Faster Real Time integration (sub-second) with CDC, Replication, and SOA

• Faster Project Setup, Design and Delivery

ODI is Simpler

• Simpler Setup, Configuration, Management, and Monitoring

21 Copyright © 2012, Oracle and/or its affiliates. All rights

reserved.

• Simpler way to do Mapping using Declarative SQL Interfaces

• Simpler Deployment with Fewer Hardware Devices

• Simpler extensibility with Knowledge Module code templates

ODI is Saves Money (Lower TCO, Higher ROI)

• Less Hardware & Energy Costs with E-LT Architecture

• Less Time Wasted on Unnecessary ETL Mappings, Scripting, and Training

• Less Integration Overhead Integrating with Applications, SOA, and Management

Page 22: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Problem - Shrinking ETL Batch Windows

TRANSFORM

2LOAD

3

EXTRACT

1

22 Copyright © 2012, Oracle and/or its affiliates. All rights

reserved.

• ETL Processing Exceeds Business Batch

windows due to increased data volumes

Challenge

Older data

Page 23: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Optimized Data Loading through E-LT• The key to improved performance and reduced costs

• E-LT provides flexible architecture for optimized performance

• Benefits

• Leverage Set-based

Conventional ETL Architecture

Extract LoadTransform

23 Copyright © 2012, Oracle and/or its affiliates. All rights

reserved.

• Leverage Set-based transformations

• Improved performance for loading, no network hop

• Takes advantage of existing

hardware

Next Generation Architecture

“E-LT”“E-LT”Load

TransformTransform

Extract

Page 24: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

ODI-EE Declarative Design• Improved Developer Productivity and Lower Maintenance Costs

Conventional ETL DesignSpecify ETL Data Flow Graph• Developer must define every step of

Complex ETL Flow Logic• And significant development and

maintenance efforts

Declarative Set-based Design

24 Copyright © 2012, Oracle and/or its affiliates. All rights

reserved.

• Simplifies the number of steps• Automatically generates the Data Flow

whatever the sources and target DB

Benefits� Significantly reduce the learning curve

� Shorter implementation times� Streamline access to non-IT pros

ODI Declarative Design

Define How: Built-in Templates

Define

WhatYou Want

Automatically GenerateDataflow

11 22

Page 25: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Knowledge ModulesHot-Pluggable: Modular, Flexible, Extensible

2525

Benefits

� Tailor to existing best practices

� Ease administration work� Reduce cost of ownership

Page 26: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Trusted Business Insight with EDQ

Match & Profile & Parsing &

Enterprise Data Quality (Product & Customer Data)

Watchlist

Master Data Management

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration26

Match & Merge

Profile & Audit

Parsing & Standardize

WatchlistScreening

Data Movement & Transformation

OLTP OLAP UnstructuredLegacy

Page 27: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

CrossCross--Sell /Sell /UpUp--SellSell

Lower Operations Lower Operations & IT Costs& IT Costs

11stst Time Accurate Time Accurate DeliveriesDeliveries

Law Enforcement Law Enforcement / Privacy / Privacy

Revenue Generation& Customer Retention

Better Operations

IT AgilityRisk Management

& Compliance

Practical Applications and Value Drivers of DQ

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.27

Foundation or Rationalization for CRMFoundation or Rationalization for CRM

An addAn add--on to CRM or ERPon to CRM or ERP Marketing/ Compliance ApplicationMarketing/ Compliance Application

Fraud Prevention / Fraud Prevention / ComplianceCompliance

Differentiated Differentiated Service DeliveryService Delivery

Accelerate New Accelerate New Product IntroductionProduct Introduction

DQ/MDM as… DQ/MDM as…

Accelerate New Accelerate New IT Projects IT Projects

Foundation or Rationalization for SCM/ ERP/Fulfillment Foundation or Rationalization for SCM/ ERP/Fulfillment

Technology FoundationTechnology Foundation

Page 28: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Data Quality Maturity Progression

DQ Process/

DQ + MDM– Single Domain

DQ + MDM– Multiple Domains

DQ ‘Masters’ data in legacy applications with

Hubs persist and sync data – leveraged impact

Greater coverage, greater impact

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.28

CHAOS!

DQ Tools

DQ Process/ System

No tools, no process – the Wild West!

Project based – benefits are short lived

DQ ‘Masters’ data in legacy applications with

repeatable process

Page 29: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Enterprise Data QualityIntegrated Solution for All Data Quality Problems

Broadest DQ offering• Best of breed capabilities for both Party Data and Product

Data• Profiling, standardization, matching, case management,

governance

Most usable DQ offering

Match

Govern

Identify & merge duplicates

Monitor effectiveness & resolve problems

Com

mon A

ccess/U

I

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.29

Most usable DQ offering• Completely integrated offering – designed to work together

• Designed for business and technical users• Transparent operation and results – no black boxes

Pervasive operation for enterprise quality governance • Within legacy systems and MDM Hubs• As part of migration/system load

• On data entry/capture• As part of data movement/transfer

Profile

Standardize

Quickly understand data content

Drive conformance to standards

Identify & merge duplicates

Com

mon A

ccess/U

I

Page 30: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Putting it all Together: Oracle Data Integration

Oracle Enterprise Data Quality

Oracle SOA Suite, BPM Suite, Oracle Business Analytics

Any Data, Any Source

Zero Downtime,

High Availability

Any Data, Any Source

Zero Downtime,

High Availability

Public CloudPrivate Cloud

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration30

Big Data Connectors

Oracle GoldenGate

Oracle Data Integrator

Data Service,Federation

Oracle Cloud Application Foundation

OLTP OLAP UnstructuredLegacy

Real-time

Analytics

On Premise,

On Cloud

Real-time

Analytics

On Premise,

On Cloud

Page 31: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Oracle Engineered Systems

Any Data, Any Source

Zero Downtime,

High Availability

Any Data, Any Source

Zero Downtime,

High AvailabilityOracle Exadata Oracle Big Data Appliance

Oracle Exalytics In-Memory Machine

• Oracle BI• Oracle TimesTen• Oracle Essbase• Oracle Endeca

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration31

Real-time

Analytics

Trusted

Business Insight

Real-time

Analytics

Trusted

Business Insight

Oracle Exadata Oracle Big Data Appliance

• Oracle Database

• Exadata Software

• Advanced Analytics

• Oracle NoSQL

• Oracle R• Apache

Hadoop

Page 32: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

If you recognize your company in one of this questions…it’s time to contact us

My architecture is

heterogeneous and complex

My batch windows aren’t

long enough anymore

I’m moving to Exa... How do I

migrate?I want to

transform my architecture but

retain legacy

I need continuous

availability and maximum HA

32 Copyright © 2012, Oracle and/or its affiliates. All rights

reserved.

Oracle Data

Integration Solutions

My DB/DW is mission critical but I need to

upgrade

I can’t take much/any downtime

My reporting needs are killing my production

systems

How can I reduce risk in

my data integration

plans?

Page 33: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Architecture Principlesand Best PracticesArchitecture Principlesand Best Practices

Page 34: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Your Enterprise ArchitectureAn Architects Approach to Emerging Areas

• Addresses Solution

– Feasibility

– Time

• Addresses Goals

– Aligned to Value

– Leverages Processes

– Leverages Infrastructure

• Addresses Risks

– Skills

– Experience

– Investment

– Cost– Leverages Infrastructure

Page 35: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

The Information Architecture Spectrum

Data Realms Structure Volume Security

Storage &

Retrieval Modeling Integration Consumption

Master data

Transactions

Analytical data

Metadata

Structured Medium -

High

Database,

app, & user

access

RDBMS / SQL Pre-defined

relational or

dimensional

modeling

ETL/ELT, CDC,

Replication,

Message

BI & Statistical

Tools,

Operational

Applications

Reference data Structured Low- Platform XML / xQuery Flexible & ETL/ELT, System-based

Evaluating Economic and Architecture Tradeoffs

Reference data Structured

and Semi-

Structured

Low-

Medium

Platform

security

XML / xQuery Flexible &

Extensible

ETL/ELT,

Message

System-based

data

consumption

Documents

and Content

Unstructured High File system

based

File System /

Search

Free Form OS-level file

movement

Content Mgmt

Page 36: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

The Information Architecture Spectrum

Data Realms Structure Volume Security

Storage &

Retrieval Modeling Integration Consumption

Master data

Transactions

Analytical data

Metadata

Structured Medium -

High

Database,

app, & user

access

RDBMS / SQL Pre-defined

relational or

dimensional

modeling

ETL/ELT, CDC,

Replication,

Message

BI & Statistical

Tools,

Operational

Applications

Reference data Structured Low- Platform XML / xQuery Flexible & ETL/ELT, System-based

Evaluating Economic and Architecture Tradeoffs

Reference data Structured

and Semi-

Structured

Low-

Medium

Platform

security

XML / xQuery Flexible &

Extensible

ETL/ELT,

Message

System-based

data

consumption

Documents

and Content

Unstructured High File system

based

File System /

Search

Free Form OS-level file

movement

Content Mgmt

Big Data

- Weblogs

- Sensors

- Social Media

Structured,

Semi-

Structured,

Unstructured

High File system &

database

Distributed FS

/ noSQL

Flexible

(Key Value)

Hadoop,

MapReduce,

ETL/ELT,

Message

BI & Statistical

Tools

Page 37: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Traditional Information Architecture

ETL/ELTETL/ELTETL/ELTETL/ELT

TransactionTransactionDataData

TransactionTransactionDataData M

an

ag

em

en

tM

an

ag

em

en

tS

ecu

rity

, G

overn

an

ce

Secu

rity

, G

overn

an

ce

Man

ag

em

en

tM

an

ag

em

en

tS

ecu

rity

, G

overn

an

ce

Secu

rity

, G

overn

an

ce

AdvancedAdvancedAnalyticsAnalyticsAdvancedAdvancedAnalyticsAnalytics

VisualVisualDiscoveryDiscovery

VisualVisualDiscoveryDiscovery

DBMS

(OLTP)

DBMS

(OLTP)

Reference & Reference & Master DataMaster DataReference & Reference & Master DataMaster Data

Str

uc

ture

d Data Warehouse

Data Warehouse EPM / BI AppsEPM / BI AppsEPM / BI AppsEPM / BI Apps

Reporting & Reporting & DashboardsDashboardsReporting & Reporting & DashboardsDashboards

CDCCDCCDCCDC

RealReal--TimeTimeRealReal--TimeTime

DB RepDB RepDB RepDB Rep

Data MartsData Marts

ODSODS

Key Architecture Principles

• Treat Data as an Asset

• Quality, Timeliness, Accuracy• Information Metadata

Key EA Responsibility

• Governance

- Center of Excellence, Training- Technical Standards

• Transparency, Compliance

Page 38: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Big Data Information Architecture

Distributed Distributed

File SystemFile System

Distributed Distributed

File SystemFile System

MachineMachineGeneratedGeneratedMachineMachine

GeneratedGenerated

SocialSocialMediaMediaSocialSocialMediaMedia

Text, ImageText, ImageVideo, AudioVideo, AudioText, ImageText, Image

Video, AudioVideo, Audio

Key-ValueData StoreKey-ValueData Store

Un

str

uc

ture

d

Structured Data Structured Data CorrelationCorrelation

Structured Data Structured Data CorrelationCorrelation

VisualizationVisualization& Discovery& DiscoveryVisualizationVisualization& Discovery& Discovery

SandboxesSandboxes Statistical Statistical AnalyticsAnalyticsStatistical Statistical AnalyticsAnalytics

VisualizationVisualization& Discovery& DiscoveryVisualizationVisualization& Discovery& Discovery

Data MartsData Marts

DWDW

MapReduceMapReduce

SolutionsSolutions

MapReduceMapReduce

SolutionsSolutions

Key Architecture Principles

• Treat Data as an Asset

− Low density, variable schema• Quality, Timeliness, Accuracy

− Drive to zero latency

• Information Metadata− Privacy

Key EA Responsibility• Governance

− Center of Excellence, Training− Technical Standards

• Transparency, Compliance

Page 39: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

The Big Data Divide

ETL/ELTETL/ELTETL/ELTETL/ELT

Biz Biz TxnTxnDataData

Biz Biz TxnTxnDataData

Man

ag

em

en

tM

an

ag

em

en

tS

ecu

rity

, G

overn

an

ce

Secu

rity

, G

overn

an

ce

Man

ag

em

en

tM

an

ag

em

en

tS

ecu

rity

, G

overn

an

ce

Secu

rity

, G

overn

an

ce

AdvancedAdvancedAnalyticsAnalyticsAdvancedAdvancedAnalyticsAnalytics

VisualVisualDiscoveryDiscovery

VisualVisualDiscoveryDiscovery

Master & Master & Ref DataRef DataMaster & Master & Ref DataRef Data

Str

uc

ture

d

EPM / BI AppEPM / BI AppEPM / BI AppEPM / BI App

Reporting & Reporting & DashboardsDashboardsReporting & Reporting & DashboardsDashboards

CDCCDCCDCCDC

RealReal--TimeTimeRealReal--TimeTime

DB RepDB RepDB RepDB RepData MartsData Marts

ODSODS

Se

mi-

str

uc

ture

d

DBMS

(OLTP)

DBMS

(OLTP)Data

WarehouseData

Warehouse

Distributed Distributed

File SystemFile System

Distributed Distributed

File SystemFile System

MapReduceMapReduce

SolutionsSolutions

MapReduceMapReduce

SolutionsSolutions

Machine Machine GeneratedGeneratedMachine Machine

GeneratedGenerated

Social Social MediaMediaSocial Social MediaMedia

Text, ImageText, ImageVideo, AudioVideo, AudioText, ImageText, Image

Video, AudioVideo, Audio

Key-ValueData StoreKey-ValueData Store

Un

str

uc

ture

dS

em

i

str

uc

ture

d

Custom CodeCustom Code

SandboxesSandboxes

Page 40: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Driving to an Integrated Architecture

TransactionTransactionDataData

TransactionTransactionDataData

Ma

na

ge

me

nt

Ma

na

ge

me

nt

Se

cu

rity

, G

ove

rna

nc

eS

ec

uri

ty,

Go

ve

rna

nc

eM

an

ag

em

en

tM

an

ag

em

en

tS

ec

uri

ty,

Go

ve

rna

nc

eS

ec

uri

ty,

Go

ve

rna

nc

e

DBMS

(OLTP)

DBMS

(OLTP)

Master & Master & Ref DataRef DataMaster & Master & Ref DataRef Data

Str

uc

ture

d

Data Warehouse

Data Warehouse

Text AnalyticsText AnalyticsText AnalyticsText Analytics

Reporting &Reporting &DashboardsDashboardsReporting &Reporting &DashboardsDashboards

Machine Machine Machine Machine

Se

mi-

str

uc

ture

d

AlertingAlertingAlertingAlerting

EPM EPM BI ApplicationsBI Applications

EPM EPM BI ApplicationsBI Applications

DB DB ReplicReplicDB DB ReplicReplic

ETL/ELTETL/ELTETL/ELTETL/ELT

ChangeDCChangeDCChangeDCChangeDC

ODSODS

CaptureCapture Store/ProcessStore/Process IntegrateIntegrate OrganizeOrganize AnalyzeAnalyze GovernGovern

Ma

na

ge

me

nt

Ma

na

ge

me

nt

Se

cu

rity

, G

ove

rna

nc

eS

ec

uri

ty,

Go

ve

rna

nc

eM

an

ag

em

en

tM

an

ag

em

en

tS

ec

uri

ty,

Go

ve

rna

nc

eS

ec

uri

ty,

Go

ve

rna

nc

e

AdvancedAdvancedAnalyticsAnalyticsAdvancedAdvancedAnalyticsAnalytics

VisualVisualDiscoveryDiscovery

VisualVisualDiscoveryDiscovery

Text AnalyticsText Analyticsand Searchand Search

Text AnalyticsText Analyticsand Searchand Search

RealReal--TimeTimeRealReal--TimeTime

Machine Machine GeneratedGeneratedMachine Machine

GeneratedGenerated

Social Social MediaMediaSocial Social MediaMedia

Text, ImageText, ImageVideo, AudioVideo, AudioText, ImageText, Image

Video, AudioVideo, AudioKey-ValueData StoreKey-ValueData Store

Un

str

uc

ture

dS

em

i

str

uc

ture

d

HadoopHadoop

Cluster wCluster wMapReduceMapReduce

HadoopHadoop

Cluster wCluster wMapReduceMapReduce

InIn--Database Database AnalyticsAnalytics

InIn--Database Database AnalyticsAnalytics

MessageMessage--

BasedBased

MessageMessage--

BasedBased

ChangeDCChangeDCChangeDCChangeDC

Data MartsData Marts

Streaming(CEP Engine)

Streaming(CEP Engine)

Page 41: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Best Practices

• Align Big Data initiative with specific business goals

• Ensure centralized IT strategy for standards and governance

• Use a center of excellence to minimize training and risk

Adopt an Enterprise Architecture Approach

• Embrace data diversity and long tail data analysis• Correlate big data and structured data• Provide high performance and scalable analytical sandboxes

Expand Your Information Architecture

Page 42: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Oracle Data Integration

Page 43: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Driving to an Integrated Architecture

TransactionTransactionDataData

TransactionTransactionDataData

Ma

na

ge

me

nt

Ma

na

ge

me

nt

Se

cu

rity

, G

ove

rna

nc

eS

ec

uri

ty,

Go

ve

rna

nc

eM

an

ag

em

en

tM

an

ag

em

en

tS

ec

uri

ty,

Go

ve

rna

nc

eS

ec

uri

ty,

Go

ve

rna

nc

e

DBMS

(OLTP)

DBMS

(OLTP)

Master & Master & Ref DataRef DataMaster & Master & Ref DataRef Data

Str

uc

ture

d

Data Warehouse

Data Warehouse

Text AnalyticsText AnalyticsText AnalyticsText Analytics

Reporting &Reporting &DashboardsDashboardsReporting &Reporting &DashboardsDashboards

Machine Machine Machine Machine

Se

mi-

str

uc

ture

d

AlertingAlertingAlertingAlerting

EPM EPM BI ApplicationsBI Applications

EPM EPM BI ApplicationsBI Applications

DB ReplicationDB ReplicationDB ReplicationDB Replication

ETL/ELTETL/ELTETL/ELTETL/ELT

ODSODS

CaptureCapture Store/ProcessStore/Process IntegrateIntegrate OrganizeOrganize AnalyzeAnalyze GovernGovern

Oracle Data Oracle Data

Ma

na

ge

me

nt

Ma

na

ge

me

nt

Se

cu

rity

, G

ove

rna

nc

eS

ec

uri

ty,

Go

ve

rna

nc

eM

an

ag

em

en

tM

an

ag

em

en

tS

ec

uri

ty,

Go

ve

rna

nc

eS

ec

uri

ty,

Go

ve

rna

nc

e

AdvancedAdvancedAnalyticsAnalyticsAdvancedAdvancedAnalyticsAnalytics

VisualVisualDiscoveryDiscovery

VisualVisualDiscoveryDiscovery

Text AnalyticsText Analyticsand Searchand Search

Text AnalyticsText Analyticsand Searchand Search

RealReal--TimeTimeRealReal--TimeTime

Machine Machine GeneratedGeneratedMachine Machine

GeneratedGenerated

Social Social MediaMediaSocial Social MediaMedia

Text, ImageText, ImageVideo, AudioVideo, AudioText, ImageText, Image

Video, AudioVideo, AudioKey-ValueData StoreKey-ValueData Store

Un

str

uc

ture

dS

em

i

str

uc

ture

d

HadoopHadoop

Cluster wCluster wMapReduceMapReduce

HadoopHadoop

Cluster wCluster wMapReduceMapReduce

InIn--Database Database AnalyticsAnalytics

InIn--Database Database AnalyticsAnalytics

CDCCDCCDCCDC

Data MartsData Marts

Streaming(CEP Engine)

Streaming(CEP Engine)

MessageMessage--

BasedBased

MessageMessage--

BasedBased

Oracle Data

Integration

Oracle Data

Integration

Page 44: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Oracle Data IntegratorBulk Data Processing and Big Data Transformation

Any Data Application

Sources

Legacy Sources

CEP, Data Services

Any Data Warehouse

Any Planning System• Optimized E-LT Architecture

• Declarative design for Lower TCO

• Extensible Knowledge Modules for

enriching functionality (i.e. big data transformation)

Relational and Non-Relational

SourcesOracle Data IntegratorOracle Data IntegratorOracle Data IntegratorOracle Data Integrator

Page 45: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Oracle GoldenGateReal-time Data Integration for Big Data

Zero Downtime Upgrade and Migration

CEP Integration,

Highly Available, Disaster Recovery

Application Sources

Legacy Sources

CEP Integration, Real-time Events

Data Synchronization across the Enterprise

Real-time BI/Data Warehousing, MDM

Relational and Non-Relational

Sources

• Log-based change data capture,

distribution, transformation, and delivery

• Bidirectional replication

• Reliable data delivery, transaction integrity

Oracle GoldenGateOracle GoldenGateOracle GoldenGateOracle GoldenGate

Page 46: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Oracle Enterprise Data QualityData Quality for Big Data

Application Sources

Legacy Sources

Cleansing, Match, Merge, Standardization

Governance, Profiling,

Relational and Non-Relational

Sources

• Complete and best-of-breed approach for both customer and product data

• Unifies business-oriented interfaces for ease-of-use, lower TCO

• Reduces risk, high match rates with low false positives

Data Preparation, Case Management

Governance, Profiling, Auditing

Oracle Enterprise Data QualityOracle Enterprise Data QualityOracle Enterprise Data QualityOracle Enterprise Data Quality

Page 47: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Oracle Data IntegrationComplete Offering for Enterprise Data Integration

Application

Legacy Sources

• Complete and best-of-breed

approach to address enterprise integration

• Maximum performance with lower cost of ownership, ease of

use, and reliability.

Oracle Enterprise Data QualityOracle Enterprise Data QualityOracle Enterprise Data QualityOracle Enterprise Data Quality

Relational and Non-Relational

Application Sources

use, and reliability.

• Certified for leading technologies to deliver fast time to value

• Oracle customers report:

– 80% lower TCO

– Five times higher performance

– 70% reduction in development costs

Oracle Data IntegratorOracle Data IntegratorOracle Data IntegratorOracle Data Integrator

Oracle GoldenGateOracle GoldenGateOracle GoldenGateOracle GoldenGate

Page 48: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

New DB/HW/OS/APP

Fully Active Distributed DB

Heterogeneous

Source Systems

ADG

ADG

ADG

Dis

aste

rR

ecovery

OGG

OGG

OGG ADG

Zero DowntimeMigrations & Upgrades

Active/ActiveHigh Availability

Query Off-Loading

Migrations&Consolidations

ODI

EDQODI

DIS Use Cases

Global Data Centers

Reporting Database

ODSData Warehouse

ADG

OGG

OGG

OGG

ODI

ODIOGG

ODIOGG

Real-Time & Batch BI

Data Distribution

SOA/EDA

ODI

BPM

BPM

CEP

EDQ

EDQ

Page 49: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Communications Finance / Banking Media Services

Data Integration in the Marketplace

Sorted by Vertical Markets (sample customers only)

49 Copyright © 2011, Oracle and/or its affiliates. All rights

reserved.

Energy/Industrial Insurance / Health Retail Other

Page 50: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Join the Oracle Community

Twitter@ORCLGoldenGate or #OracleDataIntegration

Facebookfacebook.com/OracleDataIntegration

LinkedIn

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration50

LinkedInOracle Data Integration

Oracle blogblogs.oracle.com/dataintegration

Oracle.comoracle.com/goto/dataintegration

Page 51: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration51

Page 52: 1 Copyright © 2012, Oracle and/or its affiliates. All ... · Optimized for Oracle Database 11g and Oracle Exadata environments ... Oracle Data Integrator 11 g Fastest ETL/E-LT, Simpler

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. #OracleDataIntegration52