RI –ARCHITECTUREORACLE INDIA PVT. LTD.

Puneet Kala

Key Concepts of ODI

Execution Context Knowledge Modules Models Interfaces Packages Scenarios Procedures Variables

Variables

Defined in Designer Tab of ODI Scope: Global, Project Operations

1. Refresh 2. Declare3. Set4. Evaluate

Interfaces

What1. The destination data store for the data (the

target). This will be chosen from an ODI model.2. The data stores that supply the input data (the

sources). These too will be chosen from ODI models.

How1. The transformations that are applied to the data

during the transition from sources to target (the mappings). These are expressed in SQL.

2. The physical transfer mechanisms that are used between sources and target (the flow). This role is performed by the Knowledge Modules

Interfaces

Packages

Packages are the basic element of orchestration in ODI.

Sequence your interfaces, and define what operation to perform when a step (or interface) fails

Sequential Execution

Packages

Models

Models in ODI are used to store the metadata imported from the databases.

Created using Designer Tab in ODI

Procedures

A Procedure is a set of commands that can be executed by an agent.

Procedures should be considered only when what you need to do can't be achieved in an interface

Procedures require you to develop all your code manually, as opposed to interfaces

Scenario

Scenario is the "compiled" version of an ODI object.

It is possible to generate scenarios for packages, procedures, interfaces or variables

Load Plans

Execution of scenarios in a hierarchy of sequential and parallel steps

Scenarios, Packages, interfaces, variables and Procedures can be added to load plans.

It is an executable object that contains steps that needs to be executed.

Load Plans

Execution Context

Knowledge Modules

Knowledge Modules

LKM: MySQL To Oracle and File to Oracle

IKM: Oracle Incremental Update

Knowledge Modules

Issue the join-and-filter query to extract the MySQL data into the staging area in Oracle

Extract the data from the flat file and upload that into the staging area

Combine the data in the staging area and upload it from there into the target data schema

Final Structure

Confused?

ODI general Flow

SDE SIL PLP

SDE SIL PLP

SDERetail Analytics SDE programs are source dependent extraction programs that extract data from source system, transform data, and load data to Retail Analytics staging tables. SDE programs name have "sde" as the suffix.

SILRetail Analytics SIL programs are source independent loading programs that load data from Retail Analytics staging tables to Retail Analytics base level data mart tables. SIL programs name have "sil" as the suffix.

PLPRetail Analytics PLP programs are post loading programs that load data from Retail Analytics base level tables or Retail Analytics temporary tables, created and populated during batch cycle, to Retail Analytics data mart tables. The PLP programs include ETL maintenance and fact aggregations. PLP programs name have "plp" as the suffix.

SDE SIL PLP

Batch Process and Running Jobs

SDE, SIL & PLP each have a job associated with them which are shell process

User has two options, either execute from ODI or execute the batch.

When executing the Batch, please follow:1. Verify in C_LOAD_DATES table for any existing

Entry. 2. Verify Parameter values on C_ODI_PARAMS3. Execute the Batch.

In case of Errors look into : $MMHOME/Errors/Log

Example SDE, SIL

SDE: SDE_RetailColorDimension1. Package: SDE_RetailColorDimension2. Source Table: DIFF_IDS3. Target Table: W_RTL_PRODUCT_COLOR_DS4. Batch Name: prdclrsde.ksh

SIL: SIL_RetailProductColorDimension1. Package: SIL_RetailProductColorDimension2. Source Table: W_RTL_PRODUCT_COLOR_DS3. Target Table: W_RTL_PRODUCT_D4. Batch Name: rtlprodcolorsil.ksh

Universal Adaptor Framework

Universal Adaptor Framework

UAF

UAF

The goal of the Universal Adapter Framework (UAF) is to simplify the process of moving source dependent extracts into Retail Insights staging tables for customers

In terms of implementation, the UAF first requires pipe (‘|’) separated value (DAT file) text file extracts to be provided as inputs

Once the DAT files are in place, the UAF can be used to move that data into Retail Insights staging tables through the use of sqlldr

The control files required for sqlldr will be created automatically during the process that is written in ODI.

Error Management

Error Management

Three Main Category :-1. Data Errors2. Execution Errors3. Operational Error

Data Error

These occur when we encounter "bad" data during data integration tasks

Data type issues, ODI Constraints Keys, References, Conditions Detection of Data errors is done by

setting Flow_Control option in the Flow tab of an IKM

Errors detected move to ‘Error Hospital’ Only data check in RA! ‘Dimension

Identifier Missing’

Data Error

Error Table

All columns, definitions of the corresponding table. For example: Error occurred while Loading Customer data in Customers Table, hence E$_CUSTOMER Table.

Additional column about control, “S” Static or “F” Flow

Error Message Check Date ODI Object Info Constraint Name Constraint Type ex. FK(Reference Failure) ODI Session

Correcting Error

RECYCLE_ERROR Option in IKM to True Correcting Data in Error Tables and

Executing the Packages.

Execution Errors

Example of Execution Errors

These occur when one or more steps in an interface, procedure, package, or scenario fail to complete successfully

Database is down, Which can be handled and anticipated error, hence managed through flow control using KO and OK steps in a package

Unexpected Error during the Execution of a package, Log could be found in Operator Navigator in ODI Studio example: TO_DATA function on an XML source Data Server

Operational Errors

These arise if one or more of the underlying platform-level or infrastructure components encounter some kind of exception.

This area forms part of management and monitoring, especially locating and viewing system component (for example, ODI Agent)

Restart / Recovery

What happens when a Job fails?1. All successful interfaces in the package

commit data to the database, we can restart the package from operational navigator tab. Default behavior to restart from point of failure

2. When we restart a package again, KM takes care of changes.

1. Trunk and Load2. Incremental Update

3. KM written to create a File when successful execution and verify the existence of that file before starting again.

Load Plan Restart Options

If Load Plan is created we have three options for each step.1. Restart from new session2. Restart from failed step3. Restart from Failed Task

THANKS