informatica transformations

7/29/2019 Informatica Transformations

1/255

Introduction

#209 ,Nilagiri Block ,Adithya Enclave ,Ameerpet,HYD @8801408841, 8790998182


2/255

By the end of this course you will:

Understand how to use the major PowerCentercomponents for development

Be able to build basic ETL mappings and mapplets*

Course Objectives


e a e o crea e, run an mon or wor ows

Understand available options for loading target data

Be able to troubleshoot most problems

* A mapplet is a subset of a mapping


3/255


4/255

Informatica Products

PowerCenter ETL batch and real-time data integration

PowerAnalyzer BI reporting web-browser interface with reports,

dashboards, indicators, alerts; handles real-time

metricsSu erGlue* Centralized metadata browsin cross-enter rise


including PowerCenter, PowerAnalyzer, DBMS,

BI tools, and data modeling tools

PowerExchange Data access to mainframe, mid-size system and

complex files

PowerCenter

Connect products

Data access to transactional applications and

real-time services

* Uses PowerCenter to extract metadata and PowerAnalyzer to display reports


5/255

www.informatica.com provides information (under Services) on:

Professional Services

Education Services

my.informatica.com sign up to access:

Technical Support

Product documentation (under Tools online documentation)

Informatica Resources


Velocity Methodology (under Services)

Knowledgebase

Webzine

Mapping templates

devnet.informatica.com sign up for Informatica Developers Network

Discussion forums

Web seminars

Technical papers


6/255

Informatica offers three distinct Certification titles:

Exam A: Architecture and Administration Exam C: Advanced Administration

Informatica Professional Certification


Exam B: Mapping Design Exam D: Advanced Mapping Design

Exams A, B, C, D plus Exam E: Enablement Technologies

For more information and to register to take an exam:http://www.informatica.com/services/Education+Services/Professional+Certification/
http://www.informatica.com/services/Education+Services/Professional+Certification/http://www.informatica.com/services/Education+Services/Professional+Certification/


7/255

Extract, Transform and Load

Transaction level data

Operational Systems

MainframeRDBMS Other

Aggregated data

Decision Support

Data

Warehouse

Aggregate data


Optimized for transaction

response time

Current

Normalized or

De-normalized data

Historical data

ETL Load

Transform

Extract

C eanse ata

Consolidate data

Apply business rules

De-normalize data


8/255


9/255


10/255


11/255

1. Source Object Definitions

By the end of this section you will:

Be familiar with the Designer interface


Be able to create Source Definitions Understand Source Definition properties

Be able to use the Data Preview option


12/255

Import from:- Relational database- Flat file- XML object

Create manually

Methods of Analyzing SourcesSource Analyzer

Repository


Repository

Server

Repository Agent

TCP/IP

Native


13/255

Analyzing Relational Database Sources

Table

View

Synonym

Relational DB Source

DEF

Source Analyzer

ODBC


Repository

Server

Repository Agent

TCP/IP

DEF

Native


14/255

Analyzing Relational Database

SourcesEditing Source Definition Properties



15/255

Analyzing Flat File Sources

Mapped Drive

NFS Mount

Local Directory DEF

Fixed Width Delimited

Flat FileSource Analyzer

Re ositor


Repository

Server

Repository Agent

TCP/IP

DEF

Native


16/255

Flat File Wizard

Three-step wizard

Columns can be

renamed within

wizard

Text, Numeric and


are supported

Wizard guesses

datatype


17/255

Flat File Source Properties



18/255

Data Previewer

Preview data in Relational database sources

Flat file sources

Relational database targets Flat file tar ets


Data Preview Option is available in Source Analyzer

Warehouse Designer

Mapping Designer Mapplet Designer


19/255


20/255

Using Data Previewer in Source

AnalyzerData Preview Results


Data

Display

View up to

500 rows


21/255



22/255

2. Target Object Definitions

By the end of this section you will:

Be familiar with Target Definition types

Know the supported methods of creating


Target Definition

Understand individual Target Definition

properties


23/255

Creating Target Definitions

Methods of creating Target Definitions

Import from relational database

Import from XML object


Create manually (flat file or relationaldatabase)


24/255

Creating Target Automatically from

SourceDrag-and-drop a

Source Definition

into the

Warehouse

Designer

Workspace



25/255

Target Definition Properties



26/255



27/255

3. Mappings

By the end of this section you will be familiar with:

The Mapping Designer interface

Transformation objects and views

Source Qualifier transformation


The Expression transformation

Mapping validation


28/255

Mapping Designer

Mapping List

Transformation Toolbar


Iconized Mapping


29/255

Transformations Objects Used in This

ClassSource Qualifier: reads data from flat file & relational sourcesExpression: performs row-level calculations

Filter: drops rows conditionally

Sorter: sorts data


ggrega or: per orms aggrega e ca cu a ons

Joiner: joins heterogeneous sources

Lookup: looks up values and passes them to other objects

Update Strategy: tags rows for insert, update, delete, rejectRouter: splits rows conditionally

Sequence Generator: generates unique ID values


30/255

Other Transformation Objects

Normalizer: normalizes records from relational or VSAM sources

Rank: filters the top or bottom range of records

Union: merges data from multiple pipelines into one pipeline

Transaction Control: allows user-defined commits

Stored Procedure: calls a database stored procedure


External Procedure : calls compiled code for each row

Custom: calls compiled code for multiple rows

Midstream XML Parser: reads XML from database table or message queue

Midstream XML Generator: writes XML to database table or message queue

More Source Qualifiers: read from XML, message queues and applications


31/255

Transformation Views

A transformation has

three views:

Iconized shows the

transformation in relationto the rest of the ma in


Normal shows the flow

of data through the

transformation

Edit shows

transformation ports(= table columns)

and properties;

allows editing


32/255

Source Qualifier Transformation

Ports All input/output

Represents the source record set queried by the Server.

Mandatory in Mappings using relational or flat file sources


Convert datatypes

For relational sources:

Modify SQL statement

User Defined Join

Source Filter

Sorted portsSelect DISTINCT

Pre/Post SQL


33/255

Informatica Functions Character

Character Functions

Used to manipulate character data

CHRCODE returns the numeric value (ASCII or

Unicode) of the first character of the string

passed to this function

ASCII

CHR

CHRCODE

CONCAT

INITCAP

INSTR

LENGTHLOWER


s or ac war compa y on y. se

|| insteadLPAD

LTRIM

REPLACECHR

REPLACESTR

RPAD

RTRIM

SUBSTRUPPER


34/255

TO_CHAR (numeric)

TO_DATE

Informatica Functions Conversion

Conversion Functions


TO_DECIMAL

TO_FLOAT

TO_INTEGER

Used to convert datatypes


35/255

Informatica Functions Data Cleansing

INSTR

IS_DATE

IS_NUMBER

IS_SPACES

ISNULL

LTRIM

METAPHONE

REPLACECHR

Used to process data during data cleansing

METAPHONE and SOUNDEX create indexes basedon English pronunciation (2 different standards)


REPLACESTR

RTRIM

SOUNDEX

SUBSTR

TO_CHAR

TO_DATE

TO_DECIMALTO_FLOAT

TO_INTEGER


36/255

Informatica Functions Date

Date Functions

Used to round, truncate, or

compare dates; extract one

part of a date; or perform

ADD_TO_DATE

DATE_COMPARE

DATE_DIFF

GET_DATE_PART

LAST_DAY

ROUND (Date)

SET_DATE_PART


arithmetic on a date

To pass a string to a date

function, first use the

TO_DATE function to convert

it to an date/time datatype

_

TRUNC (Date)


37/255

Informatica Functions Numerical and

Scientific

Numerical Functions

Used to perform mathematical

operations on numeric data

ABS

CEIL

CUME

EXP

FLOOR

LN

LOG


MOVINGAVG

MOVINGSUM

POWER

ROUND

SIGN

SQRT

TRUNC

COS

COSH

SIN

SINH

TANTANH

Scientific Functions

Used to calculate

geometric values of

numeric data


38/255

Informatica Functions Special and

Test

ABORT

DECODE

ERROR

Special Functions

Used to handle specific conditions within

a session; search for certain values; test

conditional statements


Test Functions

Used to test if a lookup result is null

Used to validate data

LOOKUP, ,

IS_DATE

IS_NUMBERIS_SPACES

ISNULL


39/255

Default ValuesTwo Usages

For input and I/O ports, default values are used toreplace null values

For output ports, default values are used to handle transformationcalculation errors (not-null handling)


Default value

for the

selected port

port Validate thedefault value

expression

ISNULL function is

not required


40/255

Mapping Validation



41/255

Connection Validation

Examples of invalid connections in aMapping:

Connecting ports with incompatible datatypes

Connecting output ports to a Source


Connecting a Source to anything Except a Source

Qualifier or Normalizer transformation*

Connecting an output port to an output port or an

input port to another input port

*used for Cobol files


42/255

Mapping Validation

Mappings must:

Be valid for a Session to run

Be end-to-end complete and contain valid expressions

Pass all data flow rules


app ngs are a ways va a e w en save ; can e va a e w oubeing saved

Output Window displays reason for invalidity


43/255

Example Mapping



44/255

4. Session Task

Server instructions to run the logic of ONE specific mapping

e.g. source and target data location specifications, memory allocation,optional Mapping overrides, scheduling, processing and load instructions

Becomes a component of aWorkflow (or Worklet


If configured in the Task Developer,the Session Task is reusable(optional)


45/255

5. Workflow Structure A Workflowis set of instructions for the Informatica Server to

perform data transformation and load Combines the logic of Session Tasks, other types of Tasks and

Worklets The simplest Workflow is composed of a Start Task, a Link and

one other Task


Start

TaskSession Task

Link


46/255

6. Workflow Monitor The Workflow Monitor is the tool for monitoring Workflows and

Tasks

Choose between two views:

Gantt chart

Task view


Gantt Chart view Task view


47/255

Target Load Plan

# 209 ,Nilagiri Block ,Adithya Enclave ,Ameerpet,HYD @8801408841,8790998182


48/255

Target Load Plan

Target Load plan, informs the server the order in which to readSource Qualifiers for sending data to Targets

A target load plan decides the loading order of targets in amapping (If we use multiple source qulifiers).


,transformations, and targets linked together in a mapping.

You can set the target load order if you want to maintainreferential integrity when inserting, deleting, or updatingtables that have the primary key and foreign key constraints. Itis called as Constraint-based Loading (Only one Sourcequalifier)


49/255

-



50/255

Constraint-based Loading

PK1

FK1 PK2


FK2

To maintain referential integrity, primary keys must be

loaded before their corresponding foreign keys here in

the order Target1, Target2, Target 3


51/255

Setting Constraint-based Loading



52/255


Terminology Active transformation

Can operate on groups of data rows and/orcan change the number of rows on the data flow

Examples: Source Qualifier, Aggregator, Joiner, Sorter, Filter

Active source Active transformation that generates row


Cannot match an output row with a distinct input row

Examples: Source Qualifier, Aggregator, Joiner, Sorter

(The Filter is NOT an active source)

Active group

Group of targets in a mapping being fed by the same active source


53/255

Constraint-Based Loading

RestrictionsCannot have two active groupsExample 1

With only one Active source, rows

for Targets1, 2, and 3 will be

loaded properly and maintain

referential integrity

PK1

FK1

PK2

FK2


Example 2

With two Active sources, it is notpossible to control whether rows forTarget3 will be loaded before or afterthose for Target2

PK1

FK1

PK2

FK2


54/255

Example

Target Tables

CREATE TABLE DEPT1

(DEPTNO NUMBER(2) CONSTRAINT PK_DEPT1 PRIMARY KEY,

DNAME VARCHAR2(14) ,

LOC VARCHAR2(13) ) ;


CREATE TABLE EMP1

(EMPNO NUMBER(4) CONSTRAINT PK_EMP1 PRIMARY KEY,

ENAME VARCHAR2(10),

JOB VARCHAR2(9),

MGR NUMBER(4),

HIREDATE DATE,

SAL NUMBER(7,2),

COMM NUMBER(7,2),

DEPTNO NUMBER(2) CONSTRAINT FK_DEPTNO1 REFERENCES DEPT);


55/255


56/255

Types of Transformation

1. Passive transformation1. Operates on one row of data at a time AND

2. Cannot change the number of rows on the data flow

3. Exam le: Ex ression Looku Stored Procedure


transformation

2. Active transformation1. Can operate on groups of data rows AND/OR

2. Can change the number of rows on the data flow

3. Examples: Aggregator, Filter, Source Qualifier


57/255

Example

Tax=Sal *0.1Passive


Deptno=10

14 Rows 5 Rows

14 Rows14 Rows

Active


58/255

Types of Transformation

1. Connected transformation1. Is connected in mapping

2. All active and passive can use as connected

3. Connected transformation receives multiple input ports andcan rovides multi le out uts


2. Unconnected transformation1. Is unconnected in mapping

2. Stored and lookup transformations only can use asunconnected

3. unconnected transformation receives multiple input ports andprovides only one output


59/255

Example

Connected


Unconnected


60/255

Expression Transformation



61/255

Expression Transformation

Ports

Mixed

Variables allowed

Perform calculations using non-aggregate functions (row level)


Create expression in an

output or variable port

Usage

Perform majority of datamanipulation

Click here to invoke theExpression Editor


62/255

Expression Editor

An expression formula is a calculation or conditional statement fora specific port in a transformation

Performs calculation based on ports, functions, operators,variables, constants and return values from other transformations



63/255

Expression Validation

The Validate or OK button in the Expression Editor will:

Parse the current expression

Remote port searching (resolves references to

ports in other transformations)


Parse e au t va ues

Check spelling, correct number of arguments in functions, other

syntactical errors


64/255


65/255

#209 ,Nilagiri Block ,Adithya Enclave ,Ameerpet,HYD @8801408841, 8790998182#209 ,Nilagiri Block ,Adithya Enclave ,Ameerpet,HYD @8801408841, 8790998182


66/255


67/255

1. Reusable Tasks

Three types of reusable Tasks

Session Set of instructions toexecute a specific Mapping

Command Specific shell


comman s to run ur ng anyWorkflow

Email Sends email during theWorkflow


68/255

Reusable Tasks

Use the Task Developerto create

reusable tasks

These tasks will then appear in

the Navigator and can be

dragged and dropped into any

workflow



69/255


70/255


.


71/255

Session Task

Server instructions to run the logic of ONE specificmappinge.g. source and target data location specifications,memory allocation, optional Mapping overrides,scheduling, processing and load instructions



72/255

Creating a Session Task Created to execute the logic of a mapping (one mapping

only)

Session Tasks can be created in two ways

1. Using Task Developer (Reusable)

2. Using Workflow Designer (Workflow-specific)



73/255

Using Task Developer (Reusable)

In Work flow manager Click on

Select the Session button from the Task Toolbar


Or Select menu Tasks | Create and select Session from the drop-downmenu


74/255

Using Workflow Designer (Work Flow -Specific)

In Work flow manager Click on

Select the Session button from the Task Toolbar


Or Select menu Tasks | Create and select Session from the drop-downmenu


75/255

Session TaskProperties and

ParametersSession Task

Sessionparameter

Properties Tab



76/255

Session TaskSetting Source

PropertiesSession Task

Selectsource

instance

Mapping Tab


Set

properties

Set

connection


77/255

Session TaskSetting Target

PropertiesSession Task

Select target

instance

Mapping Tab


Note: Heterogeneous targets are

supported

Set properties

Set connection


78/255

Reusable Session Configurations



79/255

Session Configuration

Define properties to be reusable across

different sessions

Defined at folder level


Must ave one o t ese too s

open in order to access


80/255

Session Configuration (cont d)


Available from menu or

Task toolbar


81/255

Session Configuration (cont d)



82/255

Session Task Config Object


Within Session task properties, choose desired

configuration


83/255

Session Task Config Object Attributes

Attributes


may be

overridden

within the

Session task


84/255

2. Command Task

Specify one or more Unix shell or DOS commands torun during the Workflow

Runs in the Informatica Server (UNIX or Windows)environment

Command task status successful com letion or


failure) is held in the pre-defined task variable$command_task_name.STATUS

Each Command Task shell command can executebefore the Session begins or after the Informatica

Server executes a Session


85/255

Command Task

Specify one (or more) Unix shell or DOS (NT, Win2000)commands to run at a specific point in the workflow

Becomes a component of a workflow (or worklet)

If created in the Task Developer, the Command task is


If created in the Workflow Designer, the Command task isnot reusable

Commands can also be invoked under the Components

tab of a Session task to run pre- or post-session


86/255

Command Task (cont d)



87/255

Command Task (cont d)

Add Cmd


Remove Cmd


88/255

Command Task(Example)



89/255

Command Task(Example)



90/255

3. Email Task

Configure to have the Informatica Server to send email atany point in the Workflow

Becomes a component in a Workflow (or Worklet) If configured in the Task Developer, the Email Task is

reusable (optional)

Emails can also be invoked under the Com onents tab of a


Session task to run pre- or post-session


91/255

Email Task (cont d)



92/255

Email Task (Example)



93/255



94/255

Non-Reusable Tasks

Six additional Tasks are available in the WorkflowDesigner

Decision


Timer

Control

Event Wait

Event Raise


95/255

1. Decision Task Specifies a condition to be evaluated in the Workflow

Use the Decision Task in branches of a Workflow

Use decision task instead of multiple link conditions .



96/255

Decision Task

The Decision task has a predefined variable

called $Decision_task_name.condition that

represents the result of the decision


.the condition in the Decision task and sets the

predefined condition variable to True (1) or

False (0).


97/255

Decision Task(Example)



98/255

2. Assignment Task

Assigns a value to a Workflow Variable

Variables are defined in the Workflow object


Expressions Tab

General Tab


99/255

Assignment Task



100/255

3. Timer Task Waits for a specified period of time to execute the next Task

General Tab

Absolute Time

Datetime Variable

Timer Tab


Relative Time


101/255

Timer Task modes

Absolute timevWe specify the exact date and time or we can choose

a user-defined workflow variable to specify the exacttime. The next task in workflow will run as per the


.

Relative timev We instruct the Power Center Server to wait for a

specified period of time after the Timer task, the

parent workflow, or the top-level workflow starts


102/255

Timer Task(Example)



103/255

4. Control Task Stop or ABORT the Workflow

General

Tab

Properties Tab



104/255


105/255

5. Event Wait Task

Pauses processing of the pipeline until a specified event

occurs

Events can be: Pre-defined file watch

User-defined created b an Event Raise task some where in the


workflow


106/255

Event Wait Task File watch



107/255

6. Event Raise Task Represents the location of a user-defined event

The Event Raise Task triggers the user-defined event when the

Informatica Server executes the Event Raise Task


Used with the Event Wait Task

General Tab Properties Tab


108/255

Event Wait user defined Example



109/255




110/255




111/255




112/255

Filter Transformation


Filt T f ti


113/255


1. Is a condition created with followingelements

1. Port or column

2. Operator


. peran

Example Deptno = 10

1 2 3



114/255


1. Is Active2. Is connected

3. Use the Filter transformation to filter outrows in a mapping


.

5. The naming convention for a Filtertransformation is FIL_TransformationName

6. Condition to reject rows of records whichhave FIRST_NAME nullIIF(ISNULL(FIRST_NAME),FALSE,TRUE)


115/255


116/255

Example Filter



117/255



118/255

Worklets

An object representing a set or grouping of Tasks

Can contain any Task available in the Workflow Manager

Worklets expand and execute inside a Workflow

A Workflow which contains a Worklet is called the parent

Workflow


Worklets CAN be nested

Reusable Worklets create in the Worklet Designer

Non-reusable Worklets create in the Workflow Designer


119/255

Re-usable Worklet In the Worklet Designer, select Worklets | Create


Tasks in a Worklet

Worklets

Node


120/255


121/255


122/255


Debugger


123/255

Debugger


Creating a Debug Session

Debugger windows and indicators


Viewing data with the Debugger

Setting and using Breakpoints

Tips for using the Debugger

D b F t


124/255

Debugger Features

Wizard driven tool that runs a test session

View source / target data

View transformation data

Set breakpoints and evaluate expressions

Initialize variables


Manually change variable values

Data can be loaded or discarded

Debug environment can be saved for later use


125/255


126/255

Set Breakpoints

2. Choose global or specific

transformation

3. Choose to break on data

condition or error. Optionally

1. Edit breakpoint


skip rows.

4. Add breakpoint(s)

5. Add data conditions

6. Continue (to next breakpoint)


127/255

Debugger Tips

Server must be running before starting a Debug Session

When the Debugger is started, a spinning icon displays. Spinning stops

when the Debugger Server is ready

The flashing yellow/green arrow points to the current active Source

Qualifier. The solid yellow arrow points to the current Transformation


Next Instance proceeds a single step at a time; one row

moves from transformation to transformation

Step to Instance

examines one transformation at a time,following successive rows through the same transformation


128/255

Sorter Transformation




129/255


Can sort data from relational tables or flat files

Sort takes place on the Informatica Server machine

Multiple sort keys are supported


The Sorter transformation is often more efficient than asort performed on a database with an ORDER BY clause

Is useful to eliminate duplicate records so it is called asactive transformation



130/255


Sorts data from any source, at any point in a data flow

Ports

Input/Output

Define one or more sort

keys

Sort Keys


Define sort order for

each key

Example of Usage

Sort data before

Aggregator to improve

performanceSort Order

Sorter Properties


131/255

Sorter Properties

Cache size can be

ad usted. Default is 8 Mb.


Ensure sufficient memory is

available on the Informatica

Server (else Session Task will

fail)


132/255

Aggregator Transformation




133/255



Basic Aggregator functionality

Creating subtotals with the Aggregator


Aggregator properties

Using sorted data

f


134/255


Ports Mixed I/O ports allowed

Variable ports allowed

Performs aggregate calculations


Group By allowed

Create expressions in variable

and output ports

Usage

Standard aggregations

Aggregate Expressions


135/255

Aggregate Expressions

Aggregate

functions are

supported only in

the Aggregator

Transformation


Conditional Aggregate

expressions are supported: Conditional SUM format: SUM(value, condition)

Aggregator Functions


136/255

Aggregator Functions

Return summary values for non-null data inselected ports

Use only in Aggregator transformations Use in output ports only Calculate a single value (and row) for all records

in a group

AVG

COUNT

FIRST

LAST

MAX

MEDIAN


within an aggregate function

Conditional statements can be used with these

functions

STDDEV

SUM

VARIANCE

Aggregator Properties


137/255

Aggregator Properties

Sorted Input Property

Instructs the

Aggregator to

expect the data to


Set Aggregator cache

sizes for Informatica

Server machine

be sorted

Aggregating Unsorted Data


138/255

Aggregating Unsorted Data

Unsorted

data

Group By:

- store

- department

- date


No rows are released from Aggregator untilall rows are aggregated

Aggregating Sorted Data


139/255

Aggregating Sorted Data

Group By:

- store

- department

- date

Data sorted by:

- store

- department

- date


Each separate group (one row) is released as soonas the last row in the group is aggregated

Components of the Aggregator


140/255

p gg g

Transformation Aggregate cache. The Integration Service stores data in the

aggregate cache until it completes aggregate calculations. It storesgroup values in an index cache and row data in the data cache.

Aggregate expression. Enter an expression in an output port. Theexpression can include non-aggregate expressions and conditionalclauses

Grou b ort. Indicate how to create rou s. The ort can be an


input, input/output, output, or variable port. When grouping data,the Aggregator transformation outputs the last row of each groupunless otherwise specified.

Sorted input. Select this option to improve session performance. Touse sorted input, you must pass data to the Aggregatortransformation sorted by group by port, in ascending or descending

order.


141/255

Monitoring Workflows


142/255

Monitoring Workflows


The Workflow Monitor GUI interface

Monitoring views

Server monitoring modes


Filtering displayed items

Actions initiated from the Workflow Monitor

Truncating Monitor Logs

W kfl M it


143/255

Workflow Monitor

The Workflow Monitor is the tool for monitoring Workflows andTasks

Choose between two views:

Gantt chart

Task view


Gantt Chart view Task view

Monitoring Current and Past


144/255

g

Workflows The Workflow Monitor displays only workflows that have

been run

Displays real-time information from the Informatica Server

and the Repository Server about current workflow runs


Monitoring Operations


145/255

Monitoring Operations

Perform operations in the Workflow Monitor Stop, Abort, or Restart a Task, Workflow or Worklet

Resume a suspended Workflow after a failed Task is corrected

Reschedule or Unschedule a Workflow

View Session and Workflow logs Abort has a 60 second timeout


e erver as no comp e e process ng an comm ngdata during the timeout period, the threads and processesassociated with the Session are killed

Stopping a Session Task means the Server stops reading data

Monitoring in Task View


146/255

Monitoring in Task ViewStart Completion

Task Server Workflow Worklet Time Time


Status Bar

Start, Stop, Abort, Resume

Tasks,Workflows and Worklets

Filtering in Task View


147/255

Filtering in Task View

Monitoring filters can

be set using drop down

menus.

Minimizes items

displayed in Task

View


Right-click on Session to retrieve the Session

Log (from the Server to the local PC Client)

Filter Toolbar


148/255

Filter Toolbar

Select type of tasks to filter


Display recent runs

Filter tasks by specified criteria

Select servers to filter

Truncating Workflow Monitor Logs


149/255

Truncating Workflow Monitor Logs

Workflow Monitor

Repository Manager

Repository Manager s


clears the Workflow

Monitor logs


150/255

Joiner Transformation




151/255


When to join in Source Qualifier and when in Joinertransformation

Homogeneous joins


Heterogeneous joins

Joiner properties

Joiner conditions

Nested joins

When to Join in Source Qualifier


152/255

If you can perform a join on the source database, then you canconfigure it in the Source Qualifier

The SQL that the Source Qualifier generates, default orcustom, executes on the source database at runtime

Example: homogeneous join 2 database tables in same


database

When You Cannot Join in Source

Qualifier


153/255

Qualifier

If you cannotperform a join on the source database, then youcannotconfigure it in the Source Qualifier

Examples: heterogeneous joins

An Oracle table and a DB2 table

A flat file and a database table


Two flat files



154/255

Active Transformation

Ports

All input or input / output

Performs heterogeneous joins on different data flows


from master source

Examples

Join two flat files

Join two tables from

different databases

Join a flat file with arelational table

Joiner Conditions


155/255


supported

Joiner Properties


156/255

Join types:

Normal (inner)

Master outer

Detail outer


Full outer

Joiner can accept sorted data (configure the join condition to use the sort

origin ports)

Set Joiner

Caches

Nested Joins


157/255

Used to join three or more heterogeneous sources


Mid-Mapping Join (Unsorted)


158/255

The unsorted Joiner does not accept input in the following

situations:

Both input pipelines begin with the same Source Qualifier

Both input pipelines begin with the same Joiner

The sorted Joiner does not have these restrictions.



159/255

Performance Tuning


Performance Tuning


160/255

g

First step in performance tuning is to identify the performance bottleneck in thefollowing order:

Target

Source

Mapping

Session


The most common performance bottleneck occurs when the Informatica Server

writes to a target database.

Target Bottlenecks


161/255

g

Identifying A target bottleneck can be identified by configuring the session to write to a

flat file target.

Optimizing


Dropping Indexes and Key Constraints before loading.

Increasing commit intervals.

Use of Bulk Loading / External Loading.

Source Bottlenecks


162/255

Source Bottlenecks

Identifying Add a filter condition after Source qualifier to false so that no

data is processed past the filter transformation. If the time it

takes to run the new session remains about the same, then there

is a source bottleneck.


n a es mapp ng remove a e rans orma ons an e

performance is similar, then there is a source bottleneck.

Optimizing

Optimizing the Query by using hints.

Use Informatica Conditional Filters if the source system lacksindexes.

Mapping Bottlenecks


163/255

Mapping Bottlenecks Identifying

If there is no source bottleneck, add a Filter

transformation in the mapping before each target

definition. Set the filter condition to false so that no

data is loaded into the target tables. If the time it takes

to run the new session is the same as the original


session, there is a mapping bottleneck.

Optimizing

Configure for Single-Pass reading

Avoid unnecessary data type conversions.

Avoid database reject errors.

Use Shared Cache / Persistent Cache

Session Bottlenecks


164/255

Session Bottlenecks

Identifying If there is no source, Target or Mapping bottleneck, then there may be a

session bottleneck.

Use Collect Performance Details. Any value other than zero in thereadfromdisk and writetodisk counters for Aggregator, Joiner, or Ranktransformations indicate a session bottleneck. Low (0-20%)BufferInput_efficiency and BufferOutput_efficiency counter values also


.

Optimizing Increase the number of partitions.

Tune session parameters.

DTM Buffer Size (6M128M)

Buffer Block Size (4K128K)

Data (2M

24 M )/ Index (1M -12M) Cache Size

Use incremental Aggregation if possible.

Session Bottlenecks - Memory


165/255

Configure the index and data cache

memory for the Aggregator, Rank,

and Joiner transformations in the

Configuration Parameters dialog box The amount of memory you

configure depends on partitioning,


largest cache, and how much memory

cache and disk cache you want to use

System Bottlenecksd f


166/255

Identifying

If there is no source, Target, Mapping or Session bottleneck, thenthere may be a system bottleneck.

Use system tools to monitor CPU usage, memory usage, and paging.

On Windows :- Task Manager

On Unix Systems tools like sar, iostat. For E.g. sar u (%usage on

user, idle time, i/o waiting time)


Optimizing

Improve network speed.

Improve CPU performance

Check hard disks on related machines

Reduce Paging

Incremental Aggregation


167/255

First Run creates idx and dat files.

Second Run performs the following actions:

For each i/p record, the Server checks historical information in the index

file for a corresponding group, then:

If it finds a corresponding group, it performs the aggregate operation

incrementall usin the a re ate data for that rou and saves the


incremental change

If it does not find a corresponding group, it creates a new group and

saves the record data

When writing to the target Informatica Server

Updates modified aggregate groups in the target

Inserts new aggregate data

Deletes removed aggregate data

Ignores unchanged aggregate data

Saves modified aggregate data in the index and data files



168/255


You can find options for

incremental aggregation

on the Transformations

tab in the session

properties


The Server Manager

displays a warning

indicating the

Informatica Server

overwrites the existing

cache and a reminder to

clear this option after

running the session

Commit Points


169/255

A commit interval is the interval at which the Informatica Servercommits data to relational targets during a session The commit point can be a factor of the commit interval, the

commit interval type, and the size of the buffer blocks The commit interval is the number of rows you want to use as a

basis for the commit point


as a basis for the commit point Can choose between the following types of commit interval

Target-based commit Source-based commit

During a source-based commit session, the Informatica Server

commits data to the target based on the number of rows from anactive source in a single pipeline

Commit Points


170/255

During a target-based commitsession, the Informatica Server

continues to fill the writer buffer

after it reaches the commit

interval

When the buffer block is filled,


commit command

As a result, the amount of data

committed at the commit point

generally exceeds the commit

interval

Commit Points


171/255

During a source-based commit session, the Informatica Server commits

data to the target based on the number of rows from an active source in a

single pipeline

These rows are referred to as source rows

A pipeline consists of a source qualifier and all the transformations and

targets that receive data from the source qualifier


Advanced External Procedure

Source Qualifier

Normalizer

Aggregator

Joiner Rank

Mapplet, if it contains one of the above transformations

Commit Points


172/255


When the Informatica Server runs a source-based commit session, it

identifies the active source for each pipeline in the mapping

The Informatica Server generates a commit row from the active source

at every commit interval

When each target in the pipeline receives the commit row, the

Informatica Server performs the commit

Commit Points


173/255




174/255

g

PK1

FK1 PK2


FK2

To maintain referential integrity, primary keys must beloaded before their corresponding foreign keys here in

the order Target1, Target2, Target 3

Setting Constraint-based Loading


175/255

g g



Terminology


176/255

Terminology Active transformation

Can operate on groups of data rows and/orcan change the number of rows on the data flow

Examples: Source Qualifier, Aggregator, Joiner, Sorter, Filter

Active source Active transformation that generates row


Cannot match an output row with a distinct input row

Examples: Source Qualifier, Aggregator, Joiner, Sorter

(The Filter is NOT an active source)

Active group Group of targets in a mapping being fed by the same active source

Constraint-Based Loading

Restrictions


177/255

RestrictionsCannot have two active groups

Example 1

With only one Active source, rows

for Targets1, 2, and 3 will be

loaded properly and maintain

referential integrity

PK1

FK1

PK2

FK2


Example 2

With two Active sources, it is notpossible to control whether rows forTarget3 will be loaded before or afterthose for Target2

PK1

FK1

PK2

FK2


178/255



179/255





180/255

Ports All input/output

Represents the source record set queried by the Server.Mandatory in Mappings using relational or flat file sources


Convert datatypes

For relational sources:

Modify SQL statement

User Defined Join

Source Filter

Sorted ports

Select DISTINCTPre/Post SQL

Source Qualifier Properties


181/255

Is active transformation

Used to read data from tables and flat files

Useful to remove duplicates


Source Qualifier can join homogenous tables

User can modify WHERE clause

User can modify join statement

Source Qualifier Properties


182/255

User can specify ORDER BY (manually or automatically)

Pre- and post-SQL can be provided

SQL properties do not apply to flat file sources



183/255

Lookup Transformation




184/255

Can use in two ways

1. Connected

2. Un connected



185/255


.



186/255


Lookup principles

Lookup properties


Lookup techniques

Caching considerations

Persistent caches

How a Lookup Transformation Works


187/255

For each mapping row, one or more port values are looked up in adatabase table or flat file

If a match is found, one or more table values are returned to the

mapping. If no match is found, NULL is returned

Lookup value(s)

Lookup transformation


Return value(s)


188/255

Lookup Conditions


189/255


Multiple conditions are supported

Lookup Properties


190/255

Lookup

table name

Native database

Lookup condition


connection object name

Source type: Database

or Flat File

Lookup Properties cont d


191/255

Polic on multi le


match:

Use first value

Use last value

Report error

Lookup Caching


192/255

Caching can significantly impact performance

Cached

Lookup table data is cached locally on the Server

Mapping rows are looked up against the cache


Uncached

Each Mapping row needs one SQL SELECT

Rule Of Thumb: Cache if the number (and size) of records in the Lookup table is

small relative to the number of mapping rows requiring the lookup

Persistent Caches


193/255

By default, Lookup caches are not persistent; when the session completes,the cache is erased

Cache can be made persistent with the Lookup properties

When Session completes, the persistent cache is stored on the server harddisk

The next time Session runs, cached data is loaded fully or partially into RAM


and reused

A named persistent cache may be shared by different sessions

Can improve performance, but stale data may pose a problem


194/255

Lookup Caching Properties (cont d)


195/255

Set Lookup

cache sizes

Make cache

persistent


Set prefix for

persistent cache file

name

Reload

persistentcache


196/255


.

Unconnected Lookups


197/255

By the end of this section you will know:

Unconnected Lookup technique

Unconnected Lookup functionality


Unconnected Lookup


198/255

Physically unconnected from other transformations NO data flow arrows

leading to or from an unconnected Lookup Lookup data is called from the point in the Mapping that needs it Lookup function can be set within any transformation that supports

expressions

Function in the Aggregator calls the


unconnected Lookup

Unconnected Lookup Technique


199/255

ConditionRow keys

(passed to Lookup)

Use lookup lookup function within a conditional statement


Condition is evaluated for each row but Lookup function is

called only if condition satisfied

_ , . _

Lookup function

Unconnected Lookup Functionality


200/255

One Lookup port value may be returned for each Lookup

Must check a

Return port in the


Ports tab, else failsat runtime

Connected versus Unconnected

Lookups


201/255

CONNECTED LOOKUP UNCONNECTED LOOKUP

Part of the mapping data flow Separate from the mapping dataflow

Returns multiple values (bylinking output ports to anothertransformation

Returns one value- by checkingthe Return (R) port option for theout ut ort that rovides the


return value

Executed for every recordpassing through thetransformation

Only executed when the lookupfunction is called

More visible, shows where the

lookup values are used

Less visible, as the lookup is

called from an expression withinanother transformation

Default values are used Default values are ignored


202/255

Update Strategy Transformation


Business Challenge


203/255

When you design a data warehouse, you need to decide whattype of information to store in targets. As part of the targettable design, you need to determine whether to maintain allthe historic data or just the most recent changes.


Update Levels


204/255

Within a session: When you configure a session, you caninstruct the Integration Service to either treat all rows inthe same way (for example, treat all rows as inserts), oruse instructions coded into the session mapping to flag


Within a mapping: Within a mapping, you use the

Update Strategy transformation to flag rows for insert,delete, update, or reject.

Flagging Rows Within a Mapping


205/255

Operation Constant Numeric Value


_

Update DD_UPDATE 1

Delete DD_DELETE 2

Reject DD_REJECT 3

Update Strategy Transformation


206/255

Used to specify how each individual row will be used to updatetarget tables (insert, update, delete, reject)

Ports All input / output Specify the Update


Strategy ExpressionIIF or DECODE logicdetermines how tohandle the record

Example Updating Slowly

Changing Dimensions

Update Strategy Expressions


207/255

IIF ( score > 69, DD_INSERT, DD_DELETE )

Expression is evaluated for each row

Rows are tagged according to the logic of the expression


Appropriate SQL (DML) is submitted to the target database:insert, delete or update

DD_REJECT means the row will not have SQL written for it.Target will not see that row

Rejected

rows may be forwarded through Mapping

Operations for Individual Target Tables


208/255

You can set the following update strategy options:

1. Insert. Select this option to insert a row into a target table.

2. Delete. Select this option to delete a row from a table.

1. Update. You have the following options in this situation:


2. Update as Update. Update each row flagged for update if it exists inthe target table.

3. Update as Insert. Insert each row flagged for update.

3. Update else Insert. Update the row if it exists. Otherwise,insert it.

4. Truncate table. Select this option to truncate the targettable before loading data.


209/255

Normalizer Transformation


Normalizer Transformation


210/255

Is Active and connected transformation

Is useful to read data from Cobol files

Is associated with Cobol file


records

Is useful to generate GCID (Generated Column id )

Example


211/255

Year Account Q1 Q2 Q3 Q4

2012 Saving 1000 2000 3000 4000

Year Account Month Amount

2012 Saving 1 1000

2012 Saving 2 2000

2012 Saving 3 3000

2012 Saving 4 4000


Normalizer


212/255

Sequence Generator Transformation




213/255

1. Is Passive and connected

2. Use the Sequence Generator to create

unique primary key values, replace missing


,range of numbers.

3. It has two ports NEXTVAL,CURRVAL.

4. We can not modify ports

Creating Keys


214/255

1. You can create primary or foreign key values

with the Sequence Generator transformation

by connecting the NEXTVAL port to a target


.2. You can use a range of values from 1 to

9,223,372,036,854,775,807 with the smallest

interval of 1.

3. Called as surrogate keys

Replacing Missing Values


215/255

1. You Use the Sequence Generator transformation toreplace missing keys by using NEXTVAL with the IIF andISNULL functions.

Example: To replace null values in the ORDER_NO column,you create a Sequence Generator transformation with the


transformation. In the Expression transformation, drag theORDER_NO port into the transformation along with anyother necessary ports. Then create an output port,ALL_ORDERS.

In ALL_ORDERS, you can then enter the following

expression to replace null orders:IIF( ISNULL( ORDER_NO ), NEXTVAL, ORDER_NO )



216/255

Generates unique keys for any port on a row

Ports

Two predefined output ports,

NEXTVAL and CURRVA


No input ports allowed

Usage

Generate sequence numbers

Shareable across mappings

Sequence Generator Properties


217/255



218/255

Union Transformation


Union TransformationUseful to combine two similar structures


219/255

Types of Groups

1. Input (two or more)

Useful to combine two similar structures

(Number of columns and corresponding data

types must be same in two source pipelines


2. Output

Union Transformation


220/255

Is active transformation

Is useful to combine multiple similar input pipelines intoone output pipe line

Supports heterogeneous data sources


It works similar to UNION ALL function in data base

MFD


221/255



222/255

Mapping Parameters and Variables




223/255

By the end of this section you will understand:

System variables

Mapping parameters and variables




224/255

Apply to all transformations within one Mapping Represent declared values

Variables can change in value during run-time

Parameters remain constant during run-time


Provide increased development flexibility

Defined in Mapping menu



225/255

Format is $$VariableName or $$ParameterName

Can be used in pre and post-SQL

Successful completion of the session the integrationservice stores the variable value in the repository .


ar a es can e use n genera ng sequence num ers Variables can be used in incremental loading


Sample declarations


226/255

Set aggregation

User-defined

names

Set datatype


Declare Mapping Variables and Parameters

in the Designer Mappings/Mapplets menu

type

Set optional

initial value



227/255


Apply parameters or variables in formula

Functions to Set Mapping Variables

Variable value


228/255

SETMAXVARIABLE($$Variable

,value

)Sets the specified variable to the higher of the currentvalue or the specified value

SETMINVARIABLE($$Variable,value)Sets the specified variable to the lower of of the current


SETVARIABLE($$Variable,value)Sets the specified variable to the specified value

SETCOUNTVARIABLE($$Variable)Increases or decreases the specified variable by the

number of rows leaving the function(+1 for each insertedrow, -1 for each deleted row, no change for updated orrejected rows)

Parameter FilesYou can specify a parameter


229/255

p y p

file for a session in the session

editor

File extension must be .prm or

.txt


Parameter file contains folder.sessionor folder.WF:workflowname.ST:session name and initializes each parameter and

variable for that session. For example:

[Production.s_m_MonthlyCalculations]$$State=MA

$$Time=10/1/2000 00:00:00$InputFile1=sales.txt$DBConnection_target=sales$PMSessionLogFile=D:/session logs/firstrun.txt

Parameters & Variables Initialization

Priority


230/255

1. Parameter file

2. Repository value

3. Declared initial value


4. Default value

Real Time Examples


231/255

if you want use a constant value in many places ofmapping then parameter or variable can be used


Real Time Examples


232/255

Variables can be used in Incremental data load

A typical example of mapping variable in informatica, Lets

say you get a full file from your source and you are

supposed to design your mapping to load only incremental

data, precisely the new and updated records after the


previous run are to be processed.Create a mapping variable $$var_date

Filter source where create/update date > $$var_date.

Assign $$$sessionstarttime value to var_date. If the

session is successful, session start time will be saved to the

variable. In the next run the transaction generated afterthe previous run will only be considered.

Real Time Examples


233/255

To pass a value from one session to another.

Use workflow variable to assign the value of mapping

variable in informatica in any session and you can access it

in to next session through workflow variable. This can be

achieved by session properties Pre-session variable


assignment , and Pos -session on success variableassignment .

Real Time Examples


234/255

Easy deployment to production server.

your informatica code needs to deployed on production

server and as always you will not have access to

roduction server, So to de lo the code ro erl we need


to provide instructions to deployment team to performcosmetic changes in informatica code. Majority of the

cases we are not so lucky that deployment team can

perform those tasks without any support.

To overcome this problem one can parameterize all the

stuffs which are subject to change on a different server.


235/255

Mapplets


236/255


Mapplet Designer

Mapplet advantages


Mapplet rules

Active and Passive Mapplets

Mapplet Parameters and Variables

Mapplet Designer


237/255

Mapplet

Input and Output

Transformation


Mapplet Output

Transformation

app et es gner o

informatica transformations

Documents