incremental aggregation
TRANSCRIPT
11/5/13 Incremental Aggregation in Informatica
www.folkstalk.com/2012/02/incremental-aggregation-in-informatica.html 1/6
Home Data Warehouse Informatica Informatica Scenarios Oracle Unix Hadoop Forum Subscribe
SoftLayer® Official SiteSoftLayer.com/Asia-Pacific
More Custom Hosting From A Trusted Source. 24x7 Support. Chat Now.
INCREMENTAL AGGREGATION IN INFORMATICA
Incremental Aggregation is the process of capturing the changes in the source and
calculating the aggregations in a session. This process makes the integration service to
update the target incrementally and avoids the process of calculating the aggregations on
the entire source. Consider the below sales table as an example and see how the
incremental aggregation works.
Source:
YEAR PRICE
----------
2010 100
2010 200
2010 300
2011 500
2011 600
2012 700
For simplicity, I have used only the year and price columns of sales table. We need to do
aggregation and find the total price in each year.
Search... Search
Know the User Shell - Unix / Linux
View Phpinfo from Command Line
Unix Sed Command to Delete Lines in File - 15Examples
UNIX RECENT POSTS
11/5/13 Incremental Aggregation in Informatica
www.folkstalk.com/2012/02/incremental-aggregation-in-informatica.html 2/6
When you run the session for the first time using the incremental aggregation, then
integration service process the entire source and stores the data in two file, index and
data file. The integration service creates the files in the cache directory specified in the
aggregator transformation properties.
After the aggregation, the target table will have the below data.
Target:
YEAR PRICE
----------
2010 600
2011 1100
2012 700
Now assume that the next day few more rows are added into the source table.
Source:
YEAR PRICE
----------
2010 100
2010 200
2010 300
2011 500
2011 600
2012 700
2010 400
2011 100
2012 200
2013 800
More Unix Tutorials
How to recover lost root user password in Unix /linux
Sort Files By Size In Unix / Linux ls Command
More Informatica Articles
Session Logs Information - Informatica
Load all records except last N - Informatica
Cumulative Sum Calculation in Informatica
Informatica PowerCenter Repository - Global andLocal
Different Types of Tracing Levels In Informatica
INFORMATICA RECENT POSTS
Cursor For Loop Example in Oracle PlSql
Max of Product N consecutive Digits - Oracle SqlQuery
SQL Query to Group / Aggregate N Consecutive Rows
Oracle Trigger After Insert Example & Create
Oracle Trigger Before Insert Create & Example
ORACLE RECENT POSTS
11/5/13 Incremental Aggregation in Informatica
www.folkstalk.com/2012/02/incremental-aggregation-in-informatica.html 3/6
Now for the second run, you have to pass only the new data changes to the incremental
aggregation. So, the source will contain the last four records. The incremental
aggregation uses the data stored in the cache and calculates the aggregation. Once the
aggregation is done, the integration service writes the changes to the target and the
cache. The target table will contains the below data.
Target:
YEAR PRICE
----------
2010 1000
2011 1200
2012 900
2013 800
Points to remember
1. When you use incremental aggregation, first time you have to run the session with
complete source data and in the subsequent runs you have to pass only the
changes in the source data.
2. Use incremental aggregation only if the target is not going to change significantly. If
the incremental aggregation process changes more than hhalf of the data in target,
then the session perfromance many not benfit. In this case go for normal
aggregation.
Note: The integration service creates a new aggregate cache when
A new version of mapping is saved
Configure the session to reinitialize the aggregate cache
Moving or deleting the aggregate files
Decreasing the number of partitions
Configuring the mapping for incremental aggregation
Top Examples of Awk Command in Unix
Sed Command in Unix and Linux Examples
Cut Command in Unix ( Linux) Examples
Informatica Scenario Based Interview Questions withAnswers - Part 1
Informatica Scenario Based Questions - Part 2
Download Informatica PowerCenter Version 9.1Tutorials (PDF Documents)
String Functions in Hive
Date Functions in Hive
Unconnected Stored Procedure TransformationExample - Informatica
SQL Transformation in Informatica with examples
POPULAR POSTS
Follow me on Google+
11/5/13 Incremental Aggregation in Informatica
www.folkstalk.com/2012/02/incremental-aggregation-in-informatica.html 4/6
Labels: Informatica
Before enabling the incremental aggregation option, make sure that you capture the
changes in the source data. You can use lookup transformation or stored procedure
transformation to remove the data which is already processed. You can also create a
trigger on the source database and can read only the source changes in the mapping.
Replies
Reply
7 comments:
Neel 08 February, 2012 23:01
Hi,
Is incremental aggregation so simple? If we implement d idea of incremental load or CDC,
and by default aggregator has caching property...why do i need to excercise incremental
aggregation as separate option.
What is the advantage of using this over normal map. (using cdc and not using
incremental aggregation property). Please explain.
Reply
Anonymous 10 February, 2012 07:24
Normal aggregator also caches the data. However, this cache will be cleared
when the session run completes. In case of incremental aggregation the cache
will not be cleared and it is reused in the next session run.
If you want to use normal aggregation, every time you run the session you have
to pass the complete source data to calculate the aggregation. In case of
incremental aggregation, as the processed data is stored in the cache, you just
need to pass only the changes in the source. This way the data in cache and
the changes form the complete source.
Anderson Schmitt 14 February, 2012 09:27
11/5/13 Incremental Aggregation in Informatica
www.folkstalk.com/2012/02/incremental-aggregation-in-informatica.html 5/6
Replies
Reply
Replies
Reply
Thanks, this solved me a big problem!
Reply
Rahul Srivastava 22 April, 2013 22:34
how to enable incremental aggregation in mapping is incremental aggregation only work
with aggregator transformation if yes then how to enable it?
Reply
vijay bhaskar 22 April, 2013 22:38
You have to use aggregator transformation. You can find the incremental
aggregation in the session properties tab under the performance section.
kalpesh 03 July, 2013 01:49
It is said that , use incremental aggregation only if the target is not going to change
significantly OR If the incremental aggregation process changes more than hhalf of the
data in target, then the session perfromance many not benfit.
But in the given example all the target records are changed hence the increamental
aggregation may not benefit. Could anyone explain why increammental aggregation may
not benefit if target records change significantly...
Reply
vijay bhaskar 03 July, 2013 02:00
Updates are lot costlier when compared to inserts. This is why if the target
records changes significantly, then truncate the target and do aggregation on
the complete data and then load into target.
11/5/13 Incremental Aggregation in Informatica
www.folkstalk.com/2012/02/incremental-aggregation-in-informatica.html 6/6
Newer Post Older PostHome
Subscribe to: Post Comments (Atom)
Enter your comment...
Comment as: Google Account
Publish
Preview
FOLLOW US ON FACEBOOK
privacy policy. Powered by Blogger.