incremental aggregation

Download Incremental Aggregation

Post on 28-Nov-2015

98 views

Category:

Documents

2 download

Embed Size (px)

TRANSCRIPT

  • 11/5/13 Incremental Aggregation in Informatica

    www.folkstalk.com/2012/02/incremental-aggregation-in-informatica.html 1/6

    Home Data Warehouse Informatica Informatica Scenarios Oracle Unix Hadoop Forum Subscribe

    SoftLayer Official SiteSoftLayer.com/Asia-Pacific

    More Custom Hosting From A Trusted Source. 24x7 Support. Chat Now.

    INCREMENTAL AGGREGATION IN INFORMATICA

    Incremental Aggregation is the process of capturing the changes in the source and

    calculating the aggregations in a session. This process makes the integration service to

    update the target incrementally and avoids the process of calculating the aggregations on

    the entire source. Consider the below sales table as an example and see how the

    incremental aggregation works.

    Source:

    YEAR PRICE

    ----------

    2010 100

    2010 200

    2010 300

    2011 500

    2011 600

    2012 700

    For simplicity, I have used only the year and price columns of sales table. We need to do

    aggregation and find the total price in each year.

    Search... Search

    Know the User Shell - Unix / Linux

    View Phpinfo from Command Line

    Unix Sed Command to Delete Lines in File - 15Examples

    UNIX RECENT POSTS

  • 11/5/13 Incremental Aggregation in Informatica

    www.folkstalk.com/2012/02/incremental-aggregation-in-informatica.html 2/6

    When you run the session for the first time using the incremental aggregation, then

    integration service process the entire source and stores the data in two file, index and

    data file. The integration service creates the files in the cache directory specified in the

    aggregator transformation properties.

    After the aggregation, the target table will have the below data.

    Target:

    YEAR PRICE

    ----------

    2010 600

    2011 1100

    2012 700

    Now assume that the next day few more rows are added into the source table.

    Source:

    YEAR PRICE

    ----------

    2010 100

    2010 200

    2010 300

    2011 500

    2011 600

    2012 700

    2010 400

    2011 100

    2012 200

    2013 800

    More Unix Tutorials

    How to recover lost root user password in Unix /linux

    Sort Files By Size In Unix / Linux ls Command

    More Informatica Articles

    Session Logs Information - Informatica

    Load all records except last N - Informatica

    Cumulative Sum Calculation in Informatica

    Informatica PowerCenter Repository - Global andLocal

    Different Types of Tracing Levels In Informatica

    INFORMATICA RECENT POSTS

    Cursor For Loop Example in Oracle PlSql

    Max of Product N consecutive Digits - Oracle SqlQuery

    SQL Query to Group / Aggregate N Consecutive Rows

    Oracle Trigger After Insert Example & Create

    Oracle Trigger Before Insert Create & Example

    ORACLE RECENT POSTS

  • 11/5/13 Incremental Aggregation in Informatica

    www.folkstalk.com/2012/02/incremental-aggregation-in-informatica.html 3/6

    Now for the second run, you have to pass only the new data changes to the incremental

    aggregation. So, the source will contain the last four records. The incremental

    aggregation uses the data stored in the cache and calculates the aggregation. Once the

    aggregation is done, the integration service writes the changes to the target and the

    cache. The target table will contains the below data.

    Target:

    YEAR PRICE

    ----------

    2010 1000

    2011 1200

    2012 900

    2013 800

    Points to remember

    1. When you use incremental aggregation, first time you have to run the session with

    complete source data and in the subsequent runs you have to pass only the

    changes in the source data.

    2. Use incremental aggregation only if the target is not going to change significantly. If

    the incremental aggregation process changes more than hhalf of the data in target,

    then the session perfromance many not benfit. In this case go for normal

    aggregation.

    Note: The integration service creates a new aggregate cache when

    A new version of mapping is saved

    Configure the session to reinitialize the aggregate cache

    Moving or deleting the aggregate files

    Decreasing the number of partitions

    Configuring the mapping for incremental aggregation

    Top Examples of Awk Command in Unix

    Sed Command in Unix and Linux Examples

    Cut Command in Unix ( Linux) Examples

    Informatica Scenario Based Interview Questions withAnswers - Part 1

    Informatica Scenario Based Questions - Part 2

    Download Informatica PowerCenter Version 9.1Tutorials (PDF Documents)

    String Functions in Hive

    Date Functions in Hive

    Unconnected Stored Procedure TransformationExample - Informatica

    SQL Transformation in Informatica with examples

    POPULAR POSTS

    Follow me on Google+

  • 11/5/13 Incremental Aggregation in Informatica

    www.folkstalk.com/2012/02/incremental-aggregation-in-informatica.html 4/6

    Labels: Informatica

    Before enabling the incremental aggregation option, make sure that you capture the

    changes in the source data. You can use lookup transformation or stored procedure

    transformation to remove the data which is already processed. You can also create a

    trigger on the source database and can read only the source changes in the mapping.

    Replies

    Reply

    7 comments:

    Neel 08 February, 2012 23:01

    Hi,

    Is incremental aggregation so simple? If we implement d idea of incremental load or CDC,

    and by default aggregator has caching property...why do i need to excercise incremental

    aggregation as separate option.

    What is the advantage of using this over normal map. (using cdc and not using

    incremental aggregation property). Please explain.

    Reply

    Anonymous 10 February, 2012 07:24

    Normal aggregator also caches the data. However, this cache will be cleared

    when the session run completes. In case of incremental aggregation the cache

    will not be cleared and it is reused in the next session run.

    If you want to use normal aggregation, every time you run the session you have

    to pass the complete source data to calculate the aggregation. In case of

    incremental aggregation, as the processed data is stored in the cache, you just

    need to pass only the changes in the source. This way the data in cache and

    the changes form the complete source.

    Anderson Schmitt 14 February, 2012 09:27

  • 11/5/13 Incremental Aggregation in Informatica

    www.folkstalk.com/2012/02/incremental-aggregation-in-informatica.html 5/6

    Replies

    Reply

    Replies

    Reply

    Thanks, this solved me a big problem!

    Reply

    Rahul Srivastava 22 April, 2013 22:34

    how to enable incremental aggregation in mapping is incremental aggregation only work

    with aggregator transformation if yes then how to enable it?

    Reply

    vijay bhaskar 22 April, 2013 22:38

    You have to use aggregator transformation. You can find the incremental

    aggregation in the session properties tab under the performance section.

    kalpesh 03 July, 2013 01:49

    It is said that , use incremental aggregation only if the target is not going to change

    significantly OR If the incremental aggregation process changes more than hhalf of the

    data in target, then the session perfromance many not benfit.

    But in the given example all the target records are changed hence the increamental

    aggregation may not benefit. Could anyone explain why increammental aggregation may

    not benefit if target records change significantly...

    Reply

    vijay bhaskar 03 July, 2013 02:00

    Updates are lot costlier when compared to inserts. This is why if the target

    records changes significantly, then truncate the target and do aggregation on

    the complete data and then load into target.

  • 11/5/13 Incremental Aggregation in Informatica

    www.folkstalk.com/2012/02/incremental-aggregation-in-informatica.html 6/6

    Newer Post Older PostHome

    Subscribe to: Post Comments (Atom)

    Enter your comment...

    Comment as: Google Account

    Publish

    Preview

    FOLLOW US ON FACEBOOK

    privacy policy. Powered by Blogger.