5 copyright © 2009, oracle. all rights reserved. defining etl mappings for staging data

5Copyright © 2009, Oracle. All rights reserved.

Defining ETL Mappings for Staging Data

Copyright © 2009, Oracle. All rights reserved.5 - 2

Objectives

After completing this lesson, you should be able to do the following:

• Explain the purpose of a staging area

• Explore the Mapping Editor interface

• Define simple mappings between source objects and staging table columns

• Use the different mapping operators such as the External Table, Filter, Expression, and the Data Generator operator


Lesson Agenda

• Purpose of a staging area• Define OWB mappings

– Mapping Editor interface: Grouping, ungrouping, and spotlighting

• Create External Tables and use it in a mapping– Create and bind and synchronization

• Use Filter operator and the Data Generator operator

• Mapping source to target attributes– Use automapper

• Expression operator– Copy-paste operators, attributes, and groups

• Set loading type and target load ordering

• Validate and generate a Mapping


Purpose of a Staging Area

• A staging area is a common platform.

• Staging tables share a common database.

• Staging tables can identify data with a point in time.

Data marts/Warehouse

Staging area

Sources

Relationalsystems

Flat files


Data Staging Area

• Is the construction site for the warehouse

• Is required by many implementations

• Can be composed of operational data stores (ODS), flat files, or relational server tables

• Is frequently configured as multitier staging

Extract

Transform

Load(Transport)

Operationalsystems

Data staging area Warehouse


Data Warehouse Implementation: Typical Steps

Part 1

Part 2

1. Define the source metadata.

2. Define staging area metadata.

3. Define a relational dimensional model.

4. Create process flows.

5. Deploy the mappings.

6. Execute or run the mappings.

7. View the data.


Metadata to Data

1. Initialize and organize the modules.

2. Import the metadata. (This gives you only the data definition.)

3. Use the metadata to define the mapping.

4. Deploy and execute the mapping to load the data.


Defining Staging Area in OWB

• Create and register an OWB user using the Security node in the Globals Navigator– Optionally create an associated location

• Create an Oracle module– Either with a new location pointing to the staging user you

registered– Or using the location created while registering the user

Setting up the staging area is no different from creating any other Oracle module pointing to a database schema. You learned about this in lesson 3.


Lesson Agenda



• Create External Tables and Use it in a mapping– Create and bind and synchronization







OWB Mapping

• Extract, transform, and load (ETL) involves the movement and transformation of data from your sources to your targets.

• Use OWB mappings to specify which source data objects provide data to which target data objects.

• Mapping definitions reside in target modules.

Warehouse Builder supports both bottom up and top-down design in mapping.


Building a Mapping

Identify the Oracle module.

Create a mapping.

Connect operators using connection lines.

Generate the mapping.

Add operators, and specify the source and target.

Deploy the mapping.

Execute the mapping.


Creating a New Mapping

Mapping Editor canvas: Here you design the mapping

Code templates mappings are created under the Template Mappings node, discussed in Part 2 of the course. You do not create the mappings under this for the Part 1 practices.


Mapping Editor Canvas

Existing objects can be directly dragged on the canvas.

Operators are contained in the Component Palette. You can drag an

operator from this palette to the canvas.

Useful tools that help you manage the Mapping Editor canvas real estate.


Tools to Manage the Mapping Editor Canvas Operators

Auto Layout

Fit In Window

Zoom Control: In, Out, Interactive

Mapping Debugger toolbar


Grouping and Ungrouping

1


Grouping and Ungrouping

2 3


Spotlighting in the Mapping Editor


Practice 5-1: Working in the Mapping Editor

In this practice, you explore the Mapping Editor interface.


Lesson Agenda










Creating Mappings for Flat File Sources

• You define two mappings:– MAP_STG_CUSTOMERS: Uses an external table as a source

to derive data from a flat file.– MAP_STG_GEOGRAPHY: Uses a flat file source directly to

extract data.

External TableFlat File


What Are External Tables?

Flat file Staging area Data mart

Flat file Data martExternal table

SQL*Loader PL/SQL

PL/SQL

Avoids this extra step

Customers_txtCUST_Ext


Creating an External Table


Create External Table: File Selection

External tables are defined on top of the flat file structures.


Create External Table: Locations


Configuring External Tables

12

3

4

By default this configuration is not required. You should configure this when the data file is

different from the sampled file or you want to add many data files.


Practice 5-2: Creating a Staging Module and an External Table

This practice covers the following topics:

• Creating a Staging Module, STAGING_AREA• Creating an External Table, CUST_EXT


Including the External Table in a Mapping

1 2


Creating a Staging Table


Create and Bind

• The STG_CUSTOMERS table operator exists only on the mapping canvas and the table metadata is actually not created.

• To create the table, right-click the operator and select Create and Bind.

Select the location where you want to create the table.


Adding an Attribute

If you want to remove an attribute from the table, right-click on the numbered column and select Remove.


Levels of Synchronizing Changes

Mapping Design Level

Design Time

Metadata Level

Database Level

Outbound Inbound

Reverse: Re-import Re-deploy


Synchronizing: Outbound and Inbound

• The LOAD_DATE attribute is only added in the Table operator and not in the table definition.

• You should synchronize the operator with the table definition. This is Outbound synchronization.

• If you add an attribute in the table definition and you want to synchronize the operator with it, perform inbound synchronization.


Synchronizing: Outbound and Inbound


Synchronize All in a Mapping


Quiz

If you define a new unbound Table operator on the Mapping Editor, you should do the following to actually create the table definition:

a. Validate the operator

b. Generate the operator

c. Create and Bind the operator

d. Synchronize the operator


Quiz

If you delete an existing table definition and then re-import a more recent version of the table, you must perform the following if the table has been used in a mapping:

a. Outbound synchronization

b. Inbound synchronization


Lesson Agenda










Using a Filter Operator

Filter operator is used to conditionally extract rows.

Set the condition in the Filter Condition property in the Property Inspector panel. Click the ellipses (…) to start the Expression Builder.


Using the Expression Builder to Definea Filter Condition


Using a Data Generator

• Use a Data Generator operator to introduce a sequence, record number, or system date into a mapping.

MAP_STG_GEOGRAPHY: In this mapping, a Flat File operator is used as a source.


Lesson Agenda










Mapping Source Columns to the Target

Individually mapping

the attributes

Automatically mapping

groups of attributes


Using the Automapper in the Mapping Editor

You can map between two operators without expanding them.

When you connect attribute groups, use this

section

When you connect individual source to target attributes. It is keyboard enabled: enter * to get

a list of all the attributes in a drop-down list


Lesson Agenda










Expression Operator

Expression operator is used to compute and transform a column value.

Edit the expression inline


You can copy-paste or cut-paste the following across mappings:

• Operators

• Attributes

• Operator groups

Copying Operators Across Mappings


Expression Editing and Copying

Copy the target attribute and paste it into the OUTGRP1 of the Expression operator to quickly define an output attribute with the same data type and properties.


Quiz

Using the automapper, you can enter * to get a list of all the attributes in a drop-down list:

a. True

b. False


Lesson Agenda










Setting Loading Type


Target Load Ordering

• Allows loading related targets in a single mapping in order

• Determines target load order based on foreign keys first, to load master before detail


Overriding the Default Load Order

You can change the order usingthe up and down arrow buttons.

Click the mapping canvas to see Target Load Order in the Property inspector panel.


Lesson Agenda










Validating the Mapping

Click the Validate tool to verify the metadata definitions and configuration parameters in a mapping before generating or deploying the mapping

Validation Warning

Validation Success


Generating the Mapping

Click the Generate tool to generate the PL/SQL code for the mapping. Note that this code is generated only for review purposes.


Quiz

If there are multiple targets in a mapping, you can define an order for loading the targets by using the following mapping property:

a. Loading Type

b. Target load order


Summary

In this lesson, you should have learned how to:

• Explain the purpose of a staging area

• Explore the Mapping Editor interface

• Define simple mappings between source objects and staging table columns

• Use the different mapping operators such as the External Table, Filter, and Data Generator operator


Practice 5-3 Overview: Defining ETL Mappings for Staging Data

This practice covers the following topics:

• Creating a simple mapping that loads the data from the FILE_GEOGRAPHY_MULTI_CSV file into two staging tables; each table stores the records of a type COUNTRY_REC or REGION_REC.

• Creating a mapping to load data from the CUST_EXT source external table to the STG_CUSTOMERS staging table. The table contains the records of customers in the UK.

5 copyright © 2009, oracle. all rights reserved. defining etl mappings for staging data

Documents

staging data slide

staging tables

staging area metadata

mapping slide

staging user

data definition

oracle module

staging table columns