5Copyright © 2009, Oracle. All rights reserved.
Defining ETL Mappings for Staging Data
Copyright © 2009, Oracle. All rights reserved.5 - 2
Objectives
After completing this lesson, you should be able to do the following:
• Explain the purpose of a staging area
• Explore the Mapping Editor interface
• Define simple mappings between source objects and staging table columns
• Use the different mapping operators such as the External Table, Filter, Expression, and the Data Generator operator
Copyright © 2009, Oracle. All rights reserved.5 - 3
Lesson Agenda
• Purpose of a staging area• Define OWB mappings
– Mapping Editor interface: Grouping, ungrouping, and spotlighting
• Create External Tables and use it in a mapping– Create and bind and synchronization
• Use Filter operator and the Data Generator operator
• Mapping source to target attributes– Use automapper
• Expression operator– Copy-paste operators, attributes, and groups
• Set loading type and target load ordering
• Validate and generate a Mapping
Copyright © 2009, Oracle. All rights reserved.5 - 4
Purpose of a Staging Area
• A staging area is a common platform.
• Staging tables share a common database.
• Staging tables can identify data with a point in time.
Data marts/Warehouse
Staging area
Sources
Relationalsystems
Flat files
Copyright © 2009, Oracle. All rights reserved.5 - 5
Data Staging Area
• Is the construction site for the warehouse
• Is required by many implementations
• Can be composed of operational data stores (ODS), flat files, or relational server tables
• Is frequently configured as multitier staging
Extract
Transform
Load(Transport)
Operationalsystems
Data staging area Warehouse
Copyright © 2009, Oracle. All rights reserved.5 - 7
Data Warehouse Implementation: Typical Steps
Part 1
Part 2
1. Define the source metadata.
2. Define staging area metadata.
3. Define a relational dimensional model.
4. Create process flows.
5. Deploy the mappings.
6. Execute or run the mappings.
7. View the data.
Copyright © 2009, Oracle. All rights reserved.5 - 8
Metadata to Data
1. Initialize and organize the modules.
2. Import the metadata. (This gives you only the data definition.)
3. Use the metadata to define the mapping.
4. Deploy and execute the mapping to load the data.
Copyright © 2009, Oracle. All rights reserved.5 - 9
Defining Staging Area in OWB
• Create and register an OWB user using the Security node in the Globals Navigator– Optionally create an associated location
• Create an Oracle module– Either with a new location pointing to the staging user you
registered– Or using the location created while registering the user
Setting up the staging area is no different from creating any other Oracle module pointing to a database schema. You learned about this in lesson 3.
Copyright © 2009, Oracle. All rights reserved.5 - 10
Lesson Agenda
• Purpose of a staging area• Define OWB mappings
– Mapping Editor interface: Grouping, ungrouping, and spotlighting
• Create External Tables and Use it in a mapping– Create and bind and synchronization
• Use Filter operator and the Data Generator operator
• Mapping source to target attributes– Use automapper
• Expression operator– Copy-paste operators, attributes, and groups
• Set loading type and target load ordering
• Validate and generate a Mapping
Copyright © 2009, Oracle. All rights reserved.5 - 11
OWB Mapping
• Extract, transform, and load (ETL) involves the movement and transformation of data from your sources to your targets.
• Use OWB mappings to specify which source data objects provide data to which target data objects.
• Mapping definitions reside in target modules.
Warehouse Builder supports both bottom up and top-down design in mapping.
Copyright © 2009, Oracle. All rights reserved.5 - 12
Building a Mapping
Identify the Oracle module.
Create a mapping.
Connect operators using connection lines.
Generate the mapping.
Add operators, and specify the source and target.
Deploy the mapping.
Execute the mapping.
Copyright © 2009, Oracle. All rights reserved.5 - 13
Creating a New Mapping
Mapping Editor canvas: Here you design the mapping
Code templates mappings are created under the Template Mappings node, discussed in Part 2 of the course. You do not create the mappings under this for the Part 1 practices.
Copyright © 2009, Oracle. All rights reserved.5 - 14
Mapping Editor Canvas
Existing objects can be directly dragged on the canvas.
Operators are contained in the Component Palette. You can drag an
operator from this palette to the canvas.
Useful tools that help you manage the Mapping Editor canvas real estate.
Copyright © 2009, Oracle. All rights reserved.5 - 15
Tools to Manage the Mapping Editor Canvas Operators
Auto Layout
Fit In Window
Zoom Control: In, Out, Interactive
Mapping Debugger toolbar
Copyright © 2009, Oracle. All rights reserved.5 - 16
Grouping and Ungrouping
1
Copyright © 2009, Oracle. All rights reserved.5 - 17
Grouping and Ungrouping
2 3
Copyright © 2009, Oracle. All rights reserved.5 - 18
Spotlighting in the Mapping Editor
Copyright © 2009, Oracle. All rights reserved.5 - 19
Practice 5-1: Working in the Mapping Editor
In this practice, you explore the Mapping Editor interface.
Copyright © 2009, Oracle. All rights reserved.5 - 20
Lesson Agenda
• Purpose of a staging area• Define OWB mappings
– Mapping Editor interface: Grouping, ungrouping, and spotlighting
• Create External Tables and use it in a mapping– Create and bind and synchronization
• Use Filter operator and the Data Generator operator
• Mapping source to target attributes– Use automapper
• Expression operator– Copy-paste operators, attributes, and groups
• Set loading type and target load ordering
• Validate and generate a Mapping
Copyright © 2009, Oracle. All rights reserved.5 - 21
Creating Mappings for Flat File Sources
• You define two mappings:– MAP_STG_CUSTOMERS: Uses an external table as a source
to derive data from a flat file.– MAP_STG_GEOGRAPHY: Uses a flat file source directly to
extract data.
External TableFlat File
Copyright © 2009, Oracle. All rights reserved.5 - 22
What Are External Tables?
Flat file Staging area Data mart
Flat file Data martExternal table
SQL*Loader PL/SQL
PL/SQL
Avoids this extra step
Customers_txtCUST_Ext
Copyright © 2009, Oracle. All rights reserved.5 - 23
Creating an External Table
Copyright © 2009, Oracle. All rights reserved.5 - 24
Create External Table: File Selection
External tables are defined on top of the flat file structures.
Copyright © 2009, Oracle. All rights reserved.5 - 25
Create External Table: Locations
Copyright © 2009, Oracle. All rights reserved.5 - 26
Configuring External Tables
12
3
4
By default this configuration is not required. You should configure this when the data file is
different from the sampled file or you want to add many data files.
Copyright © 2009, Oracle. All rights reserved.5 - 27
Practice 5-2: Creating a Staging Module and an External Table
This practice covers the following topics:
• Creating a Staging Module, STAGING_AREA• Creating an External Table, CUST_EXT
Copyright © 2009, Oracle. All rights reserved.5 - 28
Including the External Table in a Mapping
1 2
Copyright © 2009, Oracle. All rights reserved.5 - 29
Creating a Staging Table
Copyright © 2009, Oracle. All rights reserved.5 - 30
Create and Bind
• The STG_CUSTOMERS table operator exists only on the mapping canvas and the table metadata is actually not created.
• To create the table, right-click the operator and select Create and Bind.
Select the location where you want to create the table.
Copyright © 2009, Oracle. All rights reserved.5 - 31
Adding an Attribute
If you want to remove an attribute from the table, right-click on the numbered column and select Remove.
Copyright © 2009, Oracle. All rights reserved.5 - 32
Levels of Synchronizing Changes
Mapping Design Level
Design Time
Metadata Level
Database Level
Outbound Inbound
Reverse: Re-import Re-deploy
Copyright © 2009, Oracle. All rights reserved.5 - 33
Synchronizing: Outbound and Inbound
• The LOAD_DATE attribute is only added in the Table operator and not in the table definition.
• You should synchronize the operator with the table definition. This is Outbound synchronization.
• If you add an attribute in the table definition and you want to synchronize the operator with it, perform inbound synchronization.
Copyright © 2009, Oracle. All rights reserved.5 - 34
Synchronizing: Outbound and Inbound
Copyright © 2009, Oracle. All rights reserved.5 - 35
Synchronize All in a Mapping
Copyright © 2009, Oracle. All rights reserved.5 - 36
Quiz
If you define a new unbound Table operator on the Mapping Editor, you should do the following to actually create the table definition:
a. Validate the operator
b. Generate the operator
c. Create and Bind the operator
d. Synchronize the operator
Copyright © 2009, Oracle. All rights reserved.5 - 37
Quiz
If you delete an existing table definition and then re-import a more recent version of the table, you must perform the following if the table has been used in a mapping:
a. Outbound synchronization
b. Inbound synchronization
Copyright © 2009, Oracle. All rights reserved.5 - 38
Lesson Agenda
• Purpose of a staging area• Define OWB mappings
– Mapping Editor interface: Grouping, ungrouping, and spotlighting
• Create External Tables and Use it in a mapping– Create and bind and synchronization
• Use Filter operator and the Data Generator operator
• Mapping source to target attributes– Use automapper
• Expression operator– Copy-paste operators, attributes, and groups
• Set loading type and target load ordering
• Validate and generate a Mapping
Copyright © 2009, Oracle. All rights reserved.5 - 39
Using a Filter Operator
Filter operator is used to conditionally extract rows.
Set the condition in the Filter Condition property in the Property Inspector panel. Click the ellipses (…) to start the Expression Builder.
Copyright © 2009, Oracle. All rights reserved.5 - 40
Using the Expression Builder to Definea Filter Condition
Copyright © 2009, Oracle. All rights reserved.5 - 41
Using a Data Generator
• Use a Data Generator operator to introduce a sequence, record number, or system date into a mapping.
MAP_STG_GEOGRAPHY: In this mapping, a Flat File operator is used as a source.
Copyright © 2009, Oracle. All rights reserved.5 - 42
Lesson Agenda
• Purpose of a staging area• Define OWB mappings
– Mapping Editor interface: Grouping, ungrouping, and spotlighting
• Create External Tables and Use it in a mapping– Create and bind and synchronization
• Use Filter operator and the Data Generator operator
• Mapping source to target attributes– Use automapper
• Expression operator– Copy-paste operators, attributes, and groups
• Set loading type and target load ordering
• Validate and generate a Mapping
Copyright © 2009, Oracle. All rights reserved.5 - 43
Mapping Source Columns to the Target
Individually mapping
the attributes
Automatically mapping
groups of attributes
Copyright © 2009, Oracle. All rights reserved.5 - 44
Using the Automapper in the Mapping Editor
You can map between two operators without expanding them.
When you connect attribute groups, use this
section
When you connect individual source to target attributes. It is keyboard enabled: enter * to get
a list of all the attributes in a drop-down list
Copyright © 2009, Oracle. All rights reserved.5 - 45
Lesson Agenda
• Purpose of a staging area• Define OWB mappings
– Mapping Editor interface: Grouping, ungrouping, and spotlighting
• Create External Tables and Use it in a mapping– Create and bind and synchronization
• Use Filter operator and the Data Generator operator
• Mapping source to target attributes– Use automapper
• Expression operator– Copy-paste operators, attributes, and groups
• Set loading type and target load ordering
• Validate and generate a Mapping
Copyright © 2009, Oracle. All rights reserved.5 - 46
Expression Operator
Expression operator is used to compute and transform a column value.
Edit the expression inline
Copyright © 2009, Oracle. All rights reserved.5 - 47
You can copy-paste or cut-paste the following across mappings:
• Operators
• Attributes
• Operator groups
Copying Operators Across Mappings
Copyright © 2009, Oracle. All rights reserved.5 - 48
Expression Editing and Copying
Copy the target attribute and paste it into the OUTGRP1 of the Expression operator to quickly define an output attribute with the same data type and properties.
Copyright © 2009, Oracle. All rights reserved.5 - 49
Quiz
Using the automapper, you can enter * to get a list of all the attributes in a drop-down list:
a. True
b. False
Copyright © 2009, Oracle. All rights reserved.5 - 50
Lesson Agenda
• Purpose of a staging area• Define OWB mappings
– Mapping Editor interface: Grouping, ungrouping, and spotlighting
• Create External Tables and use it in a mapping– Create and bind and synchronization
• Use Filter operator and the Data Generator operator
• Mapping source to target attributes– Use automapper
• Expression operator– Copy-paste operators, attributes, and groups
• Set loading type and target load ordering
• Validate and generate a Mapping
Copyright © 2009, Oracle. All rights reserved.5 - 51
Setting Loading Type
Copyright © 2009, Oracle. All rights reserved.5 - 52
Target Load Ordering
• Allows loading related targets in a single mapping in order
• Determines target load order based on foreign keys first, to load master before detail
Copyright © 2009, Oracle. All rights reserved.5 - 53
Overriding the Default Load Order
You can change the order usingthe up and down arrow buttons.
Click the mapping canvas to see Target Load Order in the Property inspector panel.
Copyright © 2009, Oracle. All rights reserved.5 - 54
Lesson Agenda
• Purpose of a staging area• Define OWB mappings
– Mapping Editor interface: Grouping, ungrouping, and spotlighting
• Create External Tables and Use it in a mapping– Create and bind and synchronization
• Use Filter operator and the Data Generator operator
• Mapping source to target attributes– Use automapper
• Expression operator– Copy-paste operators, attributes, and groups
• Set loading type and target load ordering
• Validate and generate a Mapping
Copyright © 2009, Oracle. All rights reserved.5 - 55
Validating the Mapping
Click the Validate tool to verify the metadata definitions and configuration parameters in a mapping before generating or deploying the mapping
Validation Warning
Validation Success
Copyright © 2009, Oracle. All rights reserved.5 - 56
Generating the Mapping
Click the Generate tool to generate the PL/SQL code for the mapping. Note that this code is generated only for review purposes.
Copyright © 2009, Oracle. All rights reserved.5 - 57
Quiz
If there are multiple targets in a mapping, you can define an order for loading the targets by using the following mapping property:
a. Loading Type
b. Target load order
Copyright © 2009, Oracle. All rights reserved.5 - 58
Summary
In this lesson, you should have learned how to:
• Explain the purpose of a staging area
• Explore the Mapping Editor interface
• Define simple mappings between source objects and staging table columns
• Use the different mapping operators such as the External Table, Filter, and Data Generator operator
Copyright © 2009, Oracle. All rights reserved.5 - 59
Practice 5-3 Overview: Defining ETL Mappings for Staging Data
This practice covers the following topics:
• Creating a simple mapping that loads the data from the FILE_GEOGRAPHY_MULTI_CSV file into two staging tables; each table stores the records of a type COUNTRY_REC or REGION_REC.
• Creating a mapping to load data from the CUST_EXT source external table to the STG_CUSTOMERS staging table. The table contains the records of customers in the UK.