chapter 9 working with the loop transformations · · 2014-04-26chapter 9 working with the loop...

Chapter 9 Working with the Loop Transformations

9.1 Introduction to the Loop Transformations ..................................................................9-3

9.2 Iterating a Job ...............................................................................................................9-7

Demonstration: Exploring the Summary Statistics Transformation ........................................ 9-8

Demonstration: Creating an Iterative Job ............................................................................ 9-16

Exercises .............................................................................................................................. 9-30

9.3 Iterating a Single Transformation (Self-Study) ......................................................... 9-35

Demonstration: Iterating the Create Table Transformation .................................................. 9-39

Demonstration: Using PROC METALIB ............................................................................... 9-53

Demonstration: Enable Parallel Processing ........................................................................ 9-60

Exercises .............................................................................................................................. 9-63

9.4 Solutions to Exercises ............................................................................................... 9-64

9-2 Kompaktkurs: Data Integration

Copyright © 2012, SAS Institute Inc., Cary, North Carolina, USA. ALL RIGHTS RESERVED.

Working with the LOOP Transformations 9-3


9.1 Introduction to the Loop Transformations

3

Objectives

Explain the purpose of the Loop transformations.

Explain the function of the control table.

Explain parameter mapping.

4

About Iterative Jobs

An iterative job uses the Loop and

Loop End transformations to execute

one or more processes multiple

times.

Iterate a Single Transformation

Iterate a Job



5

Loop and Loop End Transformations

The Loop and Loop End

transformations are used to

mark the beginning and end,

respectively, of an iterative

process.

They are located in

the Control group on the

Transformations tab.

6

About Parameters

The process that is iterated contains

parameters in the form of macro

variables.

The values for the parameters are

provided by the control table.

The Loop transformation iterates

once for each row in the control table.



7

The Control Table

The control table can be any type of table. The control

table can be created in a separate job, or it can be

created as part of the iterative job.

Example of control table as separate job:

Example of control table as part of the “Loop” job:

8

Job and Transformation Parameters

Parameters can be defined for transformations and for

jobs on the Parameters tab in the Properties window.



9

Parameter Mapping

The parameters in the iterated process must be mapped

to columns in the control table. This is done on the

Parameter Mapping tab in the Loop Properties window.

Columns in the

control table

Parameters in the

iterated item

Parameter Mapping tab



9.2 Iterating a Job

12

Objectives

Review a job that uses the Summary Statistics

transformation.

Create an iterative job that uses a control table,

the Loop transformations, and a parameterized job.

13

Summary Report for United States

The country manager for the United States requested

a report on customer order information.

A job was developed using the Summary Statistics

transformation.



Exploring the Summary Statistics Transformation

In this demonstration, an existing job that uses the Summary Statistics transformation to generate a report

for United States customers is executed. The job is reviewed, including the attributes and characteristics

of the Summary Statistics transformation.

1. If necessary, access SAS Data Integration Studio using Kari’s credentials.

a. Select Start All Programs SAS SAS Data Integration Studio 4.4.

b. Verify that the connection profile is Kari’s Work Repository.

c. Click OK to close the Connection Profile window and access the Log On window.

d. Type Kari as the value for the User ID field and Student1 as the value for the Password field.

e. Click OK to close the Log On window.

2. Open the job to review it.

a. Click the Folders tab.

b. If necessary, expand Data Mart Development Orion Reports Extract and Summary.

c. Double-click the DIFT Create Report for US Customer Order Information job.

d. Right-click the Extract transformation and select Properties.

e. Click the Where tab. Notice the expression that involves the value for US.

f. Click Cancel to close the Extract Properties window.



g. Right-click the Summary Statistics transformation and select Properties.

h. Click the Options tab.

i. If necessary, click the Assign columns group in the selection pane.

j. Verify that Total_Retail_Price and Quantity are to be analyzed across the values

of Customer_Gender and Customer_Age_Group.



k. Click the Basic group (under Statistics) in the selection pane.

l. Verify that the MIN, MEAN, and MAX statistics are to be calculated for the analysis variables.

m. Click the Percentiles group (under Statistics) in the selection pane.

n. Verify that the MEDIAN statistic is to be calculated for the analysis variables.



o. Click the Titles and footnotes group in the selection pane.

p. Verify that the formatted value for US (United States) is specified as part of the Heading 2 text.

q. Click the ODS options group in the selection pane.

r. Verify that the ODS result area has Use HTML as its value. Notice that the filename specified

involves a compressed formatted value for US: UnitedStatesCustomerInfo.html.

s. Click Cancel to close the Summary Statistics Properties window.



3. Run the job.

a. Return to the DIFT Create Report for US Customer Order Information job.

b. Select Actions Run to run the job.

c. Verify that all steps completed successfully.

d. If necessary, view the log for the executed job.

4. View the HTML document.

a. Open Windows Explorer.

b. Navigate to S:\Workshop\dift\reports.

c. Double-click UnitedStatesCustomerInfo.html to open the generated report.

d. If necessary, click to close the security message in the browser window.

The generated report is displayed.

e. Select File Close to close the report.

5. Return to SAS Data Integration Studio.

6. Select File Close to close the Job Editor.



16

Report for Each Country

While the report is exactly what the United States country

manager requested, other country managers are now

requesting the same report but for their respective

country.

A separate job for each country could be created (all 45

of them!). However, if a change occurred in the requested

layout of the report, the management of all these jobs

becomes unwieldy.

A better solution would be to use an iterative job.

17

Steps for an Iterative Job

You can fulfill this request in three steps:

Create the parameterized job.

Step 3

Create the iterative job.

Create the control table.

Step 1

Step 2



18

Control Table

The control table can be any table that contains rows of

data that can be fed into an iteration. The creation of this

table can be an independent job or as part of the job flow

containing the Loop transformations.

In the previous example, that value of country is included

in three places:

The unformatted value subset data in the Extract

transformation.

The formatted value is a title for the report in the

Summary Statistics transformation.

The compressed formatted value builds the name of

the HTML file created in the Summary Statistics

transformation.

19

Step 1: Create the Control Table

This table is populated with the country values. The

Extract transformation can be used to get the distinct

countries from DIFT Customer Order Information.

The code generated by the Extract transformation

resembles the following:

proc sql;create table difttgt.distinctcountries asselect distinct put(customer_country,$country.)

as countryname,compress(put(customer_country,$country.))

as ccountryname,put(customer_country,$2.) as countryvalue

from difttgt.customerorderinfo;quit;



20

Step 2: Create the Parameterized Job

The three hardcoded values for United States are

replaced with macro variable references. The macro

variables are defined as parameters to the job.

Country values are passed into this job from the control

table.

21

Step 3: Create the Iterative Job

The iterative job uses the control table as input to the

Loop transformation. The Loop transformation executes

the parameterized job once for each row in the control

table. That is, it executes once for each country.



Creating an Iterative Job

This demonstration illustrates the steps for building an iterative job. In step 1, the control table is created.

In step 2, the parameterized job is created. In step 3, the control table and the parameterized job are

combined with the Loop transformations to form an iterative job.

Step 1: Create a Control Table.


a. Select Start All Programs SAS SAS Data Integration Studio 4.4.

b. Verify that the connection profile is Kari’s Work Repository.

c. Click OK to close the Connection Profile window and access the Log On window.

d. Type Kari as the value for the User ID field and Student1 as the value for the Password field.

e. Click OK to close the Log On window.

2. Create a new folder.


b. Right-click the Data Mart Development folder and select New Folder.

c. Type Orion Loop Jobs as the value for the Name field.

d. Verify that the Location value is /Data Mart Development.

e. Click OK to close the New Folder window.

f. Click the Checkouts tab and verify that the new folder exists.

3. Define the job to load the control table.

a. On the Checkouts tab, click the Orion Loop Jobs folder.

b. Select File New Job. The New Job window appears.

c. Type DIFT Populate Control Table - Countries as the value for the Name field.

d. Verify that the location is set to /Data Mart Development/Orion Loop Jobs.

e. Click OK.

The job is created in the Checkouts tab and the Job Editor window appears.

4. Add the source table.


b. Expand Data Mart Development Orion Target Data.



c. Drag the DIFT Customer Order Information table to the Job Editor.

5. Add the Extract transformation.

a. Click the Transformations tab.

b. Expand the SQL group and locate the Extract transformation.

c. Drag the Extract transformation to the Job Editor.

d. Connect the DIFT Customer Order Information table to the Extract transformation.

6. Select File Save to save the job metadata to this point.

7. Specify the properties of the Extract transformation.

a. Right-click the Extract and select Properties.

b. Click the Mappings tab.

c. Remove all target column definitions.

1) Click the first column.

2) Hold down the SHIFT key and then click the last column.

3) Click Delete Target Columns.

d. Add three new columns to the target table.

1) Click three times to add three new columns.

2) Type CountryName as the name of the first new column.

3) Type Country Name as the column description.

4) Type put(customer_country,$country.) as the expression.

The expressions can be copied from the HelperFile.txt.

5) Verify that the Type is set to Character.

6) Type 20 as the length.

7) Type CCountryName as the name of the second new column.

8) Type Compressed Country Name as the column description.

9) Type compress(put(customer_country,$country.)) as the expression.

10) Verify that the Type field is set to Character.




12) Type CountryValue as the name of the third new column.

13) Type 2-Character Country Value as the column description.

14) Type put(customer_country,$2.) as the expression.

15) Verify that the Type fieldis set to Character.


The target table information should resemble the following:

e. Map the Customer_Country source column to each of the three target columns.

f. Click the Options tab.

g. If necessary, click the SQL Extract group.

h. Set the value of the Add the DISTINCT option to Yes.

i. Click OK to close the Extract Properties.

8. Select File Save to save the job metadata.

9. Register the work table of the Extract transformation.

a. Right-click the Extract work table and select Register Table.

b. Type DIFT Control Table - Countries as the value for the Name field.

c. Verify that the location is set to /Data Mart Development/Orion Loop Jobs.

d. Click the Physical Storage tab.



e. Type DistinctCountries as the value for the Physical name field.

f. Click next to the Library field.

1) Expand Data Mart Development Orion Target Data.

2) Click DIFT Orion Target Tables Library.

3) Click OK to close the Select a library window.

The Physical Storage tab should resemble the following:

g. Click OK to close the Register Table window.

The job flow diagram updates to the following:


11. Run the job to generate the control table.

a. Select Actions Run.

b. Verify that the job completed successfully.

c. Click the Log tab and verify that DIFTTGT.DISTINCTCOUNTRIES is created with 45 rows

and 3 columns.

d. Click the Diagram tab.

e. Right-click DIFT Control Table - Countries and select Open.



There are rows for 45 countries in the table.

f. Select File Close to close the View Data window.

g. Select File Close to close the Job Editor.

Step 2: Create a Parameterized Job

Copy the job that creates the US Customer Order Information summary report and parameterize it by

replacing the hardcoded references to the United States with macro variable references.

1. Copy the job that creates the US Customer Order Information summary report and rename the copy.


b. Expand Data Mart Development Orion Reports Extract and Summary.

c. Right-click DIFT Create Report for US Customer Order Information and select Copy.

d. Click the Checkouts tab.

e. Right-click the Orion Loop Jobs folder and select Paste.

f. Right-click DIFT Create Report for US Customer Order Information on the Checkouts tab

and select Properties.

g. Type DIFT Parameterized Job for Country Reports as the new value for the Name field.

h. Verify that the location is set to /Data Mart Development/Orion Loop Jobs.

i. Click OK to close the Properties window.

2. Parameterize the job.

a. Double-click DIFT Parameterized Job for Country Reports to open the job for editing.

b. Right-click Extract and select Properties.

c. Click the Where tab.



d. Replace "US" with "&CtryValue" in the expression. Ensure that double quotation marks are

used.

e. Click OK to close the Extract Properties window.

f. Right-click the Summary Statistics transformation and select Properties.

g. Click the Options tab.

h. Click the Titles and footnotes group in the selection pane.

i. In the Heading 2 field, update the text to (&Country Customers).

j. Click the ODS options group.

k. In the Location field, update the filename to &CCountry._CustomerInfo.html.

Be sure to include the period that delimits the parameter name.

l. Click OK to close the Summary Statistics Properties window.

3. Select File Save to save the job.



4. Define the job parameters.

a. Right-click in the background of the job and select Properties.

b. Click the Parameters tab.

c. Click New Prompt.

1) Type Country as the value for the Name field.

2) Type Country Name as the value for the Displayed text field.

3) Click the Prompt Type and Values tab.

4) Type United States as the value for the Default value field.

5) Click OK to close the New Prompt window.

d. Click New Prompt.

1) Type CCountry as the value for the Name field.

2) Type Compressed Country Name as the value for the Displayed text field.

3) Select Prompt Type and Values tab.

4) Type UnitedStates as the value for the Default value field.


e. Click New Prompt.

1) Type CtryValue as the value for the Name field.

2) Type Country Value as the value for the Displayed text field.

3) Select Prompt Type and Values tab.

4) Type US as the value for the Default value field.




The final set of three parameters is listed on the Parameters tab.

f. Click Test Prompts to validate the prompts and default values.

g. Click OK to close the Test the Prompts window.

h. Click OK to close the DIFT Parameterized Job for Country Reports Properties window.


An ampersand on the icon that represents the DIFT Parameterized Job for Country Reports job

indicates that the job is parameterized.

6. Run the job to verify that it produces the same result after it has been parameterized.

Parameterized jobs can be tested if the parameters are supplied default values.

a. Click Run in the job toolbar.

b. Verify that all steps completed successfully.



c. Click the Log tab to view the log for the executed job.

d. Scroll toward the top of the log and notice that the parameters are created and assigned the default

values.

e. Scroll toward end of the Summary Statistics code area and verify that the correct HTML filename

is being generated, as well as the correct TITLE2 text.

f. Also, verify that the data were subset correctly.

7. Select File Close to close the Job Editor window.



8. View the HTML document.

a. Open a Windows Explorer.


c. Double-click UnitedStates_CustomerInfo.html to open the generated report.

The report is identical to the one that is produced by the unparameterized job. Next, create the

iterative job to replicate this report for all 45 countries in the control table.


1. Create the initial job metadata.

a. Click the Checkouts tab.

b. Select the Orion Loop Jobs folder.

c. Select File New Job. The New Job window appears.

d. Type DIFT Loop Job for Country Reports as the value for the Name field.

e. Verify that the location is set to /Data Mart Development/Orion Loop Jobs.

f. Click OK. The Job Editor window appears.

2. Add the control table to the job editor.

a. In the Checkouts tab locate the DIFT Control Table - Countries table.

b. Drag the table to the Job Editor.



3. Add the Loop transformation.


b. Expand the Control group.

c. Drag the Loop transformation to the Job Editor.

d. Connect the DIFT Control Table - Countries table to the Loop transformation.

4. Add the parameterized job.


b. Drag the DIFT Parameterized Job for Country Reports job to the Job Editor.

5. Add the Loop End transformation.


b. In the Control group, locate the Loop End transformation.

c. Drag the Loop End to the Job Editor.

The process flow diagram should resemble the following:

6. Edit the properties of the Loop transformation.

a. Right-click the Loop transformation and select Properties.

b. Click the Parameter Mapping tab.

c. For the Country Name parameter (the Country macro variable), select CountryName as the

value for the Mapped Source Column.

d. For the Compressed Country Name parameter (the CCountry macro variable), select

CCountryName as the value for the Mapped Source Column.

e. For the Country Value parameter (the CtryValue macro variable), select CountryValue as the

value for the Mapped Source Column.



f. Click OK to close the Loop Properties window.

7. Select File Save to save diagram and job metadata to this point.

8. Run the job and view the log.

a. Click Run in the Job Editor toolbar.


c. Click the Log tab to view the log.

For every row in the control table, the Loop transformation assigns the row values to the macro

variables in the parameterized job. The parameterized job code is then regenerated and executed.

Scrolling through the log reveals the macro variable assignments for each iteration. A few are

shown here:

Each iteration of the parameterized job produces an HTML report.



9. View the HTML reports.



c. Double-click one of the reports to view it.

d. Select File Close to close the report.

10. In SAS Data Integration Studio, select File Close to close the DIFT Loop Job for Country Reports.



11. Click the Checkouts tab. It should contain the following metadata objects:

12. Check in the metadata.

a. On the Checkouts tab, click the repository name (Kari’s Work Repository).

b. Select Check Outs Check In All. The Check In Wizard appears.

c. Type Adding jobs & table for loop demonstration to iterate job as the value for the Title

field.

d. Click Next. Verify that all objects are selected.

e. Click Next. Review the information in the Summary window.

f. Click Finish. The metadata objects should no longer be on the Checkouts tab.

g. Click the Folders tab.

h. Expand Data Mart Development Orion Loop Jobs.

The Orion Loop Jobs folder should resemble the following:



Exercises

Create a job for a summary report that shows total quantities ordered, aggregated by

product group and quarter. The report should summarize data for only the year 2007 and the Clothes and

Shoes product line. Next, replicate the report for all available values of year and product line.

1. Creating a Summary Report on the DIFT Product Order Information Table

The report should resemble the following:



Use Kari’s credentials to create the desired job. Details are as follows:

Create a job named DIFT Create Report for 2007 Clothes-Shoes Products. Store the job

metadata in the /Data Mart Development/Orion Reports/Extract and Summary folder. Use the

DIFT Product Order Information table as the source. Use the Extract transformation to subset

the source data. Use the Sort transformation to order the records. Finally, use the Summary Tables

transformation to create the report.

In the Extract transformation, use the following subsetting criteria:

Product_Line = "Clothes & Shoes" and Year=2007

In the Sort transformation, sort the data by Product_Group and Quarter.

In the Summary Tables transformation, use the following specifications:

- Assign columns group: For the Select analysis columns (VAR statement) option, select

Quantity Ordered.

- Categorize Data group: For the Select columns to subgroup data (CLASS statement)

option, select Product_Group and Quarter.

- Describe TABLE to print group: In the Specify row expression field, type

Product_Group.

- Describe TABLE to print group: In the Specify column expression field, type

Quarter*Quantity=" "*Sum. This adds the sum of Quantity across Quarter to the report

and suppresses the label for this field.

- Describe TABLE to print group: In the Specify TABLE statement options field, type

rts=20. This restricts the row-title-spacing to 20 characters.

- Label a keyword group: In the Specify KEYLABEL statement field, type sum=" ". This

uses a blank as the label for the SUM statistic.

- Other options group: In the Specify other options for OPTIONS statement field, type

ls=70 nodate nonumber. (Other options is the selection pane choice that enables this

specification.) This states to restrict the line size or number of characters per line to 70, to

suppress the date and time, and to suppress the page numbers.

- Other options group: In the Summary tables procedure options field, type

format=comma8.. This applies the comma8. Format to the report.

- Titles and footnotes group: In the Heading 1 field, type Total Quantity Ordered for

Quarters of 2007.

- Titles and footnotes group: In the Heading 2 field, type Product_Line: Clothes & Shoes.

- ODS options group: In the ODS result field, type Quantities2007ClothesAndShoes.html.



- ODS options group: For the Location field, select S:\Workshop\dift\reports as the location

and add Quantities2007ClothesAndShoes.html as the filename.

Add the following pre-processing code for easier viewing of the report on the Output tab:

options formchar="|---|-|---|";

Run the job and verify that the desired report is created.

Check in the new job.

The report above for the year 2007 and the Clothes & Shoes product line must now be replicated for

each distinct combination of year and product line. This can be accomplished with an iterative job. In

the next three exercises, you create the control table, the parameterized job, and the iterative job.

2. Creating a Control Table: Years and Product Lines

The control table must contain the values of Year and of Product_Line. These values are used to

subset the data for each report, and they are included in the report headings. Also, a third column

contains values that specify a valid filename for each report. Use Kari’s credentials for this exercise.

Details are as follows:

Create a job named DIFT Populate Control Table - Years and Product Lines. Store the job

metadata in the /Data Mart Development/Orion Loop Jobs location. Use the DIFT Product

Order Information table as the source table. Use the Extract transformation to generate the

combinations of the distinct year and product line values. Load the output from the Extract into the

control table.

Set the distinct option in the Extract transformation. Use the Mapping tab in the Extract

transformation to define the three columns in the control table and the required mappings and

expression.

Name Description Type Length

Product_Line Product Line Character 25

Year Year Numeric 8

ProductLineVN Valid FileName Product_Line Character 25

Use 1:1 mappings for Product_Line and Year.

Use the following expression for ProductLineVN:

compress(tranwrd(ProdOrders.Product_Line ,"&","And"))

Name the control table DIFT Control Table - Years and Product Lines. Name the physical table

DistinctYearsProdLines and store it in the DIFT Orion Target Tables Library.



Execute the job and verify that the control table has 20 observations and 3 columns.

3. Creating a Parameterized Job

The parameterized job is a copy of the job DIFT Create Report for 2007 Clothes - Shoes Products.

Use Kari’s credentials for this exercise.


Store the job metadata object in the folder /Data Mart Development/Orion Loop Jobs.

Name the parameterized job DIFT Parameterized Job for Year-Product Line Reports.

Replace the hardcoded values on the Extract transformation’s Where tab with macro variable

references.

Replace the hardcoded values on the Summary Tables transformation’s Option tab with macro

variable references (two titles and the HTML filename).

Define appropriate parameters. Assign valid default values so that this job can be executed and

tested outside of the Loop job.

Execute the job and verify the macro variables and parameters were defined successfully.

4. Creating and Executing the Iterative Job

The iterative job uses the control table populated in Exercise 3, the Loop transformations, and the

parameterized job created in Exercise 4. Use Bruno’s credentials for this exercise.


Store the job metadata object in the folder /Data Mart Development/Orion Loop Jobs.

Name the iterative job DIFT Loop Job for Year and Product Line Reports.

Be sure to specify valid parameter mappings for the Loop transformation.

Execute the job and verify that the HTML reports are created.



9.3 Iterating a Single Transformation (Self-Study)

27

Objectives

Discuss a job that iterates a single transformation.

Use PROC METALIB to register tables automatically.

Use parallel processing to improve performance.

28

Orion Star Marketing Request

A Marketing group at Orion Star requests individual

studies on customer orders across the age groups broken

down by gender.

To facilitate these studies, it is decided to create a

separate table for each combination of age group and

gender.

One way to create this series of tables is to use the Loop

transformations.



29

Orion Star Marketing Request – Specifics

The DIFT Customer Order Information table contains the

required data for the desired tables.

Each table is a subset based on a combination of a

gender value and an age group value.

The table names use the following convention:

CustomerOrders_<gender value><age group value>

Examples: CustomerOrders_Female15To30years

CustomerOrders_Female31To45years

CustomerOrders_Male15To30years

CustomerOrders_Male31To45years

30

Job Specifics

One job creates the control table. A second job uses the

Loop transformations to iterate over the Create table

transformation. The control table provides parameter

values to the Loop, one record at a time.

Each iteration requires the following:

a value for gender (for subsetting)

a value for age group (for subsetting)

a value that is part of the table name

Therefore, the control table should contain three columns.

Example: F

Example: 15-30 years

Example: Female15To30years



31

Create Table Transformation Specifics

The Create Table transformation is configured with macro

variable references in the WHERE clauses, as well as for

part of the name of the output table. The macro variables

are defined as parameters to the Extract transformation.

32

Loop Transformation Specifics

The Loop transformation maps the parameters defined in

the Create Table transformation to columns in the control

table.

Parameters

defined to the

Create Table

transformation

Columns

defined in the

control table



33

Demo Job Flow

Iterative process using

Control table

Loop transformations

Create Table transformation

Creation of the

Control Table



Iterating the Create Table Transformation

This demonstration illustrates a job that uses the Loop transformation to iterate a Create Table

transformation. Each iteration of the Create Table transformation produces a subset of the DIFT

Customer Order Information table for a combination of gender and age group.

If necessary, access SAS Data Integration Studio using Kari’s credentials.

1. Select Start All Programs SAS SAS Data Integration Studio 4.4.

2. Verify that the connection profile is Kari’s Work Repository.

3. Click OK to close the Connection Profile window and access the Log On window.

4. Type Kari as the value for the User ID field and Student1 as the value for the Password field.

5. Click OK to close the Log On window.

Step 1: Create a Control Table

1. Create a job to populate the control table.


b. Expand Data Mart Development Orion Loop Jobs.

c. Right-click Orion Loop Jobs and select File New Job.

d. Type DIFT Populate Control Table - Genders and Age Groups as the value for the Name

field.

e. Verify that the location is set to /Data Mart Development/Orion Loop Jobs.

f. Click OK. The Job Editor window appears.



2. Turn off automatic propagation and mapping for the job.

a. Click and clear Automatically Propagate Job.

b. Click again and clear Automatically Map Job.

3. Add the source table to the process flow.




4. Add the Extract transformation to the process flow.


b. Expand the SQL folder and locate the Extract transformation.

c. Drag the Extract transformation to the Job Editor.

d. Connect the DIFT Customer Order Information table to the Extract transformation.


6. Specify the properties of the Extract transformation.

a. Right-click Extract and select Properties.

b. Click the Mappings tab.

c. Verify that there are no Target table columns (automatic propagation was disabled for this job).



d. Click to expand the Target table pane.

e. Create three new columns in the target table.

1) Click three times to add three new columns.

2) Type GenderValue as the column name of the first new column.

3) Type Gender Value as the column description.

4) Type the following expression in the Expression field:

put(customer_gender, $1.)

5) Verify that the type is Character,


7) Type AgeGroupValue as the column name of the second new column.

8) Type Age Group Value as the column description.

No expression is specified as this column is mapped from a source column.

9) Verify that the type is Character.


11) Type GdrAgeGrp as the column name of the third new column.

12) Type Gender Age Group Value as the column description

13) Type the following expression in the Expression field:

compress(put(customer_gender,$gender.)||

ifc(index(customer_age_group, "-")>0,

tranwrd(customer_age_group,"-","To"),

tranwrd(customer_age_group,"&","And")))

Some customer_age_group values have a hyphen (-) that must be changed to

the word “To” for a valid table name, and other values have an ampersand (&)

that must be changed to the word “And”.

14) Verify that the type is Character.


16) Click to collapse the Target table pane.



f. Specify appropriate mappings.

1) Map the Customer_Gender source column to both GenderValue and GdrAgeGrp.

2) Map the Customer_Age_Group source column to both AgeGroupValue and GdrAgeGrp.

h. Click the Options tab.

i. If necessary, click the SQL Extract group.

j. Set the value for Add the DISTINCT option to Yes.

g. Click OK to close the Extract Properties.


8. Register the work table.

a. Right-click the Extract work table and select Register Table.

b. Type DIFT Control Table - Genders and Age Groups as the value for the Name field.

c. Verify that the Location value is /Data Mart Development/Orion Loop Jobs.

d. Click the Physical Storage tab.

e. Type DistinctGenderAgeGroups as the value for the Physical name field.


1) Navigate to Data Mart Development Orion Target Data.

2) Select DIFT Orion Target Tables Library.

3) Click OK to close the Select a Library window.

g. Click OK to close the Register Table window.




9. Execute the job and verify the output.

a. Click Run to execute the job.


c. Right-click DIFT Control Table - Genders and Age Groups and select Open.

d. Verify that there are 3 columns and 10 rows, and that GdrAgeGrp values are distinct

combinations of gender and age group.

e. Select File Close to close the View Data window.

10. Select File Close to close the Job Editor.


1. Define the job.


b. Expand the Data Mart Development Orion Loop Jobs folder.

c. Click the Orion Loop Jobs folder.

d. Select File New Job. The New Job window appears.

e. Type DIFT Loop Job Populate Gender-Age Group Tables as the value for the Name field.

f. Verify that the location is set to /Data Mart Development/Orion Loop Jobs.

g. Click OK. The Job Editor window appears.

2. Add the control table to the process flow.





c. Drag the DIFT Control Table - Genders and Age Groups table to the Job Editor.

3. Add the Loop transformation to the process flow.


b. Expand the Control group and locate the Loop transformation.

c. Drag the Loop transformation to the Job Editor.

d. Connect the DIFT Control Table - Genders and Age Groups table to the Loop transformation.

4. Add the Create Table transformation to the process flow.


b. Expand the SQL group and locate the Create Table transformation.

c. Drag the Create Table transformation to the Job Editor.

d. Position the Create Table transformation next to the Loop transformation, but do not connect the

Create Table transformation to the Loop transformation

5. Add the source table for the Create Table transformation to the process flow.




d. Connect the DIFT Customer Order Information table to the input of the Create Table

transformation.

6. Add the Loop End transformation to the process flow.


b. Expand the Control folder and locate the Loop End transformation template.

c. Drag the Loop End to the Job Editor. The Loop End transformation connects automatically to the

Extract transformation.

There is no visual connection between the Loop transformation and the Create Table

transformation.




For each iteration over the Create Table transformation, the name of the output table must change to

reflect the subset taken from the source table. A macro variable is used to accomplish this.

8. Specify properties of the Create Table work table.

a. Right-click the work table of the Create Table transformation and select Properties.

b. Click the Physical Storage tab.

c. Type CustomerOrders_&gdragegrp as the value for the Physical name field.

d. Clear Create as view.

e. Select Redirect to a registered library as the value for the Location field.


1) Expand Data Mart Development Orion Target Data.

2) Click DIFT Orion Target Tables Library.

3) Click OK to close the Select a Library window.

g. Click OK to close the Create Table Properties window.

9. Edit the properties of the Create Table transformation.

a. Right-click the Create Table transformation and select Properties.

b. Click the Source tab.

c. Verify that the Type value is Single Table and the Table value is CustomerOrderInformation.

d. Click the Result tab.

e. Verify that all columns are mapped.

f. Select the Remove duplicate rows (DISDINCT) option.



g. Add the subsetting criteria.

1) Click the Filter and Sort tab.

2) In the Filter (upper) panel, click New row. A row is added for a WHERE condition.

3) Click in the first Operand field and select Choose Column.

4) Expand DIFT Customer Order Information and select Customer_Gender.

5) Click OK to close the Choose Columns window.

6) Keep = (equals) as the operator.

7) Type "&GdrVal" in the second Operand field.

8) Click New row to add a second row.

9) In the second row, keep AND as the Boolean.

10) Click in the first Operand field and select Choose Column.

11) Expand DIFT Customer Order Information and select Customer_Age_Group.

12) Click OK to close the Choose Columns window.

13) Keep = (equals) as the operator.

14) Type "&AgeGrpVal" in the second Operand field.



h. Click the Parameters tab.

1) Click New Prompt.

2) Type GdrVal as the value for the Name field.

3) Type Gender Value as the value for the Displayed text field.



6) Type AgeGrpVal as the value for the Name field.

7) Type Age Group Value as the value for the Displayed text field.



10) Type GdrAgeGrp as the value for the Name field.

11) Type Gender-Age Group Value as the value for the Displayed text field.


i. Click OK to close the Create Table Properties.


The control table supplies the values for the parameters defined in the Create Table transformation. The

parameters of the Create Table transformation must be mapped to columns in the control table. This is

done in the properties of the Loop transformation.



11. Edit the properties of the Loop transformation.


b. Click the Parameter Mapping tab.

c. For the Gender Value parameter, select GenderValue for the Mapped Source Column.

d. For the Age Group Value parameter, select AgeGroupValue for the Mapped Source Column.

e. For the Gender-Age Group Value parameter, select GdrAgeGrp for the

Mapped Source Column.

f. Click OK to close the Loop Properties window.


13. Run the job.

a. Right-click in background of the job and select Run.


14. Verify that the tables were created.


b. Navigate to S:\Workshop\dift\datamart.

c. Verify that the 10 data sets were created.

d. Select File Close to close Windows Explorer.

15. Review the log.



a. Return to SAS Data Integration Studio.

b. Click the Log tab in the Job Editor window.

c. Scroll to the top of the log.

d. Select Edit Find and Replace.

e. Type let gdrval and press ENTER.

The first observation from the control table has its values assigned to the macro variables

(parameters) via a series of %LET statements.

f. Scroll down in the log to find the note that the table was created.

g. Click Close to close the Find/Replace window.

h. Click the Diagram tab in the Job Editor.

16. Select File Close to close the job.



36

Where Are the Tables?

In the previous demonstration, a series of tables is

produced in physical storage. However, these tables are

not currently registered in metadata.

The Register Tables Wizard could be used to register

these new tables interactively.

An alternative is to use the METALIB procedure to

automate the registration of the new tables.

37

Introducing the METALIB Procedure

The METALIB procedure updates the metadata in the

metadata server to match the tables in a library. By

default, the procedure performs the following tasks:

creates metadata for any table in the library that is not

yet registered

updates metadata for registered tables in the library,

including columns, indexes, and keys



38

PROC METALIB

With optional statements, PROC METALIB can perform

the following additional tasks:

select or exclude specific tables from processing

specify where new metadata is stored in SAS folders

add a prefix to the name of all new metadata objects

generate a report of changes that the procedure made

to metadata

generate a report of metadata differences without

making the changes

in the generated report, include an impact analysis

for Job and Transformation objects

in the generated report, include a list of tables that

match the metadata

How PROC METALIB Works

The procedure examines the data source (the SAS library) that is referenced by the SASLibrary object.

Then the procedure examines the SAS table names in the data source and compares them to the values

of the SASTableName= attributes in the metadata. For each SAS table name in the data source, the

procedure checks the repository association list to see whether a matching table definition exists.

If a matching table definition does not exist, one is created.

If a matching table definition exists, it is updated to match the table in the data source.

If duplicate table definitions exist, only the first table definition is updated. The additional table

definitions are ignored by default. If you specify UPDATE_RULE=(DELETE), the additional table

definitions are deleted. If you specify UPDATE_RULE=(DELETE NODELDUP), the additional table

definitions are not deleted.

If a table definition exists that does not correspond to a table in the data source, it is ignored by default.

If you specify UPDATE_RULE=(DELETE), the table definition is deleted.

For more information, see topics about managing table metadata in the SAS® Intelligence Platform:

Administration Guide.

What Metadata Is Updated?

The procedure updates all table definitions that are associated with the specified SASLibrary object.

The affected metadata objects include PhysicalTable, Column, ForeignKey, Index, KeyAssociation, and

UniqueKey. For more information about these metadata objects, see their descriptions in SAS® Metadata

Model: Reference.



39

Syntax: METALIB Procedure

PROC METALIB;OMR<=>(LIBRARY="name" |

LIBRARY="/folder-pathname/name" <server-connection-arguments>);

<FOLDER <=> "/pathname";><NOEXEC;><PREFIX<=>"text";><REPORT <<=> (report-arguments)>;><UPDATE_RULE<=><DELETE> <NOADD>

<NODELDUP> <NOUPDATE>);>RUN;

Statement Purpose

PROC METALIB statement Update metadata to match the tables in a library.

OMR statement Specify data source and connection parameters for SAS Metadata Server.

FOLDER statement Specify where new metadata is stored in SAS folders.

NOEXEC statement Suppress the metadata changes from being made.

PREFIX statement Specify a text string to add to the beginning of all new metadata object

names.

REPORT statement Create a report that summarizes metadata changes.

UPDATE_RULE statement Override default update behavior.



Using PROC METALIB

This demonstration illustrates adding PROC METALIB code as a post-processing step to register the

tables created by the iterative process automatically. The registration requires that the job is executed by a

user with Create access. The report generated by PROC METALIB is reviewed.


2. If necessary, open the job DIFT Loop Job Populate Gender-Age Group Tables.

3. Add pre-processing and post-processing code to the job.

a. Right-click in the background of the Diagram tab and select Properties.

b. Click the Precode and Postcode tab.

c. Click Precode.

d. Type the following code for pre-processing of the job:

options ls=95 nodate nonumber formchar="|---|-|---|";

The Precode area should resemble the following:

This code can be found in S:\Workshop\dift\HelperFile.txt. The FORMCHAR option is

used because the output from PROC METALIB contains tables.



e. Click Postcode.

f. Type the following code for post-processing of the job:

proc metalib;

omr (library="DIFT Orion Target Tables Library");

folder "/Data Mart Development/Orion Target Data";

prefix "DIFT ";

report;

run;

The Postcode area should resemble the following:

g. Click OK to close the job properties window.

4. Select File Save to save the job.

5. Run the job.


b. Click the Status tab in the Details area. Notice that the Postcode generated an error.

c. Double-click the word Error for the Postcode processing.

Kari does not have the required Create permission to create metadata objects in

the /Data Mart Development/Orion Target Data folder.





8. Check in the table and job objects.


b. Click the repository name (Kari’s Work Repository).

c. Select Check Outs Check In All.

d. Type Adding job and table needed for gdr/agegrp tables creation as the value for the

Title field.

e. Click Next. Verify that both objects are selected.

f. Click Next. Review the information in the Summary window.

g. Click Finish. The table objects should no longer be on the Checkouts tab.

9. Verify that the objects are checked in.





10. Log on using Ahmed’s credentials.

a. Select File Connection Profile.

b. Click Yes.

c. Select the My Server profile.

d. Click OK to close the Connection Profile window and access the Log On window.

e. Type Ahmed as the value for the User ID field.

f. Type Student1 as the value for the Password field.

g. Click OK to close the Log On window.

11. Access and execute the job.



c. Double-click DIFT Loop Job Populate Gender-Age Group Tables.

d. Right-click in background of the job and select Run.

e. Click the Status tab in the Details area. Verify that all steps completed successfully.



f. Click the Log tab.

g. Scroll to locate and review the PROC METALIB code.




13. Verify that the metadata table objects were created.


b. Right-click the Orion Target Data folder and select Refresh.

42

Parallel Processing

Parallel processing refers to processing that is handled

by multiple CPUs simultaneously. This technology takes

advantage of hardware that has multiple CPUs, called

SMP computers, and provides performance gains for two

types of SAS processes:

threaded I/O

threaded application processing.



43

Threaded Processing

SMP computers have multiple CPUs and an operating

environment that can spawn and manage multiple pieces

of executable code called threads.

A thread is a single, independent flow of control

through a program or within a process.

Threading takes advantage of multiple CPUs by

dividing processing among the available CPUs.



Enable Parallel Processing

This demonstration illustrates the benefits of parallel processing.

1. If necessary, access SAS Data Integration Studio using Marcel’s credentials.

2. Verify the number of processors available.

a. Select Tools Code Editor.

b. Type the following code:

options cpucount=actual;

proc options option=cpucount;

run;

c. Click Run.

d. Click the Log tab.

e. Verify that there is more than one CPU available.

f. Select File Close to close the Code Editor window.

g. Click No when asked to save changes.



3. Open the job DIFT Loop Job Populate Gender-Age Group Tables.



c. Double-click DIFT Loop Job Populate Gender-Age Group Tables.

4. Turn on run-time statistics collection and run job.

a. Right-click in background of the job and select Collect Runtime Statistics.

b. Right-click in background of the job and select Run.

c. Verify that the job completed successfully.

d. Click the Statistics tab in the Details area.

e. Notice the Duration value for the entire job.



5. Access the Loop transformation properties.


b. Click the Loop Options tab.

c. Click Execute iterations in parallel.

d. Click OK to close the Loop Properties window.


7. Rerun the job and check the performance statistics.



c. Click the Statistics tab in the Details area.

d. Notice the Duration value for the entire job.

Notice that even in this simple environment, the processing time was more than cut in half.

8. Select File Close to close the open job.



Exercises

5. Iterating a Transformation with Loop Transformations

Use Kari’s credentials for this exercise.

Create a job that uses the Extract transformation and Loop transformations to create and register a

series of tables that are subsets of DIFT Product Order Information.

The control table for the Loop transformation can be created as a separate job if desired. The

control table should contain a distinct set of compressed values from the Product_Group column.

These values are used for both subsetting criteria as well as naming the new tables. The

compressed values should have no punctuation or spaces (no spaces, no commas, no ampersands,

no quotation marks, no hyphens, and no periods).

Hint: The COMPRESS function returns a character string with specified

characters removed.

COMPRESS(<source><,chars>) where <source> is the source string that contains characters to remove. <chars> is a character string that are the list of characters to remove.

Name the physical tables PrdOrdrs _<compressed_product_group_value> and store the physical

tables in the DIFT Orion Target Tables Library.

Subset based on the value of Product_Group using the following WHERE expression:

COMPRESS(Product_Group," ,&'-.") = "&prdgrp"

where &prdgrp is defined as a parameter for the Extract transformation.

Create a metadata table object for each of the subset tables. Name the metadata table objects DIFT

<tablename> and store the metadata table objects in the Data Mart Development Orion

Target Data folder. PROC METALIB can be used to create the metadata table objects. How many

metadata table objects were created?

Run the loop processes in parallel. How much improvement, if any, is gained?

The DIFT Product Order Information table was created in Chapter 1, Exercise 2.



9.4 Solutions to Exercises

1. Creating a Summary Report on DIFT Product Order Information Table

a. If necessary, access SAS Data Integration Studio using Kari’s credentials.

1) Select Start All Programs SAS SAS Data Integration Studio 4.4.

2) Verify that the connection profile is Kari’s Work Repository.

3) Click OK to close the Connection Profile window and access the Log On window.

4) Type Kari as the value for the User ID field and Student1 as the value for the Password

field.

5) Click OK to close the Log On window.

b. Create the initial job metadata.

1) Click the Folders tab.

2) Expand Data Mart Development Orion Reports Extract and Summary.

3) Verify that the Extract and Summary folder is selected.

4) Select File New Job. The New Job window appears.

5) Type DIFT Create Report for 2007 Clothes-Shoes Products as the value for the

Name field.

6) Verify that the location is set to /Data Mart Development/ Orion Reports/Extract and

Summary.

7) Click OK. The Job Editor window appears.

c. Add source table.

1) In the Folders tab, expand Data Mart Development Orion Target Data.

2) Drag the DIFT Product Order Information table to the Job Editor.

d. Add the Extract transformation to the process flow.

1) Click the Transformations tab.

2) Expand the SQL group.

3) Drag the Extract transformation to the Job Editor.

4) Connect the DIFT Product Order Information table to the Extract.

e. Add the Sort transformation to the process flow.

1) In the Transformations tab, expand the Data folder

2) Drag the Sort transformation to the Job Editor.



3) Connect the Extract work table to the Sort transformation.

f. Add the Summary Tables transformation to the process flow.

1) On the Transformations tab, expand the Analysis folder.

2) Drag the Summary Tables transformation to the Job Editor.

3) Connect the Sort work table to the Summary Tables transformation.

g. Select File Save to save the job metadata.

h. Specify properties of the Extract transformation.

1) Right-click the Extract transformation and select Properties.

2) Click the Where tab.

3) In the lower panel, click the Data Sources tab.

4) Expand the ProdOrders table.

5) Select Product_Line and click Add to Expression.

6) Click = in the operators toolbar.

7) In the Expression Text area, type "Clothes & Shoes".

8) Click And in the operators toolbar.

9) In the Data Sources tab, expand the ProdOrders table.

10) Select Year and click Add to Expression.

11) Click = in the operators toolbar.

12) In the Expression Text area, type 2007.

The final expression should resemble the following:

13) Click OK to close the Extract Properties window.



i. Specify the properties of the Sort transformation.

1) Right-click the Sort transformation and select Properties.

2) Click the Sort By Columns tab.

3) Double-click Product_Line in the Available columns list to add it to the Sort by columns

list.

4) Double-click Quarter in the Available columns list to add it to the Sort by columns list.

5) Click OK to close the Sort Properties window.

j. Select File Save to save the job metadata to this point.

k. Specify properties of the Summary Tables transformation.

1) Right-click the Summary Tables transformation and select Properties.

2) On the General tab, remove the default description.

3) Click the Options tab.

4) Verify that the Assign columns group is selected in the selection pane.

5) Locate the Select analysis columns (VAR statement) option and click .

6) Select Quantity Ordered and click .

7) Click OK to close the Select Data Source Items window.

8) Click the Categorize data group in the selection pane.

9) Locate the Select columns to subgroup data (CLASS statement) option and click .

10) Select Product Group and click .

11) Select Quarter and click .

12) Click OK to close the Select Data Source Items window.

13) Click the Describe TABLE to print group in the selection pane.

14) In the Specify row expression field, type Product_Group.

15) In the Specify column expression field, type Quarter*Quantity=" "*Sum.

16) In the Specify TABLE statement options field, type rts=20.

17) Click the Label a keyword group in the selection pane.



18) In the Specify KEYLABEL statement field, type sum=" ".

19) Click the Other options group in the selection pane.

20) In the Specify other options for OPTIONS statement field, type ls=70 nodate nonumber.

21) In the Specify tables procedure options field, type format=comma8..

22) Click the Titles and footnotes group in the selection pane.

23) In the Heading 1 field, type Total Quantity Ordered for Quarters of 2007.

24) In the Heading 2 field, type Product_Line: Clothes & Shoes.

25) Click the ODS options group in the selection pane.

26) In the ODS result field, select Use HTML.

27) In the Location field, click Browse.

28) Navigate to the S:\Workshop\dift\reports folder.

29) Type Quantities2007ClothesAndShoes.html in the Name field.

30) Click OK to close the Select a file window.

l. Add precode.

1) Click the Precode and Postcode tab.

2) Check the Precode option.

3) Type the following statement in the Precode field:

options formchar="|---|-|---|";

4) Click OK to close the Summary Tables Properties window.

m. Select File Save to save the job metadata to this point.

n. Run the job.

1) Click Run in the Job Editor toolbar

2) Verify that the job completed successfully.

3) If necessary, view the log.

o. View the HTML report.

1) Open Windows Explorer.

2) Navigate to S:\Workshop\dift\reports.

3) Double-click Quantities2007ClothesAndShoes.html to open the generated report.

4) If necessary, click to close the security message in the browser.

5) After viewing the report, select File Close to close the report.



p. Check in the job.

1) In SAS Data Integration Studio, click in the Job Editor to gain focus.

2) Select File Close to close the DIFT Create Report for 2007 Clothes-Shoes Products job.

3) Click the Checkouts tab.

4) Click the repository name (Kari’s Work Repository).

5) Select Check Outs Check In All. The Check In Wizard appears.

6) Type Adding job to generate summary table report as the value for the Title field.

7) Click Next. Verify that the job object is selected.

8) Click Next. Review the Summary information.

9) Click Finish.

q. Verify that the DIFT Create Report for 2007 Clothes-Shoes Products job is in the Data Mart

Development/ Orion Reports/Extract and Summary folder.

2. Creating a Control Table: Years and Product_Lines

a. Define the initial job metadata object to load the control table.


2) Expand Data Mart Development Orion Loop Jobs.

3) Verify that the Orion Loop Jobs folder is selected.


5) Type DIFT Populate Control Table - Years and Product Lines as the value for the Name

field.

6) Verify that the location is set to /Data Mart Development/Orion Loop Jobs.


8) If necessary, select View Layout Left to Right.

9) Select File Save to save diagram and job metadata to this point.

b. Create the needed process flow diagram.



1) Add the source table to the process flow.

a) If necessary, click the Folders tab.

b) If necessary, expand Data Mart Development Orion Target Data.

c) Drag the DIFT Product Order Information table object to the Diagram tab of the

Job Editor.

2) Add the Extract transformation to the process flow.

a) Click the Transformations tab.

b) Expand the SQL folder and locate the Extract transformation template.

c) Drag the Extract transformation to the Diagram tab of the Job Editor. Place the

transformation next to the table object.

3) Connect the DIFT Product Order Information table object to the Extract transformation.

4) Specify the properties for the Extract transformation.

a) Right-click the Extract transformation and select Open.

b) Click the Options tab

c) Verify that the SQL Extract options group is selected.

d) Set the value for the Add the DISTINCT option to Yes.

e) Click the Mappings tab.

f) On the Target table side, remove all column definitions.

(1) Click the first column.

(2) Hold down the SHIFT key and then click the last column.

(3) Click Delete Target Columns.

g) On the Source table side, propagate two source columns to the target.

(1) On the Source table side, click Product_Line, hold down the CTRL key, and select

Year.

(2) Click and then select Selected Source Columns To Targets.

(3) On the Target table side, type Year as the Description value for the Year column.

h) Add a third column to Target table.

(1) Click .

(2) Type ProductLineVN as the Column value.

(3) Type COMPRESS(TRANWRD(ProdOrders.Product_Line ,"&","And")) as the

value for Expression.



(4) Verify that the type is set to Character.

(5) Type 25 as the length of the new column.

(6) Type Valid FileName Product_Line as the description of the new column.

(7) Map Product_Line to ProductLineVN.

5) Select File Close to close the Extract Properties.

6) Select File Save to save the job to this point.

c. Register the output table from the Extract.

1) Right-click the Extract work table and select Register Table.

2) Type DIFT Control Table - Years and Product Lines as the value for the Name field.

3) Click the Physical Storage tab.

4) Verify that DIFT Orion Target Tables Library is specified as the value for the

Library field.

5) Type DistinctYearsProdLines as the value for the Name field.

6) Click OK to close the Register Table window.



d. Run the job to generate the control table.

1) Right-click in background of the job and select Run.

2) Verify that the job runs successfully.

3) Click the Log tab and verify that DIFTTGT.DISTINCTYEARSPRODLINES is created with

20 observation and 3 variables.



4) Click the Diagram tab.

5) Right-click DIFT Control Table - Years and Product Lines and select Open.

6) Review the data then select File Close to close the View Data window.

e. Select File Close to close the Job Editor.

3. Creating a Parameterized Job

a. Copy the job that extracts data and creates a summary table report.



3) Right-click DIFT Create Report for 2007 Clothes - Shoes Products and select Copy.

4) Right-click the Orion Loop Jobs folder and select Paste.

b. Update the basic properties of the copied job.


2) Right-click Copy of DIFT Create Report for 2007 Clothes - Shoes Products (the copied job

located in the Loop Transforms folder) and select Properties.

3) Type DIFT Parameterized Job for Year - Product Line Reports as the value for the Name

field.




5) Click OK to close the Properties window.

c. Update the job to use parameters and not hardcoded values.

1) Double-click the job DIFT Parameterized Job for Year - Product Line Reports. The job

opens in the Job Editor window.

2) Edit the Extract transformation.

a) Right-click the Extract transformation and select Properties.

b) Click the Where tab.

c) Type &ProdLine in place of Clothes & Shoes (in the Expression Text area). Make sure

that double quotation marks are being used.

d) Type &year in place of 2007 (in the Expression Text area). Make sure that double

quotation marks are not being used.

e) Click OK to close the Extract Properties window.

3) Edit the Summary Tables transformation.

a) Right-click the Summary Tables transformation and select Properties.

b) Click the Options tab.

c) Click Titles and footnotes in the selection pane.

d) In the Heading 1 area, type &year in place of 2007.

e) In the Heading 2 area, type &ProdLine in place of Clothes & Shoes.

f) Click ODS options in the selection pane.

g) In the Location area, update the filename to Quantities_&year._&PLVN..html.



Be sure to type the period that separates the year parameter name and the

subsequent text, as well as the period that separates the PLVN parameter name

from the .html extension.

h) Click OK to close the Summary Tables Properties window.


5) Define job parameters.

a) Right-click in the background of the job and select Properties.

b) Click the Parameters tab.

c) Click New Prompt.

d) Type Year as the value for the Name field.

e) Type Year Value as the value for the Displayed text field.

f) Click the Prompt Type and Values tab.

g) Type 2007 as the value for the Default value field.

h) Click OK to close the New Prompt window.

i) Click New Prompt.

j) Type ProdLine as the value for the Name field.

k) Type Product Line Value as the value for the Displayed text field.

l) Select Prompt Type and Values tab.

m) Type Clothes & Shoes as the value for the Default value field.

n) Click OK to close the New Prompt window.

o) Click New Prompt.

p) Type PLVN as the value for the Name field.

q) Type Product Line Valid fileName Value as the value for the Displayed text field.

r) Select Prompt Type and Values tab.

s) Type ClothesAndShoes as the value for the Default value field.

t) Click OK to close the New Prompt window.

The final set of three parameters is listed on the Parameters tab.



u) Click to validate the prompts and default values.

v) Click OK to close the Test the Prompts window.

w) Click OK to close the DIFT Parameterized Job for Year - Product Line Reports Properties

window.


d. Run the job.


2) Click the Status tab in the Details area. Notice that all processes completed successfully.

3) Click to close the Details view.

4) View the log for the executed job.

5) Scroll toward the top of the log and note that the parameters are all defined default values.

6) Scroll toward end of the Summary Tables code area and verify that the correct HTML

filename is being generated, as well as the correct TITLE2 text.

7) Select File Close to close the Job Editor window.



e. View the HTML document.

1) Open a Windows Explorer.


3) Verify that Quantities_2007_ClothesAndShoes.html exists. (You can also check the date-

time stamp to verify that this HTML file was created with this job.)

4) Select File Close to close the Windows Explorer window.

4. Creating and Executing an Iterative Job

a. Create the initial job metadata.



3) Verify that the Orion Loop Jobs folder is selected.


5) Type DIFT Loop Job for Year & Product Line Reports as the value for the Name field.




b. Create the process flow diagram.

1) Add the control table to the Job Editor.

a) Click the Checkouts tab.

b) Drag the DIFT Control Table - Years and Product Lines table object to the Diagram tab

of the Job Editor.

2) Add the Loop transformation to the process flow.


b) Expand the Control folder and locate the Loop transformation template.

c) Drag the Loop transformation to the Diagram tab of the Job Editor.

d) Connect the DIFT Control Table - Years and Product Lines table to the Loop

transformation.

3) Add the parameterized job to the process flow.

a) Click the Checkouts tab.

b) Drag the DIFT Parameterized Job for Country Reports job to the Job Editor.



4) Add the Loop End transformation to the process flow.


b) If necessary, expand the Control folder and locate the Loop End transformation template.

c) Drag the Loop End transformation to the Job Editor.

5) Select File Save to save the job to this point.

c. Edit the properties of the Loop transformation.

1) Right-click the Loop transformation in the process flow diagram and select Properties.

2) Click the Parameter Mapping tab.

3) For the Year Value parameter (and macro variable Year), select Year as the value for the

Mapped Source Column.

4) For the Product Line Value parameter (and macro variable ProdLine), select Product_Line

as the value for the Mapped Source Column.

5) For the Product Line Valid fileName Value parameter (and macro variable PLVN), select

ProductLineVN as the value for the Mapped Source Column.

6) Click OK to close the Loop Properties window.


d. Run the job.






4) View the log for the executed Job.

5) Select File Close to close the DIFT Loop Job for Year & Product Line Reports.

e. Verify that the HTML files were created.

1) Open Windows Explorer.


3) View any of the reports.

4) Select File Close to close the report view.

5) Select File Close to close Windows Explorer.

f. Check in the table and job objects.


2) Click the repository name (Kari’s Work Repository).

3) Select Check Outs Check In All.

4) Type Adding jobs & table for loop exercise to iterate job as the value for the Title field.

5) Click Next. Verify that both objects are selected.



6) Click Next. Review the information in the Summary window.

7) Click Finish. The table objects should no longer be on the Checkouts tab.

5. Iterating a Transformation with the Loop Transformations

a. Create a job to populate separate product line tables.


2) Right-click Orion Loop Jobs and select File New Job.

3) Type DIFT Loop Job Populate Product Group Tables as the value for the Name field.


5) Click OK.

b. Turn off automatic propagation and mapping for the job.

1) Right-click in the background of the job and select Settings Automatically Propagate

Columns.

2) Right-click in the background of the job and select Settings Automatically Map

Columns.

c. Add the source data object to the process flow.

1) If necessary, click the Checkouts tab.

2) Select the DIFT Product Order Information table object.

3) Drag the object to the Diagram tab of the Job Editor.

d. Add the Extract transformation to the process flow.


2) Expand the SQL folder and locate the Extract transformation template.

3) Drag the Extract transformation to the Job Editor.

4) Connect the DIFT Product Order Information table object to the Extract transformation.

e. Specify the properties for the Extract transformation.

1) Right-click the Extract transformation and select Open.

2) Click the Options tab.

3) Set the Distinct option to Yes.

4) On the Select tab, add one new column.

a) Click once.

b) Double-click the untitled name field.



c) Type PrdGrp as the name and press ENTER.

d) Tab to the Expression field and specify the following expression information:

Compress(product_group, " ,&'-.")

e) Tab to the Length field, type 25, and press ENTER.

5) Specify appropriate mapping by clicking on Product_Group and dragging to PrdGrp.

6) Click OK to close the DIFT Loop Job Populate Product Group Tables Properties window.

f. Register the work table.

1) Right-click the Extract work table and select Register Table.

2) Type DIFT Control Table - Distinct Product_Group Values as the value for the Name

field.

3) Verify that the location is set to /Data Mart Development/Orion Target Data.


5) Type DistinctProductGroups as the value for the Name field.

6) If necessary, clear Create as view.

7) Click next to the Library field.

a) Expand Data Mart Development Orion Target Data.

b) Click DIFT Orion Target Tables Library.

c) Click OK.

8) Click OK to close the Register Table window.

g. Add the Loop transformation to the process flow.


2) Expand the Control folder and locate the Loop transformation.

3) Drag the Loop transformation to the Job Editor.

4) Connect the DIFT Control Table - Distinct Product Groups table to the Loop

transformation.

h. Add the Extract transformation.


2) Expand the SQL folder and locate the Extract transformation.

3) Drag the Extract transformation to the Diagram tab of the Job Editor.

4) Connect the work table of the Loop transformation to the Extract transformation.

i. Add the Loop End transformation to the process flow.




2) Expand the Control folder and locate the Loop End transformation template.

3) Drag the Loop End transformation to the Job Editor. The Loop End transformation connects

to the Extract transformation automatically.

j. Select File Save to save diagram and job metadata to this point.

k. Specify properties for the output table for the Extract transformation.

1) Right-click the Extract work table and select Properties.


3) Type &prdgrp._PrdOrdrs as the value for the Physical name field.

4) Clear Create as view.

5) Select Redirect to a registered library for the Location value.

6) Click next to the Library field.

a) Expand Data Mart Development Orion Target Data.

b) Click DIFT Orion Target Tables Library.

c) Click OK.


l. Edit the properties of the Extract transformation.

1) Right-click the Extract transformation and select Properties.

2) Click the Mappings tab.

The Source table area contains the columns produced by the Loop transformation.

3) Click to import target column definitions.

a) Click the Folders tab.

b) Navigate to Data Mart Development Orion Target Data.

c) Click DIFT Product Order Information.

d) Click . All column definitions from the selected table are moved to the Selected list.

e) Click OK to close the Import Columns window.

4) Click to propagate from targets to sources.

5) Click the Where tab.

6) Click the Functions tab (mid-window).

7) Click the Character folder to display the functions in this category.



8) Double-click the COMPRESS(source<,characters-to-remove>) function. The COMPRESS

function is added to the Expression Text area.

9) Click the Data Sources tab (mid-window) and expand the table.

10) Double-click the Product_Group column. The column appears as the first argument for the

COMPRESS function in the Expression Text area:

11) Highlight the <charValue> second argument in the Expression Text area.

12) Type " ,&'-." as the value for the second argument of the COMPRESS function.

13) Type the following after the COMPRESS function in the Expression Text area.

= "&prdgrp"

The Expression Text area should now resemble the following:

14) Click the Code tab.

15) In the Code generation mode field, click and select User written body.

16) In the PROC SQL code, replace &SYSLAST on the FROM clause with difttgt.prodorders.

17) Click the Parameters tab.

a) Click New Prompt.

b) Type PrdGrp as the value for the Name field.

c) Type Product Group as the value for the Displayed text field.

d) Type Product Group value as the value for the Description field.

e) Click OK to close the New Prompt window.


m. Select File Save to save the job metadata to this point.

n. Edit the properties of the Loop transformation.

1) Right-click the Loop transformation in the process flow diagram and select Properties.

2) Click the Parameter Mapping tab.



3) For the Product Group parameter, select PrdGrp as the value for the Mapped Source

Column.

4) Click OK to close the Loop Properties window.

o. Select File Save to save diagram and job metadata to this point.

p. Run the job.




4) View the log for the executed job.

q. Verify that the tables were created.

1) Access Windows Explorer.

2) Navigate to S:\Workshop\dift\datamart.

3) Verify that the data sets were created.

4) Select File Close to close Windows Explorer.

r. Select File Close to close the job.

s. Check in the table and job objects.


2) Click the repository name (Bruno’s Work Repository).

3) Select Check Outs Check In All.

4) Type Adding job and table needed for prod group tables creation as the value for the

Title field.

5) Click Next. Verify that both objects are selected.

6) Click Next. Review the information in the Summary window.

7) Click Finish. The table objects should no longer be on the Checkouts tab.

t. Log on using Ahmed’s credentials.

1) Select File Connection Profile.

2) Click Yes.

3) Under the Open an existing connection profile field, click and select

My Server.

4) Click OK to close the Connection Profile window and access the Log On window.

5) Type Ahmed as the value for the User ID field.



6) Type Student1 as the value for the Password field.

7) Click OK to close the Log On window.

u. Add pre-processing and post-processing code to the job.



3) Double-click the DIFT Populate Product_Group Tables job.

4) Right-click in the background of the Diagram tab and select Properties.

5) Click the Precode and Postcode tab.

6) Click Precode.

7) Type the following code for pre-processing of the job:

options ls=95 nodate nonumber formchar="|---|-|---|";

8) Click Postcode.

9) Type the following code for post-processing of the job:

proc metalib;

omr (library="DIFT Orion Target Tables Library");

folder "/Data Mart Development/Orion Target Data";

prefix "DIFT ";

report;

run;

10) Click OK to close the job properties window.


v. Run the job.


2) Click the Status tab in the Details area. Notice that Postcode generated an Error.

3) Click the Status tab in the Details area. Notice that all code segments completed successfully.

4) Click the Log tab.

5) Scroll to locate and review the PROC METALIB code.

w. Select File Close to close the job.

x. Verify that the metadata table objects were created.


2) Right-click the Orion Target Data folder and select Refresh.

y. Verify the number of processors available.



1) Select Tools Code Editor.

2) Type the following code:

options cpucount=actual;

proc options option=cpucount;

run;

3) Click Run in the Job Editor toolbar.

4) Click the Log tab.

5) Verify that there is more than one CPU available.

6) Select File Close to close the Code Editor window.

7) Click No.

z. Turn on run-time statistics collection and run the job.

1) Right-click in background of the job and select Collect Runtime Statistics.


a) Click the Statistics tab in the Details area.

b) Locate the Duration value for the entire job.

3) Access the Loop transformation properties.

a) Right-click the Loop transformation and select Properties.

b) Click the Loop Options tab.

c) Click Execute iterations in parallel.

d) Click OK to close the Loop Properties window.


5) Rerun the job and check the performance statistics.

a) Right-click in background of the job and select Generate Code.

b) Click the Diagram tab.

c) Right-click in background of the job and select Run.

d) Click the Status tab in the Details area and verify that all items ran successfully.

e) Click the Statistics tab in the Details area.

f) Locate the Duration value for the entire job.

Was the time improved if multiple CPUs are used to process?

6) Select File Close to close the open job.

chapter 9 working with the loop transformations · · 2014-04-26chapter 9 working with the loop...

Documents