oracle financial services data integration hub · remaining chapters in the user guide. this...

Oracle Financial Services Data Integration Hub

User Guide

Release 8.0.7.0.0

October 2019

PREFACE

AUDIENCE

ORACLE FINANCIAL SERVICES DATA INTEGRATION HUB | 2

Oracle Financial Services Data Integration Hub Copyright © 2019 Oracle and/or its affiliates. All rights reserved.

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable:

U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are “commercial computer software” pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government.

This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.

This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle.

For information on third party licenses, click here.

https://docs.oracle.com/cd/E61555_01/licensing.htm

PREFACE

AUDIENCE


Document Control

Version Number

Revision Date Change Log

01 February 2019 Understanding Mapping Report

External Data Store - Fields and their descriptions (Control Tab)

Using Aggregation for entire Dataset

Using Flatten table to PC hierarchy transformation for an EDD

Viewing the Execution History of a Connector

02 April 2019 Updated EDS section with Dynamic Principal name and Driver fields.

Updated Test Connection details - ODI setup validation settings.

03 October 2019 Added Scenario Regeneration details in Settings section.

PREFACE

AUDIENCE


Table of Contents 1 Preface .......................................................................................................................... 8

1.1 Audience ............................................................................................................................................ 8

1.2 Prerequisites ...................................................................................................................................... 8

1.3 Acronyms ........................................................................................................................................... 8

1.4 Glossary of Icons................................................................................................................................ 8

1.5 Related Information Sources .............................................................................................................. 9

2 Introduction to Data Integration Hub........................................................................ 10

2.1 Overview of Data Integration Hub (DIH) .......................................................................................... 10

2.2 Benefits of DIH ................................................................................................................................. 11

2.3 Summary of Chapters ...................................................................................................................... 11

3 Data Flow in DIH ......................................................................................................... 13

3.1 DIH Dataflow Diagram ..................................................................................................................... 13

4 Logging into DIH and Common Functionalities ...................................................... 14

4.1 Logging into DIH............................................................................................................................... 14

4.2 Common functionalities .................................................................................................................... 16

5 Setting up the Platform .............................................................................................. 17

5.1 Settings ............................................................................................................................................ 17

5.1.1 Editing the Mandatory Settings .................................................................................................. 18

5.1.2 Editing the Optional Settings ...................................................................................................... 20

6 Application Data Interface ......................................................................................... 22

6.1 Refreshing Application Data Interface ............................................................................................. 23

6.1.1 Model Changes abstracted for ETL process by DIH .................................................................. 26

6.1.2 Model Changes with impact on ETL process ............................................................................. 27

6.1.3 Data Model Validation Messages ............................................................................................... 28

6.2 Viewing Application Data Interface .................................................................................................. 29

6.2.1 Understanding Mapping Report ................................................................................................. 34

6.3 Refreshing Target Datastore ............................................................................................................ 36

6.4 Command Line Utilities .................................................................................................................... 38

PREFACE

AUDIENCE


6.4.1 Prerequisites ............................................................................................................................... 38

6.4.2 Invoking the Command Line Utilities .......................................................................................... 38

7 Data Mapping ............................................................................................................. 40

7.1 Parameters ....................................................................................................................................... 40

7.1.1 Understanding the Summary Screen ......................................................................................... 40

7.1.2 Fields and their descriptions ....................................................................................................... 42

7.1.3 Defining a Parameter.................................................................................................................. 42

7.1.4 Modifying and Viewing a Parameter ........................................................................................... 43

7.1.5 Deleting a Parameter.................................................................................................................. 43

7.1.6 Unpublishing a Parameter .......................................................................................................... 43

7.1.7 Dependency ............................................................................................................................... 44

7.1.8 Search and Filter ........................................................................................................................ 44

7.1.9 Parameters in EDD and Connectors .......................................................................................... 44

7.2 External Data Store .......................................................................................................................... 45



7.2.3 Defining an External Data Store ................................................................................................. 52

7.2.4 Modifying and Viewing an External Data Store .......................................................................... 52

7.2.5 Deleting an External Data Store ................................................................................................. 52

7.2.6 Unpublishing an External Data Store ......................................................................................... 53

7.2.7 Dependency ............................................................................................................................... 53


7.3 External Data Descriptor .................................................................................................................. 53



7.3.3 Defining an External Data Descriptor ......................................................................................... 60

7.3.4 Modifying and Viewing an External Data Descriptor .................................................................. 63

7.3.5 Deleting an External Data Descriptor ......................................................................................... 63

7.3.6 Dependency ............................................................................................................................... 64


7.3.8 Parameters in EDD Definition .................................................................................................... 65

7.4 Connectors ....................................................................................................................................... 66

PREFACE

AUDIENCE



7.4.2 Creating a Connector ................................................................................................................. 69

7.4.3 Understanding the Components ................................................................................................. 79

7.4.4 Connector Properties.................................................................................................................. 97

7.4.5 Modifying and Viewing a Connector ......................................................................................... 100

7.4.6 Copying a Connector ................................................................................................................ 101

7.4.7 Deleting a Connector ................................................................................................................ 101

7.4.8 Search and Filter ...................................................................................................................... 102

7.4.9 Parameters in Connector ......................................................................................................... 102

8 Publishing DIH Metadata ......................................................................................... 103

8.1 Publish and Unpublish ................................................................................................................... 103

9 Metadata Object Migration ...................................................................................... 107

9.1 Performing Object Migration .......................................................................................................... 107

9.2 Exporting Objects from Source Environment ................................................................................. 108

9.3 Importing Objects from Target Environment .................................................................................. 108

10 Executing DIH Metadata .......................................................................................... 110

11 Execution History .................................................................................................... 111

11.1 Viewing the Execution History of a Connector ............................................................................... 111

12 Metadata Browser .................................................................................................... 115

12.1 Connector ....................................................................................................................................... 115

12.2 External Data Descriptor ................................................................................................................ 115

12.3 Application Data Interface .............................................................................................................. 116

13 Appendix A: FAQ - Use Cases ................................................................................ 117

13.1 Loading Data from One File into Staging ....................................................................................... 117

13.2 Loading Data from Two Files Joined into Staging ......................................................................... 117

13.3 Loading Data into Staging from File and Performing Lookup into a Table .................................... 117

13.4 Loading Data into Staging from an XML File ................................................................................. 118

13.5 Loading Data into Staging from EBCDIC ....................................................................................... 118

13.6 Loading Data from File into Multiple ADI/Subtypes ....................................................................... 118

PREFACE

AUDIENCE


13.7 Loading Data into Results from File ............................................................................................... 118

13.8 Loading Data from HIVE ................................................................................................................ 119

13.9 Loading data into Partitioned Hive Table ....................................................................................... 119

13.10 Extracting Data from Staging into File............................................................................................ 120

13.11 Extracting Data from Result into File.............................................................................................. 120

13.12 OFSAA out of the box dimension tables as lookup entity .............................................................. 120

13.13 Migrating Connectors from Standard View to Dataflow View ........................................................ 121

13.14 When ODI connectivity fails with universal connection pool error ................................................. 121

13.15 How do I connect to External Source Server such as Db2/SQL Server/Teradata/Sybase? ......... 123

13.16 Disabling/Enabling DIH Webservice .............................................................................................. 123

PREFACE

AUDIENCE


1 Preface

1.1 Audience Following are the intended audience for the DIH user guide:

• ETL Developers: The ETL Developers from the IT Department of the financial services institution, who do the data sourcing.

• Business Analysts: The business analysts from the IT Department of the financial services institution, who do the mapping of the tables.

1.2 Prerequisites • OFSAA application should be installed for the staging tables. This can be installed before or after

DIH.

• Oracle Data Integrator (ODI) must be installed.

1.3 Acronyms

Acronym Description

DIH Data Integration Hub

UI User Interface

ODI Oracle Data Integrator

ADI Application Data Interface

KM Knowledge Module

EDD External Data Descriptor

Apps Application

EDS External Data Source

OLH Oracle Loader for Hadoop

1.4 Glossary of Icons This manual may refer to all or some of the following icons.

Icons Description

To create a function

To Delete a function

To view Dependencies

PREFACE

RELATED INFORMATION SOURCES


Icons Description

To copy a function

or To refresh a function

To start a function

To return to previous screen

To download a file

To add a Join

To remove a Join

To auto-map source and target

To filter the items

To search for an item in Source and Target list

To import mapping excel

To export mappings to excel

1.5 Related Information Sources Along with this user manual, you can also refer to the following documents in the OHC documentation library:

• Oracle Financial Services Analytical Application-OBP Interface User Guide Release

• Oracle Financial Services Analytical Application-DRM Interface User Guide Release

• Oracle Financial Services Analytical Application-FCUBS Interface User Guide Release

• Oracle Financial Services Analytical Application-FAH Interface

• Oracle Financial Services Data Integration Installation Manual Release

http://docs.oracle.com/cd/E60528_01/homepage.htm

INTRODUCTION TO DATA INTEGRATION HUB

OVERVIEW OF DATA INTEGRATION HUB (DIH)


2 Introduction to Data Integration Hub This chapter provides an overview to Data Integration Hub (DIH), its benefits, and the structure of remaining chapters in the user guide.

This chapter covers the following topics:

• Overview of Data Integration Hub

• Benefits of DIH

• Summary of Chapters

2.1 Overview of Data Integration Hub (DIH) Data Integration Hub enables data exchange between OFSAA and external systems. This is facilitated through logical abstraction of OFSAA Data Foundation (Financial Services Data Foundation and Insurance Data Foundation) exposed as Application Data Interfaces (ADI). External Data Sources (EDS) and External Data Descriptors (EDD) are defined through DIH user interface, which also facilitates mapping of EDDs to ADIs, forming Connectors.

DIH publishes information on ADI, EDS, EDD and Connectors, alongside other relevant metadata, to Oracle Data Integrator (ODI), which delivers high-performance data movement and transformation among enterprise platforms with its open and integrated E-LT architecture and extended support for Big Data.

DIH does not supplant ETL tools or processes. Instead, it provides a logical abstraction of physical attributes in FSDF and OIDF, allowing user interaction in business terms while defining data exchange tasks, without adding any additional overhead or hops upon execution of data movement. DIH also shields users from modelling complexities of Data Foundation, especially when it comes to handling data around slowly changing dimensions (SCD).

Factory integration between select Oracle applications and OFSAA is also rendered over DIH. Such applications currently include banking platforms (FLEXCUBE and OBP), change management solutions (DRM) and accounting platforms (FAH and FAHC). Note that, such factory integration is rendered through additional modules, the details of which are available from respective user guides.


BENEFITS OF DIH


2.2 Benefits of DIH The benefits of DIH are as follows:

• Provides a level of abstraction and insulates upstream ETL processes from staging model changes.

• Support of Big Data.

• Provides a simplified mapping screen for loading data into OFSAA staging/result through an abstract layer.

• Removes ETL technicalities by prepackaging and predefining ODI parameters based on different use cases.

• Download Specification is a system managed meta object for upstream data mapping.

• Process/execution optimization for OFSAA data and known use cases. For example, in a direct one-to-one mapping, there will not be any hopping.

• Hosts pre-defined integration with Oracle Products that play role of data providers to Analytical Applications.

• Helps in getting lineage right from source data to staging till results.

2.3 Summary of Chapters This user guide provides information on mapping of an External Data Descriptor (EDD) with Application Data Interface (ADI). It begins with information on how a Parameter is defined, how a Source System is defined in DIH, how an EDD and ADI are set, and how they are connected. The metadata is then published to ODI. The execution part does not happen in DIH as it depends on batches that are scheduled in the Financial Services Institutions. However, details about that are mentioned in the last chapter.

The guide has the following chapters:

• Data Flow in DIH: This chapter explains the flow of data in DIH.

• Logging into DIH and common functionalities: This chapter explains how to login to DIH and common functions such as modifying, viewing and deleting, using Parameter as an example.

• Setting up Platform: This chapter explains how to configure and edit the platform and parameters.

• ADI: This chapter explains how to configure the ADI for viewing an OFSAA Data Interface.

• Data Mapping: This chapter explains how to manage DIH Mapping activities such as:

Parameters: This chapter explains how to define and manage Parameters.

External Data Store: This chapter explains how to define the External Data Store information.

External Data Descriptor: This chapter explains how to define the source table/file, and adding the different components

Connectors: This chapter explains how to map one or more EDDs to ADI and how to create and view a connector.

• Publishing DIH Metadata: This chapter explains how the DIH metadata is transferred to Oracle Data Integrator (ODI).

• Object Migration: This chapter explains as to how to perform an Object Migration.


SUMMARY OF CHAPTERS


• Executing DIH Metadata: This chapter explains as to how after publishing, the scheduled batches are executed.

• Execution History: This chapter provides information about the execution history.

• Metadata Browser: This chapter details the metadata browser.

DATA FLOW IN DIH

DIH DATAFLOW DIAGRAM


3 Data Flow in DIH This chapter provides high-level information of the data flow used in DIH.

3.1 DIH Dataflow Diagram The following is the DIH data flow diagram:

For more information, click the topic on the diagram.

Start

Configure ODI Connectivity and DIH System Variables

Define External Data Source, including connectivity

parameters

Define External Data Descriptor(s)

Connect EDD to ADI

Choose Application Data Interface

Publish DIH Metadata to ODI as Scenarios for Job Executions

Execute DIH Metadata through ODI Agent

LOGGING INTO DIH AND COMMON FUNCTIONALITIES

LOGGING INTO DIH


4 Logging into DIH and Common Functionalities In this chapter, details on how to log into the DIH application are explained. There are a few common functionalities, which are used in different modules of DIH. To avoid repetition, these functionalities are explained under Common Functionalities section, taking Parameter as an example.

4.1 Logging into DIH Access the DIH application using the login credentials (User ID and password). The built-in security system ensures that you are permitted to access the window and actions based on the authorization only.


LOGGING INTO DIH


After logging into the application, select Financial Services Data Integration Hub.

The DIH window appears:

Click in the DIH home. The DIH has the below four sub links:

• Administration

• Application Data Interface

• Data Mapping

• Execution

The DIH home page displays the summary of the setup details and the activity details.

1. Setup Details: This screen lists the Parameters, EDS, EDD and Connector details along with ADIs used and not used. It also displays the details of published and not published connectors along with executed and not executed connectors.

2. Activity: This screen displays the user activity, which takes place between three days. You can

view the activities of any creation, updation, and deletion of the metadata. You can view the publish / unpublish or refresh activity in this screen.

In From and To fields, select Date and Time and click Fetch info to view the results.


COMMON FUNCTIONALITIES


4.2 Common functionalities You can modify, view, delete, and find dependencies of a Parameter, Source, EDD, ADI, and Connectors. Following sections explains how this is done, taking the example of a parameter.

SETTING UP THE PLATFORM

SETTINGS


5 Setting up the Platform Setting up the platform is the initial activity in DIH. The Settings option allows in maintaining the Oracle Data Integrator (ODI) setup information and application variables.

5.1 Settings NOTE It is assumed that ODI has been installed, configured and

verified as per its documentation, before carrying out tasks in this section.

1. Click icon in the DIH home screen to navigate to the Administration window.

2. Click Settings .The Settings window appears. This window captures the ODI set up information.

3. Only the DIH administrator is authorized to edit this field.


SETTINGS


5.1.1 Editing the Mandatory Settings While editing the Settings, the fields that are displayed are explained as follows.

5.1.1.1 Fields and their descriptions

Serial No.

Fields Description

1 ODI User The ODI supervisor user name you defined when creating the master repository or an ODI user name you defined in the Security Navigator after having created in the master repository.

2 ODI Password The ODI supervisor password you defined when creating the master repository or an ODI user password you defined in the Security Navigator after having created the master repository.

3 Use JNDI If Yes, enter the Master Repository JNDI. If No, enter Master Repository DB User, Master Repository DB Password, Master DB Driver and Master DB Connection.

4 Master Repository DB User Database user ID/login of the schema (database, library) that contains the ODI master repository.

5 Master Repository DB Password

Master Repository DB user password.

6 Master DB Driver Specifies the driver required to connect to the RDBMS supporting the master repository created from the dropdown list. The default value is oracle.jdbc.OracleDriver. It need not be changed if it is on Oracle database.

7 Master DB Connection The URL used to establish the JDBC connection to the database hosting the repository. The format is jdbc:oracle:thin:@<Hostname/IP Address>:<Port Number>:<Service Name>

8 Master Repository JNDI JNDI name for ODI Master Repository

9 Work Repository The name of the work repository that is created previously (Example: WorkRep1).

10 Project Enter the Project Name created in ODI.

11 Folder Enter the folder name under the project created in ODI. All the packages are created under this location.

12 Agent URL Specify the agent URL where the ODI agent is running. This is used to execute a DIH connector from OFSAAI batch/RRF. This is not needed to do data mapping. The format is http://<Hostname/IP Address where ODI agent is running>:<PORT Number>/<Agent Context Name>


SETTINGS


5.1.1.2 Procedure to Edit the Mandatory Settings To edit the mandatory settings, perform the following steps:

1. Edit the ODI User details in the following window.

2. Enter the field details as explained in Fields and their Description section.

3. To validate details specified, click . The validation summary displaying status on below details

ODI Connection

ODI Project Name

ODI Folder Name under the project

Import Status of Knowledge Module from DIH

Import Status of ODI procedures from DIH

4. Enter the details and click Save.

To know about ODI connectivity failing with universal connection pool error, refer Migrating Connectors from Standard View to Dataflow View

5. While upgrading DIH from version 8.0.4.0.0 or lower, open an existing connector, the following error message appears.


SETTINGS


Perform the following steps:

1. Login to DIH application with valid User ID and Password.

2. Open new tab in the browser and click the following link:

https://<HOST>:<PORT>/<CONTEXT>/dih-api/util/connectors/upgrade-con?infodom=<INFODOM>&checkOrUpdate=U

The response provides information on number of connectors migrated with the status of their migration.

NOTE While upgrading DIH from version 8.0.5.0.0 or higher, the connectors created in standard view must also be migrated to Dataflow view. Perform the above steps to migrate connectors from Standard View to Dataflow View.

When ODI connectivity fails with universal connection pool error in FAQ - Use Cases section.

5.1.2 Editing the Optional Settings While editing the Settings, under optional tab the fields that are displayed are explained as follows.

NOTE The following properties are optional and need not be specified, if they are already available as environment variables in the server where ODI agent is running.

5.1.2.1 Fields and their descriptions

Serial No. Fields Description

1 Character Set (Applicable for File type Source)

This field is applicable if the source system is of type File. You must specify the character set when you are using SQL loader for data loading.


SETTINGS


Serial No. Fields Description

2 ODI Oracle Home This field is applicable if the source system is of type File. You must specify the Oracle Home path where the ODI agent is located.

3 Scenario Regeneration

This field allows choice of when Scenario Regeneration is performed in ODI. If you choose ‘Publish’, Scenario Regeneration is performed during Connector Publish in DIH. If you choose ‘Execution’, Scenario Regeneration is performed during each Connector execution. Note: Scenario Regeneration is needed such that ODI tasks are refreshed or maintained up to date with

a. source, destination, mapping and related information, and

b. parameters set through ODI user interface, including those for performance optimization / tuning purposes.

Performing Scenario Regeneration during Connector execution is, therefore, a useful option. However, in certain circumstances, especially when a number of instances of the same Connector are executed concurrently, Scenario Regeneration during Connector execution may be sub-optimal and even trigger execution errors.

5.1.2.2 Procedure to Edit the Optional Settings To edit the optional settings, perform the following steps:

1. You can edit the optional ODI details in the following window.

2. Enter the field details explained in Fields and their descriptions section.

3. Click Add icon to add multiple rows for each agent. The Save Agent As window appears.


SETTINGS


4. Enter the Agent Name and URL and click OK.

NOTE It is mandatory to enter the agent name.

5. Click Edit icon to modify the saved agent.

6. Click Delete icon to delete the agent.

7. Enter the details and click Save.

APPLICATION DATA INTERFACE

SETTINGS


6 Application Data Interface Application Data Interface (ADI) data comes pre-seeded based on the application that is installed.

Application Data Interface enables to view the logical definition of OFSAA physical entities of staging and Result area. This can be viewed by selecting application and its subtype.

Click icon in the DIH home screen to navigate to the Application Data Interface window.

The Application Data Interface summary displays the following:

• Refresh ADI: The Last ADI Refresh, which details the last refresh, performed, execution date, status and the changes.

• Number of ADIs: The total number of ADIs, which are present and used.

• Target Datastore Refresh: The Last Target Data Store Refresh, which details the last refresh, performed, execution date, status and the changes.

The summary screen also displays the graph of total attributes, mandatory attributes and the mapped attributes.


REFRESHING APPLICATION DATA INTERFACE


6.1 Refreshing Application Data Interface The Refresh ADI creates ADIs by reading the uploaded data model in the setup. It works for incremental changes also. If there is any change in data model from the previous one, then ADI refresh compares the existing ADI with updated model and accommodates the changes by either adding new ADI or adding new attribute and so on.

To refresh the ADI list, perform the following steps:

1. Click Refresh ADI on Application Data Interface screen.

The Refresh Application Data Interface summary appears:

2. Click icon on the right hand side of the window for Datamodel Validation. The Datamodel Validation window appears. This validates and identifies the issues in values specified by the user-defined properties for physical/logical view in the OFSAA Data Model. Once executed, the utility log errors/issues are identified.

3. You can search for an Object name or Message. On validation, you receive a message. Refer

Data Model Validation Messages table for information on each message.

4. Click Export to Excel icon to export the results of the validation done.




5. Verify the information and click OK.

6. Click icon on the right hand side of the window to start the refresh of ADIs.

7. The ongoing ADI refresh appears as follows:

8. On successful invocation of ADI refresh, a message appears.

9. In case you need a detailed running log, click icon to download the log.

10. A zip file is downloaded containing the detailed log for the execution.

11. To view the log details, extract the log file from the zip folder.

12. You can check the status:

Failed

Successful

Aborted

Alert

Warning

Not Applicable

13. Click the Run ID link on the Refresh ADI screen. This displays the Changes, Alerts and Error Messages. Under the changes tab, you can view all the details that are made as part of ADI Refresh.




NOTE At any given time, click the refresh icon to check the status if it is complete or still in progress.




6.1.1 Model Changes abstracted for ETL process by DIH

Scenario Description

When only logical name of an attribute is changed

ADI refresh updates the logical name in DIH repository. Post ADI refresh, no action is expected from user, changes are reflected automatically in connector/ADI.

When only description of an attribute is changed

ADI refresh updates the description in DIH repository. Post ADI refresh, no action is expected from user, changes are reflected automatically in connector/ADI.

When only domain of an attribute is changed

ADI refresh updates the domain in DIH repository. Post ADI refresh, no action is expected from user, changes are reflected automatically in connector/ADI.

When both logical name and domain of an attribute are changed

ADI refresh updates the logical name and domain in DIH repository. Post ADI refresh, no action is expected from user, changes are reflected automatically in connector/ADI.

When physical name of an attribute is changed

ADI refresh updates the physical name in DIH repository. Post ADI refresh, user needs to perform refresh target data store, and unpublish and publish a connector for changes to be reflected in ODI.

When data type of an attribute is changed

ADI refresh updates the data type in DIH repository. Post ADI refresh, user needs to perform refresh target data store, and unpublish and publish a connector for changes to be reflected in ODI.

When precision/scale of an attribute is changed

ADI refresh updates the precision/scale in DIH repository. Post ADI refresh, user needs to perform refresh target data store, and unpublish and publish a connector for changes to be reflected in ODI.

When physical name of an entity is changed

ADI refresh updates the physical name in DIH repository. Post ADI refresh, user needs to perform refresh target data store, and unpublish and publish a connector for changes to be reflected in ODI.

When logical name (OFSAA Data Interset or SubType Name) of an entity is changed

ADI refresh updates the logical name in DIH repository. Post ADI refresh, no action is expected from user, changes are reflected automatically in connector/ADI.




6.1.2 Model Changes with impact on ETL process The following model changes will have an impact for connectors during ADI refresh.

Column used in connector mapping are dropped from table.

Table used in connector are dropped.

The ADI refresh process displays status as Impact Identified. The affected connectors and the respective used objects (which are dropped in the model) are list under the Alerts tab as displayed in the below image.

Message Action

Table being dropped is already used in the connector

Unpublish and remove the ADI from the connector.

Column being dropped is already used in the connector

Unpublish and remove the attribute references from the connector. In case of Insert type connector, remove attribute reference from mapping and truncate filter expression. In case of Extract type connector, remove attribute reference from filter, join, look up, derived column, mapping and aggregation components.




6.1.3 Data Model Validation Messages

Message Reason Resolution

Table Classification Missing

User defined Property “OFSAA Data Interface Class” is not specified in logical view of the table in OFSA Data Model.

Specify the value for User Defined Properties in the OFSA Data Model in ERWIN

Sub Type Name Missing

User defined Property “OFSAA Data Interface Sub-Type” is not specified in logical view of the table in OFSA Data Model


Duplicate ADI Name

User defined Property “OFSAA Data Interface Name” must be different for the specified tables.

Specify unique value for OFSAA Data Interface Name UDP in the OFSA Data Model in ERWIN

No enabled Application mapped

The Application user defined properties for all columns of table does not have the value as “DL-MAN” or “DL-OPT”

The value for application user defined properties must be set to mentioned value.

ADI Name Missing

User defined Property “OFSAA Data Interface Name” is mandatory and is not specified in logical view of the table in OFSA Data Model


Invalid Table Classification

User defined Property “OFSAA Data Interface Class” can have the value as R ,S or D only

Specify correct value for mentioned user defined property

Invalid Subtype Name

User defined Property “OFSAA Data Interface Sub-Type” is specified when there is no subType for ADI

OFSAA Data Interface Sub-Type UDP is applicable when there are multiple subtypes for given ADI. Please leave it blank or same as ADI name otherwise.


VIEWING APPLICATION DATA INTERFACE


6.2 Viewing Application Data Interface This tile displays the total number of ADIs that are available in the set up and the number of used ADIs in connectors.

To view the total number of ADIs that are present and used, perform the following steps:

1. Click Total or Used icon on the Application Data Interface.

The Application Data Interface summary screen appears:

2. Click to view the Mapping Report for that particular ADI. For more information, see Understanding Mapping Report section.

3. You can view the summary details of all the ADIs that are present or used in either Card view

or List view .

4. The Search bar helps you to find the required information. You can enter the nearest matching keywords to search, and filter the results by entering information on the search box. You can search for an ADI name using either the name or description.

5. Click to filter the ADI. The RHS displays the applications you can select to filter. Select the required application and the feature.




6. Select the required application and then click Apply. The summary screen displays the filtered

ADIs.

7. Click Reset to deselect the filter options and clear the Subject area.

8. Click to view the Attributes for that particular ADI on the summary screen. For each ADI, you can view the following details, number of attributes, number of physical entities, number of applications and the number of connectors defined on that ADI.

9. Depending on the ADI selected, there may or may not be additional subtype filters. Such as, for

Transactions: Customer Account, there is a Product Class list as subtype filters available. You can choose one or more Product class to filter the attributes listed below.

10. The selected ADI details are displayed. There are two views for each ADI:

Logical View: The Logical view shows all the attributes and its associated description with additional information. For example, if the attribute is mandatory or not for the selected application, its domain and LOV (List of values) that are possible for the particular attribute.




Physical View: The Physical view shows the underlying physical table name of the selected ADI. On selecting the physical table name, it shows the mapping between the logical attribute name and its corresponding physical column name.

At any given time, you can switch between logical and physical view.

11. In the Logical View, you can see the attribute details as follows:

List with the logical name

Description

Domain

List of values

12. Attributes which contain,

Sensitive information PII (Personally Identifiable Information) is displayed with icon prior the attribute.

Attributes, which are deprecated, are shown as strike through text.

Attributes which are mandatory have * next to them.

Attributes which are mandatory for selected applications have ** next to them.




13. In the Logical View, you can search for an attribute name or description.

14. The following progress bar display the mapping status in percentage for the ADI. The progress can also be viewed based on the filters applied in this screen.

NOTE The percentage is calculated based on the number of attributes sourced verses the total number attributes.

.

15. In Physical view, click the table name. You can view the attribute name, field name, data type, length, precision and .format.

For example, in case of ADI with subtype such as Customer Account, the physical table name is based on the Subtype. Hence, one or more physical table names may appear.

When you select the table, the respective attribute is displayed.

16. In the Physical View, you can search with either an attribute name or physical column name.

In both logical and physical view, you can click the filter icon.

A filter drawer appears with options to filter based on applications, OFSAA Module, Logical Domain and other properties.




17. Select the required application and then click Apply. The summary screen displays the filtered

ADIs.




18. Click Reset to deselect all filter options.

19. Click Apply to filter the attribute list.

6.2.1 Understanding Mapping Report To understand the mapping Report, perform the following steps:

1. In the Application Data interface screen, click to view the Mapping Report for that particular ADI.




The Mapping Report window appears. This displays the number of attributes mapped for the ADI (subtype) in each connectors created for the same.

2. Select the Sub Type Name from the list of sub types present in the ADI if applicable.

3. Select the Connector Name from the list of connectors, which are populating that particular ADI.

Depending on the connector and sub type you select, the attribute report is displayed.

4. For example, Loan Contracts are mapped to the applications present in the above screen. The report displays the total number of mapped attributes and total number of mandatory attributes for that particular application.

In case the mapped attributes are,

Less than mandatory attributes, they are displayed in Red.

Greater than mandatory attributes they are displayed in Yellow.

Same as the total number of attributes they are displayed in Green.


REFRESHING TARGET DATASTORE


It also displays the EDS name, the number of attributes from that particular EDS and the derived expression count.

5. Click to open it in View mode.

6.3 Refreshing Target Datastore It represents underlying physical tables of every available ADI as data store in ODI. This process also works on incremental way.

Post data model upload, if the changes are to be made available in DIH, then first perform Refresh ADI. Once Refresh ADI is done, perform Refresh Target Datastore.

To refresh the Target Datastore list, perform the following steps:

1. Click Target Datastore Refresh on Application Data Interface screen.


REFRESHING TARGET DATASTORE


The Refresh Target Datastore Interface summary appears:

2. Click icon on the right hand side of the window to start the refresh of Target Datastore. The ongoing Target Datastore refresh appears as follows:

3. On successful invocation of TDS refresh, a message appears.

4. In case you need a detailed running log, click icon to download the log.

A zip file is downloaded containing the detailed log for the execution.


NOTE At any given time, click the refresh icon to check the status if it is complete or still in progress.

NOTE This is not a day-to-day activity.


COMMAND LINE UTILITIES


6.4 Command Line Utilities Using the command line utility, you can invoke both Refresh Application Data Interface and Refresh Target Datastore process.

6.4.1 Prerequisites • You must have access and execute permission to the following directory:

$FIC_HOME/ficdb/bin

• If secured protocol is enabled for accessing OFSAA application then “CURL_CA_BUNDLE” environment variable must be set where application is installed. The variable points to the path where CA certificate is available that is generated during application deployment.

For example: CURL_CA_BUNDLE=/usr/share/ssl/certs/ca-bundle.crt.

6.4.2 Invoking the Command Line Utilities To invoke the command line utility for refreshing ADI and Target Datastore, perform the following steps:

1. Once the prerequisites are met, navigate to $FIC_HOME/ficdb/bin path and execute RefreshADI.sh and then RefreshDS.sh. This expects two mandatory parameters:

a. Information Domain name where DIH is installed. This should be mandatorily mentioned in uppercase.

b. OFSAA Login ID, which can be specified in either lower or uppercase.

2. For example, refer the following screens.

a. For Refreshing Application Data Interface, use the following command:

./RefreshADI.sh <INFODOM_NAME> <OFSAA_USER>

b. For Refreshing Target Datastore, use the following command:

./RefreshDS.sh <INFODOM_NAME> <OFSAA_USER>


COMMAND LINE UTILITIES


NOTE If the webserver is of type Websphere, an error message appears in the console as “Refresh Target Datastore as fail”.

Verify this from the application screen, and ignore the message in the command prompt and proceed.

If the webserver is of type Weblogic, an error message appears "curl: (35) SSL connect error". In this case, invoke the process from application screen and proceed.

DATA MAPPING

PARAMETERS


7 Data Mapping To manage DIH Mapping activities, we have the following:

• Parameters: You can define and manage Parameters.

• External Data Store: You can define and manage the External Data Store information.

• External Data Descriptor: You can define and manage the External Data Descriptor information.

• Connectors: You can define and manage the Connectors for the data load process.

For detailed information, click the required activity.

7.1 Parameters Parameters are placeholders and constant values that have different uses in DIH.

7.1.1 Understanding the Summary Screen

1. To understand the summary screen, perform the following steps. Click icon in the DIH home screen to navigate to the Data Mapping window.

2. Click Parameters.

The Parameters Summary appears.

DATA MAPPING

PARAMETERS


3. You can make use of the Search option to search for a specific Source.

4. Click Export to Excel icon. The List of Parameters are exported to an excel sheet with the following information:

a. Parameter IDs

b. Parameter Name

c. Description

d. Type

e. Value

f. Default Value

g. Date Format

h. Status

i. Last Modified By and

j. Last Modified Date

5. Click button to create a Parameter. For more information, refer section Defining a Parameter.

This chapter includes the following sections:

• Fields and Descriptions

• Defining a Parameter

• Modifying and Viewing a Parameter

• Deleting a Parameter

• Unpublishing a Parameter

DATA MAPPING

PARAMETERS


• Dependency

• Search and Filter

• Parameter in EDD Definition

• Parameters in Connector

7.1.2 Fields and their descriptions Fields displayed in Parameters screen are explained in the following table.

Fields Description

Fields marked in red asterisk(*) are mandatory

Parameter Name The name for the placeholder that you want to define. For example, MISDATE, which can be used as a placeholder for Date.

Parameter Description

The description for the parameter you want to define. In this example, the description can be, “MISDATE can be used to substitute the date values for each day, dynamically, in mmddyyyy format.”

Parameter Type There are three parameter data types: • Constant: Constant data type is selected, for substituting a

constant value. • RunTime: RunTime data type is selected for substituting a value

dynamically, in run time. In the example that is used here, MISDATE can be selected as Run Time, because, it is used to make a substitution, dynamically.

• CurrDate: CurrDate data type is selected for substituting a value as Current System Date.

Value Only for constant types. Holds the actual value that of the parameter

7.1.3 Defining a Parameter To define a new Parameter, perform the following steps:

1. Click the Add button to define a parameter on the Parameters Summary. The Parameters screen appears.

2. Enter the fields, which are explained and tabulated in the Fields and their Description section.

3. Enter the above details and click Save to save the changes made.

DATA MAPPING

PARAMETERS


4. The Audit Trail section at the bottom of the screen displays the information of the parameter created.

7.1.4 Modifying and Viewing a Parameter You can edit or view an existing Parameter, other than the Parameters, which are in published status.

NOTE You cannot edit the parameter if the parameter is in published state.

1. To edit or view a parameter, perform the following steps:

2. To edit or view a Parameter, you can select the required parameter from the Parameters Summary.

3. The details of the selected Parameter is displayed. You can modify or view the details.

4. Only the Parameter description, Parameter Type and the Value / Default Value / Date Format can be edited on this screen. Update the required details.

5. Click Save to save the changes made.

7.1.5 Deleting a Parameter To delete an existing parameter, perform the following steps:

1. On the Parameters Summary, click Delete button. A confirmation dialogue appears.

2. Click Yes. The Parameter details are deleted.

NOTE The Delete button is enabled only in the following cases:

• If the parameter is not in published state.

• If it is not used by any higher object, for example: Connector/EDD.

• If it is pre-seeded.

7.1.6 Unpublishing a Parameter You can unpublish a parameter only when all the following conditions are met:

1. The parameter is in published state.

2. All the higher objects using the parameter are in unpublished state, for example: Connector/EDD.

To unpublish a parameter, perform the following steps:

3. Select the required parameter from the parameter summary. The details of the selected parameters are displayed.

4. Click Unpublish.

NOTE Parameters are published automatically by the system whenever the higher objects (EDD/Connector) which are using it, are published.

DATA MAPPING

PARAMETERS


7.1.7 Dependency

As the name suggests, on clicking the Dependency icon , it lists where the entire parent Parameter has dependency.

7.1.8 Search and Filter The Search and Filter option in the user interface helps you to find the required information. You can enter the nearest matching keywords to search, and filter the results by entering information on the search box. You can search for a parameter using either the name, description, status or type.

For example, enter the Parameter key word as ‘ODI’ in the search box. The entire Parameter name with ODI is listed.

7.1.9 Parameters in EDD and Connectors For information on parameters in EDD and Connectors, refer sections Parameters in EDD Definition and Parameters in Connector respectively.

DATA MAPPING

EXTERNAL DATA STORE


7.2 External Data Store This option enables to define the External Data Store (EDS) information. DIH supports Oracle database, XML, EBCDIC, Flat file data sources, DB2, SQL server, Sybase, Teradata, and Big data sources such as HDFS and Hive.

7.2.1 Understanding the Summary Screen To understand the summary screen, perform the following steps:

1. Click icon in the DIH home screen to navigate to the Data Mapping window.

2. Click External Data Store.

The External Data Store Summary appears.

3. In the Source Systems section of the External Data Store Summary, you can define, edit, and

delete a source.

4. You can make use of the Search option to search for a specific Source.

DATA MAPPING

EXTERNAL DATA STORE


5. Click Export to Excel icon. The List of EDSs are exported to an excel sheet with the following information:

a. EDS IDs

b. EDS Name

c. Description

d. EDS Type

e. JDBS URL

f. File Location

g. Status

h. Last Modified By and

i. Last Modified Date

6. Click button to create an EDS. For more information, refer section Defining an External Data Store.

This chapter has the following sections:

Fields and Descriptions

Defining an External Data Store

Modifying and Viewing an External Data Store

Deleting an External Data Store

Unpublishing an External Data Store

Dependency

Search and Filter

7.2.2 Fields and their descriptions

Fields Description


Name Enter the name of the Source.

Description Enter a description for the Source.

Type The following options are available. Click each source type for information on the additional fields specific to them.

• DB2 Type • EBCDIC Type • File Type • HDFS Type • Hive Type

DATA MAPPING

EXTERNAL DATA STORE


Fields Description


• Oracle Type • SQL Server Type • Sybase Type • Teradata Type • XML Type

7.2.2.1 DB2 Type IBM DB2 is a family of database server products. These products support the relational model.

When you select EDS Type as DB2, the following fields appear:

Field Description Comments

JDBC URL

In this field, you need to mention the URL of the DB

Example: jdbc:db2://<hostname>[:<port>]/<database>

User ID Enter the User ID

Password Enter password

Schema Enter the schema name

7.2.2.2 EBCDIC Type Extended Binary Coded Decimal Interchange Code (EBCDIC) File is a binary code for alphabetic and numeric characters.

When you select EDS Type as EBCDIC, the following fields appear:


File Location Enter the absolute path of the data file landing area.

ODI agent must be available and running in the server from where the data file is located.

7.2.2.3 File Type American Standard Code for Information Interchange (ASCII) File is a character-encoding scheme.

When you select EDS Type as File, the following fields appear:

DATA MAPPING

EXTERNAL DATA STORE





Encryption at Rest

In case source file is encrypted, choose Encryption at Rest option and enter the Encryption Key File Path.

• DIH must have access to the source file landing area.

• The UNIX user, which is used for starting the agent, must have execution permission to DMT utility.

Example: /landingzone/inputfiles

7.2.2.4 HDFS Type Hadoop Distributed File System (HDFS) is an open source and fundamentally a new way of storing and processing data. It enables distributed processing of huge amounts of data across industry-standard servers that both store and process the data, and can scale without any limits HDFS is Big Data in a raw format.

When you select EDS Type as HDFS, the following fields appear:


JDBC URL

In this field, you need to mention the URL of the DB.

Example: hdfs://<Host>:<Port>

File Location

Enter the file location.

7.2.2.5 Hive Type Hive provides a mechanism to project structure onto the data in Hadoop. Big data in tabulated format.

When you select EDS Type as Hive, the following fields appear:


JDBC URL


Example: jdbc:hive://<Host>:<Port>/<Schema>

Driver Enter the driver for HIVE datastore

For example, to connect to Cloudera Hive server with JDBC 4.0 data standards, specify “com.cloudera.hive.jdbc4.HS2Driver” as driver. Refer Cloudera document for more details about Cloudera JDBC drivers.

https://www.cloudera.com/documentation/other/connectors/hive-jdbc/latest/Cloudera-JDBC-Driver-for-Apache-Hive-Install-Guide.pdf

DATA MAPPING

EXTERNAL DATA STORE



If Kerberos is enabled:

Principal User Name

Enter the Principal User Name

Conf File Path

Enter the Kerberos Conf File path

Key Tab Path

Enter the path of the Key Tab file, generated for the principal user

If Kerberos is not enabled:


Password Enter the Password

7.2.2.6 Oracle Type An Oracle database (DB) is a collection of data treated as a unit. The purpose of a database is to store and retrieve related information.

When you select EDS Type as Oracle, the following fields appear:


JDBC URL In this field you need to mention the URL of the DB

Example: jdbc:oracle:thin:@//<hostname>:<port>/<servicename>


Password Enter a password

Schema Enter the Schema name in upper case

Encryption in Transit

Choose this option in case you want the data to be encrypted while reading from source.

DATA MAPPING

EXTERNAL DATA STORE


7.2.2.7 SQL Server Type Microsoft SQL Server is a relational database management system. It is a software product with the primary function of storing and retrieving data as requested by other software applications, which may run either on the same computer or on another computer across a network.

When you select EDS Type as SQL Server, the following fields appear:


JDBC URL


Example: jdbc:sqlserver://<hostname>\SQLExpress



Schema Enter the Schema name

7.2.2.8 Sybase Type Sybase produces software to manage and analyze information in relational databases.

When you select EDS Type as Sybase, the following fields appear:


JDBC URL

In this field, you need to mention the URL of the DB

Example: jdbc:sybase:Tds:<hostname>:<port>




7.2.2.9 Teradata Type Teradata Corporation provides analytic data platforms, applications and related services. Its products enable users to consolidate data from different sources and make the data available for analysis.

When you select EDS Type as EBCDIC, the following fields appear:

DATA MAPPING

EXTERNAL DATA STORE



JDBC URL

Enter URL of the DB

Example: jdbc:teradata://<hostname>

User ID: Enter the User ID



7.2.2.10 XML Type Extensible Markup Language (XML) is a markup language that defines a set of rules for encoding documents.

When you select EDS Type as XML, the following fields appear:




DATA MAPPING

EXTERNAL DATA STORE


7.2.3 Defining an External Data Store While defining a Source, the fields that are displayed are explained as follows.

To define a new source from External Data Store Summary, perform the following steps:

1. Click Add button to define a new External Data Store on the External Data Store Summary. The External Data Store screen appears.

2. Enter the values in the fields as described and tabulated in the Fields and their Description

section.

The fields will change as per the option selected for Type. For example, If Source Type is selected as File, The File Location field needs to me mentioned.

3. Click “Test Connection” to test the connection details (User ID/ Password) for the database types DB2, HIVE, Oracle DB, SQL Server, Sybase, and Teradata.

4. Enter the above details and click Save.

7.2.4 Modifying and Viewing an External Data Store To edit and view an external data store, perform the following steps:

1. To edit or view an EDS, you can select the required EDS from the EDS parameter summary.

NOTE Edit operation is restricted for Published objects.

2. The Audit Trail section at the bottom of the screen displays the information of the source created.

3. EDS Name and Type cannot be edited.


7.2.5 Deleting an External Data Store To delete an existing EDS, perform the following steps:

1. On the EDS Summary, click Delete button. A confirmation dialogue appears.

2. Click Yes. The EDS details are deleted.

DATA MAPPING

EXTERNAL DATA DESCRIPTOR



• If the EDS is not in published state,

• If it is not used by any object.

7.2.6 Unpublishing an External Data Store You can unpublish an EDS only when all the following conditions are met:

1. The EDS is in published state.

2. All the higher objects using the EDS are in unpublished state, for example: Connector/EDD.

To unpublish an EDS, perform the following steps:

1. Select the required EDS from the EDS summary. The details of the selected EDS are displayed.

2. Click Unpublish.

NOTE EDS are published automatically by the system whenever the higher objects (EDD/Connector) which are using it, are published.

7.2.7 Dependency

As the name suggests, on clicking the Dependency icon , it lists where the entire parent EDS has dependency. That is, you cannot delete a child file without deleting the parent file.

7.2.8 Search and Filter The Search and Filter option in the user interface helps you to find the required information. You can enter the nearest matching keywords to search, and filter the results by entering information on the search box. You can search for a parameter using either the name, description, status or type.

For example, enter the Parameter key word as ‘DRM in the search box. All the EDS names with DRM are listed.

7.3 External Data Descriptor The External Data Descriptor (EDD) allows defining or registering external source data structures in DIH. The EDD definition can be used for loading and extracting data into/from OFSAA. The control tab is not used for extraction. It is only used when reconciliation is required while loading data into OFSAA.

DATA MAPPING



7.3.1 Understanding the Summary Screen


2. Click External Data Descriptor.

The External Data Descriptor Summary appears.

3. Click Export to Excel icon. The List of EDDs are exported to an excel sheet with the following information:

a. EDD IDs

b. EDD Name

c. Description

d. EDS Name

e. EDS Type

f. Status

DATA MAPPING



g. Last Modified By and

h. Last Modified Date

4. Click button to create an EDD. For more information, refer section Defining an External Data Descriptor.


• Fields and Descriptions

• Defining an EDD

• Modifying and Viewing an EDD

• Deleting an EDD

• Unpublishing an External Data Descriptor

• Dependency

• Search and Filter

7.3.2 Fields and their descriptions

7.3.2.1 Data tab

Fields Description


Data File Name You can add multiple data files to an EDD. For example, You need to add Term Deposits Contracts data file. There are Term Deposits Contracts data files for Retail as well as Corporate accounts. Therefore, to get both these details, you first add the Term Deposits Contracts data file for Retail accounts, say, td_contracts%#MISDATE%_1.csv and as the next record, add Term Deposits Contracts data file for Corporate accounts. Example: td_contracts%#MISDATE%_1.csv

Record Delimiter The records are stored differently in different operating systems. The options available are:

• MS-DOS • Unix • No Record Delimiter • Other

With respect to the example, select Unix.

DATA MAPPING



Fields Description


File Format There are 2 options: • Fixed Length: The file will have records and columns

with a fixed length. Each column will have a predetermined and unchanging size, set when the record layout is designed, and the sum of the column sizes will add up to the record size.

• Delimited: There will be a separation of the records and columns using a delimiter character like comma, semicolon, hyphen and so on.

With respect to the above example, select Delimited.

Text Qualifier It’s a character that identifies a text. This is used when some characters exists within a text. Generally, double quotes are used, prefixed and suffixed with a text. This is optional.

Skip Number Of Records The number of records to be skipped is to be mentioned. The records are skipped from the top. Generally, this is to skip the Headers.

Decimal Separator This mentions up to which decimal digit you want to view the result in.

Read from template A template contains all the values and is in Excel file format. If the template is not available, you need to create it manually by clicking

the Add button, under Source Data Elements tab. If the template is available, you can browse for the template. Refer to the File EDD Template.

You can also drop the template in the area “ “.

Select Template (*.xls,*.xlsx,*.csv Files Only

You can click Browse and select the required template.

Data Elements

Name Name of the field in EDD. Example: Field name in a file or column name in a table. Note: Field name of XML type should not be more than 25 characters and for others should not be more than 30 characters.

Type This shows the Data type of the field. Example: String, Number, EBCDIC and so on.

Length This is applicable only for EBCDIC format. This is the length of EBCDIC data type. In case of file, it is length only.

Precision It is to specify the decimal point. Example: 10.3.

Format It is the format to specify the date format. Example: YYYY-MM-DD. The format should be in capitals.

https://docs.oracle.com/cd/E92630_01/PDF/common/File_EDD_Template.xlsx

DATA MAPPING



Fields Description


Record Type Code This identifies the Record type in a file where Header, Trailer and Data are of different record length and type. The values can be any string available in the text file. This value is only possible for the first field in a file. Example: The values can be DATA; CTRL to specify it is a control record.

7.3.2.2 Control tab In Control tab, DIH allows to:

1. Reconcile between source data received and data loaded into OFSAA, using the control information available from source system. This feature is used to implement record count validation or amount reconciliation from a source file. It also allows to specify a threshold for validation. The specified threshold is compared with the difference (record count or amount) to perform the validation. Only in case the difference is more than the threshold value the execution fails. Detailed report on control information from both source and the final data loaded along with threshold breach is available in Execution History. The threshold can be absolute or percentage value. The connector execution process considers threshold type while performing reconciliation.

2. Generate control information for File extracts from OFSAA. Specifying control for extract EDDs generates control details, based on the configuration in the EDD. Both record count and amount value (sum, average, max and so on) for specific columns are recorded into control file.

NOTE This option is applicable only for File type EDDs (ASCII and EBCDIC).

The following fields must be configured, to achieve the above functionality:

Fields Description


When Separate File is selected as Yes.

File Name Specify the name of the file.

File Format There are 2 options: Fixed Length: The file will have records and columns with a

fixed length. Each column will have a predetermined and unchanging size, set when the record layout is designed, and the sum of the column sizes will add up to the record size.

Delimited: There will be a separation of the records and columns using a delimiter character like comma, semicolon, hyphen and so on.

With respect to the above example, select Delimited.

DATA MAPPING



Fields Description


Column Delimiter If the File Format is selected as Fixed Length, the Column Delimiter would by default be Other. If the File format is selected as Delimited, the following options are available in the drop down list. Other

Space

Semicolon

Comma

Tab

With respect to the above example, select Comma.

Record Type Code Used to uniquely identify a record within a file. Many times, Bank provides file having data and control record within same file. In that case, in order to distinguish between data record and control record, the first field is Record Type. It has a specific value to identify that. Here, specify the value that identifies the Data. Values can be ‘DATA’ and so on. For Control record, the value is specified under control tab. Only the first field of a file is used for Record Type.

Record Delimitter The records are stored differently in different operating systems. The options available are: MS-DOS

Unix

No Record Delimiter

Other

With respect to the example, select Unix.

Skip number of records

The number of records to be skipped is to be mentioned. The records are skipped from the top. Generally, this is to skip the Headers.

Text Qualifier It’s a character that identifies a text. This is used when some characters exists within a text. Generally, double quotes are used, prefixed and suffixed with a text. This is optional.

Decimal separator This mentions up to which decimal digit you want to view the result in.

Record Type Length This is applicable only for Control records that are of Fixed length. The length of the record type value to pick up the correct record. For example, if control record is “DATATotal Records400” .and DATA is the Record type, the length is ‘4’.

Control Name Length Based on the above example, Control name is “Total Records”. Hence, the Control Name Length is ‘13’.

DATA MAPPING



Fields Description


Control Value Length Based on the above example, the Control value is 400. Hence, the length of the control Value is ‘3’

When Separate File is selected as No.

Record Type Code Used to uniquely identify a record within a file. Many times, Bank provides file having data and control record within same file. In that case, in order to distinguish between data record and control record, the first field is Record Type. It has a specific value to identify that. Here, specify the value that identifies the Data. Values can be ‘DATA’ and so on. For Control record, the value is specified under control tab. Only the first field of a file is used for Record Type.

Control Value Length Based on the above example, the Control value is 400. Hence, the length of the control Value is ‘3’

Control Name Length Based on the above example, Control name is “Total Records”. Hence, the Control Name Length is ‘13’.

Controls

Control Name Specify the name of the control.

Aggregation Method Select either Aggregation Method or Count. The supported aggregation methods are: Min

Max

Average

Sum

Aggregation Column Name

Select the column on which aggregation method is applied. Note: For count, no column needs to be selected.

Threshold Type This field is optional. There are two selections of threshold, percentage or absolute. If percentage is selected the reconciliation difference in percent will be matched against this threshold value. If absolute is selected the absolute difference in percent will be matched against this threshold value.

Threshold Value Specify the difference value in percent or absolute.

DATA MAPPING



7.3.2.3 Transformation Tab

Fields Description


Transformation Type This is a drop-down listing different types of transformation supported. Currently, only Aggregation is supported.

If the Transformation Type is selected as None:

Derived Data Elements

Name Name of the derived field in EDD. Note: Field name should not be more than 30 characters.

Type This shows the Data type of the field. Example: Varchar2, Number, Date, and so on.

Expression On selecting ‘Add option’, the Specify Expression window appears. Here, you can select the required Entities, Functions and Operators. That is, you can write your own expression. Enter the field name and click OK. Now the newly created field name will be listed.

If the Transformation Type is selected as Aggregation:

Derived Data Elements

Name Name of the derived field in EDD. Note: Field name should not be more than 30 characters.

Type This shows the Data type of the field. Example: Varchar2, Number, Date, and so on.

Expression On selecting ‘Add option’, the Specify Expression window appears. Here, you can select the required Entities, Functions and Operators. That is, you can write your own expression. Enter the field name and click OK. Now the newly created field name will be listed.

Aggregation Properties

Group By This is available when Aggregation is selected.

Having This is available when Aggregation is selected.

7.3.3 Defining an External Data Descriptor To define a new EDD from the External Data Descriptor window, perform the following steps:

1. Click the Add button from the tool bar. The External Data Descriptor Specifications window appears.

2. In the External Data Store Name section, select Data Source from the drop down list. The Data Source is the Source you had created. In this example it is, DRM_SRC_FILES. The values in Defining an External Data Store example are used. The description comes up automatically.

DATA MAPPING



NOTE The fields in the External Data Descriptor Specification Details section changes as per the Source Code selected.

For HDFS data, the working date format is yyyy-MM-dd.

For Hive table, the working date format is yyyy-MM-dd.

For Sybase source database, date data type is not supported. It has to be timestamp.

3. You can click button to refresh the data elements.

4. Enter the values in the fields as described and tabulated in the Fields and their Description section.

5. If data needs to be reconciled post loading, then control has to be defined for the EDD by clicking

on Control Tab. In this version, only Number of records control is possible.

6. To transform the EDD, click the Transformation tab.

DATA MAPPING



7. You can add derivation to few data elements of the EDD.

a. Create derived data elements by clicking Add button.

b. To edit the derived data element, click Edit button. The Expression window appears.

c. Expression can be specified using the data elements defined in the Data tab and functions.

d. To delete the derived data element and click Delete button.

8. Click the Transformation tab and select the Transformation Type.

a. Select Aggregation, and click Edit button to view Expression Window.

DATA MAPPING



b. Specify the Group by clause and Having expression, if applicable.

c. Define Derived Data Elements for field to be aggregated under previous tab.

NOTE It is applicable for only in the case of DB2, Oracle, Sybase, My SQL and Hive.

9. Click Save.

7.3.4 Modifying and Viewing an External Data Descriptor You can edit or view an existing EDD, other than the EDDs, which are in published status.

NOTE You cannot edit the EDD if the EDD is in published state.

To edit or view an EDD, perform the following steps:

1. Select the required EDD from the EDD Summary.

2. The details of the selected EDD is displayed. You can modify or view the details.

3. Update the required details.


5. Click Save as Draft in case you wish to save it and update it later. The status shows as draft.

7.3.5 Deleting an External Data Descriptor This option only checks the higher order object. That is, if the order has dependency, you cannot delete unless the dependency is removed.

For example, assume EDD is used in Connector. Then, unless the Connector is deleted, the used EDD cannot be deleted.

To delete an existing EDD, perform the following steps:

1. On the EDD Summary, click Delete button. A confirmation dialogue appears.

2. Click Yes. The EDD details are deleted.


a. If the EDD is not in published state,

b. If it is not used by any object.

DATA MAPPING



You can unpublish an EDD only when all the following conditions are met:

1. The EDD is in published state.

2. All the connectors using the EDD are in unpublished state.

To unpublish an EDD, perform the following steps:

1. Select the required EDD from the EDD summary. The details of the selected EDD are displayed.

2. Click Unpublish.

NOTE EDD gets published automatically by the system whenever the higher objects (Connector) which are using it, are published.

7.3.6 Dependency

As the name suggests, on clicking the Dependency icon , it lists where the entire parent EDD has dependency.

7.3.7 Search and Filter The Search and Filter option in the user interface helps you to find the required information. You can enter the nearest matching keywords to search, and filter the results by entering information on the search box. You can search for an EDD using either the name, description, status or type.

For example, enter EDD key word as ‘AATB_ACCT’ in the search box. The entire EDD name with AATB_ACCT is listed.

DATA MAPPING



7.3.8 Parameters in EDD Definition While defining an EDD, parameter can be used as a placeholder in a data filename.

For example:

Consider a table with two columns, such as Account number and Balance.

Account Number Balance

A1 1000

A2 1000

A3 1000

A1 1000

A2 1500

A3 1500

In the above example, a customer has three accounts (A1, A2 and A3).

The customer has deposited different amounts on January 1st and 2nd 2014. The csv data files can be created for those two dates as follows:

• The account transaction for January 1st 2014 is saved as

• td_contracts_/01012014/.csv

• The account transaction for January 2nd 2014 is saved as

• td_contracts_/01022014/.csv

If a parameter, MISDATE is defined as a runtime, this can be used as a placeholder that substitutes date in mmddyyyy format. That is, the data filename can be mentioned as td_contracts_%#MISDATE%.csv. When this file is called, it substitutes the date in the file name, dynamically, in the run time.

Parameters Data Types need not always be RunTime. They can be Constants or values like Current Date, which can also be used to substitute a value in a data filename.

DATA MAPPING

CONNECTORS


7.4 Connectors Connector allows mapping one or more External Data Descriptor with Application Data Interface. It allows mapping of one or more ADI with EDD as well, in case of extract type connector.

There are pre-built Connectors for Oracle applications such as, Flexcube, Oracle Banking Platform, and so on. For other applications, you need to define Connectors for your EDDs.

7.4.1 Understanding the Summary Screen To understand the summary screen, perform the following steps:


2. Click Connector.

The Connector window appears.

DATA MAPPING

CONNECTORS


3. The summary screen list down all the connectors that are defined in the set up. It displays the entire insert and extract type connector details. It gives the information about the number of parameters, EDS, EDDs, and ADI used in a specific connector.

4. Click Export to Excel icon. The Select Export Columns window appears.

5. Select the required ID and click Download. The list of connectors is exported to an excel sheet

with Connector IDs and Connector Name This lists both insert and extract type connector details.

6. Click to view the connectors in card view. It gives the information about the number of parameters, EDS, EDDs, and ADI used in a specific connector.

In case, the connector is published:

DATA MAPPING

CONNECTORS


7. Click to view the connectors in list view.

In case, the connector is published:

8. Click button to create a connector. For more information, refer section Creating a Connector.

DATA MAPPING

CONNECTORS


7.4.2 Creating a Connector Connector allows mapping one or more External Data Descriptor with Application Data Interface. It allows mapping of one or more ADI with EDD as well, in case of extract type connector.

There are pre-built Connectors for Oracle applications such as, Flexcube, Oracle Banking Platform, and so on. For other applications, you need to define Connectors for your EDDs.

This section includes the following sections:

7.4.2.1 Icons and Description To create a connector you must understand each of the icons at the beginning. While creating a Connector, the icons that are displayed are explained as follows:

Icon Description

Click this icon to view the list of all External Data Descriptors created in the setup. You can drag the desired EDD on the canvas.

Click this icon to view the list of all ADIs created in the setup. You can drag the desired ADI on the canvas.

Click this icon to open the mapping window. You can map the source column to target column in the window.

This component is used for defining join between two entities. Click this icon to open the window where you can define the join condition between two entities.

This component is used for defining filter a given entity. Click this icon to open the window where you can define the filter condition.

This component is used for defining look up condition. Click this icon to open the window where you can define the join condition between two entities.

This component is used for defining Derived column. Click this icon to open the window where you can define an expression, which can be mapped to the target column.

This component is used for transforming flattened hierarchy entities into parent child hierarchy entities.

This component is used for Transpose (Rows to Columns) for a given entity. Click this icon to open the window where you can define the pivot data element and the new columns, which will be transposed from multiple rows of source entity.

This component is used for Transpose (Columns to rows) for a given entity. Click this icon to open the window where you can define the unpivot data element and new rows which will be transposed from columns of source entity.

DATA MAPPING

CONNECTORS


Icon Description

This component is used for defining group by and having clause for Aggregation. Click this icon to open the window where you can define a group by and having clause for aggregation.

Click this button to remove all the nodes added into the canvas.

This appears on the connector window when the connector is published and is opened in view mode. The connector is not editable.

This appears on the connector window when the connector is not published. The connector is editable.

7.4.2.2 Creating a Connector for Loading Data into OFSAA To create a Connector for loading data into OFSAA perform the following steps:

1. On the Connectors Summary window, click Insert tab.

2. Click Add button. The New Connectors Definition window appears.

3. To define a connector, you must have source with EDD and a target, which is ADI.

4. Click Source to select the required EDDs.

5. Here, you can filter your selection based on the EDS selected. The EDD node’s color depend on the source system type.

DATA MAPPING

CONNECTORS


For example:

File types are in blue.

Oracle types are in red.

HDFS types are in orange.

6. In case you select ‘OBP_STAGE_SRC’ as the EDS, it displays the EDDs for that particular EDS selected.

7. Click the search icon to search for a particular EDD. You can select multiple EDS.

8. Select the required EDD and drag it to the canvas.

9. Click Target. Here you can filter ADIs based on the application selected.

10. Click the search icon to search for a particular ADI.

11. Select the required ADI. Drag it to the canvas and then link the input and output nodes.

12. Click the input white circle. The anchor symbol appears. Drag and drop the line to link it to the required component.

DATA MAPPING

CONNECTORS


13. At any given time, you can right-click the node to either delink or remove inlinks / outlink or delete

a node.

14. To edit or view the properties, on the Connector window, click .

15. In Connector Details, enter the name and description for the connector.

DATA MAPPING

CONNECTORS


16. In Pre Load Options, select the truncate option to be defined in the target. Selecting truncate will remove data from table as per the truncate option specified.

Select No, if you do not wish to truncate the table prior loading.

In case you select Partial Truncate, provide Partition Name. The parameters name can be provided here. In case, you want to truncate a partition, the option partial truncate must be selected. Specify the partition to be truncated prior load.

NOTE In case of multi target load, truncate type has to be same for all target however; truncate expression may vary.

Select Full Truncate to fully truncate. Here no expression is required. In case you want to truncate the entire table, the full truncate option must be selected.

Select Selected Rows to truncate on the selected expressions. In case you remove specific rows, the selected rows option must be selected. Specify the filter condition for the rows to be deleted. Specific rows are removed from the table prior load.

NOTE If OFSAA is hosted on Big Data environment then Truncate with Selected Rows option is not supported.

DATA MAPPING

CONNECTORS


Click Edit icon to filter the selected rows.

Select the required entity and click Validate. This validates the expression.

Click Ok once the expressions are selected.

In the above example, truncate details are selected for Account Address.

17. In Properties, select the Default Properties, File Properties and Table Properties in case you have selected a default type or file type or a table type respectively.

NOTE Refer section Connector Properties section for more details on the properties.

7.4.2.3 Creating a Connector for Extracting Data from OFSAA To create a Connector for extracting data from OFSAA perform the following steps:

1. On the Connectors Summary window, click Extract tab.

2. Click Add button. The Select OFSAA Module window appears.

DATA MAPPING

CONNECTORS


3. Choose Dataflow View and select the module you want to extract data from and click Ok. The

New Connectors Definition window appears.

4. To define a connector, you must have source with EDD and a target, which is ADI.

5. Click Source to select the required ADIs.

6. Here, you can filter your selection based on the ADI selected. The ADI node’s color depend on the source system type.

7. Click the search icon to search for a particular ADI.

8. Select the required ADI. Drag it to the canvas and then link the input and output nodes.

For example:

DATA MAPPING

CONNECTORS


File types are in blue.

Oracle types are in red.

HIVE types are in brown.

9. Click Target. Here you can filter EDS based on the application selected.

10. In case you select ‘OBP_STAGE_SRC’ as the EDS, it displays the EDDs for that particular EDS

selected.

11. Click the search icon to search for a particular EDD. You can select multiple EDS.

12. Select the required EDD and drag it to the canvas.

13. Click the input white circle. The anchor symbol appears. Drag and drop the line to link it to the required component.

14. At any given time, you can right-click the node to either delink or remove inlinks / outlink or delete

a node.

DATA MAPPING

CONNECTORS


15. To edit or view the properties, on the Connector window, click .

16. In Connector Details, enter the name and description for the connector.

17. In Pre Load Options, select the truncate option to be defined in the target. Selecting truncate will remove data from table as per the truncate option specified.

Select No, if you do not wish to truncate the table prior loading.

DATA MAPPING

CONNECTORS


In case you select Partial Truncate, provide Partition Name. The parameters name can be provided here. In case, you want to truncate a partition, the option partial truncate must be selected. Specify the partition to be truncated prior load.

Select Full Truncate to fully truncate. Here no expression is required. In case you want to

truncate the entire table, the full truncate option must be selected.

Select Selected Rows to truncate on the selected expressions. In case you remove specific rows, the selected rows option must be selected. Specify the filter condition for the rows to be deleted. Specific rows are removed from the table prior load.

NOTE If data is extracted into Big Data environment then Truncate with Selected Rows option is not supported.

Click Edit icon to filter the selected rows.

Select the required entity and click Validate. This validates the expression.

Click Ok once the expressions are selected.

In the above example, truncate details are selected for Account Address.

18. In Properties, select the Default Properties, File Properties and Table Properties in case you have selected a default type or file type or a table type respectively.

DATA MAPPING

CONNECTORS


NOTE Refer section Connector Properties section for more details on the properties.

7.4.3 Understanding the Components

7.4.3.1 Using Filter To use the filter component, perform the following steps:

1. Drag and drop the Filter component on the canvas to define a filter on an entity. For example, EDD (insert connector) / ADI (extract connector)

2. It accepts input only from an entity and it can have only one output.

3. In case you have multiple entities selected, and you want to have filter for more than one entity, then you must select as many number of filters, connect to the respective entity, and then define their expressions.

For example, to add filter to three entities, drag three filters.

4. At any given time, right-click the filter component to either delink or remove inlinks / outlinks or

delete the filter component.

DATA MAPPING

CONNECTORS


5. Double-click filter component. The Filter Expression window appears:

The selected entities and parameters are displayed in the filter expression screen.

6. Specify the required filter expression using columns and parameters.

7. Click Validate in case you wish to verify the correctness of the SQL expression.

8. Click OK.

NOTE You do not need to add ‘WHERE’ clause for the filter.

1. For File data loading, use filter expression of Number type along with single quotes. For example: N_DRAWN_AMOUNT ='40000'.

2. For Date field refer To_CHAR function for comparison.

3. Parameters can also be used in filter expression. Date format must be a valid SQL date format.

For Example:

[EDD_GL_DATA].[EXTRACTION_DATE] = TO_DATE(#DIHDEV.MIS_DATE,’dd-MM-yyyy’)

9. If Source type is Hive, the filter expressions must conform to the following restrictions:

Expression must be valid HiveQL

DATA MAPPING

CONNECTORS


Does not include Oracle built in or user defined functions

Does not include Sub queries

Includes Hive built in functions only

Parameters can also be used in filter expression. MISDATE can also be passed dynamically so that it is loaded from Batch Execution Screen. The date format specified must be valid Hive Date format i.e. yyyy-MM-DD

For Example: Filter Expression in Connector:-[EDD_GL_DATA].[EXTRACTION_DATE] = #DIHDEV.MIS_DATE

MISDATE Parameter in Batch:

MISDATE=$MISDATE:yyyy-MM-dd

7.4.3.2 Using Join To use the join component, perform the following steps:

1. Drag and drop the Join component on the connector window to link multiple entities. For example, EDDs (insert connector) / ADIs (extract connector)

2. The Join component accepts input from two entities.

3. To join more than two entities, drag another join component. Link the output of the first join to the

input of the second join and then connect the other entities. You can repeat this for multiple entities. Select the Source Entity and click Ok.

DATA MAPPING

CONNECTORS


4. At any given time, right-click the join component to either delink or remove inlinks / outlinks or delete a join component.

5. Double-click the join component to define a join condition. The Join window appears:

Here you see the selected entities in the left and right tab.

6. You can drag and reorder the left and right tab to choose the right/left entity in a join condition.

7. To join entities, select column from left and right tab and click Add Join icon. This displays the joined entities. You can join multiple entities.

8. To remove two joined conditions, select two columns from left and right tab and click Remove

Join icon. The joined condition is removed from the list.

9. Click Reset in case you wish to reset all the joined condition.

DATA MAPPING

CONNECTORS


10. Click Ok once you are completed.

NOTE This creates an inner join between the connected EDDs.

7.4.3.3 Using Lookup To use the lookup component, perform the following steps:

1. Drag and drop the Lookup component on the canvas to define a filter on an entity. For example, EDD (insert connector) / ADI (extract connector).

2. You can lookup values from an entity using this component.

NOTE Lookup component accepts input from two entities. One from Value Entity and the other one from Lookup Entity.

3. At any given time, right-click the lookup component to either delink or remove inlinks / outlinks or

delete a lookup component.

4. Double-click the lookup component to define a lookup condition. The Lookup window appears:

Here you see the connected entities in the left and right tab.

DATA MAPPING

CONNECTORS


5. The entity that is on the right side of the window is the look up entity. You can change the lookup

entity by moving it to the right side. The “LookUp Entity” field displays the entity specified for lookup.

6. To specify lookup condition, select data elements from left and right entities and click Add Join

icon. Lookup condition is displayed as follows:

7. To remove a lookup condition, select data elements from left and right entities and click Remove

Join icon. The lookup condition is removed from the list.

8. Click Reset in case you wish to reset the lookup condition.

9. Click Ok.

NOTE This creates a left outer join between the connected entities.

7.4.3.4 Using Aggregation for an EDD To use the aggregation component, perform the following steps:

1. Drag and drop the Aggregation component on the canvas to define an aggregation on an EDD.

2. It accepts input only from an EDD and it can have only one output.

3. In case you have multiple EDDs to be aggregated then you must select as many number of aggregation components, connect to the respective EDD, and then define their group by and having clauses.

For example, to add aggregation to three EDDs, drag three aggregation components.

DATA MAPPING

CONNECTORS


4. At any given time, right-click the aggregation component to either delink or remove inlinks /

outlinks or delete aggregation component.

5. Double-click the aggregation component to define an aggregation condition. The Aggregation

window appears:

Here you see the selected EDD under the entity tab.

6. Select the group by columns and specify expression for the having clause.

7. Click Reset in case you wish to reset all the aggregation condition.



7.4.3.5 Using Aggregation for entire Dataset To use the aggregation component, perform the following steps:

1. Drag and drop the Aggregation component on the canvas to define an aggregation on entire dataset.

DATA MAPPING

CONNECTORS


2. It accepts input only from mapping component or filter that is connected to mapping component.

3. At any given time, right-click the aggregation component to either delink or remove inlinks /

outlinks or delete aggregation component.

4. Double-click the aggregation component to define an aggregation condition. The Aggregation

window appears:

Here you see the selected EDD under the entity tab.

DATA MAPPING

CONNECTORS


5. Select the group by columns and specify expression for the having clause.

6. Click Reset in case you wish to reset all the aggregation condition.



7.4.3.6 Using Transpose (Rows to Columns) for an EDD To use the Transpose (Rows to Columns) component, perform the following steps:

1. Drag and drop the Transpose (Rows to Columns) component on the canvas to define a Transpose (Rows to Columns) component on an EDD.


3. In case you have multiple EDDs selected, and you want to have Transpose (Rows to Columns) component for more than one EDD, then you must select as many number of Transpose (Rows to Columns) components, connect to the respective EDD, and then define their expressions.

NOTE The output can be connected to Join, Lookup and Mapping.

For example, to add Transpose (Rows to Columns) component to three EDDs, drag three Transpose (Rows to Columns) components.

DATA MAPPING

CONNECTORS


4. At any given time, right-click the Transpose (Rows to Columns) component to either delink or remove inlinks / outlinks or delete a Transpose (Rows to Columns) component.

5. Double-click the component to transpose the entity rows into columns. The Transpose Row to

Column window appears.

Here you see the selected EDD and parameters.

6. Specify the pivot data element to transpose rows into columns.

7. Specify the Row Value – Transposed Column – Expression combination. Ensure to have a minimum of two combinations.

8. Click Review to review the transformation. The Review Transformation window appears displaying the sample of the transformation data.

DATA MAPPING

CONNECTORS


9. Click Ok.

7.4.3.7 Using Transpose (Columns to Rows) for an EDD To use the Transpose (Columns to Rows) component, perform the following steps:

1. Drag and drop the Transpose (Columns to Rows) component on the connector window to define a Transpose (Columns to Rows) component on an EDD.


3. In case you have multiple EDDs selected, and you want to have Transpose (Columns to Rows) component for more than one EDD, then you must select as many number of Transpose (Columns to Rows) components, connect to the respective EDD, and then define their expressions.

NOTE The output can be connected to Join, Lookup and Mapping.

For example, to add Transpose (Columns to Rows) component to three EDDs, drag three Transpose (Columns to Rows) components.

4. At any given time, right-click the Transpose (Columns to Rows) component to either delink or

remove inlinks / outlinks or delete a Transpose (Columns to Rows) component.

5. Double-click the component to transpose the entity columns into rows. The Transpose Column to

Row window appears.

Here you see the selected EDD and its parameters.

DATA MAPPING

CONNECTORS


6. Specify the Unpivot Data Element to transpose columns into rows.

7. Specify the Header Column Name and Value Column Name.

8. Specify the Column Value (Header column) and Expression Pair (Value column) for each transposed rows. Ensure to have a minimum of two pairs.

9. After specifying the Unpivot Data Elements, click Auto Transpose. This will transpose columns into rows based on the unpivot data elements selected.

10. You can also click to drag and drop the columns.

11. Click Review to review the transformation. The Review Transformation window appears displaying the sample of the transformation data.

DATA MAPPING

CONNECTORS


12. Click Ok.

DATA MAPPING

CONNECTORS


7.4.3.8 Using Derived Column To use the Derived Column component, perform the following steps:

1. Drag and drop one more Derived Column component on the canvas.

2. Connect the Derived Column to the mapping.

NOTE The output must be connected to the mapping.

3. At any given time, right-click the expression component to either delink or remove outlinks or

delete an expression component.

4. To define the expression, double-click the Derived Column component. The Derived Expression

window for Derived Column appears:

Here you see the selected EDDs in the left tab.

DATA MAPPING

CONNECTORS


5. Specify the name and expression for the Derived Column.


7. Click Ok.

7.4.3.9 Using Mapping 1. Double-click Mapping component. The Mapping window appears.

NOTE The input and output for the mapping component must be connected before specifying the mappings.

2. The mapping window displays the EDDs and ADIs and their respective data / derived data elements.

3. Click a Data Element under Source, Attribute under Target and then click Map icon. On the RHS column mapping is displayed.

4. The following are the validations done for the mapping

a. Data Type Validation

b. Data Length Validation

DATA MAPPING

CONNECTORS


c. Data Precession Validation

5. In case validation is successful, it displays Successful icon next to the mapping.

6. In case any of the above (step 4) validation fails, it displays Warning icon next to the mapping.

7. At any given time, you can select Unmap icon to unmap the source and target.

8. Click Auto-Map icon to auto map a source and target.

NOTE Auto-mapping is done by matching logical / physical column name of both source or target.

9. In the Source column, click filter icon. Enable it to view the unmapped items.

NOTE The mapped columns are displayed in red.

10. In the Target column, click the filter icon. Enable it to view the unmapped, mandatory and valid for applications.

11. Under the target column, you can hover on each item to see the details. It provides the

description, length and scale information.

12. Click Search icon to search for a column name under Source or Target column list.

13. Click Delete to delete all the mappings. You can also delete individual mappings by selecting the cross symbol next to the column mapping.

14. Click Import Mapping icon to import mapping excel. Choose a mapping excel from the file browser.

DATA MAPPING

CONNECTORS


15. Click Export Mapping icon to export the mapping information. This downloads an excel file.

16. Click Search icon to search for a column mapping. You can search for an item based on source column name, target column name, source or target entity or a remark.

7.4.3.10 Using Flatten table to PC hierarchy transformation for an EDD

1. Drag and drop PC Hierarchy component on the canvas.

2. Connect the Derived Column to the mapping.

NOTE The output must be connected to the mapping.

3. At any given time, right-click the expression component to either delink or remove outlinks or

delete an expression component.

4. To define the expression, double-click the Flatten table to PC hierarchy. The Flattened Table to

Hierarchy window appears.

DATA MAPPING

CONNECTORS


5. Choose the Hierarchy Type. The types of hierarchy supported are Balanced, Ragged and

Skipped. Click the help icon to view the details and understand how the hierarchies are defined.

6. Specify the Number of levels in the hierarchy. This field accept only numbers.

7. Specify the Parent Node Column name and Child Node Column name, which are used in mapping.

8. Select the Key Elements from the drop-down list.

9. Select all nodes. You can change the date and or other details from the drop-down list.

10. Click Review to view the transformation changes.

11. Click Ok.

DATA MAPPING

CONNECTORS


7.4.4 Connector Properties The properties by default have some values. For every connector, it needs to be reviewed. Generally, no change is required.

Fields Applicable Values

Description Default

Value

Loading mechanism

External Table

SQLLDR

There are two options External Table and SQLLDR. External Table - If the loading

mechanism is selected as External table, then the file-landing zone must be located/mounted on database server.

SQLLDR - This option is only applicable when OFSA is hosted in Oracle DB. The file-landing zone must be located or mounted on the server where ODI agent is running. Oracle Database Client must be installed in the server where ODI agent is running.

Note: If the loading mechanism is selected as External Table, the file should be located in the same place as the database server. In case target database type is HDFS, only external table option is enabled. In case target database type is Oracle, provide CREATE DIRECTORY role to the atomic schema. In addition, the path/folder used in the directory should be having read, write permission.

SQLLDR

DIRECT True

False

Direct path load of SQLLDR. Values can be True and False.

TRUE

Parallel True

False

Parallel option. True means the loading happens with parallel option and False means it happens in sequential way.

TRUE

Degree of Parallel

Integer Decides the degree of parallelism. The Values specified here must be a number.

0

No: of Errors Integer Shows the number of errors allowed for the SQLLDR and External Table to proceed. That means single record fails the job fails. The Values specified here must be a number.

0

DATA MAPPING

CONNECTORS



Description Default

Value

Maximum Discard

Integer Discarded records allowed for SQLLDR. This needs to be set to a very high number when using multiple subtypes under an ADI. That means it is multiple targets.

1

XML date Format

Valid XML Date format In this field, you can define the format of the XML Date. Example: MMDDYYYY.

MMDDYYYY

Avoid Partition Exchange

Yes

No

The property option decides whether Partition Exchange method is used during data load. Note - Sometimes because of some unknown reason, if the connector execution fails during partition exchange, then it can be avoided by choosing the option as "Yes".

No

Do you want to use Data Pump?

Yes

No

There are two values ‘Yes’ and ‘No’. If the value is ‘Yes’, it indicates that the Oracle Database source is loaded into OFSAA using the Data Pump method. Alternatively, the standard way of using DBLink method is followed. Note: The following access is required for data pump option. Grant create any directory to Source schema Grant create any directory to target schema Grant execute on DBMS_FILE_TRANSFER to target schema Grant execute on utl_file to source schema

No

Source and Target in Same Environment?

Yes

No

This parameter is used only if Data Pump is used. If the value is, ‘Yes’ then file transfer step is not performed during loading. Alternatively, it will transfer file from source to target folder-using DBLink.

Yes

Source Dump Location

/src/tmp Specify a folder/path that is accessible by Source Oracle Database to create the dump file.

/src/tmp

Target Dump Location

/target/tmp Specify a folder/path that is accessible by Target Oracle Database to read the dump file.

/target/tmp

DATA MAPPING

CONNECTORS



Description Default

Value

Number of Splits for Dump File

Integer Specify a number to transfer the files in parallel chunks. For example, three, indicates that dump file will be split in three and transferred separately. This is to improve performance of file transfer. The Values specified here must be a number.

0

Effective Dated Key for Result Area?

Yes

No

This parameter is used for loading data into the result area. It decides whether it will perform lookup into dimension for latest record, or effective dated record. To get the latest record, it appends f_latest_record_indicator =’Y’ and for effective dated it appends mis_date between d_record_start_date and d_record_end_date. Note: For extraction data or any date attribute, effective dating does not work. If this parameter is selected as “Yes” and any date field needs to be loaded into Extraction date or any other date field, then do not rely on surrogate key generation. Use derived column and enter the value in the format “YYYYMMDD” as a number.

No

Do you want to use DBLink?

Yes

No

This parameter is used to specify the source database connection method. There are two values ‘Yes’ and ‘No’. If the value is ‘Yes’, it indicates connection source database will be created using DBLink method. If the value is ‘No’ , it indicates connection to source database will be created using JDBC url

No

Hive Date Format

yyyy-MM-dd This parameter is used to specify the date format for date columns in Hive source.

yyyy-MM-dd

DATA MAPPING

CONNECTORS



Description Default

Value

Do you want to use Big Data SQL?

Yes

No

This parameter is used to specify loading mechanism from HIVE source. If you have selected Yes, Oracle Big Data SQL is used. Note: The following are the pre-requisites for the above method:

• The Oracle Big Data SQL is installed and configured on Oracle Big Data Appliance and Database (OFSAA) machines.

• In OFSAA database, a directory named DEFAULT_DIR is created.

• The folder/system path specified for the above directory must be accessible to database (OFSAA).

For more details refer: https://docs.oracle.com/bigdata/bda45/BDSUG/installing.htm#BDSUG-GUID-5CDA332E-8CB8-42CC-8922-09295E22B0E5 If you have selected No, Oracle Loader for Hadoop is used. Note: The following are the pre-requisites for the above method:

Oracle Loader of Hadoop connectors are installed and configured in the source system.

ODI agent is also configured for OLH.

For more details refer: https://docs.oracle.com/cd/E72987_01/odi/odi-big-data/setup.htm#ODIBD117

No

7.4.5 Modifying and Viewing a Connector To edit or view a connector, you can perform the following steps:

1. To edit or view a connector, you can select the required connector from the connector summary.

NOTE You can edit an existing connector, other than the connectors, which are in published status. To edit a connector, which is in publish status, you must first un-publish it and then open it.

2. The connectors created opens in their respective view mode.

https://docs.oracle.com/bigdata/bda45/BDSUG/installing.htm#BDSUG-GUID-5CDA332E-8CB8-42CC-8922-09295E22B0E5




https://docs.oracle.com/cd/E72987_01/odi/odi-big-data/setup.htm#ODIBD117

https://docs.oracle.com/cd/E72987_01/odi/odi-big-data/setup.htm#ODIBD117

DATA MAPPING

CONNECTORS


3. The details of the selected connector is displayed. You can modify or view the details.

4. Connector Name cannot be edited. Update the other required details.


6. For published connector, if you wish to make any changes click ‘Unpublish’. The ‘Unpublish’ option clears ODI metadata that has been created during publish. Update the required changes and then click Publish. The updated changes are synced in ODI.

7.4.6 Copying a Connector To copy an existing Connector, perform the following steps:

1. Click Copy button for the required connector. A Save As dialog box appears.

Depending on the view in which the original connector is created, the copied connector will have the same view.

2. Enter the name and description.

3. Click Save. The Connector details are saved with a new specified connector name. The existing connector remains unmodified.

7.4.7 Deleting a Connector To delete an existing Connector, perform the following steps:

1. Click Delete button for the required connector. A confirmation dialogue appears.

2. Click Yes to delete a connector. The Connector is deleted. In case you do not wish to delete click

No.

DATA MAPPING

CONNECTORS


7.4.8 Search and Filter The Search and Filter option in the user interface helps you to find the required information. You can enter the nearest matching keyword to search, and filter the results by entering information on the search box. You can search for a connector with either the name, description, or status of the connector.

For example, enter the key word as ‘CON_DRM_GL’ in the search box. All the connector names with 'CON_DRM_GL’ are listed.

You can sort the list by connector name or modified date (ascending or descending order).

7.4.9 Parameters in Connector Parameters are used while defining the mapping, between EDD to ADI. While mapping the ADI to EDD, the fields or columns within the ADI needs to be mapped to the fields in EDD. If there are no corresponding extracts in EDD, parameters can be used to identify the default values for certain ADI elements. In addition, parameters can be used while defining derived columns during mapping.

For example, if you want to use the Runtime MIS date as parameter then, it must be converted to a date first. The following is the expression:

To_char(to_date(#DIHDEV.MIS_DATE,’dd-MON-YYYY’),’MM’)

NOTE Runtime batch MIS date will be in string format. It needs to be converted to DATE. The date format used here should be a valid SQL date format.

PUBLISHING DIH METADATA

PUBLISH AND UNPUBLISH


8 Publishing DIH Metadata Publishing is a process of transferring the DIH metadata to Oracle Data Integrator (ODI). It converts the DIH metadata into ODI metadata by creating required ODI objects such as DataServer, Logical/Physical Schema, Datastores, Interface, Package and Scenario. You cannot edit any DIH object after publishing. In order to edit, you need to unpublish, make the necessary changes and then re- Publish.

NOTE The flow diagram represents the activities undertaken upon publish. No action is required apart from triggering the Publish task.

8.1 Publish and Unpublish You can batch publish/unpublish the selected connectors to ODI at any given point. You can also unpublish all the objects using this option.

To publish, perform the following steps:

1. Click icon in the DIH home screen to navigate to the Administration window.

Start Publish

Load EDDs and Load ADIs

Load Tables into Heap Memory

Overwrite/Rename the Source and Target Object on the ODI

Convert EDD and ADI into ODI Repository

Check if Source and Target available in ODI

FALSE

TRUE

Check if Interface available in ODI

Overwrite/Rename the Interface on the ODI

Convert Connector to Interface

TRUE

FALSE

Include Reconciliation ProcedureAdditional Control File Interface

Create Package




2. Click Publish/Unpublish. The Publish/Unpublish window appears.

3. When you select the option as Publish, all the saved and unpublished connectors are displayed.

4. Double-click the connectors or press Ctrl to select multiple connectors and click to move it to the next column.

NOTE

To deselect a connector click .




5. Click to select all the connectors.

NOTE

To deselect all the connectors click .

6. Click Publish after the items are selected. The status of the published connects are displayed. You can view the connectors, which are successfully connected or failed, under Status field.

7. When you select the option as Publish/Unpublish, all the published connectors are displayed.

8. Double-click the connectors or press Ctrl to select multiple connectors and click to move it to the next column.

NOTE

To deselect a connector click .

9. Click to select all the connectors.

NOTE

To deselect all the connectors click .

10. In case you select All Objects, then by default all the connectors and their corresponding EDD, EDS and parameters are unpublished.




11. Click Unpublish after the items are selected. The status of the selected connectors are displayed. You can view the connectors, which are successfully published/unpublished or failed under Status section.

12. Search the connector to check the status of the publish/unpublished process.

METADATA OBJECT MIGRATION

PERFORMING OBJECT MIGRATION


9 Metadata Object Migration Metadata objects created by DIH may also be migrated leveraging this mechanism, with the list including DIH Parameters, EDS/EDD/Connector definitions and batches.

Using the object migration utility, you can migrate (export/ import) DIH metadata objects across different setups. You can specify one or more objects within an object type or within multiple object types. You can choose from where the object migration utility reads the data, that is, from CSV files or OBJECTMIGRATION.xml file. For migrating objects using CSV files for migrating objects using OBJECTMIGRATION.xml file, refer section Command Line Utility to Migrate Objects from OFSAAI user guide.

The following section details how such Metadata Object migration can be achieved.


• Performing Object Migration

• Exporting Object from Source Environment

• Importing Object into Target Environment

9.1 Performing Object Migration Offline Object Migration is a two-step process:

• Exporting Objects from Source Environment

• Importing Objects into Target Environment

For both of these steps, refer sample file OBJECTMIGRATION.xml, which is also present at $MIGRATION_HOME/conf/ in the OFSAAI setup.

Assumptions and Pre-requisites for this process are listed as follows:

• Both Source and Target environment should be on the same data model version.

• Ensure that the source and target are at same DIH application level as well as at same model level before migrating objects from source to target.

• ADI refresh to be completed before this process.

• ODI settings are correctly specified in target environment prior to import of objects.

• Target data store refresh is also performed and successful.

• If exported object already exists in the target environment or an object with same name already exists in target environment then it will be overwritten with new definition details.

If exported Parameter already exists in target Environment, they will not be overwritten unless the new definition is of different Parameter Type. If exported EDS already exist in target Environment, they will not be overwritten unless the new definition is of different EDS Type.

• Any exported object, if already exists in target or an object with same name already exists in target then that object and all its dependent objects must be in unpublished state for migration to go through successfully.

Dependent objects for a Connector are EDS, EDD and Parameter. Dependent objects for EDD are EDS and Parameter. Parameter and EDS do not have dependent objects.

https://docs.oracle.com/cd/E60058_01/PDF/8.0.7.x/8.0.7.0.0/OFSAAI_User_Guide_8.0.7.0.0.pdf

https://docs.oracle.com/cd/E60058_01/PDF/8.0.7.x/8.0.7.0.0/OFSAAI_User_Guide_8.0.7.0.0.pdf


EXPORTING OBJECTS FROM SOURCE ENVIRONMENT


9.2 Exporting Objects from Source Environment Follow the below procedure to export object from source environment.

1. Replace placeholders of UserId, Infodom with Source UserId & Infodom.

2. For $Folder put segment name for the infodom provided above. Mention locale as ‘en_US’.

3. $FILE_NAME: Specify the file name which is created under "metadata/archive" folder. For example, mention ‘rules’ in place of $FILE_NAME and you get rules.dmp in archive folder.

Fail On error: Fail on any error occurs while restoring metadata. Mention ‘Y’ for Yes and ‘N’ for No.

OVERWRITE: If Metadata exist in the system then Overwrite while restoring metadata. Mention ‘Y’ for Yes and ‘N’ for No.

In Mode tag: mention EXPORT.

For FAILONERROR and OVERWRITE, it’s recommended to mention ‘Y’.

4. In OBJECT tag, mention “*” for Code property, to export all definitions. Else, for each definitions put equal number of OBJECT tag with map ID and version number in comma separated format.

Type: Use following for definitions:

70 for Parameter Definitions

71 for EDS Definition

72 for EDD Definition

74 for Connector Definition

5. The format for All OBJECTS tag is: <OBJECTS TargetFolder="DIHUSERS"><OBJECT Code="*" Type="73" /></OBJECTS>

6. For three definitions, OBJECTS tag is: <OBJECTS TargetFolder="DIHUSERS">

<OBJECT Code="123221" Type="73" />



</OBJECTS>

7. Execute $MIGRATION_HOME/bin/ObjectMigration.sh after providing executable permissions.

8. A file $FILE_NAME.dmp (rules.dmp) is created in $MIGRATION_HOME/metadata/archive

Move this file to $MIGRATION_HOME/metadata/restore folder. You can copy the file in the Target environment by creating "restore" folder under "metadata" directory (if not available).

9. Exporting definitions from Source environment is done successfully.

9.3 Importing Objects from Target Environment Follow the below procedure to import object to target environment.

1. Repeat 1-3 steps as followed in export mode.


IMPORTING OBJECTS FROM TARGET ENVIRONMENT


In Mode tag: mention IMPORT.

2. In OBJECT CODE property, mention “312321”.

NOTE Everything that is exported is imported. You cannot choose only certain definitions to move

3. Format for OBJECTS Tag is: <OBJECTS TargetFolder="DIHUSERS">

<OBJECT Code="312321" Type="73" /></OBJECTS>

4. Execute $MIGRATION_HOME/bin/ObjectMigration.sh after providing executable permissions.

5. Check DIH.log for logs. It provides details such as, number of definitions that have successfully moved and other errors. Importing objects to target environment is done successfully.

EXECUTING DIH METADATA

IMPORTING OBJECTS FROM TARGET ENVIRONMENT


10 Executing DIH Metadata A published DIH Connector can be executed in the following two methods:

• External Batch Management Tool: A DIH connector, with ETL job instructions are published into ODI, as a Package object. The ODI package object shares the same name as a DIH connector. These packages can be executed by generating a scenario out of it. External Batch Management tools can be used to execute these scenarios. Refer ODI documents for more details on generating and executing an ODI scenario.

• From OFSAA Batch: Creates a typical OFSAA Batch and executes it. For more information, refer to the AAI User Guide in OHC documentation library. The component is DIH Connector. You can select all the published connectors during task creation. DIH Connectors can be configured in Run Rules Framework (RRF) also with the same component.

NOTE • This can be done only on a published connector and not a saved connector.

• Currently, there is a limitation of passing parameter in Run Rule Framework for DIH connectors. Batch framework can be used in case runtime parameter needs to be passed.

• If the connector contains any Runtime parameters, they can be set in the Variables input field of the Task Definition Screen.

Example: MISDATE=’10-Jan-2015’

NOTE • In case of an upgrade, all the existing batches continues to perform as is without any update.

• In case you edit the existing task in a batch then it is mandatory to select an agent.

If there are multiple parameters, they can be passed by separating them with a comma.

Example: MISDATE=’10-Jan-2015’, BATCHID=22015

• MISDATE and BATCH ID can also be passed dynamically so that it is loaded from Batch Execution Screen as follows:

Example: MISDATE=$MISDATE:dd-MM-yyyy, BATCHID=$BATCHID

In the above example, the date format appended to MISDATE has to conform to Simple Date Format. If no date format is specified, the default date format used is yyyymmdd.

• If variables are being used as part of connector mappings or filter expressions, they should be passed within single quotes as follows:

Example: MISDATE=‘$MISDATE:dd-MM-yyyy’, BATCHID=‘$BATCHID’

• If the date format is expected in dd-MON-yyyy format, then in Batch Task it has to be specified in the following format. Note the difference in month format in the following example:

Example: MISDATE=‘$MISDATE:dd-MMM-yyyy’

• If parameter is used in connector filter expression for an EDD of source type Hive, date format is expected in yyyy-MM-dd format.

Example: MISDATE=‘$MISDATE:yyyy-MM-dd’

http://docs.oracle.com/cd/E60058_01/homepage.htm

EXECUTION HISTORY

VIEWING THE EXECUTION HISTORY OF A CONNECTOR


11 Execution History Execution History option, provides the status of the DIH Connector executions, number of records loaded, and error messages if any.

11.1 Viewing the Execution History of a Connector

Click icon in the DIH home screen to navigate to the Execution window.

Click Execution History. The Execution History window appears.

To view the Execution Histories of a Connector, perform the following steps:

1. Click Execution History. The Execution History summary window appears.

2. You can view the summary details of all the connectors that are executed in either Card view

or List view .

3. The Search bar helps you to find the connector for which you can view the executions. You can enter the nearest matching keywords to search, and filter the results by entering information on the search box. You can search for a connector using either the name or description.

4. Click the required Connector under Execution History. The Execution screen displays the list of executions for the selected connector with the latest displayed on the top.

EXECUTION HISTORY



Here you can view the number of records loaded, duration, start time, end time and the status of last execution.

NOTE Due to limitations in ODI, the number records while loading data into Hive might display as zero.

5. Click Show details icon to view details about the execution. The error details, failed source command and failed target command are displayed.

6. In case you wish to see the detailed execution report click Download Log icon. A zip file is downloaded containing the detailed log for the execution.


8. To view the Execution Load Details, click . The Data Load Info screen appears, with the execution details.

The Data Load Info grid displays the parameters and the runtime values for parameters and the EDDs with the file and table names of the EDD used for the execution.

In case the control definition is defined for file type EDD, the report of the expected and actual results loaded is displayed.

EXECUTION HISTORY



9. Click Execution Report icon under the Execution History. This screen displays the following details:

Batch Run ID of the latest execution

Number of records loaded in the last execution

Total number of controls defined for all EDDs in the connector

Number of controls breached for the last execution

Connector execution frequency

NOTE The frequency is classified as Daily, Weekly, Monthly, Bi-Monthly and Quarterly. This value is decided automatically depending on the frequency of the connector execution in the specified environment.

10. Click Report under Execution History to view the report for the required connector.

EXECUTION HISTORY



11. In case you wish to view the failed executions, select yes to view them.

METADATA BROWSER

CONNECTOR


12 Metadata Browser The DIH metadata/objects available in the Meta Data Browser (MDB) are:

• Connector

• Application Data Interface

• External Data Descriptor

12.1 Connector The connector link lists down the individual connectors.

When you click a connector name, it displays the definition of that connector.

• The name and description of the connector is displayed at the top.

• The Selected EDD grid, which shows the EDD, associated with the connector.

• The Selected ADI grid shows ADI for the connector.

• The Joins grid shows the join expression between EDDs if applicable.

• The Mapping grid shows mapping of the Left field with the Right attributes. Additionally, it displays the underlying physical columns.

• The Depends On grid shows EDD and ADI used in the connector.

• Click Export to PDF icon to generate PDF report for the Connector. The report will also contain all used EDDs and ADI details.

12.2 External Data Descriptor The External Data Descriptor window lists all the available External Data Store. Under each EDS, it has all EDDs. When you click a particular EDD, it shows the definition of the EDD, with the name and Description displayed at the top.

• The Properties sections shows properties such as file name, file format and so on.

• The Data Element sections show all the fields that are part of this EDD.

METADATA BROWSER



• Click Export to PDF icon to generate PDF report for the EDD.

12.3 Application Data Interface The Application Data Interface window shows all the attributes along with descriptions of the List of Values (LOVs). Click Export to PDF icon to generate PDF report for the ADI.

APPENDIX A: FAQ - USE CASES

LOADING DATA FROM ONE FILE INTO STAGING


13 Appendix A: FAQ - Use Cases

13.1 Loading Data from One File into Staging To load data from one file to Staging, follow the below steps:

1. Create an EDS of the type File.

2. Create EDD by selecting the pre-defined EDS. Provide all required information while creating the EDD. If post loading reconciliation is required then go to Control tab and provide a control record.

3. Create a Connector for loading data into staging. Select both the EDDs, establish a join, and click Lookup. If SQLLDR option is enabled, then the file must be available in the server where ODI agent is running. If External Table option is selected, then file has to be available in target database server.

4. Publish the Connector.

5. Execute the Connector.

13.2 Loading Data from Two Files Joined into Staging To load data from two files which are joined, to Staging, follow the below steps:

1. Create EDS of type File.

2. Create two EDD by selecting the pre-defined EDS. Provide all required information while creating EDD. If post loading reconciliation is required then go to Control tab and provide a control record.




13.3 Loading Data into Staging from File and Performing Lookup into a Table To load data from a file, to Staging, and perform lookup into a table, follow the below steps:

1. Create one EDS of type File and another EDS of type Database.

2. Create two EDD by selecting the predefined EDS. Provide all required information while creating the EDD. If post loading reconciliation is required then go to Control tab and provide a control record. Post loading reconciliation is only applicable for file type Data Loading.





LOADING DATA INTO STAGING FROM AN XML FILE


13.4 Loading Data into Staging from an XML File To load data to Staging from an XML file, follow the below steps:

1. Create an EDS of type XML.

2. Create EDD by selecting the predefined EDS. Provide all required information while creating the EDD. To get the XML structure, use XSD file. If post loading reconciliation is required then go to Control tab and provide a control record. The control record should be in a separate file.




13.5 Loading Data into Staging from EBCDIC To load data to Staging from EBCDIC, follow the below steps:

1. Create an EDS of type EBCDIC.

2. Create EDD by selecting the predefined EDS. Provide all required information while creating the EDD. To define the EBCDIC structure, use Cobol Copy book file. If post loading reconciliation is required then go to Control tab and provide a control record. The control record should be in a separate file.

3. Create a Connector for loading data into staging.



13.6 Loading Data from File into Multiple ADI/Subtypes To load data from File into multiple ADI/Subtypes, follow the below steps:

1. Create an EDS of type File.

2. Create EDD by selecting the predefined EDS. Provide all required information while creating the EDD. To define the file structure, you can use Excel template. If post loading reconciliation is required then go to Control tab and provide a control record.

3. Create a Connector for loading data into staging. Select multiple ADIs/Subtypes. Set filter against each ADI/Subtype selected to identify which record goes where.



13.7 Loading Data into Results from File To load data into Results from File, follow the below steps:


2. Create EDD by selecting the predefined EDS. Provide all required information while creating the EDD. To define the File structure, you can use Excel template for defining the columns. If post


LOADING DATA FROM HIVE


loading reconciliation is required then go to Control tab and provide a control record. The control record has to be in a separate file.

3. Create a Connector for loading data into results. Select EDD and ADI (appears only relevant for Result area). During mapping, code values that are available in file should be mapped to the attributes (that are internally number columns). While loading it converts the code value to surrogate key by performing a lookup in the related dimension table. Dimension data should be populated before this loading occurs using SCD process.



13.8 Loading Data from HIVE The following Jars needs to be copied to /ficweb/WEB-INF/lib:

1. commons-logging-1.1.3.jar

2. guava-15.0.jar

3. hadoop-auth-2.6.0-cdh5.8.0.jar

4. hadoop-auth.jar

5. hadoop-common-2.6.0-cdh5.8.0.jar

6. hadoop-common.jar

7. hive-common-1.1.0-cdh5.8.0.jar

8. hive-jdbc-1.1.0-cdh5.8.0.jar

9. hive-metastore-1.1.0-cdh5.8.0.jar

10. hive-service-1.1.0-cdh5.8.0.jar

11. HiveJDBC4.jar

12. httpclient-4.3.jar

13. httpcore-4.3.jar

14. libthrift-0.9.0.jar

15. slf4j-api-1.7.5.jar

For example, to connect to Cloudera Hive server with JDBC 4.0 data standards, specify “com.cloudera.hive.jdbc4.HS2Driver” as driver. Refer Cloudera document for more details about Cloudera JDBC drivers.

NOTE Contact support for more information.

13.9 Loading data into Partitioned Hive Table In case data is loaded into partitioned table in Hive, the following configuration must be done in ODI by logging into ODI studio:

1. Navigate to Designer Tab > Models > <INFODOM_NAME>.

2. Right click the target table to be loaded and click Open.

3. Navigate to Attributes Tab.

https://www.cloudera.com/documentation/other/connectors/hive-jdbc/latest/Cloudera-JDBC-Driver-for-Apache-Hive-Install-Guide.pdf


EXTRACTING DATA FROM STAGING INTO FILE


Select the “Partition by” check box for the column on which partition is defined.

13.10 Extracting Data from Staging into File To extract data from Staging into File, follow the below steps:


2. Create EDD by selecting the predefined EDS. Provide all required information while creating the EDD. To define the File structure, you can use Excel template for defining the columns. If post loading reconciliation is required then go to Control tab and provide a control record. The control record has to be in a separate file.

3. Create Connector for extracting data from staging. ADI becomes source and EDD becomes target. File structure will be according to EDD.



13.11 Extracting Data from Result into File To extract data from Results into File, follow the below steps:


2. Create EDD by selecting the predefined external data store. Provide all required information while creating the EDD. To define the File structure, you can use Excel template for defining the columns. If post loading reconciliation is required then go to Control tab and provide a control record. The control record should be in a separate file.

3. Create a Connector for extracting data from results. ADI becomes source and EDD becomes target. File structure will be according to EDD. During extract, internal surrogate keys are converted into code values by performing looking up into dimension table.



13.12 OFSAA out of the box dimension tables as lookup entity Out of the box, dimension tables will be used in the following scenarios:

• Loading Data into results

• Extracting data from results

While defining mapping, if results ADI surrogate key column is mapped, then corresponding dimension table will be used as lookup.

For Example:

• Loading data into results

Example: If N_Product_Skey of Common Account Summary ADI is mapped to an EDD product code column, then DIM_PRODUCT Table is used as lookup for getting the N_Product_Skey value and V_PROD_CODE will be used in the join expression. The EDD does a left outer join with the DIM_PRODUCT.

• Extracting Data from Results


MIGRATING CONNECTORS FROM STANDARD VIEW TO DATAFLOW VIEW


Example: If N_Product_Skey of Common Account Summary ADI is mapped to an EDD product code column, then DIM_Product Table is used as lookup for getting the V_PROD_CODE value and N_Product_Skey will be used in the join expression. The Common Account Summary lookup does a left outer join with DIM_PRODUCT.

NOTE The lookup is established identifying the Surrogate key relationship between Results ADI and Dimension table automatically. You need not define the lookup in the connector.

13.13 Migrating Connectors from Standard View to Dataflow View While upgrading DIH from version 8.0.4.0.0 or lower, open an existing connector, the following error message appears.

Perform the following steps:

6. Login to DIH application with valid User ID and Password.

7. Open new tab in the browser and click the following link:

https://<HOST>:<PORT>/<CONTEXT>/dih-api/util/connectors/upgrade-con?infodom=<INFODOM>&checkOrUpdate=U

The response provides information on number of connectors migrated with the status of their migration.

NOTE While upgrading DIH from version 8.0.5.0.0 or higher, the connectors created in standard view must also be migrated to Dataflow view. Perform the above steps to migrate connectors from Standard View to Dataflow View.

13.14 When ODI connectivity fails with universal connection pool error Use Web Service for ODI connectivity and perform the following steps:


WHEN ODI CONNECTIVITY FAILS WITH UNIVERSAL CONNECTION POOL ERROR


a. Navigate to <FIC_WEB_HOME>/webroot/WEB-INF/props and open DIHWSDetails.conf

b. Update the following properties in the above file.

c. Navigate to <FIC_DB_HOME>/bin and trigger StartDIHWS.sh and check DIHWebService.log file under <FIC_DB_HOME>/log for web service status.

Property Value Remarks

USE_WS Y/N Y – If you want to use web service

N – If you don’t want to use web service

HTTPS 0/1 1 – If you want to enable https protocol for web service

0 – If you want http protocol for webservice

PORT Unused port number Please specify a unused port number in OFSAA installed machine


HOW DO I CONNECT TO EXTERNAL SOURCE SERVER SUCH AS DB2/SQL SERVER/TERADATA/SYBASE?


13.15 How do I connect to External Source Server such as Db2/SQL Server/Teradata/Sybase? DIH uses JDBC mechanism to connect to source such sources. Hence the JDBC driver for the source server must be made available in deployed location for DIH and ODI agent.

Perform as follows:

Navigate to $FIC_WEB_HOME/webroot/WEB-INF/lib and copy the JDBC driver..

13.16 Disabling/Enabling DIH Webservice 1. To enable web service, V_COMPONENT_VALUE should be set to 1 for V_COMPONENT_CODE

= 'USE_WEBSERVICE' in DIH_SETUP_MASTER table present in atomic schema.

2. Start webservice.

3. To disable webservice, value should be set to 0.


DISABLING/ENABLING DIH WEBSERVICE


OFSAAI Support Contact Details

• Contact Infrastructure support at https://flexsupp.oracle.com if you have installed ERM and FCCM applications.

• Raise an SR in https://support.oracle.com if you have any queries related to EPM applications.

https://flexsupp.oracle.com/

https://support.oracle.com/

APPENDIX A: FAQ - USE CASES | DISABLING/ENABLING DIH WEBSERVICE


Send Us Your Comments Oracle welcomes your comments and suggestions on the quality and usefulness of this publication. Your input is an important part of the information used for revision.

• Did you find any errors?

• Is the information clearly presented?

• Do you need more information? If so, where?

• Are the examples correct? Do you need more examples?

• What features did you like most about this manual?

If you find any errors or have any other suggestions for improvement, indicate the title and part number of the documentation along with the chapter/section/page number (if available) and contact the Oracle Support.

Before sending us your comments, you might like to ensure that you have the latest version of the document wherein any of your concerns have already been addressed. You can access My Oracle Support site which has all the revised/recently released documents.

oracle financial services data integration hub · remaining chapters in the user guide. this...

Documents