odi 11g - multiple flat files to oracle db table by taking file name dynamically

ODI Multiple Flat Files to Table 2016

ODI (11.1.1.6) Multiple CSV to Table via Interface with dynamically getting of File Name with Status and File Moved to Archive Folder

Document Made by: Darshankumar Prajapati & Ravindrakumar

Document Version: 1.0

Document Date: 07-May-2016

Document Last Updated: 07-May-2016

1


Purpose: Suppose we have multiple CSV Files, that we have to load in to Single Table via Single Interface, also File Name must taken Dynamic; also we have to make a table with File Name’s List with Loaded in to Table Status along with their Record Count.

Prerequisites : It was assumed that before this Tutorial you are able to load a Data from Flat File to Table via Single Interface in ODI(11.1.1.6 or Greater).

Software and Hardware Requirements (Optional):The following is a list of software requirements:

The system should include the following installed products:

Oracle Database 11g

Oracle Data Integrator 11gR1

If not done before, start the services and components for Oracle Database 11g.

Created Model: You are going to make this type of Model, so each and every component can be explained in detail in below (Detailed Technical Steps) section.

Detailed Technical Steps:

We have to make one Package under Packages name used here is: Multiple File Load.

2


1) Go to Diagram Tab and Select: ODI OS Command.

This will used for Taking All Files Name in One Flat File to achieve Dynamically Getting of File Name.

Put Below command in Command to Execute Tab: cmd /c dir D:\ODI_TRAINING\*.* /b /a:-d > D:\ODI_TRAINING\pq.csv

In case of Linux:Cd /u01/files/Ls > pq.csv

In above command assume that Our Multiple Flat Files Source Location is: D:\ODI_TRAINING\*.*

And We are doing only Listing of all files in to Flat File called PQ.CSV.

Screen Shot of files in My Directory:

After executing this you will have these values in PQ.CSV.

3


2) Now we have to create 1 Interface, which will load Data from above created File (PQ.CSV) to Table.For this we need to create one table like below:

CREATE TABLE FILELIST_STATUS(FILENAME VARCHAR2(240 BYTE),STATUS VARCHAR2(10 BYTE),DOP DATE,RECORD_COUNT NUMBER)

Now you will have to create On Interface in which PQ.CSV as a Source and FILELIST_STATUS as Target.

Screenshot of Interface created:

- In Below Interface Filter Condition is nothing but we don’t want to load PQ.CSV in FILELIST_STATUS table, as my PQ.CSV is also putted on same location.

- Screenshot of Filtered Condition:

4


- In Status you can Hardcode ‘N’ for a time.- DOP, you can give sysdate, to know Date and Time of that file on which it processed.

3) Now we need to create on GLOBAL VARIABLE as shown in Below Screen Shots:

This is Refresh Variable:

Step1:

Step2:

In this Below query is used:

select count(*) from filelist_status

It will just Counts HOW MANY TOTAL FILES ARE!

4) Now we are using same variable as Evaluate Variable like shown below:

5


5) Now we need to create one Global Variable for Getting File Name. And we need to take it as REFRESH VARIABLE.

Step1:

Step2:

Query to be used for refreshing this is :

select filename from filelist_status where rownum < 2 and status='N'

Step3:

6


Important Note: We need to pass this variable in Model in which we are using to Load Data from Flat File to Table like as shown below:

Also It was observed in ODI(11.1.1.6) that while Running This as a Single Interface, ODI will not able to Get File Name via Variable, but while You are going to Execute Same via PACKAGE then ODI will able to read the File Name from Variable.

6) Now we need to have One Single Interface which will load Data from Flat File to Table.

In our case we have simply made table same as File, and map each and every columns.

Step1:

Step2:

Flow Control Details:

7


This interface will simply insert Data from Flat File to our Target Table.

7) Now we need to have One PROCEDURE which will Update File Status which was by default ‘N’ in step 2, this will set that as ‘Y’, and Will Also put RECORD COUNT in that FILELIST_STATUS table.

Query used to achieve this is:

update filelist_status set status='Y',DOP=sysdate,record_count='<%=odiRef.getPrevStepLog("INSERT_COUNT")%>' where rownum<2 and status='N'

In this odiRef.getPrevStepLog("INSERT_COUNT") will simply gets Record Count for Particular Step in ODI.

Step1:

Step2:

Step3:

8


8) Now we need to Move this processed flat file to Archive Directory.

For this we need to use “ODIFILEMOVE” as below:

Step1:

Step2:

9) Now we need to cross check How Many Files are Still Remains to Load.

For this we need to Reuse Variable File_Move as SET VARIBALE as shown below:

9


This will simply Decreased by 1. So it will be automatically stopped when File_Count NOT > 0.

So at the end You have created the below Model.

After running of this Procedure you can check file status in FILELIST_STATUS Table as below:

You can cross check Record Count in your Final Table in which you have loaded Data.

Summary:

In this tutorial, you have learned how to:

Verify the Prerequisites.

Create one ODI_OS_COMMAND for just listing of you all flat flies into Single Flat File.

Create a New ODI Model for the Flat File Source to Target Database Table.

Create a New ODI Source Datastore for Use with ODI Interface

Create a New ODI Target Datastore for Use with ODI Interface

10


Create a Variable for FILE COUNT.

Create a Global Variable for dynamically passing File Name in Data Models in ODI.

Create an Interface which will Load Data from Flat file to Oracle Table.

Create a procedure which will Updates a File Status in Table along with Record Count.

Create a ODI_FILE_MOVE for moving your processed file to ARCHIVE Directory.

Hope you have understood which we have developed with the help of this document.

11

odi 11g - multiple flat files to oracle db table by taking file name dynamically

Data & Analytics