ODI JDBC EXCEL

BY ODI EXPERTS

V1.0 DOCUMENTATION

CONTENTS

READING AND WRITING EXCEL WITHOUT ODBC ..................................................... 2

LIMITATION OF THIS DRIVER ............................................................................................ 2

PRELIMINARY STEPS .............................................................................................................. 3

REVERSING THE EXCEL FILE (RKM)................................................................................ 6

SINGLE SHEET HEADER PRESENT AT 5TH ROW ................................................... 7

MULTIPLE SHEETS ............................................................................................................... 9

SELECTIVE MULTIPLE SHEETS ................................................................................... 10

LOADING FROM EXCEL TO DB (LKM) .......................................................................... 11

LOADING EXCEL FILE INTO DB ................................................................................... 11

HANDLING EMPTY ROWS .............................................................................................. 13

INTEGRATION INTO EXCEL (IKM) ................................................................................. 14

WRITING INTO 2003 (XLS) ............................................................................................ 16

REWRITING INTO EXISTING EXCEL FILE ............................................................... 20

WRITING HUGE DATA INTO EXCEL FILE ................................................................ 22

LOADING DIRECTLY FROM FILE ................................................................................. 24

ODI EXCEL JDBC DATATYPE ............................................................................................. 25

CURRENCY, NUMERIC, SCIENTIFIC AND PERCENTAGE DATATYPE

SCALE LEVEL........................................................................................................................ 25

HANDLING DATE & DATETIME ................................................................................... 26

HANDLING CURRENCY_SYMBOLS .............................................................................. 27

API ................................................................................................................................................ 28

COMMON ERROR/ISSUES .................................................................................................. 31

READING AND WRITING EXCEL WITHOUT ODBC

For a long time, Reading and Writing Excel was a big challenge especially in

the Non-windows environment. Today from ODIExperts we are providing a

unique solution ODI JDBC EXCEL, which will enable you to Reverse, Load and

Integrate (Read/Write) Excel XLS and XLSX files.

LIMITATION OF THIS DRIVER

In ODI Data / View Data on the data store will not work

Excel (XLS) format only allows 65,536 Rows per Sheet and Excel (XLSX)

format allows 1,048,576 rows per sheet.

Although not a limitation but while writing Rows greater than 100,000

should go for XLSX format, as the process will be Faster and File created

will be compressed and smaller in size. Excel 2003 (xls) lacks such

capabilities.

PRELIMINARY STEPS

Extract the ODI Excel JDBC Zip file and you will find these files

KM_IKM_EXCEL_INSERT_V1.xml

KM_LKM_EXCEL_to_SQL_V1.xml

KM_RKM_EXCEL_V1.xml

ODI_JDBC_Excel_V1.jar

TECH_ODI_JDBC_Excel.xml

Copy the ODI_JDBC_Excel_v1.jar under your ODI Agent’s Driver Folder and

restart your Agent .For local, please copy in to your respective USERLIB

Folder.

Next Import the ODI JDBC Excel Technology in Duplication Mode.

Create a DataServer with just an appropriate Name for it. No Need to

provide any other details.

Create a Physical Schema and provide the Directory where the Excel Files are present.

[ Note: - You can create ‘n’ Number of Physical Schema under the

same Data Server ]

Finally create a Logical Schema and link them through the appropriate

Context.

Now you are ready to use this Technology.

Login into Designer and Import the RKM, LKM and IKM respectively.

REVERSING THE EXCEL FILE (RKM)

The different options of the RKM are

HEADER_ROW – By default most of the excel have the first row as the header

but there might be chance that header row can be from 3rd or Nth line in that case

provide the appropriate Value to it.

SHEET_NO – If one of the Excel Sheet is to be reversed then specify the Sheet

No. By default the Sheet No.

To Reverse Multiple Sheets please use %.

First sheet will be 1; Second sheet will be 2 and so on...

Note: - If different Sheets have different Header Row then please reverse them

individually.

Empty sheets will not be ignored, while reversing.

Model Setup

SINGLE SHEET HEADER PRESENT AT 5TH ROW

SAMPLE FILE

SELECT REVERSE ENGINEER – CUSTOMIZED

KNOWLEDGE MODULE – RKM EXCEL_V1

MASK - << File Name with Extension>>

SHEET_NO –1, Specify the sheet No.

HEADER_ROW – 5 [Row No ,where the header is present, as you can

see in the above screenshot ]

Once reversed the datastore will be similar as shown below,

Name - << File Name with Extension >>

Resource Name – Excel File Name

[Note – Please don’t modify anything in the First line of the description]

Data type is detected depending on the content.

MULTIPLE SHEETS

SAMPLE FILE

Provide the Options depending on your requirement

MASK - << File Name with Extension>>

SHEET_NO – %

HEADER_ROW – 1

[NOTE: - The same header row Value will be used for all the Sheets.

In case if different sheets have different values please reverse

individually]

As you can see from the below image, all the sheets are reversed.

SELECTIVE MULTIPLE SHEETS

In case you wish to reverse selective multiple sheets you can provide the

Sheet-No separated by comma. A sample screenshot is shown below.

LOADING FROM EXCEL TO DB (LKM)

The different options of LKM are

TRIM_COLUMNS: If true the columns will be trimmed while loading into C$

table else column will retain spaces as present in File.

DELETE_TEMPORARY_OBJECTS: As present in all the KM options, this option

is to retain C$.

Restrictions

Filters has to be only on the Staging

Joins between two Sheets also have to be on Staging

Any SQL Operation on source has to be performed on the Staging or Target

side only.

LOADING EXCEL FILE INTO DB

The below shown sample data will be Inserted into the Target Table.

Map the column accordingly.

Note: - All the columns need to be mapped to staging only. Column mapped

on Source is not supported and will throw error.

Select the LKM Excel to SQL_V1 and appropriate IKM depending on your

Target Technology.

Save and Execute and ones the Execution is successful. Please look into the

Target Table for Data.

Operator Session Execution of the above Interface

Data on the Target Table.

HANDLING EMPTY ROWS

The LKM automatically takes care of the empty rows while loading into the Target. As you can see from the below screenshot, the lines where the row is empty is not loaded into target

INTEGRATION INTO EXCEL (IKM)

This IKM is Multi Technology IKM and so does not create I$ table but load

directly from Source or C$ table depending the Source.

Enable the Option Staging Area Different from Target and select the

appropriate Source Logical Schema.

The different options of the IKM are

HEADER_ALIGNMENT: You can set the Header Text Alignment .The valid

values are LEFT, RIGHT and CENTER. By default LEFT is used.

HEADER_BOLD: If true the Header will be bold else Normal text.

HEADER_COLOR: You can set the Header color here. By default the plum color

is used.

The valid color values are AQUA, AUTOMATIC, BLACK, BLUE, BLUE_GREY,

BRIGHT_GREEN, BROWN, CORAL, CORNFLOWER_BLUE, DARK_BLUE,

DARK_GREEN, DARK_RED, DARK_TEAL, DARK_YELLOW, GOLD, GREEN,

GREY_25_PERCENT, GREY_40_PERCENT, GREY_50_PERCENT,

GREY_80_PERCENT, INDIGO, LAVENDER, LEMON_CHIFFON, LIGHT_BLUE,

LIGHT_CORNFLOWER_BLUE, LIGHT_GREEN, LIGHT_ORANGE,

LIGHT_TURQUOISE, LIGHT_YELLOW, LIME, MAROON, OLIVE_GREEN, ORANGE,

ORCHID, PALE_BLUE, PINK, PLUM, RED, ROSE, ROYAL_BLUE, SEA_GREEN,

SKY_BLUE, TAN, TEAL, TURQUOISE, VIOLET, WHITE, YELLOW

REWRITE_EXISTING_FILE: Enable this option if you want to rewrite existing

Excel Sheet.

WRITE_LARGE_XLSX: If you are loading huge data then enable this option as

then JDBC driver uses a different algorithm to load faster and with less

memory.

DATE_FORMAT: Date format while writing into Excel.

d- Day

m - Month

y - Year

Some of the Valid Format

------------------------------

m/d/yy

d-mmm-yy

d-mmm

mmm-yy

DATETIME_FORMAT: Date time format while writing into Excel

d - Day

m - Month

y - Year

s - Second

h - Hour

mm - Minutes

Some of the Valid Format

-------------------------------

h:mm AM/PM

h:mm:ss AM/PM

h:mm

h:mm:ss

m/d/yy h:mm

CURRENCY_SYMBOL: The currency symbol to be used if the datatype is

Currency. The default currency is $.

WRITING INTO 2003 (XLS)

You will need to manually Create Target datastore with appropriate Column

Name and Datatype.

Data Store Name : <<Sheet Name>>.

The Sheets created in Excel will have data Store Name.

Resource Name : <<File Name with Extensions>>

Make sure you provide the file with appropriate Extension xls / xlsx.

Note: - Without Extension IKM will throw error.

Select the appropriate Datatype and don’t worry about the logical length but

make sure you select the appropriate Scale as Excel created will use that for

creating required decimals data.

Map the Column accordingly.

Sample Source data to be loaded into Excel

Sample Target data for the above source data. Since my Scale for Salary and

com_pct was 2, the decimals were added accordingly.

Now the Scale level is increased to 3

As you can see the Decimal are created with 3 Decimal points.

If Decimal points are not entered then data is create is with no Decimal Values.

Make sure you provide the Options else Excel file will be created using default options.

REWRITING INTO EXISTING EXCEL FILE

The below example illustrates rewriting an existing File. The Driver will automatically create a new Sheet in the File and Start Writing into it.

Note: - The Present driver cannot rewrite into an existing Sheet, but only create a new sheet in the existing File. Also rewriting does not work if WRITE_LARGE_XLSX is enabled

This is a sample file with one Sheet .Now we will Insert data into this File.

Create a Datastore and datatype accordingly.

In the KM options change the option REWRITE_EXISTING_FILE to True

As you can see from the below screenshot, the New sheet is created next to

exiting Sheet in the same file.

WRITING HUGE DATA INTO EXCEL FILE

The below example illustrates writing huge data into an Excel File.

Note: - It’s highly recommended to use only XLSX format to write huge file and

enable option WRITING_LARGE_XLSX to be true which uses a different

algorithm to write huge data in less time and memory .

WRITING_LARGE_XLSX will purge the existing data in the File.

We are going to load the below table data (around 3.6 million) into an Excel

file.

In the IKM Options change the options WRITE_LARGE_XLSX to True

As you can see from the screenshot,

it only took 96 seconds to write

around 3.6 Million Rows.

1048575 * 3 Sheets + 506247 = 3,651,972 Rows which matches the above

total Count.

[Note: - In above calculation Header row is not included, that’s why its

506247]

LOADING DIRECTLY FROM FILE

The IKM can load directly from a delimited file too, but none of the Header styling will work.

Sample output

In the Interface just select the Source File logical schema as Staging Area

ODI EXCEL JDBC DATATYPE

CURRENCY

DATE

DATETIME

NUMERIC

PERCENTAGE

SCIENTIFIC

VARCHAR

CURRENCY, NUMERIC, SCIENTIFIC AND PERCENTAGE

DATATYPE SCALE LEVEL

Depending on the scale and the datatype the data is displayed accordingly.

For currency we have scale-4, so currency displayed with 4 decimal points.

For Numeric we have scale-3, so the numbers is displayed with 3 decimal

points.

For Percentage the scale -2, so the percentage is displayed with 2 decimal

points.

In case if the decimal value in the database is more than the scale specified in

the Excel the data is displayed accordingly. For example now all the scale is

changed to 2.

Now as you can see from the screenshot, the decimal values are displayed to 2

points. Even though the original data is of different scale, but the data is

displayed accordingly and no data is truncated as you can see in the formula

bar.

In case you wish to display the data as it is in the Database, then have the

datatype as VARCHAR, which is shown in the below screenshot

While loading data from Excel (LKM), all the above datatype are read as

NUMBER and accordingly loaded as number/Integer into Staging or Target

Side.

HANDLING DATE & DATETIME

Date datatype displays only Day, Month and Year and does not support time.

For both Date and Time support please use the DATETIME datatype.

Note: - If you have trouble loading from the Excel when the datatype is

DATE/DATETIME, please use VARCHAR and convert accordingly at the

staging or target side.

HANDLING CURRENCY_SYMBOLS

For any currency symbol other than $ please provide the appropriate

currency symbol in the options as shown in the below examples.

API

UnloadTables – You can use this API to unload multiple tables into same or different Excel File. The method is as follows

In the Command on Source – Specify the Technology and the Schema. In the

Command on Target Select the Technology Jython.

Here is an example to unload multiple tables of HR Schema.

UnloadTables(String filename,Connection conn,String

schema, String tableNames)

import api.api as Excelapi

conn=odiRef.getJDBCConnection("SRC")

fileName='C:/HR_TABLES.xlsx'

schema='HR'

tableNames='COUNTRIES,DEPARTMENTS,EMPLOYEES,JOB_HISTORY,JOBS,LOCATIO

NS,REGIONS'

Excelapi.UnloadTables(fileName,conn,schema,tableNames)

Sample Output

In case you wish to have different schema / different excel File; you can repeat

the steps in single or different Command.

COMMON ERROR/ISSUES

We have documented some of the common error/issues which can occur due

to invalid entry/options in KM options or Excel File.

: - This exception will occur when you provided an FileNotFoundException

invalid File Name or File without correct Extension.

: - While trying to Reverse multiple or Sheet does not display in the Operator

Individual Sheets and if the session in the operator is successful, yet you don’t

see the Sheet being reversed there is high chance that Sheet is empty or you

. have provide an Header Row which is empty

- If you have provided a sheet No which does not Sheet index is out of range:

exists in Excel File Sheet index out of range error will occur.

: - In case if the Excel data type is Cannot get a numeric value from a text cell

Numeric, Currency, Percentage and Scientific but some of the data contains

Non numeric values or String in nature this Error can occur. Change the

and try reloading the Content. datatype of the Excel to VARCHAR

While using LKM Excel to SQL, if Data does not get loaded from Excel (LKM):-

your interface is successful yet you don’t see any data in target, it means you

have set the column mapped to Source. Please set the appropriate column

mapping to and try again Staging

: - If you are loading huge data Loading data into Excel is taking too much time

and taking too much. Try enabling WRITING_LARGE_XLSX .

C O N T A C T U S

We hope that this documentation has helped in the initial steps of

using and understanding the ODI JDBC EXCEL driver and KM’s.

In case if you still have issues or question please feel free to contact

us at www.odiexperts.com.