Download - Odi Jdbc Excel
ODI JDBC EXCEL
BY ODI EXPERTS
V1.0 DOCUMENTATION
CONTENTS
READING AND WRITING EXCEL WITHOUT ODBC ..................................................... 2
LIMITATION OF THIS DRIVER ............................................................................................ 2
PRELIMINARY STEPS .............................................................................................................. 3
REVERSING THE EXCEL FILE (RKM)................................................................................ 6
SINGLE SHEET HEADER PRESENT AT 5TH ROW ................................................... 7
MULTIPLE SHEETS ............................................................................................................... 9
SELECTIVE MULTIPLE SHEETS ................................................................................... 10
LOADING FROM EXCEL TO DB (LKM) .......................................................................... 11
LOADING EXCEL FILE INTO DB ................................................................................... 11
HANDLING EMPTY ROWS .............................................................................................. 13
INTEGRATION INTO EXCEL (IKM) ................................................................................. 14
WRITING INTO 2003 (XLS) ............................................................................................ 16
REWRITING INTO EXISTING EXCEL FILE ............................................................... 20
WRITING HUGE DATA INTO EXCEL FILE ................................................................ 22
LOADING DIRECTLY FROM FILE ................................................................................. 24
ODI EXCEL JDBC DATATYPE ............................................................................................. 25
CURRENCY, NUMERIC, SCIENTIFIC AND PERCENTAGE DATATYPE
SCALE LEVEL........................................................................................................................ 25
HANDLING DATE & DATETIME ................................................................................... 26
HANDLING CURRENCY_SYMBOLS .............................................................................. 27
API ................................................................................................................................................ 28
COMMON ERROR/ISSUES .................................................................................................. 31
READING AND WRITING EXCEL WITHOUT ODBC
For a long time, Reading and Writing Excel was a big challenge especially in
the Non-windows environment. Today from ODIExperts we are providing a
unique solution ODI JDBC EXCEL, which will enable you to Reverse, Load and
Integrate (Read/Write) Excel XLS and XLSX files.
LIMITATION OF THIS DRIVER
In ODI Data / View Data on the data store will not work
Excel (XLS) format only allows 65,536 Rows per Sheet and Excel (XLSX)
format allows 1,048,576 rows per sheet.
Although not a limitation but while writing Rows greater than 100,000
should go for XLSX format, as the process will be Faster and File created
will be compressed and smaller in size. Excel 2003 (xls) lacks such
capabilities.
PRELIMINARY STEPS
Extract the ODI Excel JDBC Zip file and you will find these files
KM_IKM_EXCEL_INSERT_V1.xml
KM_LKM_EXCEL_to_SQL_V1.xml
KM_RKM_EXCEL_V1.xml
ODI_JDBC_Excel_V1.jar
TECH_ODI_JDBC_Excel.xml
Copy the ODI_JDBC_Excel_v1.jar under your ODI Agent’s Driver Folder and
restart your Agent .For local, please copy in to your respective USERLIB
Folder.
Next Import the ODI JDBC Excel Technology in Duplication Mode.
Create a DataServer with just an appropriate Name for it. No Need to
provide any other details.
Create a Physical Schema and provide the Directory where the Excel Files are present.
[ Note: - You can create ‘n’ Number of Physical Schema under the
same Data Server ]
Finally create a Logical Schema and link them through the appropriate
Context.
Now you are ready to use this Technology.
Login into Designer and Import the RKM, LKM and IKM respectively.
REVERSING THE EXCEL FILE (RKM)
The different options of the RKM are
HEADER_ROW – By default most of the excel have the first row as the header
but there might be chance that header row can be from 3rd or Nth line in that case
provide the appropriate Value to it.
SHEET_NO – If one of the Excel Sheet is to be reversed then specify the Sheet
No. By default the Sheet No.
To Reverse Multiple Sheets please use %.
First sheet will be 1; Second sheet will be 2 and so on...
Note: - If different Sheets have different Header Row then please reverse them
individually.
Empty sheets will not be ignored, while reversing.
Model Setup
SINGLE SHEET HEADER PRESENT AT 5TH ROW
SAMPLE FILE
SELECT REVERSE ENGINEER – CUSTOMIZED
KNOWLEDGE MODULE – RKM EXCEL_V1
MASK - << File Name with Extension>>
SHEET_NO –1, Specify the sheet No.
HEADER_ROW – 5 [Row No ,where the header is present, as you can
see in the above screenshot ]
Once reversed the datastore will be similar as shown below,
Name - << File Name with Extension >>
Resource Name – Excel File Name
[Note – Please don’t modify anything in the First line of the description]
Data type is detected depending on the content.
MULTIPLE SHEETS
SAMPLE FILE
Provide the Options depending on your requirement
MASK - << File Name with Extension>>
SHEET_NO – %
HEADER_ROW – 1
[NOTE: - The same header row Value will be used for all the Sheets.
In case if different sheets have different values please reverse
individually]
As you can see from the below image, all the sheets are reversed.
SELECTIVE MULTIPLE SHEETS
In case you wish to reverse selective multiple sheets you can provide the
Sheet-No separated by comma. A sample screenshot is shown below.
LOADING FROM EXCEL TO DB (LKM)
The different options of LKM are
TRIM_COLUMNS: If true the columns will be trimmed while loading into C$
table else column will retain spaces as present in File.
DELETE_TEMPORARY_OBJECTS: As present in all the KM options, this option
is to retain C$.
Restrictions
Filters has to be only on the Staging
Joins between two Sheets also have to be on Staging
Any SQL Operation on source has to be performed on the Staging or Target
side only.
LOADING EXCEL FILE INTO DB
The below shown sample data will be Inserted into the Target Table.
Map the column accordingly.
Note: - All the columns need to be mapped to staging only. Column mapped
on Source is not supported and will throw error.
Select the LKM Excel to SQL_V1 and appropriate IKM depending on your
Target Technology.
Save and Execute and ones the Execution is successful. Please look into the
Target Table for Data.
Operator Session Execution of the above Interface
Data on the Target Table.
HANDLING EMPTY ROWS
The LKM automatically takes care of the empty rows while loading into the Target. As you can see from the below screenshot, the lines where the row is empty is not loaded into target
INTEGRATION INTO EXCEL (IKM)
This IKM is Multi Technology IKM and so does not create I$ table but load
directly from Source or C$ table depending the Source.
Enable the Option Staging Area Different from Target and select the
appropriate Source Logical Schema.
The different options of the IKM are
HEADER_ALIGNMENT: You can set the Header Text Alignment .The valid
values are LEFT, RIGHT and CENTER. By default LEFT is used.
HEADER_BOLD: If true the Header will be bold else Normal text.
HEADER_COLOR: You can set the Header color here. By default the plum color
is used.
The valid color values are AQUA, AUTOMATIC, BLACK, BLUE, BLUE_GREY,
BRIGHT_GREEN, BROWN, CORAL, CORNFLOWER_BLUE, DARK_BLUE,
DARK_GREEN, DARK_RED, DARK_TEAL, DARK_YELLOW, GOLD, GREEN,
GREY_25_PERCENT, GREY_40_PERCENT, GREY_50_PERCENT,
GREY_80_PERCENT, INDIGO, LAVENDER, LEMON_CHIFFON, LIGHT_BLUE,
LIGHT_CORNFLOWER_BLUE, LIGHT_GREEN, LIGHT_ORANGE,
LIGHT_TURQUOISE, LIGHT_YELLOW, LIME, MAROON, OLIVE_GREEN, ORANGE,
ORCHID, PALE_BLUE, PINK, PLUM, RED, ROSE, ROYAL_BLUE, SEA_GREEN,
SKY_BLUE, TAN, TEAL, TURQUOISE, VIOLET, WHITE, YELLOW
REWRITE_EXISTING_FILE: Enable this option if you want to rewrite existing
Excel Sheet.
WRITE_LARGE_XLSX: If you are loading huge data then enable this option as
then JDBC driver uses a different algorithm to load faster and with less
memory.
DATE_FORMAT: Date format while writing into Excel.
d- Day
m - Month
y - Year
Some of the Valid Format
------------------------------
m/d/yy
d-mmm-yy
d-mmm
mmm-yy
DATETIME_FORMAT: Date time format while writing into Excel
d - Day
m - Month
y - Year
s - Second
h - Hour
mm - Minutes
Some of the Valid Format
-------------------------------
h:mm AM/PM
h:mm:ss AM/PM
h:mm
h:mm:ss
m/d/yy h:mm
CURRENCY_SYMBOL: The currency symbol to be used if the datatype is
Currency. The default currency is $.
WRITING INTO 2003 (XLS)
You will need to manually Create Target datastore with appropriate Column
Name and Datatype.
Data Store Name : <<Sheet Name>>.
The Sheets created in Excel will have data Store Name.
Resource Name : <<File Name with Extensions>>
Make sure you provide the file with appropriate Extension xls / xlsx.
Note: - Without Extension IKM will throw error.
Select the appropriate Datatype and don’t worry about the logical length but
make sure you select the appropriate Scale as Excel created will use that for
creating required decimals data.
Map the Column accordingly.
Sample Source data to be loaded into Excel
Sample Target data for the above source data. Since my Scale for Salary and
com_pct was 2, the decimals were added accordingly.
Now the Scale level is increased to 3
As you can see the Decimal are created with 3 Decimal points.
If Decimal points are not entered then data is create is with no Decimal Values.
Make sure you provide the Options else Excel file will be created using default options.
REWRITING INTO EXISTING EXCEL FILE
The below example illustrates rewriting an existing File. The Driver will automatically create a new Sheet in the File and Start Writing into it.
Note: - The Present driver cannot rewrite into an existing Sheet, but only create a new sheet in the existing File. Also rewriting does not work if WRITE_LARGE_XLSX is enabled
This is a sample file with one Sheet .Now we will Insert data into this File.
Create a Datastore and datatype accordingly.
In the KM options change the option REWRITE_EXISTING_FILE to True
As you can see from the below screenshot, the New sheet is created next to
exiting Sheet in the same file.
WRITING HUGE DATA INTO EXCEL FILE
The below example illustrates writing huge data into an Excel File.
Note: - It’s highly recommended to use only XLSX format to write huge file and
enable option WRITING_LARGE_XLSX to be true which uses a different
algorithm to write huge data in less time and memory .
WRITING_LARGE_XLSX will purge the existing data in the File.
We are going to load the below table data (around 3.6 million) into an Excel
file.
In the IKM Options change the options WRITE_LARGE_XLSX to True
As you can see from the screenshot,
it only took 96 seconds to write
around 3.6 Million Rows.
1048575 * 3 Sheets + 506247 = 3,651,972 Rows which matches the above
total Count.
[Note: - In above calculation Header row is not included, that’s why its
506247]
LOADING DIRECTLY FROM FILE
The IKM can load directly from a delimited file too, but none of the Header styling will work.
Sample output
In the Interface just select the Source File logical schema as Staging Area
ODI EXCEL JDBC DATATYPE
CURRENCY
DATE
DATETIME
NUMERIC
PERCENTAGE
SCIENTIFIC
VARCHAR
CURRENCY, NUMERIC, SCIENTIFIC AND PERCENTAGE
DATATYPE SCALE LEVEL
Depending on the scale and the datatype the data is displayed accordingly.
For currency we have scale-4, so currency displayed with 4 decimal points.
For Numeric we have scale-3, so the numbers is displayed with 3 decimal
points.
For Percentage the scale -2, so the percentage is displayed with 2 decimal
points.
In case if the decimal value in the database is more than the scale specified in
the Excel the data is displayed accordingly. For example now all the scale is
changed to 2.
Now as you can see from the screenshot, the decimal values are displayed to 2
points. Even though the original data is of different scale, but the data is
displayed accordingly and no data is truncated as you can see in the formula
bar.
In case you wish to display the data as it is in the Database, then have the
datatype as VARCHAR, which is shown in the below screenshot
While loading data from Excel (LKM), all the above datatype are read as
NUMBER and accordingly loaded as number/Integer into Staging or Target
Side.
HANDLING DATE & DATETIME
Date datatype displays only Day, Month and Year and does not support time.
For both Date and Time support please use the DATETIME datatype.
Note: - If you have trouble loading from the Excel when the datatype is
DATE/DATETIME, please use VARCHAR and convert accordingly at the
staging or target side.
HANDLING CURRENCY_SYMBOLS
For any currency symbol other than $ please provide the appropriate
currency symbol in the options as shown in the below examples.
API
UnloadTables – You can use this API to unload multiple tables into same or different Excel File. The method is as follows
In the Command on Source – Specify the Technology and the Schema. In the
Command on Target Select the Technology Jython.
Here is an example to unload multiple tables of HR Schema.
UnloadTables(String filename,Connection conn,String
schema, String tableNames)
import api.api as Excelapi
conn=odiRef.getJDBCConnection("SRC")
fileName='C:/HR_TABLES.xlsx'
schema='HR'
tableNames='COUNTRIES,DEPARTMENTS,EMPLOYEES,JOB_HISTORY,JOBS,LOCATIO
NS,REGIONS'
Excelapi.UnloadTables(fileName,conn,schema,tableNames)
Sample Output
In case you wish to have different schema / different excel File; you can repeat
the steps in single or different Command.
COMMON ERROR/ISSUES
We have documented some of the common error/issues which can occur due
to invalid entry/options in KM options or Excel File.
: - This exception will occur when you provided an FileNotFoundException
invalid File Name or File without correct Extension.
: - While trying to Reverse multiple or Sheet does not display in the Operator
Individual Sheets and if the session in the operator is successful, yet you don’t
see the Sheet being reversed there is high chance that Sheet is empty or you
. have provide an Header Row which is empty
- If you have provided a sheet No which does not Sheet index is out of range:
exists in Excel File Sheet index out of range error will occur.
: - In case if the Excel data type is Cannot get a numeric value from a text cell
Numeric, Currency, Percentage and Scientific but some of the data contains
Non numeric values or String in nature this Error can occur. Change the
and try reloading the Content. datatype of the Excel to VARCHAR
While using LKM Excel to SQL, if Data does not get loaded from Excel (LKM):-
your interface is successful yet you don’t see any data in target, it means you
have set the column mapped to Source. Please set the appropriate column
mapping to and try again Staging
: - If you are loading huge data Loading data into Excel is taking too much time
and taking too much. Try enabling WRITING_LARGE_XLSX .
C O N T A C T U S
We hope that this documentation has helped in the initial steps of
using and understanding the ODI JDBC EXCEL driver and KM’s.
In case if you still have issues or question please feel free to contact
us at www.odiexperts.com.