db2 xml @ euroclear - gse belux | think global - act local xml in euroclear v1.0.pdf• pay extra...

1

GSE DB2 Working Group meeting 04 December 2014

DB2 XML @ Euroclear

About Euroclear

• Active in the financial transaction processing, Euroclear is the world’s largest provider of domestic and cross-border settlement & related services for bonds, equity and funds.

• Our clients are banks and financial institutions, brokers, dealers from over 80 countries.

2

Euroclear facts & figures

3

• Number of employees: 3600 worldwide► 2,500 in Belgium, and approximately 900 in IT

• Nationalities: >80

• Average age: 37

• Number of offices worldwide: 12

• Countries in which we have clients: 80

• Headquarters: Belgium►Main office in Brussels, a 2nd office in Braine L’Alleud

Our offices and hubs worldwide

4

Who are we?

XML in Euroclear

6

How did it start?

After migration to DB2 v10 (Sept. 2012) we started to present to internal teams the new features of DB2 and XML (with demo SQLs)

•We started easy•Generation of XML from Relational Tables (Cobol) •Mixed approach : some information are stored in XML and in the relational columns

•Information about eligibility criteria for financial assets (Reports)•Sparse data with variable structure•Relational representation involving excessive number of tables

•No update of the XML generated•Creation of XML fragments merge together based on queries•XML shared with external parties

•Funny but true: We already had extraction of data in XML

Why in DB2?

7

Testing of different solutions:

•Cobol 1:•starting from completely populated data structure

•declare cursor, open cursor, fetch rows, copy to memory, close cursor•generate xml with XML GENERATE XML-BUFFER

•Cobol 2:•appending individual elements and manually closing the tags•declare cursor, open cursor, fetch rows, copy to memory, close cursor•generate xml with XML GENERATE XML-BUFFER •generate xml with XML GENERATE XML-BUFFER for each row fetched

•DB2:•code the query using XMLELEMENT, XMLFOREST, XMLAGGREGATE, …•executing the query generates the XML document

Why in DB2?

8

COBOL 1 COBOL 2 DB2Type of xml Only Simple

documentsLimitations on attributes, elementnames

Only Simple documentsLimitations on attributes, element names

Any type of document, no limitations

CodingEffort/maintainability

Medium Cursor handling

Medium to HighCursor handlingPointer manipulation and manual tag closing

Low

Query complexity Low Low Medium to High Depending on xmlcomplexity

Memory consumption High Low Low

CPU Consumption

100% 120% 120%

Collection of lessons learnt

9

•Zparms•Utilities•SQL•Restrictions•Issues•Tools

XMLVALA with LOBVALA

10

The XMLVALA subsystem parameter specifies, in KB, an upper limit for the amount of storage that each user is to have for storing XML values.

•Issue: XML sometimes is a CLOB•Before parsing of the XML•After Serialization•When displayed in query tools

•XML(Binary format) is quite compact•XML(Text format) is very large

•Ratio can vary from 1:10 to 1:100 ( the sky is the limit )•Depending on your tag names

•You need to know this ratio (more or less) to estimate how big should be the zParms•Evaluate the repetitive values, they will make the difference in the size of the XML

•At the moment, we have enlarge the XMLVALA and LOBVALA

XML_RESTRICT_EMPTY_TAG

11

Specifies whether DB2 always serializes an empty XML element using a start-element tag followed by an end-element tag.

•NO ( Default )•<element></element>•Longer serialization output

•YES•<element/>•Shorter serialization output

•Remember: the information stored in XML format has an infinite possibilities of representation in XML Text format•Don’t compare XML by it’s text representation

•Extract data and compare the data•Solution: hashing (kind of) of the data and comparison of the data

Size of XML for logging

12

XML and LOB are different for logging

•To LOG or NOT To LOG?…this is the question…•With LOB you had a choice

•Possible to avoid to LOG the LOB independently from the base table •With XML the logging option is inherited from the base table

•You may modify the XML Document and DB2 needs to handle the information concerning the changes

•Corrective actions:•Increase the number of logs•Monitor the unavailable output log buffer

•No issue encountered concerning the logging

•If you need to insert very large XML, use the LOAD LOG NO

Utilities

13

You need to exercise for a while before get it right…

•Best suggestion:•Use TEMPLATE and LISTDEF as much as you can•Remember:

•XML are stored (like LOBS) in a separate tablespace•Very important to remember when you use COPY or RECOVER utilities

TEMPLATE LOCALDDN DSN 'DB2K.&DB..&TS..D&DATE..T&TIME.'

DISP(NEW,CATLG,DELETE) VOLCNT(20) UNIT TAPC RETPD 21 STACK YES

LISTDEF DB01A INCLUDE TABLESPACE EXSYMM00.PPECT01T ALL

COPY LIST DB01A FULL YES COPYDDN(LOCALDDN)

Options:• BASE• LOB• XML• ALL

Remember always

14

Work with LISTDEF and add option

ALL

Work only withUniversal TBS

Unload

15

Several options to unload data from a table

•Unloading the data you can redirect the XML to a File or keep it in the SYSREC•If you keep it in the SYSREC you are limited by the record length-> Spanned YES

•You can unload data in Binary format ( faster )

UNLOAD DATA SPANNED YES FROM TABLE EXSYMM.MIXTABLEI (NAME VARCHAR, XMLVALUE XML)

UNLOAD DATA SPANNED YES FROM TABLE EXSYMM.MIXTABLEI (NAME VARCHAR, XMLVALUE XML BINARYXML)

Spanned YES

000001 ....ROW1.....<?xml version="1.0" encoding="IBM500"?><ROOT/>

0100DDEF0000246A994A89A89977F4F748989889877CCDFFF7664DDDE66

060496610000ECF74305592965EF1B0F055364957EF924500FFEC96631E

----+----10---+----2----+----3----+----4----+----5----+----

000002 ....ROW2.....<?xml version="1.0" encoding="IBM500"?><R><T/></R>

0100DDEF0000346A994A89A89977F4F748989889877CCDFFF7664D64E6646D6

0604966200002CF74305592965EF1B0F055364957EF924500FFEC9EC31EC19E

----+----10---+----2----+----3----+----4----+----5----+----6---

16

Organization . . . : PS Record format . . . : VBS Record length . . . : 32768Block size . . . . : 27998

Spanned YES - BINARYXML

000001 ....ROW1............ì.ê!!èh...:] 0100DDEF00001C3000000505445810075

0604966100004AB510002842FF48800AA

----+----10---+----2----+----3---

000002 ....ROW2............ì.êh...ì.èh...::] 0100DDEF00001C300000050582005058200775

0604966200009AB51000281287008148800AAA

----+----10---+----2----+----3----+---

17

Organization . . . : PS Record format . . . : VBS Record length . . . : 32768Block size . . . . : 27998

Fastest and the mostcompact

Unload

18


•Externalize the XML as a CLOB to a file

TEMPLATE TCLOBF UNIT(SYSDA) DISP(MOD,CATLG,DELETE)DSN(&USERID..&DB..&TS..T&TI..UFILEREF) UNLOAD DATA FROM TABLE EXSYMM.MIXTABLEI (NAME VARCHAR, XMLVALUE VARCHAR CLOBF TCLOBF)

VARCHAR CLOBF TCLOBF

SYSREC:

****** ********************************************* Top of Data *******

000001 ROW1 EXSYMM.EXSYMM00.XMIX0001.T124943.UFILEREF(EYEABJZP)

000002 ROW2 EXSYMM.EXSYMM00.XMIX0001.T124943.UFILEREF(EYEABKIN)

****** ******************************************** Bottom of Data *****

EXSYMM.EXSYMM00.XMIX0001.T124943.UFILEREF(EYEABKIN):

<?xml version="1.0" encoding="IBM500"?><R><T/></R>

19

Organization . . . : PS Record format . . . : VB Record length . . . : 32756 Block size . . . . : 32760

Unload

20


•DSNTIAUL

//UNLOAD EXEC PGM=IKJEFT01,DYNAMNBR=20 //SYSTSPRT DD SYSOUT=* //SYSTSIN DD * DSN SYSTEM(DB2X) RUN PROGRAM(DSNTIAUL) PLAN(DSNTIAUL) PARMS('SQL,LOBFILE(EXSYMM)') -

LIB(‘DB2X.RUNLIB.LOAD') //SYSPRINT DD SYSOUT=* //SYSUDUMP DD SYSOUT=* //SYSREC00 DD SYSOUT=* //SYSPUNCH DD SYSOUT=* //SYSIN DD * SELECT * FROM EXSYMM.MIXTABLEI;

DSNTIAUL - LOBFILE

SYSREC00:

****** ************************************************ Top of Data ****

000001 ROW1 EXSYMM.Q0000000.C0000001.R0000000

000002 ROW2 EXSYMM.Q0000000.C0000001.R0000001

****** ********************************************* Bottom of Data ****

EXSYMM.Q0000000.C0000001.R0000001:

<?xml version="1.0" encoding="IBM500"?><R><T/></R>

21

Organization . . . : PS Record format . . . : VB Record length . . . : 27994 Block size . . . . : 27998

Load

22

The Load utility is the best to insert big XML• Avoid the Logging• No application needed for the insert (not possible in REXX)• Break the barrier of the SQL statement size

►You could push a XML document in the insert if smaller than 2MB• IC needed after the LOAD if LOG NO

• You can specify USS path►Huge advantage when you ftp from distributed

Load example = Insert

23

//DSNUPROC.SYSREC00 DD * XX EXSYMM.Q.C1.R0000002//DSNUPROC.SYSIN DD *LOAD DATA LOG NO RESUME YES INDDN SYSREC00 INTO TABLE

EXSYMM.PP_XML_TABLE ( COL1 POSITION(1) CHAR(2), XML_TX POSITION( 5 ) CHAR( 44) CLOBF )

Favourite solution to insert XML in DB2 from a dataset without developing an application

PS containing the XML Document

It could be a USS file

Image Copy

24

If you use LISTDEF ,the COPY RBA point will be closer between Base and Auxiliary.The consistency of the Base Tablespace and XML Tablespace will not be at risk.

TEMPLATE LOCALDDN DSN 'DB2K.&DB..&TS..D&DATE..T&TIME.'

DISP(NEW,CATLG,DELETE) VOLCNT(20) UNIT TAPC RETPD 21 STACK YES



If the base tablespace is restored to a different point than the XML tablespace, the data will be in check pending à Run the Check data utility

Image Copy

25

TEMPLATE STATEMENT PROCESSED SUCCESSFULLY


LISTDEF STATEMENT PROCESSED SUCCESSFULLY


PROCESSING LIST ITEM: TABLESPACE EXSYMM00.PPECT01T

PROCESSING LIST ITEM: TABLESPACE EXSYMM00.XPPR0000

DSNU1038I 288 15:12:05.49 DSNUGDYN - DATASET ALLOCATED.

DSN=DB2K.EXSYMM00.PPECT01T.D2014288.T131205

DSNU1038I 288 15:12:06.47 DSNUGDYN - DATASET ALLOCATED. DSN=DB2K.EXSYMM00.XPPR0000.D2014288.T131205

DSNU428I -DB2K 288 15:12:16.80 DSNUBAFI - DB2 IMAGE COPY SUCCESSFUL FOR TABLESPACE EXSYMM00.PPECT01T

DSNU428I -DB2K 288 15:12:16.82 DSNUBAFI - DB2 IMAGE COPY SUCCESSFUL FOR TABLESPACE EXSYMM00.XPPR0000

Quiesce

26


QUIESCE LIST DB01A


LISTDEF STATEMENT PROCESSED SUCCESSFULLY

QUIESCE LIST DB01A

PROCESSING LIST ITEM: TABLESPACE EXSYMM00.PPECT01T

PROCESSING LIST ITEM: TABLESPACE EXSYMM00.XPPR0000

QUIESCE SUCCESSFUL FOR TABLESPACE EXSYMM00.PPECT01T

QUIESCE SUCCESSFUL FOR TABLESPACE EXSYMM00.XPPR0000

QUIESCE AT RBA 01FF5CD81C4B AND AT LRSN 01FF5CD81C4B

QUIESCE UTILITY COMPLETE, ELAPSED TIME= 00:00:00

UTILITY EXECUTION COMPLETE, HIGHEST RETURN CODE=0

Recover

27

You need to exercise for a while before get it right…

LISTDEF safe choice, unless you know what you are doing…


RECOVER LIST DB01A BACKOUT YES TORBA X'01FF5CD81C4B' VERIFYSET YES

•VERIFYSET YES = checks that all related objects that are required for the point-in-time recovery are included in the RECOVER control statement (DEFAULT )

Remember the indexes, check that all the indexes are rebuilt on the base table and auxiliary à -DIS DB(EXSYMM00) SPACENAM(*) LIMIT(*)

X'01FF5CD81C4B' Recovery point of Quiesceor IC(SYSIBM.SYSCOPY)

Reorg

28

Not different from any other utility•Reorg Base and/or XML Tablespace

Restriction: Base and XML TBS cannot be reorganized in parallel by two different jobs.

LISTDEF REOLIST INCLUDE TABLESPACE EXSYMM00.PPECT01T ALL

TEMPLATE ICCOPY DSN(DB2K.&DB..&TS..REOIC.D&JD..T&HO.&MI.) UNIT VTAPE STACK YES DISP (MOD,CATLG,CATLG) RETPD 25 REORG TABLESPACE LIST REOLIST LOG NO SHRLEVEL REFERENCE COPYDDN ICCOPY

Forget about…

29

• DSN1COPY►Not supported

–SYSIBM. SYSXMLSTRINGS• Cross LOAD

►DB2 v10–Not Supported–Issue for replication

►DB2 v11–Supported

XMLTABLE

30

Used to extract XML information to relation columnsUsed for join between XML and Relational TablesCreation of relational views of the XML information

• Pay extra attention to the two different part of the parameters• First part, the XPath will:

• position in the XML tree,• determine the cardinality of your result set• accept search pattern

• ‘//employee’• Second part, will:

•determine the information you extract based on XPath•determine the definition of the relational columns

Suggestion: Start from the deepest and most repetitive elements and exploit the parents of those nodes. In XPath the ‘..’ indicates the parent of the node.(Not a good approach for performance)

XMLTABLE

31

SELECT X."Instrument_ID",X."Rating_Class_Number", X."Rating_Date”, X."ISO_Currency_Code"

FROM EXSYMM.PP_MM_SRC_2 AS C, XMLTABLE( XMLNAMESPACES(DEFAULT '▒▒▒▒▒▒▒▒', 'http://www.w3.org/2001/XMLSchema-instance' as "xsi"),

'/Instrument_Roots/Instrument_Root/Instrument_Ratings/Instrument_Rating' PASSING C.XML_TX Columns "Instrument_ID" VARCHAR(30) PATH './Instrument_ID', "Rating_Class_Number" integer PATH './Rating_Class_Number', "Rating_Date" CHAR(10) PATH './Rating_Date', "ISO_Currency_Code" CHAR(5) PATH '../../ISO_Currency_Code' ) AS X

; Instrument_ID Rating_Class_Number Rating_Date ISO_Currency_Code

# Row

sS

tart Point

http://www.w3.org/2001/XMLSchema-instance

XMLQUERY

32

XQuery is a query and programming language designed to transform structured and untrusted data.

•The rule at Euroclear:• Application performs data transformation

• XQuery:• Allowed to perform simple transformation of data of XML• Forbidden to implement in XQuery any business logic

•The Business logic is always executed and managed in developed software

IBM Infosphere Data Replication

33

For the friend QREP:• No Automatic load

• No cross load supported ( v10 )• Only Manual Subscription

• Pay extra attention to:• MQ message size (if you reduced it)• In the definition of the queue map:

• Maximum message length• Equal to your MAX(LENGTH(XMLSERIALIZE(XML_C AS CLOB)))

• Memory buffer for receive queue• 3 x MAX(LENGTH(XMLSERIALIZE(XML_C AS CLOB)))

• Additional rights to be granted ( the XML document comes from the table and not the LOG)

•Tested with Business Temporal and XML Tables

DB2 Connect – ODBC

34

• How to handle DB2 XML from distributed?

• LONGDATACOMPAT=1• Transformation of DB2 CLOB in SQL_LONGVARCHAR• This option is useful when running ODBC applications that cannot handle the

large object data types• Spool to file and you have your XML

DRIVER={IBM DB2 ODBC DRIVER};DSN=<dsn-name>;UID=<user-id>; MODE=SHARE;DBALIAS=RDBWC;PATCH2=6,18,42;PATCH1=1024;DISABLEUNICODE=1;LONGDATACOMPAT=1;

SELECT CLOB('<?xml version="1.0" encoding="UTF-8"?>' CONCAT '<?xml-stylesheet type="text/xsl" href="file://y://yourstylesheet.xsl"?>') CONCAT XMLSERIALIZE(XMLELEMENT(NAME "ROOT",XMLAGG(XMLELEMENT(NAME "TABLE",NAME))) AS CLOB(20M)) FROM SYSIBM.SYSTABLES;

Excel and IE are your best friends

35

Sub Macro1()Dim cnDB As New ADODB.ConnectionDim strSQL As StringDim rsRecords As New ADODB.RecordsetDim objIE As InternetExplorerSet objIE = New InternetExplorer'Open the ODBC Connection using this statementstrSQL = ""strSQL = strSQL & sqlq'strSQL = strSQL & "SELECT XMLSERIALIZE(XML_TX AS CLOB(20M) INCLUDING XMLDECLARATION) FROM EXSYMM.PP_MM_SRC "cnDB.Provider = "MSDASQL.1"cnDB.ConnectionString = "Driver={IBM DB2 ODBC DRIVER};Database=" & ssid _& ";Hostname=<Host Name>;Port=<Host Port>;Protocol=TCPIP;UID= " _& usr_n & ";Password=" & pwd_n & ";DISABLEUNICODE=1;LONGDATACOMPAT=1;"

cnDB.OpenrsRecords.Open strSQL, cnDBSet fs = CreateObject("Scripting.FileSystemObject")Set a = fs.CreateTextFile("Y:\testfile.xml", True)While Not rsRecords.EOF

a.Write (rsRecords(0))rsRecords.MoveNext

Wenda.Close'Close everything and set the references to nothingrsRecords.CloseSet rsRecords = NothingcnDB.CloseSet cnDB = Nothing'Open with IEWith objIE.Visible = True.Navigate "Y:\testfile.xml"End With

End Sub

* Some references are required in the Excel Project

36

Further questions?

• For questions► [email protected]

mailto:[email protected]

db2 xml @ euroclear - gse belux | think global - act local xml in euroclear v1.0.pdf• pay extra...

Documents