complex legacy system archiving/data retention with mongodb and xquery

Legacy System Archiving With XML, XQueryand MongoDB

Dave Watson

SVP, iWay Software

@watsondaveny

watson.dave@gmail.com

Agenda

XML Archive Overview and Business Use Cases

XML Archive Technical Discussion

iWay Archive

What is XML Archive

An extension of ESB for archiving data

Leverage ESB process-oriented integration and data

federation capabilities

Long term data retention

Large repository, large index (Big Data)

Search and retrieve capabilities (High performance)

Business use examples

Satisfy regulatory requirement

e-Discovery (e.g. research, forensic)

Business analytics

Archive – Solving Business Needs

Regulations / Reqrs Example Data Retention

Federal Record

Retention Requirement

Patient health records 75 years (after last

episode of care)

FDA 21 CFR Part 11 Clinical trials and FDA

approval

35 years

HIPAA (Healthcare) Pediatric medical records 21 years

Sarbanes-Oxley (public

companies)

Audit 7 years

SEC 17a-4 (Financial

services)

Account records

Corporate documentation

6 years

Life of the enterprise

Research Life science Long-term

Analytics Financial / Legal Long-term

Examples of Business Requirements:

Archive – Types of Data

Can handles all types of data, for example: Electronic Documents

Word, Excel, EDI, HL7, XML, …

Applications

ERPs, CRMs, SAP, SFDC, …

Database Data

IMS, DB2, Oracle, Sybase, SQL Server, MUMPS, …

Electronic Files

VSAM, Unix, Logs, …

Outlook, Lotus Notes

Others

Multimedia files, Paper, Blueprints, Forms, Claims, …

ESB adapter components can be used to connect to the different types of

Archive – Archiving Needs

Archive Requirements

Policy Based – Logical selection of DB records/transactions to be archived

Store very large amounts of data in archive

Keep data for a very long periods of time

Become independent from Applications/DBMS/Systems – future proof

Protect authenticity of data – regulation and compliance

Access archived data when needed / as needed

Quickly search huge numbers of archived documents

Discard data after retention period – regulation and compliance

Examples of Archiving Requirements:

Store 75 years worth of patient data

Diverse Sources

Oracle

Support archive, query and integration scenarios

XML to remain unchanged and exist outside the data store

Ability to query documents

Ability to retrieve original XML or part of XML using XQuery

Ability to integrate XML archived data in federated services

with operational sources (e.g. MUMPS, HL7, Oracle)

Archive – Example Business Use Case

Highly scalable high performance document

management database

Easily integrates into a ESB architecture

Multi-threaded parallel processing

Distributed processing

Just another data source along with, e.g., Oracle and

MUMPS databases

Leverage ESB Tools for process orchestration,

process monitoring, data mapping/transformation,

security and data aggregation capabilities.

Implementation and vendor neutral – archived data (e.g.

XML) stored in the operating system‟s native file system

Archive – Example Business Requirements

XML Archive Technical Discussion

Overview

Load Channel

Reads XML documents and loads them into the

document repository.

Query Channel

Handles query request and response against the

document repository.

Test Channel

Simple visual interface displaying functionality and

usage of the Query API.

Highly configurable ESB Java application that can be

customized to specific needs.

Technology Involved

iWay Service Manager (commercial)

IBM WebSphere ESB (commercial)

Oracle Service Bus (commercial)

WS02 ESB (open source)

mongoDB - http://www.mongodb.org/

JSON - Java Script Object Notation

XQuery - XML query language

mongoDB

“Humongous”

Scalable, high-performance, document-oriented database.

JSON-style documents.

Mirror capable.

Auto-Sharding (clustering), horizontal scaling, automatic

failover, zero single point failure.

MapReduce support for complex processing. Work is

distributed among the cluster.

GridFS support.

A distributed file system.

Commercial support from 10gen (OEM by iWay Software)

XQuery

A query and functional programming language for XML

documents.

Is to XML documents what SQL is to databases.

“FLWOR” expressions.

FOR, LET, WHERE, ORDER BY, RETURN

Example:

for $x in /FEDREG/CNTNTS/AGCY where

$x/EAR=„Agricultural‟ order by $x ascending

return $x

Supports syntax for constructing new documents.

JSON – JavaScript Object Notation

The new data-interchange language of the web.

www.json.org

Base Loading Architecture

mongoDB

Listener Flow

XML to

Binary

Storage

GridFS

Base Query Architecture

mongoDB

Listener Flow

Binary

Storage

(Optional)

Get XML

GridFS

Requester

Loading ModificationExternal Storage

mongoDB

Listener Flow

XML to

File System

Loading ModificationSAP Loading Architecture

mongoDB

XML to

Binary

Storage

GridFS

Server

System

IDOC to

Loading ModificationChange Data Capture Loading Architecture

mongoDB

XML to

Binary

Storage

GridFS

Listener

Loading ModificationSalesforce.com Loading Architecture

mongoDB

XML to

Binary

Storage

GridFS

Listener

Salesforce

System

Loading ModificationFTP Loading Architecture

mongoDB

XML to

Binary

Storage

GridFS

Server

System

Query ModificationWeb Service SOAP Query Architecture

mongoDB

Listener Flow

Binary

Storage

(Optional)

Get XML/

GridFS

Service

Client

The Test Client

Note: The archive is designed to be called from other

flows or programs.

A simple AJAX based human interface for querying the XML

Archive.

Provides examples of the HTTP query interface provided by

the base XML Archive.

Installed with the base implementation of the XML Archive.

Simple Example

Loaded this simple XML Doc:

Displaying the Document

XML Link:

JSON Link:

Basic Query

Return all documents who have the name attribute of

the <a> element equal to “bob”.

Advanced Queries

Support for:

Regular Expressions

Ranges

Query handler is a wrapper around the mongoDB

query language.

Basic XQUERY

Return only the <b> element from the document.

Formatted Result:

complex legacy system archiving/data retention with mongodb and xquery

Technology

xquery & asp.net

xml query: xquery reference: xquery by priscilla walmsley,...

xquery rocks

xquery overview

implémentation xquery

xquery tut

common uses for xquery xquery: contents › public ›...

manual xquery

xquery basics tutorial

xquery extensions

xmlii xschema xschema xquery xquery oracle xsu oracle xsu

xquery for db2

xml query: xquery

xpath to xquery

xquery tips tricks

xquery style conventions - xqdoc...

xquery: joining content and data · xquery 1.0 provides the...

xquery design patterns

xquery helena galhardas dei ist. agenda xpath – exercício...

xquery updates in monetdb/xquery