mlw 2014 - data governance for regulated industries

© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Data Governance for Regulated Industries Amir Halfon, CTO Financial Services Jim Clark, Senior Director, Product Management MarkLogic World, June2014

© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. SLIDE: 2

Hello, my name is Big Data focused for the last 8 years Enterprise Database & Start-ups in NoSQL & Big Data Focus on Secutiry, Bitemporal and Hadoop

Hello, my name is CTO for Financial Services at MarkLogic for 2 years Previously at Oracle, Sun Microsystems Many years in the FinServe sector


Agenda

Data governance considerations Legacy approaches: How we got here Case studies: Solutions Q&A


Data Governance Considerations

Security

Privacy Continuity

Provenance Compliance

Retention



Security



Security

Privacy



Security

Privacy

Provenance



Security

Privacy

Provenance

Retention



Security

Privacy Continuity

Provenance

Retention



Security

Privacy Continuity


Retention


Why is this difficult? And risky?

And expensive? And behind schedule?


Last Generation

OLTP

Warehouse

Data Marts Archives

“Unstructured”

“ ”

Video Audio

Signals, Logs, Streams

Social

Documents, Messages

{ } Metadata

Search 🔍

Reference Data


Can anything be done?


Enterprise NoSQL

Flexible data model, comprehensive indexes o Documents: Hierarchy, text, values, tags—schema “on-read” o Scalars: Aggregates and range filters, including geospatial o Triples: Linked facts and inferencing o Permissions: Users, roles, compartments, and privileges o Queries: Reverse indexes for alerting, matching

In-memory writes, lock-free reads Ad hoc dimensions, real-time transformation Strict consistency throughout


Case Studies


Financial Services Case Studies

Records Retention and Investigations Trade Operational Data

Store Regulatory Compliance Customer On-Boarding


Case Study: Records Retention and Investigations

Accurately respond to litigation Hold, review, produce data across current, legacy systems Repatriate and reconcile distributed data Demonstrate fidelity and audit trail Reduce infrastructure and maintenance costs


Old Generation Records Retention and Investigations

Oracle

Mainframe

Sybase

87 total systems


New Generation Records Retention and Investigations

Oracle

MarkLogic Mainframe

Sybase

87 total systems

Shared Storage NAS HDFS

Ingest

Offline

Query

Replication

MarkLogic

100TB 40TB


Case Study: Operational Trade Data Store

Comply with regulations requiring operational insights Quickly operationalize business innovation Support risk management requirements Reduce costs per trade

Trade processing exceptions

infrastructure and maintenance costs


ETL

Old Generation Operations and Analytics

Multiple Relational Data Stores for different instrument types

…

Limited, fragmented analytics and reporting capabilities

Long, costly development cycles

Derivatives Rates

FX

…etc.

Expensive, error-prone post-trade processing

…etc.

Matching Clearing

Settlement

ETL ETL


New Generation Operations, Compliance and Analytics

Executed Trades

HDFS

Clearing Settlement …etc. Matching

Exceptions Management

Simplified workflow architecture

Tiered Storage using Hadoop

Post Trade Processing

Surveillance, Risk & Compliance

Historical Analysis

MarkLogic

Single ODS for all instrument types persisted as-is off a message bus

Single source of truth


Case Study: On-Boarding Compliance

Thousands of rules, 1–2M accounts, 30–40M documents Encoding, adjusting, and matching rules must scale Impossible to pre-define dimensions, relationships Vet new accounts and “show your work” Real-time decision-making


Old Generation On-Boarding Compliance

Documents Policies Regulations


New Generation On-Boarding Compliance

Documents

MarkLogic

Onboarding Workflow

Policies Regulations


Case Study: Dodd Frank Compliance

Trace lineage of order lifecycle for OTC derivatives Search, link supporting communications, documents Strict reporting and retention rules, response times Existing policies, point solutions don’t scale


Old Generation Regulatory Compliance

email

Reference Data

Reporting

Categorization Linking

Trade Records

Operations


New Generation Regulatory Compliance

MarkLogic

Operations

Reporting Surveillance Ad hoc analysis

Categorization Enrichment Linking

email

Reference Data

Trade Records

{ } Metadata


Enrichment and Linking


Management Dashboard


What now?


New Generation Data Governance

Security

Privacy Continuity


Retention


Take-Aways

New and more data is both an opportunity and a threat Last generation of data management is not sufficient More copies, representations, transformations increase risk ETL and up-front modeling reduce agility Index once and reuse across workloads, lifecycle


SECURE Minimize duplication,

costly ETL, reduce risk

REAL-TIME Interactive search, delivery & analytics

MARKLOGIC

OPERATIONAL Run enterprise, mission-critical

applications

mlw 2014 - data governance for regulated industries

Technology