how to build and promote a successful mdm solution on a shoestring

Pablo Riboldi, PhDSolution Manager for Information Governance & Quality

The Church or Jesus Christ of Latter-day Saints

How to build and promote a successful MDM solution on a shoestring

Master Data Management (MDM) is a set of processes, technologies, strategies to securely provide accurate and consistent enterprise data.

2© IRI 2011

Master Data Management - Definition

• Large– 13 million members in 160 countries– 53,000 full-time missionaries– Materials translated in 83 languages– 264 centrally managed databases– Largest Genealogy system in the world

• www.lds.org or www.mormon.org

© IRI 2010 3

My Organization

• Large• Organization (Hierarchical)– Governing Council (18 members)– Departments (23)– Silo Organization– Independent information systems

• www.lds.org or www.mormon.org

© IRI 2010 4

My Organization

• Large• Organization (Hierarchical)• Non Profit– Invite people to come to Jesus Christ by

• Proclaiming the gospel• Perfecting the saints• Redeeming the dead

– Clearly defined common purpose and– Great people to work with

• www.lds.org or www.mormon.org

© IRI 2010 5

My Organization

Common Method MDM Sharing

MDM

Systems of Record (Enterprise info)

Pass-through accounts

2800+

ViewsDB Links

Master Data Management

Repository

Consumer Systems

6© IRI 2011

Data Management

• Provide accurate data to Church applications • Improve consistency of data reporting across the Church • Ensure the security of data • Enforce policies for use of data • Reduce development time • Eliminate the need to maintain master data in each

application • Decrease maintenance costs

7© IRI 2011

MDM Goals

• Designate authoritative sources of master data and select stewards

• Provide interfaces for cleansing and enrichment of data • Share master data from the source with other applications • Create a common data model and interfaces for

using master data in applications • Centralize administration of security and data use policies

(at the application level)

8© IRI 2011

MDM Strategy

Table 1

Relational data models Dimensional data models

Table3

Rel 1 2

Table4

Table 2

Rel 1 2 3

Dim 1

Dim 2

Dim 3

FactETL

How to store data in the MDM RepositoryRelational – Dimensional – Canonical Models

• Used for transactional systems• Normalized (at least 3NF, usually at 5NF)• Optimized for CRUD transactions (insert,

update, delete)

• Used for data warehouses• Denormalized: Star schema with

dimensions (entities), facts (measures)• Optimized for Reporting (ad-hoc queries)

Canonical data models

Table 1 2

Table 3 4

ETL

How to store data in the MDM RepositoryRelational – Dimensional – Canonical Models

Table 1

Relational data models

Table3

Rel 1 2

Table4

Table 2

Rel 1 2 3

• Used for transactional systems• Normalized (at least 3NF, usually at 5NF)• Optimized for CRUD transactions (insert,

update, delete)

• Used for reference by other systems• Semi-denormalized (1NF or 2NF)• Optimized for fast access by systems

(usually used in dropdowns)

MDM_ORG

ORG_ID ORG_NAME ORG_STATUS PARENT_ORG …

1 HR Department Active CPBO …

2 ICS Department Active CPBO …

3 ISD Department Discontinued CPBO …

… … … …MDM_ORG

ORG_ID ORG_NAME ORG_STATUS PARENT_ORG …

1 HR Department Active (NULL) …

2 ICS Department Active (NULL) …

MDM Repository

Target Application

• The target receives only “Active” organizations and it does not receive PARENT_ORG.

• Every target receives the same data model.• The target receives the information via Replication,

Database Link, or Web Service.

Need to deliver only the information (rows andcolumns) authorized by the Data Steward…

• We need to constrain the results at both the row and column level to meet data sharing agreement constraints.

• The MDM Repository is hosted on an Oracle database.

• Oracle Virtual Private Database (VPD) permits the creation of policies to control database access at the row and column level.– These policies essentially add a dynamic WHERE clause to

the SQL statement issued against the table, view, or synonym.

Why use Oracle VPD capabilities?

• Establishing the repository:– Selection of a data source (MDM_User or another source)

• Publishing the repository:– Create MDMR schema with a common data model in each

consuming application database • Build applications that use foreign keys to the master data instead of

copying the data– Use Data Integrator to replicate data to MDMR schemas

• Re-use same code for every application– Use VPD to enforce data use policies specific to each application

• Create a passthrough account on the source– Administer MDMR data privileges for each application centrally

(Integration team)

13© IRI 2011

MDM Implementation includes:

Systems of Record (Master data)

MDM_USERschema

MDM Source

OrganizationsLeaders

Finance

Other systems

14© IRI 2011

1) Master data is integrated, cleansed, enriched, and transformed to the MDM data structures from the systems of record.

Master Data Management – Architecture

Systems of Record (Master data)

MDM_USERschema

MDM Source

MDM_SECschemaOrganizations

Leaders

Finance

Other systems

After Login

Trigger

15© IRI 2011

1) Master data is integrated, cleansed, enriched, and transformed to the MDM data structures from the systems of record.

2) A set of security packages is defined to restrict columns and rows.

Master Data Management – Architecture

Systems of Record (Master data)

Business Objects

Data Integrator

MDM_USERschema

MDM Source

MDM_SECschemaOrganizations

Leaders

Finance

Other systems

After Login

TriggerAPP_PTM

3) The After-login trigger applies security policies to the MDM accounts.

16© IRI 2011

1) Master data is integrated, cleansed, enriched, and transformed to the MDM data structures from the systems of record.

2) A set of security packages is defined to restrict columns and rows.

Master Data Management – Architecture

Systems of Record (Master data)

Business Objects

Data Integrator

MDM_USERschema

MDM Source

MDM_SECschemaOrganizations

Leaders

Finance

Other systems

After Login

TriggerAPP_PTM

4) Security policies limit the rows and columns available to each MDM account.

3) The After-login trigger applies security policies to the MDM accounts.

17© IRI 2011

1) Master data is integrated, cleansed, enriched, and transformed to the MDM data structures from the systems of record.

2) A set of security packages is defined to restrict columns and rows.

Master Data Management – Architecture

Systems of Record (Master data)

Business Objects

Data Integrator

Requesting Application Instance

MDMRschemaMDM_USER

schema

MDM Source

MDM_SECschemaOrganizations

Leaders

Finance

Other systems

After Login

TriggerAPP_PTM

4) Security policies limit the rows and columns available to each MDM account.

3) The After-login trigger applies security policies to the MDM accounts.

5) Data is published to the MDMR at set frequencies.

18© IRI 2011

1) Master data is integrated, cleansed, enriched, and transformed to the MDM data structures from the systems of record.

2) A set of security packages is defined to restrict columns and rows.

Master Data Management – Architecture

Systems of Record (Master data)

App.schema

Business Objects

Data Integrator

Requesting Application Instance

MDMRschemaMDM_USER

schema

MDM Source

MDM_SECschemaOrganizations

Leaders

Finance

Other systems

After Login

TriggerAPP_PTM

APP_PTC Application

4) Security policies limit the rows and columns available to each MDM account.

3) The After-login trigger applies security policies to the MDM accounts.

5) Data is published to the MDMR at set frequencies.

6) Requesting App can use IDs as FKs, but should not copy data.

.

19© IRI 2011

1) Master data is integrated, cleansed, enriched, and transformed to the MDM data structures from the systems of record.

2) A set of security packages is defined to restrict columns and rows.

Master Data Management – Architecture

• A database package is defined for each object to be protected.

• The database package functions reference information stored in the Application Context of the current database session and returns the SQL fragment to be either:– Appended to the Where Clause or– Used to determine whether or not to show the

column contents in the result.

How are the policies defined?

CREATE OR REPLACE package MDM_ORG_SEC_PKG as function SET_PREDICATE (OBJECT_SCHEMA VARCHAR2,OBJECT_NAME VARCHAR2) return VARCHAR2; -- row policies... function VIEW_TIMEZONE (OBJECT_SCHEMA VARCHAR2,OBJECT_NAME VARCHAR2) return VARCHAR2; -- column policiesend MDM_ORG_SEC_PKG;/CREATE OR REPLACE package body MDM_ORG_SEC_PKG as CTX_VALUE VARCHAR2(2000) :=NULL; CTX_NAME VARCHAR2(30) :='MDM_SEC_CTX'; -- defines the application context function SET_PREDICATE (OBJECT_SCHEMA VARCHAR2,OBJECT_NAME VARCHAR2) return VARCHAR2 as begin CTX_VALUE:=NULL; CTX_VALUE:=SYS_CONTEXT('USERENV','SESSION_USER'); -- identifies the user associated with this session IF CTX_VALUE = OBJECT_SCHEMA THEN RETURN '1=1'; -- if the user is the same as the object owner, enable all access END IF; CTX_VALUE:=NULL; CTX_VALUE:=SYS_CONTEXT(CTX_NAME,'ORG_PREDICATE',2000); -- gets the where clause fragment to be appended IF CTX_VALUE IS NULL THEN RETURN '1=0'; -- if there is no information in the context, deny all access ELSE RETURN CTX_VALUE; -- return the row constraints END IF; EXCEPTION WHEN OTHERS THEN RETURN '1=0'; end;... function VIEW_TIMEZONE (OBJECT_SCHEMA VARCHAR2,OBJECT_NAME VARCHAR2) return VARCHAR2 as begin…

Sample policy package

• First, we define the objects and protected column lists in the MDM repository security tables.

• Then we execute a set of scripts to:– Create a stored procedure that will be used to

populated the application context when the user connects.

– Create the application context and tie to the above stored procedure.

– Tie the VPD policies to the database objects.

How do we connect the policy to thedatabase object?

Defines the objects (tables, views, synonyms) to be protected

Defines the columns to be protected

MDM_VPD_TABLE_PK TABLE_SCHEMA TABLE_NAME DESCRIPTION APP_CONTEXT POLICY_GROUP

POLICY_FUNCTION_SCHEMA PREDICATE_POLICY_NAME PREDICATE_FUNCTION_NAME LAST_UPDATE

1… MDM_ORG MDM Organization Table MDM_SEC_CTX MDM_ORG_SEC_POLICY_GRP … ORG_PREDICATE MDM_ORG_SEC_PKG.SET_PREDICATE …… … … … … … … … … …

MDM_VPD_COL_SET_PK MDM_VPD_TABLE_PK COL_SET_POLICY_NAME COL_SET_FUNCTION_NAME SECURED_COLUMN_LIST LAST_UPDATE68 1 ORG_TIMEZONE MDM_ORG_SEC_PKG.VIEW_TIMEZONE TIME_ZONE_CODE, TZ_DATABASE_CODE …

… … … … … …

MDM Repository Security Tables (part 1)

For the specific user, thisdefines the where clausefragment …

and the column sets to includewith the default columns.

USER_NAME MDM_VPD_TABLE_PK VPD_PREDICATE LAST_UPDATE DSA_NAMESOME_USER 1 ORG_STATUS = 'Active' … …… … … … …

USER_NAME MDM_VPD_COL_SET_PK LAST_UPDATE RULESOME_USER 68 … org_sensitive_0_1_flag=0… … … …

MDM Repository Security Tables (part 2)

• Now, when the SQL statement is executed:SELECT * FROM MDM_ORGOnly those rows / columns authorized are returned in the result set.

We’re done!

• Reference Data Sets– Languages– Geopolitical Locations (Countries, Regions, etc.)– Currencies– Exchange Rates

• Master Data Sets– Organizations– Leaders– Employees & reporting hierarchy – in progress– Physical Facilities (Churches, Temples, Seminaries, etc.) – in

progress

What do we have in our MDM?

• We replicate authorized master data to 54 other productions systems (230+ instances) daily.

• Developers become familiar with the canonical models for master data, which reduces the development cost of using master data.

• Web services deliver master data from the MDM repository.

• It takes about 20 minutes to provision master data to a new consumer.

• Changes in source systems are completely transparent to the consuming systems.

• Reduce number of DB links to source systems.

How successful is our MDM?

• Questions & Answers

• Pablo Riboldi Solution Manager for Information Governance & Quality riboldipj@ldschurch.org The Church or Jesus Christ of Latter-day Saints Visit us at www.mormon.org

Thank you!