dqs & mds in sql server 2016

Data Quality Services and Master Data Services in SQL Server 2016

2017-03-15

Agenda

• Introduction : 5 min.• Reality check, do you have these issues?• What is master data management?

• Master data management in SQL Server 2016 : 15 min.• Data Quality Services (DQS)• Master Data Services (MDS)• What’s new in SQL Server 2016 for MDS?

• Data Quality Services demo : 15 min.• Cleansing & Matching data using Excel MDS Add-in• Cleansing data using SSIS

• Master Data Services demo: 20 min.• Creating model, entities, attributes and business rules• Load data using SSIS

33, Prince Street, Suite 284, Montreal (Quebec) H3C 2M7

Sébastien NotebaertSenior Consultant

[email protected]

Microsoft Data Platform expert with 15+ years experience in deploying data warehousing, analytics, data integration and MDM solutions.

• Services :• Projets BI• Conseil BI

• Fondée en 2010• 35 experts• Clientèle :

• Grande entreprise• PMEs

• Services :• Infra Cloud (Azure)• Solutions Cloud

• Nouvelle pratique• Issue de l’acquisition

GTO• 2 experts, recrutement

actif• Clientèle :

• Grande entreprise• PMEs

• Services gérés• Cloud privé• Cloud Public


• PMEs• Domaine Médical

• Produits :• Solution BI

Financier• Solution processus

budgétaire• Reporting ERO


• PMEs

Who we are…

Reality check: do you have these issues?

• Do you have instances of invalid data impacting business processes?

• Do you wish your business users could manage data themselves such as Customer and Product?

• Do you have IT resources spending time investigating data quality issues and/or applying data fixes?

• Do you have the need for consolidation and distribution of data to other systems?

• Do you have an environment of heterogeneous systems which all could benefit from a single view of domain data such as Customer or Product?

-> You need master data management!

What is master data management?

• The technology, tools, and processes required to create and maintain consistent and accurate master data

• Master data is a set of data objects that are at the center of business activities like: Customers, Products, Cost Centers, Locations, …

• Master data is not transactional data like a sale or inventory movement

• To succeed an MDM initiative must include business processes, people AND data

MDM Architecture Style

Style Description Which

System is

Master

Update

Source

Systems

Comple

xity

REGISTRY Master data is not consolidated, but is maintained as a set of “stub”

records mapped to attributes stored in the source systems. There

is little data movement other than create or delete global stub

records in the registry. The golden record is assembled dynamically

using complex distributed queries. The upside is that a real time

central reference can be made with little or no infrastructure

investment. The downside is that without central governance of the

data the golden record isn’t highly reliable.

Source

system

Not

applicable

Data stays

in source

systems

1

CONSOLIDATED Master data is consolidated from multiple source systems into a

physical golden record for downstream consumption; however, any

updates made to the master data are not returned to the original

sources. Consolidated MDM hubs are quick and inexpensive to set

up, and offer a big return by enabling reliable enterprise-wide

reporting. Data movement is typically tolerant of inter-day latency

and managed through inexpensive batch processes, but it’s a one-

way street from source systems.

Source

system

No 2

MDM Architecture Style (continued)

Style Description Which

System is

Master

Update

Source

Systems

Comple

xity

COEXISTENCE Just like consolidated, coexistence harmonizes multiple sources into a

physical golden record for downstream consumption. Coexistence

adds the important step of updating the source systems. These

requirements are typically high latency and can usually be met at an

acceptable time and cost through bi-directional batch processes.

Coexistence is a natural evolution of the consolidated architecture

with the added benefit of linking centrally governed data back to the

source systems. Interfacing with complex data sources, such as ERP

systems, can become a costly drawback.

Source

system

Yes 3

TRANSACTIO

OR

CENTRALIZED

While this approach also creates a physical golden record, the key

differentiator from the consolidated and coexistence styles is the

MDM hub extends enterprise governance over the source systems.

This introduces shorter latency requirements that are typically

addressed with a combination of web services integration and an

authoring application that bridges centralized and application-

specific governance needs. In most cases, this is a big step up in cost,

complexity and implementation time. The pay-off is more

comprehensive governance in real-time; however, too many complex

sources can push cost and complexity beyond the value created by

the architecture.

MDM

system

Yes 4

Master Data Management Entity Examples

People Things Places Abstract

Customers

Vendors

Sales People

Employees

Partners

Patients

Products

Business Units

Bill of Materials

Parts

Storage Bins

Equipment

Locations

Stores

Wells

Power Lines

Geo Areas

Warehouses

Accounts

Warranties

Time

Metrics

Securities

Contracts

Master Data Management Solution Areas

Data Quality

Improve Efficiency

Compliance

Retain Customers

Merger & Acquisition

Cross Reference

Golden Records

• Human typographical errors; incomplete information; spreadsheet data management• Mergers and consolidation; ERP implementations, consolidation or migration• New purposes for old data; retire old applications such as mainframe applications• Single point of data maintenance; BI reporting

• Different types of customer accounts• Accurate view of data by implementing MDM and DQ • Single point of data maintenance• Cross sell and upsell

• Tracking spends by customer • Province and federal legislation

• Single view of customer spend, channels, cross sell and upsell

• Cross reference of same customers across multiple systems• Survivorship of best consolidated data across multiple systems

• Single view of anything that has attributes that can be matched• Cleanup of source systems with business rules and golden records pushed back

• Merging chart of accounts; consolidate financial reporting • Single view of product• Single view of customers

Master Data Management in SQL Server 2016

Master Data Management in SQL 2016

MDS & DQS in a Consolidated MDM Architecture

Data Quality Services

Data Quality Services

• SQL Server Data Quality Services (DQS) is a knowledge-driven data quality tool.

• DQS enables you to build a knowledge base and use it to perform a variety of critical data quality tasks including: correction, enrichment, standardization, and de-duplication of your data.

• DQS is like a spell checker for your data

• DQS was shipped in SQL Server 2012 (ent. & BI ed.)

Data Quality Services (continued)

• DQS consists of Data Quality Server and Data Quality Client, both of which are installed as part of SQL Server 2016.

• Data Quality Server is a SQL Server instance feature that consists of three SQL Server catalogs with data-quality functionality and storage

Data Quality Services Building Blocks

• DQS building blocks• Knowledge base

• Domains• Knowledge discovery• Matching policy

• Data Quality projects• Cleansing project• Matching (deduplication)

project

Using Data Quality Services

Create DomainsKnowledge Discovery

Clean Data Match Data

• Create a domainlike product brand or category

• Set it properties • Set it values• Set it rules• Set its term

based relations

• Train your domains by loading data

• Accept / reject / enrich / correct your domains

• Clean data setsusing your trained knowledge base

• Match and deduplicateyour data usinga matching policy

Data Quality Services Features & Tasks

Knowledge Bases and Domains•Building a KB•Importing and Exporting Knowledge•Managing a domain•Managing a composite domain

Data Quality Projects•Create a Data Quality project•Open, Unlock, … a DQ project•Open an SSIS Project in DQ client

Data Cleansing•Cleanse Data Using DQS•Cleanse Data in a Composite Domain

Source: https://msdn.microsoft.com/en-us/library/hh213042.aspx

Data Matching•Create a matching policy•Run a Matching Project

Reference Data Services in DQS•Configure DQS to use reference data•Attach domain to reference data•Cleanse data using reference data

Data Profiling and Notifications

DQS Administration

DQS Security

-> Topics in bold will be covered during the demo

Master Data Services

Master Data Services

• SQL Server Master Data Services is an Master Data Management (MDM) tool to define and manage non-transactional lists of data, with the goal of compiling maintainable and validated master lists for the enterprise.

• An MDM project generally includes an evaluation and restructuring of internal business processes along with the implementation of MDM technology. The result of a successful MDM solution is reliable, centralized data that can be analyzed, resulting in better business decisions.

• MDS was shipped in SQL Server 2008 R2 (datacenter & ent. ed.)

Master Data Services (continued)

Master Data Services has the following components:• Master Data Services Configuration Manager, a tool you use to create and

configure MDS databases and web applications.

• Master Data Manager, a web application you use to perform administrative tasks (like creating a model or business rule), and that users access to update data.

• MDSModelDeploy.exe, a tool you use to create packages of your model objects and data so you can deploy them to other environments.

• Master Data Services web service (WCF), which developers can use to extend or develop custom solutions for Master Data Services.

• Master Data Services Add-in for Excel, which you use to manage data and create new entities and attributes.

Master Data Services Building Blocks• The model is the most fundamental object in an MDS solution• Models are the containers that encapsulate all other MDS objects (i.e. entities, attributes and business rules)

Interacting with Master Data Services

• Master Data Manager Web User Interface• MDS Web Services API (WCF API)• Subscription views via T-SQL• Staging stored procedures• MDS Add-in for Excel

SSIS package that calls MDS stored procedures ->

Master Data Manager Web Application

Work with master data

Work with models, entities, attributes, business rules

Master Data Services Excel Add-in• The Excel Add-in for MDS offers users functions that can be found in the Master Data

Manager web app

• Users can update and view master data, as well as modify or create entities

• A major benefit of the Excel Add-in is the ability to quickly bulk load data into MDS

• The Excel Add-in provides users the ability to use Data Quality Services to clean data

before it moves into MDS

Using Master Data Services

Create Entities Load Members Validate Review Version

• Createentities and attributes to hold your master data

• Define business rules to ensure quality

• Load members using the staging process, Excel add-in or the API

• Validate the model’s data loaded to ensure it complies to all business rules

• Correct and enrich data that fails validation

• Version your model so that subscribing apps access only valid data

Massive improvements to performance and scalability

15x

MDS: What’s new in SQL Server 2016?

Transaction log retentionConfigurable settings for retaining the MDS transaction history table to enable automatic truncation. New member retention mode.

Display name for each objectGives more control over the names displayed for a given object – including the Code and Name attributes.

Entity Sync

Modeling and Management


Type 2 subscription viewsSee attribute change history in an easy to consume format.

Custom and compound IndexesAdd a custom index on commonly used attributes to improve overall performance.

Business rules supporting custom SQL Scripts

One level approval workflows

Modeling and Management


Granular security permissionsAllows permissions to be set around read, write, create, and delete.

Multiple administrator rolesSupport for Super User and Model Admin roles allows for multiple system administrators, and model level admins.


User Experience Improvements


-> Silverlight dependency implies using IE, no Chrome or Safari support

Master Data Services Features & Tasks

Create Structures to Contain Data•Models•Entities•Custom Index•Attributes•Domain-Based Attributes •Attribute Groups

Maintain Master Data•MDS Add-in for Microsoft Excel•Members•Business Rules•Transactions•Annotations•Hierarchies

Source: https://msdn.microsoft.com/en-us/library/hh231022.aspx

Improve Data Quality•Validation•Versions•Notifications •Security

Move Data•Importing Data •Exporting Data •Deploying Models

Develop a Custom Application•Developer's Guide •Microsoft.MasterDataServiceNamespace

-> Topics in bold will be covered during the demo

References

Master Data Services• Microsoft Master Data Services in SQL Server 2012 by James Serra: http://bit.ly/QW6kpQ• Master Data Overview: https://msdn.microsoft.com/en-us/library/ff487003.aspx• Master Data Features and Tasks: https://msdn.microsoft.com/en-us/library/hh231022.aspx• Master Data Developer’s Guide: https://msdn.microsoft.com/en-us/library/hh230994.aspx• What’s New in MDS 2016 : https://msdn.microsoft.com/en-us/library/ff929136.aspx

Data Quality Services• Data Quality Services: https://msdn.microsoft.com/en-us/library/ff877925.aspx• DQS Matching Transform for SSIS: https://ssisdqsmatching.codeplex.com/• MS DQS Blog using DQS Matching Transform:

https://blogs.msdn.microsoft.com/dqs/2013/06/25/automating-the-data-matching-process-in-sql-server-data-quality-services-dqs/

http://bit.ly/QW6kpQ

https://msdn.microsoft.com/en-us/library/ff487003.aspx

https://msdn.microsoft.com/en-us/library/hh231022.aspx

https://msdn.microsoft.com/en-us/library/hh230994.aspx



https://ssisdqsmatching.codeplex.com/

https://blogs.msdn.microsoft.com/dqs/2013/06/25/automating-the-data-matching-process-in-sql-server-data-quality-services-dqs/

We are recruiting!

PROFILES

BI and Analytics Architect

Data Architect

Microsoft BI Developper

Analytics and Data Visuzlization Specialist

2

TECHNOLOGIES

Microsoft BI (SSAS, SSIS, SSRS)

Power BI

Tableau

MDM

SQL Server

[email protected]

mailto:[email protected]

THANK YOU FOR YOUR TIME!

THANK YOU TO THE DATA SCIENCE INSTITUTE FOR THIS OPPORTUNITY!

DRIVE SAFELY ON YOUR WAY HOME!

☺

33, Prince Street, Suite 284, Montreal (Quebec) H3C 2M7

dqs & mds in sql server 2016

Technology