microsoft sql server distributing data with r2 bertucci

DistributingDistributing Data Techniques withData Techniques withDistributing Distributing Data Techniques with Data Techniques with SQL Server 2008 R2SQL Server 2008 R2QQ

Speaker: Paul Bertuccid b hi hFounder, Database Architechs

Silicon Valley SQL Server User GroupOctober 19, 2010

Mark Ginnebaugh, User Group Leader, [email protected]

Distributing Data TechniquesDistributing Data Techniques with SQL Server 2008 R2

Copyright 2010 All Rights Reserved

Agendag• Determining where data is needed

• What SQL Server options do you have

• How to get it there

• Work within your constraints Security

Regulatory (like PII…)

Ad i i t ti l Administratively

Data volatility/sizes

SLA’s you must deliver

Disaster Recovery/Business Continuity

Skills/Knowledge of your team

Resources (like $$’s, so on)

Reasons to Distribute DataReasons to Distribute Data

19FIG02OLTP DB

SQL Server 2008

PrimaryOLTP

Rpt DB

SQL Server 2008

ReportingServer

Reporting/ODS

OLTP DB pt

SQL Server 2008

EuropeEnabling/Partitioning

xyz DB

SQL Server 2008

USA(Headquarters)

xyz DB

Server

SQL Server 2008

AsiaServerxyz DB

xyz DB

Server

xyz DB

SQL Server 2008

North AmericaRegion

xyz DB

SQL Server 2008

EuropeRegion

Regionalization

(multiple owners)y y

SQL Server 2008

Primary

SQL Server 2008

Hot Spare

(multiple owners)

Failover

xyz DB

Primary

xyz DB

Hot Spare(Fail-over)

Distributing periodic updates toData Marts from a “master” Data Warehouse

DataMart

SQL Server2008

MasterData Warehouse

DataDataMart

DataMart

SQL Server2000

SQL Server2005

ORACLE

Enabling a Data Warehousefrom Legacy Systems

MasterData Warehouse

SQL Server2005

SQL Server2008

HeterogeneousInitialization of tables HeterogeneousSubscriber

Initialization of tablesfor a heterogeneous Subscriber

in Replication

ORACLE

Publisher

Subscriber

SQL Server 2008

SQL Server2008

Subscriber

SQL Server 2005

Pulling in Sales Order informationfrom disparate sources

for import into SQL Server 2008Sales Force Automation

Application for import into SQL Server 2008

Access

Application

Access

SQL Server2008Excel 2008Excel

Data Mart

SQL Server2008

OLTP Database

2008

Export from the OLTP Database

ExcelSQL Server

2008

Export from the OLTP Database to a business intelligence

Data Mart & Excel

Connection

SQL Server2008

SQL Server2008

Results Results

2008

Client Local Server2008

Remote Server

Decision‐Tree Path Traversal

I t ti

DatabaseMirroring

With S h tIntegrationServices

With Snapshots

1a2c3 1a2d3 Detach/

1a2

1a2c3

1 2 3

1a2b3

1a2d3

1a2e3

ab c d

e

BCP

Detach/Attach

1 LogShippingSync

Framework

1b2bac

de

1c2

1e2

1a2a3

Framework 1c21d2

Backup/Restore

Data Replication

LinkedServersServers

DataReplication

LogShipping

Database Mirroringwith Snapshots

LinkedServers

SSIS BCPBackup/Restore

Detach/Attach

SyncFramework

LargeData

HighVolatilityVolatility

On Line

Many Targets

LowLatency

Transformation/Transformation/Filtering

Data ReplicationData Replication

Central Publisher(default option)SQL Server

2008SQL Server

20052008

AdventureWorks2008

translog

2005

AdventureWorks2008

translogSQL Server

19FIG10

translog translogSQL Server2008

AdventureWorks2008

SubscriptionServer

SubscriptionServer

translogPublication

Server

Oracle

SQL Server2000

distribution

DistributionServer

AdventureWorks2008

AdventureWorks2008

translog

Server

Subscription Subscription

1515

SubscriptionServer Server

19FIG11

Central PublisherRemote DistributorSQL Server

2008SQL Server

2005

SQL Server2008

2008

AdventureWorks2008

translog

2005

AdventureWorks2008

translog2008

AdventureWorks2008

translog

translogSubscription

Server


Server

translogPublication

Server

distribution

SQL Server2008

Oracle

SQL Server2000

Remote DistributionServerAdventure

Works2008

Subscription

AdventureWorks2008

translogSubscriptionSubscription

Server Server

Publishing Subscriber

SQL Server2008

SQL Server2005

19FIG12

SQL Server2008

AdventureWorks2008bl

icat

ion

Serv

er

2008

AdventureWorks2008

translog

2005

AdventureWorks2008

translogtranslog

distribution

PuS

istr

ibut

ion

Serv

er


Server


Server

Dpt

ion

erSQL Server

2008

tion

erSu

bscr

iSe

rve

AdventureWorks2008

translogOracle

SQL Server2000

2008

Publ

icat

Serv

e translogAdventureWorks2008

Subscription

AdventureWorks2008

translogSubscriptionio

nr

1717

SubscriptionServer Serverdistribution

Dis

trib

uti

Serv

er

19FIG13 CentralSQL Server

2008tio

ner

SQL Server2008

tion

er19FIG13 CentralSubscriber

AdventureWorks2008

translogPubl

ica

Serv

eio

nr

AdventureWorks2008

translogPubl

icat

Serv

eon

SQL Server 2008

SubscriptionServer

distribution

Dis

trib

uti

Serv

er

distribution

Dis

trib

uti

Serv

er

AdventureWorks2008

Server

SQL Server2008

atio

nve

r

SQL Server2008

atio

nve

r

AdventureWorks2008

translogPubl

icSe

rvut

ion

er

AdventureWorks2008

translogPubl

ica

Serv

utio

ner

distribution

Dis

trib

uSe

rve

distributionD

istr

ibu

Serv

e

19FIG14

Multiple Publishersof a Single Table

on onSQL S SQL S19FIG14

on rSu

bscr

iptio

Serv

er

AdventureWorks2008 on

Subs

crip

tioSe

rver

AdventureWorks2008

SQL Server2008

SQL Server2008

Publ

icat

ioSe

rver translog

Publ

icat

ioSe

rver translog

distribution

Dis

trib

utio

nSe

rver

distribution

Dis

trib

utio

nSe

rver

Territory 1

CustomerID TerritoryID AccountNumberCustomerType rowguid ModifiedDateCustomers Table

Territory 2

1345 1 AW1345 I X69G9.. 120203

1356 2 AW1356 I W211G.. 051605

2354 1 AW2354 S 7SQ78K.. 106705

3346 2 AW3346 I W12DV.. 022305

7643 2 AW7643 S WZ8R4.. 122205

7901 2 AW7901 I S2345X.. 041506

8921 1 AW8921 I RT66Y.. 031206

U d ti

19FIG15

Updating Subscriber

SQL Server2008

AdventureWorks2008bl

icat

ion

Serv

er

SQL Server Queue(Queue Reader Agent)

Works2008

translog

distributionPu

b Sst

ribut

ion

Serv

er

Update Customersset AccountNumber = null

where CustomerID = 12345

Dis S

SQL Server2008

Adventure

SQL Server2008

AdventureW k 2008Works2008


Server

Works2008


Server

19FIG16SQL Server

2008SQL Server

200819FIG16

Peer-to-PeerAdventureWorks2008

translogPublication

AdventureWorks2008

translogPublicationPublication

ServerPublication

Server

North American Asiao e caServer

s aServerSQL Server 2008

distribution

Distribution ServerPeer Originator id: 1 Peer Originator id: 2

C t ID T it ID A tN b C T id M difi dD tCustomers Table

1345 1 AW1345 I X69G9.. 120203

1356 2 AW1356 I W211G.. 051605

2354 1 AW2354 S 7SQ78K.. 106705

3346 2 AW3346 I W12DV 022305

CustomerID TerritoryID AccountNumberCustomerType rowguid ModifiedDate

3346 2 AW3346 I W12DV.. 022305

7643 3 AW7643 S WZ8R4.. 122205

7901 5 AW7901 I S2345X.. 041506

8921 4 AW8921 I RT66Y.. 031206

Publisher

P bli ti

19FIG05AdventureWorks2008

AW_VerticalPublication

19FIG05SQL Server

2008

translog CustomerV (Article)

1345 1 AW1345 I X69G9.. 120203


1345 1 I

CustomerID TerritoryID CustomerType

1356 2 AW1356 I W211G.. 051605

2354 1 AW2354 S 7SQ78K.. 106705

3346 2 AW3346 I W12DV.. 022305

7643 3 AW7643 S WZ8R4.. 122205

1356 2 I

2354 1 S

3346 2 I

7643 3 S

7901 5 AW7901 I S2345X.. 041506

8921 4 AW8921 I RT66Y.. 031206

7901 5 I

8921 4 I

Publisher

AW_HorizontalPublication

SQL Server

AdventureWorks2008

translog

19FIG06CustomerH (Article)

1345 1 AW1345 I X69G9.. 120203


SQL Server2008

1345 1 AW1345 I X69G9.. 120203

1356 2 AW1356 I W211G.. 051605

2354 1 AW2354 S 7SQ78K.. 106705

3346 2 AW3346 I W12DV.. 022305

7643 3 AW7643 S WZ8R4.. 122205

Only these Rows!7643 3 AW7643 S WZ8R4.. 122205

7901 5 AW7901 I S2345X.. 041506

8921 4 AW8921 I RT66Y.. 031206


2354 1 AW2354 S 7SQ78K.. 106705

7643 3 AW7643 S WZ8R4.. 122205

yp g

Publisher

19FIG07Ad t

AW_H_and_V

Publication

SQL Server

AdventureWorks2008

translog07FIG08

CustomerHV (Article)

1345 1 AW1345 I X69G9.. 120203


SQL Server2008

1345 1 AW1345 I X69G9.. 120203

1356 2 AW1356 I W211G.. 051605

2354 1 AW2354 S 7SQ78K.. 106705

3346 2 AW3346 I W12DV.. 022305

7643 3 AW7643 S WZ8R4.. 122205

Only these Columns and these Rows!

7643 3 AW7643 S WZ8R4.. 122205

7901 5 AW7901 I S2345X.. 041506

8921 4 AW8921 I RT66Y.. 031206

CustomerID TerritoryID CustomerType

2354 1 S

7643 3 S

y CustomerType

AW_H_and_J (Publication)

19FIG08

07FIG09CustomerH (Article)

OrdersHJ (Article)

1345 1 AW1345 I X69G9 120203

CustomerID TerritoryID AccountNumberCustomerType rowguid ModifiedDate Only the orders for1345 1 AW1345 I X69G9.. 120203

1356 2 AW1356 I W211G.. 051605

2354 1 AW2354 S 7SQ78K.. 106705

3346 2 AW3346 I W12DV.. 022305

7643 3 AW7643 S WZ8R4 122205

2354

7643

CustomerID

for customer type “S”

SalesOrderID RevisionNumber OrderDate DueDate Status CustomerID

7643 3 AW7643 S WZ8R4.. 122205

7901 5 AW7901 I S2345X.. 041506

8921 4 AW8921 I RT66Y.. 031206

7643

join

43659 1 7/1/2001 7/13/2001 5 1345

43660 1 7/1/2001 7/13/2001 5 1356

43661 1 7/1/2001 7/13/2001 5 2354

43662 2 7/1/2001 7/13/2001 5 3346SalesOrderID OrderDate CustomerID

2525

43663 1 7/1/2001 7/13/2001 5 7643

43664 1 7/1/2001 7/13/2001 5 7901

43665 1 7/1/2001 7/13/2001 5 8921

43661 7/1/2001 2354

43663 7/1/2001 7643

Publisher SubscriberAW (P bli ti )

19FIG17

SubscriberSQL Server

2008

Adventurecatio

nrv

er

AW (Publication)

Customers (Article)SQL Server

2008AdventureWorks2008

translog

di t ib ti

Publ

icSe

rbu

tion

rver PUSH

2008

AdventureWorks2008

translog

PUSH: initiated by Publisher

distribution

Dis

trib

Ser

SubscriptionServer

PULL(agent will be on the Publisher if local distribution,

or on the Distributor if remote distribution) PULL: initiated by Subscribery

(agent will be on the subscriber)

Distributing DataData Access Latency Autonomy Sites(locations) Frequency Network Machines Owner Other REPLICATIONDistributing Data(locations)

Read OnlyReporting short high many high fast/

stablen

server/site1 OLTP

site

Each site only needs regionaldata

Central PublisherTransactional replfilter by region

Read OnlyReporting long high many low fast/

stablen

server/site1 OLTP

site


Central PublisherSnapshot replfilter by region

Read Mostly fast/ n Regional updates Central Publisheread ost yA few updates short high < 10 medium fast/

stablen

server/site1 OLTP

site

g pon one table Transactional repl

Updating Subs

Read MostlyA few updates medium high < 10 medium slow/

unreliabn

server/siteAll

update

Regional updateall tables

Central PublisherMerge repl

Read equalEqual updates short high < 10 medium fast/

stable1

server/siteAll

update

Regional updateall tables

Peer-to-PeerTransactional

repl

Inserts (new orders) short high many high fast/

stable1

server/site

1reportsite


Central SubscriberTransactional repl

/ f / Central PublisherHot/WarmSpare

Veryshort

high < 2 high fast/stable

1server/site

1 OLTPsite

Fail-overCentral PublisherRemote DistributorTransactional repl

Log ShippingYes, it is still here..,

Log Shipping

SQL Server 2008

“Source”

SQL Server 2008

“Destination”

T LCallOne DB

CallOne DB

Q

SecondaryServertranslog

TxnLogbackupsPrimary

Server

CallOne DB

\Backup\CallOne_tlog_200405141120.TRN

T L \LogShare\CallOne tlog 200405141120 TRN TxnLogpped

wer

TxnLogCopies

\LogShare\CallOne_tlog_200405141120.TRN TxnLog Restores

SQL S 2008

“Monitor”

Last

log

ship

Del

ay A

nsw

MSDB DB

SQL Server 2008

MonitorServer

Delay between logs loaded

Delay Answer

Server

Don’t build on this, it is being deprecated !

Database Mirroringwith Snapshotswith Snapshots

Network

Client Client ClientClient

SQL Server 2008 SQL Server 2008Mi S

A

DB MirroringAdventure Works DB

Mirror Server

t l

PrincipalServer

Adventure Works DBB

DD

translog translogC

SQL Server xyz

Witness

3131

MSDB DBt ess

Server

10SQL Server 2008 SQL Server 2008

Mirror ServerPrincipal

AdventureWorksDB

translog

Server

Database

AdventureWorksDB

translogDatabase

SQL Server 2008

SnapshotDatabase Snapshot

node

m

irror

.

MSDB DBWitnessServer

this

ser

ver n

com

es th

e m

If t

bec

3232

01SQL Server 2008

01AdventureWorks DB

translog

SourceServer

D t b

CREATE DB ‘xyz’AS SNAPSHOT OF

AdventureWorks

Database Snapshot

--- Reverting --- if needed ---Reverting if needed RESTORE DB

AdventureWorksFROM DATABASE_SNAPSHOT = ‘xyz’

Source DataPPages

Source

02SQL Server 2008

SQL SparseServer

AdventureWorksDB

SnapshotAdventureWorks

DB

FilePages

DB

Empty Sparse fileof a Snapshot just created

(no updates to original

3434

( p gdata pages have occurred yet)

SourceData

PPages

Source

03SQL Server 2008

SQL SparseServer

AdventureWorksDB


DB

FilePages

DB

System Catalog Copy of originalSystem Catalogof changed pages

Copy of originalpages for snapshot

only when a page is changed(Copy-on-write)

3535

(Copy-on-write)

Source DataPages

Snapshot Users

PagesSELECT …..data……. FROM AdventureWorks

SNAPSHOT

04SQL Server 2008

SQLSServer

AdventureWorksDB


DB

System Catalog SparseSystem Catalogof changed pages

SparseFile

Pages

3636

10SQL Server 2008 SQL Server 2008

Mirror ServerPrincipal

AdventureWorksDB

translog

Server AdventureWorksDB

translogDatabase

SQL Server 2008

Database Snapshot

Database Snapshot

node

m

irror

.

MSDB DBWitnessServer

this

ser

ver n

com

es th

e m

If t

bec

3737

PH Topology With SnapshotsPH Topology With SnapshotsReal timeR ti

SQL Server 2008

ReportingUsers

Net

wor

k

SQL Server 2008

PrincipalServer

n

N

Adventure Works DB

SQL Server 2008

Mirror Server

Adventure Works DB

Active

ster

ed

pplic

atio

n

icat

ion

translog translog

PHPassive

Clu

s

OLT

P A

Rep

l

Database SnapshotPH

TopologyPrincipalServer

Network

3838SQL Server 2008

Point in timeReporting Users

DB Mirror and ReplicationSQL Server 2008 SQL Server 2008

Publisher

PrincipalServer

MirrorServer

SQL Server 2008

Subscriber

Distributor

SQL Server 2008

Principal

SQL Server 2008

Mirror

s bu o

SQL Server 2008

PrincipalServer

MirrorServer

WitnessServer

SQL Server 2008

Subscriber

3939

Integration Services

Excel

OLAP Data Warehouse

Access

MultidimensionalMultidimensionalCubeOLTP

Databases

Other Data Sources

OLTP

Other Data Sources

OLTPDatabases

SQL Server

Flat filesWeb Services

XML

P k

Task

Package

Task

Task

aine

rSource e

Con

taData Flow Task

Source

Transformation

D ti ti

Cac

he

Task

Destination

Task

Task

Data MartSSIS

e

SQL Server2008

OLTP Database

sk to

d

Agg

rega

te

AdventureWorks 2008

Tas

Join

and

UnleashedDataMart

AdventureWorks

Export from the OLTP Database

Excel.csv flat file

SQL Server2008

Export from the OLTP Database to a business intelligence

Data Mart & Excel/.csv flat file

Linked Servers

SQL Server

OLE DB Providerfor MS SQL Server

SQL Server6.5 – 2008

ODBC

OLE DB Providerfor ODBC

SybaseInformixDB2Othersde

r DLL

’s

ODBC

MS OLE DBP id

OthersOLE DB Provider

for JetExcel spreadsheetFoxPro DatabaseDbase fileAccess Database

Jet database

fileOLE DB Providerf O lE

DB

Pro

viSQL Server

2008

Provider for Oracle

ORACLE

OLE DB P id

OLE

OLE DB Providerfor Indexing Service Active Directory Services

Index ServerOthers

Client Tier Server Tier Remote Data Source Tier (can be “Linked”)

Might be to easy?Might be to easy?--------------------------------------------------------------- Selecting data from Excel spreadsheet via Linked Server --Selecting data from Excel spreadsheet via Linked Server -------------------------------------------------------------SELECT substring(CustomerID,1,5) as 'CustID', substring(ContactName,1,18) as ContactName,

b t i (Cit 1 10) 'Cit ' b t i (C t 1 15) 'C t 'substring(City,1,10) as 'City', substring(Country,1,15) as 'Country'FROM [Linked ExcelSW]...[SWCustomers$]ORDER BY ContactNamegogCustID ContactName City Country------ ------------------ ---------- ---------------PICCO Adam Greifer Los Angele USABLAUS John Monroe Concord USABLAUS John Monroe Concord USACHOPS Martin Sommer Berkeley USA

Microsoft Sync FrameworkMicrosoft Sync FrameworkA comprehensive synchronization platform that enables collaboration and offline access forenables collaboration and offline access for applications, services, and devices. It features technologies and tools that enable roaming, sharing, and taking data offline.

B i S F k d l b ildBy using Sync Framework, developers can build sync ecosystems that integrate any application with any data from any store that uses any y y yprotocol over any network. Mentioned here because of its replication‐like behavior for “occasionally connected” applicationsoccasionally connected applications.

OthersOthers

• BCPBCP

• Detach/Attach

k /• Backup/Restore

Copy on Write Technology

DB Mirroring

Network

Client Client ClientClient

SQL Server 2008 SQL Server 2008Mi S

A

Adventure Works DB

Mirror Server

t l

PrincipalServer

Adventure Works DBB

DD

translog translogC

SQL Server xyz

Witness

5050

MSDB DBt ess

Server

SourceData

PPages

Source

03SQL Server 2008

SQL SparseServer

AdventureWorksDB


DB

FilePages

DB

System Catalog Copy of originalSystem Catalogof changed pages

Copy of originalpages for snapshot

only when a page is changed(Copy-on-write)

5151

(Copy-on-write)

Database Architechs

Paul BertucciPaul Bertucci Founder Database ArchitechsFounder Database Architechs Former Former Chief Data Architect Chief Data Architect –– VeritasVeritas & Symantec& Symantec C rrentl Chief Architect A todesk Currently Chief Architect - Autodesk Author of SQL Server 2000, 2005 & 2008/R2 Unleashed CoCo--Author of ADO.NET in 24 hours Author of ADO.NET in 24 hours Author MS SQL Server High AvailabilityAuthor MS SQL Server High Availability Author Sybase Performance & TuningAuthor Sybase Performance & Tuning Author Sybase Performance & TuningAuthor Sybase Performance & Tuning Author Sybase Physical DB Author Sybase Physical DB DesignDesign Author Chen Entity/Relationship ModelingAuthor Chen Entity/Relationship Modeling VeritasVeritas SQL Server Performance Series SQL Server Performance Series

[email protected]@dbarchitechs.com

To learn more or inquire about speaking opportunities, please contact:

Mark Ginnebaugh User Group LeaderMark Ginnebaugh, User Group [email protected]