Service-Oriented Data Architecture (SODA) Service-Oriented Data Architecture (SODA)

Jon JahrenJon JahrenProduktsjef, Microsoft NorgeProduktsjef, Microsoft Norgejonjah@microsoft.com

AgendaAgenda

Natural forces in dataNatural forces in data

Data in a services worldData in a services world

Introducing SODAIntroducing SODA

SQL 2005 SODA featuresSQL 2005 SODA features

Wrap upWrap up

Wave 1Wave 1

Processor

Code

Data New Data

“Data Processing”

Wave 2Wave 2

Enough storage to Enough storage to keep some data keep some data “online”“online”Processor

CodeData

“OLTP”

Wave 3Wave 3Enterprise Process and Enterprise Process and Data models led to Data models led to “Enterprise “Enterprise Applications”Applications”

Enterprise Applications Enterprise Applications linked via linked via “Integration” (EAI)“Integration” (EAI)EAI

ERP

HR

The Data MonolithThe Data Monolith

Previous 30 years Previous 30 years trended towards trended towards data consolidationdata consolidation

SOA, connected SOA, connected systems, common systems, common schema, pulling schema, pulling data apartdata apart

Data ProcessingData Processing

OLTPOLTP

EAEA

InternetInternet

B2B ServicesB2B Services

DevicesDevices

SOASOA

Dat

a C

onve

rgen

ce

Dat

a C

onve

rgen

ce

Data D

istribution

Data D

istribution

Data in a SOA WorldData in a SOA World

Implications of SOA on dataImplications of SOA on dataBreaking of data monolithsBreaking of data monoliths

Data becomes more distributedData becomes more distributed

Recognition of data classesRecognition of data classesDifferent treatment for different dataDifferent treatment for different data

SOA SOA → Event Driven Data Systems→ Event Driven Data Systems

Loose couplingLoose couplingHeterogeneous systemsHeterogeneous systems

TimeTime

SecuritySecurity

Consistency vs. DistanceConsistency vs. Distance

Transactions take Transactions take DB’s from one DB’s from one consistent state to consistent state to anotheranother

Doing this across Doing this across space and time is space and time is hard!hard!

AttemptsAttempts2-phase commit2-phase commit

Optimistic CCOptimistic CC

ReplicationReplication

DataDataConsistencyConsistency

““Distance”Distance”SpaceSpaceTimeTime

AuthorityAuthorityConnectednessConnectedness

Resilience and CorrectnessResilience and Correctness

Challenge:Challenge:• System CorrectnessSystem Correctness• Distance:Distance:

• SpaceSpace• TimeTime• AuthorityAuthority• ImplementationImplementation

• Loose couplingLoose coupling

Approach:Approach:• Acknowledge Acknowledge Separation Separation

• AuthorityAuthority• TransactionTransaction• ImplementationImplementation

• Recognize data classesRecognize data classes• Reference dataReference data

Result:Result:• Service OrientedService Oriented Database Architecture Database Architecture (SODA) (SODA)

• Native Web ServicesNative Web Services• Service BrokerService Broker• Service Logic HostService Logic Host (CLR) (CLR)• Query NotificationsQuery Notifications

Order Processing circa 1950Order Processing circa 1950

PaymentPayment

CatalogCatalogUpdatesUpdates

OrderOrderFormForm

CatalogCatalog

OrderOrderAckAck

InvoiceInvoice

OrderOrderCopyCopy

PickPickListList

InventoryInventory

InventoryInventory

LedgerLedger

Reference DataReference DataResource DataResource DataActivity DataActivity DataService Interaction DataService Interaction Data

Reference Data DefinedReference Data Defined

Used to create service requestsUsed to create service requestsFormat interpretable by all partiesFormat interpretable by all parties

Anyone can read the catalogAnyone can read the catalog

Has a stable identifierHas a stable identifierMarch 2005 catalogMarch 2005 catalog

Referenced by service requestReferenced by service requestItem 23, Page 64, March 2005 catalogItem 23, Page 64, March 2005 catalog

Can be infinitely replicated and cachedCan be infinitely replicated and cachedDoesn’t need to be 100% consistentDoesn’t need to be 100% consistent

Might be able to order from March catalog on Might be able to order from March catalog on April 1April 1st.st.

CatalogCatalog

Activity Data DefinedActivity Data Defined

Lifetime generally bounded by Lifetime generally bounded by business activity (order, itinerary, business activity (order, itinerary, etc.)etc.)

May be retained for reporting, etc.May be retained for reporting, etc.

Format is private to serviceFormat is private to service

Generally keyed to business activityGenerally keyed to business activityBackorder for order #268Backorder for order #268

Can be partitionedCan be partitioned

Can change partitioning without Can change partitioning without moving datamoving data

(Ephemeral – evens out over time)(Ephemeral – evens out over time)

PickPickListList

Resource Data DefinedResource Data Defined

Very long lifetimeVery long lifetimeSKU’s, Customers, Accounts…SKU’s, Customers, Accounts…

Format and update private to serviceFormat and update private to service

Highly concurrentHighly concurrentOne item’s “stock on hand” modified by One item’s “stock on hand” modified by many ordersmany orders

Re-partitioning requires moving Re-partitioning requires moving entriesentries

InventoryInventory

Service Interaction Data DefinedService Interaction Data Defined

The only way services communicateThe only way services communicate

Format interpreted by all partiesFormat interpreted by all partiesEveryone can read and interpret the Everyone can read and interpret the “order form”“order form”

It may get lostIt may get lostWhether mail, e-mail, internet, etc.Whether mail, e-mail, internet, etc.

If it gets lost, you send another copyIf it gets lost, you send another copyThe copy better be an exact replica of the The copy better be an exact replica of the lost onelost one

OrderOrderFormForm

Observations on Data TypesObservations on Data Types

Different Service Data Classes Have Different Service Data Classes Have Different CharacteristicsDifferent Characteristics

NoReadOnly

None NoneYesNo: HasOpen Schema

NoReadOnly

None NoneYesNo: Has

Open Schema

YesVeryLow

ObjectPersistance

OptimisticMaybeYes

YesMay Be

HighCOM+

TransactionalPessimistic(Locking)

MaybeYes

ServiceInteraction

Reference

Activity

Resource

Private toService? XML?

Con-currency

Needs En-capsulation?

ObjectStyle

ConcurrencyApproach

Kind ofData

Deployment TopologiesSingle Tier & Client/ServerDeployment TopologiesSingle Tier & Client/Server

All data updated All data updated directlydirectlyService interaction Service interaction data implicitdata implicit

(Order Data comes (Order Data comes from form)from form)

All data treated All data treated alikealike

CatalogCatalog

OrderOrderCopyCopy

PickPickListList

InventoryInventory

LedgerLedger

Deployment TopologiesMulti-TierDeployment TopologiesMulti-Tier

Reference Data Cloned Reference Data Cloned & Cached& Cached

Activity Data “Tightly Activity Data “Tightly Coupled” to back endCoupled” to back end

Application connected Application connected to app server in a to app server in a “Session”“Session”

InventoryInventory

LedgerLedger

CatalogCatalogCacheCache

CatalogCatalogMasterMaster

OrderOrderCopyCopy

PickPickListList

Application ServerApplication Server

ReplicationReplication

CatalogCatalogCacheCacheCatalogCatalog

CacheCache

Deployment TopologiesFully Distributed Connected ApplicationDeployment TopologiesFully Distributed Connected Application

InventoryInventoryServiceService

LedgerLedgerServiceService

CatalogCatalogMaint.Maint.

ServiceService

OrderOrderServiceService

KittingKittingServiceService

CatalogCatalogUpdatesUpdates

OrderOrder

OrderOrderAckAck

InvoiceInvoice

OrderOrder

PaymentPayment

PaymentPayment

Reference Data Cached Reference Data Cached RemotelyRemotely

Activity Data Managed by Activity Data Managed by servicesservices

All communications via Service All communications via Service Interaction Data (Messages)Interaction Data (Messages)

Supports Heterogeneous Supports Heterogeneous ImplementationsImplementations

What Role Databases?What Role Databases?

Role of databases in SOA?Role of databases in SOA?Reference Data Cache ManagementReference Data Cache Management

Service Interaction (Message) Data Service Interaction (Message) Data HandlingHandling

Activity Data Management Activity Data Management

Connected Sessions Connected Sessions → Event Driven→ Event Driven

Need Service Oriented Database Need Service Oriented Database Architecture (SODA)Architecture (SODA)

ScalingScaling

Techniques differ with dataTechniques differ with dataReference data Reference data

Shared Scalable DatabaseShared Scalable Database

Snapshot replication Snapshot replication

Query notificationsQuery notifications

Activity data Activity data Dynamic partitioningDynamic partitioning

Data dependent routingData dependent routing

Service interaction data Service interaction data Inherently asynchronousInherently asynchronous

Resource data Resource data Static partitioningStatic partitioning

Peer-to-peer or bi-directional replicationPeer-to-peer or bi-directional replication

Storage strategiesStorage strategies

Techniques differ with dataTechniques differ with dataReference data Reference data

Master copy in central databaseMaster copy in central database

Replicas onlineReplicas online

Activity dataActivity dataKeep copies for activity tracking, trending, miningKeep copies for activity tracking, trending, mining

Service interaction dataService interaction dataKeep EXACT copies of request and responseKeep EXACT copies of request and response

For retry and recoveryFor retry and recovery

Sometimes a legal requirementSometimes a legal requirement

Resource dataResource dataDivide processesDivide processes

Updates can be asynch with other categoriesUpdates can be asynch with other categories

Session Oriented DatabaseSession Oriented Database

Input & Events Input & Events processed by processed by applicationapplication

Passive DataPassive Data

Little binding Little binding between data and between data and “logic”“logic”

Database “Application”Database “Application”

Database SessionDatabase Session

Database “Application”Database “Application”

Service Oriented DatabaseService Oriented Database

Service Logic Service Logic bound to:bound to:

ServicesServices

Query NotificationsQuery Notifications

EventsEvents

Service Logic Service Logic Triggered:Triggered:

By data changesBy data changes

By message arrivalBy message arrival

By timer firingBy timer firing

Data becomes Data becomes “active”“active”

EventsEvents

Service LogicService Logic

Service LogicService Logic

Service LogicService Logic

Database “Application”Database “Application”

SODA RequirementsSODA Requirements

Event DrivenEvent DrivenCommunication endpoint for servicesCommunication endpoint for services

Great host for service logicGreat host for service logicDecoupled from messaging (asynch)Decoupled from messaging (asynch)

Support logic inside or outside DB Support logic inside or outside DB processprocess

Needs to support reference and Needs to support reference and activity data managementactivity data management

Support service centric programming Support service centric programming modelmodel

What about App Servers?What about App Servers?

Some SODA requirements Some SODA requirements implemented in Application Servers…implemented in Application Servers…

Message BrokeringMessage Brokering

Complex Logic HostingComplex Logic Hosting

Web Service HostingWeb Service HostingWeb Service Comm. EndpointsWeb Service Comm. Endpoints

Web Service MechanismsWeb Service Mechanisms

Caching supportCaching support

Analogy between TP-Lite and Analogy between TP-Lite and traditional TPtraditional TP

TP-Heavy vs. TP-LiteTP-Heavy vs. TP-Lite

TP-LiteTP-LiteEarly to mid 1990’s DB Vendors add TP Early to mid 1990’s DB Vendors add TP features to databasefeatures to database

Thread pooling/managementThread pooling/management

Logic hosting (stored procedures)Logic hosting (stored procedures)

Communication ManagementCommunication Management

Transaction coordinationTransaction coordination

Became “Good Enough” for most Became “Good Enough” for most needsneeds

Same thing will happen with SODASame thing will happen with SODA

SQL Server 2005 SODA featuresSQL Server 2005 SODA features

Query notificationsQuery notificationsFor scaling out data & presentation cachesFor scaling out data & presentation caches

Reference data scalingReference data scaling

Native web servicesNative web servicesService EndpointService Endpoint

Service brokerService brokerService centric architectureService centric architecture

Reliable messaging with complete database Reliable messaging with complete database integrationintegration

Service logic hostService logic hostCLR IntegrationCLR Integration

Service Broker

Query Notifications for Web CachingQuery Notifications for Web Caching

Query Processor

Response

Response

Request

Request

Query/Subscription

Result

Notification

IIS / ASP.Net SQL Server 2005

Dispatcher

Native SOAP accessNative SOAP accessStandards based access to SQL Standards based access to SQL ServerServer

No client dependencyNo client dependency

Improved InteroperabilityImproved Interoperability

New “ENDPOINT AS HTTP” objectNew “ENDPOINT AS HTTP” objectConfigure connection infoConfigure connection info

Configure authenticationConfigure authentication

Expose Functions & SPsExpose Functions & SPs

Expose TSQL BatchesExpose TSQL Batches

http://server1/aspnet/default.aspxhttp://server1/aspnet/default.aspx

http://server1/sql/pubs?wsdlhttp://server1/sql/pubs?wsdl

KernelKernelModeMode

ListenerListener

Asynchronous, Distributed, Decoupled Asynchronous, Distributed, Decoupled programming environmentprogramming environment

Fully integrated into the database engineFully integrated into the database engine

Programming Model - DDL and DMLProgramming Model - DDL and DML

Integrated Management and DeploymentIntegrated Management and Deployment

Dependent SQL Server 2005 features – Events, Dependent SQL Server 2005 features – Events, Query Notifications, SQLiMailQuery Notifications, SQLiMail

Customer Usage, benefits: Enables new Customer Usage, benefits: Enables new scenarios – async SPs, Scalable distributed scenarios – async SPs, Scalable distributed multi-db applicationsmulti-db applications

TransactionTransaction

Tran

saction

Tran

saction

Stored proc runs onStored proc runs on Different transactionDifferent transaction Different threadDifferent thread Different securityDifferent security Different timeDifferent time

ClientClientAppApp

Persisted Dialog

LaunchedLaunchedStoredStoredProcProc

Services Live In The Databases

Services Live In The Databases

Ongoing work in the databaseOngoing work in the databaseEach Service “instance” is kept in a databaseEach Service “instance” is kept in a databaseState is kept in the databaseState is kept in the databaseMessages are stored in the databaseMessages are stored in the database

Routing to a databaseRouting to a databaseIncoming messages are kept in the databaseIncoming messages are kept in the databaseThe message is matched to the state and the The message is matched to the state and the computation is performedcomputation is performedRouting incoming web service messages means delivering to the Routing incoming web service messages means delivering to the correct databasecorrect database

TransactionTransaction

ServiceService

SharedSharedStuffStuff

PrivatePrivateStateState

TransactionTransaction TransactionTransaction

SQLCLR IntegrationSQLCLR IntegrationCLR Deeply IntegratedCLR Deeply IntegratedHosting layer provides Hosting layer provides coordination of:coordination of:

ThreadsThreadsMemoryMemoryGarbage CollectionGarbage Collection

DDL, Metadata, DDL, Metadata, SecuritySecurityAllows System Allows System ExtensionExtension

Scalar FunctionsScalar FunctionsTable Value FunctionsTable Value FunctionsProceduresProceduresTriggersTriggersData AccessData Access

WindowsWindows

SQL-OSSQL-OS

Host LayerHost Layer

SQL CLRSQL CLR

SQL EngineSQL Engine

ConclusionConclusion

Must consider classes of data in SOAMust consider classes of data in SOA

Databases will evolveDatabases will evolveFrom:From: Connected & session oriented Connected & session oriented

To:To: Event driven & service oriented Event driven & service oriented

TP-Heavy TP-Heavy → TP-Lite evolution will → TP-Lite evolution will repeat itself from App Server → SODArepeat itself from App Server → SODA

SQL 2005 is a SODA platformSQL 2005 is a SODA platform