Advanced SQL Azure

PresenterTitleCompany

Session Objectives and Takeaways

Session Objectives:Understand the challenges and opportunities that SQL Azure presents from a performance and scalability perspectiveUnderstand the capabilities, services and tools available in the Windows Azure Platform to address them

Get up to speed on roadmap for scaling database applicationsGet up to speed on SQL Azure Data SyncGet up to speed on Microsoft Sync Framework (Sync Fx)

Key Takeaways:Engineer for elastic scale with SQL Azure todayEngineer for low latency and concurrency with SQL Azure Data Sync and Sync Fx

SQL Azure: Performance and Scalability

Database Design• Table design• Index design• Query design• Transaction design

Capacity• Size limits• Partitioning (shard)

Latency• Redundancy• Replica overhead

• Distance• Affinity Groups• SQL Azure Data

Sync• Sync Fx

Concurrency• Replication• SQL Azure Data

Sync• Partitioning (shard)

Scalability• Scale-up• Not an option

• Scale-out• Partitioning (shard)• SQL Azure Data

Sync

Scaling Database Applications

Scaling database applicationsScale up

Buy large-enough server for the jobBut big servers are expensive!

Try to load it as much as you canBut what if the load changes? Provisioning for peaks is expensive!

Scale-outPartition data and load across many servers

Small servers are cheap! Scale linearly

Bring computational resources of many to bear800 little servers is very fast

Load spikes don’t upset usLoad balancing across the entire data center

Scale-out with SQL Azure TodayElastic Provisioning of Databases

CREATE DATABASE and goNo VMs, no servers

Pay-as-you-go business modelDon’t need it --- DROP it

Zero Physical AdministrationBuilt-in High Availability, patching, maintenance

Database Copy, SQL Azure Data Sync

Ticket Direct Solution

Elastic scale – database as a servicePay as you grow and shrinkEasy to provision and manage databaseNo hardware, no manual database administration required

Promotions, events, ticket selling businesses are “bursts – bound” by natureCapacity constraints limit business agilityHigh costs of entry into new businessDifficult to roll out extra capacity quicklyIdle capacity “off-bursts” is cost prohibitive

Cap

acit

y

Time

“Capacity Bursting“

Average Usage

Average Usage Com

pu

te

# o

f Hr’

sChallenges Today

Scale-out for Multi-tenant applicationsPut everything into one DB? Too big…Create a database per tenant? Not bad…Sharding Pattern: better

Application is already prepared for it!

T1 T2 T3 T4 T5

T6 T7 T8 T9T10

T11

T12

T13

T14

T15

T16

T17

T18

T19

T20

T1 T2 T3 T4 T5

T6 T7 T8 T9 T10

T11 T12 T13 T14 T15

T16 T17 T18 T19 T20

All my data is handled by

one DB on one server

Sharding PatternLinear scaling through database independence

No need for distributed transactions in common cases

Engineered partitioningRather than complete transparency

Local access for mostConnection routingQuery, transaction scoping

Distributed access for someFan-out expensive computation

App

Engineered Partitioning: Platform CapabilitiesProvisioning

Growing and shrinking capacity

ManagingUpgrading, patching, HA for lots of databases

RoutingWhere is the directory?How to scale it and use it?

Partition ManagementSplitting and Merging, without loss of availabilityFan-out

Covered by

SQL Azure today

Coming up in SQL

Azure:Federatio

ns

2011

Distribution of DataPartitioned

Spread across member machinesEach piece is on one machine (+HA)Most of the data!

CentralizedOnly available in one placeRead and write, but not too much

ReplicatedCopied to all member machinesCan be read anywhere (reference)Should not be written too much

Data1ref

Data2ref

Data3ref

Data4ref

Data5ref

Con-fig

SQL Azure Federations: ConceptsFederation

Represents the data being sharded

Federation KeyThe value that determines the routing of a piece of data

Atomic UnitAll rows with the same federation key value: always together!

Federation Member (aka Shard)A physical container for a range of atomic units

Federation RootThe database that houses federation directory

Root

Federation “CustData”

Member: [min, 100)

AUPK=5

AUPK=25

AUPK=35

Member: [100, 488)

AUPK=10

5

AUPK=23

5

AUPK=36

5Member: [488, max)

AUPK=55

5

AUPK=25

45

AUPK=35

65

(Federation Key: CustID)

Creating a FederationCreate a root database

CREATE DATABASE SalesDBLocation of partition mapHouses centralized data

Create the federation in rootCREATE FEDERATION Orders_Fed (RANGE BIGINT)Specify name, federation key type

Start with integral, guid types

Creates the first member, covering the entire range

SalesDB

Federation “Orders_Fed”

(Federation Key: CustID)Member: [min,

max)

Creating the schemaFederated tables

CREATE TABLE orders (…) FEDERATE ON (customerId)Federation key must be in all unique indices

Part of the primary key

Value of federation key will determine the member

Reference tablesCREATE TABLE zipcodes (…)Absence of FEDERATE ON indicates reference

Centralized tablesCreate in root database

Federation “Orders_Fed”(Federation Key: CustID)

Member: [min, max)

SalesDB

orders

Products

zipcode

Splitting and MergingSplitting a member

When too big or too hotALTER FEDERATION Orders_Fed SPLIT (100)Creates two new members

Splits (filtered copy) federated dataCopies reference data to both

Online!

Merging membersWhen too smallALTER FEDERATION Orders_Fed MERGE (200)Creates new member, drops old ones

Federation “Orders_Fed”(Federation Key: CustID)

Member: [min, max)

SalesDB

orders

Products

zipcode

Member: [min, 100)

orders zipcode

Member: [100, max)

zipcodeorders

Connecting and OperatingConnect to atomic unit

USE FEDERATION Orders_Fed (56) WITH FILTERING=ONConnection routed to member containing 56Only data with federation key value 56 is visible

Plus reference data

Safe: atomic unit can never be split

Connect to entire federation memberUSE FEDERATION Orders_Fed (56) WITH FILTERING=OFFConnection routed to member containing 56All data within the member database is visibleDangerous: federation member can be split

Member: [min, 100)

AUPK=

5

AUPK=25

AUPK=56

App

zipcode

Schema DistributionFederation members can have different schemas at a point in time:

Temporary, while schemas are being upgradedTemporary, while customer is testing new schema on some shardsPermanently, because shards are different

To alter schema:Manually

Connect to each federation member (USE FEDERATION Orders(56) WITH FILTER=OFF)Alter it (ALTER TABLE Customers …)

Future: schema-distribution serviceConnect to rootManage and apply schemas asynchronously

Sharding in SQL Azure: Beyond v1Schema Management

Allow multi version schema deployment and management across federation members.

Fan-out QueriesAllow single query that can process results across large number of federation members.

Auto RepartitioningSQL Azure manages the federated databases for you through splits/merges based on some policy (query response time, db size etc)

Multi Column Federation KeysFederate on enterprise_customer_id+account_id

Walk-through: setup

CREATE FEDERATION Orders_Fed (RANGE BIGINT)

USE FEDERATION Orders_Fed(0) WITH FILTERING=OFFCREATE TABLE orders(orderidbigint, odatedatetime, customeridbigint, primary key (orderid, customerid)) FEDERATE ON (customerid)

CREATE UNIQUE INDEX o_idx1 on orders(customerid, odate)CREATE INDEX o_idx2 on orders(odate)

CREATE TABLE orderdetails(orderdetailidbigint, orderidbigint, partidbigint, customeridbigint,

primary key (orderdetailid, customerid)) FEDERATE ON (customerid)

ALTER TABLE orderdetails add constraint orderdetails_fk1 foreign key(orderid,customerid) references orders(orderid,customerid)

CREATE TABLE uszipcodes(zipcodenvarchar(128) primary key, state nvarchar(128))

Walk-through: real workConnect to: InitialCatalog=‘SalesDB’

– get some regular work done (within customer 239)USE FEDERATION Orders_fed(239) WITH FILTERING=ON

SELECT * FROM Orders JOIN OrderDetailsON …INSERT INTO Orders (customerid, orderid, odate) VALUES (239, 2, ‘5/7/2010’)

– get some cross-customer work doneUSE FEDERATION Orders_fed(0) WITH FILTERING=OFF

DELETE from Orders WHERE odate < ‘1/1/2000’- Repeat for other members…

-- go back to rootUSE FEDERATION ROOT

UPDATE CleanupSchedule set LastCleanupDate = GETSYSTIME()

Walk-through: ups and downs!--Day#2 business grows!

ALTER FEDERATION Orders_FedSPLIT AT(1000)

--Day#3 black friday!ALTER FEDERATION Orders_FedSPLIT AT(100)

ALTER FEDERATION Orders_FedSPLIT AT(200,300,400…)

--Day#4 recession hits!ALTER FEDERATION Orders_FedMERGE AT(100)

--Day#5 oh boy… double dip.ALTER FEDERATION Orders_FedMERGE AT(200,300,400…)

SQL Azure Data Sync

SQL Azure Data Sync – Key FeaturesElastic Scale

Service scales as resources requirements growNo-Code Sync Configuration

Easily define data to be synchronizedSchedule Sync

Choose how often data is synchronizedConflict Handling

Handle issues where same data is changed in multiple locations

Logging and MonitoringAdministration capabilities for tracking data and monitoring potential issues

On-Premises to Cloud Symmetry

SQL Server to SQL Azure Sync

Sync

Benefits• Makes cloud extension

rather than replacement

• Enables moving workload to cloud in stages preserving investment in existing infrastructure

• New scenarios spanning enterprise, cloud

SQL Azure

SQL Azure to SQL Azure Synchronization

Benefits• Geo-

synchronization of data across data centers

• Scale-out read or read/write

Extending Data to Retail and Remote Offices

SQL Azure to SQL Server Sync

Benefits• Share data with remote and

retail offices, bringing data closer to the end-users

• Lower latency – data access doesn’t require round trips

• Higher availability – app still runs if server is unreachable

• Reduced network utilization – most data access is local

SQL Azure

Sync End to End Scenarios

On-Premises Applications

Sync

SyncSync

Offline Applications

Sync

SyncSQL Azure Database

Sync Sync

Sync

SQL Azure Data Sync

http://azure.com

SQL Azure Data Sync CTP2

(Coming in near future)

Microsoft Sync Framework 4.0 CTP available now!

Sync Sync

Retail & Remote Offices

SQL Azure Data Sync CTP1(Available now from http://azure.com)

Microsoft Sync Framework

Why Build Offline Capable AppsOffline/Cached mode apps enjoy lots of value

Client apps get better UX through Lower latency – data access doesn’t require round tripsHigher availability – app still runs if server is unreachable

Reduced network utilization – most data access is localServers gain better ability to schedule work asynchronously

Benefits of cached mode are magnified on the Internet

Server/service is further awayNetwork is less tuned/reliable

Offline Client

Server / Service

Remote Store

Local Data

Cache

Client App

Sync

Query / Update

Sync

WINDOWS PHONE 7- ISOLATED STORAGE- OTHER STORES

WINDOWS- SQL SERVER EXPRESS- SQL COMPACT

WINDOWS MOBILESQL COMPACT -

IPHONE / ANY CLIENTSQLITE / ANY STORE -

BROWSER / HTML5- HTML5 STORES

Simple protocol(OData& Sync)

Minimal client & store requirements

Client APIsupport

Windows Server / IIS

SyncEndpoints

Auth / Mgmt / Bus Logic

SyncFx

SQL Server

WindowsAzure

SyncEndpoints

Auth / Mgmt / Bus Logic

SyncFx

SQL Azure

Sync smarts on server, not on client

Easy to develop the sync endpoints

SILVERLIGHTISOLATED STORAGE -

OTHER STORES - Client APIsupport

Sync Framework v4 OverviewC

LIE

NT O

FFLIN

E A

PP

SS

YN

C S

ER

VIC

E

Cloud

On-Premises

Offline Applications Architecture - Server

Isolated Storage Provider

OData Sync Proxy

Cache Controller

Windows Azure Application

SQL Azure

SQL AzureProvider

Sync Logic

Silverlight Offline Application

Isolated Storage

Collections

Silverlight Offline Application

Sync

OData Sync Endpoint

Business Logic

Sync Service OverviewData Store

Expose data from SQL Azure or SQL Server

HostHosted on Windows Azure or IIS/Windows ServerExposed using a WCF sync endpoint

ProtocolExpose data for synchronization via a protocol and allow 3rd parties to build offline clients

FeaturesSupport business logic extensity Custom authentication / authorizationFiltering

ToolingProvide a Tooling Wizard experience to configuring server and client

Offline Applications in Silverlight

Isolated Storage Provider

Silverlight Offline Application

OData Sync Proxy

Cache Controller

• Sync Logic Moves to Server/Service• Enables Offline In Isolated Storage with

Silverlight 3 & 4• Extensibility to allow 3rd party storage to be hooked in

Isolated Storage Sync Logic

Collections

Silverlight Offline Client

Windows Azure Application

SQL Azure

SQL AzureProvider

Sync Logic

OData Sync Endpoint

Business Logic

Sync

Offline Applications for all platforms

Client Needs To:• Track changes on local store (samples for common

stores).• Implement client side sync proxy (samples for

common platforms)

Sync Application

OData Sync Proxy

Store

Collections

Offline application on any platform

Windows Azure Application

SQL Azure

SQL AzureProvider

Sync Logic

OData Sync Service

Business Logic

Sync

Offline Applications Architecture - Protocol

Sync Application

OData Sync Proxy

Store

Collections

Offline application on any platform

Windows Azure Application

SQL Azure

SQL AzureProvider

Sync Logic

OData Sync Service

Business Logic

Sync

Simple protocol (OData & Sync)

OData Protocol & SyncApply the principals of OData to the problem of data-syncStandardize on protocol not on componentsMinimal client sync logic or algorithmsService manages sync keeping client simpleProvide samples to show how to consume protocol for any platformProvide components for richer experience in Silverlight& WP7Full interop details defined in SDK

Sync Framework – Offline SupportReleases and Roadmap

Release Availability

Overview

SyncFx v4.0 October CTP

October, 2010

• Public Preview release with sole focus being the new support for offline scenarios and extended client reach Components for sync service Components for Silverlight and WP7• Samples for WM6.5, HTML5

• Wizard Tooling and iPhone sample (in few weeks)

SyncFx v4.0 RTW

2011 • New support for offline scenarios and extended client reach

• Targeted database provider improvements

Summary

SQL Azure presents unique challenges and opportunities from a performance and scalability

perspective.

Engineer for elastic scale with SQL Azure todayEngineer for low latency and concurrency with SQL Azure Data Sync and Sync Fx

© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the

date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.