abolade gbadegesin principal software architect microsoft corporation bb06

Mesh Services Architecture And Concepts

Abolade GbadegesinPrincipal Software ArchitectMicrosoft Corporation

BB06

Live ServicesUser-centric data and services with more than 460M Users

Mesh

S

erv

ices

Identity Directory StorageCommunicatio

ns and Presence

Search & Geospatial

Users DevicesSynchronizatio

nApplications

Agenda

User Experiences Concepts Architecture Planning and Deployment Implementation Operations, Experiences, Lessons Learned Future Work

Using the Live Mesh software

demo

Key Capabilities

Provisioned state User, devices, applications, relationships

User-data storage Feeds and content Feed synchronization and content replication

Communications and presence Device presence and remote UI access Peer connectivity

Notifications Single channel for asynchronous push Signaling channel for session establishment

User-data storage

Accounts

Notifications

Communications& Presence

deviceclaim

device

identity

Provisioning In The MeshConceptual elements

,

provisioned during software install

provisioned during web sign-up

1

2

3

builds on Live ID authentication tocapture user-device relationships

Understanding REST In The MeshResources, collections and links

resource

collection

resource

UriEntityTagSubscriptionUri…

UriLastUpdateTime…

resource

linkUriLabel…

scope for grouping resourcesand tracking changes

metadata common to all resources

primitive for representing &navigating relationships

represented as feeds and items

1

2

3

4

Provisioned State ServicesConceptual view and building blocks

Mesh Service

Accounts Live Mesh Client

Accounts

Live ID Service Live IDClient Runtime Library

appapp

users, devices, apps (local cache)

HTTP

HTTP

mesh object

Folder Synchronization In The MeshConceptual elements

localfilesystem

data feed mapping

,

meshdatafeed

‘maps’ data feed to a file system location

1

2

items in data feedrepresent files

mesh object defines permission scope for sharing feeds

3

Understanding FeedSync In The MeshEnabling bidirectional feed synchronization

<rss version="2.0"> <channel> <title>To Do List</title> <description>A list of items to do</description> <link> http://example.com/partial.xml </link> <item>

<title>Buy groceries</title> <description>Get milk, ... </description> <sx:sync id="item_1_myapp_2005-05-21T11:43:33Z" updates="3">

<sx:history sequence="1" when="2005-05-21T09:43:33Z" by="a9"/> <sx:conflicts ...>

<item>...</item>

</sx:conflicts></sx:sync>

</item> </channel>

</rss>

version vector and conflicts for each feed item

localfile

system

Synchronized User-Data Storage ServicesConceptual view and building blocks

Mesh Service

User-data storage Live fx Client

User-data storage

Mesh folders app

,

(local replica)

010010010100111100

Scalable storage

feeds, content,permissions

HTTP, FeedSync

handles mesh folders

Mesh Services Architecture

Mesh Service

User-data storage

Live fx Client

User-data storage

010101

Scalable storage

Services fabric

Local file system

Accountsappapp

Microsoft Update

Live ID Service

Accounts Live IDClient

Runtime Library

MessengerNAT

Traversal

MessengerRelay

Service

Mesh UX


Notifications

Notifications


Mesh Folders

MeshWeb Desktop

MeshRemote Desktop

Derive workloads from dominant application flows

Identify key scalability factors Decide on the partitioning of state Assign machine roles across tiers

Mesh Services ArchitectureGetting from concepts to implementation

Mesh AccountsLive fx Client

Mesh Services PlanningWorkload analysis: Startup and file updates

update local cachesubscribe for change notifications

Startup

File change

Mesh User-Data Storage

synchronize feedupload updated files

synchronize local replicassubscribe for change notifications

Mesh NotificationsLive fx Client Mesh User-Data Storage

Mesh Services PlanningWorkload analysis: Content replication

fire change notifications

Contentreplication synchronize local replica

query availability of updated files

fetch updated files from peers

File change

continued

Mesh Communications and Presence

deliver change notifications

query availability of peers

relay or direct connection

Partition by user, device, and mesh object Use soft state to minimize I/O load Leverage HTTP 1.1 semantics for caching, change

notification, and incremental state transfer Leverage client-side resources for holding state Leverage peer connectivity for content replication

Mesh Services PlanningScale-out strategy

Mesh Services PlanningPartitioning resources

https://accounts.mesh.com/Identities/FKEXBSLJR3VUXEXMTPSSOVO6GM/DeviceClaims/CSOW2TEBYUSE5BD3I5QOS724UU

each resource lives in a ‘home’ datacenter

each resource has a stable canonical URI based on its resource identifier

each resource has a location URI that routes requests directly to its ‘home’ datacenter

https://accounts.clus9.bay.mesh.com/Identities/FKEXBSLJR3VUXEXMTPSSOVO6GM/DeviceClaims/CSOW2TEBYUSE5BD3I5QOS724UU

requests to the ‘wrong’ datacenter get redirected, and clients cache the redirect URI for future use

mapping a resource identifier to its ‘home’ datacenter requires a lookup in a replicated database

Mesh Services PlanningGeneral service design principles

Commodity hardware Partitioning for scaling out,

redundancy for availability Loose coupling across roles Xcopy deployment and configuration Fail-fast, recovery-oriented error handling Self-monitoring and self-healing

Mesh Services DeploymentInfrastructure and roles

Account APIs

Services fabric

User-Data Storage

Subscription store

Notificationqueue store

Device presence store

Dictionary state store

Partitioning & Recovery Manager

Watchdogs

front endroles

back endroles

infrastructureroles

commodity hardwarexcopy deployment

automated rollbackself-monitoring

fail-fast recoveryself-healing

scaleunit

Scalable storageServices fabric SKU:2P Dual Core8GB RAM4x 500GB SATA II HDDWindows Server

parti

tioni

ng u

nits

Web Desktop

Content upload & download

Front end roleBack end roleBuilding block

Services fabric

Mesh Services DeploymentScale unit layout

Subscription store



Dictionary state store


Wat

chdo

gs

Web Desktop

Content upload & download

Notifications API

Accounts Scalable Storage

User-Data Structured

Storage

Live ID Messenger Relay Hotmail

User-Data Blob

Storage

reliable state

stores

soft state stores

building block services

Accounts API

User-Data Storage API

Communications & Presence

soft state stores

Front end roleBack end roleBuilding block

Communications and Presence

Mesh Services ImplementationSoft state partitioning and recovery

Node 1 Node 2

{ECCA8793-3208-48A9-A1A1-038D0AD704C5}

/MeshObjects/FJ3GZXUNDC4EPPPPUMQ2QNCG6I/Subscriptions/GLW6BC6HRWFU5LEW7HTX5FPC3Y

hash nodes & resourcesto the same output space

node ‘nearest’ a resourceis responsible for it




Owner library

Lookup library

Owner library

Notifications API

Lookup library

consistent hashingnode loss tracking

resource owner nodes

resource client nodes

owner library obtains leaseslookup library queries assignments

Mesh Services ImplementationCommon software stack

Services fabric

controller

Mesh services host process

Windows Server

Mesh service applicationMesh services runtime

AP configuration

NETFX 3.5 SP1

Tracing & log

collection

multiple processes mesh service process

Mesh service codeBuilding block code

singleton process

SMB.sys

Mesh Services ImplementationFront end software stack

Services fabric

controller


Windows Server

HTTP.sys sockets

HTTP request dispatcher

TLS/TCP stream

dispatcher

Mesh resource handlers

Mesh content handlers

TLS/TCPhandlers

Mesh services runtime

AP configuration

Subscriptions

Soft state lookup

NETFX 3.5 SP1

Tracing & log

collection

SMB.sys

WCF syndication

Accounts

User-data storage

Communications & Presence

Notifications

Scalable storage client

User-data content

upload & download

TCP notification streaming

multiple processes mesh service process singleton process


Mesh Services ImplementationBack end software stack

Services fabric

controller


Windows Server

sockets

Sockets dispatcher

Mesh resource managersMesh services runtime

AP configuration

Subscriptions

Soft state owner/lookup

NETFX 3.5 SP1

Tracing & log

collection

SMB.sys

Scalable storage client

multiple process mesh service process singleton process

Work queues

Device presence manager

Notification queue manager

Subscription manager

Dictionary store manager

Partitioning and recovery manager


Monitoring The Mesh Services

demo

Mesh Services OperationsExperiences and lessons learned

On availability:Design for loosely coupled dependence on building blocksDiligently validate client/cloud upgrade scenariosInvest in pre-production stress and functional coverage in environments that look like productionDesign for throttling based on both dynamic thresholds and static bounds


On monitoring:Continuously refine performance counters, logs, and log processing toolsMonitor end-user-visible operations (Keynote)Build end-to-end tracing across tiersSelf-healing is hard: Invest in tuning watchdogs and thresholds


On deployment:Deployments every other week, client upgrades every monthMajor functionality roughly each quarterTook advantage of gradual ramp to learn lessons early

Mesh Services FuturesExtending the mesh cloud

synchronize user-data storage with non-Microsoft services

(e.g. Smugmug)

federate user authentication with other identity providers

(e.g. Active Directory)

personal and enterprise user-data storage

offerings:on-premises software

and online service

You saw this working today:

Microsoft Services

Connector

You saw this in the Sync Framework session

Stay tuned!

Microsoft Services ConnectorFederating identity

Desktop

Windows Azure

ISV Apps

Live Mesh

EnterpriseApps

Azure Platform Services

Microsoft Federation Gateway

Live Identity Provider

Exchange Your AppsSharepoint

ActiveDirectory

Enterprise On-PremisesMicrosoft Services

Connector

Online Services

CRM Online

Live Services

LiveMesh

Soon: enterprise credentials in Live Mesh

Future: enterprise policy on Mesh Services

and policy

Mesh Services

Sync Framework and Live Framework Synchronizing clouds

Mesh Services

User-data storage010101

Accountsappapp


Notifications

CloudSync

Live Framework

SyncFX store providers

Sync apps

Future: Mesh Services as a sync hub for your apps

The Mesh Services deliver a comprehensive and compelling suite of key capabilities

You can build on the Mesh Services today using the Live Framework and Sync Framework

Your feedback will help define the roadmap for delivering personal and enterprise offerings of the Mesh Services

Summary

Live Mesh http://www.mesh.com http://blogs.msdn.com/livemesh

Live Framework and Mesh Services BB04: Live Services: A Lap Around the Live

Framework and Mesh Services BB34: Live Services: Notifications, Awareness, and

Communications Sync Framework

http://msdn.microsoft.com/sync TL30: Microsoft Sync Framework Advances

Related Sessions and Resources

http://www.mesh.com/

http://blogs.msdn.com/livemesh

http://msdn.microsoft.com/sync

Evals & Recordings

Please fill

out your

evaluation for

this session at:

This session will be available as a recording at:

www.microsoftpdc.com

Please use the microphones provided

Q&A

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market

conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

abolade gbadegesin principal software architect microsoft corporation bb06

Documents

mesh accounts live fx

feed item slide

files mesh object

presence node

direct connection slide

replicated database

apps local cache http

end role building block