SPS – Scale to 15k RPS
Patrice PellandMicrosoft
Overview and Goals of SPS• SPS (Shared Personalization Service)• It is a backend storage and service• Enables following scenarios:
• Explicit personalization• Implicit content optimization• Geo based customization
Scenario #1Scenario#1 – WL Anonymous ID and Machine Anonymous ID - based Explicit Personalization Examples: Locations for weather, news, events, favorite sports team, personal shopping list, customized page settings, etc.
Scenario #2Scenario#2 – WL Anonymous ID and Machine Anonymous ID - based Implicit Content Optimization Examples: User demographic & behavior based content optimizations and/or personalization (e.g. personal recommendation)
Scenario #3Scenario#3 – GEO based customization SPS provides a Geolookup service that allows partner to enable IP based customizations (e.g. default location, Location based contents, GEO fencing, etc.)
Scaling? Availability? Perf?• Why? 150 Million users visit US Home Page /
month and with peeks of 15,000 RPS and up to 75 million users on other HP.
• Latency goals: Read < 25 ms – update < 50 ms• Pages have to be up - $$$ loss if not• Need to be stateless
Overall ArchitectureSPS
Webstore Config Server
SPS Configuration
SPS Deployment DataLookup
Deployment Data
SPS FE Cluster
CMS Rendering System
Partner web serverSPS Logic
Cache Access
Webstore DB
Access
WCFService
AppFabric Cache Cluster
Cache BoxCache
BoxCache BoxCache
BoxCached Data
Database Access
Lookup System
Database Partitions
Load Balancer
SPSAdapter (SPS MSN
CMS service wrapper)
GeoService
How?• Everything is Stateless• Windows AppFabric Caching service with
many nodes – reliable and redundant– Similar to memcache– 240 GB of memory cache in the US
• SQL Server DB Partitioning with lookup system master/backup at each level
12
Facts• Availability
– Designed with no single point of failure– Web - multiple web servers behind a LB.– DB
• Each DB partition has a primary & secondary DB setup with multi master topology. • Transactional replication is used by SQL to sync the primary & secondary. If a primary DB server goes down, requests are handled by secondary DB server.
– File share: WAN Sync is used to replicate critical files across primary & secondary file server. VIP ensures automatic read availability for SPS Service when primary goes down. Write availability for backend services is ensured by manual fail over.
– Throttling to prevent outage from abnormal traffic – throttling is configurable both at server level and at partner level. Partner level throttling is based on around 200% of normal peak traffic
– Load balancer also has a secondary backup
• Scalability– Web & AppFabric cache: Scalability is achieved by adding new nodes. Everything is stateless…– DB: Databases are hosted as webstore application. Scalability is achieved by partitioning. Adding additional data partition is very easy.
• Live site metrics– Latency: 10 ms read, 30 ms update, 12ms (async update)– US: 39 web servers, 15 AppFabric caching server, 10 SQL lookup server and 12 SQL backend (data) servers – Asia: 17 web servers, 8 AppFabric caching server, 8 SQL lookup server and 10 SQL backend (data) servers – Europe: 16 web servers, 8 AppFabric caching server, 8 SQL lookup server and 10 SQL backend (data) servers – Current Peak RPS per web box in US is 375 (14.7K RPS US), Peak CPU 40%. Server capacity is around 600RPS with 70% CPU
13
High-level Features• Support shared namespace definition – reduce # of calls• Support multiple levels of access control of shared namespace
– Behind corp firewall
• Plug-in smart defaults for namespace– Smart Defaults return faster for cases where the user doesn’t have
customizations yet.
14
High-level Features• Plug-in smart data validation for namespace
– Small DLLs validate pre-compiled on the server
• Bulk upload of implicit user preference or clustering info• Geolookup service – One stop shop – reduce calls• Support both netTCP calls and WCF calls – if in the same DC
then netTCP 35% faster than normal TCP• Service is available globally: US, Europe and Asia – Closer to the
user.
15
High-level Features• Introduction of an API for Async update
– Designed to support implicit updates or storing session data. In this case, user does not explicitly make an effort to update his/her setting. Instead, by just browsing a page, or click a link, corresponding settings are stored on SPS.
• Examples: Recent stock list from doing stock quotes on MSN Money site, Search History, Article List where user clicked thumb up/down, etc.
– Two stage updates: 1) data from client request is first saved in cache; then 2) batch updates to DB, thus allowing faster response time to client. Optimized for writes
– Data is in memory for a short period of time before being written to DB. We are using AppFabric high availability mode (i.e. dual cache copy) to minimize potential data loss. Data loss may occur only if both cache servers are down at the same time.
– Async update can be turned on/off at attribute level via admin UI. E.g. User’s preferred locations are not using Async update, but Money Recent Quotes may be.
16
Anatomy of a Get API Call
SPSEndpoint(WCF, CF)
AppFabric Cache
PartitionLookup
CoreCoreCoreCore
(1)Lookup Data in Cache
(2)Return Data Found in
Cache
(3)UserId for lookup
(Cache miss) (4)
Core Partition Information for User Record
(5)Query for records
(6)User Records
MSN Geo Lookup Service
PartitionLookup
(7)User IP
(8)User
Location/Connection Info
Smart Defaults ProviderSmart Defaults
ProviderSmart Defaults
Provider
(9)User
RevIPInfo and Data missing
from DB
(10)Defaults for
Missing Data
(0)Partner Request
(12)Response
(11)Write to Cache
17
Anatomy of an Update API Call
SPSEndpoint(WCF, CF)
AppFabric Cache
PartitionLookup
CoreCoreCoreCore
(3)UserId for lookup
(6)Core Partition Information
for User Record (8)Write records
(9)Success/Fail
PartitionLookup
Smart Defaults Provider
Smart Defaults Provider
Smart Validator Provider
(1)Validate Request
(2)Success/Fail
(0)Partner Request
(10)Response
(7)Invalidate Cache
(5)Create lookup record
(4)User Not Found
18
Anatomy of an Async Write Call
CacheSweeper
SPSEndpoint(WCF, CF)
Main Cache
CoreCoreCore
Async Cache
1. Async Write Request
2. Invalidate Main cache
3. Write to Async cache
a. Batch Read for DB Loading
b. DB Load
5. Response
4. Return (success)
c. Invalidate Async cache
19
Anatomy of an Async Read Call
SPSEndpoint(WCF, CF)
Main Cache
CoreCoreCoreCore
Async Cache
1.Read Request
4. Read from Main Cache
2. Read from Async cachePartitionLookupPartition
Lookup3. Cache miss from Async cache
5. Cache miss from Main cache
8. Query for records
9. User Records
10. Write to Main Cache
11. Response
6. UserId for lookup(Cache miss)
7. Core Partition Information for User Record
Q&AThank you!