exchange server 2013 database and store changes

Exchange Server 2013Database and Store ChangesScott SchnollPrincipal Technical WriterMicrosoft Corporation

Agenda

Mailbox Server Platform EvolutionExchange 2013 GoalsDatabase and Store Changes

Mailbox Server Platform Evolution

Exchange 2010 Exchange 2013 Challenges Solutions

Processor

8 core(2 socket)

20+ cores(2 socket)

• Store scales poorly past 12 cores

• Eggs/basket increase

• Multi-role servers• Multi-process Store

Memory32GB(8x4GB DIMM)

96GB(12x8GB DIMM)

• n/a • n/a

Disk

3.5” 7.2K 2TB (35/server)

3.5” 7.2K 6TB(12/server)

• Disks getting larger but not faster

• DB Maintenance times• Seed reliability/duration• Eggs/basket increase

• Reduce IOPS by +50%• Multiple Databases/disk• Tune DB maintenance

Network

4 x 1GBit Nics 2 x 10Gbit Nics • How do you leverage the bandwidth for seeding

• Faster seeding with multiple databases/disk

Exchange Server 2013 Goals

Decrease hardware costsIncrease reliability and availabilityProvide data protection enhancementsEnable faster root cause analysis through better diagnosticsDeliver core platform investments for future innovations

Decrease Hardware Costs

Decrease Hardware Costs

Reduce IOPS by 50% compared with Exchange 2010Disk sizes increasing (8TB) with no corresponding increase in IOPSLarger mailboxes (100GB)Mailbox schema and ESE pre-read optimizations

Support multiple databases per volumeMaximize disk space utilization without increased reseed timesDistribute active users across available database volumes

Drive higher adoption of JBOD deploymentTake advantage of low-cost locally attached storage

IOPS Reductions

Improvements to logical contiguity of store schemaProperty blobs are used to store actual message propertiesSeveral messages per page means fewer large IOs to retrieve message propertiesUse of long-value storage is reduced, though when accessed, large sequential IOs are used

Reduction in passive copy IO100MB checkpoint depth reduces write IOTransaction log code refactored for faster failover

Tables Optimized for Sequential IOGlobal Tables Catalog – registry of tables existing in database Globals – database version, etc Mailbox – MailboxNumber, Owner Info, Locale, LastLogonTime, etc DeliveredTo – duplicate delivery information Events – reliable events for assistants

Tables partitioned by MailboxNumber Folder - FolderId, Item Count, Size, PropertyBlob Message – DocumentId, MessageId, FolderId, PropertyBlob,

OffPagePropertyBlob, MessageClass ordered by DateReceived Attachment – AttachmentId, Name, Size, CreationTime, etc PhysicalIndexes (partitioned by LogicalIndex)

Message Table Property Storage

Blobs used to store collection of MAPI properties Referred to as On-page and Off-page property blobs ESE compression optimizes physical storage of blob data Compression more efficient when input contains more properties

PropertyBlob Contains properties previously stored in Header table in message table

column Property promotion OffPagePropertyBlob PropertyBlob possible Blob size limited to eliminate LV tree access for core message properties

OffPagePropertyBlob ESE LV Hints push storage of this blob into separate LV tree Reading LV tree involves large sequential I/O

Higher Reliability and Availability

Higher Reliability and Availability

Improved isolation from hardware/software failuresStore process per database, faster failover and disk failure handlingBuilt-in monitoring and availability managementBest copy and server selection includes health of entire protocol stackService recovery through failover and/or restartNon-stop operationsNo scheduled mailbox database maintenance Autoreseed automatically restores redundancy on disk failureMaintain data protection without manual interventionDynamically uses spare disks to restore database copy health

Exchange Information Store

Previously a single monolithic process

Unmanaged code

Very nested code that made it difficult to debug

Managed Store

Store service/process (Microsoft.Exchange.Store.Service.exe)Microsoft Information Store serviceManages worker process lifetime based on mount/dismountLogs failure item when store worker process problems detectedTerminates store worker process in response to “dirty” dismount during failover

Store worker process (Microsoft.Exchange.Store.Worker.exe)One process per database, RPC endpoint instance is database GUIDResponsible for block-mode replication for passive databasesFast transition to active when mountedTransition from passive to active increases ESE cache size 5X

Microsoft Exchange Replication serviceReplication service process (MSExchangeRepl.exe)Detecting unexpected database failuresIssues mount/dismount operations to StoreProvides administrative interface for management tasksInitiates failovers on failures reported by ESE, Store and Responders

ESE Cache Management

Allocates 25% of memory for store worker process ESE cacheThis is referred to as the max cache targetAmount allocated to each store worker process based on number of hosted database copies and value of MaximumActiveDatabasesStatic amount of cache allocated to passive and active copies

Store worker process will only use max cache target when copy is activePassive database copies allocate 20% of max cache target

Max cache target computed at service process startupRestart Store service process when adding/removing copies or changing value of MaximumActiveDatabases

Store Worker Process Example

MountedOnServer: EXHV-1125.EXHV-44159dom.extest.microsoft.com

Name WorkerProcessId Mounted ActivationPreference EdbFilePath

---- --------------- ------- -------------------- -----------

MDB01C 11328 True {[EXHV-1125, 1], [EXHV-94234, 2], [EXHV-96496, 3]} C:\Databases\MDB01\MDB01C\MDB01C.edb





---- --------------- ------- -------------------- -----------

MDB01A 2920 True {[EXHV-94234, 1], [EXHV-96496, 2], [EXHV-1125, 3]} C:\Databases\MDB01\MDB01A\MDB01A.edb





---- --------------- ------- -------------------- -----------

MDB01B 16692 True {[EXHV-96496, 1], [EXHV-1125, 2], [EXHV-94234, 3]} C:\Databases\MDB01\MDB01B\MDB01B.edb



Recurring Maintenance

Scheduled maintenance is eliminated in Exchange 2013Recurring maintenance now part of time-based assistant (TBA) infrastructureStoreMaintenance: lazy index maintenance, isintegStoreDirectoryServiceMaintenance: disconnected mailbox expiration

Workload Management monitors CPU, RPC latency, and replication health Task execution throttled/deferred when resource pressure exists

Background ESE database scanning further throttledBased on datacenter disk failure analysis, target to complete background database scan within 4 weeks (using multiple databases on 8 TB disks)

Managed Availability

Tests determine viability of various components on Mailbox serverDatabase connectivity and replicationProtocol services (Outlook, OWA, EAS, IMAP, POP)

Recommend HA actions when service-impacting condition foundDatabase failoverRestart serviceRestart computer

Escalate when auto recovery unsuccessful and service not restoredIntegration with System Center to raise awareness of service-impacting conditions that cannot be automatically resolved

Managed Availability

Name Trigger/Recovery sequence

Database Availability 12 logon failures in 16 minutes Escalate

Store service not running Restart service Bugcheck Escalate

Database Free space Free disk space drops below 10% Escalate

Store service process repeatedly crashing

3 crashes for store service in 1 hour Escalate

Store worker process repeatedly crashing

3 crashes for store work (across all workers) in 1 hour Escalate

Percent RPC requests 90% of available threads per database Database Failover Escalate

70ms RPC latency 70ms RPC Avg latency Determine impact scope Id/quarantine mailbox Escalate

150ms RPC latency 150ms RPC Avg latency Determine impact scope Id/quarantine mailbox Escalate

Mailbox quarantined More than 1 mailbox quarantined on database Escalate

Assistants service not running Restart service Escalate

Event assistants behind watermarks* Assistant watermark age exceeds threshold Escalate

Number of search tasks* Count of search tasks exceeds threshold Escalate

Mailbox Quota Management

Reduction in overhead to generate over-quota notificationAt logon time, system evaluates mailbox quota against policySends over-quota notification message once per notification interval, notifications are NOT sent to inactive mailboxes

Mailbox size calculation is more accurate measurement of mailbox database storage usedIncludes both internal and end-user items/properties

Mailbox size will likely increase when moved to Exchange 2013Search metadata stored on items increases overall mailbox sizeNo increase in database footprintShould plan to increase quota per mailbox

Data Protection Improvements


Autoreseed automatically restores redundancy on disk failure

Lag copies can “care for themselves”Play down when low on space, during page patching, and when required for availability (no other copies available)Lagged copy activation can be simplified with transport enhancements

Lagged Copy Example

MailboxServer: EXHV-1125

Name Status ReplayLagStatus---- ------ ---------------MDB01A\EXHV-1125 Healthy Enabled:True; PlayDownReason:None; Percentage:47; Configured:1.00:00:00; Actual:11:20:49MDB02A\EXHV-1125 Healthy Enabled:True; PlayDownReason:None; Percentage:47; Configured:1.00:00:00; Actual:11:20:14MDB03A\EXHV-1125 Healthy Enabled:True; PlayDownReason:None; Percentage:47; Configured:1.00:00:00; Actual:11:19:53

MailboxServer: EXHV-94234

Name Status ReplayLagStatus---- ------ ---------------MDB01B\EXHV-94234 Healthy Enabled:True; PlayDownReason:None; Percentage:48; Configured:1.00:00:00; Actual:11:36:23MDB02B\EXHV-94234 Healthy Enabled:True; PlayDownReason:None; Percentage:48; Configured:1.00:00:00; Actual:11:36:24MDB03B\EXHV-94234 Healthy Enabled:True; PlayDownReason:None; Percentage:48; Configured:1.00:00:00; Actual:11:36:18


Continued support of VSS backup APIWindows Server Backup support, supports both active and passive database copies3rd party VSS applications should be compatible with Exchange 2013 without major changes1

VSS full, copy, incremental, and differential backup/restore supported

Windows Server Backup supports backup of both active and passive databasesScheduled backup succeeds regardless of mount state

1 Backup vendors responsible for integration and supportability statements

Diagnostic Improvements

Diagnostic Improvements

Insight into runtime without dumps or external clientsPowerShell access to mailbox database internals and in-memory state

Eliminate need for end-user reproAlways on tracing to capture “outlier” operational behavior

Get-MailboxStatistics Example

Get-MailboxStatistics –Database DB1 sort ItemCount -Descending | select -first 10 | FT -a MailboxGuid,DisplayName,IsArchiveMailbox,TotalItemSize,ItemCount

MailboxGuid DisplayName IsArchiveMailbox TotalItemSize ItemCount----------- ----------- ---------------- ------------- ---------ac2dcfd6-555b-460e-85bf-1c656367dc2c Todd Luttinen False 13.24 GB (14,219,331,170 bytes) 616162d6928c49-19d8-481d-acbf-139e86cbbfa7 Ananth Ramanathan False 12.59 GB (13,514,775,720 bytes) 337668c672c655-cbb2-4684-a366-35dc9c2e95e1 Online Archive - Ananth Ramanathan True 7.717 GB (8,285,648,981 bytes) 27588112c16313-df0f-43e5-abb7-06d420f922f5 Archive - Danilo Benzatti True 7.788 GB (8,362,091,456 bytes) 244379b037b150-9abe-46ba-83f8-c65cc4973f0d Paul Bowden False 7.474 GB (8,025,245,880 bytes) 22672167894514-a60a-4050-94f7-566c196154fa Danilo Benzatti False 5.565 GB (5,975,363,784 bytes) 206090a32d480c-6aac-41f5-9915-df1cdd70c045 William Rall False 9.239 GB (9,920,247,961 bytes) 2032793b85bb49-936e-4f9c-83d1-7c3ee4e4b1ee Ashish Mishra False 4.495 GB (4,826,655,382 bytes) 1990345796fb94-482f-4f54-ba15-c516c718afcf Liang Mou False 8.151 GB (8,751,575,868 bytes) 1977678b493423-a362-4a0e-b97c-0ec6bfc483cf Mihai Jalobeanu False 8.989 GB (9,652,362,586 bytes) 180590

Get-StoreUsageStatistics Example[PS] C:\>$mailbox = get-mailbox toddlutt[PS] C:\>$statistics = Get-StoreUsageStatistics -Database $mailbox.database[PS] C:\>$statistics | ? {$_.DigestCategory -eq 'timeInServer'} | sort TimeInServer -descending | select -first 10 | FT -a DisplayName,SampleTime,TimeIn*,LogRecord*,RopCount,LDAP*

DisplayName SampleTime TimeInServer TimeInCPU LogRecordCount LogRecordBytes ROPCount LdapReads LdapSearches----------- ---------- ------------ --------- -------------- -------------- -------- --------- ------------Jack Xiao 9/4/2012 3:11:53 PM 13905 0 0 0 1514 0 0Ananth Ramanathan 9/4/2012 3:16:53 PM 7074 0 290 38893 1620 0 0Liang Mou 9/4/2012 3:16:53 PM 5884 0 89 4623 983 0 0Ananth Ramanathan 9/4/2012 3:10:53 PM 5651 0 142 209590 1520 0 0Ananth Ramanathan 9/4/2012 3:13:53 PM 5058 0 155 15729 1580 0 0Ananth Ramanathan 9/4/2012 3:12:53 PM 4643 0 0 0 1131 0 0Ananth Ramanathan 9/4/2012 3:15:53 PM 4575 0 146 12036 1306 0 0Ananth Ramanathan 9/4/2012 3:19:53 PM 4528 0 9 299 1163 0 0Ananth Ramanathan 9/4/2012 3:18:53 PM 4306 0 216 163766 1380 0 0Ananth Ramanathan 9/4/2012 3:11:53 PM 3999 0 0 0 1092 0 0

[PS] C:\>$statistics | ? {$_.DigestCategory -eq 'LogBytes'} | sort LogRecordBytes -descending | select -first 10 | FT -a DisplayName,SampleTime,TimeIn*,LogRecord*,RopCount,LDAP*

DisplayName SampleTime TimeInServer TimeInCPU LogRecordCount LogRecordBytes ROPCount LdapReads LdapSearches----------- ---------- ------------ --------- -------------- -------------- -------- --------- ------------Ben Rinaca 9/4/2012 1:43:51 PM 205649 0 1150007 413624055 68559 0 9Neil Joyer 9/4/2012 2:33:52 PM 67961 0 379201 121880991 27142 0 0Neil Joyer 9/4/2012 2:43:52 PM 90086 0 248803 83060705 18797 0 6William Rall 9/4/2012 1:53:51 PM 15971 0 14232 35585703 4301 0 0Syed Ali (Sapphire Technologies Canada) 9/4/2012 3:02:53 PM 16822 0 340227 10458217 5224 0 0Paul Bowden 9/4/2012 2:12:52 PM 24531 0 8885 7390547 21883 0 4Todd Luttinen 9/4/2012 2:03:51 PM 133244 0 28227 6846890 23852 0 2Paul Bowden 9/4/2012 2:53:53 PM 12208 0 5539 5198231 15663 0 4Daniel Popper 9/4/2012 2:43:52 PM 9472 0 5621 4396546 9600 0 5Paul Bowden 9/4/2012 2:33:52 PM 11543 0 4504 3754540 15831 0 0

Core Investments for Future Innovations

Core Investments

Integration of new search engine – Search FoundationSame search engine used by SharePoint 2013

C# Development PlatformImproved developer productivity through better toolsLarger community of developers within team available to contribute

Better layering of implementationImplementation of physical layer isolates underlying database engine from upper (logical) layers and MAPI implementation

Virus Scanning API (VSAPI)

Exchange 2013 does not support VSAPIIt does include transport extensibility to scan messages in-flight3rd party A/V extensions no longer run in Store process

On-demand scanning not considered an effective solution with clients that cache data (Outlook, OWA, EAS, POP, IMAP)EWS is available for scheduled and on-demand scan scenarios

Results

E14 vs. E15: DITL Performance Comparison

• 48 | 76% reduction in disk IOPS

• 18 | 41% reduction in Average RPC Latency

• 17 | 34% increase in CPU per RPC processed

• ~4x increase in Store memory overhead

DB IOPS/Mailbox0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.65

0.16

RPC Average La-tency

Mcycles per RPC packet

Store Memory per Mailbox (MB)

0

0.5

1

1.5

2

2.5

3

3.5

4

Online Mode | Cached Mode

E14 vs. E15: DITL Performance Comparison

• 48 | 76% reduction in disk IOPS• 18 | 41% reduction in Average RPC Latency• 17 | 34% increase in CPU per RPC

processed• ~4x increase in Store memory overhead

DB IOPS/Mailbox0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.65

0.16

RPC A

vera

ge Lat

ency

Mcy

cles

per

RPC

pac

ket

Stor

e Mem

ory

per M

ailb

ox (M

B)0

0.51

1.52

2.53

3.54

Online Mode | Cached Mode

LoadGen Simulation – 10 DBs/1000 usersTwo profiles: Online and Cached (Default/Optimized)

Perf gains are not free – increase in CPU and memory

CPU increase is factor of optimizing for two-socket servers and moving to multi-process architecture

Enables us to scale out using multi-core processors without having to cross processor bridge to access shared L2 cacheSome CPU overhead comes from using managed code

Memory increase is also factor of multi-processor architecture

Most of the memory is in small and large object heaps in .NET primarily used for object allocation and cleanup

IOPS Reductions

Exchange 2003 Exchange 2007 Exchange 2010 Exchange 20130

0.2

0.4

0.6

0.8

1

DB IOPS/Mailbox

IOPS/Mailbox

~95.5% Reduction!

Summary

Mailbox storage has…Reduced IOPS by 50-70%...again!Optimized for large disks (8TB) and larger mailboxes (+100GB) Better isolation leading to higher reliabilityBuilt-in monitoring and recovery to drive higher availabilityImproved data protection to reduce risk of data loss

Questions?

Scott SchnollPrincipal Technical [email protected]://aka.ms/schnoll

schnoll

exchange server 2013 database and store changes

Documents