UNC301 - (Microsoft IT) Microsoft Exchange 2007 Architecture and Design at MicrosoftMartin CoetzerTechnical ConsultantMicrosoft

Agenda

• Microsoft IT Exchange environment overview• Transport and topology• Server roles and hardware planning• Mobile messaging scenarios – Client Access

Server• Internet mail connectivity – Edge Transport

Server• Mailbox server and storage designs• Backup approach (current and future)

Microsoft IT Exchange Environment Overview

• Multiple Exchange organizations (Corp, Dogfood, Windeploy, WinSE, Extranet, Managed Solutions)

• Prior to transition to Exchange 2007– ~100 legacy Exchange 2003 servers / 30 mailbox servers

• After transition to Exchange 2007– ~64 Exchange 2007 mailbox clusters running in the Corp forest at

RTM– Goal to reduce it to ~35 in the nearest future

• Mailboxes – 130,000+ • Public Folders – 240,000 folders• Distribution Groups – 170,000+• Maximum message size = 10 MB• Availability Goal – 99.99%• Mailbox size on Exchange 2003: 200 MB • Mailbox size on Exchange 2007: 500 MB and 2 GB

Post-Consolidation EnvironmentPost-Consolidation Environment

4 Sites with Exchangemailbox servers

100 Exchange servers - 30 mailbox server (clustered) - 4000x200MB per cluster node

99.99% availability goal

Microsoft IT Messaging Trends and Needs

Microsoft IT Exchange Server 2003 Deployment Limitations of 32-bit platform• Scalability limitations

– Virtual memory is limited to 4GB• Exchange Server 2003 can use up to 3GB max

– Kernel Memory limitations• Paged Pool 256MB max with /3GB enabled• Client connections, Access tokens• Non Paged Pool (256MB max without /3GB and 128MB

max with /3GB)• HTTPs and TCP connections to the Exchange Front

End or Gateway

• Storage Performance demands– Small Database cache size = higher disk I/O and

storage demands

Goals of Exchange Server 2007 Deployment at Microsoft

• Overcome scalability limitations of 32-bit platform

• Maintain 99.99% availability goal• Increase the user mailbox size from 200MB to

500MB and 2GB• Reduce hardware, storage, and backup costs• Establish mailbox server resilience from

storage level failures• Establish flexible and scalable middle-tier

services (e.g. Client Access, Transport)

Routing Routing Group Group

ConnectorsConnectors

RGCRGC

RGCRGC

RGCRGC

Legacy Exchange Server 2003 EnvironmentLegacy Exchange Server 2003 Environment

DublinDublin

SingaporeSingapore

Sao PauloSao Paulo

RedmondRedmond

Redmond-Redmond-ExchangeExchange

AD Site LinkAD Site Link

Exchange Server 2007 EnvironmentExchange Server 2007 EnvironmentAD Site withAD Site with

Exchange Servers Exchange Servers

AD Site withoutAD Site without Exchange Servers Exchange Servers

DublinDublin

SingaporeSingapore

Sao PauloSao Paulo

RedmondRedmond

Redmond-Redmond-ExchangeExchange

AD Site LinkAD Site Link

Exchange Server 2007 EnvironmentExchange Server 2007 Environment

Exchange Exchange RoutingRouting

AD Site withAD Site with Exchange Servers Exchange Servers

AD Site withoutAD Site without Exchange Servers Exchange Servers

DublinDublin

SingaporeSingapore

Sao PauloSao Paulo

Redmond-Redmond-ExchangeExchange

AD Site LinkAD Site Link

Custom Site LinkCustom Site LinkExchangeCost=10ExchangeCost=10ADCost=999ADCost=999

RedmondRedmond

Exchange Server 2007 EnvironmentExchange Server 2007 Environment

set-adsitelink Dublin-to-RedmondExchange -ExchangeCost 10set-adsitelink SaoPaulo-to-RedmondExchange -ExchangeCost 10set-adsitelink Singapore-to-RedmondExchange -ExchangeCost 10

Microsoft IT Exchange Server 2007 Server Roles• In Exchange Server 2003 timeframe Microsoft IT deployed servers roles dedicated for specific functions

• Exchange Server 2007 deployment – similar approach– Structured system

administration andmaintenance

– Optimized hardware andsoftware configuration

– Targeted load balancing and fault tolerance

– Flexible scaling approach– Independent and flexible – server provisioning– Higher Availability

Hub Transport ServersEdge Transport Servers

Client Access Servers

Clustered Mailbox Servers

Unified Messaging Servers

Inte

rnet

Exchange 2007 Load Balancing and Fault Tolerance at Microsoft IT

• Enterprise scenario – must build load balancing and fault tolerance provisions– Resilience to server level failures– Tolerance to increased load due to planned server downtime and

environmental conditions (spam attacks, “snow day”)

Component Load Balancing Fault Tolerance

Mailbox Server N/A Clustering (CCR)NIC Teaming

Hub Transport Server MBX HUB: Built inEdge HUB: Send Connector – multiple bridgeheads

Edge Transport Server Internet Edge: MX equal cost & DNS Round RobinHUB Edge: Edge Subscriptions

Client Access Server External: Network Load Balancing + ISA 2006 WPLBInternal: Network Load Balancing

Unified Messaging UM IP Gateway: Multiple IP Gateways per dial planIP Gateway UM: Built in (Round Robin between UM)

Hardware ConfigurationsHardware Configurations

RoleRole ConfigurationConfiguration

BHD/Edge BHD/Edge 11 4x2.2 GHz CPU*, 8GB RAM 4x2.2 GHz CPU*, 8GB RAM

CAS CAS 22 4x2.2 GHz CPU*, 4GB RAM 4x2.2 GHz CPU*, 4GB RAM

UMUM 2x2.2 GHz CPU*, 4GB RAM 2x2.2 GHz CPU*, 4GB RAM

MBX 2000 @ 500MB (CCR)MBX 2000 @ 500MB (CCR) 4x2.6 GHz CPU*, 12GB RAM4x2.6 GHz CPU*, 12GB RAM

MBX 2400 @ 2GB (CCR)MBX 2400 @ 2GB (CCR) 4x3.0 GHz CPU*, 16GB RAM4x3.0 GHz CPU*, 16GB RAM

MBX 3600 @ 2GB (CCR)MBX 3600 @ 2GB (CCR) 8x2.6 GHz CPU*, 24GB RAM8x2.6 GHz CPU*, 24GB RAM

MBX 6000 @ 500MB (CCR)MBX 6000 @ 500MB (CCR) 8x2.66 GHz CPU8x2.66 GHz CPU**, 16GB RAM, 16GB RAM

** 2 way = One Dual Core CPUs; 4 way = Two Dual Core CPUs ; 8 way = Four Dual Core CPUs

** 8 way = Two Quad Core CPUs

11 7:1 Mailbox-to-HUB server performance ratio

22 4:1 Mailbox-to-CAS server performance ratio

SPECint ® 2000 rates were used to compare different processors (www.spec.org)

• Common URL namespace for mobile messaging clients– OWA, Exchange Active Sync (EAS), Outlook Anywhere (RPC/HTTP) (e.g.

https://mail.company.com)

• Integration – all mobile messaging services on a common Exchange Server 2007 CAS platform

• Mobile Messaging Service Usage– OWA: ~60,000 unique users/month– RPC/HTTP: ~60,000 unique users/month– EAS: ~30,000 unique users/month

• Highly scalable Exchange Server 2007 CAS infrastructure – Load balancing and fault tolerance

OverviewOverview

Messaging TopologyMessaging Topology

Exchange 2007CAS Servers

ISA Server2006

Exchange 2007 Mailbox Servers

Exchange 2007CAS Servers

ISA Server2006

Exchange 2007 Mailbox Servers

Exchange 2007CAS Servers

ISA Server2006

Exchange 2007 Mailbox Servers

Dublin:-ExternalURL “https://emeamsg.microsoft.com/...”

Singapore:-ExternalURL “https://apsmsg.microsoft.com/...”

Sao Paulo:-ExternalURL “https://spamsg.microsoft.com/...”

Exchange 2007CAS Servers

ISA Server2006

Exchange 2007 Mailbox Servers

Internet

Redmond:-ExternalURL “https://msg.microsoft.com/...”-Autodiscover Service URL “https://autodiscover.microsoft.com/autodiscover”

DesignDesign

ISA02

ISA03

CAS-02

CAS-03

Externally trustedSSL Certificate

msg.microsoft.com

CAS-01ISA01

Exchange RPC

Exchange 2007 CAS Servers

RPC Proxy

RPC Proxy

RPC Proxy

ISA 2006Array

Internally trustedSSL Certificate

msg.microsoft.com

LO

AD

BA

LA

NC

ING

(single

affi

nity)

SSL

External users Internal users

SSL

LO

AD

BA

LA

NC

ING

(single

affi

nity)

Exchange 2007Mailbox Servers

Published Paths

/exchange/*

/owa/*

/public/*

/Microsoft-Server-ActiveSync/*

/RPC/*

/Autodiscover/*

/EWS/*

/UnifiedMessaging/*

Cookie-based load

balancing

IP-based load

balancing

CO

NN

EC

TIV

ITY V

ER

IFIE

RS

Connection/Sender/Recipient FilteringRecipient LookupAntispamAttachment managementAntivirus

Perimeter Forest

Microsoft IT Exchange 2007 Edge Server

Exchange 2007 Mailbox servers

MOMExchange 2007 HUB servers

Corp Forest

DC / GC

Internet

DC / GC

SMS MOM

Port 25 (SMTP)

Port 25 (SMTP)Port 53 (DNS)Port 80 (AV Signature Updates)

Port 25 (SMTP)Port 3389 (Terminal Services)Port 50389/50636 (EdgeSync)

Active Directory Forest in Perimeter Network - Non corporate AD joined OR trusted

Port 25 (SMTP)

Exchange 2007 Edge Servers

Region 1

Region 2

• Deployment Goals: Mailbox size: 2GB; Deleted items retention time: 14 days• Daily Transaction Log generation rate per mailbox per day: 25MB (average)• Database overhead (fluff) factor: 20%• Desired max DB size - 200GB:

– 200 GB / (2GB + 20%) = 85 mailboxes/DB • One Database per Storage Group• Databases per LUN: 7 (to facilitate weekly backup schedule)

– One Backup Stream Per source LUN• DB LUN Capacity requirements for 85mbx * 7DBs= 600 mailboxes/LUN

– 600 mailboxes * 2GB + 20%[DB overhead] + 5%[Ci] + 10%[Free space] = 1,656GB

– 1,656GB / 275GB * 2 [RAID1+0] = 12 disks (@300GB)• DB LUN Performance requirements

– Assuming 1.0 IOPS/mbx at 2GB for very heavy user profile with 5MB of RAM/user– 600 mailboxes * 1 IOPS/mbx = 600 IOPS / LUN– 1:1 Read:Write mix (300 Reads + 300 Writes = 600)

• 300[R] + 2x300[W] = 900 IOPS• 900 / 130 IOPS = 7 disks (10K RPM)

• Conclusion: Design is capacity bound - 12 disks required for capacity [RAID1+0]

Hardware Design: Mailbox Storage ProvisioningHardware Design: Mailbox Storage Provisioning

Hardware Design: Universal Storage Building Block – 2GBHardware Design: Universal Storage Building Block – 2GB• Goal: Design simple and scalable storage building blockGoal: Design simple and scalable storage building block

Approach:Two disk enclosures

Cross-shelf mirroring [RAID 1+0]

LUN Capacity:600 mailboxes = 12 disks [RAID 1+0]

7 Storage Groups/Databases

Single Logical Drive

Two Database LUNs total1,200 x 2GB mailboxes

One Log LUN2 logical volumes (L:\ + M:\)

825GB of log space

30 x 300GB drives24 for databases

6 for transaction logs

Data:Log – 4:1

1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

En

clo

su

re 1

2 3 4 5 6 7 8 9 10 11 12 13 14 15

En

clo

su

re 2

• Goal: Design simple and scalable storage building blockGoal: Design simple and scalable storage building block

DATA1RAID 1+01650GB

DATA2RAID 1+01650GB

LOG RAID 1+0

800GB

Hardware Design: Universal Storage Building Hardware Design: Universal Storage Building Block – 500MBBlock – 500MBLUN Capacity:LUN Capacity:

830 mailboxes = 14 disks [RAID 830 mailboxes = 14 disks [RAID 1+0]1+0]

7 Storage Groups/Databases7 Storage Groups/Databases

Single Logical DriveSingle Logical Drive

Three Database LUNs totalThree Database LUNs total2,500 x 500MB mailboxes2,500 x 500MB mailboxes

One Log LUNOne Log LUN550GB of log space550GB of log space

50 x 146GB SFF SAS drives50 x 146GB SFF SAS drives42 for databases42 for databases

8 for transaction logs8 for transaction logs

6

En

clo

sure

1E

ncl

osu

re 2

7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 251 2 3 4 56 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 251 2 3 4 5

Hardware Design: 2,400 x 2GB mailboxes (SAS DAS)Hardware Design: 2,400 x 2GB mailboxes (SAS DAS)

Passive NodeActive Node

SAS

Log Shipping

Database StorageDB: 48x300GB 10K RPM SAS (6.6TB R10)

LOG: 12x300GB 10K RPM SAS (1.6TB R10)

Database StorageDB: 48x300GB 10K RPM SAS (6.6TB R10)

LOG: 12x300GB 10K RPM SAS (1.6TB R10)

SAS SAS SAS

Hardware Design: 3,600 x 2GB mailboxes (FC SAN)Hardware Design: 3,600 x 2GB mailboxes (FC SAN)

Passive NodeActive Node

FC4GB

Log Shipping

FC4GB

Database StorageDB: 72x300GB FC (10TB R10)

LOG: 18x300GB FC (2.5TB R10)

FC4GB

FC4GB

Database StorageDB: 72x300GB FC (10TB R10)

LOG: 18x300GB FC (2.5TB R10)

Passive NodeActive Node

SAS

Log Shipping

SAS SAS SASSAS

Backup StorageSATA RAID-5

BACKUP

Database Storage Replica Storage

Streaming BackupStreaming Backup

• Backup Window: 4 hours• Schedule: Full – weekly, Incremental - daily

System Center Data Protection Manager 2007System Center Data Protection Manager 2007

Passive NodeActive Node

SAS

Passive NodeActive Node

Passive NodeActive Node

DPM SRVER1GigE

1GigE

1GigE

DPM Agent

DPM Agent

DPM AgentDPM Agent

DPM Agent

DPM Agent

SAS

• VSS based backup from “Active” or “Passive”• CCR aware requestor• Express Full backup technology (deltas)• Incremental backup - every 15 minutes

www.microsoft.com/dpm

Q&A

Thank you

http://www.microsoft.com/southafrica/ucs/2007