Building a Dynamic and Flexible Exchange Architecture

Scott Drummonds

Group Manager, Performance Marketing

16 October, 2008

Performance

Cost

Exchange Performs Well on VMwareIO is not an issueScales out better on shared hardware

Reduce infrastructure costsConsolidate Exchange server rolesEliminate need for dedicated standby servers

ManagementDynamic Management

Streamline testing & troubleshooting with snapshots and clonesProvision in minutesScale dynamically and right-size infrastructureAchieve high availability without complexity of clustering

Why Run Exchange on VMware

Microsoft SVVP Support and LicensingSupport

8K mailboxes

Native VMware Infrastructure

8 VM

16K mailboxes

16 core

128 GB 16 core

128 GB

Double Capacity of Exchange 2007 Infrastructure

Microsoft Support Making Significant Progress

Licensing Allows VMotion

> Reassign licenses between servers as frequently as needed

> Covers 41 server apps including

> Exchange 2007> SQL Server 2008> SharePoint 2007> Dynamics CRM 4.0

Support for Applications

> Microsoft SVVP Program to validate virtualization products

> Technical support for Windows and 31 apps on SVVP validated software

> Exchange 2007> SQL Server 2008> SharePoint 2007> Dynamics CRM 4.0

VMware Validated in SVVP

> 9/3 announcement

Native

Mailbox Mailbox

MSCCR

VMware Infrastructure

Network Load Balancing

Edge Hub Client Mailbox

Simpler, Cost-Efficient Availability for Exchange 2007

Edge Hub Client

MSLCR

VMware HA & VMotion

SimpleEliminate complexity of MS clusteringOne HA solution for all Exchange roles

Cost-effectiveWindows and Exchange Standard EditionNo dedicated standby servers

ReliableVMware HA for unplanned downtimeVMotion for planned downtimeLCR for data replication

Improve Testing Efficiency with Snapshots and Clones

Exchange ExchangeBlackBerry

ServerBlackBerry

DB

ProductionTest

Archive for Fast Roll-back

Exact copy of production

12

Run more tests faster

3

Move changes into production

4

> Faster testing> More accurate testing on exact production copy> Lower cost testing infrastructure

Scale Exchange Dynamically

Scale upRe-size Virtual MachinesVMotion to more powerful host

Scale outProvision new instances

VMware Exchange Deployment

Business and User Characteristics� Business Type: Software Company

� Typically 1 million internal / 100k external / 40k spam messages per day� Primary datacenter in Palo Alto

� No Business Continuity or Disaster Recovery capabilities

� Single Active Directory site

� Production Mailboxes: ~9000

� Mailbox profiles� Unlimited mailbox sizes for users (largest ~54GB)

� Total mailbox data: 10TB

� Average mailbox size: ~1GB

� Average IOP rating per user: 2.25

Exchange 2003 ArchitectureSystem-Wide

� Active Directory Windows Server 2003 Enterprise deployed in a single site

� Exchange Server 2003 Enterprise Edition

� No site resiliency or disaster recovery in place

� 1 GigE network connection

Palo Alto Datacenter

� 2 Microsoft Single Copy Clusters (SCC)

� 14 physical hosts

� 9 Exchange Virtual Servers for 9000 mailboxes

� 2 load balanced Front-End multiple protocol servers (IMAP, OWA, POP3, etc.)

� 2 Public Folder Server replicas

� Fibre channel SAN (EMC CLARiiON storage)

� Streaming backups performed on active mailbox server stores

Exchange 2003 Architecture

Exchange 2007 Architecture� 3 physical locations (Multi-Site configuration)

� Primary datacenters in Palo Alto and Santa Clara

� Fail-over between primary datacenters

� Second Palo Alto site for file share witness

� 1 Gig connection between facilities

� Mailboxes (9,000 and rapidly growing)� Extra-large or non-quota’d mailboxes (>2 GB): ~2000

� Standard mailboxes (1-2 GB): ~7000

� Mailbox profiles� 1 and 2 GB mailbox quotas for regular mailboxes

� 3, 4 and 5 GB mailbox quotas for extra-large mailboxes

� Average 1,200 mailboxes per VM

� No mailbox sizes greater than 5GB

Exchange 2007 ArchitectureSystem-Wide� Active Directory Windows Server 2003 R2 x64 Enterprise Edition (Multi-Site configuration)

� Exchange 2007 Enterprise Edition

� Clustered Continuous Replication (CCR) over stretched VLAN used for database replication

� 1 GB connection between sites (Bandwidth is tunable depending on requirements)

� 11 application-aware (2 –node) High Availability VMs using CCR

� 9000 mailboxes transitioned to the Exchange Server 2007 environment

� Project completion date: 30-June-2008

� Entire Exchange infrastructure is virtualized using VMware Infrastructure 3

Palo Alto Datacenter• 17 Exchange Server VMs

• 6 Hub/CAS

• 11 Mailbox VMs

• Fiber Channel SAN

• 2 EMC Networker storage node VMs

Santa Clara Datacenter• 14 Exchange Server VMs

• 3 Hub/CAS (proxy and transport only)

• 11 Mailbox VMs

• Fiber Channel SAN

• 2 EMC Networker storage node VMs

• 2 Windows 2003 Std 32 bit DC/GC VMs

Exchange 2007 Architecture

Addressing General Concerns

� Performance� Exchange 2007 x64 bit architecture allows you to trade I/O for memory.

� 8K block size and write coalescing allows larger but fewer writes to storage.

� Enhancements to VMware Infrastructure 3 support 32/64 bit clustering and improved storage layer

� Availability� VMware VMotion, HA and DRS allow dynamic balancing of the Exchange configuration

across multiple ESX hosts and also facilitates planned maintenance.

� Flexibility in use of Clustered Continuous Replication (CCR) to allow immediate failover and reverse replication without the complexities of scripting. Two copies of the Exchange data are available for immediate recovery

� Management and Maintainability � VirtualCenter provides a single pane of glass for managing and maintaining ESX as well

as the Exchange VMs. Templates allow for quick provisioning

Addressing General Concerns

� Backups� Flexible backup options using array-based, in-guest, streaming, or VMware Consolidated

Backups. VCB now supports the VSS framework.

� Scaling� VMware Infrastructure 3 allows us to run multiple Exchange VMs on the same ESX hosts

� More efficiently utilize the physical resources of your “Big Iron” servers

� Overcome the 8 core/32GB Microsoft maximum recommended Exchange configuration

� Resize VMs with just a reboot (add more memory, cpu, network, storage)

Recoverability� VMware HA restarts VMs on a different ESX host in a VMware Cluster when a failure is

detected

� No single point of failure

Business Requirements

� Multi-Site Resiliency

� Leverage VMware Infrastructure 3 features to create a robust, reliable and mobile virtualized messaging infrastructure capable of withstanding datacenter/site failure

� High Availability

� Leverage application-aware high availability/replication technology for immediate failover of messaging systems with no single point of failure and minimal impact to connected clients

� Improved performance � Take advantage of Exchange 2007 architecture enhancements to provide better I/O and

application performance for an improved end-user experience

� Scalability� Utilize VMware Infrastructure 3 to provide more efficient hardware utilization as well as

provide the ability to overcome Microsoft’s maximum recommended configuration of 8 cores/32GB of memory for the Exchange application

Business Requirements� Immediate Disaster Recovery

� Leverage the VMware feature set ensure fast recovery from ESX host outages

� Infrastructure Upgrades� Leverage VMware to free up critical space in the datacenter, cut power and cooling

requirements. � Provide redundant power, network capabilities

� Quota Implementation� Leverage quotas to reduce backup window schedules, offline defragmentation due to

constant mailbox movement� Maintain Exchange performance though controlled mailbox growth

� Flexible Backup Options � Leverage VMware feature set to give the IT department flexible backup options

� VMware Consolidated Backups (Now supports VSS framework U2)� In Guest backups (streaming) performed using backup application of your choice� Array based snapshots� Passive node backups allow you to be less concerned with backup windows

How We Did It

� Built a proof-of-concept environment� Deployed on a test hardware platform� Identified POC participants (Core Team, IT, Friends of IT and Other Business users)

� Followed PSO Virtual Infrastructure Methodology (VIM) � Developed a PSO SOW - VIM Assessment (Assess, Plan, Build, Manage)

� Created a pilot user base � Create a mixed community of users (POC consisted of Approximately 100 users)

� Conducted a rigorous UAT� Accepted or rejected based on the testing� Design a scalable infrastructure (Based on business/technical requirements and

information collected from existing environment)

� Pilot program� IT Training� Performance Measurement, # of VM per Core, per ESX server

VM Sizing: Mailbox Server����������� ���������������������������������

���� ����!������� �����������

"�������� ������ ����

�# ��$ ����

��� ��%&��'��� ����������� ����

���� �!����������� � "!���#$���%�$ #

���� �!����������� ��������#$���%�$ #

���� �!����������� ���& #�#$���%�$ #

����������(���������

�'��$���$!!���( )�*&(�+�,-,�. ��(��

)�*&��+/,-,��$ ����$����0���$��1�2�� #�)�����$���*�+3)����4*���!��$��-

)�*&��+5,-,�2���*����$��1�2���*#$��6��$�$�$!�5$%%��%�+3)����4*���!��$��-

�'��$���$!!���� )�*&��+/,78��9�� �:�)�(�;-,�)���"�*��+3)����4*���!��$��-

)�*&��+/,78��9�� �:�)�(�;-,�)���"�*��+3)����4*���!��$��-

���< +*�����)���$! � �*��$��!-

VM Sizing: Peripheral Roles

)�*&��+5,-,�2���*����$��1�2���*#$��7��$�$�$!�5$%%��%�+3)����4*���!��$��-

�����������������&�����)���*����'���������

���� ����!������� �����������

"�������� ������ ����

�# ��$ ���

��� ��%&��'��� ����������� ����

���� �!����������� � "!���#$���%�$ #

����������(���������

�'��$���$!!���( )�*&(�+�,-,�. �� ��

�'��$���$!!���� )�*&��+/,-,�)���"�*��1�2�� #�)�����$���*�+3)����4*���!��$��-

&�����+ �������$+�# �������������

���� ����!������� �����������

"�������� ������

�# ��$ ����

��� ��%&��'��� �����������9��

���� �!����������� � "!���#$���%�$ #

����������(���������

�'��$���$!!���( )�*&��+�,-,�. �� ��

Exchange 2003 Storage Profile

� Configured for very heavy profiles

� All Outlook users configured for online mailbox access

� Large IMAP population

� Mailbox ranging from 1GB - > 10GB in size

� 2500 Online Mobile users (RIM, Good and ActiveSync) Technologies

� Read intensive configuration (70/30)

� Average user profile for Exchange Server 2003 was ~2.5 - 3.0 IOPS per

mailbox

Exchange 2007 Storage Profile

� Configured for extremely heavy profiles

� Used Exchange calculator to calculate required storage IOPS/mailbox

� Exchange 2007 reduces IO by ~50 – 75%

� Exchange 2007 calculator yielded 1.15 IOPS/mailbox

� Added additional .6 IOPS to 1.15 IOP number for a total 1.75 IOPS

� Profile used messages sent/received (50/300) This is unheard of !!

� 5MB cache per user mailbox

Storage Design

Availability and Recovery� Exchange 2003 environment utilized Single Copy Clustering

� Single point of failure was storage

� Only allowed streaming backups on the Active mailbox stores

� Difficulty in meeting backup windows

� Difficult recovery processes and procedures for clusters and data

� Memory fragmentation and performance problems as we scaled Required frequent reboots of hardware.

New Availability and Recovery Strategy

VI3 Design Considerations

� VMware VMotion

� Allows Exchange VMs to be migrated to other ESX Server hosts during host patching and maintenance

� Uninterrupted GuestOS and Host patching of Exchange infrastructure

� VMware DRS

� Allows even distribution of the various Exchange workloads across the VMware Infrastructure

� VMware HA

� Allows automatic restart of Exchange VMs on surviving HA cluster nodes in the event of a host outage.

New Availability and Recovery Strategy

MS Exchange Design Considerations

� Application-Aware High Availability (CCR)

� Maintain duplicate copies of the data

� Flexible backup options (backup from passive node of cluster) in order to meet tight production schedule windows.

� New Backup strategy uses Virtual Tape libraries (VTL) to perform staggered (weekly full, daily differential) backups

� De-Duplication performed at the VTL enables 30 day onsite retention

� Multi-Site Business Continuity and Disaster Recoverability

� Uninterrupted GuestOS and Host patching of Exchange infrastructure

Lessons Learned

� Know your existing environment

� Characterize your workloads (normal and especially peak)

� Define a Proof Of Concept

� Proof of concept is a chance to make sure the application meets or exceeds your Business Requirements (include users from all business units in POC)

� Test Proof of Concept using appropriate test toolset

� Exchange toolset consists of (Jetstress, LoadGen and ESP)

� Define attainable objectives when designing your new environment

� Don’t bite off more than you can chew

Lessons Learned

� Testing is critical (Measure Twice Cut Once)

� Get close to your application vendors

� Continuously monitor performance in your virtualized environment

� Don’t skimp on hardware !! Virtualized environments are highly dependent on the “Core Four” (CPU, RAM, Storage and Network)

� Keep growth and capacity requirements in mind when designing your environment

� Know and characterize your workloads before tackling any virtualization project

Summary

� We’re a satisfied user of our own products and professional services

� We are virtualizing business critical apps� To maintain high availability

� To provide us better DR solutions

� To save money

� Virtualization has changed how we operate

� We have a strategic view of virtualization

Additional InformationFor more information visit us at the following websites:

VMware

http://www.vmware.com

Exchange Solutions and Performance Studies

http://www.vmware.com/go/exchange

Backup

Backup: University of Plymouth Overview

University of Plymouth Overview

Located in South West of UK

Over 30,000 students

3,000 Staff

Annual income of $300m

4 Centres of Excellence in Teaching and Learning

University of Plymouth IT Infrastructure

PlymSERVE

Virtualized server environment

Consolidation of 280 Physical Servers to 30

2 Data Centers

Use of VMotion, DRS and HA to provide highly resilient environment

Projected savings: $90k per annum on energy (75% reduction on non-virtualized)

177 tons of CO2

94% saving in floor space (32 racks to 2)

University of Plymouth IT Infrastructure

PlymSTORE

Virtualized Storage Infrastructure

Based around FalconStor IPStor Appliance (Resilient)

2 HP EVA 8000 SANs

2 IBM DS4800 SANs

2 SATA Arrays

Synchronous Mirroring of data between Data Centers

UoP Exchange Implementation

4 E2K7 Malibox Servers (4 vCPU, 16GB RAM)

Each Server has 17 StoresApprox 12,500 (1,500 – 2,000 MAPI) users per server

1 E2K7 CAS Server (2 vCPU, 2GB)

Approx 5-6K peak concurrent OWA connections

3 E2K7 Hub Transport Servers (2 vCPU, 2GB)

Up to 500K SMTP Messages handled per week

2 E2K3 Back End Database Servers (2 vCPU, 4GB RAM)

Providing Public Folder support for legacy Intranet Portal

UoP Exchange 2007 Topology

Internet

Hub Transport Mailbox

Client Access Server

MAPI Clients

SMTPOWA

UoP Virtualized Exchange

VMware ESX 3.02 HA Cluster

8 x HP BL685c Blades (4x2.8GHz DC CPU, 32GB RAM)

Storage provided on mirrored HP EVA8000 through FalconStorIPstor appliance

Split across 2 Data Centers

Anti-affinity rules set to prevent Mailbox servers from co-existing on same physical host

HA Cluster also supports other services as well as Exchange (HR System, SQL databases, Filestore)

UoP Exchange 2007 Topology

Mailbox

Mailbox

Mailbox

Mailbox

EVA8000

HA Cluster

Hub Transport

Filestore

Hub Transport

Hub Transport

HR DB

SQL

IPStorData Centre 1 Data Centre 2

Filestore

EVA 8000

Exchange and VMware Benefits

High Availability

Hardware FailureMailbox Servers restart in <5 minutes

CAS and Hub Transport restart in 2 minutes

VMotion allows maintenance to happen without interruption to service

Reduced Complexity

No O/S-level or Application-level Cluster Service requiredSimpler configuration

Easier Troubleshooting (not that we’ve had to troubleshoot much!)

Clustering requires comparable hardware – VMware doesn’t!

Exchange and VMware Benefits (contd)

Hardware Independence

VMs are hardware independent

Could upgrade/switch hardware without requiring new drivers

Scalability

Possible to scale each VM to 64GB RAM (with ESX 3.5)

Can start with smaller RAM then increase until sweet spot found

Enhanced Testing and Rollback

Use of Snapshot and Clone

High user satisfaction

80% reduction in user complaints for Exchange service

Virtualised Exchange Performance – Real World

CPU Memory

Average Max Average Max

E2K7 Mailbox

620MHz 10GHz 2GB 7GB

E2K7 CAS 620MHz 4.6GHz 700MB 1.1GB

E2K7 Edge Transport

180MHz 970MHZ 200MB 600MB

E2K3 Public Folders

470MHz 5GHz 300MB 1GB

Data gathered from live production environment: Mailbox and CAS Serversover 11 months, Edge Transport over 4 months and E2K3 over one month

Summary

You can run Exchange 2007 with 50,000 mailboxes under VMware and achieve high user satisfaction

Take notice of Microsoft’s sizing recommendations

In our experience there are a number of reasons why it is better to run Exchange under VMware

Configuration is simpler

VMotion and HA provide superior levels of HA

Improved testing and rollback

Enhanced Management