Best Practices for Virtualization of Microsoft BizTalk Server 2009 with Hyper-V

Ewan Fairweather & Petr KratochvilBPD Customer Experience TeamMicrosoft

Microsoft Confidential

Session Objectives

Quantify BizTalk Server 2009 performance vs. BizTalk Server 2006 R2 Quantify performance of BizTalk Server 2009 on “Hyper-V”Provide guidance on how to effectively performance test BizTalk 2009

Microsoft Confidential

Session Agenda

BackgroundGoalsUse casesR2 vs. 2009 Performance Comparison

Optimizing platformOptimizing application design

Hyper-V PerformanceConclusion

4

Petr KratochvilLeadHQ

Focused on:

Performance

Focused on:Operational Readiness

EwanFairweather(PM)US- West

Tim Wieman (Senior PM)US-Central

PaoloSalvatori(Principal PM)EMEA

Focused on:Architecture Design

Focused on:RFID, Architecture Design

RamaRamani(PM)US

Focused on:Architecture Design, Operational Readiness

Quoc Bui( Senior PM)Asia

Who we are

Microsoft Confidential

BizTalk 2009 MSDN Guidance

All Guides are published to:MSDN, TechNet ,CHM

End to end samples:Sample applicationBizUnit scriptsLogParser scriptsVisual Studio Load Tests

RTM + 90RTMBeta

Product Release Cycle

CA

T &

UE

Deliver ADR to TAP Customers

Deliver MSDN Hyper-V Guidance

Performance and Hyper-V Boot Camp

Enterprise Customer Engagements

Deliver MSDN Performance Guidance

3

1 2

Deliver MSDN Operational Guidance

4

Denotes investment

Denotes delivery

Microsoft Confidential

BizTalk Performance Story

The usual storyWe all know testing is important for every solutionTesting is neglected

This is a problemSolutions using BizTalk tend to be business critical and tolerate little downtimePoor testing increases the risk profile of your projectThis runs the risk of huge technical and political problemsOur labs have shown that infrastructure tuning alone can double performance in some cases

Microsoft Confidential

Session Agenda

BackgroundGoalsUse casesR2 vs. 2009 Performance Comparison

Optimizing infrastructureOptimizing application design

Hyper-V PerformanceConclusion

8

BizTalk 2009 Performance Lab GoalsTechnical Goals:• Quantify BizTalk Server 2009 (2009) vs. BizTalk Server 2006 R2 (R2)• Quantify performance of physical vs. “Hyper-V” BizTalk Server 2009• Quantify performance of physical vs. “ Hyper-V”SQL Server 2008• Quantify performance of consolidating BizTalk/SQL on “Hyper-V”Use Case 1:

Logical Ports WCF Orchestration scenarioUse Case 2:

Inline Sends Orchestration ScenarioConditions:• Perform optimizations on 2009, R2, Hyper-V • Determine delta of:

• Throughput # of calls within 8 hours• Latency

• Identical infrastructure for all platforms• Apply infrastructure optimizations first, then application optimizations

Microsoft Confidential

Session Agenda

BackgroundGoalsUse casesR2 vs. 2009 Performance Comparison

Optimizing infrastructureOptimizing application design

Hyper-V PerformanceConclusion

10

Inline Sends and Logical Ports

Vs.

Logical Port Orchestration

Inline Sends Orchestration

11

Calculator Request<CalculatorRequest mlns="http://Microsoft.BizTalk.CAT.Samples.Schemas.CalculatorRequest">  <Method>XmlDocumentOrchestration</Method>  <Operations>    <Operation>      <Operator>+</Operator>      <Operand1>82</Operand1>      <Operand2>18</Operand2>    </Operation>    <Operation>      <Operator>-</Operator>      <Operand1>30</Operand1>      <Operand2>12</Operand2>    </Operation>    <Operation>      <Operator>*</Operator>      <Operand1>25</Operand1>      <Operand2>8</Operand2>    </Operation>    <Operation>      <Operator>\</Operator>      <Operand1>100</Operand1>      <Operand2>25</Operand2>    </Operation>      <Operation>      <Operator>+</Operator>      <Operand1>100</Operand1>      <Operand2>32</Operand2>    </Operation>  </Operations></CalculatorRequest>

1

2

3

4

12

Calculator Response<CalculatorResponse xmlns="http://Microsoft.BizTalk.CAT.Samples.Schemas.CalculatorResponse">      <Status>Ok</Status>      <Results>            <Result>                  <Value>100</Value>                  <Error>None</Error>            </Result>            <Result>                  <Value>18</Value>                  <Error>None</Error>            </Result>            <Result>                  <Value>200</Value>                  <Error>None</Error>            </Result>            <Result>                  <Value>4</Value>                  <Error>None</Error>            </Result>            <Result>                  <Value>132</Value>                  <Error>None</Error>            </Result>      </Results></CalculatorResponse>

1

13

Code Walk-ThroughInline Sends and Logical Ports examples

Demo

Microsoft Confidential

Session Agenda

BackgroundGoalsUse cases

R2 vs. 2009 Performance ComparisonOptimizing platformOptimizing application design

Hyper-V PerformanceConclusion

Lab Hardware Kit - Overview

1

2

3

4

5

6

7

8 9

Detailed SAN Configuration

1

2

3

R2 vs. 2009 Performance Comparison

Name Model CPU Type # of CPUs # of Cores/CPU Architecture RAM Min Local Disks OS SoftwareTAP2X-L04 DL380 G5 Intel Xeon 2 x 2.33 Ghz 4 x64 8 GB 3 x 72gb 10k SAS Win2k8 EE 64bit BizTalk 2009

TAP2X-L03 DL380 G5 Intel Xeon 2 x 2.33 Ghz 4 x64 8 GB 3 x 72gb 10k SAS Win2k8 EE 64bit BizTalk 2009

TAP2X-L02 DL380 G5 Intel Xeon 2 x 2.33 Ghz 4 x64 8 GB 3 x 72gb 10k SAS Win2k3 R2 SP2 EE 64bit BizTalk 2006 R2

TAP2X-L01 DL380 G5 Intel Xeon 2 x 2.33 Ghz 4 x64 8 GB 3 x 72gb 10k SAS Win2k3 R2 SP2 EE 64bit BizTalk 2006 R2

TAP4x-M01 BL680 Intel Xeon 4 Proc (2.4 Ghz) 4 x64 32GB 3 x 72gb 10k SAS Win2k3 R2 SP2 EE 64bit SQL Server 2005

TAP4x-O01 BL680 Intel Xeon 4 Proc (2.4 Ghz ) 4 x64 32GB 3 x 72gb 10k SAS Win2k8 EE 64bit SQL Server 2008

Vs.

1

2

3

18

Platform Optimizations

They are grouped into the following categories: Platform OptimizationsSQL Server Optimizations: General (all SQL Servers)SQL Server Optimizations: BizTalk DatabasesBizTalk Optimizations

If appropriate:SQL Server Optimizations: Custom Databases

Microsoft Confidential

With 64 bit defaults - Results are not always what they seem

Baseline Comparison~27% throughput improvement R2~77% throughput improvement 2009

*Excessive dehydration was occurring, dehydration settings were non-optimal for 64 bit host

~25% latency improvement R2~45% latency improvement 2009

TestConcurrent Test Client Users

Messages Per Second

Avg Response (s)

% < 3 seconds

Messages Per Day

Test Length

Test Case 1 Baseline

R2 – 32 Bit Orch

100 45.4 2.24 94.36 ~1,309,000 10 minutes

2009 - 64 bit Orch

100 34.3 2.89 75.45 ~987,000 10 minutes

Test Case 1 Optimized

R2 64 bit Orch 100 58.06 1.69 97.5 ~1,672,000 10 minutes

2009* 64 Bit Orch

100 60.64 1.61 97.97 ~1,746,000 10 minutes

1 2 3

5 6 7

4

8

20

Troubleshooting Dehydration

Counters to use:Process\Virtual Bytes VirtualMemoryThrottlingCriteriaProcess\Private Bytes PrivateMemoryThrottlingCriteria

1 2

Setting Original (MB) Optimization Applied (MB)

Virtual Memory 900 6000

Private Memory 50 700

21

1

2

Determining Dehydration Values Using Perfmon

Virtual Bytes ~ 5.4 GB

Private Bytes increases under load max ~600MB

22

Optimizing DehydrationDefault <xlangs><Configuration MaxThreshold="1800"

MinThreshold="1" ConstantThreshold="-1"><Dehydration><VirtualMemoryThrottlingCriteria

OptimalUsage=“900" MaximalUsage=“1300" IsActive=“true" />

<PrivateMemoryThrottlingCriteria OptimalUsage=“50" MaximalUsage=“350" IsActive="true" />

<PhysicalMemoryThrottlingCriteria OptimalUsage="700" MaximalUsage="1000" IsActive="false" /> </Dehydration>

</Configuration> </xlangs>

Optimized<xlangs><Configuration MaxThreshold="1800"

MinThreshold="1" ConstantThreshold="-1”><Dehydration><VirtualMemoryThrottlingCriteria

OptimalUsage="6000" MaximalUsage="6300" IsActive="true" />

<PrivateMemoryThrottlingCriteria OptimalUsage="700" MaximalUsage="1000" IsActive="true" />

<PhysicalMemoryThrottlingCriteria OptimalUsage="700" MaximalUsage="1000" IsActive="false" /> </Dehydration>

</Configuration> </xlangs>

Setting Counter To Use

VirtualMemoryThrottlingCriteria Process\Virtual Bytes

PrivateMemoryThrottlingCriteria Process\Private Bytes

1

2

Microsoft Confidential

Optimizing 64 Bit Dehydration SettingsTest

Concurrent Test Client Users

Messages Per Second

Avg Response (seconds)

% < 3 seconds

Messages Per Day

Test Length

32 Bit Orchestration Host Default Config File

R2 100 55.82 1.75 95.57 ~1,607,000 10 minutes

2009 100 56.25 1.74 95.31 ~1,620,000 10 minutes

64 Bit Orchestration Host Default Config File

R2 100 45.63 2.15 95.85 ~1,314,000 10 minutes

2009 100 45.49 2.16 88.95 ~1,310,000 10 minutes

64 Bit Orchestration Host –Optimized Config File

R2 100 58.06 1.69 97.5 ~1,672,000 10 minutes

2009 100 60.64 1.61 97.97 ~1,746,000 10 minutes

1 2

3

5 6

4

7

• Tuning dehydration settings on 64 bit host:• Improved throughput by 27% (R2) & 33% (2009)

Microsoft Confidential

Session Agenda

BackgroundGoalsUse casesR2 vs. 2009 Performance Comparison

Optimizing platformOptimizing application design

Hyper-V PerformanceConclusion

TestConcurrent Test Client Users

Messages Per Second

Avg Response (sec)

% < 3 seconds

Messages Per Day

Test Length

Logical Ports – Stair Step Load

2009 100-800 64.9 6.51 18.43 ~1,869,000 30 minutes

Inline Sends – Stair Step Load

2009 100-800 148.15 0.23 99.98 ~4,262,000 30 minutes

Inline Sends Performance Improvement

Low Latency4x throughput improvement in this scenario under constant loadUseful for scatter-gather pattern

TestConcurrent Test Client Users

Messages Per Second

Avg Response (sec)

% < 3 seconds

Messages Per Day

Test Length

Logical Ports – Constant Load

2009 100 60.64 1.61 97.97 ~1,746,000 10 minutes

Inline Sends – Constant Load

2009 100 236.96 0.24 99.98 ~6,824,000 10 minutes

1 2 3 4

Constant Load

Stair Step Load

Microsoft Confidential

Session Agenda

BackgroundGoalsUse casesR2 vs. 2009 Performance Comparison

Optimizing platformOptimizing application design

Hyper-V Performance Conclusion

27

The BizTalk 2009 Hyper-V Guideannouncing

MSDNhttp://msdn.microsoft.com/en-us/library/dd722825.aspx

28

Hyper-V Architecture

VHD - single file sits on root NTFS volumeFixed, Dynamic, Differencing

Pass-through – exclusive disk access to guest

Root Partition

I/OStackI/O

Stack

DriversDrivers

Child Partition

I/OStackI/O

Stack

VSCsVSCs

ServerServer

Child Partition

I/OStackI/O

Stack

VSCsVSCs

ServerServer

HypervisorHypervisor

Devices Processors Memory

VMBusVMBus VMBusVMBusVMBusVMBusShared Memory

VSPsVSPsVSPsVSPs

OS Kernel Enlightenments

(WS08+)

1

2

3

4

5

29

BizTalk Server

Virtual Processors

0

BizTalk Server

Virtual Processors

0

BizTalk Server

Virtual Processors

0

BizTalk Server

Virtual Processors

0

Logical Processors

0 1 2 3

1 to 1 Mapping Overload Scenario

BizTalk Server

Virtual Processors

0 1

BizTalk Server

Virtual Processors

0 1

BizTalk Server

Virtual Processors

0 1

BizTalk Server

Virtual Processors

0 1

Logical Processors

0 1 2 3

Virtual Processor Allocation

1 2

Physical BizTalk 2009 vs. Hyper-V BizTalk 2009

Vs.

•Physical and Hyper-V BizTalk Servers part of the same Group Name Model CPU Type # of CPUs # of Cores/CPU Architecture RAM Min Local Disks OS SoftwareTAP2X-

L06 DL380 G5 Intel Xeon 2 x 2.33 Ghz 4 x64 8 GB 3 x 72gb 10k SAS Win2k8 EE 64bit BizTalk 2009

Hyper-V ImagesTAP2X-

L05 DL380 G5 Intel Xeon 2 x 2.33 Ghz 4 x64 8 GB 3 x 72gb 10k SAS Win2k8 EE 64bit BizTalk 2009

Hyper-V ImagesTAP2X-L04

DL380 G5 Intel Xeon 2 x 2.33 Ghz 4 x64 8 GB 3 x 72gb 10k SAS Win2k8 EE 64bit BizTalk 2009

TAP2X-L03

DL380 G5 Intel Xeon 2 x 2.33 Ghz 4 x64 8 GB 3 x 72gb 10k SAS Win2k8 EE 64bit BizTalk 2009

TAP4X-O01 BL680 Intel Xeon 4 Proc (Ghz TBC) 4 x64 32GB 3 x 72gb 10k

SAS Win2k8 EE 64bit SQL Server 2008

1 2

3

TestConcurrent Test Client Users

Messages Per Second

Avg Response (s)

% < 3 seconds

Messages Per 8-hour Day

Test Length

Logical Ports on Optimized Platform

2009 100 60.64 1.61 97.97 ~1,746,000 10 minutes

2009 Hyper-V 100 52.96 1.86 94.8 ~1,520,000 10 minutes

Inline Sends on Optimized Platform

2009 100 236.96 0.24 99.98 ~6,824,000 10 minutes

2009 Hyper-V 100 223.54 0.28 99.87 ~6,438,000 10 minutes

1 2 3

5 6 7

4

Hyper-V delivers ~87% throughput* Hyper V delivers ~85% latency*

*Note: Calculated from Logical Ports Test

8

Hyper-V Results – BizTalk Tier

Hyper-V Summary – BizTalk Tier

Baseline / BizTalk and SQL on separate Physical HardwareBizTalk on Hyper-V / SQL on Physical Hardware

BizTalk and SQL on Hyper-V / Separate Hyper-V Host ComputersBizTalk and SQL on Hyper-V / Consolidated Hyper-V Host Computer

0

5

10

15

20

25

30

35

2.162.29

2.713.57

34.3

32.36

27.38

20.8533333333333

Key BizTalk Performance Indicators

Latency / Request-Response La-tency (sec)

Throughput / Documents processed per second

12

3

4

Hyper-V Summary – SQL Tier

Baseline / BizTalk and SQL on separate Physical HardwareBizTalk on Hyper-V / SQL on Physical Hardware

BizTalk and SQL on Hyper-V / Separate Hyper-V Host ComputersBizTalk and SQL on Hyper-V / Consolidated Hyper-V Host Computer

0

10

20

30

40

50

60

70

80

90

100

11

21

63.4761.65

52.88

41.39

90.188 88.7

89.5

Key SQL Performance Indicators

Totals Avg Disk Read (ms)

Total Drives Disk Write (ms)

SQL Server:SQL StatisticsBatch Requests/sec /100

SQL Processor Utilization %

34

Processor Lessons Learned

To measure Guest processors:\Hyper-V Hypervisor Virtual Processor(*)\% Guest Run Time\Hyper-V Hypervisor Virtual Processor(*)\% Total Run Time

To measure Physical processors:\Hyper-V Hypervisor Logical Processor(*)\% Total Run Time

To measure Host processors:\Hyper-V Hypervisor Root Virtual Processor(*)\% Total Run Time

35

Disk Lessons Learned

For VHD’s use:<RootPartition>\Logical Disk(*)\Avg. sec/Read<RootPartition>\Logical Disk(*)\Avg. sec/Write<Guest>\Logical Disk(*)\Avg. sec/Read<Guest>\Logical Disk(*)\Avg. sec/Write

For PassThrough Disk Analysis use:“Hyper-V Virtual Storage Device” countersxPerf Tool http://perftools

36

Pass-Through Disk Perf

Physical SQL Physical Disk

Virtual SQL Pass-Through Disk

0 50 100150200250300

Transfers (reads/writes) per second

Transfers (reads/writes) per second

Measurement Physical SQL Physical Disk

Virtual SQL Pass-through Disk

Difference

Transfers/Sec 269.73 250.47 7%*

* Other factors such as SAN performance could have been a factor

37

Disk Lessons Learned

Use PassThrough disks for high disk I/O:BizTalk MessageBox DatabaseBizTalk Tracking DatabaseBizTalk File Adapter %temp directory if BizTalk is streaming large files

38

Key TakeawaysQuantify BizTalk Server 2009 performance vs. BizTalk Server 2006 R2

3-5% improvement in 2009W2K3 Networking optimizations did not improve performance on W2K8 in our lab

Quantify performance of BizTalk Server 2009 on “Hyper-V”Approx 13% Hyper-V overhead for BizTalk tier. This is comparable to other products.SQL becomes CPU bound due to 4 VP limit

Provide guidance on how to effectively performance test BizTalk 2009

Visual Studio Testing Tools are a great driver for testing processSample code to be provided:

LoadTest Plug-In for BizUnitTest applicationVS Load Test Project

39

MSDNhttp://msdn.microsoft.com/en-us/library/dd722825.aspx

TechNethttp://technet.microsoft.com/en-us/library/dd722825.aspx

Microsoft Confidential

Helpful Tools

BizTalk BPAPerfMonLoadGenBizUnitLog ManRelogLog ParserPALSQLIO BizTalk Orchestration Profiler

Microsoft Confidential

SQL Hyper-V Session

DAT 318 – Microsoft SQL Server 2008 Virtualization Considerations and Best Practices

Room: 152 Time: 9AM – 10:15AM

42

Q & A

Microsoft Confidential

ResourcesBizTalk Server 2006: Managing a Successful Performance Lab

http://msdn2.microsoft.com/en-us/library/aa972201.aspx

Scaling Your Solutionshttp://msdn2.microsoft.com/en-us/library/aa561567.aspx

Configuration Parameters that Affect Adapter Performancehttp://msdn2.microsoft.com/en-us/library/aa561380.aspx

Persistence and the Orchestration Enginehttp://msdn2.microsoft.com/en-us/library/aa547090.aspx

Troubleshooting MessageBox Latency Issueshttp://msdn2.microsoft.com/en-us/library/aa561709.aspx

Performance Tips and Trickshttp://msdn2.microsoft.com/en-us/library/aa560694.aspx

Rule Engine Configuration and Tuning Parametershttp://technet.microsoft.com/en-us/library/aa548008.aspx

Performance Considerations When Using the Rule Enginehttp://msdn2.microsoft.com/en-us/library/aa952929.aspx

Microsoft Confidential

Filegroup – ResourcesBizTalk Server Database Optimization

http://technet.microsoft.com/en-us/library/bb743398.aspx Understanding Files and Filegroups

http://msdn2.microsoft.com/en-us/library/ms189563.aspxHow to: Add Data or Log Files to a Database (SQL Server Management Studio)

http://msdn2.microsoft.com/en-us/library/ms189253.aspxHow to: Move an Existing Index to a Different Filegroup (SQL Server Management Studio)

http://msdn2.microsoft.com/en-us/library/ms175905.aspxPhysical Database Files and Filegroups

http://msdn2.microsoft.com/en-us/library/ms179316.aspxWorking with tempdb in SQL Server 2005

http://www.microsoft.com/technet/prodtechnol/sql/2005/workingwithtempdb.mspx

Optimizing Server Performance Using I/O Configuration Optionshttp://msdn2.microsoft.com/en-us/library/ms189892.aspx

45

www.microsoft.com/teched Sessions On-Demand & Community

http://microsoft.com/technet Resources for IT Professionals

http://microsoft.com/msdn Resources for Developers

www.microsoft.com/learning Microsoft Certification & Training Resources

Resources

www.microsoft.com/learningMicrosoft Certification and Training Resources

46

Complete an evaluation on CommNet and enter to win!

Microsoft Confidential

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED

OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Ongoing Orchestration Performance Testing

Message processing performanceXMLDocumentXMLReader/XMLWriter and Virtual StreamCustom Message Class

Test Messages5 operations, 2 KB500 operations, 125 KB5000 operations 1243 KB

Ongoing Pipeline Performance Testing

Pipeline processing performanceXMLDocumentXMLReaderXPathReader

Test Messages5 operations, 2 KB500 operations, 125 KB5000 operations 1243 KB

50

Appendix Memory Lessons Learned

VM’s require non-paged memory (physical memory)Ensure you have enough physical memory on both the host and guest computersUse Traditional Counters:

<RootPartition>\Memory\Available Mbytes<Guest>\Memory\Available Mbytes<RootPartition>\Memory\Pages/sec<Guest>\Memory\Pages/sec

51

Appendix Network Lessons Learned

For high network I/O:Use a 1:1 mapping of physical to virtual network adapters

For measuring network adapters use:<RootPartition>\Network Interface

Bytes Total Per SecondOutput Queue Length

<RootPartition>\Hyper-V Virtual Network AdapterBytes Per Second

<RootPartition>\Hyper-V Virtual Switch

Vs.

BizTalk Comparison Tier

SQL Comparison Tier

1 2

3 4

5