jetstress field guide v1.0.0.6 by microsoft

8/8/2019 Jetstress Field Guide v1.0.0.6 By Microsoft

http://slidepdf.com/reader/full/jetstress-field-guide-v1006-by-microsoft 1/47

Jetstress 2010

Exchange 2010 Jetstress Field Guide

15:14:05

Version 1.0.0.6Released

Prepared by

Neil Johnson

Senior Consultant



MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under

copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or

transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for

any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights

covering subject matter in this document. Except as expressly provided in any written license agreement from

Microsoft, our provision of this document does not give you any license to these patents, trademarks,

copyrights, or other intellectual property.

The descriptions of other companies’ products in this document, if any, are provided only as a convenience to

you. Any such references should not be considered an endorsement or support by Microsoft. Microsoft cannot

guarantee their accuracy, and the products may change over time. Also, the descriptions are intended as brief

highlights to aid understanding, rather than as thorough coverage. For authoritative descriptions of these

products, please consult their respective manufacturers.

© 2010 Microsoft Corporation. All rights reserved. Any use or distribution of these materials without express

authorization of Microsoft Corp. is strictly prohibited.

Microsoft and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United

States and/or other countries.

The names of actual companies and products mentioned herein may be the trademarks of their respective

owners.

Page iiExchange Jetstress 2010, Field Guide, Version 1.0.0.6Released

Prepared by Neil Johnson

"45334382" last modified on 15/11/2010, Rev 7



Revision and Signoff Sheet

Change Record

Date Author Version Change reference

19/08/20

10

Neil

Johnson

0.1.0.7 • First draft sent out for review

26/08/20

10

Neil

Johnson

0.1.0.8 • Updates from MCS UK Messaging

team

27/08/20

10

Neil

Johnson

0.1.0.9 • Updates from Alex Costa

• Added in lab test data

29/08/20

10

Neil

Johnson

0.1.0.10 • UK Messaging team feedback

12/09/20

10

Neil

Johnson

0.1.0.12 • Ross Smith IV review feedback

• Robert Gillies review feedback

13/09/20

10

Neil

Johnson

1.0.0.0 • Released

16/09/20

10

Neil

Johnson

1.0.0.1 • Jeff Mealiffe review feedback

• Updated Jetstress test types 6.1

20/09/2010

Neil Johnson

1.0.0.2 • Scott Scholl review feedback

29/09/20

10

Ross Smith

IV

1.0.0.3 • Incorporated troubleshooting and

cmdline appendix sections

29/09/20

10

Ramon B.

Infante

1.0.0.4 • Review and Update on Log

Replication

11/10/20

10

Neil

Johnson

1.0.0.5 • Incorporated Ramon b. Infante’s

comments

•Updated Table 8 - Quick resultsanalysis table

15/11/20

10

Neil

Johnson

1.0.0.6 • Updated for version 14.01.0225.017

Page iiiExchange Jetstress 2010, Field Guide, Version 1.0.0.6Final





Document Contributors

Name Position Section

Neil Johnson Senior Consultant, UK MCS Author

Alexandre

Costa

Senior SDET, Exchange Test Jetstress internals

Ross Smith IV Principal Program Manager, Exchange CXP Configuring

Jetstress

Ramon b.

Infante

Director, WW Communities, UC Various

Page ivExchange Jetstress 2010, Field Guide, Version 1.0.0.6Final





Reviewers

Name Versio

n

Position Date

Doug

Gowans

0.1.0.7 Senior Consultant, UK MCS 13/08/2010

Michael

Currie

0.1.0.7 Senior Consultant, UK MCS 18/08/2010

Alexandre

Costa

0.1.0.7 Senior SDET, Exchange Test 19/08/2010

Neil Hobson 0.1.0.9 Infrastructure Consultant, UK MCS 27/08/2010

Ross Smith

IV

0.1.0.1

0

Principal Program Manager, Exchange CXP 10/09/2010

Robert Gillies 0.1.0.1

0

Solution Architect, US MCS 10/09/2010

Jeff Mealiffe 1.0.0.0 Senior Program Manager, Exchange CXP 16/09/2010

Scott Schnoll 1.0.0.1 Principal Technical Writer, Office Content

Publishing (UA)

20/09/2010

Ramon b.

Infante

1.0.0.4 Director, WW Communities, UC 29/09/2010

Special Thanks

Mark Jacklin at the Microsoft UK Services lab for providing the storage and server

hardware used within this document.

Page vExchange Jetstress 2010, Field Guide, Version 1.0.0.6Final





Table of Contents

1 Purpose...............................................................................1

2 Introduction to Jetstress.......................................................2

3 Jetstress Internals................................................................3

3.1 Main Jetstress Components..........................................................................3

3.1.1 Auto Tuning Component........................................................................3

3.1.2 Thread Dispatcher.................................................................................4

3.1.3 Background Log Checksummer.............................................................4

3.1.4 Offline Log and Database Checksummer..............................................4

3.1.5 Reporting and Verification.....................................................................4

4 Planning for Jetstress...........................................................5

4.1 Jetstress testing flow chart...........................................................................54.2 When should I run Jetstress in my project?..................................................6

4.3 Where should I run Jetstress in my infrastructure?......................................7

4.4 How much time should I allocate for Jetstress testing?................................8

4.4.1 Initialisation...........................................................................................8

4.4.2 Testing..................................................................................................8

4.4.3 Cleanup.................................................................................................8

4.5 Preparing for the Jetstress test....................................................................9

4.6 File Level Anti-Virus is configured to exclude all Exchange data locations

and any directories that Jetstress has been configured to use. What happens if

the test fails?.........................................................................................................9

5 Installing Jetstress.............................................................11

5.1 Online Documentation...............................................................................11

5.2 Jetstress Version and Download.................................................................11

5.3 Prerequisites..............................................................................................11

5.4 Getting ESE Files necessary for Jetstress...................................................13

5.4.1 File locations from an installed Exchange Server................................13

5.4.2 File locations from the installation media............................................13

5.5 Installation.................................................................................................14

5.5.1 Application Installation........................................................................14

5.5.2 ESE File Installation.............................................................................16

6 Configuring Jetstress..........................................................19

6.1 Jetstress Test Types...................................................................................19

6.1.1 Test a disk subsystem throughput......................................................19

6.1.2 Test an Exchange mailbox profile.......................................................19

6.2 Initial configuration....................................................................................20

Page viExchange Jetstress 2010, Field Guide, Version 1.0.0.6Final





7 Jetstress Output Files.........................................................27

8 Reading Jetstress report data.............................................28

8.1 Target design values..................................................................................28

8.2 Reading the Jetstress Test Result Report...................................................28

8.2.1 Test Summary.....................................................................................288.2.2 Database Sizing and Throughput........................................................29

8.2.3 Jetstress System Parameters...............................................................29

8.2.4 Database Configuration.......................................................................29

8.2.5 Transactional I/O Performance............................................................30

8.2.6 Background Database Maintenance I/O Performance.........................30

8.2.7 Log Replication I/O Performance.........................................................30

8.2.8 Total I/O Performance..........................................................................31

8.2.9 Host System Performance...................................................................32

8.2.10 Test Log...............................................................................................32

8.3 Interpreting Jetstress test result................................................................338.4 Test evaluation..........................................................................................34

9 Appendix A – Configuring thread count................................35

10 Appendix B – Configuring sluggishsessions........................36

10.1 Lab test data for SluggishSessions............................................................37

11 Appendix C - Running a Jetstress Test with JetstressCmd.exe

38

12 Common Issues...............................................................40

12.1 Log or Data Volumes cannot be overlapped..............................................40

12.2 Troubleshooting Jetstress...........................................................................40

12.2.1 Jetstress cannot attach to or create a database..................................40

12.2.2 Error loading Performance Monitor counters.......................................41

12.2.3 Database Performance counters not working after using Jetstress... ..41

12.2.4 Unable to tune for the parameters......................................................41

Unable to tune for the parameters

Page viiExchange Jetstress 2010, Field Guide, Version 1.0.0.6Final





1 Purpose

This document is intended to explain the process and requirements for validating

an Exchange storage solution prior to releasingan Exchange deployment into

production.

It will explain how Jetstress works, how to plan for and perform a Jetstress test, and

how to automate the process.

This document is not intended to provide Exchange storage design guidance. For

guidance on Exchange server design and planning refer toMailbox Server Storage

Design.

Page iExchange Jetstress 2010, Field Guide, Version 1.0.0.6Final



http://technet.microsoft.com/en-gb/library/dd346703.aspx






2 Introduction to Jetstress

Jetstress is a tool for simulating Exchange database I/O load without requiring

Exchangeto be installed. It is primarily used to validate physical deployments

against the theoretical design targets that were derived during the design phase.

To accurately simulate the complex Exchange database I/O pattern, Jetstress makes

use of the same ESE.DLL that Exchange uses in production. It is therefore vital that

Jetstress use the same version of the Extensible Storage Engine (ESE) files that your

Exchange infrastructure will be built with in production.

Ideally, Jetstress testing will be part of the overall project plan. The best time to

schedule Jetstress testing is just before Exchange will be physically installed onto

the servers.

Jetstress testing provides the following benefits prior to deploying live users.

• Validates that the physical deployment is capable of meeting specific

performance requirements

• Validates that the storage design is capable of meeting specific

performance requirements

• Finds weak storage components prior to deploying in production

• Proves storage and I/O stability

The most important aspect of Jetstress testing is that it allows you to see how the

physically deployed storage and server infrastructure will behave once a real

Exchange workload is applied. This often works out differently from expectations,

especially in scenarios where shared storage infrastructure is deployed or where

the storage design is complex.

Often the Jetstress test will not provide the results that were expected. Sometimes

by making subtle configuration changes to the storage infrastructure(for example,

driver or firmware updates) it is then possible to get the test to pass.

Fundamentally, a successful Jetstress test validates that all of the hardware and

software components within the I/O stack from the operating system down to the

physical disk drive are working to a sufficient level to meet the predicted

performance required by Exchange to operate successfully.

Note: The validity of your Jetstress testing is only as good as the user profileanalysis and workload prediction that was completed during the design phase of

the project.






1 Jetstress Internals

1.1 Main Jetstress Components

Like Exchange, Jetstress is an ESE-based application. It runs in user memory space,makes API calls to ESE, which in turn makes calls to the Windows File system and

I/O Manager to gain access to the data stored on disk. During each of these tasks

Windows records performance information about the specific task and the operating

system as a whole. Once the test is completed, Jetstress analyses the performance

data to determine if the system meets the targets specified at the beginning of the

test.

Extensible Storage Engine(ESE)

Storage Subsystem

Background DatabaseMaintenance

Transactional I/O

W i n d o w s I / O M a n a g e r

D e v i c e D r i v e r s

JetstressApplication

Auto tuning

Thread Dispatcher

Background Log Checksummer

Offline Log & Database Checksummer

Windows Operating System Hardware

WindowsPerformance Counters

Reporting and Verification P e r f o r m a n c e D a t a

Figure 1 - Main Jetstress Components

1.1.1 Auto Tuning Component

This component is responsible for auto tuning within Jetstress. It attempts to

determine the maximum thread count that the solution can support. Each thread

performs a set amount of ESE calls, which generates a set amount of disk I/O. By

raising or lowering the thread count per database the storage workload can be

modified. The auto tuning component attempts to programmatically determine the

maximum thread count that the storage solution can support; whilst remaining

within the published disk latency guidelines for Exchange Server. The Jetstress test

parameters for disk latency are shown in section Interpreting Jetstress test result.

Note: Autotuning is generally unsuccessful if more than a single database is stored on the same set of physical disk spindles.






1.1.2 Thread Dispatcher

The thread dispatcher is responsible for managing workload within Jetstress. The

main areas of interest within the thread dispatcher are as follows:

• ThreadCount: number of transactional threads per database (prior to

Exchange 2010, it used to be the number of threads per storage group).

• ThreadTypes: each of those threads chooses to do one type of work against

the database. The same thread can perform different types of work during a

given run. There are four types: insert, read, update and delete (all of those

against records on a table). The default operation mix for an Exchange 2010

simulation is: 40%, 35%, 5% and 20%, respectively.

• SluggishSessions: the default is 1 for Exchange 2010. This is usually used to

fine tune the amount of work performed by a given thread. Internally, athread sleeps for (SluggishSessions * TaskRunTime) before picking up the

next task to run. For example, if you have 3 for SluggishSessions and an

insert thread took 100 msec in the last cycle, it will sleep for 300 msec

before moving on to the next cycle. Of course, 0 means “go full throttle”.

1.1.1 Background Log Checksummer

This component simulates the I/O overhead of additional database copies. This

copy operation has an I/O cost which increases with each additional copy.

1.1.2 Offline Log and Database Checksummer

This process checksums all database and log files at the end of a Jetstress run to

ensure that all data is intact. It also provides performance data for CRC checksum

speed should VSS copies require a checksum prior to backup.

1.1.3 Reporting and Verification

At the end of a Jetstress test this process compares the observed performance

results against a set of acceptable values. These results are then written to a HTML

file. During the test binary performance data is written out to a BLG file.






2 Planning for Jetstress

Jetstress testing is often misunderstood from a planning perspective. Particularly,

how much time to allocate for testing and which part of the project should Jetstress

testing occur?

2.1 Jetstress testing flow chart

The aim of the following process is to find the maximum workload while still passing

the test. Fundamentally we need to increase workload until the test fails. The last

value before failure is the highest workload that the system can support. If this

value is below the design target, we then use sluggishsessions to fine tune the test.

Increase threadcount until test fails

Compare highestIOPS from passed test

to design targets Testing Begins

Increasesluggishsessionsuntil test pases

FAIL

Record resultsPASS Testing Ends

Test failingdue to latency but not

reaching designtarget

NO

Storage unable tomeet designrequirements

YES

Figure 2 - Jetstress test flowchart






2.2 When should I run Jetstress in my project?

Jetstress testing can often take place at multiple phaseswithin the project plan.

Depending on the design approach taken, Jetstress testing may be performed

during both the planning (design) and build phases of a project.

Figure 3 - SDM phase overview

So, why would you run Jetstress during the planning/design phase of a project? The

simple answer is that with today’s powerful hardware it is often the case that the

Exchange design team must use standard “chunks” of hardware to create their

design. Rather than attempt to guess what the I/O limits are of the hardware it is

usually preferable to perform some Jetstress tests on the hardware to determine

the maximum storage IO capacity of the system. This allows the design team to

specify the bill of materials much more precisely, thereby saving money and

reducing risk.

But if you have already proven the solution in the lab, why test again at build time?

This is a common question; many projects only schedule sufficient time for testinga

single server and its storage solution with the belief that they only need to validate

the design. The problem with this approach is that it assumes a zero error rate in

the build out. What happens if someone forgets a part of the build on one server?

Or deploys a different device driver from the one used in the lab? What happens if a

faulty piece of hardware has been deployed? Jetstress testing at build time is a

great way to validate that the physically deployed hardware and software are

capable of providing the required I/O performance for Exchange. Jetstress testing

at build time is also a way to identify failing components such as disk drives; it is

much less stressful to identify a weak batch of disks during a Jetstress test than ona Monday morning after a large user migration!

If the project plan will allow it, build in sufficient time to test each and every server

and storage chassis that will bedeployed before migrating user mailboxes to it.

Remember that Jetstress can be fully automated, so with a little bit of planning it

can be left to run overnight and may not actually add any significant overhead to

the project.






2.4 How much time should I allocate for Jetstress testing?

Jetstress testing can take a long time to complete and it is vital that this time is

correctly planned for within your Exchange project plan.

Generally the test procedure can be broken up into three parts.

• Initialisation

• Testing

• Cleanup

1.1.1 Initialisation

This phase includes installation, prerequisites and initial database creation. Of

these tasks the initial database creation will take the longest amount of time.

Database creation time varies between hardware deployments however expect

around 24 hours for 10TB of data.

1.1.2 Testing

The actual testing phase will vary depending on the complexity and maturity of the

design. If your design is based on complex, cutting-edge storage technology it is

highly likely that you will need to allocate more time for testing. If your design is

based on common direct attached components the testing phase is likely to be

quite short. For simple direct attached solutions allow between 2-5 days, for

complex SAN solutions try to allocate up to 10 working days. Troubleshooting

storage performance issues can often be very time-consuming.

1.1.3 Cleanup

Before the server can be put into production it is necessary to remove the Jetstress

application and the test databases that were created. The recommended

procedure is as follows

• Uninstall Jetstress and Reboot

• Copy the Jetstress data to a safe location

• Delete the Jetstress installation folder

• Remove all test databases

Depending on complexity, allow between 1 and2 hours per Exchange server that

needs to have Jetstress uninstalled.






1.1 Preparing for the Jetstress test

Jetstress simulates an Exchange database workload. To ensure that the

environment is ready it should be configured according to both the hardware

vendor’s and Microsoft recommendations.

Refer to Understanding Exchange 2010 Storage Configuration for further detail.

As a starting point ensure that the following conditions have been met.

1. All devices on the storage system must be listed on the Windows Hardware

Compatibility List (HCL) if you are running Windows Server 2008 R2 and

Windows Server 2008. For more information, see Products Designed for

Microsoft Windows - Windows Catalog, Windows Compatibility Center, and

Windows Logo'd Product List.

2. If multiple clusters will be sharing any aspect of the disk subsystem, theserver/storage configuration must be Cluster/Multi-Cluster Certified.

3. Verify with vendors that drivers and firmware are current. Drivers and

firmware include, but are not limited to, the following items:

a. Server BIOS/firmware

b. SCSI/Array Controller firmware and driver

c. Fibre Host Bus Adapter (HBA) firmware and driver

d. Fibre switch/hub firmware

e. SAN (Storage Area Network) enclosure Operating

System/Microcode/firmware

f. Hard disk firmware

4. Verify that the HBA/SAN specific configuration is set correctly. Many HBAs

use registry keys to customize the configuration to a specific SAN platform

(for example, Queue Depth).

5. Raid Controller Stripe size is 256Kb or greater (refer to hardware vendor for

guidance).

6. Read/Write Cache is 75% Write and 25% Read on all LUN’s.

7. Configure the storage logical unit numbers (LUNs) (consider Exchange log

devices and database devices).

8. Format the LUNs within Windows with NTFS file system. Best practice = 64k

allocation unit size.

9. NTFS Compression is not enabled.10.




http://technet.microsoft.com/en-gb/library/ee832792.aspx

http://go.microsoft.com/fwlink/?linkid=26219



http://technet.microsoft.com/en-gb/library/ee832792.aspx






1.1 File Level Anti-Virus is configured to exclude all

Exchange data locations and any directories that

Jetstress has been configured to use. What happens if

the test fails?

It is important to determine the pass and fail criteria for the test. The test will find

the peak working load that the storage is able to provide at the I/O latency targets

recommended by the Microsoft Exchange Team. These are defined in section

Interpreting Jetstress test result.

If the recorded IOPS target from the Jetstress test is above the targets documented

within the Exchange design then the storage solution is deemed to have passed the

test. If it does not meet the design targets, then the storage solution is deemed to

have failed the test.

If the test shows that the storage has failed to meet its design targets it will be

necessary to perform remediation. This usually involves a combination of resources

from the design/project, build, hardware, and storage vendor teams. The aim of

remediation is to determine why the IOPS target was below the design target and to

provide a remediation plan before submitting the solution for a re-test.






2 Installing Jetstress

2.1 Online Documentation

The documentation for the Exchange Server 2010 version of Jetstress is availableon TechNet at the following location.

http://technet.microsoft.com/en-us/library/ff706601.aspx

2.2 Jetstress Version and Download

Version Build Usage Link

14.01.0225.

017

32

bit

• Exchange

2003

http://www.microsoft.com/downloads/en/

details.aspx?FamilyID=6c9c1180-4dd8-

49c4-85fe-

ca1cdcb2453c&displayLang=us14.01.0225.

017

64

bit

• Exchange

2007

• Exchange

2010

http://www.microsoft.com/downloads/en/

details.aspx?

displaylang=en&FamilyID=13267027-

8120-48ed-931b-29eb0aa52aa6

Table 1 - Jetstress version and download table

Note: Although there is a 32-bit build of Exchange 2007 it is not recommended or

supported to use these ESE files to run a Jetstress test. This is due to the

requirement for a 64-bit address space to accurately simulate a realistic Exchange

I/O pattern.

Always ensure that you use the same version of Jetstress to initialise the databases

and to perform the testing.

1.1 Prerequisites

• Microsoft .NET Framework 3.5 or higher

• A copy of your production ESE files2

○ ese.dll○ eseperf.dll○ eseperf.hxx○ eseperf.ini○ eseperf.xml

It is extremely important that the version of ESE that is used for the test is the

same version that will be used in production.

2 See section Getting ESE Files necessary for Jetstress for the locations of these files.




http://www.microsoft.com/downloads/en/details.aspx?FamilyID=6c9c1180-4dd8-49c4-85fe-ca1cdcb2453c&displayLang=us




http://www.microsoft.com/downloads/en/details.aspx?displaylang=en&FamilyID=13267027-8120-48ed-931b-29eb0aa52aa6












1.1 Getting ESE Files necessary for Jetstress

Jetstress requires ESE to function. The needed files are available from an installed

Exchange server or from the Exchange installation media. It is recommended to

get the files from an installed Exchange server that has been fully updated andpatched. If you are deploying an Exchange server that is based on either RTM or

SP1 code however, it is possible to get the necessary files directly from the

installation media without requiring an Exchange installation.

1.1.1 File locations from an installed Exchange Server3

File Path

ESE.DLL C:\Program Files\Microsoft\Exchange Server\V14\Bin

ESEPERF.DLL C:\Program Files\Microsoft\Exchange

Server\V14\Bin\perf\AMD64

ESEPERF.HXX C:\Program Files\Microsoft\Exchange


ESEPERF.INI C:\Program Files\Microsoft\Exchange


ESEPERF.XML C:\Program Files\Microsoft\Exchange


Table 2 - ESE file locations on running Exchange server

1.1.2 File locations from the installation media4

File Path

ESE.DLL \setup\serverroles\common

ESEPERF.DLL \setup\serverroles\common\perf\amd64

ESEPERF.HXX \setup\serverroles\common\perf\amd64

ESEPERF.INI \setup\serverroles\common\perf\amd64

3 These paths are for Exchange server 2010 SP14 These paths are for Exchange server 2010 SP1






3

.

Leave the installation options

as default unless you have a

good reason to change them.

Note: All performance data

and HTML reports will be

stored in the installation

folder.

4

.

This is the last chance to stop

the installation. Click on

“Next” to install…






5

.

Once installation is completed

click on “Close”.

Table 4 - Jetstress installation instructions

1.1.1 ESE File Installation

# Instruction Screenshot

1

.

Copy ESE prerequisite files

into the Jetstress installation

folder.

By default this is “c:\Program

Files\Exchange Jetstress”






2

.

Click on the “Start” menu and

open “Exchange Jetstress

2010”

Note: Jetstress requires local

Administrator access. If user

access control is enabled

ensure that you start the

JetstressWin.EXE process as

an administrator.

3

.

Click on “Start new test”

4

.

Jetstress will attempt to use

the ESE files that were copied

over in step 1. The first time

that this occurs Jetstress must

be restarted. Verify in the

output on this screen that the

ESE version is correct and that

the last line of the status

output requires that Jetstress

be restarted.






Close Jetstress

This is the end of the Jetstress

installation.

Table 5 - ESE installation instructions






1 Configuring Jetstress

For the purposes of this document we will be configuring a disk subsystem

throughput test. The goal of this test is to identify the peak working IOPS value that

the storage subsystem can sustain while remaining within the disk latency targetsestablished by the Exchange Product Group.

1.1 Jetstress Test Types

1.1.1 Test a disk subsystem throughput

This test uses some fixed parameters to determine the maximum storage

performance at maximum working capacity (80%). This is the recommended test

type since it identifies the maximum working load of the storage solution for use

with Exchange Server 2010. The values observed from this test can be used both

to qualify the solution ready for production and to calculate available system I/O

headroom once the service is in production. This test should be regarded as

mandatory for each and every Exchange server released into production.

1.1.2 Test an Exchange mailbox profile

Helps you determine whether your storage system meets or exceeds the planned

Exchange mailbox profile. In the Exchange mailbox profile test scenario, you can

specify the number of mailbox users, IOPS per mailbox and quota size to simulate

the profiled Exchange mailbox load. This test type can be useful if your storage has

been specifically designed to operate only at a specific disk capacity5.

Note: Even if this test type is used, it is still recommended to complete the disk subsystem throughput test to determine the maximum working load of the storage

solution at full capacity.

5 It is not recommended to design Exchange storage performance based on less than 80%utilisation capacity.






1.2 Initial configuration

# Instruction Screenshot

1

.

Click on the “Start” menu and

open “Exchange Jetstress

2010”

2

.

Click on “Start new test”






3

.

Check that the status text

does not ask for a restart and

that the last two lines state

that the ESE engine and

performance libraries weredetected.

4

.

Since this is the first time we

are configuring a test we will

accept the defaults and clicknext.

This will create a new

configuration file called

JetstressConfig.xml in the

default installation directory.

5

.

Select the “Test disk

subsystem throughput” test

and click “next”






6

.

Check supress tuning and use

the thread count per

database. To configure this

value correctly refer to

Appendix A – Configuring Thread Count.

7

.

Configure the test for

“performance”, do not

configure “multi-host” unlessthe test is running against a

shared storage solution. If the

Exchange database design will

have Background Database

Maintenance (BDM)disabled (it

is enabled by default), then

uncheck the Run BDM

checkbox.

For DAS deployments accept

the defaults.

8

.

Enter in the folder for storing

the test results and set the

correct duration for Jetstress.

Performance tests should be

run a minimum of 2 hours.

Note: While you are figuring

out the correct thread count to

use, you can set a shorter

than 2 hour test by typing

directly into the window.

• 0.75 = 45m

• 0.50 = 30m






• 0.25 = 15m

1

.

Configure the test to represent

theproduction deployment.

Number of databases should

be the total on this server

including all database copies.

Number of copies per

database represents the

number of total copies that

will exist for each

uniquedatabase. This value

simply simulates some LOG

I/O reads to account for thelog shipping between active

and passive databases – it

does NOT actually copy logs

between servers.

For example, if your 6 server

DAG contained 30 databases,

with 1 active copy, 2 passive

HA copies and 1 lagged copy

per database (or 120

database copies spread across

6 servers, with each server

hosting 20 copies), you would

set the Number of Databases

to 20 and the Number of

copies per database to 4.

2

.

Configure the database and

log file paths appropriately.

Scroll to the bottom of this

page to find the “next” link.






3

.

If this is the first time the test

has been run select to “create

new databases”, otherwise

select “Attach existing

databases”.

4

.

Verify that the paths are as

expected and click “Prepare

test”

5

.

This will begin database

initialisation – this process will

vary but plan on 24 hours for

every 10TB worth of data to

be initialised.

This value should equate to

80% of the available storage.

Refer to section DatabaseSizing and Throughput for

further information on

database sizes.






6

.

Once the test has been

initialised, click “Execute

Test”.

7

.

Once the test has completed,

close Jetstress and copy the

Jetstress report and

performance data somewhere

for analysis.

Each performance test willgenerate the following files.

• Performance_<date>.X

ML

• Performance_<date>.H

TML

• Performance_<date>.B

LG

• DBChecksum_<date>.X

ML

• DBChecksum_<date>.H TML

• DBChecksum_<date>.B

LG

• XMLConfig_<date>.XML

Ensure that you make a copy






of all of these files.

Note: In addition you may also

wish to make a copy of the

*.EVT files which contain

event log data taken during

the test.

Table 6 - Jetstress initial configuration

Page iiExchange Jetstress 2010, Field Guide, Version 1.0.0.6Final





1 Jetstress Output Files

This section will explain what output files will be created after the test and what

they are intended for.

File Content PurposePerformance_<date>.BLG Binary performance data

captured during the

performance test.

To provide detailed data for

analysis. Open this file in

perfmon and examine the

counters manually to

understand reasons for

failure.Performance_<date>.XML XML Report for the

performance test

Provides the status report

data in XML format.Performance_<date>.HTM

L

HTML Report for the

performance test

Provides an easy to read

status report for the test.DBChecksum_<date>.BLG Binary performance data

captured during the

checksum test.

Provides binary

performance data gathered

during the CRC checksum of

the database. Useful if the

checksum fails or takes a

long time to complete.DBChecksum_<date>.XML XML Report for the

checksum test

Provides status report data

in XML format.DBChecksum_<date>.HTM

L

HTML Report for the

checksum test

Provides an easy to read

status report for the

checksum test.XMLConfig_<date>.XML XML Configuration File Provides a backup of the

Jetstress Configuration file

used for the test.

Table 7 - Jetstress output files

Page ivExchange Jetstress 2010, Field Guide, Version 1.0.0.6Final





2 Reading Jetstress report data

This section will walk through a very simple sample report, and explain where the

key values are stored and how to interpret the data.

2.1 Target design values

Before we can evaluate our Jetstress data we need to know what our design targets

are. Assuming that the storage design was based on data from the Mailbox Role

calculator, the information we need is in the following table on the Role

Requirements tab.

Make a note of the following values.

• Total Database Required IOPS / Server

1.1 Reading the Jetstress Test Result Report

The following report is for a very simple test with a single database and a single

thread.

1.1.1 Test Summary

This section is a basic summary of the test, when it started, finished and which

versions of operating system and ESE were used.

The most important part of this section is the overall test result!






1.1.2 Database Sizing and Throughput

This section shows some more detailed parameters regarding the test. A

performance test should always have 100% for Capacity Percentage and

Throughput Percentage. In this example a 20GB LUN was used, Jetstress created a

16GB Database to test against which is only 80% of the available space. This is

normal behaviour; by default, in performance mode Jetstress will use 80% of the

disk capacity to allow room for growth during the test process.

The most important value in this section is the Achieved Transactional I/O per

Second .

This value represents the level of IOPS achieved for the database alone, without

BDM I/O or LOG I/O. This value directly maps to the Total Database Required

IOPS / Server value recorded in section Target design values.

1.1.3 Jetstress System Parameters

This section displays some system values that Jetstress used for this test. The

important values for analysis here are the thread count and number of copies per

database.

1.1.4 Database Configuration

This section lists the paths for each database and log combination. In this example

only one database was configured. Check that all of the test databases are listed

here and the path names are correct.






1.1.5 Transactional I/O Performance

This section of the report displays the Transactional I/O values that were achieved

for each database. Transactional I/O does not include I/O for Background Database

Maintenance.

BDM I/O is mostly sequential so it is not usually considered during the design phase.

Note: To validate that the test has met the design requirements compare the I/O

Database totals in this section to the design targets defined within the Mailbox Role

Calculator.

In this example,

• Design Target = 60 OPS (from storage calculator)

• Achieved IOPS = 31.5 + 33.3 = 64.8 IOPS (from jetstress report)

• 64.8 is greater than 60so the test has validated the storage subsystem.

1.1.1 Background Database Maintenance I/O Performance

This section displays the I/O that was used to perform Background Database

Maintenance only. The combination of these values plus Transactional I/O will be

the total IOPS generated during the test.

1.1.2 Log Replication I/O Performance

This section displays the I/O overhead for LOG file replication. In this example there

were no replica copies, this is shown by a zero count for I/O Log Reads/sec. If this

value is greater than zero it confirms that database replication is being simulated.






1.1.3 Total I/O Performance

This table shows all I/O that was recorded during the test (transactional I/O plus

BDM I/O). The summation of I/O values from this table should agree with those

observed at the storage subsystem.

Total Test IOPS = 53.3 + 33.3 + 33.4 = 120 IOPS

The following chart shows the observed IOPS from the Windows test host during the

Jetstress test. This counter includes all system IOPS as well as the test IOPS;

however there should be a strong correlation between the two metrics.

Figure 4 - Host observed IOPS

In this example, the storage should have been designed to meet a target of 60

IOPS, the discrepancy between design target and total achieved IOPS is caused by

the sequential BDM IOPS and LOG IOPS. If the storage team believes that BDM is

causing a test to fail, perform a test with BDM disabled and compare the results.

BDM does not usually add any significant overhead to the test latency (0.05% is

typical).






1.1.4 Host System Performance

This section of the report shows the observed system performance during the test.

This section is most often used for troubleshooting. The most important thing to

note from this section is that the CPU load from Jetstress is usually minimal.

Jetstress has been optimised to evaluate the storage subsystem and not the host

performance itself.

1.1.5 Test Log

This section of the report is a log of the Jetstress test. It is most often used for

troubleshooting or to record how long a test took to complete various stages.







1.1 Test evaluation

Evaluate the following criterion for each test run. The first two tests are validated

against the design targets and must be performed manually; Jetstress does not

validate these values. The third and fourth are against pre-defined latency targetsfor Exchange, if these values are not within tolerance, Jetstress will report the test

as failed.

1. DB IOPS Target: Is the Achieved Transactional I/O per Second in the test

report higher than the Total Database Required IOPS / Server predicted in the

Mailbox Role Calculator?

2. Is the I/O Database Reads Average Latency in the test report <20ms?

3. Is the I/O Log Writes Average Latency in the test report <10ms?

DB IOPS Target

DBRead

Latency

LOGWrite

Latency

Action

PASS PASS PASS Test successful

FAIL PASS PASS The test is failing to meet the IOPS target, but the latencyvalues are good. Increase the thread count by 1 and re-test.Use sluggishsessions to fine tune if necessary.

PASS FAIL FAIL At least one database has recorded latency over threshold.If the latency values are very close to limits increase sluggishsessions by 1, if both target IOPS and latency values aremuch higher decrease the thread count.

PASS PASS FAIL

PASS FAIL PASS

FAIL FAIL FAIL If the test shows that Achieved IOPS is below the designtarget AND the test latency values are above limits thestorage solution is unable to meet the requirements. At thisstage it is necessary to re-evaluate the storage design andbegin troubleshooting the physical deployment to determinethe correct remediation.

FAIL FAIL PASS

FAIL PASS FAIL

Table 8 - Quick results analysis table








1 Appendix B – Configuring sluggishsessions

If it is not possible to achieve the right IOPS value by modifying the thread count it

becomes necessary to modify the sluggishsessions value within the

JetstressConfig.xml file.

The sluggishsessions value adds a pause between each task. This allows a level of

fine-tuning over the workload dispatched by Jetstress.

As sluggishsessions is increased the achieved IOPS value decreases.

To change the value, open the JetstressConfig.xml file and look for the default

configuration option

<SluggishSessions>1</SluggishSessions>

Modify the value, save the configuration file and then re-start Jetstress.






1.1 Lab test data for SluggishSessions

The following test data is aimed as a guide to help configuring thread count and

sluggishsessions. It is provided as a guide to the target IOPS values that Jetstress

will attempt to reach for a given configuration. As latency increases the achievedIOPS values will diminish. This data was taken from a single database running on a

Raid10 LUN with 24x2.5” 10k SAS HDD’s.

DB IOPSLOG

IOPS

Total

IOPS

(excl

BDM)

BDM

IOPS

Avg DB

Read

Latency

Avg DB

Write

Latency

DB IOPSLOG

IOPS

Total

IOPS

(excl

BDM)

BDM

IOPS

Avg DB

Read

Latency

Avg DB

Write

Latency

DB IOPSLOG

IOPS

Total

IOPS

(excl

BDM)

BDM

IOPS

Avg DB

Read

Latency

Avg DB

Write

Latency

1 120 42 162 29 7.5 2.2 89 31 120 28 7.2 4.4 64 20 84 28 7.2 5

2 214 70 284 30 7.7 1.1 146 48 194 30 7.3 1.4 106 34.4 140.4 28 7.9 3.2

3 490 105 595 30 5.9 1.1 218 67 285 30 7.8 1.3 154 48 202 29 8.1 1.7

4 679 136 815 30 6.1 1.5 289 84 373 30 7.9 1 235 69 304 30 7.6 1.2

5 811 153 964 30 6.2 1.1 534 110 644 30 6.3 1.3 289 84 373 30 7.8 1.2

6 947 165 1112 30 6.3 1.2 631 125 756 30 6.4 1.2 486 99 585 30 6.3 1.4

7 1117 189 1306 30 6.4 1.2 740 146 886 30 6.5 1.1 544 109 653 30 6.3 1.2

8 1220 195 1415 30 6.6 1 835 156 991 30 6.5 1.2 610 118 728 30 6.5 1.4

9 1305 204 1509 30 7 1 944 160 1104 30 6.5 1 723 136 859 30 6.3 1.1

SluggishSessions=1 SluggishSessions=2 SluggishSessions=3

Thread

Count

Figure 5 - Effects of SluggishSessions on IOPS workload

2 Appendix C - Running a Jetstress Test with

JetstressCmd.exe

Both JetstressWin.exe and JetstressCmd.exe use the common Jetstress core library

files, which means you will have comparable test results with the same XMLconfiguration file. We recommend that you use JetstressWin.exe to create new test

scenarios, and JetstressCmd.exe to open and run the test scenarios by using the

/config command-line option. You can also see all the other available options by

using the /? (help) command-line option.

Action Argument Example of Use Description

help /? The help for the command-

line program

Config /c JetstressConfig.xml Open a configuration file

Generate /g Generate a sample XML

configuration file

TimeOut /TimeOut 2H0M0S Test Duration. Default is 2

hours.






Output /output c:\output Path for test output.

Default is the current

directory.

DBPath /dbpath m:\sg1\mdb

/dbpath n:\sg2\mdb

Database paths for each

storage group

LogPath /log x:\sg1\log y:\sg2\log Log path for each storage

group

PctCapacity /pctcapacity 100 Specify capacity

percentage

Throughput /throughput 100 Specify throughput

percentage

Threads /threads Suppress auto tuning and

specify thread count

DoNotRunDBPerform

ance

Do not run background

database maintenance

during performance/stress

test

RunDBPerformance Run background database

maintenance during soft

recovery test

New /new Create new databases

Open /open Open existing databases

Bak /bak Restore backup database

Recovery /recovery Run soft recovery test

Streaming Run streaming backup test

Transaction Run transaction

performance test

VerifyCheckSum Run database checksums

Page iiExchange Jetstress 2010, Field Guide, Version 1.0.0.6Final





3 Common Issues

3.1 Log or Data Volumes cannot be overlapped

This issue can arise when using Exchange Server 2007 ESE files and you attempt tostore multiple LOG streams on the same logical disk volume. Although this is a

valid Exchange server deployment, Jetstress uses the logical disk counters to record

I/O latency values for each Database and Log stream. If multiple Databases and

Logs are using the same logical disk then Jetstress is unable to isolate the

performance counters required.

To work around this issue, edit the JetstressConfig.XML file and change

<useEseCounters> to be true.

<UseEseCounters>true</UseEseCounters>

Save the XML File and attempt to re-configure the test with overlapped files.

Jetstress will now allow the configuration since it has been configured to use the I/O

performance data from ESE rather than the Operating System Logical Disk

Counters.

3.2 Troubleshooting Jetstress

While using Jetstress, you may encounter some known issues with Jetstress. This

section provides possible causes, and the recommended solutions.

3.2.1 Jetstress cannot attach to or create a database

Event log error that may display: Error -1023

• Possible cause: The path of the database or log files is incorrect.

• Solution: Ensure that the paths and file names are correct.

Event log error that may display: Error -1032

• Possible cause: Permissions are insufficient to access the .edb file or the log

files.

• Solution: Verify that permissions are sufficient for the account under which

Jetstress is running. Jetstress requires read/write permission to the directories

it is using.

Event log error that may display: Error -550 (0)

• Possible cause: The last time Jetstress was run, it was ended uncleanly. This

caused the log files to become unsynchronized with the database.

• Solution: Delete the Jetstress database (*.edb), log files (*.log), and check file

(*.chk), and re-create the Jetstress database. You can also use Eseutil.exe

with the /r switch to resynchronize the logs and database.

Event log error that may display: Error -1022Page iii

Exchange Jetstress 2010, Field Guide, Version 1.0.0.6Final





• Possible cause: The failure is caused by circular logging by Jetstress.

• Solution: Check the log drive for the log file name that is identified in the

event log. Delete that log file and all the log files that have a higher number

in the file name. Then, run Eseutil.exe /r to recover Jetstress.edb. When the

database is in a good state, delete all the log files in the log directory, and

rerun Jetstress.

1.1.1 Error loading Performance Monitor counters

JetstressWin.exe relies on performance counters to monitor the system.

JetstressWin.exe requires the ESE database counters to be installed.

• Cause: When the counters are not loaded correctly, you may see exception

errors related to performance counters.

•

Solution: To reload the counters, exit from JetstressWin.exe. Locate thedirectory where JetstressWin.exe was installed and verify that eseperf.dll,

eseperf.hxx, and eseperf.ini files exist in the directory. In a command shell

window, type the command unlodctr ESE and then click Enter. This will

unregister the ESE Database performance counters. Start JetstressWin.exe

and allow it to reload the performance counters.

1.1.1 Database Performance counters not working after using

Jetstress

Uninstalling Jetstress causes performance counter issues.

• Cause: In certain cases when Exchange was installed before Jetstress,uninstalling Jetstress could result in unstable Database Performance

counters.

• Solution: To restore access to database counters after Jetstress has been

removed, go to the Exchange installation directory (eg. D:\*\Exchange

Server\V14\bin) from a command prompt.

○ From the command prompt, run the command unlodctr ESE to ensure

that the database performance counter registration information has

been completely removed from the registry.

○ Run the command lodctr esperf.ini, which should correctly register the

database counters. The database performance counters will be available the next time.

1.1.1 Unable to tune for the parameters

This error indicates that Jetstress could not find appropriate parameters that could

be used to run a performance or stress test at the desired level of I/O load.






• Cause: This can be caused by several factors. The most common reason is

that the storage subsystem has multiple hosts attached to it, and those hosts

are competing for common resources during the tuning process.

• Solution: When you are running in a scenario such as this, you can run

Jetstress on a single host with tuning enabled to generate the appropriate

load parameters, and then rerun the test on the other hosts with the

Suppress Tuning option enabled and the tuning parameters entered manually

from the results of the first test.

jetstress field guide v1.0.0.6 by microsoft

Documents