vmworld09 - lab 4 - vsphere 4 - performance and troubleshooting - v8

vSphere 4: Performance and Troubleshooting

Contents i

Contents Lab Summary ............................................................................................................... 1

Section 1: Storage Performance Bottlenecks ............................................................... 1 Section 2: Managing VM Sizes In Aggressively Consolidated Environments ............... 1 Section 3: Understanding the Monitor .......................................................................... 1

Section 1: Storage Performance Bottlenecks ................................................................. 2 Background Information ............................................................................................... 2 Activities ....................................................................................................................... 2

Part 1: Storage Performance Troubleshooting Tools ...................................................... 2 Part 2: Using and Interpreting vscsiStats ........................................................................ 8 Part 3: Improving Storage Performance ....................................................................... 12

Section 2: Managing VM Sizes In Aggressively Consolidated Environments ................. 13 Background Information ............................................................................................. 13 Activities ..................................................................................................................... 13

Part 1: Starting esxtop, Virtual Machines and Generating Load .................................... 13 Part 2: Over commit CPU and observe impact on performance and esxtop.................. 20 Part 3: Over commit RAM and observe impact on performance and esxtop. ................ 25 Part 4: Power down and clean up ................................................................................. 32

Section 3: Understanding the Monitor ......................................................................... 34 Background Information ............................................................................................. 34 Activities ..................................................................................................................... 35

First Run: Paravirtualization (VMI) ................................................................................ 35 Second Run: Binary Translation (BT) ........................................................................... 36 Third RUN: Hardware Assist with SW MMU ................................................................. 38 Fourth Run: Hardware Assist with Hardware MMU ....................................................... 39 Optional Investigation: How to detect monitor mode. .................................................... 41

Appendix A: Important Names ..................................................................................... 42 Appendix B: Virtual Machine, Host, and Storage Configuration .................................... 43

Contents ii

Server Configurations ................................................................................................. 43 Virtual Machine Configurations ................................................................................... 43

Appendix C: References ............................................................................................. 45 From Section 2 ........................................................................................................... 45 From Section 3 ........................................................................................................... 45

Instructors: Michael Burnett, Systems Engineer, VMware (Canada)

Todd Day, Systems Engineer, VMware (Connecticut)

Scott Drummonds, Technical Marketing, VMware (Palo Alto)

Jon Hall, Technical Training, VMware (Palo Alto)

Henrik Moenster, System Engineer, VMware (Denmark)

Chris Romano, Systems Engineer, VMware (Virginia)

Mike Szafranski, System Engineer, VMware (Maryland)

Stein Wilhelmsen, Systems Engineer, VMware (Norway)

Lab 4: vSphere 4 – Performance and Troubleshooting

VMworld 2009 1

Lab Summary This lab covers performance monitoring, troubleshooting, and optimization in VMware vSphere 4. Key concepts of performance will be discussed in the presentation, with this manual leading you through hands-on activities to demonstrate the concepts. The content of this lab was created by VMware’s performance team and is being delivered by many of the company’s most performance-savvy customer-facing elements, most of who have worked on versions of this lab in previous years.

In addition to the presentation and hands-on elements of this lab, please feel free to ask questions about any performance issues that you feel are important. Challenge us in areas where you need more information. We should be able to answer most of your questions immediately. Difficult questions will require more time but will inform our continued efforts to improve the performance of our product, the quality of the performance management tools, and the research that drives the content for this lab and other performance material.

Thanks for your attendance and enjoy the lab.

Section 1: Storage Performance Bottlenecks We find that sub-optimal storage is the cause of the majority of performance problems in a virtual environment. vSphere Administrators have access to tools that can identify performance problems and sometimes correct bottlenecks without the support of a storage administrator.

This section begins with high IO VMs running IOmeter, a common storage performance test tool, which is capable of driving more storage traffic than the array can support. You will see how the vSphere client, esxtop, and vscsiStats can be used to characterize the storage problem. Finally, you will see the impact of optimal Datastore configuration on the performance of the virtual machines.

Section 2: Managing VM Sizes In Aggressively Consolidated Environments In lightly consolidated environments, slight inefficiencies in virtual machine sizing go unnoticed by application users. But for vSphere administrators that are aggressively consolidated virtual machines to minimize costs, right-sizing VMs is a requirement to meet the needs of demanding users.

In this section you will investigate environments with multiple VMs that are sub-optimally configured with respect to their applications’ needs. You will use the vSphere client and esxtop to identify problems with memory performance, kernel scheduler stress, and wasted resources. You will see the impact of corrections to the VMs’ memory sizes, virtual CPU (vCPU) count, and resource allocations.

Section 3: Understanding the Monitor ESX is comprised of two discrete components that serve different functions in a virtualized environment. The instructor will describe these two components, the monitor and VMkernel, and show the impact of the various modes of the monitor on virtualized workloads.


VMworld 2009 2

Section 1: Storage Performance Bottlenecks Background Information When working with a storage intensive virtualized environment, storage array configuration is critical for optimal performance. When identifying performance issues, sub-optimal configurations are the most likely cause of a storage bottleneck. vSphere Administrators have access to tools that can identify performance problems and sometimes correct bottlenecks without the support of a storage administrator. This lab will introduce and compare the various available tools for troubleshooting storage performance.

This lab begins with an introduction to the various performance tools available to the administrator using a virtual machine generating large amounts of storage traffic using IOmeter, a common storage performance test tool. You will see how tools inside the guest like Perfmon, as well as tools external to the guest including the vSphere client, esxtop, and vscsiStats, can be used to characterize the storage problem. You will then generate a performance issue by creating heavy storage load from two virtual machines sharing a poorly configured Datastore. To solve the issue, you will migrate one of these virtual machines to an alternative, high performance Datastore. The tools introduced to you will then be used to compare the performance of the virtual machine disks running on the sub-optimal Datastore with the virtual machine disks running on the high-performance Datastore. This will show the performance impact to a virtual machine with both the standard SCSI driver and the new high performance PVSCSI driver. You will also see how this operation relieves congestion on the sub-optimal Datastore, allowing both virtual machines to perform better.

Activities

Part 1: Storage Performance Troubleshooting Tools To demonstrate the various troubleshooting tools available to the vSphere Administrator, you will generate a disk load on a virtual machine and record the output using each of the tools available to you.

To start the virtual machine and generate disk load

1. Open the vSphere Client and connect to your vCenter Server

2. Power on your win-iometer-1 virtual machine.


VMworld 2009 3

3. Open a console to this virtual machine, log in (user: administrator, pass: vmware123) and launch IOmeter by double-clicking the iometer-D.icf

file. This will generate disk load on drive D.

4. Highlight the win-iometer-1 manager. Expand the manager by clicking the plus sign, then confirm that the worker shows the D: Drive as selected (Note: The D: drive should not have a red slash through it. Alert a lab staff member if this occurs).

5. Click the Results Display tab to monitor the results.


VMworld 2009 4

6. Click the green start flag in IOmeter to start the test. This test will generate storage activity on the first VM’s D: drive.

7. The test will prompt you to save the results as a CSV file. Accept the default save file, which will put results on your desktop. You may be prompted to overwrite an existing file. If so, allow this action.

//////////

8. Allow the test to run for 1-2 minutes. Record the Write MBs and Average Write Response Time values.

9. Stop the IOmeter test for the D Drive and close IOmeter.

10. Repeat the process for the E: drive using iometer-E.icf file. The E: drive has been configured to use the VMware PVSCSI driver included with

ESX 4.0. Record the Write MBs and Average Write Response Time values for this drive.


VMworld 2009 5

11. Open Perfmon by clicking on the perfmon.msc file:

12. View the output for the virtual machine, then record the values for each machine below. How do these values compare with the values recorded in IOmeter?

13. Both IOmeter and Perfmon are running in the guest. As a result, both should show approximately the same performance data.

14. Using the vSphere Client, select the VMware ESX Server host that these virtual machines are running on and click the Performance tab, then click the Advanced button.

15. Click the Chart Options link. You will create a storage chart for use in this lab. Select the Disk Real Time option under Chart Options.

16. In the Objects section, click None to clear the objects, then select the ESX Host, and the two naa disks indicated on your datasheet.

17. In the Counters section, click None to clear the counters, then select Write rate and Write latency.

18. Click Save Chart Settings, and name the chart Storage Chart 1. Click Apply, then OK.


VMworld 2009 6

19. Select the pop-up chart icon to display the performance chart in its own window. The resulting graph should be similar to the one below:

20. The naa device in the graph that shows activity is the disk used by the win-iometer-1 E Drive. Monitor and record the Average value for Write rate and Write latency from the chart output.

21. Using putty, open a session to your ESX host, using root and vmware123 for the username and password:


VMworld 2009 7

22. Type in esxtop. esxtop has been customized to display disk metrics specific to this lab. To see the individual metrics for a Datastore, you must expand, or maximize the display. Identify the vmhba# with high activity (if more than one vmhba# has high activity, this is due to multipathing, and either vmhba# may be chosen). This is the adapter for your virtual machine. Expand the adapter using the e command, followed by the appropriate vmhba#.

23. The CID number should now be displayed. Identify the CID (controller ID) with activity, then type a, followed by vmhba#:# using the controller

ID.

24. Finally, locate the TID (target ID) with activity, then type t, followed by vmhba#:#:# using the target ID. The output should look similar to the image below:

25. Record the MBWRTN/s (MB written per second) and DAVG/wr (Average Device latency per write) values for the virtual machine.

26. Press q to Quit esxtop. Remain in putty. The vSphere Client performance graphs and esxtop both display performance data relative to the ESX Host. This data is likely different from the data recorded in the guest. To obtain further information, we will use vscsiStats.


VMworld 2009 8

Part 2: Using and Interpreting vscsiStats vscsiStats collects and reports counters on storage activity, specifically activity at the virtual SCSI device level in the kernel. This allows information to be gathered on specific virtual disks. The following section will demonstrate how to use vscsiStats to gather this information and then how the resulting data can be interpreted.

To start vscsiStats and gather statistics about your virtual machine’s virtual disks

1. You should still be connected to your ESX Host with putty. If not, reconnect now.

2. Determine the worldGroupID (WIDs) of your virtual machine. This is needed to collect data specific to the two virtual machines you are currently using in this lab. To do this, run the following command:

cd /usr/lib/vmware/bin (press Enter)

./vscsiStats –l (case sensitive, lower case L)

3. The output should list the WIDs of each of your virtual machines in an output similar to the following:

Virtual Machine worldGroupID: 4716, Virtual Machine Display Name: win-iometer-1 {

Virtual SCSI Disk handleID: 8249



}

4. Record the WIDs of your win-iometer-1 virtual machine. 5. Now execute the vscsiStats command using the syntax below:

./vscsiStats –s –w xxxx


VMworld 2009 9

Where xxxx is the WID for the win-iometer-1 virtual machine. The output should look like the following:

vscsiStats: Starting Vscsi stats collection for worldGroup 4716, handleID 8251

Success.

6. View the seekDistance metrics (the number of blocks between successive reads) for this virtual machine by entering:

./vscsiStats -w xxxx -p seekDistance

7. View the latency metrics for this virtual machine by entering:

./vscsiStats -w xxxx -p latency

8. Identify the Virtual SCSI Disk handleID with the largest count number. This disk is the disk being used by IOmeter. The information about this

disk can be captured to a .csv file for use with a tool like Excel. Several metrics are collected, you will obtain the latency statistics. The command to output latency statistics to a file called latency.csv is:

./vscsiStats -w xxxx -p latency –c > /root/latency.csv

Where xxxx is the WID for win-iometer-1.

9. Copy the file to your desktop using winscp. To do this, open winscp and connect to your ESX Host , using your ESX Host name, as shown

below:


VMworld 2009 10

10. Drag the latency.csv file from the /root directory on your ESX Host to your desktop. Note: You must change the directory on the left side to

the Desktop folder.

11. Open the e2Spreadsheet application:


VMworld 2009 11

12. Click Import to import the latency.csv file from the desktop:

13. Drag the frame line for columns A and B to see the mean virtual machine write latency. Locate the handleID identified in step 8.

14. The resulting data shows the latency for reads, writes and total IOs. Record the write latency as shown below, converting to milliseconds (ex.

1542=15.42):


VMworld 2009 12

Part 3: Improving Storage Performance To demonstrate the benefits of an optimally configured Datastore, we have created a high performance Datastore to use as a comparison. The virtual machine used in the previous section has data disks in a sub-optimal Datastore. You will power the virtual machines on and create storage load with IOmeter. This will allow you to compare the performance of the two virtual machines.

1. Open the console to win-iometer-1 and close perfmon and IOmeter. Re-launch IOmeter using the iometer-All.icf file and start the test.

2. Power on win-iometer-2 (win-iometer-1 should still be powered on). Open a console to the virtual machine, launch IOmeter by double-clicking

the iometer-All.icf file and start the test.

3. After 2-3 minutes, record the Write MBs and Average Write Response Time values for the D drive and E drive for both virtual machines; win-iometer-1 and win-iometer2.

4. How is the performance for win-iometer-1 (sub-optimal Datastore) compared to win-iometer-2 (high performance Datastore)?

5. How does the performance of the D drive (worker (standard SCSI driver) compare with the performance of the E drives (VMware PVSCSI

driver)?

6. Return to the Storage Chart 1 and review the results:


VMworld 2009 13

Section 2: Managing VM Sizes In Aggressively Consolidated Environments

Background Information In under committed environments, it is often difficult to notice the impact of improperly configured virtual machines, particularly when the VM’s have been configured with excessive resources. In environments where CPU and RAM are overcommitted, it is possible to observe better performance with optimally configured virtual machines using appropriate resource configurations. We will be using 3 VM’s with identical Databases and configured with the same amount of RAM allocated to the databases. We will vary the amount of RAM and vCPUs allocated to the virtual machines.

In this lab, you will:

1. Observe the normal performance of 3 copies of an application:

a. 1vCPU 2GB RAM virtual machine running a 1GB RAM Oracle database

b. 4vCPU 4GB RAM virtual machine running a 1GB RAM Oracle database

c. 7vCPU 7GB RAM virtual machine running a 1GB RAM Oracle database

2. Over commit CPU and observe performance and esxtop statistics

3. Over commit RAM and observe performance and esxtop statistics

Activities

Part 1: Starting esxtop, Virtual Machines and Generating Load 1. Power down the two win-iometer virtual machines used in the previous lab.

2. Power on the database vApp. Verify that the database vApp is not in any Resource Pool.

3. Verify that the ram-hog vApp is powered off and is in the ram-limit Resource Pool.


VMworld 2009 14

4. Verify that the cpu-hog vApp is powered off and is in the cpu-limit Resource Pool.

5. Close any open putty sessions, then connect 2 instances of putty to your ESXxxb server (Use putty as described in Section 1).

6. Launch esxtop in each putty session.

7. In the first putty session, type c to show CPU information, then V to display only virtual machines.

3

2

4


VMworld 2009 15

8. In the second session, press m to show memory information, then V to display only virtual machines. We will use the second instance in Part 3.

9. Display Configuration settings of the database virtual machines and take note of Memory Size and CPU count.

a. Click on the database vApp.

b. Click on the Virtual Machines tab.

c. Right-Click on the column headings and add CPU Count, Memory Size and Guest Mem%. Keep Name, Host CPU – MHz, and Host Mem – MB. Deselect all other items.


VMworld 2009 16

d. Note Memory Size, CPU count and utilizations of the Virtual Machines

10. Generate load on the databases.

a. Open a Console to the db-client VM:

b. On the desktop there are 3 ‘apple’ icons. Double-Click the one on the left.


VMworld 2009 17

The icon launches the Swingbench load generator against the first Database virtual machine. We will launch 2 more instances in the next step. Note: There will be a pause before data shows up in the window.

c. Arrange the window for easy viewing of the Tx/Min (transactions per minute) and Benchmark Name fields.


VMworld 2009 18

d. Repeat this process for the other database VM’s. Use the other 2 ‘apple’ icons.

e. Arrange the Swingbench windows such that the Tx/Min and the Benchmark Name are visible for all 3 databases.


VMworld 2009 19

11. Open up the esxtop session configured for CPU and observe the baseline performance statistics as shown below:

a. Observe the following statistics (these are the statistics under normal operating conditions):

Are the VM’s saturating the CPU? (%RUN =100% * Number vCPU)?

Are the VM’s waiting for I/O (%WAIT - %IDLE)?

Are the VM’s waiting for CPU Resources (%RDY high and %MLMTD for limits)?

Are the VM’s waiting for RAM to swap (%SWPWT)?

Are the VM’s using all the CPU’s they have low (%CSTP)?


VMworld 2009 20

Part 2: Over commit CPU and observe impact on performance and esxtop.

1. Right-Click the cpu-limit Resource Pool and Edit Settings. Verify the CPU Limit value is as indicated below. This pool limits the CPU. The Reservations are not expandable meaning the VM’s will not use any resources outside the pool.


VMworld 2009 21

2. Richt-Click the cpu-hog vApp and Edit Settings, then click the Start Order tab. Make sure the ‘Startup Sequence Procedes When’ is 5 for each VM. Make sure all the VM’s are in Group 1. Once the startup sequence is started, there is no way to change the delay period. Adding VM’s to the vApp will reset the startup delay to a 2 minute default per VM.

3. Power On the cpu-hog vApp.


VMworld 2009 22

4. Observe the impact on performance using esxtop and Swingbench.

5. Using the esxtop session configured for CPU, look at the %RUN and %MLMTD. %MLMTD (Max Limited) is the amount of time the cpu is not running due to the Resource Pool Limit.

6. View the Swingbench consoles, look at the TX/Min and take note of any impact.


VMworld 2009 23

7. Drag the cpu-hog vApp out of the cpu-limit Resource Pool and drop onto the ESXxxb Host.

8. Return to the esxtop CPU session. Note the %Run, %RDY, and %CSTP (Co-Scheduling Stop Time). Are the Database virtual machines impacted asymmetrically? Is the server at 100% utilization?


VMworld 2009 24

9. View the Swingbench output and observe the impact of CPU over commit on database performance. Do the relative increases on %RDY and %CSTP roughly correspond to the decrease in performance?

10. Power off the cpu-hog vApp.


VMworld 2009 25

Part 3: Over commit RAM and observe impact on performance and esxtop.

1. Arrange the second esxtop window for easy viewing of memory parameters.

a. Note MEMSZ (Memory Size) and GRANT (Memory Granted). Memory Size is the amount of RAM set in the VM’s properties. Grant is the amount of RAM given to the VM by ESX.

b. Type ‘f’ to modify the fields. By pressing the letters on the keyboard, toggle on C, G, H, I, J. Press the space bar.


VMworld 2009 26

c. Arrange the two esxtop windows such that both RAM and CPU can be observed at the same time.

2. Go into vCenter, click on the db-1 virtual machine and then the Resource Allocation tab. Note the ballooning, and swapping levels. These are the same values as in esxtop. They take about 30 seconds to make it out to vCenter.


VMworld 2009 27

3. Verify that the ram-hog vApp is in the ram-limit Resource Pool.

4. Right click on the ram-hog vApp and choose Edit Settings.


VMworld 2009 28

5. Verify that the ram-hog vApp ‘Startup sequence proceeds when:’ is 5.

6. Power On the ram-hog vApp.


VMworld 2009 29

7. Return to the esxtop sessions. Note the SWTGT (swap target) and GRANT values in the memory session and the SWPWT (the amount of time the CPU is waiting for swap) value in the CPU session. Are these values the same for all the Database virtual machines?

8. Drag database vApp into the ram-limit Resource Pool.


VMworld 2009 30

9. Using Swingbench, observe the impact of over committed RAM on the database virtual machines.

10. Using esxtop, view the following statistics:

a. Are the database virtual machines impacted symmetrically with respect to the SWCUR (Swap current)?

b. Are the %SWPWT values (the amount of time the CPU is waiting for swapping) equal on all database virtual machines?

c. Do the virtual machines have the same amount of RAM taken by the Balloon Driver (MCTLSZ)? (For every page of active RAM 3 pages of idle RAM are taken)


VMworld 2009 31

11. Power Off the ram-hog vApp


VMworld 2009 32

12. Flip between esxtop and the db-client console. Observe the changes in the Ballooning and swapping for a few minutes and how they correlate to the performance of the Database virtual machines.

13. Optional – Ask instructor for win-iometer-2 step

Part 4: Power down and clean up

1. Power Off the ram-hog vApp


VMworld 2009 33

2. Shut Down the database vApp.

3. Verify that the virtual machines for Section 2 are all shut down.

4. Close any open putty sessions.


VMworld 2009 34

Section 3: Understanding the Monitor Background Information The Virtual Machine Monitor (VMM) is, together with VMkernel, responsible for execution of the virtual machine. Each virtual machine gets its own VMM instance. As a result, multiple monitor instances exist in a consolidated environment. The monitor can use either software or hardware techniques to virtualize the CPU and Memory Management Unit (MMU). Different Monitor Modes are:

Binary translation (BT): full software virtualization to handle privileged instructions and memory management. Paravirtualization via the Virtual Machine Interface (VMI): the guest runs a modified kernel that is aware of the presence of a hypervisor. Hardware assist (HW): virtualization assist techniques in the CPU are leveraged for privileged instructions. If the hardware contains a

virtualization-aware memory management unit (MMU) it will be used. Virtualization-aware MMUs, such as AMD-V/RVI and Intel VT/EPT, can handle certain guest operations as effectively as paravirtualization. Paravirtualization is a software approach that requires a special kernel in the guest operating system. VMware’s approach to paravirtualization, VMI, is available in several 32-bit Linux distributions. Although paravirtualization of the guest operating system is not considered a technology of the future, we will cover this for completeness of the lab. Different types of guest operating systems and applications put different loads on the ESX Server. With the correct monitor settings, vSphere administrators can optimize the performance of virtual machines. To simulate some different workloads in the VM’s, we will test the performance of a virtual machine with 4 microbenchmarks using different Monitor settings. The microbenchmarks do not represent real life application workloads, but they illustrate the different types of workloads a VMM has to handle. You will see that a flexible VMM is crucial to meet all kinds of workloads. Section 3 LAB activities include checking and altering VM settings, powering on the VM, running the benchmarks, taking notes of the performance numbers, and powering off the VM again. At the very end of this chapter, there is a short description of the different techniques and methods together with credits and links to more information. Please proceed to the next page to start the Section 3 LAB activities.


VMworld 2009 35

Activities To demonstrate the effect of the different monitor settings, you will generate workload in the virtual machine by using a script that starts the benchmarks.

First Run: Paravirtualization (VMI)

1. Right-Click the virtual machine lin-monitor, select Edit Settings, then select the Options tab.

2. Select Paravirtualization, and check that Support VMI Paravirtualization is selected.

3. Select CPU/MMU Virtualization and check that Use software for instruction set and MMU virtualization is selected.


VMworld 2009 36

4. Click OK, then power on the lin-monitor virtual machine. 5. Right-Click lin-monitor and select Open Console

6. Click inside the Console window and log in as root with password vmware123.

7. Start the benchmarks with the command: ./benchmark.

8. The benchmarking will run through four microbenchmarks: SUM, GETPPID, FORKWAIT and MEMSWEEP

9. Write down the four REAL time numbers in the VMI column in the table on the handed out data sheet in section 3, Understanding the Monitor.

10. When complete, power off the lin-monitor virtual machine, leaving the Console window open.

Second Run: Binary Translation (BT)

1. Right-Click the virtual machine lin-monitor, and select Edit Settings, then select the Options tab.

2. Select Paravirtualization, and check that Support VMI Paravirtualization is not selected.


VMworld 2009 37

3. Then, select CPU/MMU Virtualization and check that Use software for instruction set and MMU virtualization is selected.

4. Click OK, then power on the lin-monitor virtual machine.




8. Write down the four REAL time numbers in the BT column on the data sheet in section 3, Understanding the Monitor.

9. When completed, power off the lin-monitor virtual machine, leaving the Console window open.


VMworld 2009 38

Third RUN: Hardware Assist with SW MMU

1. Right-Click the virtual machine lin-monitor, and select Edit Settings, then select the Options Tab.


3. Then, select CPU/MMU Virtualization and check that Use Intel VT-x/AMD-V for instruction set virtualization and software for MMU virtualization is selected.


VMworld 2009 39

4. Click OK and power on the lin-monitor virtual machine.




8. Write down the four REAL time numbers in the VT column on the data sheet in section 3, Understanding the Monitor.

9. When completed, power of the lin-monitor virtual machine, leaving the Console window open.

Fourth Run: Hardware Assist with Hardware MMU

1. Right-Click the lin-monitor virtual machine, select Edit Settings, then select the Options tab.


3. Then, select CPU/MMU Virtualization and check that Use Intel VT-x/AMD-V for instruction set virtualization and Intel EPT/AMD RVI for MMU virtualization is selected.


VMworld 2009 40

4. Click OK, then power on the lin-monitor virtual machine.




8. Write down the four REAL time numbers in the EPT column on the data sheet in section 3, Understanding the Monitor.

9. When completed, power off the lin-monitor virtual machine.

This concludes the Section 3 LAB activities!


VMworld 2009 41

Optional Investigation: How to detect monitor mode. You can detect the monitor setting by looking at vmware.log file for the VM and looking for the string: MONITOR From the ESX console: [root@esx SLES 11]# grep MONITOR vmware.log Aug 11 22:12:42.255: vmx| MONITOR MODE: allowed modes : BT Aug 11 22:12:42.255: vmx| MONITOR MODE: user requested modes : BT Aug 11 22:12:42.257: vmx| MONITOR MODE: guestOS preferred modes: HWMMU BT HV Aug 11 22:12:42.257: vmx| MONITOR MODE: filtered list : BT If you do not have access to the ESX console or you are running ESXi, you can download vmware.log to your desktop using datastore browser or RCLI and use e.g. the Windows version of grep. Try and change the monitor setting, grep again and note the changes. You can detect if VMI is enabled by looking at vmware.log file for the VM. This does not tell you if VMI is in use, just that it is presented to the guest OS. Look for the string: vmi.present From the ESX console: [root@esx SLES 11]# grep vmi.present vmware.log Aug 11 22:12:42.228: vmx| DICT vmi.present = TRUE To check the status of VMI from within the VM, and to see if VMI is in use, look in dmesg of the guest OS: vmmtest:~ # dmesg | grep VMI VMI: Found VMware, Inc. Hypervisor OPROM, API version 3.0, ROM version 1.0 If VMI is not enabled, nothing will be returned. For more in depth information on VMM modes and techniques, consult the references mentioned in Appendix C – from section 4.


VMworld 2009 42

Appendix A: Important Names The following names occur throughout this manual.

Name Object

win-iometer-1 The first VM from section 1, storage performance analysis.

win-iometer-2 The second VM from section 1, storage performance analysis.

db-1 ‘Small’ Oracle database VM

db-2 ‘Medium’ Oracle database VM

db-3 ‘Large’ Oracle database VM

db-client Oracle Database Client – runs load generation

ram-hog(1-6) Diskless VM’s that boot from ISO image – consume RAM

cpu-hog(1-10) Diskless VM’s that boot from ISO image – consume CPU cycles

win-netperf-client-1 The first client VM (load-generating) in section 3, network bandwidth management.

win-netperf-client-2 The second client VM (load-generating) in section 3, network bandwidth management.

win-netpef-server The server VM (load receiving) in section 3, network bandwidth management.

lin-monitor The Linux VM used in section four to showcase the monitor’s impact on performance.


VMworld 2009 43

Appendix B: Virtual Machine, Host, and Storage Configuration Server Configurations

Server count: two per station. Six student stations share one NetApp array.

Memory per host: 48 GB

LUN configuration on storage:

o App01 and App02: Contain all OS and application VMDKs.

o VMFS01a: “slow” LUN.

o VMFS01b: “fast” LUN.

Virtual Machine Configurations Virtual Machine Operating System vCPU Count Memory Amount Virtual Storage

Configuration win-iometer-1 Windows Server 2003,

64-bit 1 500 MB C: 15 GB (OS)

D: 1 GB (LSI)

E: 1 GB (PV)

win-iometer-2 Windows Server 2003, 64-bit

1 500 MB C: 15 GB (OS)

D: 1 GB (LSI)

E: 1 GB (PV)

db-1 CENT 5.3, 64-bit 1 2 GB /dev/sda: 10 GB

/dev/sdb: 10 GB Oracle installation

/dev/sdc: 10 GB Database files

/dev/sde: 500 MB redo-logs


VMworld 2009 44

db-2 CENT5.3, 64-bit 4 4 GB /dev/sda: 10 GB




db-3 CENT5.3, 64-bit 7 7 GB /dev/sda: 10 GB




db-client CENT5.3, 64-bit 2 2 GB /dev/sda: 12 GB (OS + db files)

cpu-hog(1-10) DOS 1 256 MB No Disk

ram-hog(1-6) DOS 1 4 GB No Disk

lin-monitor SLES 11, 32-bit, with VMI enabled kernel

1 256 MB /dev/sda: 8 GB thin provisioned


VMworld 2009 45

Appendix C: References

From Section 2 Interpreting esxtop statistics: http://communities.vmware.com/docs/DOC-9279

Using esxtop: http://inet-gw.maphis.homeip.net/training/VMWare/VMWorld-2008/TA1440_Locked_Loaded.pdf

From Section 3

Software and Hardware Techniques for x86 Virtualization, Ole Agesen, VMware Inc., http://www.vmware.com/resources/techresources/10036

AP12 VMworld Europe 2008, Performance Aspects of x86 Virtualization, Ole Agesen, VMware Inc.

VMI, Alok Kataria, VMware.

vmworld09 - lab 4 - vsphere 4 - performance and troubleshooting - v8

Documents