vital signs: performance monitoring windows server module 6: processor microsoft confidential

Vital Signs: Performance Monitoring Windows

Server

Module 6:Processor

Microsoft Confidential

Conditions and Terms of Use

This training package is proprietary and confidential, and is intended only for uses described in the training materials. Content and software is provided to you under a Non-Disclosure Agreement and cannot be distributed. Copying or disclosing all or any portion of the content and/or software included in such packages is strictly prohibited.

The contents of this package are for informational and training purposes only and are provided "as is" without warranty of any kind, whether express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, and non-infringement.

Training package content, including URLs and other Internet Web site references, is subject to change without notice. Because Microsoft must respond to changing market conditions, the content should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. Unless otherwise noted, the companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

For more information, see Use of Microsoft Copyrighted Content athttp://www.microsoft.com/about/legal/permissions/

Microsoft®, Internet Explorer®, and Windows® are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Other Microsoft products mentioned herein may be either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks are property of their respective owners.

Copyright and Trademarks © 2011 Microsoft Corporation. All rights reserved.


http://www.microsoft.com/about/legal/permissions/

http://www.microsoft.com/about/legal/permissions/

Students: How to View this Presentation

Switch to Notes Page view Click View on the ribbon and select Notes Page Use page up or page down to navigate Zoom in or out as needed

Most slides will have supporting text that you can view now or after the delivery

Add notes to your copy of the presentation if you want to.

You take the presentation files home with you.


Module 6: Processor

Section 1: Measuring processor usageSection 2: Privileged modeSection 3: User modeSection 4: Power management


Section 1: Measuring processor usage

Processes and threadsIdle processPrivileged mode and User mode% Processor Time formulaThresholds for % Processor TimeThe \Process\% Processor Time counterReal-time toolsProcessor affinityNUMA computer systems


Processes and threads

Process: A container for a set of resources that are used when running an instance of the programThread: An entity within a process that the operating system schedules to run


Idle process

All CPU clock cycles are consumed by the operating systemWhen no runnable thread exists on a CPU, the operating system dispatches the per-CPU idle threadIt shows up as System Idle Process, but it’s not a real process\Process(Idle)\% Processor Time measures the total idle time of all processors


Privileged mode and User mode

A thread can be in one of two modes:Privileged mode: Synonymous with Kernel mode. Code is running on a processor in which all memory is accessible and all CPU instructions can be issued. Operating-system code (such as system services and device drivers) runs in Kernel mode. • Measured by \Processor(*)\% Privileged Time

User mode: The non-privileged processor mode, which applications run in. A limited set of interfaces is available in this mode, and the access to system data is limited. This is the mode where application code is executed.• Measured by \Processor(*)\% User Time

(continued)Microsoft Confidential

Privileged mode and User mode (continued)

For each of these activities, which processor mode would you expect them to be in most often? (Kernel, User, or both)1. Backup software2. Database server executing complex calculations3. Database server scanning large amounts of file system data4. 3D games5. Browsing the Internet


Privileged mode and User mode (answers)

For each of these activities, which processor mode would you expect them to be in most often? (Kernel, User, or both)1. Backup software: Kernel2. Database server executing complex calculations: User3. Database server scanning large amounts of file system data: Kernel4. 3D games: Both5. Browsing the Internet: User


% Processor Time formula

% Processor Time = % User Time + % Privileged Time

If any two values are known, the third can be calculatedExample:• If % Processor Time is 50 and % Privileged Time is 5,

% User Time must be 45


% Processor Time formula (continued)

% Processor Time = % User Time + % Privileged Time

% Processor Time: 56

% User Time: 46

% Privileged Time: 10


Thresholds for % Processor Time

Threshold EffectLess than 50%

Informational Normal usage

> 50%WarningMonitor the processor time

> 80%

CriticalThe system might seem sluggish

Common next steps:

Identify the processes that are consuming high CPUIdentify which is higher: privileged time or user time

Monitor the \Processor(*)\% Processor Time counter for all processor instances


The \Process\% Processor Time counter

Each process can potentially consume up to 100% CPU times the number of processors, if it has enough active threadsA single threaded application can consume only up to 100% CPUExample (Demo 004_Cpustres):• 1 thread at maximum activity = \Process(cpustres)\% Processor

Time of 100• With 8 logical processors, any process can consume up to

800% CPU, assuming it has at least 8 active threads


The \Process\% Processor Time counter (continued)

In this example, \Process(w3wp#4)\% Processor Time is consuming up to 367% CPUThis server has 4 logical processors, so any process can potentially consume up to 400% CPU

This process consumes up to 367% CPU


Real-time tools: Using Task Manager

To measure overall processor usage, use the Performance tab of Task Manager.The green line indicates overall usage, and the red line indicates privileged/kernel mode usage.To show privileged mode usage (red line), click View, Show Kernel Times. Not enabled by default.Generally, high CPU usage on average = poor performance.

To show privileged mode CPU


Real-time tools: Resource Monitor

Provides live processor usage informationResource Monitor is built in to Windows Vista, Windows Server 2008, and laterMore detailed information is in Windows 7 and Windows Server 2008 R2Can be launched from the Performance tab in Task Manager

Launch Resource Monitor

Process selection displayed in orange


Real-time tools: Resource Monitor (continued)

Processes

Handles

Modules

Graphical view

of processor activity


Processor affinity

Processor instancesProcessor queue lengthProcessor affinity and thread priority


Processor instances

A thread can run on only one processor at a time, or threads might have affinity to a single processorSingle-threaded applications cannot take advantage of multiple processorsThe _Total instance is an average, so look at all processor instances

_Total shows 50% usage CPU 0 is at 90%


Processor queue length

\System\Processor Queue Length is the total number of threads in all of the processor queues• Unlike the disk counters, this counter shows ready threads only, not

threads that are running

Thread Dispatcher

Processor

Queue

Processor

Queue

Each processor has its own thread queue, which is sorted by priorityA high processor queue length might not indicate a problem, but it might contribute to high context switching (discussed later)Virtual guest computers commonly report high processor queue lengths incorrectly


Processor affinity and thread priority

You can use Task Manager to adjust the base thread priority for a processA process can be assigned affinity to one or more processorsThread priority and processor affinity can both greatly affect the performance of an application

Assign base priority Assign processor affinity


NUMA computer systems

Some computers have a Non-Uniform Memory Access (NUMA) hardware architecture

Processors are grouped together in smaller units called nodes

Each node has its own processors and memory, and is connected to the larger system through a cache-coherent interchange bus

Although any processor in any node can access all of the memory, node-local memory is much faster to access

For efficiency and speed, Windows Server 2008 R2 and Windows 7 try to keep a thread on the same processor or node

RAM

Bus

P P P P

RAM

Bus

P P P P

Cache-coherent interchange

bus

Node 0 Node 1Slow

Node-local memory is

fast


Section 1 review

1. What is the difference between privileged mode and user mode?

2. If % Processor Time is 75 and % Privileged Time is 10, what is the value of % User Time?

3. What tools can you use to analyze the CPU load in real time?

4. Why is it more efficient to keep a thread on the same processor?


Section 1 review (answers)

1. Privileged mode processes operating system code, and user mode processes application code.

2. If % Processor Time is 75 and % Privileged Time is 10, % User Time must be 65.

3. You can use Task Manager and Resource Monitor to analyze the CPU load in real time.

4. It is often less efficient to frequently move a thread between processors than to leave it on the same processor, because of loading of the thread stack and accessing memory on other NUMA nodes.


Section 2: Privileged mode

Identifying Privileged mode CPU usageThresholds for % Privileged Time

Interrupts Deferred procedure calls (DPCs)Thresholds for % Interrupt Time and % DPC Time

Context switches and transitionsThresholds for Context Switches/sec

\Process\% Privileged Time

Unexpected high Privileged mode usageUsing xperf for high Privileged CPU


Identifying Privileged mode CPU usage

\Processor(*)\% Privileged Time is the percentage of time that an individual CPU (or all CPUs) has run in Kernel mode during a specified intervalA high amount of % Privileged Time may warrant more investigation into kernel-related processor usageCommon causes:• Poorly written drivers or faulty hardware• High amounts of driver-related tasks, such as file and network I/O• High amounts of context switching


Thresholds for % Privileged Time

Threshold Effect

Less than 30% Informational Normal usage

> 30%WarningMonitor I/O and thread counts

> 50%

CriticalHigh processor usage is likely, with too much time servicing hardware interrupts

Common next steps:

Investigate disk and network related I/OInvestigate % DPC Time and % Interrupt Time

Investigate Context Switches/sec

The following thresholds apply to \Processor(*)\% Privileged Time


Interrupts

Asynchronous events (that can occur at any time) that are unrelated to what the processor is currently running• The active thread on a processor is interrupted• Interrupts are generated primarily by hardware I/O devices, processor

clocks, or timers, and they can be enabled or disabled• \Processor(*)\% Interrupt Time is the time the processor spends

receiving and servicing hardware interrupts during sample intervals


Deferred procedure calls (DPCs)

Routines that perform most of the work involved in handling a device interrupt after the interrupt service routine (ISR) runs• The DPC routine executes at a lower interrupt request level (IRQL)

than standard interrupts, in order to avoid unnecessary blocking• \Processor(*)\% DPC Time is the percentage of time that the

processor spends receiving and servicing DPCs during the sample interval


Thresholds for % Interrupt Time and % DPC Time

Threshold Effect

Less than 10% Informational Normal usage

> 10%WarningMonitor I/O and thread counts

> 20%

CriticalHigh processor usage likely with too much time servicing hardware interrupts

Common next steps:

Investigate I/O that is related to disk and networkUpdate device driversRemove faulty hardwareProfile the kernel to determine CPU usage by the driver

The following thresholds apply to % Interrupt Time or % DPC Time only if there is high % Privileged Time


Context switches and transitions

Context switch: The procedure of saving the volatile machine state that is associated with a running thread, loading another thread’s volatile state, and starting the new thread’s execution• \System\Context Switches/sec is the combined rate at which all

processors on the computer are switched from one thread to another• High context switching can be caused by high disk or network I/O, too

many active threads, and other causesTransition: The act of a thread moving to or from user mode or kernel mode• \System\System Calls/sec is the combined rate of calls to operating-

system service routines by all processes running on the computer• Transitions to service I/Os commonly result in a context switch

Quantum: The time a thread is allowed to run before another thread at the same priority level is given a turn to run


Thresholds for Context Switches/sec

Threshold EffectLess than 5,000

Informational Normal usage

> 2,500 x NumOfProcs

WarningMonitor I/O and thread counts

5,000 x NumOfProcs or > 20,000

CriticalHigh processor usage is likely, with too much time switching between threads

Common next steps:

Investigate I/O related to disk and networkIdentify and reduce active threads by using \Process(*)\Thread Count and \Process(*)\% Privileged Time

Adjust for best performance of background services—longer quantums (more clock cycles before switching)

The following thresholds apply only if there is high % Privileged Time


\Process\% Privileged Time

\Process(*)\% Privileged Time is the percentage of elapsed time that the threads of a process spend executing code in privileged (kernel) modeHelps identify which processes are consuming the most privileged timeThe system process has only kernel-based threads and is exclusive to % Privileged Time

Processes using privileged time


Unexpected high Privileged mode usage

Backup software running during business hours can cause unexpected privileged mode CPU usage

30% privileged time from CPU 1 and 6

Backup software has 30% privileged time for more than 30 minutes

Backup software


Using xperf for high Privileged CPU

Microsoft xperf is a tool based on Event Tracing for Windows (ETW) that can identify processes and drivers using the most Privileged mode CPUPart of the Windows Server 2008 Performance Tool KitFor more information, go to the Windows Performance Analysis Developer Centerhttp://msdn.microsoft.com/performance/default.aspx


http://msdn.microsoft.com/performance/default.aspx




Section 2 review

1. What counter can you use to identify kernel-mode CPU usage?

2. What can cause high privileged mode CPU usage?3. What can cause frequent context switching?



1. \Processor(*)\% Privileged Time can help identify kernel-mode CPU usage

2. High privileged mode CPU usage can be caused by poorly written drivers, high amounts of I/O, hardware usage, and high context switching

3. High context switching can be caused by high disk or network I/O as well as by large numbers of active threads


Lab 6a: The case of the over-privileged processor

GoalsGoals

ScenarioScenario

Use the Processor and System counter objects to learn more about the conditionFocus on privileged-mode CPU usageUse the Process object counters to identify processes consuming the most CPU

Use the Processor and System counter objects to learn more about the conditionFocus on privileged-mode CPU usageUse the Process object counters to identify processes consuming the most CPU

Microsoft BizTalk Server is running with very high processor usage with little to no load.Microsoft BizTalk Server is running with very high processor usage with little to no load.


Section 3: User mode

Identifying User mode CPU usageAnalysis tools• Process Explorer for Thread CPU• Microsoft xperf on process profiling• Visual Studio Command Line Profiler


Identifying User mode CPU usage

\Processor(*)\% User Time is the percentage of elapsed time the processor spends in the user modeA high amount of \Processor(*)\% User Time might warrant more investigation into process application code \Process(*)\% User Time is the percentage of elapsed time that the threads of a process spend executing code in User modeCommon causes:• Poorly written or heavily used application code• High amounts calculations, object sorting, and any other tasks that

use non-privileged resources


Process Explorer for thread CPU

Process Explorer (SysInternals) can show the modules and functions (symbols needed) of threads that are actively using CPUShows only native and .NET Framework v4.0 or later modules and functionshttp://live.sysinternals.com/procexp.exe

Tools like Process Explorer can show active thread modules and functions

(symbols needed)


http://live.sysinternals.com/procexp.exe

Microsoft xperf on process profiling

Microsoft xperf shows the CPUs, processes, threads, and functions of high CPU usageShows only native (not .NET) modules and functions

xperf can show modules and functions consuming

the most CPU


Visual Studio Command Line Profiler

This command-line tool can profile native and .NET applicationsRequires the data to be captured while CLRProfiling mode is enabledProvides a report showing the functions found on the call stack the most often“How To” article about the VSProfilerhttp://perftesting.codeplex.com/wikipage?title=How%20To:%20Identify%20a%20Disk%20Performance%20Bottleneck%20Using%20SPA&referringTitle=How%20Tos

Shows the native and .NET functions


http://perftesting.codeplex.com/wikipage?title=How%20To:%20Identify%20a%20Disk%20Performance%20Bottleneck%20Using%20SPA&referringTitle=How%20Tos




Section 3 review

1. What counter can you use to identify user-mode CPU usage?

2. What can cause high user-mode CPU usage?3. What tools can you use to identify modules and functions

found most often on active thread stacks?



1. \Processor(*)\% User Time can be used to identify high user-mode CPU conditions

2. High user-mode CPU conditions can be caused by application code

3. Tools such as Process Explorer, Microsoft xperf, and the Visual Studio Profiler can identify modules and function found most often on active thread stacks


Lab 6b: The case of the very busy web server

GoalsGoals

ScenarioScenario

Use the Processor and System counter objects to learn more about the conditionFocus on user-mode CPU usageUse the Process counters to identify processes consuming the most CPU

Use the Processor and System counter objects to learn more about the conditionFocus on user-mode CPU usageUse the Process counters to identify processes consuming the most CPU

A web server has very poor response times, and users are complainingA web server has very poor response times, and users are complaining


Section 4: Power management

\Processor\% Cx Time

Processor frequency and core parkingResource Monitor and processor frequency


Processor\% Cx Time

\Processor(*)\% Cx Time is the percentage of time that the processor spends in the Cx low-power idle state% C1 Time, % C2 Time, and % C3 Time are subsets of the total processor idle timeThese counters are in all versions of Windows NTC1: Enables the processor to maintain its entire context and quickly return to the running stateC2: A lower power and higher exit latency state than C1C3: A lower power and higher exit latency state than C2


Processor frequency and core parking

Windows Vista, Windows Server 2008, and later include counters for processor clock speed• \Processor Information(*)\% of Maximum Frequency is the

percentage of the maximum frequency of the current processor• Helps determine the load and power efficiency of the processor

Windows 7 and Windows Server 2008 R2 uses core parking to consolidate processing onto the fewest number of possible processor cores, and it suspends inactive processor cores• \Processor Information(*)\Parking Status represents whether a

processor is parked or noto 0 = Not parkedo 1 = Parked


Processor frequency and core parking (continued)

In Windows Server 2008 R2, the processor power state changes based on load

Low load

Core parked

High load

Not core parked


Resource Monitor and processor frequency

Microsoft Resource Monitor (built into Windows Vista, Windows Server 2008, and later) shows the average percentage of maximum frequency of the processors

Processor frequency changes based on load (blue line)


Section 4 review

1. If % C3 Time has a high value, what might this indicate?2. Why is adjusting the clock frequency or power state of idle

processors helpful?



1. If % C3 Time has a high value, it indicates that the processor is in a low power state

2. Adjusting the clock frequency or power state of idle processors reduces the electrical power that the server uses, which saves money without sacrificing performance


References

Windows Internals, 5th edition, by Mark Russinovich and David A. Solomonhttp://technet.microsoft.com/en-us/sysinternals/bb963901.aspx

Taking Your Server’s Pulsehttp://technet.microsoft.com/en-us/magazine/cc718984.aspx

Windows Sysinternals Administator’s Referencehttp://technet.microsoft.com/sysinternals/hh290819 “Key OS Performance Metrics_v7.xps” • In the Supporting Documents folder with your classroom material.


http://technet.microsoft.com/en-us/sysinternals/bb963901.aspx

http://technet.microsoft.com/en-us/sysinternals/bb963901.aspx

http://technet.microsoft.com/en-us/magazine/cc718984.aspx

http://technet.microsoft.com/en-us/magazine/cc718984.aspx

http://technet.microsoft.com/sysinternals/hh290819

http://technet.microsoft.com/sysinternals/hh290819

Questions?


vital signs: performance monitoring windows server module 6: processor microsoft confidential

Documents

microsoft products

microsoft confidential2

registered trademarks

training package content

content andor software

intellectual property

training purposes

training materials