32-bit virtual memory constraints in windows: an update

Demand Technology Software, Inc.

32-bit Virtual Memory Constraints in

Windows: an UpdateMark FriedmanDemand Technology

1020 Eighth Avenue South, Suite 6, Naples, FL 34102 USA

phone: (239) 261-8945 fax: (239) 261-5456

e-mail: markf@demandtech.comhttp://www.demandtech.com

2Demand Technology, Inc. 32-bit VM Constraints

Outline Topics

Virtual Memory concepts

Virtual Memory constraints in 32-bit Windows

/3 GB option Physical Address

Extension (PAE) Address Windowing

Extensions (AWE) 64-bit relief

Vir t ualaddress

space

V ir t ualaddress

space

P hysical M em ory

Page Tables

Intel x860100010011 0000011010 000001101100

IA -32 Virtual address translat ion

Page tableSelector

Page tableentry

byte offset

Control Register 3

Page Directory

Private

System

10000

7F F E F F F F

A 4000000

C1000000

E 0000000

F F F FF F F F

PTEs/ cache extension

System data cache

Paged system pool

(offset of per process Page Table)

Page Tables Physical pages

Paging file

(invalid)

80000000

C0000000

4Demand Technology, Inc. 32-bit VM Constraints

Intel x86 in PAE mode

(ptr) Page Directory Index Page Table Index Byte Offset

Virtual Address Translation (PAE)

PDE

PageDirectory

PageDirectoryPointers

PDE

PDE

PDE

PDE

64-bit PTE

PageTable

RAM(Physical Memory)

CR3

31 29 20 011

5Demand Technology, Inc. 32-bit VM Constraints

Virtual Memory Manager

4 GB virtual per process address space Lower 2 GB - Private

Page 0 reserved Code pages Heap

Upper 2 GB - System System code Shared dlls System cache

System

User

0

x'1 0000'16

No Access

x'8000 0000'16

x'ffff 0000'16

System Code

Device Driver Code

Nonpaged Pool

Paged Pool

PTEs

File Cache

6Demand Technology, Inc. 32-bit VM Constraints

Virtual Memory Manager

All address spaces share the virtual addresses from the same upper 2 GB of the System area

All virtual addresses in the lower 2 GB User area are unique

Shared memory in the System range can be used for IPC

User

UserUser

System

User

0

x'1 0000'16

x'8000 0000'16

x'ffff 0000'16

System Code

Device Driver Code

Nonpaged Pool

Paged Pool

PTEs

File Cache

7Demand Technology, Inc. 32-bit VM Constraints

Private

System

10000

7F F E F F F F

A 4000000

C1000000

E 0000000

F F F F F F F F

PTEs/ cache extension

System data cache

Paged system pool

80000000

C0000000

Virtual Memory map

Paged Memory mapped

files Shared DLLs

File Server IIS (html, jpg, gif)

Page tables; kernel thread stacks

nonPaged: I/O buffers used by

device drivers TCP Session data

8Demand Technology, Inc. 32-bit VM Constraints

Virtual Memory Manager

Commit Limit A hard limit on the number of

Virtual Memory pages that the system will allocate

Sizeof (RAM) + paging file(s) Memory allocations start to fail

as % Committed Bytes in Use 100%

For smaller systems (< 2 GB RAM), Committed Bytes tends to become exhausted prior to running out of VM

9Demand Technology, Inc. 32-bit VM Constraints

Monitoring Committed Bytes

10Demand Technology, Inc. 32-bit VM Constraints

Monitoring Process\(*)Virtual Bytes

11Demand Technology, Inc. 32-bit VM Constraints

Virtual Memory Manager

Commit Limit But page files are

extendible!

Should you or shouldn’t you?

12Demand Technology, Inc. 32-bit VM Constraints

Paging file extension

13Demand Technology, Inc. 32-bit VM Constraints

System memory pools

No hard limits on the sizes of the system pools

One reserved area of virtual memory can fill up faster than the others;

Area Max Size

file cache 960 MB

PTEs 660 MB

Paged pool 512 MB

nonPaged Pools

256 MB

14Demand Technology, Inc. 32-bit VM Constraints

Monitoring the System Pools

15Demand Technology, Inc. 32-bit VM Constraints

Private

System

10000

7F F E F F F F

A 4000000

C1000000

E 0000000

F F F F F F F F

PTEs/ cache extension

System data cache

Paged system pool

80000000

C0000000

Virtual Memory map

Virtual memory map is dynamic, subject to adjustment by the OS

One reserved area of virtual memory can fill up faster than the others

When any one area fills, VMM routines can empty the file cache to acquire more vm for the paged/nonpaged pools

Paged Pool

nonPaged PoolPTEs

Cache

16Demand Technology, Inc. 32-bit VM Constraints

Default Paged and Nonpaged pool size can be overridden:

17Demand Technology, Inc. 32-bit VM Constraints

Default Paged and Nonpaged pool sizes can be overridden:

Dynamic adjustment of system virtual memory when one pool is exhausted.

Workloads most vulnerable: Terminal server with an excess of processes &

threads Servers booted with the /3 GB option

NonPagedPoolSize, NonPagedPoolSize, and SystemPages can be set explicitly. Max pool size is only available from the debugger!

Virtual Memory constraints in Terminal Server

1

10

100

1000

10000

100000

1000000

0 50 100 150 200 250

Terminal Server Sessions (x86)

Free

Sys

tem

PTE

s

0

10

20

30

40

50

60

% P

roce

ssor

Bus

yFree System PTEs

% Processor Time

19Demand Technology, Inc. 32-bit VM Constraints

Extended Virtual Addressing

Boot.ini /3 GB switch Virtual storage constraint relief for some

applications Squeezes the System VM into 1 GB

Physical Address Extension (PAE) Supports 37-bit real addresses on Xeon processors

Address Windowing Extensions (AWE) Permits processes to address real memory > 4 GB

20Demand Technology, Inc. 32-bit VM Constraints

Virtual Memory Manager

Boot.ini /3 GB switch Virtual storage

constraint relief for some server applications

SQL Server Exchange 2000 Etc.

/userva=SizeInMB subparameter, where SizeinMB can be any value between 2048 and 3072

System

User

0

x'1 0000'16No Access

x'c000 0000'16

x'ffff 0000'16

System Code

Device Driver Code

Nonpaged Pool

Paged Pool

PTEs

File Cache

/userva

21Demand Technology, Inc. 32-bit VM Constraints

Exchange 2003 memory tuning Exchange Server 2003

recommendations/3 GB /Userva = 3030SystemPages = -1HeapDeCommitFreeBlockThreshold = 0x00040000msExchESEParamCacheSizeMax = 311296 (1.2 GB)

See KB #815372

22Demand Technology, Inc. 32-bit VM Constraints

Virtual Memory constraints

Try the /3 GB switch 1 GB system area is not enough virtual memory for some

applications Possible shortage of Free System Page Table Entries Possible shortage of Nonpaged Pool

– Where Session data from TCP connections is stored

Due to fragmentation, it may not be possible to failover a 2 GB+ private address space (e.g., SQL Server, MS Exchange database – store.exe) using Microsoft Cluster Server (MCS)

During address space recovery on the standby node, the entire virtual memory allocation is acquired at one time

23Demand Technology, Inc. 32-bit VM Constraints

Physical Address Extension

PAE is supported on most recent Intel servers Physical addresses are 37-bits Up to 128 GB of RAM can be installed

New format Page Table Entries (PTEs) are 64-bits

Process address spaces are still limited to 4 GB. How can they exploit 64 GB of RAM? Expand sideways, using multiple address spaces Large Memory Enabled (LME) device drivers Address Windowing Extensions (AWE)

Boot.ini /pae switch Limitation: system addresses can be no higher than 16 GB

24Demand Technology, Inc. 32-bit VM Constraints

Intel x86 in PAE mode

(ptr) Page Directory Index Page Table Index Byte Offset

Virtual Address Translation (PAE)

PDE

PageDirectory

PageDirectoryPointers

PDE

PDE

PDE

PDE

64-bit PTE

PageTable

RAM(Physical Memory)

CR3

31 29 20 011

25Demand Technology, Inc. 32-bit VM Constraints

Physical Address Extension

Expand sideways, using multiple address spaces e.g., MS SQL

Server

SQL Server instance #1

0

3 GB

4 GB(System Range)

SQL Server instance #2

0

3 GB

4 GB(System Range)

SQL Server instance #3

0

3 GB

4 GB(System Range)

System Range

0

12 GB

RAM VM

26Demand Technology, Inc. 32-bit VM Constraints

IIS 6.0 (expand sideways)

User

Kernel

HTTP

TCP

IP

Network Interface

HTTP Kernel Mode Driver

(http.sys)

HTTP Response Cache

(Physical Memory)

LSSAS

WindowsAuthentication

SSL

Inetinfo

IIS Administration

FTP

Metabase

SMTP

NNTP

SVCHOST

W3SVC

W3wpW3wp

Aspnet_filter.dll

Mscoree.dll

Aspnet_isapi.dll

<code-behind>.dll

Default.aspx

Application Pool

W3wpW3wp

Aspnet_filter.dll

Mscoree.dll

Aspnet_isapi.dll

<code-behind>.dll

Default.aspx

Application Pool

27Demand Technology, Inc. 32-bit VM Constraints

Configuring IIS 6.0 Application Pools

28Demand Technology, Inc. 32-bit VM Constraints

Address Windowing Extensions (AWE)

API that allows processes to address real memory locations outside their 4 GB virtual addressing range Create and manage memory Overlays Used in conjunction with PAE

AllocateUserPhysicalPages

MapUserPhysicalPages

Virtual Alloc

29Demand Technology, Inc. 32-bit VM Constraints

AWESystem

User

0

x'1 0000'16No Access

x'c000 0000'16

x'ffff 0000'16

System Code

Device Driver Code

Nonpaged Pool

Paged Pool

PTEs

File Cache

x'8000 0000'16

AWE Region (Nonpaged memory)

Physical Address Range(Reserved)

Physical Address Range(Reserved)

Physical Address Range(Reserved)

MapUserPhysicalPage

Physical Address Range(Reserved)

Note: Frequent unmapping and remapping of Physical Memory blocks is expensive!

30Demand Technology, Inc. 32-bit VM Constraints

AWE support

MS SQL Server 2000: /3 GB, PAE, & AWE Multiple process instances awe enabled = 1 AWE activity counters in the Buffer Manager object

Oracle: /3 GB, PAE, & AWE AWE_WINDOW_MEMORY Also Set max server Oraclememory

SAS: /3 GB, PAE Work library can be placed in PAE memory

31Demand Technology, Inc. 32-bit VM Constraints

System Virtual Memory tuning

HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management settings

NonpagedPoolSize Defaults based on the size of RAM, up to 256 MB.

Can be set explicitly.-1 extends NonpagedPoolSize to its maximum

PagedPoolSize Defaults based on the size of RAM, Up to 650 MB.

Can be set explicitly; -1 extends PagedPoolSize to its maximum

SystemPages Defaults based on the size of RAM. Can be set explicitly; -1 extends SystemPages to its maximum

LargeSystemCache 960 MB Zero value allocates minimum sized file cache

32Demand Technology, Inc. 32-bit VM Constraints

Default Paged and Nonpaged pool size can be overridden:

33Demand Technology, Inc. 32-bit VM Constraints

Default Paged and Nonpaged pool size can be overridden (Win 2003):

System Pages = x’ffff ffff’16 or -1 Maximizes the number of PTEs that can be built Potentially useful with /PAE or Terminal Services

PagedPoolSize = x’ffff ffff’16 or -1 Allows the OS maximum flexibility to determine the size of

the Paged Pool Or PagedPoolSize and NonPagedPoolSize can be set

explicitly

LargeSystemCache controls the maximum virtual address range of the file cache 512 MB – 1 GB

34Demand Technology, Inc. 32-bit VM Constraints

!vm Debugger command runs the Page Frame Number (PFN) database

Available Pages: 4920 ( 19680 Kb)ResAvail Pages: 358 ( 1432 Kb)Locked IO Pages: 251 ( 1004 Kb)Free System PTEs: 204387 ( 817548 Kb)Free NP PTEs: 28645 ( 114580 Kb)Free Special NP: 0 ( 0 Kb)Modified Pages: 596 ( 2384 Kb)Modified PF Pages: 660 ( 2640 Kb)NonPagedPool Usage: 2750 ( 11000 Kb)NonPagedPool Max: 33768 ( 135072 Kb)PagedPool 0 Usage: 3544 ( 14176 Kb)PagedPool 1 Usage: 1359 ( 5436 Kb)PagedPool 2 Usage: 1340 ( 5360 Kb)PagedPool Usage: 6243 ( 24972 Kb)PagedPool Maximum: 138240 ( 552960 Kb)Shared Commit: 6842 ( 27368 Kb)Special Pool: 0 ( 0 Kb)Shared Process: 3688 ( 14752 Kb)PagedPool Commit: 6398 ( 25592 Kb)Driver Commit: 1630 ( 6520 Kb)Committed pages: 211846 ( 847384 Kb)Commit limit: 320257 ( 1281028 Kb)

35Demand Technology, Inc. 32-bit VM Constraints

!poolused Debugger command accountsfor all system pool allocations(plus pooltags.txt documentation)

lkd> !poolused 2 Sorting by NonPaged Pool Consumed

Pool Used: NonPaged Paged Tag Allocs Used Allocs Used LSwi 1 2576384 0 0 NV 287 1379120 14 55272 File 2983 504920 0 0 MmCm 16 435248 0 0 LSwr 128 406528 0 0 Devi 267 377472 0 0 Thre 452 296512 0 0 PcNw 12 278880 0 0 Irp 669 222304 0 0

36Demand Technology, Inc. 32-bit VM Constraints

Poolmon utility (from DDK)

37Demand Technology, Inc. 32-bit VM Constraints

64-bit Virtual Memory Relief

Component 64-bit 32-bit

Process Virtual Address Space 16 TB 4 GB

User Address Space 8 TB 2 GB

System Paged Pool 128 GB 512 MB*

System Non-paged Pool 128 GB 256 MB*

System cache 1 TB 1 GB

System PTEs 128 GB 660 MB*

Hyperspace 8 GB 4 MB

Paging File size 512 TB 4 GB

38Demand Technology, Inc. 32-bit VM Constraints

64-bit Relief

Windows supports two flavors of 64-bit hardware IA-64 (Itanium) AMD64 and Intel x64 (aka EMT)

Runs 32-bit applications with almost no performance penalty

32-bit applications can allocate their entire 4 GB virtual memory range

39Demand Technology, Inc. 32-bit VM Constraints

Questions

?