32-bit virtual memory constraints in windows: an update
DESCRIPTION
32-bit Virtual Memory Constraints in Windows: an Update. Mark Friedman Demand Technology 1020 Eighth Avenue South, Suite 6, Naples, FL 34102 USA phone: (239) 261-8945 fax: (239) 261-5456 e-mail: [email protected] http://www.demandtech.com. Outline. Topics Virtual Memory concepts - PowerPoint PPT PresentationTRANSCRIPT
Demand Technology Software, Inc.
32-bit Virtual Memory Constraints in
Windows: an UpdateMark FriedmanDemand Technology
1020 Eighth Avenue South, Suite 6, Naples, FL 34102 USA
phone: (239) 261-8945 fax: (239) 261-5456
e-mail: [email protected]://www.demandtech.com
2Demand Technology, Inc. 32-bit VM Constraints
Outline Topics
Virtual Memory concepts
Virtual Memory constraints in 32-bit Windows
/3 GB option Physical Address
Extension (PAE) Address Windowing
Extensions (AWE) 64-bit relief
Vir t ualaddress
space
V ir t ualaddress
space
P hysical M em ory
Page Tables
Intel x860100010011 0000011010 000001101100
IA -32 Virtual address translat ion
Page tableSelector
Page tableentry
byte offset
Control Register 3
Page Directory
Private
System
10000
7F F E F F F F
A 4000000
C1000000
E 0000000
F F F FF F F F
PTEs/ cache extension
System data cache
Paged system pool
(offset of per process Page Table)
Page Tables Physical pages
Paging file
(invalid)
80000000
C0000000
4Demand Technology, Inc. 32-bit VM Constraints
Intel x86 in PAE mode
(ptr) Page Directory Index Page Table Index Byte Offset
Virtual Address Translation (PAE)
PDE
PageDirectory
PageDirectoryPointers
PDE
PDE
PDE
PDE
64-bit PTE
PageTable
RAM(Physical Memory)
CR3
31 29 20 011
5Demand Technology, Inc. 32-bit VM Constraints
Virtual Memory Manager
4 GB virtual per process address space Lower 2 GB - Private
Page 0 reserved Code pages Heap
Upper 2 GB - System System code Shared dlls System cache
System
User
0
x'1 0000'16
No Access
x'8000 0000'16
x'ffff 0000'16
System Code
Device Driver Code
Nonpaged Pool
Paged Pool
PTEs
File Cache
6Demand Technology, Inc. 32-bit VM Constraints
Virtual Memory Manager
All address spaces share the virtual addresses from the same upper 2 GB of the System area
All virtual addresses in the lower 2 GB User area are unique
Shared memory in the System range can be used for IPC
User
UserUser
System
User
0
x'1 0000'16
x'8000 0000'16
x'ffff 0000'16
System Code
Device Driver Code
Nonpaged Pool
Paged Pool
PTEs
File Cache
7Demand Technology, Inc. 32-bit VM Constraints
Private
System
10000
7F F E F F F F
A 4000000
C1000000
E 0000000
F F F F F F F F
PTEs/ cache extension
System data cache
Paged system pool
80000000
C0000000
Virtual Memory map
Paged Memory mapped
files Shared DLLs
File Server IIS (html, jpg, gif)
Page tables; kernel thread stacks
nonPaged: I/O buffers used by
device drivers TCP Session data
8Demand Technology, Inc. 32-bit VM Constraints
Virtual Memory Manager
Commit Limit A hard limit on the number of
Virtual Memory pages that the system will allocate
Sizeof (RAM) + paging file(s) Memory allocations start to fail
as % Committed Bytes in Use 100%
For smaller systems (< 2 GB RAM), Committed Bytes tends to become exhausted prior to running out of VM
9Demand Technology, Inc. 32-bit VM Constraints
Monitoring Committed Bytes
10Demand Technology, Inc. 32-bit VM Constraints
Monitoring Process\(*)Virtual Bytes
11Demand Technology, Inc. 32-bit VM Constraints
Virtual Memory Manager
Commit Limit But page files are
extendible!
Should you or shouldn’t you?
12Demand Technology, Inc. 32-bit VM Constraints
Paging file extension
13Demand Technology, Inc. 32-bit VM Constraints
System memory pools
No hard limits on the sizes of the system pools
One reserved area of virtual memory can fill up faster than the others;
Area Max Size
file cache 960 MB
PTEs 660 MB
Paged pool 512 MB
nonPaged Pools
256 MB
14Demand Technology, Inc. 32-bit VM Constraints
Monitoring the System Pools
15Demand Technology, Inc. 32-bit VM Constraints
Private
System
10000
7F F E F F F F
A 4000000
C1000000
E 0000000
F F F F F F F F
PTEs/ cache extension
System data cache
Paged system pool
80000000
C0000000
Virtual Memory map
Virtual memory map is dynamic, subject to adjustment by the OS
One reserved area of virtual memory can fill up faster than the others
When any one area fills, VMM routines can empty the file cache to acquire more vm for the paged/nonpaged pools
Paged Pool
nonPaged PoolPTEs
Cache
16Demand Technology, Inc. 32-bit VM Constraints
Default Paged and Nonpaged pool size can be overridden:
17Demand Technology, Inc. 32-bit VM Constraints
Default Paged and Nonpaged pool sizes can be overridden:
Dynamic adjustment of system virtual memory when one pool is exhausted.
Workloads most vulnerable: Terminal server with an excess of processes &
threads Servers booted with the /3 GB option
NonPagedPoolSize, NonPagedPoolSize, and SystemPages can be set explicitly. Max pool size is only available from the debugger!
Virtual Memory constraints in Terminal Server
1
10
100
1000
10000
100000
1000000
0 50 100 150 200 250
Terminal Server Sessions (x86)
Free
Sys
tem
PTE
s
0
10
20
30
40
50
60
% P
roce
ssor
Bus
yFree System PTEs
% Processor Time
19Demand Technology, Inc. 32-bit VM Constraints
Extended Virtual Addressing
Boot.ini /3 GB switch Virtual storage constraint relief for some
applications Squeezes the System VM into 1 GB
Physical Address Extension (PAE) Supports 37-bit real addresses on Xeon processors
Address Windowing Extensions (AWE) Permits processes to address real memory > 4 GB
20Demand Technology, Inc. 32-bit VM Constraints
Virtual Memory Manager
Boot.ini /3 GB switch Virtual storage
constraint relief for some server applications
SQL Server Exchange 2000 Etc.
/userva=SizeInMB subparameter, where SizeinMB can be any value between 2048 and 3072
System
User
0
x'1 0000'16No Access
x'c000 0000'16
x'ffff 0000'16
System Code
Device Driver Code
Nonpaged Pool
Paged Pool
PTEs
File Cache
/userva
21Demand Technology, Inc. 32-bit VM Constraints
Exchange 2003 memory tuning Exchange Server 2003
recommendations/3 GB /Userva = 3030SystemPages = -1HeapDeCommitFreeBlockThreshold = 0x00040000msExchESEParamCacheSizeMax = 311296 (1.2 GB)
See KB #815372
22Demand Technology, Inc. 32-bit VM Constraints
Virtual Memory constraints
Try the /3 GB switch 1 GB system area is not enough virtual memory for some
applications Possible shortage of Free System Page Table Entries Possible shortage of Nonpaged Pool
– Where Session data from TCP connections is stored
Due to fragmentation, it may not be possible to failover a 2 GB+ private address space (e.g., SQL Server, MS Exchange database – store.exe) using Microsoft Cluster Server (MCS)
During address space recovery on the standby node, the entire virtual memory allocation is acquired at one time
23Demand Technology, Inc. 32-bit VM Constraints
Physical Address Extension
PAE is supported on most recent Intel servers Physical addresses are 37-bits Up to 128 GB of RAM can be installed
New format Page Table Entries (PTEs) are 64-bits
Process address spaces are still limited to 4 GB. How can they exploit 64 GB of RAM? Expand sideways, using multiple address spaces Large Memory Enabled (LME) device drivers Address Windowing Extensions (AWE)
Boot.ini /pae switch Limitation: system addresses can be no higher than 16 GB
24Demand Technology, Inc. 32-bit VM Constraints
Intel x86 in PAE mode
(ptr) Page Directory Index Page Table Index Byte Offset
Virtual Address Translation (PAE)
PDE
PageDirectory
PageDirectoryPointers
PDE
PDE
PDE
PDE
64-bit PTE
PageTable
RAM(Physical Memory)
CR3
31 29 20 011
25Demand Technology, Inc. 32-bit VM Constraints
Physical Address Extension
Expand sideways, using multiple address spaces e.g., MS SQL
Server
SQL Server instance #1
0
3 GB
4 GB(System Range)
SQL Server instance #2
0
3 GB
4 GB(System Range)
SQL Server instance #3
0
3 GB
4 GB(System Range)
System Range
0
12 GB
RAM VM
26Demand Technology, Inc. 32-bit VM Constraints
IIS 6.0 (expand sideways)
User
Kernel
HTTP
TCP
IP
Network Interface
HTTP Kernel Mode Driver
(http.sys)
HTTP Response Cache
(Physical Memory)
LSSAS
WindowsAuthentication
SSL
Inetinfo
IIS Administration
FTP
Metabase
SMTP
NNTP
SVCHOST
W3SVC
W3wpW3wp
Aspnet_filter.dll
Mscoree.dll
Aspnet_isapi.dll
<code-behind>.dll
Default.aspx
Application Pool
W3wpW3wp
Aspnet_filter.dll
Mscoree.dll
Aspnet_isapi.dll
<code-behind>.dll
Default.aspx
Application Pool
27Demand Technology, Inc. 32-bit VM Constraints
Configuring IIS 6.0 Application Pools
28Demand Technology, Inc. 32-bit VM Constraints
Address Windowing Extensions (AWE)
API that allows processes to address real memory locations outside their 4 GB virtual addressing range Create and manage memory Overlays Used in conjunction with PAE
AllocateUserPhysicalPages
MapUserPhysicalPages
Virtual Alloc
29Demand Technology, Inc. 32-bit VM Constraints
AWESystem
User
0
x'1 0000'16No Access
x'c000 0000'16
x'ffff 0000'16
System Code
Device Driver Code
Nonpaged Pool
Paged Pool
PTEs
File Cache
x'8000 0000'16
AWE Region (Nonpaged memory)
Physical Address Range(Reserved)
Physical Address Range(Reserved)
Physical Address Range(Reserved)
MapUserPhysicalPage
Physical Address Range(Reserved)
Note: Frequent unmapping and remapping of Physical Memory blocks is expensive!
30Demand Technology, Inc. 32-bit VM Constraints
AWE support
MS SQL Server 2000: /3 GB, PAE, & AWE Multiple process instances awe enabled = 1 AWE activity counters in the Buffer Manager object
Oracle: /3 GB, PAE, & AWE AWE_WINDOW_MEMORY Also Set max server Oraclememory
SAS: /3 GB, PAE Work library can be placed in PAE memory
31Demand Technology, Inc. 32-bit VM Constraints
System Virtual Memory tuning
HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management settings
NonpagedPoolSize Defaults based on the size of RAM, up to 256 MB.
Can be set explicitly.-1 extends NonpagedPoolSize to its maximum
PagedPoolSize Defaults based on the size of RAM, Up to 650 MB.
Can be set explicitly; -1 extends PagedPoolSize to its maximum
SystemPages Defaults based on the size of RAM. Can be set explicitly; -1 extends SystemPages to its maximum
LargeSystemCache 960 MB Zero value allocates minimum sized file cache
32Demand Technology, Inc. 32-bit VM Constraints
Default Paged and Nonpaged pool size can be overridden:
33Demand Technology, Inc. 32-bit VM Constraints
Default Paged and Nonpaged pool size can be overridden (Win 2003):
System Pages = x’ffff ffff’16 or -1 Maximizes the number of PTEs that can be built Potentially useful with /PAE or Terminal Services
PagedPoolSize = x’ffff ffff’16 or -1 Allows the OS maximum flexibility to determine the size of
the Paged Pool Or PagedPoolSize and NonPagedPoolSize can be set
explicitly
LargeSystemCache controls the maximum virtual address range of the file cache 512 MB – 1 GB
34Demand Technology, Inc. 32-bit VM Constraints
!vm Debugger command runs the Page Frame Number (PFN) database
Available Pages: 4920 ( 19680 Kb)ResAvail Pages: 358 ( 1432 Kb)Locked IO Pages: 251 ( 1004 Kb)Free System PTEs: 204387 ( 817548 Kb)Free NP PTEs: 28645 ( 114580 Kb)Free Special NP: 0 ( 0 Kb)Modified Pages: 596 ( 2384 Kb)Modified PF Pages: 660 ( 2640 Kb)NonPagedPool Usage: 2750 ( 11000 Kb)NonPagedPool Max: 33768 ( 135072 Kb)PagedPool 0 Usage: 3544 ( 14176 Kb)PagedPool 1 Usage: 1359 ( 5436 Kb)PagedPool 2 Usage: 1340 ( 5360 Kb)PagedPool Usage: 6243 ( 24972 Kb)PagedPool Maximum: 138240 ( 552960 Kb)Shared Commit: 6842 ( 27368 Kb)Special Pool: 0 ( 0 Kb)Shared Process: 3688 ( 14752 Kb)PagedPool Commit: 6398 ( 25592 Kb)Driver Commit: 1630 ( 6520 Kb)Committed pages: 211846 ( 847384 Kb)Commit limit: 320257 ( 1281028 Kb)
35Demand Technology, Inc. 32-bit VM Constraints
!poolused Debugger command accountsfor all system pool allocations(plus pooltags.txt documentation)
lkd> !poolused 2 Sorting by NonPaged Pool Consumed
Pool Used: NonPaged Paged Tag Allocs Used Allocs Used LSwi 1 2576384 0 0 NV 287 1379120 14 55272 File 2983 504920 0 0 MmCm 16 435248 0 0 LSwr 128 406528 0 0 Devi 267 377472 0 0 Thre 452 296512 0 0 PcNw 12 278880 0 0 Irp 669 222304 0 0
36Demand Technology, Inc. 32-bit VM Constraints
Poolmon utility (from DDK)
37Demand Technology, Inc. 32-bit VM Constraints
64-bit Virtual Memory Relief
Component 64-bit 32-bit
Process Virtual Address Space 16 TB 4 GB
User Address Space 8 TB 2 GB
System Paged Pool 128 GB 512 MB*
System Non-paged Pool 128 GB 256 MB*
System cache 1 TB 1 GB
System PTEs 128 GB 660 MB*
Hyperspace 8 GB 4 MB
Paging File size 512 TB 4 GB
38Demand Technology, Inc. 32-bit VM Constraints
64-bit Relief
Windows supports two flavors of 64-bit hardware IA-64 (Itanium) AMD64 and Intel x64 (aka EMT)
Runs 32-bit applications with almost no performance penalty
32-bit applications can allocate their entire 4 GB virtual memory range
39Demand Technology, Inc. 32-bit VM Constraints
Questions
?