xenapp load balancing
DESCRIPTION
TRANSCRIPT
Load Balancing and Intelligent Load BalancingLoad Balancing and Intelligent Load Balancing
Ivan LorenteEscalation EngineerIvan LorenteEscalation Engineer
2© 2007 Citrix Systems, Inc. — All rights reserved
Objectives
• Deep understanding of load balancing
• Understanding Intelligent load balancing
• Typical scenarios
• Troubleshooting techniques
3© 2007 Citrix Systems, Inc. — All rights reserved
Agenda
Load Balancing on an Application Delivery Scenario
Architecture
Where can it go wrong?
Troubleshooting
4© 2007 Citrix Systems, Inc. — All rights reserved
Load Balancing on an Application Delivery Scenario
Agenda item number 1
Architecture
Where can it go wrong?
Troubleshooting
Load Balancing on an Application Delivery Scenario
5© 2007 Citrix Systems, Inc. — All rights reserved
Load Balancing on an Application Delivery Scenario
Citrix® NetScaler®
Deliver WebApps
Citrix Presentation Server™
Deliver Windows Apps
Citrix Desktop Server™
Deliver Windows Desktops
Users AppsCitrix
EdgeSight™
Monitor Real-Time User Experience
Citrix WANScaler™
Accelerate Apps to Branch Offices
Citrix Access Gateway™
Enable Secure App Access
6© 2007 Citrix Systems, Inc. — All rights reserved
Load Balancing on an Application Delivery Scenario
Users Apps
PS Farm
7© 2007 Citrix Systems, Inc. — All rights reserved
Architecture
Agenda item number 1Load Balancing on an Application Delivery Scenario
Where can it go wrong?
Troubleshooting
Architecture
8© 2007 Citrix Systems, Inc. — All rights reserved
Architecture
• Data Collector Updates
• Understanding Bias
• Load Balancing during logon
• Performance Data Helper
• Citrix and the PDH
• Load balancing in the Data Store
9© 2007 Citrix Systems, Inc. — All rights reserved
Architecture
Data Collector DCDynamic Store DS
LmsSS.dllIMA Subsystem
LMS20Rules.dllSystem counters
MFRules.dllMetaFrame counters
PDH.dllPerformance Data Helper
10© 2007 Citrix Systems, Inc. — All rights reserved
Data Collector Updates
• Connection logon or logoff
• Change in the load evaluator +/- 500
• DC(DS) is updated every 30 s
• If nothing changes in 1 minute the DC(DS) sends 40 byte ping, to make sure they are alive.
• If nothing changes in 5 minutes the DC(DS) request a server load update.
11© 2007 Citrix Systems, Inc. — All rights reserved
Understanding Bias
• Definition of Bias in Science
Bias is defined as the difference between the measurement result and its unknown 'true value'
12© 2007 Citrix Systems, Inc. — All rights reserved
Understanding Bias
queryds /table:LMS_ServerLoadTable
5: Scheduling
6: Page Swaps
7: Disk Data I/O
8: Disk operations
9: IP Range
a: Application user load
b: Server User load
1: CPU Utilization
2: Context switches
3: Memory Usage
4: Page Faults
Bias = 10000/(40 USERS) = 250 DEC = fa HEX
Load = 3E8/FA HEX = 4 USERS
13© 2007 Citrix Systems, Inc. — All rights reserved
Load Balancing during logon
IMA
Ser
ver
BCtxnotif.dll
LMSSS.dll
ICA Client
Data Collector
Server A
Server B
MFSrvSS.dll
Dynamic Store
Ser
ver
A
WEVENT_LOGON
14© 2007 Citrix Systems, Inc. — All rights reserved
Performance Data Helper
• The performance data helper (PDH) interface calls the registry interface to retrieve performance data
• Uses the PDH.DLL to access the PDH API
• http://msdn.microsoft.com/archive/default.asp?url=/archive/en-us/dnarperfmo/html/msdn_pdhlib.asp
• http://msdn2.microsoft.com/en-us/library/aa373083.aspx
15© 2007 Citrix Systems, Inc. — All rights reserved
Performance Data Helper
• Code examples
http://msdn2.microsoft.com/en-us/library/aa373169.aspx
…
TCHAR szCounterPath[45] = TEXT("\\Processor(0)\\% Processor Time");
…
pdhStatus = PdhAddCounter (hQuery, szCounterPath, 0, &hCounter);
…
16© 2007 Citrix Systems, Inc. — All rights reserved
Performance Data Helper
• The performance counters information is stored under:
• HKLM\SOFTWARE\Microsoft\WindowsNT\CurrentVersion\Perflib\009\Counter
• HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib\009\Help
…
6
% Processor Time
8
…
…
7
% Processor Time is the percentage of elapsed time that the processor
spends to execute a non-Idle thread. It is calculated by measuring the
duration of the idle thread is active in the sample interval, and
subtracting that time from interval duration. (Each processor has an
idle thread that consumes cycles when no other threads are ready
to run). This counter is the primary indicator of processor activity, and
displays the average percentage of busy time observed during the
sample interval. It is calculated by monitoring the time that the service
is inactive, and subtracting that value from 100%.
9
…
18© 2007 Citrix Systems, Inc. — All rights reserved
Performance Data Helper
• The same information can be found in the following files
• %Systemroot%\System32\Perfc009.dat
• %Systemroot%\System32\Perfh009.dat
• For instance those files will be used to recreate the performance counters again
• http://support.microsoft.com/kb/300956/en-us
19© 2007 Citrix Systems, Inc. — All rights reserved
Performance Data Helper
• Perfmon.exe reads the registry keys
• IMA reads the Perfc009.dat
• Only once when IMA is starting
20© 2007 Citrix Systems, Inc. — All rights reserved
Citrix and the PDH
• We also provide some performance counters
• %Systemroot%\inf\009\IMAService\imaperf.ini
• %Systemroot%\inf\009\MetaFrameXP\mfxpperf.ini
21© 2007 Citrix Systems, Inc. — All rights reserved
Citrix and the PDH
• HKLM\SYSTEM\CurrentControlSet\Services\IMAService\Performance
22© 2007 Citrix Systems, Inc. — All rights reserved
Citrix and the PDH
• List of counters that LMS20Rules.dll uses
• \\Processor(_Total)\\% Processor Time
• \\System\\Context Switches/sec
• \\Memory\\% Committed Bytes In Use
• \\Memory\\Page Faults/sec
• \\Memory\\Pages/sec
• \\PhysicalDisk(_Total)\\Disk Bytes/sec
• \\PhysicalDisk(_Total)\\Disk Reads/sec
• \\PhysicalDisk(_Total)\\Disk Writes/sec
23© 2007 Citrix Systems, Inc. — All rights reserved
Citrix and the PDH
• Same counter with different name depending on the OS version
• English W2K
\\System\\% Total Processor Time
• English W2K3\\Processor(_Total)\\% Processor Time
24© 2007 Citrix Systems, Inc. — All rights reserved
Load balancing - Data Store
012C Hex = 300 dec = 5 min
3C Hex = 60 dec = 1 min
25© 2007 Citrix Systems, Inc. — All rights reserved
Load balancing - Data Store
26© 2007 Citrix Systems, Inc. — All rights reserved
Where can it go wrong?
Agenda item number 1Load Balancing on an Application Delivery Scenario
Architecture
Troubleshooting
Where can it go wrong?
27© 2007 Citrix Systems, Inc. — All rights reserved
Where can it go wrong?
• Black hole effect• Load Throttling - Intelligent Load Balancing
• Failing to read performance counters
• English OS with and MUI
28© 2007 Citrix Systems, Inc. — All rights reserved
Black hole effect
• Problem
• At Peak logon times a recently boot up server will get all connections and might become unresponsive
• Description
• When a server comes into a farm its load will be 0 and it will be getting all new connections to the farm. At peak logon times this could make the server unresponsive if overloaded
29© 2007 Citrix Systems, Inc. — All rights reserved
Black hole effect
• Solution
Load Throttling
???
30© 2007 Citrix Systems, Inc. — All rights reserved
Load Throttling
• Load Throttling uses Intelligent Load Biasing (ILB)
• HKLM\SOFTWARE\Citrix\IMA\LMS\UseILB
Default =1 (0,1)
• HKLM\SOFTWARE\Citrix\IMA\LMS\ILBMultiplier
Default =2 (2, 3, 4…)
Current Resolution Load += [(Max Load – Current Resolution Load) / ILBMultiplier]
31© 2007 Citrix Systems, Inc. — All rights reserved
Load Throttling - Low logon rate
• 2 server farm (server A and server B) using the Default load evaluator
32© 2007 Citrix Systems, Inc. — All rights reserved
Load Throttling - Low logon rate
ICA Client
Data Collector
Server A
Server BDynamic Store
Server B
Server A
Server Load BIAS
0
0 0
0(10000-0)/2+0 = 5000100
(10000-0)/2+0 = 5000100
34© 2007 Citrix Systems, Inc. — All rights reserved
Load Throttling - Low logon rate
• Server load if Low logon rate
0
1000
2000
3000
4000
5000
6000
ILB=1
ILB=0
35© 2007 Citrix Systems, Inc. — All rights reserved
Load Throttling - Black hole effect
• 2 server farm (server A and server B) using the Default load evaluator
• Server B is running normally (60 users) while at Peak logon times the server A is booted up
36© 2007 Citrix Systems, Inc. — All rights reserved
Load Throttling - Black hole effect
ICA Client
Data Collector
Server A
Server BDynamic Store
Server B
Server A
Server Load BIAS
0
6000 0
0(10000-0)/2+0 = 5000(10000-5000)/2+5000 = 7500
(10000-6000)/2+6000 = 8000
(10000-7500)/2+7500 = 8750
x60
38© 2007 Citrix Systems, Inc. — All rights reserved
Load Throttling
• Server load at Peak logon times
0
2000
4000
6000
8000
10000
12000
1 8
15
22
29
36
43
50
57
64
71
78
85
92
99
ILBMultiplier=2
ILBMultiplier = 10
ILBMultiplier = 20
ILB=0
39© 2007 Citrix Systems, Inc. — All rights reserved
Failing to read performance counters
• Problem
One server in the farm gets all the connections
• Cause
If a farm is balanced using advance load evaluators and we fail to read those performance counters in one server, it will report 0 load and so it will get all the connections and will get unresponsive, even crashing eventually
40© 2007 Citrix Systems, Inc. — All rights reserved
Failing to read performance counters
• Solution
• From CPS4.0 HRP02 we have change the way the load is calculated
• If we fail to read some performance counters we will report full load
Note: If you were using custom load evaluators like Advance + Default after HRP02 you might report 10000 load while with HRP01 seemed to be OK
41© 2007 Citrix Systems, Inc. — All rights reserved
English OS with and MUI
• Problem
Advance load evaluators report full load when an MUI is installed
• Cause
We were expecting English counters only
Note: To know if your problem is due to the MUI and not to a problem reading performance counters you can swap the files perfc009.dat and perfc007.dat. If that resolves the issue you are in the right direction
42© 2007 Citrix Systems, Inc. — All rights reserved
English OS with and MUI
• Solution
• %Program Files%/citrix/system32/citrix/ima/lms/lms20rules.dll
• HKLM\Software\Citrix\IMA\LMS\EnableTranslation=1 (dword)
• To be manually added
• First introduced in PSE400R02W2k3016
43© 2007 Citrix Systems, Inc. — All rights reserved
Architecture
Agenda item number 1Load Balancing on an Application Delivery Scenario
Architecture
Where can it go wrong?
Troubleshooting
44© 2007 Citrix Systems, Inc. — All rights reserved
Troubleshooting
• Failing to read performance counters => full load
• Use filemon while rebooting IMA to see possible problems accessing perf*.dat files. (leave it running for a while if they don't come up)
• Consider rebuilding performance counters
• Check that no performance counters are disabled
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfDisk\Performance]
"Disable Performance Counters“ = 1 (DWORD)
45© 2007 Citrix Systems, Inc. — All rights reserved
Troubleshooting
• Failing to read performance counters => full load
• CDF traceslms20rules ERROR!!!!! _CreatePdhQueryHelper() failed for
m_hDiskWriteCounter,error = -1073738823
lms20rules LMSRuleDll_Interface::CreatePDHQuery() ERROR!!!!! rc =
-1073738823
• -1073738823 in HEX is FFFFFFFFC0000BB9
http://msdn2.microsoft.com/en-us/library/aa373046.aspx
0xC0000BB9 (PDH_CSTATUS_NO_COUNTER) The specified counter could not be found.
46© 2007 Citrix Systems, Inc. — All rights reserved
Summary
• Deep understanding of load balancing• BIAS, PDH…
• Understanding Intelligent load balancing• Load Throttling
• Typical scenarios• Full load versus 0 load, MUI…
• Troubleshooting techniques• So far we are resolving all the problems with those tips
47© 2007 Citrix Systems, Inc. — All rights reserved
Q&A
?