in-depth troubleshooting on netscaler using command line tools
DESCRIPTION
Webinar recording - https://www1.gotomeeting.com/register/753997104 Citrix NetScaler has a rich Web-based management suite of tools available. To dig deep troubleshooting NetScaler, sometimes it’s best to roll up your sleeves and dig out the command line! The goal of this session is to demystify some useful command line tools and provide a tactical approach to troubleshooting of NetScaler. In this session we will demonstrate troubleshooting approaches using the command line and many tips for common issues seen in customer deployments. In this session you will learn about: · Differences between NetScaler kernel and BSD · Processes and disk layout · Look up stats and statuses · Troubleshoot using various different logs · Use counters to help identify issuesTRANSCRIPT
Andrew Sandford
Senior Readiness Specialist, Worldwide Support Readiness EMEA
Citrix Support Secrets Webinar SeriesIn-depth Troubleshooting on NetScaler using Command Line Tools
27 March 2014
© 2014 Citrix | Confidential – Do Not Distribute
Agenda
NetScaler CLI recap
What’s my NetScaler?
How is my NetScaler configured?
What’s my NetScaler doing?
Time for Technical Support?
Q&A
NetScaler CLIRecap
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
NetScaler Native CLI
> show
>add
>remove
>set
>enable
>disable
>force
>bind
>unbind
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
NetScaler Native CLI
>help <command>
>man <command>
>set cli prompt %u@%h-%T
>set cli mode -color ON
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
NetScaler BSD Shell
#
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
# tar
# head
# less
# more
# cat
# zcat
# ls
# find
What’s my NetScaler?
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
What’s my NetScaler?
> show version
> show ns hostname
> show hardware
> show interface –summary
>stat ssl
#sysctl –a netscaler | more
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show version/show ns hostname
> show version
NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27
> show ns hostname
Hostname: nstipster
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show hardware
> show hardware
Platform: NSMPX-10500 8*CPU+2*E1K+8*E1K+2*IX+8*CVM 1620 760100
Manufactured on: 10/7/2010
CPU: 2832MHZ
Host Id: 1234567890
Serial no: M123456789
Encoded serial no: M123456789
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show interface -summary
> show interface -summary
--------------------------------------------------------------------------------
Interface MTU MAC Suffix
--------------------------------------------------------------------------------
1 0/1 1500 00:25:90:12:eb:5a Gig Ethernet 10/100/1000 MBits
2 0/2 1500 00:25:90:12:eb:5b Gig Ethernet 10/100/1000 MBits
3 1/1 1500 00:e0:ed:1a:24:97 Gig Ethernet, copper SFP
4 1/2 1500 00:e0:ed:1a:24:96 Gig Ethernet, copper SFP
5 1/3 1500 00:e0:ed:1a:24:95 Gig Ethernet, no SFP found
6 1/4 1500 00:e0:ed:1a:24:94 Gig Ethernet, no SFP found
7 1/5 1500 00:e0:ed:1a:24:a3 Gig Ethernet, copper SFP
8 1/6 1500 00:e0:ed:1a:24:a2 Gig Ethernet, no SFP found
9 1/7 1500 00:e0:ed:1a:24:a1 Gig Ethernet, no SFP found
10 1/8 1500 00:e0:ed:1a:24:a0 Gig Ethernet, no SFP found
11 10/1 1500 00:1b:21:77:c0:35 10G Ethernet,...SFP+/SFP found
12 10/2 1500 00:1b:21:77:c0:34 10G Ethernet,...SFP+/SFP found
13 LO/1 1500 00:25:90:12:eb:5a NetScaler Loopback interface
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
stat ssl
> stat ssl
SSL Summary
# SSL cards present 8
# SSL cards UP 8
SSL engine status 1
SSL sessions (Rate) 0
System
Transactions Rate (/s) Total
SSL transactions 0 301
SSLv2 transactions 0 0
SSLv3 transactions 0 0
TLSv1 transactions 0 301
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
sysctl
# sysctl -a netscaler
netscaler.developer: 0
netscaler.recovery: 0
netscaler.sysid: 450000
netscaler.serial: 98310000cb254307ee78
netscaler.descr: NetScaler Virtual Appliance 3G
netscaler.num_pe_running: 1
netscaler.version: NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27
netscaler.model: 3000
netscaler.vmpe_max_cpus: 2
netscaler.Classic: 0
netscaler.nCore: 1
netscaler.descr: NetScaler Virtual Appliance 3Gnetscaler.num_pe_running: 1netscaler.version: NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27netscaler.model: 3000netscaler.vmpe_max_cpus: 2netscaler.nCore: 1
How is my NetScaler configured?
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
How is my NetScaler configured?
> show ip
> show feature
> show ns mode
> show info
> show license
> show run | more
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show ip
> show ip
Ipaddress TD Type Mode Arp Icmp Vserver State
--------- -- ---- ---- --- ---- ------- ------
1) 192.168.196.45 0 NetScaler IP Active Enabled Enabled NA Enabled
2) 192.168.196.146 0 SNIP Active Enabled Enabled NA Enabled
3) 192.168.196.147 0 VIP Active Enabled Enabled Enabled Enabled
4) 22.22.22.2 0 VIP Active Enabled Enabled Enabled Enabled
5) 192.168.100.44 0 SNIP Active Enabled Enabled NA Enabled
6) 192.168.1.2 0 SNIP Active Enabled Enabled NA Enabled
7) 192.168.196.148 0 VIP Active Enabled Enabled Enabled Enabled
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show feature
> show feature
Feature Acronym Status
------- ------- ------
1) Web Logging WL ON
2) Surge Protection SP OFF
3) Load Balancing LB ON
4) Content Switching CS ON
5) Cache Redirection CR OFF
6) Sure Connect SC ON
8) Priority Queuing PQ ON
9) SSL Offloading SSL ON
10) Global Server Load Balancing GSLB ON
11) Http DoS Protection HDOSP OFF
12) Content Filtering CF ON
13) Integrated Caching IC OFF
19) Rewrite REWRITE ON
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show ns mode
> show ns mode
Mode Acronym Status
------- ------- ------
1) Fast Ramp FR ON
2) Layer 2 mode L2 OFF
3) Use Source IP USIP OFF
4) Client Keep-alive CKA OFF
5) TCP Buffering TCPB OFF
6) MAC-based forwarding MBF OFF
7) Edge configuration Edge ON
8) Use Subnet IP USNIP ON
9) Layer 3 mode (ip forwarding) L3 ON
10) Path MTU Discovery PMTUD ON
16) Bridge BPDUs BridgeBPDUs OFF
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show info
> show info
NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27
NetScaler IP: 192.168.47.6 (mask: 255.255.240.0)
NW FWMODE: NOFIREWALL
Number of MappedIP(s): 0
Node: Secondary (Primary is 192.168.47.1)
System Time: Thu Mar 27 08:52:06 2014
Last Config Changed Time: Thu Mar 27 07:52:50 2014
Last Config Saved Time: Tue Mar 25 13:28:21 2014
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show run | more
#NS10.1 Build 124.13
# Last modified Thu Mar 27 10:08:20 2014
set ns config -IPAddress 192.168.47.1 -netmask 255.255.240.0
enable ns feature LB CS SSL SSLVPN OSPF REWRITE AppFw RESPONDER HTMLInjection AppFlow
enable ns mode FR L3 Edge USNIP SRADV DRADV PMTUD
set system parameter -natPcbForceFlushLimit 4294967295
set system user nsroot -----------------SNIP---------------- -encrypted
set rsskeytype -rsstype ASYMMETRIC
set lacp -sysPriority 32768 -mac 11:22:33:aa:bb:cc
set ns hostName nstipster
set interface 1/1 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype "Xen Virtual" -ifnum 1/1
set interface 1/2 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype "Xen Virtual" -ifnum 1/2
set interface 1/3 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype "Xen Virtual" -ifnum 1/3
set interface LO/1 -haMonitor OFF -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype Loopback -ifnum LO/1
add ns ip6 fe80::40da:5dff:fe08:296b/64 -scope link-local -type NSIP -vlan 1 -vServer DISABLED -mgmtAccess ENABLED -dynamicRouting ENABLED
What’s my NetScaler doing?
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
Processes and Uptime
> shell top
#ps –ax | more
#uptime
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
#top/nsppe
> shell top
last pid: 13825; load averages: 1.00, 1.04, 1.02 up 1+19:48:58 08:38:17
60 processes: 2 running, 58 sleeping
CPU states: 18.0% user, 0.0% nice, 32.1% system, 0.0% interrupt, 49.9% idle
Mem: 99M Active, 51M Inact, 1492M Wired, 15M Cache, 165M Buf, 2564K Free
Swap: 4500M Total, 12K Used, 4500M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
50185 root 1 44 -52 814M 815M CPU1 1 20.2H 100.00% NSPPE-00
11834 nobody 1 4 0 26448K 18276K accept 0 0:02 0.05% httpd
50206 root 1 4 0 61364K 11632K kqread 0 1:03 0.00% nsaggregator
50251 root 1 4 0 30504K 7964K kqread 0 0:47 0.00% nsconfigd
44 root 1 4 0 15880K 1828K kqread 0 0:21 0.00% pitboss
995 root 1 8 0 25660K 17672K nanslp 0 0:13 0.00% httpd
987 root 1 96 0 3668K 876K select 0 0:11 0.00% syslogd
50188 root 1 4 0 59184K 9016K kqread 0 0:10 0.00% nsnetsvc
100%
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
#ps –ax | more
root@ns# ps -ax | more
PID TT STAT TIME COMMAND
0 ?? WLs 0:00.44 [swapper]
1 ?? ILs 0:00.09 /sbin/init --
2 ?? DL 0:02.35 [g_event]
3 ?? DL 0:09.00 [g_up]
4 ?? DL 0:02.21 [g_down]
5 ?? DL 0:00.00 [xpt_thrd]
6 ?? DL 0:00.00 [acpi_task_0]
7 ?? DL 0:00.00 [acpi_task_1]
8 ?? DL 0:00.00 [acpi_task_2]
9 ?? DL 0:00.00 [kqueue taskq]
10 ?? RL 0:10.39 [idle: cpu1]
--More—(byte 933)
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
IP related
> show route
> show ip
> show dns addrec -type proxy
# ping
# traceroute
# telnet
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
> show route
> show route
Network Netmask Gateway/OwnedIP State TD Type
------- ------- --------------- ----- -- ----
1) 0.0.0.0 0.0.0.0 192.168.32.1 UP 0 STATIC
2) 127.0.0.0 255.0.0.0 127.0.0.1 UP 0 PERMANENT
3) 192.168.32.0 255.255.240.0 192.168.47.1 UP 0 DIRECT
4) 172.16.200.0 255.255.255.0 192.168.47.2 UP 0 STATIC|ADV
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
Load Balancing
> show lb vserver
> show service
> show connectiontable | grep <IP Address|port>
> show persistentSessions
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show lb vserver
> show lb vserver
1) LB_RGB (192.168.47.3:80) - HTTP Type: ADDRESS
State: UP
Last state change was at Wed Aug 14 09:17:14 2013
Time since last state change: 0 days, 00:30:42.140
Effective State: UP
Client Idle Timeout: 180 sec
Down state flush: ENABLED
Disable Primary Vserver On Down : DISABLED
Appflow logging: ENABLED
Port Rewrite : DISABLED
No. of Bound Services : 1 (Total) 1 (Active)
Configured Method: ROUNDROBIN
Mode: IP
Persistence: NONE
Vserver IP and Port insertion: OFF
IcmpResponse: PASSIVE
New Service Startup Request Rate: 0 PER_SECOND, Increment Interval: 0
Warning: Feature(s) not enabled [LB]
LB_RGB (192.168.47.3:80) - HTTP Type: ADDRESS
State: UPClient Idle Timeout: 180 sec
Down state flush: ENABLED
Disable Primary Vserver On Down : DISABLED
Appflow logging: ENABLED
Port Rewrite : DISABLED
No. of Bound Services : 1 (Total) 1 (Active)
Configured Method: ROUNDROBIN
Mode: IP
Persistence: NONE
Vserver IP and Port insertion: OFFWarning: Feature(s) not enabled [LB]
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show lb vserver [vservername]
> sh lb vserver LB_RGB
LB_RGB (192.168.47.3:80) - HTTP Type: ADDRESS
State: UP
Last state change was at Wed Aug 14 11:56:19 2013
Time since last state change: 0 days, 20:33:37.60
Effective State: UP
Client Idle Timeout: 180 sec
Down state flush: ENABLED
No. of Bound Services : 1 (Total) 1 (Active)
Configured Method: ROUNDROBIN
Mode: IP
Persistence: NONE
1) svc_blue (192.168.196.62: 80) - HTTP State: UP Weight: 1
1) svc_blue (192.168.196.62: 80) - HTTP State: UP Weight: 1
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show service [servicename]
> show service svc_blue
svc_blue (192.168.196.62:80) - HTTP
State: UP
Last state change was at Wed Aug 14 11:56:19 2013
Time since last state change: 0 days, 20:38:31.880
Server Name: Blue
Use Source IP: NO
Idle timeout: Client: 180 sec Server: 360 sec
Client IP: ENABLED ClientIP
1) Monitor Name: mon-http-ecv
State: UP Weight: 1 Passive: 0
Probes: 14887 Failed [Total: 124 Current: 0]
Last response: Success - Pattern found in response.
Response Time: 10.220 millisec
1) Monitor Name: mon-http-ecv
State: UP Weight: 1 Passive: 0
Probes: 14887 Failed [Total: 124 Current: 0]
Last response: Success - Pattern found in response.
Response Time: 10.220 millisec
State: UP
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show persistentSessions
> sh persistentSessions
Type SRC-IP DST-IP PORT VSNAMETIMEOUT PERSISTENCE-PARAMETER
SOURCEIP 192.168.119.81 192.168.196.61 80 LB_RGB 118192.168.119.81
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
Authentication
> show vpn vserver
> show aaa session
> show aaa stats
# cat /tmp/aaad.debug
How do I troubleshoot deeper?
Time for Technical Support?
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
Core dump directories
# ls -la
total 8
drwxrwxr-x 4 root nobody 512 Mar 4 09:41 .
drwxr-xr-x 31 root wheel 1024 Dec 4 10:06 ..
drwxrwxr-x 2 root nobody 512 Oct 8 21:28 1
-rw-r--r-- 1 root nobody 2 Mar 4 09:36 bounds
# ls –la 1/
total 10292
drwxrwxr-x 2 root nobody 512 Jan 22 11:24 .
drwxrwxr-x 4 root nobody 512 Mar 4 09:41 ..
-rw------- 1 root nobody 9881665 Dec 18 11:15 NSPPE-00-1094.gz
-rw------- 1 root nobody 603327 Dec 18 11:15 nscac64p-1189.gz
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
Packet Engine (nsppe) Process Crashing
Oct 19 08:52:46 <local0.alert> vpx1 nsppe: PE 0 (pid 1077) got signal 6; signal mask is 0x0 0x0 0x0 0x0
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NSPPE-00 (1077) unexpectedly died due to receiving signal
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 There may be a delay restarting process while collecting core dump on NSPPE-00 (1077)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 proc NSPPE-00 (1077) failure. Therefore initiating nCore NetScaler restart according to policy setting (0x29ac)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NetScaler restart may be delayed if collecting core dump for NSPPE-00 (1077)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 Pitboss declaring system failure: NSPPE-00 (1077) exited
Oct 19 08:53:44 <local0.info> vpx1 [44]: pitboss Sat Oct 19 08:53:44 2013 Deleting watch on NSPPE-00 (1077) for ()
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
/var/core
# cd /var/core/
# ls -la
total 14
drwxrwxr-x 6 root nobody 512 Oct 25 07:54 .
drwxr-xr-x 30 root wheel 1024 Oct 25 10:31 ..
drwxrwxr-x 2 root nobody 512 Sep 30 13:39 1
-rw-r--r-- 1 root nobody 2 Oct 25 07:54 bounds
root@vpx1# ls 1/
NSPPE-00-1077.gz nscac64p-1177.gz nsnetsvc-1086.gz
aslearn-1148.gz nscfsyncd-1158.gz nsrised-1164.gz
imi-1129.gz nsclfsyncd-1160.gz provserverd-1162.gz
monuploadd-1154.gz nsclusterd-1105.gz snmpd-1152.gz
nsaaad-1131.gz nsconfigd-1156.gz
nsaggregatord-1107.gz nsfsyncd-1110.gz
© 2014 Citrix | Confidential – Do Not Distribute
Show commandsSystem show node
show info
show licenseVserver/Service show lb vserver
show cs vserver
show service
show persistencesession
show connectiontableIP related show route
show ip
show dns addrec -type proxyDiagnostic show techsupport
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show node
> show node
1) Node ID: 0
IP: 192.168.1.145 (NS145)
Node State: NOT UP
Master State: Secondary
Fail-Safe Mode: OFF
INC State: DISABLED
Sync State: ENABLED
Propagation: ENABLED
Enabled Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1
Disabled Interfaces : None
HA MON ON Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1
Interfaces on which heartbeats are not seen : 1/8 1/6 1/5 1/4 1/3 1/2 1/1
Interfaces causing Partial Failure: 1/8 1/6 1/5 1/4 1/3 1/2 1/1
SSL Card Status: UP
Hello Interval: 200 msecs
Dead Interval: 3 secs
Node in this Master State for: 0:15:2:13 (days:hrs:min:sec)
2) Node ID: 1
IP: 192.168.1.45
Node State: STAYSECONDARY
Master State: Secondary
Fail-Safe Mode: OFF
INC State: DISABLED
Sync State: ENABLED
Propagation: ENABLED
Enabled Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1
Disabled Interfaces : None
HA MON ON Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1
Interfaces on which heartbeats are not seen : 1/8 1/6 1/5 1/4 1/3 1/2 1/1
Interfaces causing Partial Failure: 1/8 1/6 1/5 1/4 1/3 1/2 1/1
SSL Card Status: UP
Local node information:
Critical Interfaces: 1/8 1/6 1/5 1/4 1/3 1/2 1/1
>
Node State: NOT UP
Master State: SecondaryMaster Enabled Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1
Disabled Interfaces : None
HA MON ON Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1
Interfaces on which heartbeats are not seen : 1/8 1/6 1/5 1/4 1/3 1/2 1/1
Interfaces causing Partial Failure: 1/8 1/6 1/5 1/4 1/3 1/2 1/1
Node State: STAYSECONDARY
Master State: Secondary
© 2014 Citrix | Confidential – Do Not Distribute
NetScaler ProcessesProcess Description Process Description
nsppe NetScaler Packet Engine nsfsyncd Sync bookmarks and SSL certificates
nsaaadRBA and SSL VPN External Auth nsnetsvc
Used by the GUI for config changes
nsconf Writes the ns.conf file nsumond Runs the scriptable monitors
nslog.sh Controls Logging for newnslog nsconmsg Controls writing of newnslog
nssync HA sync nscollect Statistics gathering for historical reporting
nsreadfile Used to read SSL Cert Files imi/ripd/
ospfd/bgpdRouting processes
nscrlrefresh SSL CRL list update
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
stat commands
>stat ns
>stat cpu
>stat interface
>stat lb vserver
>stat cs vserver
>stat service
>stat ssl
>stat dns
>stat http
System
Entities
Protocols
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
stat ns
> stat ns
System overview
Up since Wed Aug 14 11:53:33 2013
CPU usage (%) 0.60
Packet CPU usage (%) 0.60
Management CPU usage (%) 0.80
Memory usage (MB) 190
InUse Memory (%) 18.73
Last Transition time We...013
System state UP
Master state Primary
# SSL cards UP 0
# SSL cards present 0
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
stat cpu
> stat cpu
CPU statistics
ID Usage
1 0
7 0
6 0
5 0
4 0
3 1
2 0
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
stat interface
> stat interface
Interface Summary
ID IntfState IntfAlias Rx Bytes Tx Bytes Rx Pkts Tx Pkts
1/8 DOWN 0 0 0 0
1/7 DOWN 0 0 0 0
1/6 DOWN 0 0 0 0
1/5 DOWN 0 0 0 0
1/4 DOWN 0 0 0 0
1/3 DOWN 0 0 0 0
1/2 DOWN 0 0 0 0
1/1 UP 14476M 21813M 442178k 56718611
10/2 DOWN 0 0 0 0
10/1 DOWN 0 0 0 0
0/1 UP 32027M 18048M 292060k 67610607
0/2 DOWN 0 0 0 0
LO/1 UP 831255M 1218G 6624M 13125M
LA/1 DOWN LA2 0 0 0 0
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
stat interface [interfacename]> stat interface 1/1
Interface [1/1]:Interface State UPLink uptime 00:40:21Link downtime 00:00:00
Throughput Statistics Rate (/s) TotalBytes received 42393 54497294Bytes transmitted 2584 20222135Packets received 629 710246Packets transmitted 47 69066Packet Statistics Rate (/s) TotalMulticast packets 18 41219NetScaler packets 85 98954
LACP Statistics Rate (/s) TotalLACPDUs received 0 0LACPDUs transmitted 0 0Error Statistics Rate (/s) TotalError packets received (hw) 0 0Error packets transmitted (hw) 0 0Inbound packets discarded (hw) 0 0Outbound packets discarded (hw) 0 0Packets dropped in Rx (sw) 539 599904Packets dropped in Tx (sw) 0 0NIC hangs -- 0Status stalls -- 0Transmit stalls -- 0Receive stalls -- 0Error-disables -- 0Duplex mismatches -- 0Link re-initializations -- 0MAC moves registered 0 0Times NIC became muted -- 0
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
stat dns
> stat dns
DNS Statistics
Runtime Statistics
Dns queries 106983
Multi queries 0
Dns responses 0
Server responses 102334
Total Record updates 0
Auth answers 0
Server queries 102347
Cache flush called 0
Cache entries flushed 0
Configuration Statistics
Non-authoritative entries 0
Authoritative entries 98
Error Statistics
Nonexistent domain 102359
Response class unsupported 0
Invalid query format 0
Stray answers 0
Incorrect RD length 0
Requests refused 0
Response type unsupported 0
Query class unsupported 0
Invalid response format 0
No answer responses 102334
Multi queries disabled 0
Other errors 0
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
stat http
> stat http
HTTP Statistics - Summary
Rate (/s) Total
Total requests 1 6251
Total responses 1 5885
Request bytes received 27 293191
Response bytes received 286 1744835
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
stat lb vserver [name]
> stat lb vserver LB_RGB
Virtual Server Summary
vsvrIP port Protocol State Health actSvcs
LB_RGB 192.168.47.3 80 HTTP DOWN 0 0
inactSvcs
LB_RGB 1
Virtual Server Statistics
Rate (/s) Total
Vserver hits 0 0
Requests 0 0
Responses 0 0
Request bytes 0 65
Response bytes 0 188
Total Packets rcvd 0 5
Total Packets sent 0 4
Vserver hits 0 0
Requests 0 0
Responses 0 0
Request bytes 0 65
Response bytes 0 188
Total Packets rcvd 0 5
Total Packets sent 0 4
Current client connections -- 0
Current Client Est connections -- 0
Current server connections -- 0
Requests in surge queue -- 0
Requests in vserver's surgeQ -- 0
Requests in service's surgeQs -- 0
Spill Over Threshold -- 0
Bound Service(s) Summary
IP port Type State Hits Hits/s
svc_andrews 192.168.33.130 80 HTTP DOWN 0 0/s
Req Req/s Rsp Rsp/s Throughp ClntConn SurgeQ
svc_andrews 0 0/s 0 0/s 0 0 0
SvrConn ReuseP MaxConn ActvTran SvrTTFB Load
svc_andrews 0 0 0 0 0 0
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
stat service [servicename]
> stat service svc_blue
Service Summary
IP port Type State
svc_blue 192.168.196.62 80 HTTP UP
Service Stats:
Rate (/s) Total
Requests 0 865
Responses 0 855
Request bytes 22 67683
Response bytes 767 2304810
Current client connections -- 0
Requests in surge queue -- 0
Current server connections -- 9
Current Server Est connections -- 0
Connections in reuse pool -- 0
Maximum server connections -- 0
Average server TTFB -- 0
Current load on the service -- 0
Requests 0 865
Responses 0 855
Request bytes 22 67683
Response bytes 767 2304810
Current client connections -- 0
Requests in surge queue -- 0
Current server connections -- 9
Current Server Est connections -- 0
Connections in reuse pool -- 0
Maximum server connections -- 0
Average server TTFB -- 0
Current load on the service -- 0
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
NetScaler Disk Partitions
# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/md0c 286M 245M 35M 88% /
devfs 1.0K 1.0K 0B 100% /dev
procfs 4.0K 4.0K 0B 100% /proc
/dev/ad0s1a 1.4G 965M 368M 72% /flash
/dev/ad0s1e 14G 3.1G 9.5G 24% /var
NetScaler File System
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
NetScaler File System examples
/var/log/ns.log
/var/nslog/newnslog/newnslog.ppe.x
/var/nsinstall/build-10.1-124.13_nc.tgz
/flash/ns-10.1-124.13.gz
/var/core/NSPPE-00-353.gz
/var/crash/vmcore.0
/nsconfig/ns.conf
/nsconfig/ssl/ns-root.cert
/nsconfig/monitors/nssmtp.pl
/nsconfig/license/FID__b0d70c6_13b16ab7034_573f.lic
/netscaler/nsconmsg
/var/nstrace/nstrace1.cap
Logs
Firmware
Crash FilesNetScaler Configuration
SSL CertificatesMonitor Scripts
License FilesNetScaler BinariesPacket Trace Files
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
dmesg
root@ns# dmesg
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.3-NETSCALER-10.1 #0: Thu Feb 20 18:54:22 PST 2014
[email protected]:/usr/obj/amd64/usr/home/build/rs_101_124_8/usr.src/sys/NS64
Preloaded elf kernel "/ns-10.1-124.13" at 0xffffffff930b3000.
Calibrating clock(s) ... i8254 clock: 1189606 Hz
CLK_USE_I8254_CALIBRATION not specified - using default frequency
Timecounter "i8254" frequency 1193182 Hz quality 0
Calibrating TSC clock ... TSC clock: 3325066248 Hz
CPU: Intel(R) Core(TM)2 Duo CPU E8600 @ 3.33GHz (3325.07-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x1067a Stepping = 10
Features=0x789fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CLFLUSH,MMX,FXSR,SSE,SSE2>
Features2=0x81282201<SSE3,SSSE3,CX16,<b19>,<b21>,<b24>,<b31>>
AMD Features=0x20000800<SYSCALL,LM>
AMD Features2=0x1<LAHF>
real memory = 2143289344 (2044 MB)
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
/var/log
# ls
auth.log httperror.log ns.log
auth.log.0.gz httperror.log.0 ns.log.0
callhome.log lastlog nscollect.log
callhomedebug.log license.log nscollect_cl.log
cron lpd-errs nsvpn.log
cron.0.gz maillog nsvpnd.log
ctxslsboc.log maillog.0.gz ntpd.log
db messages security
httpaccess.log messages.0 snmpd.log
httpaccess.log.0.gz nitro.log wicmd.log
httpd.scoreboard nitro.log.0.gz
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
/var/nslog
# cd /var/nslog/
# ls
asl lspci_tv.last nsagg.conf
aslearn.log lspci_tv.prev nsagg.log
aslearn_old_db.tar.gz lspci_vvvxxx.boot nslog.nextfile
conmsg.log lspci_vvvxxx.last nsumond
dmesg.boot lspci_vvvxxx.prev nsumond.log
dmesg.last newnslog snmpdebug.log
dmesg.prev newnslog.0.gz
lspci_tv.boot ns.log
# cd newnslog
# ls
newnslog.ppe.0
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
Troubleshooting Techniques
View eventsView console messagesView statisticsDebug system countersDebug load balancing issuesDebug CPU/Memory utilization
Use cases
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
Troubleshooting Techniques: nsconmsg
# cd /var/nslog
# nsconmsg –K newnslog -d event
# nsconmsg –K newnslog -d consmsg
# nsconmsg –K newnslog -d oldconmsg
# nsconmsg –K newnslog -s ConLb=2 –d oldconmsg
# nsconmsg –K newnslog -s ConDebug=1 –d oldconmsg
nsconmsg Common Syntax
Make sure to use –K, NOT -k
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
nsconmsg examples
# nsconmsg -d current -g cpu_use
# nsconmsg -K <newnslog-filename> -d event
# nsconmsg -d current -g ha_cur_master_state
# nsconmsg -s ConLb=2 -d oldconmsg
# nsconmsg -s ConCSW=2 -d oldconmsg
# nsconmsg -d current -g pol_hits
# nsconmsg -s ConSSL=2 -d oldconmsg
# nsconmsg -s ConCMP=2 -d oldconmsg
Live CPU related statsArchived events
HA Failover causeLB stats
CS related countersReal-time policy hitsSSL related counters
Compression related counters
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
Policy Hits
# cd /var/nslog/newnslog
# nsconmsg -K newnslog.ppe.0 -d current -g pol_hits
Displaying performance information
NetScaler V20 Performance Data
NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27
reltime:mili second between two records Fri Oct 25 09:45:52 2013
Index rtime totalcount-val delta rate/sec symbol-name&device-no
0 3038060 2 2 0 pol_hits Policy(CTX-LDAP)
1 3500159 15 15 2 pol_hits Policy(__ESNS_PREBODY_POLICY)
2 0 15 15 2 pol_hits Policy(__ESNS_POSTBODY_POLICY)
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
nsconmsg –K newnslog -d event
# nsconmsg -K newnslog -d event | more
Displaying event information
NetScaler V20 Performance Data
NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27
rtime: Relative time between two records in milliseconds
seqno rtime event-message event-time
1906 3910 PPE-0 'interface(1/1)' has been disabled Wed Aug 14 12:27:16 2013
1907 0 PPE-0 'interface(1/1)' DOWN Wed Aug 14 12:27:16 2013
1908 7 PPE-0 MonServiceBinding_192.168.47.16:4739_(ping-default)(service_192.168.47.16_33554): DOWN; Last response: Failure - Probe timed out. Wed Aug 14 12:27:25 2013
1909 0 PPE-0 MonServiceBinding_192.168.224.5:53_(ping-default)(SVC_CTX_DNS1): DOWN; Last response: Failure - Probe timed out. Wed Aug 14 12:27:25 2013
1910 0 PPE-0 MonServiceBinding_192.168.204.51:53_(ping-default)(SVC_CTXANG_DNS2): DOWN; Last response: Failure - Probe timed out. Wed Aug 14 12:27:25 2013
1911 0 PPE-0 MonServiceBinding_192.168.204.50:53_(ping-default)(SVC_CTXANG_DNS1): DOWN; Last response: Failure - Probe timed out. Wed Aug 14 12:27:25 2013
1912 0 PPE-0 'server_svc_cfg_NSSVC_DNS_192.168.204.50:53(SVC_CTXANG_DNS1)' DOWN Wed Aug 14 12:27:25 2013
1913 0 PPE-0 'server_svc_cfg_NSSVC_DNS_192.168.204.51:53(SVC_CTXANG_DNS2)' DOWN Wed Aug 14 12:27:25 2013
1906 3910 PPE-0 'interface(1/1)' has been disabled Wed Aug 14 12:27:16 2013
1907 0 PPE-0 'interface(1/1)' DOWN Wed Aug 14 12:27:16 2013
1912 0 PPE-0 'server_svc_cfg_NSSVC_DNS_192.168.204.50:53(SVC_CTXANG_DNS1)' DOWN Wed Aug 14 12:27:25 2013
1913 0 PPE-0 'server_svc_cfg_NSSVC_DNS_192.168.204.51:53(SVC_CTXANG_DNS2)' DOWN Wed Aug 14 12:27:25 2013
1910 0 PPE-0 MonServiceBinding_192.168.204.51:53_(ping-default)(SVC_CTXANG_DNS2): DOWN; Last response: Failure - Probe timed out. Wed Aug 14 12:27:25 2013
1911 0 PPE-0 MonServiceBinding_192.168.204.50:53_(ping-default)(SVC_CTXANG_DNS1): DOWN; Last response: Failure - Probe timed out. Wed Aug 14 12:27:25 2013
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
nsconmsg –K newnslog –d consmsg
# nsconmsg -K newnslog -d consmsg
Displaying console message information
NetScaler V20 Performance Data
NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27
current time is Wed Aug 14 12:53:43 2013
LSM 1/1: Mon->Up on Link_Up
platform: NetScaler Virtual Appliance 450000 (9), manufactured at 2/17/2009
platform: serial HE2H91SCZ6
platform: unknown platform - using defaults!
LSM LO/1: Mon->Up on Link_Up
NetScaler: 420Mb of memory allocated
cell_pool: 0, (2 MB, 2 MB)
cell_pool: 0, map_addr 0xd2200000
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
nsconmsg –K newnslog –d oldconmsg
# nsconmsg -K newnslog.ppe.0 -d oldconmsg
Displaying debug performance information
NetScaler V20 Performance Data
NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27
current time is Wed Aug 14 10:17:10 2013
CPU:2.4% MEM:199419576 UP:00.00:00:14 since:Wed Aug 14 10:16:57 2013
current time is Wed Aug 14 10:17:17 2013
CPU:2.4% MEM:199462204 UP:00.00:00:21 since:Wed Aug 14 10:16:57 2013
current time is Wed Aug 14 10:17:24 2013
CPU:2.4% MEM:199481988 UP:00.00:00:28 since:Wed Aug 14 10:16:57 2013
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
nsconmsg –K newnslog –s ConLb=2 –d oldconmsg
# nsconmsg -K /var/nslog/newnslog -s ConLb=2 -d oldconmsg
Displaying debug performance information
NetScaler V20 Performance Data
NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27
current time is Wed Aug 14 10:17:10 2013
-------------------------------------------------------
NATSession : Free(6552)A(6553)InUse(1)
NATSession: Cur(Tcp[0] Udp[1] Icmp[0] Other[0])
NATSession: Op/s(Tcp[0] Udp[0] Icmp[0] Other[0])
Session: A:0 F:0 IUse:0 SEs: SIP:0 C:0 SSL:0 Svr:0 UserId:0 SIPDIP:0 DIP:0 SO:0
SSF: Conn (Srvr 0 Clnt 0) U:0
CM: Conn (Srvr 0 Clnt 0) Sessions PCB 0 NATPCB 0
Z(SIP[0], C[0], SSL[0] Server[0] SIPDIP[0] DIP[0] SO[0])
Mon: Probes: 0, Failed: 0
VIP(127.0.0.2:53:DOWN:WEIGHTEDRR): Hits(0, 0/sec) Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 1024
Pkt(0/sec, 0 bytes) actSvc(0) DefPol(NONE) override(0) newlyUP(0)
Conn: Clt(0, 0/sec, OE[0]) Svr(0) SQ(Total: 0 OnVserver: 0 OnServices: 0)
slimit_SO: (Sothreshhold: 0 [Ex: 0] Consumed: [Ex: 0 Borrowed: 0 TotActiveConn: 0] Available: 0
VIP(127.0.0.2:53:DOWN:WEIGHTEDRR): Hits(0, 0/sec) Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 1024
Pkt(0/sec, 0 bytes) actSvc(0) DefPol(NONE) override(0) newlyUP(0)
Conn: Clt(0, 0/sec, OE[0]) Svr(0) SQ(Total: 0 OnVserver: 0 OnServices: 0)
slimit_SO: (Sothreshhold: 0 [Ex: 0] Consumed: [Ex: 0 Borrowed: 0 TotActiveConn: 0] Available: 0-
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
IP Conflict
# cat /var/log/ns.log
Aug 14 12:39:11 <local0.info> 192.168.47.1 ipConflict (ipConflictAddr = 192.168.47.1, sysIpAddress = 192.168.47.1)
Aug 14 12:39:42 <local0.info> 192.168.47.1 last message repeated 169 times
© 2014 Citrix | Confidential – Do Not Distribute
Troubleshooting TechniquesCommon NetScaler issues/resolutions
HDD Issues Logging fails/var missing
Flash Issues Config fails to saveConfig saves partiallySync failsDevice fails to boot
Memory starvation Dropped sessionsCPU starvation All services failing
All VIPS downDegraded performance
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
NetScaler Data Collection
NetScaler software version including build – (from the ‘show version’ NetScaler Command Line Interface (NSCLI) command) – example: 10.1.124.13
Production setup or new installation
Whether an application/service that was working is now broken or whether the user wants to configure an application/service
Network topology information
What changes were performed on the NetScaler appliance prior to the issue
Any change(s) on the connected Switches, upstream Router, or backend server prior to experiencing the issue
ns.conf after saving the configuration - (from the save configuration NSCLI command)
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show techsupport I
> show techsupport
showtechsupport data collector tool - $Revision: #1 $!
NetScaler version 10.1
Creating /var/tmp/support ....
The NS IP of this box is 192.168.47.1
Current HA state: Primary (or this is not part of HA pair!)
All the data will be collected under
/var/tmp/support/collector_P_192.168.47.1_25Oct2013_11_12
Copying selected configuration files from nsconfig ....
Copying WebInterface configuration files (if WI is installed)...
... WI is not installed on this system. Nothing to copy.
Running shell commands ....
Running CLI show commands ....
Running CLI stat commands ....
Running vtysh commands ....
Determining newnslog files to archive....
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
... copied 6 files from this directory.
Copying core files from /var/core ...(last 5 files created within the last week)
NSPPE core (2/NSPPE-00-1077.gz) file present! Skipping this file because of size restrictions..
... copied 5 files from this directory.
Copying core files from /var/crash ...(last 5 files created within the last week)
... Nothing to copy...No files created within the last one week
Copying messages,ns.log,dmesg and other log files ....
Copying imported files and mapping files ...
Copying GSLB location database files ....
Archiving all the data into "/var/tmp/support/collector_P_192.168.47.1_25Oct2013_11_12.tar.gz"....Done.
Created a symbolic link for the archive with /var/tmp/support/support.tgz
/var/tmp/support/support.tgz ---- points to ---> /var/tmp/support/collector_P_192.168.47.1_25Oct2013_11_12.tar.gz
If this node is part of HA pair, please run it on the other node also!!
show techsupport II
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
techsupport archive structure
/etc
/flash
/nsconfig
/shell
/var/cfsynclog/clusterd/core/crash/download/log/netscaler/nslog/nsproflog/nssynclog
© 2014 Citrix | Confidential – Do Not Distribute
nstrace & nstcpdump
nstrace
Common syntax:Nstrace.sh –sz 0Nstrace.sh –sz 0 –filter “SOURCEIP = 10.198.4.10” –link enabled
Filter qualifiers and operators:SOURCEIP, SOURCEPORT, DESTIP, DESTPORT, SVCNAME, VSVRNAME, STATE==, eq, !=, neq, >, gt, <, lt, >=, ge, <=, le, BETWEENCompound filters using || and &&
nstcpdump
Common syntax:Nstcpdump.sh –X tcp port 80Nstcpdump.sh –w testcapture.cap –X src host 10.198.4.10 tcp port 80
Filter qualifiers and operators:tcpdump standard
© 2014 Citrix | Confidential – Do Not Distribute
Packet TracingDifferences between nstrace.sh & nstcpdump.sh
Nstcpdump.sh Nstrace.sh
Useful if traces are to be viewed on standard output
nstcpdump.sh –w <filename> option helps writing output to file
Useful for offline collection
nstrace.sh –sz 0 Saves traces in /var/nstrace in cap format
can be used with expressions so that you get to see filtered traffic
nstcpdump.sh host <IP> nstcpdump.sh port 21
useful for collection of traces in separate log files based on NICs
nstrace.sh –tcpdump1 –nic 1
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
nstrace.sh
# nstrace.sh --?
nstrace - utility to start NetScaler packets trace
usage: nstrace.sh [-h] [-nf <number_of_files] [-time <time>] [-m <mode>] [-nic <boolean>]
-h - prints this message - exclusive option
-nf - number of files to be generated in cycle (def. 24)
-time - seconds per file (def. 3600) (could be an expression)
-sz - size of the captured data (bytes from 60 to 1514)
-m - Capturing mode: sum of the values (def. 18):
-tcpdump - 0=nstrace-format (default) or 1=tcpdump-format
-nic - use separate trace files for each interface
(only works if -tcpdump option is set)
-name - name of the trace file
-filter - Filter expression for nstrace. The maximum length of filter expression is 255 and it can of following format: <expression> [<relop> <expression>]
-link - Log filtered connection's peer's (linked connection's) traffic. Works only with -filter option
-id - ID for the trace file name for uniqueness. Should be used only with -name option
-stop - can be used to disable tracing (when 'nstrace.sh' is run in the background)
#
-h - prints this message - exclusive option
-nf - number of files to be generated in cycle (def. 24)
-time - seconds per file (def. 3600) (could be an expression)
-sz - size of the captured data (bytes from 60 to 1514)
-tcpdump - 0=nstrace-format (default) or 1=tcpdump-format
-nic - use separate trace files for each interface (only works if -tcpdump option is set)
-name - name of the trace file
-filter - Filter expression for nstrace. The maximum length of filter expression is 255 and it can of following format: <expression> [<relop> <expression>]
-link - Log filtered connection's peer's (linked connection's) traffic. Works only with -filter option
© 2014 Citrix | Confidential – Do Not Distribute
Packet Tracing
nstrace.sh nstcpdump.shProprietary capture format TCPdump PCAP capture formatNative format captures more information Useful for live capture from CLIFiles are stored in /var/nstrace Option to write to a fileNeeds custom dissector in Wireshark (1.6+) Most TCPdump options supported
Works in standard Wireshark
© 2014 Citrix | Confidential – Do Not Distribute
Trace analysisnstrace.sh
By Default if nstrace.sh is executed from shell prompt
Trace files are stored in NetScaler proprietary (.cap) format
The trace capture runs for 1 hr. (3600 sec) if not interrupted
Files are cyclically numbered from 1-24. Trace mode is 6 (that is the TXB and RX packets are captured)
The size of the captured data for each packet is 164
© 2014 Citrix | Confidential – Do Not Distribute
Trace Analysisnstrace syntax examples
Command Purpose
# nstrace.sh Stores the traces in default (proprietary) format
# nstrace.sh -tcpdump 1 Begins to save the traces in the TCPDUMP format for a default 3600 seconds
# nstrace.sh -tcpdump 1 -nic 1 Logs the traces (in TCPDUMP format) into separate log files based on the NIC IDs
# nstrace.sh –nf <value> No of files to be generated in cycle by default is 24
# nstrace.sh –time <value> Seconds per file by default 3600 seconds
# nstrace.sh -sz 0 Size of the captured data( by default it is 164), -sz 0 is the entire packet length
# nstrace.sh –m Capturing mode: sum of the values (def. 6):1-Transmitted packets (TX)2 - Packets buffered for transmission (TXB)4 - Received packets (RX)
# nstrace.sh -stop Can be used to disable tracing (when 'nstrace.sh' is run in the background
© 2014 Citrix | Confidential – Do Not Distribute
Trace Analysisnstcpdump.sh example syntax
Command Purpose
# nstcpdump.sh <type> Possible types are host, net and port .If there is no type qualifier host assumed
# nstcpdump.sh <dir> Qualifiers specify a particular transfer direction to and/or from id. Possible directions are src, dst, src or dst and src and dst. If there is no dir qualifier, src or dst is assumed. src foo', `dst net 128.3', `src or dst port ftp-data'
# nstcpdump.sh <proto> ether, fddi, ip, arp, rarp, decnet, tcp and udp.
# nstcpdump.sh –c <value> Exit after receiving ‘value’ number of packets
# nstcpdump.sh –F –I –r Not supported and not to give them as options to the script
# nstcpdump.sh –w <file name>
Write the raw packets to file rather than parsing and printing them out. Read by typing tcpdump –r test
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
nstcpdump.sh
# nstcpdump.sh --?
Setting 1000 pages (8000 KB) of trace buffers ... Done.
Enabling all nic trace mode=6 ... Done.
Changing trace packet length from 0 to 0 ... Done.
tcpdump version 3.9.4
libpcap version 0.9.4
Usage: tcpdump [-aAdDeflLnNOpqRStuUvxX] [-c count] [ -C file_size ]
[ -E algo:secret ] [ -F file ] [ -i interface ] [ -M secret ]
[ -r file ] [ -s snaplen ] [ -T type ] [ -w file ]
[ -W filecount ] [ -y datalinktype ] [ -Z user ]
[ expression ]
Saving current trace data in file 'pipe' ... in TCPDUMP format
Disabling all nic trace ... Done.
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
Trace analysis
# nstcpdump.sh host 10.102.12.250
# nstcpdump.sh port 21
# nstcpdump.sh src host 10.102.12.250 and dst host 10.102.12.204
# nstcpdump.sh ‘(host 10.102.12.250 and host 10.102.12.204) or (host 10.102.12.204 and host 10.102.12.31)’
# nstcpdump.sh host storefront.nstipster.lab
Core Dumps/Crashes
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
Core dump directories
# ls -la
total 8
drwxrwxr-x 4 root nobody 512 Mar 4 09:41 .
drwxr-xr-x 31 root wheel 1024 Dec 4 10:06 ..
drwxrwxr-x 2 root nobody 512 Oct 8 21:28 1
-rw-r--r-- 1 root nobody 2 Mar 4 09:36 bounds
# ls –la 1/
total 10292
drwxrwxr-x 2 root nobody 512 Jan 22 11:24 .
drwxrwxr-x 4 root nobody 512 Mar 4 09:41 ..
-rw------- 1 root nobody 9881665 Dec 18 11:15 NSPPE-00-1094.gz
-rw------- 1 root nobody 603327 Dec 18 11:15 nscac64p-1189.gz
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
Packet Engine (nsppe) Process Crashing
Oct 19 08:52:46 <local0.alert> vpx1 nsppe: PE 0 (pid 1077) got signal 6; signal mask is 0x0 0x0 0x0 0x0
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NSPPE-00 (1077) unexpectedly died due to receiving signal
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 There may be a delay restarting process while collecting core dump on NSPPE-00 (1077)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 proc NSPPE-00 (1077) failure. Therefore initiating nCore NetScaler restart according to policy setting (0x29ac)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NetScaler restart may be delayed if collecting core dump for NSPPE-00 (1077)
Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 Pitboss declaring system failure: NSPPE-00 (1077) exited
Oct 19 08:53:44 <local0.info> vpx1 [44]: pitboss Sat Oct 19 08:53:44 2013 Deleting watch on NSPPE-00 (1077) for ()
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
/var/core
# cd /var/core/
# ls -la
total 14
drwxrwxr-x 6 root nobody 512 Oct 25 07:54 .
drwxr-xr-x 30 root wheel 1024 Oct 25 10:31 ..
drwxrwxr-x 2 root nobody 512 Sep 30 13:39 1
-rw-r--r-- 1 root nobody 2 Oct 25 07:54 bounds
root@vpx1# ls 1/
NSPPE-00-1077.gz nscac64p-1177.gz nsnetsvc-1086.gz
aslearn-1148.gz nscfsyncd-1158.gz nsrised-1164.gz
imi-1129.gz nsclfsyncd-1160.gz provserverd-1162.gz
monuploadd-1154.gz nsclusterd-1105.gz snmpd-1152.gz
nsaaad-1131.gz nsconfigd-1156.gz
nsaggregatord-1107.gz nsfsyncd-1110.gz
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
show techsupport
> show techsupport
----SNIP----
Copying core files from /var/core ...(last 5 files created within the last week)
NSPPE core (1/NSPPE-00-1077.gz) file present! Skipping this file because of size restrictions..
... copied 5 files from this directory.
Copying core files from /var/crash ...(last 5 files created within the last week)
... Nothing to copy...No files created within the last one week
----SNIP—---
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
Backup conf files
# cd /nsconfig/
# ls -lath ns*
-rw-r--r-- 1 root wheel 16K Oct 30 14:01 ns.conf
-rw------- 1 root wheel 16K Oct 30 14:01 ns.conf.0
-rw------- 1 root wheel 12K Oct 25 09:34 ns.conf.1
-rw------- 1 root wheel 12K Oct 25 08:09 ns.conf.NS10.1-124.13
-rw------- 1 root wheel 12K Oct 25 07:52 ns.conf.2
-rw------- 1 root wheel 12K Oct 24 15:17 ns.conf.3
-rw------- 1 root wheel 13K Oct 24 08:56 ns.conf.4
© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
Useful Links
CTX109304 Data Collection Procedure to Troubleshoot NetScaler Related Issueshttp://www.slideshare.net/davidmcg/common-pitfalls-when-setting-up-a-net-scaler-for-the-first-time http://support.citrix.com/search/basic?searchQuery=counters&refinement=Content+Type,Technotes&refinement=Product+Family,NetScaler CTX114999 How to Troubleshoot Authentication with Aaad.debughttps://taas.citrix.com/
© 2014 Citrix | Confidential – Do Not Distribute
Maximize your knowledge.Continue your journey with Citrix Education.
Recommended next step for hands-on technical training:
CNS-205 Citrix NetScaler 10 Essentials and Networking Identify the capabilities and functionality of the NetScaler Explain basic NetScaler network architectureObtain, install, and manage NetScaler licensesExplain how SSL is used to secure the NetScalerImplement NetScaler TriScale Technology, including Clustering
Visit bit.ly/NSCOURSE to save 10% now through April 30.**Not valid with any other promotions, packages or discounts. Applies only to new purchases. Regional limitations may apply.
© 2014 Citrix | Confidential – Do Not Distribute
AboutCitrix Services
Citrix Services make sureyou succeed with yourvirtualization programs.
How we can help
Citrix Education – The fastest, most efficient way toget your team the virtualization skills they need. Online,on-site or in class.citrix.com/training
Citrix Consulting – Intensive engagements forcomplex, critical or just plain massive projects.citrix.com/consulting
Citrix Support – Always-on support services thatleverage everything we know about best-practicedeployment and maintenance.citrix.com/support
Educate | Guide | Support | Succeed
© 2014 Citrix | Confidential – Do Not Distribute
• 40 insider troubleshooting tips
• Covering XenDesktop, XenServer, XenApp and NetScaler
• Citrix Support top engineers
• FREE eBook
• Citrix Auto Support
• Now available!
Secrets of the Citrix Support Ninjas
© 2014 Citrix | Confidential – Do Not Distribute
Premier Support Calculator
Check it out
Work better. Live better. Use NetScaler