1. Advanced Tools and Techniques forTroubleshooting NetScaler AppliancesAndrew Redman | Lead Escalation EngineerCitrix Support WebinarAugust 26 & 27, 2014

2. 2 2014 Citrix. Confidential.NetScaler System OverviewTroubleshooting Tools & TechniquesCase StudiesResourcesConclusionQ&AAgenda 3. NetScaler System Overview 4. Key NetScaler ProcessesProcess Descriptionns_master/NSPPE4 2014 Citrix. Confidential.nsvpndnsaaadnsconfnsauthdnslog.shnssyncnsreadfilenslcdnsfsyncdnsnetsvcnsconmsgnscollectRuns Citrix NetScaler OSSSL VPN File TransferRBA and SSL VPN external authorizationWrites the ns.conf fileCLI authenticationControls logging for the newnslogHA synchronizationUsed to read SSL certificate filesRuns the front panel LCDSynchronizes bookmarks and SSL certificatesUsed by the GUI for configuration changesControls writing of the newnslogStatistics gathering for historical purposes 5. NetScaler File System/var (hard drive) Logs - /var/log & /var/nslog5 2014 Citrix. Confidential.Install - /var/nsinstallTrace - /var/nstraceCore Dumps - /var/crash & /var/core/flash (flash drive) Config - /flash/nsconfigSSL Certificates - /flash/nsconfig/ssl 6. NetScaler File System (cont.)/flash (cont.) User Monitors - /flash/nsconfig/monitors6 2014 Citrix. Confidential.Custom Options - /flash/nsconfig/ (ram drive) OS - (operating system) 7. Troubleshooting Tools & Techniques 8. NetScaler Tech Support Bundle> show techsupportCriticalSystem DataIn-DepthPerformanceMonitoringStatsDetailed LogFilesUSERCommandLogging/var/tmp/support/collector_P_10.10.10.10_21Apr2014_21_42_tar.gz 9. The NetScaler Tech Support Bundle9 2014 Citrix. Confidential. 10. Citrix Insight Services10 2014 Citrix. Confidential. 11. 11 2014 Citrix. Confidential.Data Collection Analyze DataSingle Mission Data CollectionFAQ: http://support.citrix.com/article/CTX131233TailoredRecommendations 12. https://taas.citrix.com12 2014 Citrix. Confidential. 13. Technical Issues Flagged13 2014 Citrix. Confidential. 14. Investigate Issues FurtherThe BELL14 2014 Citrix. Confidential. 15. Crash File(s) Location15 2014 Citrix. Confidential.nscollect-542.gz 16. Intuitive Navigation16 2014 Citrix. Confidential.Select a differentnewnslog file to view 17. Detailed Graphs17 2014 Citrix. Confidential.Mouse over graphto see more detailInformativereference legendDownload the dataas an excel sheet 18. IPMI Intelligent Platform Management InterfaceChange NetScaler18 2014 Citrix. Confidential.Default LOM IP Address: http://192.168.1.3IP AddressObtain HealthMonitoring DetailHarvest SerialNumberDetermine MACAddress 19. Common CLI Show CommandsCommon show commands for system information:show node, show info, show licenseCommon show commands for vserver and service:show lb vserver, show cs vserver, show service, show persistencesessionshow connectiontableOther common show commands:show route, show ip19 2014 Citrix. Confidential. 20. Common CLI Stat CommandsCommon stat commands for system information:stat ns, stat cpu, stat interfaceCommon stat commands for vserver and service:stat lb vserver, stat cs vserver, stat serviceOther common stat commands:stat dns, stat ssl, stat http20 2014 Citrix. Confidential. 21. Leveraging nsconmsgNsconmsg common use cases:View eventsView console messagesView statisticsDebug system countersDebug load balancing issuesDebug CPU/Memory utilization21 2014 Citrix. Confidential.Make absolutelysure that youuse a capital -Kand NOT alower-case -k 22. Example nsconmsg Usage# cd /var/nslog# nsconmsg -K newnslog -j fqdn-ssl-vip -s ConLb=1 -d oldconmsg22 2014 Citrix. Confidential.newnslogcurrent log filefqdn-ssl-vipname of vserverConLb=1LB stats 23. Displaying debug performance informationNetScaler current V20 time Performance is Sun Mar Data23 18:33:43 2014NetScaler NS10.1: Build 123.11.nc, Date: Feb 24 2014, 17:30:43current time is Sun Mar 23 18:33:43 2014-------------------------------------------------------NATSession : Free(6553)A(6553)InUse(0)NATSession: Cur(Tcp[0] Udp[0] Icmp[0] Other[0])NATSession: Op/s(Tcp[0] Udp[0] Icmp[0] Other[0])Session: A:0 F:0 IUse:0 SEs: SIP:0 C:0 SSL:0 Svr:0 UserId:0 SIPDIP:0 DIP:0 SO:0SSF: Conn (Srvr 0 Clnt 0) U:0Mon: Probes: 434562009, Failed: 15CM: VIP(Conn 10.54.169.75:(Srvr 0 Clnt 0) Sessions 443:UP:PCB LEASTCONNS): 0 NATPCB 0Hits(7317, 0/sec) Mbps(0.00) Pers(OFF)Z(SIP[S(10.54.148.201:0], C[0], SSL[0] Server[80:0] UP) SIPDIP[Hits(0] 7317, DIP[0] SO[0/sec, 0])P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0Mon: Probes: 434562009, Failed: 15VIP(10.54.169.75:443:UP:LEASTCONNS): Hits(7317, 0/sec) Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 0S(10.54.148.201:80:UP) Hits(7317, 0/sec, P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0 kbits) RspTime(0.00 ms)Load(0) LConn_Idx: (C:0; V:0,I:1)-------------------------------------------------------CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 201423 2014 Citrix. Confidential.kbits) RspTime(0.00 ms)CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 2014 24. # nsconmsg -K newnslog -j -s ConLb=1(2 or 3) -d oldconmsg | more# nsconmsg -K newnslog -s ConMon=1 -d oldconmsg# nsconmsg -K newnslog -s ConMEM=1 -d oldconmsg# nsconmsg -K newnslog -s ConSSL=1 -d oldconmsgConDebug - DebuggingConLb - Load BalancingConMon - Monitoring ProbesConMEM - Memory ManagementConCSW - Content SwitchingConSSL - SSL OffloadConCMP - CompressionConIC - Integrated Caching24 2014 Citrix. Confidential. 25. Log File Analysis# cd /var/log# zgrep -i cmd_executed ns.log* | more (the -i means ignore CASE)ns.log:Mar 20 16:45:06 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED2947 0 : User nsroot - Remote_ip 10.13.73.65 - Command "login nsroot "********"" - Status "Success"ns.log:Mar 20 16:45:06 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED2948 0 : User nsroot - Remote_ip 10.13.73.65 - Command "show ns license" - Status "Success"25 2014 Citrix. Confidential.interface down vServer down panic signaled 26. NetScaler + Wireshark = thumbs uphttp://www.wireshark.org26 2014 Citrix. Confidential. 27. RED HOT Wireshark Tip27 2014 Citrix. Confidential.CustomColumnsCustom MenuOptionsPre-buildCustom FiltersMuch FasterAnalysisGet the red hot details on how to empower your default Wiresharkconfiguration in the Reference Section at the end of this presentation. 28. Troubleshooting Techniques & Case Studies 29. Top Tips29 2014 Citrix. Confidential.Use Citrix Insight ServicesPay attention to the issues!Note the highlighted counter(s)Use nsconmsg to see even more detailCorrelate time-frames in other log filesAchieve root cause analysis faster! 30. Case #1 - High Availability Synchronization 31. Same type ofappliance31 2014 Citrix. Confidential.HA Pre-requisitesSame firmwareversionRecommendsame nsrootpasswordSame RPCNode passwordOpen requisiteTCP portsPrimary NetScaler Secondary NetScaler 32. Insight Services Flags The Issues32 2014 Citrix. Confidential. 33. The HA Pair Struggled To Synchronize# nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved (nic_err_bdg_muted)57520 0 71837018 nic_tot_bdg_mac_moved interface(0/1)57521 0 71837018 nic_tot_bdg_mac_moved interface(0/2)9861 0 65 nic_err_bdg_muted interface(0/1)9862 0 65 nic_err_bdg_muted interface(0/2)33 2014 Citrix. Confidential.71,837,018 MAC Moves65 Interface Mutes 34. The newnslog Time-Frame# nsconmsg -K newnslog -d setimeDisplaying start and end time informationNetScaler V20 Performance DataNetScaler NS9.3: Build 54.4.nc, Date: Dec 20 2011, 22:44:41start time Fri Feb 28 21:49:58 2014total duration 00.00:03:30end time Fri Feb 28 21:53:28 2014total duration 00.00:03:30data size 1,718,949 bytes34 2014 Citrix. Confidential. 35. Case #2 - XA/XD Slow Performance 36. Smartphones XenDesktopTablets XenApp36 2014 Citrix. Confidential.InsightServicesCritical InsightGleanedPreventativeApproachDontUnderestimateXA/XD Slow Performance 37. Insight Services Again Flags The Issues37 2014 Citrix. Confidential.http://support.citrix.com/article/CTX136926 38. Performance Was Extremely Latent# nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved4263 0 23 nic_tot_bdg_mac_moved interface(0/1)4264 0 51 nic_tot_bdg_mac_moved interface(1/1)4265 0 28 nic_tot_bdg_mac_moved interface(1/2)38 2014 Citrix. Confidential.23, 51 & 28 MAC Moves 39. Networking Issues Again?# nsconmsg -K newnslog -d statswt0 | grep nic_err4274 0 1995 nic_err_rl_pkt_drops interface(1/1)4275 0 40736 nic_err_rl_pkt_drops interface(1/2)4276 0 1995 nic_err_rl_rate_pkt_drops interface(1/1)4277 0 40736 nic_err_rl_rate_pkt_drops interface(1/2)4678 0 42731 allnic_err_rl_rate_pkt_drops39 2014 Citrix. Confidential.System Limits ExceededRate-limited Packets! 40. 40 2014 Citrix. Confidential.The Moral of the StoryLeverage Citrix Insight ServicesLeverageInsight ServicesPay AttentionGain QuickInsightDig IntonsconmsgOn Target forSuccess! 41. Resources 42. Helpful ResourcesComprehensive NetScaler CountersWireshark Developer EditionsCustomizing Wireshark TutorialCitrix Insight Services ForumNSTRACE OptionsHow To Manage VLANs, Interfaces and Subnets42 2014 Citrix. Confidential. 43. Conclusion 44. What Weve Actually CoveredAn Overview of the NetScaler System to give you a high-level understanding of the core system.I shared with you some excellent Troubleshooting Tools that are available at your disposal.I also discussed a few key Troubleshooting Techniques that you can use to diagnose issues.I then highlighted two different Case Studies leveraging the tools & techniques that I shared withyou in the presentation.In addition I provided you with a few Resources for your future reference and edification.44 2014 Citrix. Confidential. 45. Fuel your talent with continuous learning.93% of Citrix Education students became more effective in their role after attending a course.TVID: CFB-61B-A26Citrix Education offers the following technical training for Networking professionals:CNS-205: Citrix Netscaler 10 Essentials and NetworkingCPE-350: Citrix NetScaler 10 Essentials and Networking Practice ExamCNS-301: Citrix NetScaler 10 Advanced Implementation45 2014 Citrix. Confidential.Visit (bit.ly/05Webinar) to save 10% off through September 30**Not valid with any other promotions, packages, discounts or practice exams.. Applies only to new purchases. Regional limitations may apply. 46. Simplify your journey, let us guide you.Accelerate your implementation and minimize risk by taking advantage of CitrixConsulting. Youll get the expertise of certified Citrix Consulting Architects tosuccessfully deploy Citrix solutions in any phase of your project.93% of Citrix Education studentsbecame more effective in their roleafter attending a course.TVID: CFB-61B-A26Visit bit.ly/CTXConsulting to learn more about our proven methodology.46 2014 Citrix. Confidential. 47. 47 2014 Citrix. Confidential.WORK BETTER. LIVE BETTER.