Download - SharePoint Troubleshooting Tools & Techniques

1 | SharePoint Saturday Chicago 2013Twitter: @SPSChicago Hashtag #SPSChicago

Available tools/techniques used by Microsoft Premier Support Engineers.

Manuel LongoSenior Manager – SharePoint Consultant – Sogeti US

SharePoint Troubleshooting Tools and Techniques


Manuel Longo Over 13 years of experience on Microsoft

Products & Technologies. Microsoft SharePoint Premier Field

Engineer for over 5 years. Microsoft Technical Solutions Professional

for over 3 years. Microsoft Consulting Services Developer

for over 1 year. Developer Consultant since 1998.


Outline & Agenda Troubleshooting Overview SharePoint Interdependencies SharePoint Performance/Capacity Analysis Troubleshooting Tools

Diagnostics Logging Data Collection Data Analysis Debugging


Troubleshooting Overview


Troubleshooting Overview Understand the system

Purpose Configuration Interdependencies

What is the issue/problem? End user experience IT Pro experience

Narrow the scope Happen to everyone or just a few users? All clients? Only ones in remote site? What’s common when it happens? What’s not when the problem is

absent?

Reproduce in Test Environment as possible


Root Cause Analysis (RCA) Identify the factors that resulted in the

nature, the magnitude, the location, and the timing of the harmful outcomes (consequences) of one or more past events in order to identify what behaviors, actions, inactions, or conditions need to be changed to prevent recurrence of similar harmful outcomes and to identify the lessons to be learned to promote the achievement of better consequences. "Success" is defined as the near-certain

prevention of recurrence.

http://en.wikipedia.org/wiki/Root_cause_analysis




Root Cause Analysis (RCA)1. Define the problem.2. Gather data and evidence.3. Identify the causes.4. Classify causes into causal factors

and root causes.5. Identify harmful factors.6. Identify corrective actions.7. Identify solutions.8. Implement recommended root

cause corrections.9. Ensure effectiveness.

Gather

Analyze

Correct


Diagnosing What is Changed“it was working 2 days ago, and suddenly started failing…”

…if it was working before, something must have changed.

Four questions to ask:1. What’s changed?2. What’s changed?3. No really…what’s changed?4. Something must have changed, what is it ?!?


SharePoint Interdependencies

10 | SharePoint Saturday Chicago 2013

Twitter: @SPSChicago Hashtag #SPSChicago

SharePoint Interdependencies SQL Server Active Directory (Kerberos) Name Resolution (DNS) Network IIS Service Applications Client side

HTTP/HTML/JavaScript



SharePoint Performance/Capacity



Low Performance/CapacityCheck SharePoint System Requirements Hardware and Software Browser Support IP Support Software Boundaries and limits

Boundaries vs. thresholds vs. limits Limits by hierarchy, by feature

Capacity Management vs. Capacity Planning Performance and Capacity Test Results Storage and SQL Server Capacity

Planning Monitoring!

http://technet.microsoft.com/en-us/library/cc262485.aspx



Troubleshooting Tools



Troubleshooting Tools Diagnostics Logging Tools

Event logs, IIS logs, Setup logs*, Upgrade logs*, Configuration logs*, ULS logs*, SharePoint Logging Database*.

Data Collection Tools SPSFarmReport*, Process Monitor, Process Explorer, Performance

Monitor.

Data Analysis Tools Log Parser, PAL, Fiddler,, Network Monitor/Wireshark, SQL profiler,

SQLDiag, Kerberos Tools, SPDiag*, ULS Viewer*.

Debugging Tools MIISClient , SPDisposeCheck*, MSOCAF, Developer Dashboard*,

IFilter View*, Debug Diag, WinDbg.

* - SharePoint specific.



Diagnostics Logging Tools Event logs IIS logs Setup logs* Upgrade logs* Configuration logs* ULS logs* SharePoint Logging Database*



Event Logs Have the same

purpose as any operating system or application logs.

Help predict and identify the sources of a problem.

Can help diagnose problems with SharePoint.

An error from the SharePoint Health Analyzer showing the need for an update



Finding the issue…(which one?!)

Keep environment clean



IIS Log Files Record data for a SharePoint web server in

various fields, such as Client IP Address, Server IP Address, Method, HTTP Status, Cookie, Protocol Substatus, etc. Contain a hyphen (-) in fields with no values and a plus sign (+) in fields with nonprintable characters.

Can be analyzed using Log Parser.



IIS Log Configuration

Configuring IIS Logs to include additional data columns not included by default and how identify logs for particular web sites.

Demo



SharePoint Setup Logs SharePoint setup log

Helps examine setup failures. Contains the list and status of the MSI packages

installed for SharePoint 2010. Stored in the temp directory of the user who initiated

setup. Foundation log named: Microsoft SharePoint Foundation

2010 Setup (YYYYMMDD-HHMMSS-SSS).log Server log named: SharePoint Server

Setup(YYYYMMDD-HHMMSS-SSS).log Uninstall log

Help determine if the MSI packages chained by the catalyst are uninstalled completely.



SharePoint Configuration Logs Generated by the post-setup configuration tool,

PSConfig.exe and its GUI based tool, “SharePoint Configuration Wizard”.

Are created in the %CommonProgramFiles%\Microsoft Shared\web server extensions\14\LOGS folder.

Contain errors that occur during the post-setup configuration process.

Are easiest to review when imported to Microsoft Excel or another third-party log file analyzer.

Indicate whether each entry is of information or error type. Can be referenced, along with event logs, to obtain more

information about post-configuration errors displayed on the screen.



SharePoint Configuration Logs Error on the

screen.

Error details in the configuration log.

Error details in the event log.

Begin trace logging for SharePoint 2010 Products Configuration Wizard. Version 14.0.4762.100007/10/2010 12:51:44 1 INF Entering function PsconfigUserInterfaceMain.Main07/10/2010 12:51:44 1 INF Entering function Common.SetCurrentThreadCultureToInstalledCulture07/10/2010 12:51:44 1 INF Entering function Common.SetThreadCultureToInstalledCulture07/10/2010 12:51:44 1 INF Current thread culture is English (United States), current thread ui culture is English (United States), installed culture is English (United States)07/10/2010 12:51:44 1 INF Leaving function Common.SetThreadCultureToInstalledCulture



SharePoint Unified Logging System Are also referred to as trace logs in the

SharePoint GUI. Must have consistent location across all servers in

a farm. Contain over 266 categories of events. Contain nine columns: Timestamp, Process, TID,

Area, Category, EventID, Level, Message, Correlation.



ULS Logging – Best Practices Change the drive to which the logs write. Restrict log disk space usage. Use the Verbose setting sparingly. Backup the logs regularly. Enable Event Log Flood Protection.

Use ULSViewer (demo later on in this module)



SharePoint Logging Database Stores all SharePoint usage and health data, such as ULS

trace log data, event log data, blocking SQL queries, crawl and query statistics, feature usage, page requests, etc.

Is part of the Usage and Health Data Collection service application.

Has a public documented schema. Can be queried directly or written to by third-party

applications. Allows creating custom logging and diagnostic

providers. Has a customizable retention policy.

Retains detailed data for 14 days by default. Maximum 31 days of retention.

Configure it before you create other service applications. Otherwise it will be created automatically.



Logging Database - Schema Logging database is tuned to support a heavy

load of simultaneous writes Up to 5000 transactions/second in parallel Should be put on a separate disk, if possible

The database can be used in various scenarios, including: Poor crawl or query performance SQL queries that are causing blocking Determining how widely a feature is actually

used Basic configuration is divided into two categories

—usage data and health data



Logging Database - Reporting Table partitioning makes it easier to

use the provided views or create custom queries to view data in the logging database.

Most tables are partitioned for performance reasons.

Views are provided for usability. Excel can be used with the Excel Web

App or Web parts to create quick custom reports.

Some out-of-the-box reports make use of the logging database (next section).



SharePoint Correlation IDs GUID identifiers that are internally associated with every

request, and are displayed with error messages. Using Windows PowerShell administrators can track down

the correlation ID in the ULS log and discovers the cause of the error.

Correlations IDs can be found in all the different Diagnostic Logs including: Event Logs. ULS Logs. SharePoint Logging Database. SQL Server (via Profiler) under events RPC:Starting and

SP:Starting events.



Data Collection Tools SPSFarmReport* Process Monitor Process Explorer Performance Monitor



SPSFarmReport Executable that lists all farm configuration

settings. Get it from CodePlex:

spsfarmreport.codeplex.com Available for MOSS 2007 and SharePoint 2010. Reports are best viewed in IE, as you can show

and hide each section (if you enable scripts) Useful for creating a baseline,

and then running periodically to check what may have changed.



Process Monitor Combination of old Filemon and Regmon

utility. Catalogs interaction between the file

system and registry with filtering.



Process Explorer Useful for understanding what is going on inside the

process. Shows currently active processes in the top windows. Bottom window shows either:

Handles for the selected process. DLLs and memory mapped files opened by the process.



IIS Failed Request Tracing XML based files stored by default in \

inetpub\logs\FailedReqLogFiles\<W3SVC instance> Enabling consists of two steps:1. Enable Failed

Request Tracing.2. Configure your failure

definitions.



Configuring IIS FREB

How to enable Failed-Request Tracing for the Site and configure the Log File Directory and failure definitions.

Demo



Performance Monitor Captures data from the local machine or a remote

machine on different performance counters in either real time or regularly scheduled intervals.

Data can be stored and read in comma separated, tab separated, SQL, or binary format.

Common Counters: CPU. Disk. Memory. Network Interface.



Data Analysis Tools Log Parser Performance Analysis of Logs (PAL) Fiddler Network Monitor or Wireshark SQL profiler SQLDiag Kerberos Tools SPDiag* ULS Viewer*



Log Parser Provides universal query access to txt

based data sources; such as log files, XML files, and CSV files.

It was intended for use with the Windows operating system.

Free download from downloads.microsoft.com and was included with the IIS 6.0 Resource Kit tools.$ logparser <options> <SQL expression> $ logparser <options> <SQL expression>

$ logparser -e:IISW3C -q "SELECT date, time, cs-username FROM *.log WHERE cs-uri-stem LIKE '%.aspx' ORDER BY date, time;"

$ logparser -e:IISW3C -q "SELECT date, time, cs-username FROM *.log WHERE cs-uri-stem LIKE '%.aspx' ORDER BY date, time;"



Logs parsing with LogParser

Use LogParser to query IIS and Diagnostic logs.

Demo



Performance Analysis of Logs Wizard-based tool that generates a Windows

PowerShell script to analyze perfmon logs against best practice thresholds.

Thresholds are included for major Microsoft products like IIS, SharePoint, SQL Server, Exchange, etc.

Some thresholds change based on user entered criteria or computer role.

Results are created as a HTML based report.



PAL - Reports



Performance Analysis with PAL

Use PAL to process performance monitor logs, scheduling the report generation.

Demo



Fiddler A GUI-based tool to examine HTTP traffic

to a browser. Looks at additional statistic information.



Analysis of HTTP Traffic with Fiddler

Analysis of HTTP traffic opening Central Administration and SharePoint Sites and the effect of caching.

Demo



Network Monitor (or Wireshark) Network packet sniffing tool. Free to download from

downloads.microsoft.com Many features for filtering, and adding rules to

make displaying data clear. Process tracking. Grouping by network

conversation. Support for over 300 public

and Microsoft proprietary protocols.

Simultaneous capture sessions.

Wireless Monitor Mode with supported wireless NICs.

Real-time capture and display of frames.

Reassembly of fragmented data.

Sniffing of promiscuous mode traffic.

Can read libpcap capture files. API to access capture and

parsing engine.



SQL Server Profiler Using SQL Server Profiler, you can

do the following: Create a trace that is based on a

reusable template. Watch the trace results as the

trace runs. Store the trace results in a table. Start, stop, pause, and modify

the trace results as necessary. Replay the trace results. Correlate performance monitor

data. Analyze deadlocks.

Users must have the ALTER TRACE permission to run.



SQL Diag SQLdiag can collect the following types of

diagnostic information: Windows performance logs. Windows event logs. SQL Server Profiler traces. SQL Server blocking information. SQL Server configuration

information.



Kerberos Troubleshooting Examines common settings that

contribute towards Kerberos authentication and delegation DelegConfig

Verify kerberos tickets on a client machine KerbTray Klist (WS2008/R2)

SPN configuration SetSPN –x and -a



SharePoint Diagnostic Studio Aggregates multiple sources of information

together about a farm for both collection and presentation of the data.

Can be installed on a farm server or remotely. Must have Farm Administrator rights. Projects are created within the tool to access the

data.



ULS Viewer Parse and open multiple SharePoint ULS

logs (will concatenate them if you select multiple)

Reorder and resize columns, sort on any column.

Filter by Severity, Category, and Process or a custom text filter/search.

Group multi-line stack traces into single log entries for easy viewing and copy/paste.

View message easily in separate pane (No more scrolling in notepad).



Analysis of SharePoint Logs with ULSViewer

Reviewing the filtering and highlighting features of the ULSViewer analyzing SharePoint Logs in real time.

Demo



Debugging Tools MIISClient SPDisposeCheck* MSOCAF Developer Dashboard* IFilter View* Debug Diag WinDbg



FIM User Interface (MIISClient) User Profile Imports are handled by Forefront Identity

Manager in the background. Central Administration’s user interface does not provide a

high level of detail during profile synchronization. Enter MIISClient

Located at: C:\Program Files\ Microsoft Office Servers\14.0\Synchronization Service\UIShell\miisclient.exe

Allows you to see what issues SharePoint may have when attempting to sync with Active Directory in real time.

Provides error codes and user friendly messages.



SPDisposeCheck Command line tool that checks custom solutions

using the SharePoint object model to help ensure best practices and remove common causes of memory leaks.

It recursively searches and analyzes each managed module.

An updated set of “Do Not Dispose” rules have been added recently to improve the tool.

Can be integrated into Visual Studio. Add to the Tools menu Add SPDisposeCheck to your post build

commands.



MSOCAF Microsoft SharePoint Online Code Analysis

Framework. Required to use before uploading solutions to

Office 365. Can be used on premise as well. Picks up where SPDisposeCheck leaves off. An Internet connection is required to run the tool

(Clickonce Installer)



Developer Dashboard Helps perform advanced troubleshooting directly from the

page being rendered. Displays detailed diagnostics information, consolidated into

sections, for each page request. Helps test page customizations before deployment to

production. Is a farm-wide setting; turned off by default.

Can be set to on, off, or ondemand (2010 only). Can be configured via stsadm, which is easier than configuration with

Windows PowerShell:stsadm -o setproperty -pn developer-dashboard -pv On

Can be viewed by site collection administrators. Can be configured via Windows PowerShell for ondemand

usage.



Developer Dashboard

How to turn on Development Dashboard and use it for request diagnosis and troubleshooting.

Demo



iFilterView Use for troubleshooting search crawl issues

related to iFilters. It loads the iFilter for the file(s) with issues. It is also helpful for determining the names of

Crawled Properties returned by the iFilter. Download from

http://ifiltershop.com/downloads/IFilterView.zip



Debug Diag GUI based tool for analyzing process dumps to

examine for cases of crashes and hangs. Contains:

Debugger Service. Debugger Host.

Built-in reports for memory pressure, crash/hang analysis, and SharePoint analysis.



WinDbg Debugging tool used to capture and/or analyze

process dumps. A common tool for troubleshooting issues with

ASP.NET applications such as SharePoint.



Lessons Learned (…sort of)



YammerHow to Join the Conversation1. Download the Yammer phone app from the iTunes, Windows Phone, or

Android App Store.2. Join your company network, if you haven't already, at

http://www.yammer.com.3. Go to https://www.yammer.com/microsoftmidwestcustomers and click

"request an invitation". When you receive the invitation click "accept."4. Go to

https://www.yammer.com/microsoftmidwestcustomers/groups/spschicago and click "join." You will receive a welcome message in 24-72 hrs.

5. Once you have access you will be able to download posted presentations, ask questions of any of the subject matter experts and participate in any conversations going on in the group

http://www.yammer.com/

https://www.yammer.com/microsoftmidwestcustomers

https://www.yammer.com/microsoftmidwestcustomers/groups/spschicago

https://www.yammer.com/microsoftmidwestcustomers/groups/spschicago



Housekeeping Please remember to submit your session

evaluation using our mobile application Follow SharePoint Saturday Chicago on

Twitter @spschicago & use hashtag #spschicago



Thanks to Our Sponsors!

Download - SharePoint Troubleshooting Tools & Techniques

Top Related