2440: 141 web site administration web server monitoring and analysis instructor: enoch e. damson
TRANSCRIPT
The University of AkronSummit College
Business Technology Dept.2440: 141
Web Site Administration Web Server Monitoring and Analysis
Instructor: Enoch E. Damson
Monitoring and Analyzing SystemsMonitoring operating systems, Web servers,
applications, etc typically involves analyzing log filesLog files – contain information recorded by
the operating system in response to certain events
Monitoring and Analyzing the Web Server Environment 2
Monitoring Operating SystemsLogs are used to detect problems
OS, application, or security problemsVarious tools can monitor performance
Monitoring and Analyzing the Web Server Environment 3
Monitoring WindowsPerformance monitoring allows you to
compare system performance over timeWindows Task Manager highlights CPU and
memory usageYou can modify services to notify you if a
service fails
Monitoring and Analyzing the Web Server Environment 4
Windows Event Viewer The event viewer contains six event types shown in the left
pane
Monitoring and Analyzing the Web Server Environment 5
Windows Event LogsSystem and application events display
three levels of messagesInformationWarningError
Because many messages can be generated, a filter focuses on what you want to see
Over time, the logs fill up so you should clear them or save them
Monitoring and Analyzing the Web Server Environment 6
Monitoring LinuxLogging is controlled by the syslogd daemonBelow are some facilities which represent
daemons using syslogd
Monitoring and Analyzing the Web Server Environment 7
Eight Levels of Message Priorities in syslogd
Monitoring and Analyzing the Web Server Environment 8
Web Server Log FilesFiles that keep track of Web server
transactionsMost Web servers write two log files to
disk:Access log – contains a line for each Web
server requestError log – contains a line for each generated
error responseWhen log files grow:
A common practice is to put the log files on a separate drive or partition
A better solution is to rotate the log files Rename or remove the log files at regular intervals
(weekly, monthly, etc)Monitoring and Analyzing the Web Server Environment 9
Web Server Log File FormatsMost Web servers support at least two
logging formats:Common Logfile Format (CLF)Extended Logfile Format (ELF)
Most Web servers also allow the administrator to specify a custom format, along with the above formats
A standard logfile format makes it easier for users to understand files from different serversAllows third-party logfile analysis tools to support
many different Web serversMonitoring and Analyzing the Web Server Environment 10
Common Logfile Format (CLF)The NCSA and CERN Web servers first used this
file formatMany Web servers now support this format (IIS,
Apache, Netscape Enterprise, etc)Each line in the file represents a unique requestHas a fixed format with seven fields to be logged:
remotehostrfc1413authuser [date]“request”statusbytes
Monitoring and Analyzing the Web Server Environment 11
Common Logfile Format… remotehost – remote (client) hostname or IP number rfc1413 – remote username
rfc1413 defines a protocol used to determine the identity of a client that requests a resource from the server
Seldom used on Internet servers because it slows the server’s response A “-” is entered into the log if the server is unable to determine a
userid authuser – when required, the username by which the user has
authenticated is provided A “-” is used for normal unrestricted requests
[date] – date and time of the request Enclosed in brackets for potential spaces
“request” – HTTP request line exactly as it came from the client Enclosed in quotes for potential spaces
status – HTTP status code returned to the client bytes – content length of document transferred Example:
127.0.0.1 - - [24/Oct/2006:09:11:55 -0500] "GET /test.asp HTTP/1.1" 200 626Monitoring and Analyzing the Web Server
Environment 12
Extended Logfile Format (ELF)Used to log more information or omit
certain fieldsAllows the administrator to specify exactly
which fields to log and in what orderEach represents a request like CLFs but
the beginning of the file also contains some configuration directivesEach directive line begins with a #Two directives are required and must precede
all entries in the log file: Version – specifies the version of the ELF to use Fields – specifies what data to record in the logfile
Monitoring and Analyzing the Web Server Environment 13
Extended Logfile Format…Example:
#Software: Microsoft Internet Information Services 5.1#Version: 1.0#Date: 2006-10-27 03:04:57#Fields: date time c-ip cs-method cs-uri-stem sc-status sc-bytes cs-version 2006-10-27 03:04:57 127.0.0.1 GET /test.asp 200 626 HTTP/1.1
The fields directive here specifies 8 out of several available fields: date – client request date time – client request time c-ip – client IP address cs-method – HTTP request method cs-uri-stem – file requested by client sc-status – HTTP status code returned to the client sc-bytes – number of bytes sent from server to client cs-version – version of HTTP used by client to connect to the server
Monitoring and Analyzing the Web Server Environment 14
Error LogsContains informational messages and debugging
informationUseful for:
Finding problems with the serverDebugging server-side programs and new configurations
Most server packages allow the administrator to control what types of messages are logged to the error log fileThe format is usually not configurable like ELFs but
allows some flexibility in choosing the severity and type of messages to log
E.g only critical messages may be logged if a server is running smoothly
Monitoring and Analyzing the Web Server Environment 15
ReferrersDetermines what Web page was used by
the client to access a serverMay be the URL of a search engine or any Web
site with a link to the Web serverA “-” is used if there was no Referrer header
sentThe Referrer header is not sent in the following
circumstances: The users enters the URL by hand The user clicked on a link to regular file and not a Web
page on a public site The user loaded the URL from a bookmark file The Referrer URL is on a private (internal) Web site The user or browser has disabled sending the Referrer
header Monitoring and Analyzing the Web Server Environment 16
Monitoring IISIIS has specific counters for use in the
Performance MonitorThe System event viewer provides specific
information IIS has extensive logging capabilities
There are default log formats used by various third-party applications that analyze logs
Monitoring and Analyzing the Web Server Environment 17
Monitoring ApacheError LogsBy default, syslogd sends Apache messages
to /var/log/boot.logLocation of the error log
ErrorLog logs/error_logLogs refers to /var/log/httpd
You can create a different error log for each virtual host
Monitoring and Analyzing the Web Server Environment 18
Monitoring ApacheTransfer LogsTransfer logs tell you about the use of
your Web siteThe default log is based on a combined
formatDetermined by the CustomLog directive in the
configuration file (httpd.conf)There are a number of sample formats
By default, logs are stored in /var/log/httpd/access_log
Monitoring and Analyzing the Web Server Environment 19
Monitoring DNSBIND uses a logging statement that you
configure in named.confBIND defines logging in two parts:
Channel defines where logging is sentCategory defines what will be sent
If the channel is going to a file, use the versions option to define the number of backupsSize option sets maximum size of the fileprint-time adds the date and time to the file
Monitoring and Analyzing the Web Server Environment 20
BIND Categories
Monitoring and Analyzing the Web Server Environment 21
Monitoring Exchange ServerExchange server uses the application
portion of Event viewerYou can enable four types of logs
audit – access to mailboxes protocol – commands used for SMTP, etcmessage tracking – senders and receiversdiagnostic – analyze detailed problems
Monitoring and Analyzing the Web Server Environment 22
Analysis Tools for the Web ServerAnalysis tools extract system data from
logs and format the dataFor IIS, one of the popular tools is
WebTrendsHelps you determine the source of Web
trafficDetermines which pages are most popularSeveral different reports
123LogAnalyzer is available for both IIS and ApacheMany reports are similar to WebTrends
Monitoring and Analyzing the Web Server Environment 23
Log File AnalysisSimply looking at log files can provide a lot of
information about activities or requests on a serverSimply counting the number of lines in an access
log file can help determine the number of hitsLog files may be reviewed regularly to find the
common errors loggedSome of the common errors include:
Dead links Requests for non-existing files CGI scripts not working properly Permissions problems
Some of the open-source log analyzers are: Analog (http://www.analog.cx) Webalizer (http://www.mrunnix.net/webalizer) Report Magic (http://www.reportmagic.org)
Monitoring and Analyzing the Web Server Environment 24
Statistics With the help of several log analyzer programs, some of the
statistical information that can be extracted include: Most requested pages Top entry pages (the first page clients enter a site through) Most used browsers Bandwidth utilization Most active domains Top referring sites and URLs Error counts Information about search engines (most common search engines,
common queries, etc) Some of the widely used commercial log analyzer products
include: WebTrends (http://www.webtrends.com) Wusage (http://www.boutell.com/wusage)
A database could also be used to store log information to increase efficiency of logging and report generation Not all Web servers support logging to a database
Monitoring and Analyzing the Web Server Environment 25