Central Web Services at Fermilab

Presented

byJim Fromm

October 27,2006

Presentation Overview Why we have a central web farm Configuration of farm

Hardware Software

Automated tools for administration Monitoring of web farm Log processing Futures

Why a Central Webserver Farm? Eliminate experiments from worrying about

configuration/maintaining their own web server Keep on top of security issues Maintain up-to-date versions of apache/perl/python etc Maintain valid ssl certs Leverage our expertise to provide basic consulting for cgi

scripts, web page development etc.. The web managers are responsible ultimately, but if they ask nice…

Manage file system space (AFS has a quota mechanism, we keep track of when an area is getting full).

Overview of Central Webserver Farm 5 Computers – 1 Load balancing switch. 84 vhosts (and rising fast) 53 Additional web areas (conferences,

projects, computing…) 1300 Web Content Managers Web hits/year: > 220 million Staff:

Basically 0.5 of one person right now.

Hardware Configuration

Alteon

www01 www02 www03 www04(cgi)

www05(cgi)

AFS

Hardware Configuration - Details Alteon Load Balancing Switch

Configuration allows for traffic to be directed depending on type.

CGI scripts only executable on www04/05. Alteon AD3 switch, 8x 100Base-T ports with a

single GigE uplink Web Servers

Sun Netra X1 AFS file system

Alteon Configuration 51: 131.225.70.46, enabled, name www05-vhosts, weight 1, timeout 10 mins, maxcon 200000 backup none, inter 2, retry 4, restr 8 remote disabled, proxy enabled, submac disabled handle URL cookie: disabled exclusionary string matching: disabled 1: any 2: /cgi-bin 7: .php real ports: http: vport http, group 11 HTTP Application: urlslb virtual server: 4, 131.225.70.20, enabled

http: vport 8875, group 11 virtual server: 4, 131.225.70.20, enabled

http: vport http, group 11 HTTP Application: urlslb virtual server: 5, 131.225.70.52, enabled

https: vport https, group 12 virtual server: 4, 131.225.70.20, enabled

4443: vport https, group 99, pbind sslid

Software Apache v1_3

Mid-range plans to upgrade to v2_0 as security requires.

Perl Python PHP Wiki support on Plone server (on separate

set of servers outside of the webfarm)

Automated Tools No way we can keep on top of everything the old

fashioned way. Tools (perl scripts mostly) written to automate

routine tasks Create vhosts Symlink check Password check File perms check File space check

Automated Tools(cont) Symlink check

Run 1x week Check for symlinks to areas that are sharing data

that should not be shared Check for symlinks pointing to non-existent data Check for circular links Sends email to web admin team

Automated Tools (cont) Permission check

1x per week Scan for vulnerable cgi scripts, weak

permissions on files and directories This can be any variation that leave security holes

or fit the profile of known exploits: wide read-access permissions to area where passwords

are stored write-access to cgi area wide read-access permissions to top level directories

Automated Tools(cont)

Passwords (1x week) Password files, although encrypted, should not

be shared algorithm is not particularly smart: looks for

variations on the word "PASSWORD" in file name and reports these if file permissions or locations are problematic

Web Server Monitoring NGOP (FNAL developed monitoring system)

remotely monitors and alerts on the following: Ping of Alteon switch Ping of each web host machine Ping of each web server IP address Fetch pages for each web server and virtual host Fetching pages for commonly used services

(telephone,stock) and checking correct results Verifying that the httpd for each server is running on at

least one webserver host

Web Server Monitoring (cont) Verifying that "fs examine" succeeds on the main

web volume for each server (/afs/fnal/files/expwww/*), and any specific other volumes associated with it

Watching each webserver's error log on each web host, and reporting important error messages (i.e. "out of memory")

Generally an error will cut a helpdesk ticket, and page the primary.

Log Processing with Urchin Urchin v5

This product provides accurate web site analytics which supplies the executives, marketers, webmasters, and the web designers at your firm with the vital up-to-date information they need to make informed business decisions. Blah blah blah.

Recently taken over by google. Mixed opinions about the product…

Urchin (cont) Urchin likes to just stop processing without

notification. Lots of pretty pictures and gizmos. Overkill for

what we need, but if you are running an e-commerce business….

We bought Urchin to remove dependence on a home grown log processing package that was very difficult to maintain. Urchin is easier to maintain when it works.

Security Requirements Run Nessus Scans 2x year.

Nessus comes with a library of known exploits scanning profiles.

Scan various newsgroups looking for announcements.

Rely on our security team to alert us. Follow defined baseline of apache

webservers developed at Fermilab.

Future Plans We are looking to upgrade to using the Cisco load

balancer switch SunFire V240 servers

Considered moving to Linux, but… Not confident of support model. Great for general purpose

farms, not as confident for server level service. Lot’s of work to convert. Software installs, possibly broken

cgi scripts etc… Apache 2.0 Conscious of user communities in content

management systems. CMS are hot items.

Thanks for your attention! That’s all One more thing before I go….

GO CARDS!!!!