gareth smith ral ppd gareth smith ral ppd hepix autumn 2003 triumf, vancouver mainly windows issues....
TRANSCRIPT
Gareth Smith
RAL PPD
Gareth Smith
RAL PPD
HEPiX Autumn 2003
Triumf, Vancouver
Mainly Windows issues.
Gareth Smith. RAL PPD.
Gareth Smith
RAL PPD
Gareth Smith
RAL PPD
Gareth Smith
RAL PPD
Gareth Smith
RAL PPD
Overview
• HEPiX/HEPNT web pages at:http://wwwhepix.web.cern.ch/wwwhepix/
Contain links to this and recent meetings.– Summary by Alan Silverman– Videos of presentations as well as slides.
• 73 attendees• Vendor talks/exhibits
(RedHat, Microsoft, Parnasus, Ibrix)
Gareth Smith
RAL PPD
Gareth Smith
RAL PPD
Timetable
• HEPiX-HEPNT first three days.– (first day largely site reports).
• ‘Large Systems SIG’ /Security Workshop Thursday/Friday.– Parallel sessions Friday morning.
Gareth Smith
RAL PPD
Gareth Smith
RAL PPD
Windows in Site reports (1)
• Oxford University– WTS (2000, 2003), Exchange (to 2003)– 200 PCs Win 2000 / XP.
• SLAC– XP migration about complete (total 1700
systems).– Exchange from 5.5 to 2003.
• TRIUMF– Use of SAMBA, WTS 2003 starting,
Docushare.
Gareth Smith
RAL PPD
Gareth Smith
RAL PPD
Windows in Site reports (2)
• LAL– IN2P3 forest across multiple sites (7 labs so far, 4 to
join).– SMS for upgrades
• CERN– New PCs with WXP (and/or LINUX)– Mail migration from Solaris servers to Exchange– Pilot WTS 2003; WebDAV– CPU cycles from Windows Screen saver for
simulation.
Gareth Smith
RAL PPD
Gareth Smith
RAL PPD
Windows in Site reports (3)
• GSI– Windows 200 AD. Testing W2003.
• DESY– Test migration to Windows XP summer 2003.– Install via RIS.
• JLAB– Windows 2000 domain upgrade done.
• NIKHEF– SUS used to update.– Install via RIS or GHOST
First Experiences using Windows First Experiences using Windows Terminal Services on Server 2003Terminal Services on Server 2003
Alberto Pace for the IS group
Terminal Service Pilot at CERNTerminal Service Pilot at CERN
Approved by CERN Management on June 2003 3 standard computers
desktop 2.4 GHz, 1 GB RAM, 40 GB mirrored disk Usual scale out architecture Built-in load balancing
Supported freeware clients Linux Redhat, Solaris being tested Mac OS X All recent Windows versions (98, Me, 2000, XP)
Thin clients simple to install & use Internet Explorer 4 is enough on Windows Simpler than the current ongoing effort on supporting
Hummingbird Exceed
Options that were droppedOptions that were dropped
Platform-independent clients HOBLink JWT Java applet, http://www.hob.de/www_us/
Not freeware, License cost prohibitive Citrix ICA (http://www.citrix.com/)
Uniquely X11 based No additional client software required on UNIX clients Performance issue Complex Licensing mode
Linux clientsLinux clients
rdesktop freeware client www.rdesktop.org
Source available Compiled on Redhat
standard IT version and Mandrake 9.0
tsclient freeware front-end for
rdesktop (XP look) www.gnomepro.com/tsclient
Discussion with user representativesDiscussion with user representatives
A large majority of delegates requested to continue and extend the service
Continue the standard service for the core applications A subset of the existing one
Envisage the possibility of having instances of TS nodes centrally maintained where a particular service provider could install his own software LHCB build service AB/CO controls applications, with managed JVM ST/MA Asset Tracking and Maintenance Management EP/SFT for several custom applications IT/PS for some engineering applications TH to read mail attachments for non-windows users
The proposed “standard Service”The proposed “standard Service”
Core set of applications for the standard service Microsoft Office XP with Frontpage Office XP Professional Multilanguage Pack (French, German, Italian) Adobe Acrobat, Distiller, PDFMaker, Adobe PostScript Printer Driver Putty 0.53b CERN Client Printing Package CERN Phonebook 2000 Zephyr Symantec Antivirus Client
To be discussed ActiveState Perl Python Visual Studio .NET OpenAfs
OpenAFS has been one of the most welcome application but it had several technical issues
Microsoft MS Project 98 / MS Project 2002
ConclusionConclusion
A step forward in Linux / Windows / Mac integration Freeware clients exists for all platforms
(except legacy Mac OS 8-9) STOP or GO decision in November, based on
manpower cost LONG TERM COMMITMENT of 0.5 – 1 FTE
Web-based file systems and WebDAV gateway services to Web-based file systems and WebDAV gateway services to CERN DFS file systemCERN DFS file system
Alexandre Lossent, Alberto Pace
The “Web” is part of the solutionThe “Web” is part of the solution
Standard extensions to the HTTP protocol allow managing files on web servers as if these would be part of the local file system
HTTP Extensions for Distributed Authoring (WebDAV IETF RFC 2518) have been widely adopted on all major OS
Several commercial and public-domain implementations exists
WebDAVWebDAV
Web Distributed Authoring and Versioning IETF RFC 2518 (February 1999)
http://ietf.org/rfc/rfc2518.txt An extension to the HTTP protocol
New verbs (PROPFIND, MKCOL, LOCK...), headers and status codes Uses XML to format information
Initially designed as a way to author web sites Redundant with FPSE in the Windows world Versioning is limited to file locking (check in/out) Can be used as a low-end network filesystem
WebDAV Home page http://webdav.org See it also for related open-source projects
WebDAVWebDAV today today
File access: Create / delete files and folders Read / write files Copy / Move / Delete / rename files and folders
Document locking prevent the overwrite problem, where two or more
collaborators write to the same resource without first merging changes
Allow implementation of offline folders Properties
XML properties provide storage for arbitrary metadata
WebDAV tomorrow ?WebDAV tomorrow ?
Access control Set / View / Modify Access Control lists using http
Versioning and Configuration Management The V in WebDAV means “Versioning” Document check-out, check-in Retrieval of the history list Offline files and folders
Other advanced features Symbolic links Ordered collections Aggregated operations
WebDAV serversWebDAV servers
Supported by all common web servers Apache module mod_dav WebDAV package in PHP PEAR
Built-in support in IIS 5 and 6 Need to activate appropriate HTTP verbs: PUT (write
setting), PROPFIND (directory browsing setting) Permissions are managed by NTFS ACLs Microsoft adds a header to the WebDAV protocol for a
HTTP GET to return a script’s output or its source (source access setting)
WebDAV serversWebDAV servers
Supported by all common web servers Apache module mod_dav WebDAV package in PHP PEAR
Built-in support in IIS 5 and 6 Need to activate appropriate HTTP verbs: PUT (write
setting), PROPFIND (directory browsing setting) Permissions are managed by NTFS ACLs Microsoft adds a header to the WebDAV protocol for a
HTTP GET to return a script’s output or its source (source access setting)
SummarySummary
Use of WebDAV as interoperable network filesystem possible today Can be applied to collaborative tools as well (Exchange)
Takes advantage of HTTP and XML ubiquity Excellent level of interoperability for file access Really reachable from any device / anywhere
Very simple to implement But...
Still few implementation glitches https support is still limited Not a high-performance file system Not a replacement for native file system (eg NTFS) Permission management still require custom implementations
CERN Print Manager Approach
• 1 central database describing all printers– Printer server (in a dedicated DNS zone)
– Driver to be used for each printer• Per OS version (currently W95, WNT, W2K)
– Printer default settings
• 1 client with 3 main components– PrntTray : Printing Control Center (main application)
– LPRServ : LPR client (ability to show LPR transactions)
– PrinterWizard : add/remove printers, change defaults
Client : PrntTray GUI
Multi-sites Configuration
• Allow to switch between different sets of parameters– Central database locations, LPR parameters, …
• No conflict between sites– Differents directories for data files– Differents registry paths
• Site definition in an INI file– Client can be distributed with several sites
preconfigured– Easy addition of a new site
More information
• http://printpackage.web.cern.ch/PrintPackage
Installation of W2K/WXP using theunattended.sourceforge.net
project
INFN - Napoli1
INFM - UDR Napoli2
HEPiX/HEPNT 2003 – Vancouver
Rosario Esposito1
Francesco Maria Taurino1,2
Gennaro Tortone1
HEPiX/HEPNT 2003 – Vancouver
Unattended installation systems [2/3]
Unattended.sourceforge.netIt’s an OpenSource project to manage unattended installations of Windows 2K/XP workstations
Advantages: No need of Windows and Active Directory at server
side Supports a large number of network adapters Customizable partition scheme No need of .msi format to deploy applications
HEPiX/HEPNT 2003 – Vancouver
Unattended installation systems [3/3]
Unattended.sourceforge.net
Disadvantages: No user-friendly interfaces Tuning of some perl scripts and batch files is
required at server side to obtain a good site dependent installation system
No support for disk imaging based installations
HEPiX/HEPNT 2003 – Vancouver
Conclusion
Unattended.sourceforge.net is a valid alternative for Remote Installation Service (~OpenRIS !), primarily in a Unix-oriented server environment
It’s completely FREE and presents all of the advantages (and flaws) of an OpenSource project
It has interesting features, like the extreme flexibility of installation scripts
It’s not the optimal choice in the case of homogeneous hardware
No support for application deployment after the installation
Windows and UNIX Windows and UNIX InteroperabilityInteroperability - tips, tricks, and secrets - tips, tricks, and secrets
Peter SkjPeter Skjøøtt Larsentt LarsenLead PMLead PMMicrosoft CorporationMicrosoft Corporation
Client Options for UNIX Client Options for UNIX codecode
A number of alternatives exist today:A number of alternatives exist today:Improved UNIX clients with better Improved UNIX clients with better applicationsapplications
Better desktops apps for Linux, etc.Better desktops apps for Linux, etc.
UNIX like environments on Win32 APIUNIX like environments on Win32 APICygwin, uwin, mksCygwin, uwin, mks
UNIX emulation on Windows KernelUNIX emulation on Windows KernelMicrosoft Services for UnixMicrosoft Services for Unix
Virtual MachinesVirtual MachinesMicrosoft Virtual ServerMicrosoft Virtual Server
Windows like environment on UNIXWindows like environment on UNIXWineWine
All the comforts of home All the comforts of home ……
Replaces Posix subsystem (in Replaces Posix subsystem (in Windows)Windows)C Shell and Korn shellC Shell and Korn shellSingle-rooted file systemSingle-rooted file systemSymbolic linksSymbolic linksWin32Win32®® programs programsTerminals and other devicesTerminals and other devicesServices and daemonsServices and daemonsMan pagesMan pagesX windowsX windows
Windows And SFUWindows And SFU
Other device driversOther device drivers CDFSCDFS FATFAT NTFSNTFS NFSNFSClient Server GatewayClient Server Gateway
SFU/Interix SFU/Interix
WindowsWindows
Color Legend
3rd Party3rd Party
Hardware Abstraction LayerHardware Abstraction Layer
Interix SubsystemInterix Subsystem
UNIX /POSIX APIsUNIX /POSIX APIsBSDBSD
SocketsSockets
UNIX, XPG,UNIX, XPG,POSIX.2POSIX.2
commands commands & utilities& utilities
UNIXUNIXshellsshells
telnetdtelnetd
Open SourceOpen Sourcetools:tools: Apache, Apache,
Tcl/Tk, bash, etc.Tcl/Tk, bash, etc. X11X11
Win32 SubsystemWin32 Subsystem
Windows APIsWindows APIs
Windows systemWindows systemadmin, commandsadmin, commands
& networking& networking
WindowsWindowsGUIGUI
winsockwinsock
WindowsWindowscommandcommand
ShellShell
X11X11R6R6
serverserver
WindowsWindowsAppli-Appli-
cationscations
WindowsWindowsAppli-Appli-
cationscations
MotifMotif
UNIXUNIXApplicationsApplications
UUNNIIXX
SSDDKK
(gcc)(gcc)
Windows KernelWindows Kernel win32k.syswin32k.sys
Managed Co-ExistenceManaged Co-Existencewith Virtual Serverwith Virtual Server
Hardware Abstraction LayerHardware Abstraction Layer
Virtual ServerVirtual Server
UNIX KernelUNIX Kernel
UNIX APIUNIX API
CmdCmd& Util& Util
X11X11
ShellShell
UNIX APPUNIX APP
NT 4.0 KernelNT 4.0 Kernel
NT 4.0 APINT 4.0 API
CmdCmd& Util& Util
GuiGui
ShellShell
NT 4.0 APPNT 4.0 APP
Windows 2003 KernelWindows 2003 Kernel
Windows 2003 APIWindows 2003 API
CmdCmd& Util& Util
GuiGui
ShellShell
Windows APPWindows APP
Vir
tual S
erv
er
Vir
tual S
erv
er
Virtualization Virtualization ResultsResults
Linux app runs in the Windows Linux app runs in the Windows environment with integrated …environment with integrated …
User file storeUser file storeSecurity contextSecurity contextCommand execution environmentCommand execution environment
Access Linux transparently from Access Linux transparently from WindowsWindowsLinux / UNIX apps run out of the boxLinux / UNIX apps run out of the boxPerformance acceptable for many Performance acceptable for many classes of appsclasses of apps
More info …More info … http://www.microsoft.com/windows2000/migrate/unix
Email …Email … [email protected] [email protected]
Gareth Smith
RAL PPD
Gareth Smith
RAL PPD
Windows Discussion (1)
• Software Update Services.– Good results reported.– Care if using more than one way to update (SUS,
SMS etc.). Varied internal mechanisms to decide if patch applied….
– Need to reboot when requiredby SUS otherwise possibility of SUS blocking and not caching more updates.
– Synchronize with Microsoft’s updates (Tuesdays).– Maybe issues of handling Windows 2000 and XP
clients at same time.
Gareth Smith
RAL PPD
Gareth Smith
RAL PPD
Windows Discussion (2)
• Suggestion of putting personal firewalls on all systems….– (Felt to be too complicated).
• SLAC have contracted Microsoft to write a dll that will synchronize passwords between Active Directory and Kerberos.
[email protected] – mailing [email protected] – to join.
Computer Security Update
Bob Cowles, [email protected]
Presented at HEPiX - TRIUMF23 Oct 2003
Work supported by U. S. Department of Energy contract DE-AC03-76SF00515
Slammer Impact
Application of Patches to Windows
0
200
400
600
800
1000
1200
1400
1600
1800
2000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31
Days Since Patch Released
Vu
lner
able
Sys
tem
s
MS03-026
MS03-039
MS03-043
Internet Avg
MSBlaster Released
MSBlaster at SLAC
Microsoft @ Stanford
• Universities tend to be a worst case• Diverse, unmanaged
– Population– Hardware– Software
• Unlikely to fit into AD model
• Stanford had 8000 machines compromised by Blaster BEFORE students returned for classes
Conclusions
[Unchanged from last year]
• Poor administration is still a major problem
• Firewalls cannot substitute for patches
• Multiple levels of virus/worm protection are necessary
• Clue is more important than open source
CERN’s Computer Security Challenge
Denise Heagerty,CERN Computer Security Officer
Incident Summary, 2001-2003
2001 2002 2003-Sep
Incident Type
59 31 26 System compromised (intruder has control) security holes in software (e.g. ssh, kernel, ICQ, IE)
42 25 27 Compromised CERN accounts sniffed or guessed passwords
11 21 305 Serious Viruses and worms Blaster/Welchia (290), Sobig (12) , Slammer(3)
13 21 119 Unauthorised use of file servers insufficient access controls, P2P file-sharing
15 16 1 Serious SPAM incidents CERN email addresses are regularly forged
11 9 6 Miscellaneous security alerts
151 123 484 Total Incidents
Site Security: actions in progress
Hardware address registration enforced for computers using DHCP (wireless, portables)
Allows the user to be informed of problems Started for some buildings, rest of site before Xmas
Off-site FTP closure Firewall block planned for 20 Jan 2004
AFS password expiry enforcement Forced annual password changes + email warnings Already enforced for Windows/Mail passwords
Network connection Rules Defines acceptable network and security practice System admins must agree before connecting systems
Worrying Trends
Break-ins are devious and difficult to detect E.g. SucKIT rootkit
Worms are spreading within seconds Welchia infected new PCs during installation sequence
Poorly secured systems are being targeted Home and privately managed computers are a huge risk
Break-ins occur before the fix is out SPAM relays used a new hole before a patch and anti-virus
available People are often the weakest link
Infected laptops are physically carried on site Users continue to download malware and open tricked
attachments Intruders and worms can do more damage
When?
What more can be done?
Restrict/eliminate direct modem access Firewall protection has proved to be necessary Modem access is provided by ISPs
Reduce the need for VPN to access CERN services Offer popular services to the general Internet: mail,
authenticated web sites, file access, … Further enhance firewall protections
database driven and based on requirements Enhance system and application security
Some patches need deadlines and forced reboots Security & anti-virus updates should not rely on home site
access Personal firewalls can reduce risk and buy time
Improve security awareness Common messages across the HEP community would help
How CERN reacted to the How CERN reacted to the Blaster and Sobig virus attackBlaster and Sobig virus attack
Christian Boissat, Alberto Pace, Andreas Wagner
CERN results and effort involvedCERN results and effort involved
Action Preventive Repair
Apply patch to 5000 machines via NICE 0.1
Security 4.0
Network group 6.0
User Support 3.5
Coordination 0.5
Local support 4.0
Total 0.1 18
NB: Does not include effort in other Divisions
The hotfix webpage was visited 12’200 times in AugustThe emergency measures page 2600 times in second half of August
Infected Systems: Blaster/Welchia (~300), Sobig (12) (At end of August in FTE weeks)(At end of August in FTE weeks)
ConclusionConclusion
Despite this “negative” presentation, all CERN Central computing services and its network continued to work without interruption
Standard users (more than 95 %) also continued to work as usual
Unmanaged computers were heavily affected Many visitor computers were not up-to-date for virus and patches Owners of unregistered computers could not be contacted and
informed This is the lesson to learn
However, this has triggered additional efforts to further improve patch distribution methods and to reduce further the deployment time
Everybody now takes security more seriously and we did not need a catastrophic disaster to achieve this
A walk through a Grid Security Incident
HEPiX
Vancouver, October 24,2004
Dane Skow, Fermilab
AFS and User Private Keys
• Many users have home areas in AFS.• Many users do not understand how AFS access
control lists work. It is easy for users to leave their private keys
world readable in AFS space.
• Should one proactively create a .globus directory in all users $HOME with the proper permissions ?
• What about SSH RSA keys, browser credential caches, PGP keys, …
The Stats
• Of 18 directories, 14 were world readable. 11 had valid certificates.
• After 40 days, 8 had still not been revoked. 3 directories were still readable. 1 new exposure had occurred.
• Distribution of sources5 DOEGrids
5 DOESciencegrids
1 Princeton self-signed
Matt CrawfordFermilab
HEPiX, October 2003
Opportunities for collective incident
response... and prevention
Receive report or detect activity.Gather additional information.
Evaluate.Take immediate steps, if indicated.
Estimate effects on/implications for other sites.Plan corrective action.
Notify (or consult) management.Notify affected and other concerned parties.
Carry out corrective plan.Assess performance and current security posture.
Collective Incident Response
The common internet threat model is trusted endpoints on an insecure network.
SSL, SSH, ipsec, and a myriad of host vulnerabilities have turned this backwards. We’ve got more communication security than host security.
... and it’s natural to believe that a message received on a secure channel can be trusted.
See also: “The Internet is Too Secure Already,” by Eric Rescorla.
A Problem Statement
That’s not so bad, in relative terms.
At the last meeting, 6x the people exposed 18x the passwords in the same time period.
The bad news: that was GGF.
Live It?cm----97 r6----4b go----ng la----28
lu----le ca----th fz----00 fr----mp
tr----u5 hy----mj ma----_8
Gareth Smith
RAL PPD
Gareth Smith
RAL PPD
Security Discussion
• Concern about GRID firewall holes.• Idea of information page(s) for visitors to a
site.• Set-up e-mail list for Security information.
– (Contact [email protected]).– Note: This is not for Security alerts.
• Need laptops updated before they leave home institute.– And ability to update them when away.
Gareth Smith
RAL PPD
Gareth Smith
RAL PPD
Lots of Other Interesting Talks
• Root Kit Protection and Detection• SPAM fighting (two talks – GSI, Triumf)• Console management on farms• ……..
Next meeting in Edinburgh.