Index

10 Gigabit Ethernet standard, 19819-inch racks, 152, 155802.3 Spanning Tree Protocol

bridge, 45

AAcceptable-use policy, 318, 320,

579Access

data centers, 134databases, 904–905monitoring, 534–535

Access control policy, 229–230Accidental file deletion, 621–623Account names, 223Accountability and shared accounts,

290, 292Accounting policy, 568–569Accounts, longevity policy, 230–231Acquisitions overview, 8–9Active Directory lookups, 720Active listening, 376

mirroring, 792–794reflection, 795–796standardizing on phrases, 793–794summary statements, 794–795

Active monitoring systems, 532–534Active Server Pages, 691ActiveDirectory, 237, 332Ad hoc solution finder, 921–922Add-ons and preloaded operating

systems, 53Administration

centralization, 507

Administrative functions, separatenetworks for, 89

Administrator access, 327Administrator account, 291AJAX, 691–692Alerting to failure, 524Alerts, 530–532

real-time monitoring, 527Algorithms and high-latency networks,

102–103Aliases, 231

email servers, 549Always make backups, 786Always-on Internet technology, 664“An Analysis of UNIX System

Configuration” (Evard), 41–42Anonymizer service, 335Anonymizing redirection service,

258ANS.1 format, 529Antispam software, 550Anti-virus software, 550AOLServer, 691Apache, 691, 720AppleTalk, 569Appliances, 84–85Application servers, upgrading, 211Applications

centralizing, 116configuring properly, 32–33critical servers lists, 34high latency, 101new configuration information and,

426–428

955

956 Index

Applications (continued )optimizing RAID usage, 611–612response-time monitoring, 537security, 709–710streamlining write path, 612updating, 54–57

Architects, 401, 736Archival backups, 624–625Archival restores, 624Archive tapes

obsolescence, 624separating from other backups, 624

Archives, 624, 627Archiving

email, 784logs, 299

Asking for help, 808–809Assessing sites overview, 7–8Asset management, 513Assumer, 379Asynchronous JavaScript, 692ATM (Asynchronous Transfer Mode),

187, 212ATS, 139–140, 177Attackers

contacts in industry, 301logs, 299mail relay hosts, 556–557mean time to, 289responding to, 303–307site used to launch, 307spoofing real-time monitoring

system, 525Audio latency, 103Audit trail, 415Auditing, 298, 318–319

security consultants, 308–309Auditor, 302AUP (acceptable-use policy), 276–277,

326–327AUSCERT (Australian Computer

Emergency Response Team), 289Authentication, 290, 318

Apache, 720biometric mechanism, 291CGI-based applications, 720–721handheld token-based system, 291

inflexibility, 292information used for, 292over phone, 292remote access service, 661shared accounts, 290–291strong system of, 291web server software, 720

Authentication and authorizationservice, 97

Authentication servicescustomer requirements, 96full redundancy, 122

Authorization, 290–293Authorization matrix, 293–295, 320AutoLoad, 47, 50Automated front ends, 428Automated installation, 43, 47–49,

53–54Automated inventory, 238Automated operating system

installation, 32–33Automated services, 737Automated update system, 57Automatic failover, 573, 577Automatic network configuration,

469Automating

backups, 639–641combining with cloning, 51completely automated, 47–49done a little at a time, 413email service, 552fixing root problem, 413fixing symptoms and alerting

SA, 412fixing symptoms without fixing root

cause, 412hidden costs, 46manual steps and, 764–765monitoring, 535operating system, 46–47testing, 764updating servers, 463verification tests, 441

AutoPatch system, 54, 56Availability monitoring, 527Awards wall, 810–811

Index 957

BBack door, 906Back-out plans, 417

backups, 443relying on, 448service conversions, 465–466testing after use, 448–449when initiated, 444when to execute, 466writing, 443

Backup and restore system, 621basics, 620–643speed of interconnections, 635

Backup media, 622Backup policies, 230Backup software

automation, 628homegrown, 641installation, 744–745scheduling algorithms, 639

Backup tapeschanging M-W-F, 786file-by-file inventory of, 642–643passing cost to customer, 625tracking reuse, 643

Backups, 583, 619always making, 786automating, 639–641back-out plan, 443bubble-up dynamic schedule, 632centralization, 641–642commands, 639consumables planning, 635–637corporate guidelines, 625–626D2D2T (disk-to-disk-to-tape), 635data storage service, 598–601data-recovery SLA and policy, 626delegating, 641disk drive as buffer, 635DLTs (digital linear tapes), 635–637email, 559–560fire drills, 643–644full backups, 620, 627–628high-availability databases, 647–648homegrown backup software, 641incremental backups, 620, 627–628,

633

Internet-based systems, 647jukeboxes, 639, 642length of cycle, 628–631locally replicated software, 683manual backups, 639, 641media, 644–647minimal tape index, 642mirrored disks, 84mirrors, 599–600mistimed, 626NAS, 600network-based backups, 641networked off-site, 646–647nine-track tape drives, 649no substitute for, 598–599off-site storage, 644–647RAID mirrors to speed, 600risks, 417SANs, 600–601scheduling, 627–633, 639SLAs, 625–626speed of, 633–634tape inventory, 639, 642–643tape usage, 628–633technology changes, 648–649thinking aspect of, 640–641time and capacity planning,

633–635true incrementals or differentials,

633Balancing work and personal life,

809–810Bandwidth

addictiveness of increases, 657hijacked, 703–704versus latency, 101–103local area networks, 524

Bell Labs, 45–46, 234, 244AutoPatch system, 56Computer Science Research

group, 65demo schedule, 419laptop subnet, 65network-split project, 461pillars versus layers approach, 461Rioting-Mob Technique, 459–460UNIX Room, 412

958 Index

BGP (Border Gateway Protocol),187

Biometric mechanism, 291Blade servers, 91–92Bleeding edge, 218bleeding edger, 932Blind guessing, 604Bonuses, 825Boot disks, mirroring, 83Boot server, 121Booting critical systems, 483Boss philosophy, 811Bot farms, 704Bounced email, 409Break, 599Brick, 205“Bring me a rock” management

technique, 843British Telecom, 465Broadcast domain, 197Browsers, 689Budget administrator, 926–927Budgets

nontechnical manager, 860–862technical managers, 834–835technical staff, 860–862

Bugtraq, 289Build licenses, administrating,

332Building

generator backups, 143rewiring, 202

Bulk-license popular packages, 331Business applications support team,

312Business desktop computers, 73Business partners relationship, 757Businesses

constraints, 476security meeting needs, 285–287security through infrastructure,

288Business-specific services, 95Buy-versus-build decision,

845–848Buzzword-compliant tools, 399Buzzwords, 376

CCable bundling, 165Cables

categories, 198color coding, 161hiding, 159labeling, 167–169, 182, 206lengths, 163managing in racks, 156–158networks, 163organizing, 157–158patch cables, 161–163prelabeled, 168raised floor, 159–160slack in, 163testing after installation, 202value of test printouts, 203

Cage nut, 153calendar command, 419calendar global alias, 98Calendar program, 33–34Calendar server, 109, 231CamelCase, 249Canned solutions, 845CAP (Columbia Appletalk Protocol)

server, 121Capacity monitoring, 527–528Capacity planner, 926Capacity planning, 524Capturing command line, 245Capturing screen shots, 244–245Career goals, 812–813Career paths, 833–834Careful planner, 925–926Carpenters, 410–412The Case of the 500-Mile Email,

402Cat-5 cable, 161Cat-6 cable, 161–162CDP (continuous data protection), 598,

614–615Cellphones, 488Center-of-the-universe host, 122Central funnel architecture,

572–573Central host, 210Central machine, 121

Index 959

Centralization, 501–502110 percent, 504access, 504administration, 507asset management, 513backups, 641–642balance, 504basics, 502–512candidates, 505–510commodity, 509–510consolidating purchasing, 513–515consolidation, 506–507cost savings, 505distributed systems, 506easier-to manage architecture, 505experience counts, 503giving up control, 505guiding principles, 502–505helpdesk, 741impediment management decisions

or politics, 505improving efficiency, 501increased purchasing power, 509introducing new economies of scale,

505involvement, 503issues similar to new service, 504left hand, right hand, 508–509motivation, 502–503outsourcing, 515–518printing, 566–568problem-solving, 502remote access service, 658single points of failure, 512specialization, 508tape changes, 641veto power, 505

Centralized file servers, 509Centralized funnel, 573Centralized group for services, 508Centralized model for customer

support, 740–741Centralized storage, 597–598Centralizing

network management, 738services, 98, 737

CERT/CC, 289

Certificates, 704–706CFO (chief financial officer), 734CGI (Common Gateway Interface), 691

programs, 701–702scripts, 691servers, 695

CGI-based applications andauthentication, 720–721

Change advisory board, 417Change completion deadlines, 488–489Change control namespace, 230Change log, 451Change management

audit trail, 415automated checks, 426–428automated front ends, 428basics, 416–428categories of systems changed, 416communication and scheduling, 416communications structure, 418–419documentation, 422, 424e-commerce companies, 415ITIL (Infrastructure Library), 417locking, 424–426managing risk, 415Nagano Olympics, 430–431planning and testing, 416process and documentation, 416processes, 422, 424reboot test, 427–428revision control and automation, 416revision history, 424–426risk management, 417–418scheduling, 419–422specific procedures for each

combination, 416streamline processing, 431–432successful large-scale events, 431technical aspects, 424–428types of changes made, 416

Change procedures, 236Change proposals, managing, 479–481Change-control forms, 422Change-freeze times, 422, 423CHANGELOG file, 453Change-management meetings,

428–431

960 Index

Change-proposal forms, 422Chaos topology, 195Checklists, 246–247, 821Christine’s dream data center, 183–184CIAC (Computer Incident Advisory

Capability), 289Cisco

NetAid, 431routers, 395

Classifier role, 368Classifying problems, 368–369Clean Desk Policy, 315Clean network architecture, 190–191Clean state, 42Clear directions, 842–843Clerks

installing software, 761–762managed by SAs versus by

customers, 761simple automation, 763–764solving performance problem,

762–763Client servers and OS configuration,

79–80Clients

email, 553moving away from resources, 64redundancy, 553–554services, 97

Clones, upgrading, 443Cloning hard disks, 50–51Closed cable management, 158Closed services, 104Closed source security-sensitive

products, 296Cluster file systems, 588Clusters and namespace databases, 232CMS (content-management system), 253Code control systems, 425Code red emergencies, 32Code yellow emergencies, 32Colocation (colo) center, 71, 743Colocation facility, 129–130Color coding

cables, 161network cables, 167–168network jacks, 200

Commands, listing last executed, 245Commercial encryption packages, 559Commercial software, 684Commodity centralization, 509–510Commodity service, 510Communicating priorities, 820–821Communication

within company, 551company culture, 419customers, 837data centers, 170email service, 557–558emergencies, 488mail user agents, 551maintenance windows, 495nontechnical managers, 857–858plan updates, 57post maintenance, 490–491radios or mobile phones, 170scheduling and, 416sensitive updates, 420–421service conversions, 461–462stalled processes, 822technical issues, 791

Communication change, 418–419Communication policy, 307Communication skills

active listening, 792–796happy SAs (system administrators),

790–796I statements, 791–792my problems, 791other people’s problems, 791our problems, 791your problems, 791

Communications closets, 300Community strings, 529CommVault, 622Companies

culture and communication, 419defending right to information, 310security, 314

Company policy, enforcing, 828–829Company-confidential information,

274Compensation (comp) time, 358Competitive advantage, 847–848

Index 961

Complete restores, 625Complexity of networks, 190Components

failure, 597hot-swap, 86–87used by other applications, 115

Compression, 189Computer closets, 35–36, 129Computer room, 129Computers

building and initializing processes, 42centralizing purchasing process,

513–514clean desktop, 785clean state, 42configured state, 42coping with big influx of, 16–17debug process, 42early delivery to customer, 515entropy, 42function-based primary name, 109life cycle, 41–44new state, 42off state, 42preloading operating system, 51–53rebuild process, 42retiring, 44reviewing software on, 437service-based aliases, 469solid infrastructure, 287–288standardizing components, 514states and transitions exist, 43support time, 730tying services to, 98unknown state, 42updating, 42usable only in configured state, 43warranties, 76

Concurrent Versions System, 425Condensing data, 525–526Configuration files

automated checks, 426–428locking, 424–426manually modified, 426–428master copies, 237separate for web site, 715tracking changes to, 453

Configuration fixes, 704Configured state, 42–43conf.v file, 425ConServer, 80Consistency policy, 233–234Console access in data centers, 171Console servers, 121, 171Console window programs, 245Consolidation

centralization, 506–507purchasing, 513–515

Constraints, 476Consultants, 743–745, 756Consumables, 621

planning and backups, 635–637Contacts and security, 316–317Containment, 63–64Content scanning, 557Contractors, 743–745Contributing software policy,

671–672Conversions, 465, 468COO (chief opperating officer), 734Cooling

air output from ceiling, 137computer closets, 35–36costs, 146data centers, 136–148humidity control, 137–138IDF closets, 201network devices, 201providing sufficient, 35–36racks, 151raised floors, 137rules, 137smaller solutions, 146spot coolers, 146UPS, 139

Coordination, ensuring, 483–488CopyExact, 411Copyright-adherence policy, 330–332Corporate culture

help desks reflecting, 346maintenance windows, 477

Corporate guidelines and backups,625–626

Corporate namespaces, 543

962 Index

Corporate network and third-partyaccess, 279

Corporationsapplication response time, 537ethical policies and controls, 323helpdesks, 368staffing helpdesks, 347

Cost/benefit analysis, 823Costs, decreasing, 21CPU

chip sets and L2 cache, 606–607monitoring and, 524–525monitoring usage, 601–602servers, 70

Craft worker, 376–377Crashes

coping with, 9monitoring system, 36

Critical DNS server, upgrading,453–454

Critical host maintenance contracts,75–76

Critical inner voice, 805–807Critical servers

dependencies, 483lists of, 34stringent change-management

processes, 424Critical services, 122Critical systems, booting, 483Criticism, 807–808Crontabs, 78Cross-functional teams, 310–313Cross-shipping, 77Cryptographic certificates, 705–706CTO (chief technical officer), 733–734CTRL-ALT-DEL, 81The Cuckoo’s Egg (Stoll), 402Customer advocate, 927Customer dependency check, 437Customer requests

basics, 364–380frequent time-consuming requests,

383greeting to, 364–367

Customer support, 735–736, 739–741,931

centralized model, 740–741decentralized model, 740dedicated personnel, 739hybrid models, 741marketing-driven, 369solutions, 847

Customers, 756aligning priorities with expectations,

758–760announcing upgrade to, 445–446attitude of SAs, 756–758becoming craft worker, 376building confidence, 22classifying problems, 368–369communicating change to, 418–419communicating conversion plan to,

461–462communicating upgrade or back-out

plan, 448–449communication, 837compelled to lie, 370consultants, 756conversion having little impact on,

458–459decentralization and, 511–512defining emergencies for, 31digging into problem before

reporting it, 392–394feature creep, 837generating most tickets, 382giving up control, 505good first impression, 752–755group statistics, 601high and small things, 758–759ignored requests, 28ignoring messages from system

administrators, 449importance of printing, 565incorrect jargon, 392increased familiarity with, 381inexperienced, 375involving in standardization process,

66keeping happy, 15listening to concerns of, 503locking out for server upgrade,

446–447

Index 963

meeting with groups, 766–767meeting with single point of contact,

866–868opportunities to interact with,

757perceptions, 751, 760physical access to data center, 135policies associated with email service,

558prioritizing solutions, 375processes to help themselves,

347–348questions in particular category,

383relationship with support team,

740–741relying on services, 438reporting same issue, 382–383requirements, 837restoring access after upgrade, 448SA email to, 770–773self-service requests, 383service requirements, 98–100service rollout, 120setting hostnames, 62–63standards, 66task-length perception, 29–30town hall meetings, 768–770training, 462usage guidelines, 326–327useful feedback, 375verifying repair of problem, 378weekly meetings with, 867

Customer/SA, 931Customization and decentralization,

511Customizing striping, 611–612Cutting edge, 218Cylinders, 584

DD2D2T (disk-to-disk-to-tape), 635DAD (disk access density), 613Daemons, 115Daily planning, 782–783Daily tasks, 785DAS (directly attached storage), 587

Databackups, 619–620block optimization, 607condensing, 525–526corruption, 267expiring, 526length of time to keep, 526protection, 614restoring, 619–620security, 271–272

Data cables, 166Data centers, 129

access, 134basics, 130–176biometric locks, 135–136booting machines, 483cleaned power, 138communication, 170communication backups, 131console access, 171cooling, 136–148costs, 129directing airflow, 137duplicating critical services across, 268dust and, 173earthquake zone, 132equipment, 130extra electrical capacity, 144–145extra space in, 179extrawide doors, 134fire suppression, 149–150flooding, 132heat sensors, 142heating, 137high security requirements, 135high-reliability, 177–178hot spots, 142humidity control, 137–138HVAC system, 142ideal, 179–185interruption of service, 473keyboards, 171keys, 135labeling, 166–169lightning protection, 132–133locating servers in, 110location, 131–132

964 Index

Data centers (continued )locking, 135maintenance window, 130MDF (main distribution frame),

204minimum aisle width, 154mobile items, 175–176monitoring temperature, 143monitors, 171moving overview, 5natural disasters, 131–132physical checks, 300planning for future, 130political boundary, 131–132power, 136–148proximity badges, 135racks, 150–159raised floor, 134redundancy, 176–177redundant locations, 133–134reliability, 110restricting access, 135security, 134–136servers, 78–79tools and supplies, 173–175visitor policy, 136wasted space, 156wiring, 159–166workbench, 172–173working in, 173

Data flow analysis and scaling,124–125

Data format, 189Data integrity, 78, 267Data pipeline optimization, 606–608Data storage, 583, 864

basics, 584–611CDP (continuous data protection),

614–615cost, 589current usage, 590DAS (directly attached storage), 587departments and groups assessment,

589evaluating new solutions, 608–609filesystems, 587inventory and spares policy, 593

key individual disk components,584–585

less-desirable hardware, 608limits, 613–614managing, 588–596mapping groups onto storage

infrastructure, 592–593NAS (network-attached storage),

587–588performance, 604–608physical infrastructure, 609–610pipeline optimization, 606–608planning for future, 593–594problems, 609–611quotas, 592–593RAID (Redundant Array of

Independent Disks), 585–587reframing as community resource,

588–589resource difficulties, 592SAN (storage area networks), 588saturation behavior, 610–611standards, 594–596storage-needs assessment, 590–591terminology, 584–588testing new system, 608timeouts, 610unexpected events, 591usage model, 608volumes, 587

Data storage service, 596backups, 598–601historical monitoring, 601monitoring, 601–603reliability, 597–598storage SLA, 596–597

Data transfer path saturation,610–611

Data writes, 607Database-driven web sites, 695–696,

716Databases

automating data access, 710high-availability and backups,

647–648preparation function, 710read-only views, 702

Index 965

read-write views, 702scaling usage, 702tuning block size, 611–612web sites, 701

Dataflow analysis example, 126Dataflow model, 124–125Data-recovery SLA and policy, 626dbadmin account, 291dbadmin group, 291Deadlines for change completion,

488–489Debug process, 42Debugging

active monitoring systems, 533basics, 391–398better tools for, 399–400email, 553end-to-end understanding of sysem,

400–402fixing cause, not symptom, 393–394follow-the-path, 395learn customer’s problem, 392–393Microsoft Windows, 396networks, 190right tools for, 395–398Sun RPC-based protocols, 397–398systematic about finding cause,

394–395TCP-based protocols, 397–398turning as, 399UNIX systems, 396

Decentralization, 501110 percent, 504access, 504balance, 504basics, 502–512candidates, 510–512customization, 511democratizing control, 510diversity in systems, 512fault tolerance, 510–511guiding principles, 502–505issues similar to building new service,

504many single points of failure, 512meeting customers’ needs, 511–512motivation, 502–503

opportunity to improve responsetimes, 510

problem-solving, 502veto power, 505

Decentralized model, 501, 740Decision point, 417–418Decisions

precompiling, 785–787technical manager, 843–848

Decreasing costs, 21Dedicated machines services,

120–122Dedicated network router, 84Deexecutioner, 379Defense in depth, 272Defining emergencies, 31Defining scope of SA team’s

responsibility policy, 31Definition of emergency policy, 821Defragmenting hard disks, 614Delegation, 831Deleting files and UNIX shells,

410–411Deletion policy, 671–672Demarcation points, 205Dependency chains, 539Depots, 672Descriptive names, 225–226Desk location and visibility, 767Desktop computers

cost in early 1990s, 90early, 130

Desktops, rolling out new softwareto, 120

Developer’s tool chain, 685Device discovery, 535Device drivers, 53Devices

labeling, 34monitoring discovery, 535naming standards, 206networks, 209–211parts not hot swappable, 88SNMP requests, 529UPS (uninterruptible power

supply), 35Devices Control Panel, 410

966 Index

DHCPautomatically generating

configuration, 59dynamic DNS servers, 61–65dynamic leases, 60–61hidden costs, 58lease times, 64–65moving clients away from

resources, 64network configuration, 58public networks, 61templates rather than per-host

configuration, 58–60DHCP: A Guide to Dynamic TCP/IP

Network Configuration(Kercheval), 65

The DHCP Handbook (Lemon andDroms), 65

DHCP servers, 58Diagnostic services and maintenance

contracts, 75Diagnostic tools, 395–398Diameter, 232diff command, 377, 440Disaster worrrier, 925Disaster-recovery plan

archives, 624basics, 261–267damage limitation, 264–265data integrity, 267lack and risk-taking, 262legal obligations, 263–264media relations, 269preparation, 265–267recreating system, 266redundant site, 268requirements for, 264risk analysis, 262–263security disasters, 268–269

Disastersbeing prepared for, 265–266damage limitation, 264–265damage prevention, 263defining, 262restoring services after, 265–266risk analysis, 262–263

Disconnection policy, 306–307

Disk failures, 602, 623Disk-cloning system, 32Disposable servers, 91Distributed network support, 738Distributed parity, 586Distributed systems and centralization,

506Distribution-server model, 668–669Diversity in systems, 512DLTs (digital linear tapes), 635–637DNS, 96–97

appliances, 84authenticating updates, 63hosts with static leases, 62MX (Mail eXchanger) records,

553no customer requirements, 98round-robin name server records,

699–700updates and TTL (time to live)

field, 467zones and subzones, 233

DNS hosting, 717DNS master, 121DNS names, 225Document repository, 247–248

dynamic, 252important documents and, 266rollout issues, 251rules or policies, 248self-management versus explicit

management, 251–252source code control, 248

Document retention policy, 560Document root, 695Document storage area, 247–248Documentation, 241, 253

accounts requiring specialhandling, 763

basics, 242–252capturing command line, 245capturing screen shots, 244–245change management, 422, 424change procedures, 236checklists, 34, 246–247creation as you work, 34culture of respect, 253–254

Index 967

device names, 206document repository, 247–248dynamic repository, 252email, 245–246email service, 557–558enabling comments, 254feedback, 243–244labeling, 206LAN connections, 207making work easier, 241maps of physical and logical

networks, 205–206metadata, 243monitoring, 534–535networks, 205–207online, 206partially automated installation, 49print service, 573–574QA (quality assurance), 243quick guide, 244redundancy, 241request-tracking system, 246restores, 638revision control, 254rollout issues, 251routers, 207search facility, 250–251shared directory, 248software depots, 672–673sources for, 244–246storage, 247–248template, 243–244title, 243trouble-ticket system, 246WAN connections, 207what to document, 242–243wikis, 249–250

Documentation repository, web-based,249–250

Documentingdisliked processes, 242–243job description, 243security policies, 276–283

Doers of repetitive tasks, 936DokuWiki, 253Domain registration, 717DOS, 587

DoS (denial-of-service) attack, 273,309, 320

Double component failure, 87Draft server, 717Dress rehearsal, 451–452Drive controller, 585Drive protocol, 585Drivers and preloaded operating

systems, 53Drupal, 253Dual-boot updates, 56Due-diligence assessments, 7–8Dumb pipelining algorithm, 607Dumpster diving, 229, 334Duplex printing, 576Duplexing units, 569Dynamic DNS servers and DHCP,

61–65Dynamic leases, 60–62Dynamic routing, 208Dynamic to-do lists, 779Dynamically generated web pages,

691

EEAP (employee assistance program),

807echo command, 410–411ECMAScript, 691E-commerce sites

application response time, 537authorization matrix, 320backups, 625change management, 415end-to-end testing, 537helpdesks, 347, 368IT and system administration, 742layers and pillars conversions, 461maintenance windows, 475namespaces, 233pervasive monitoring, 535privacy laws, 337SA function of maintaining site, 742SA (system administrators) team,

746–747security programs, 319–320verifying problems, 373

968 Index

EDA, 311Educating customers, 384Educator, 923EIGRP (Enhanced Interior Gateway

Routing Protocol), 187Eircom, 169Electronic accomplishment wall, 811Email, 543

as alerting mechanism, 530all customers, 770–773archiving, 784arriving in unexpected place,

548backups, 559–560bounced, 409company-confidential information,

544consistent look and feel, 739content scanning, 557debugging, 553documentation, 245–246filtering, 284forwarding policy, 338, 552handling only once, 784internal and external email

addresses, 545message sizes, 555message storage, 543monitoring, 337namespace, 544privacy policy, 544reading someone else’s, 339–340reliability, 543remote access service, 654retention policy, 559–560risks associated with, 558saving copy, 245scalability, 543SEC violations, 337traffic levels, 554working well, 33–34

Email access servers, 547Email accounts, 552Email addresses, 545

name conflicts, 226–227reuse policy, 235

Email appliances, 84

Email clientschecking for email at predefined

interval, 555encryption, 559protocols, 551

Email machines and hot spares, 547Email servers, 121, 503, 547

aliases, 549monitoring, 552–553

Email serviceadvanced monitoring, 560–561automation, 552bad mail-delivery scheme, 548–549basic monitoring, 552–553basics, 543–558beta test, 546communication, 557–558documentation, 557–558encryption, 559gateways and email translation

devices, 549generality, 550–551high-volume list processing,

561–562lack of standardization, 549large bursts of traffic, 554machines involved in, 547–548message size limits, 556namespaces, 544–546policies, 558redundancy, 553–554reliability, 123, 546–547scaling, 554–556security, 544, 556–557simplicity, 547–549spam, 549–550spare spool space, 556virus blocking, 549–550

Email software, 106Email system

architecture, 543costs of malfunctioning, 546failure, 546namespace management system, 543open protocols, 543, 550–551proprietary, 107viruses, 557

Index 969

Emergencies, 29, 31–32communication during, 488defining in writing, 353–354planning and, 354

Emergency facility, 266–267Emergency lighting, 143Employees

explaining failure to, 839feedback, 839in-person orientation, 755–756listening to, 840–841publicly acknowledging, 838recognition, 838–839reprimands, 839–840respecting, 838–841retention, 401, 893–894

Encrypted tunnels, 212Encryption, 189, 559, 656Encryption system, 559End-to-end expert, 937End-to-end monitoring, 561End-to-end testing, 536–537End-to-end understanding of system,

400–402Enjoying what you do, 804Entropy, 42Environment

identifying fundamental problems in,13

services, 110–111Environment variables, 406Environmental issues and printers,

575–576EPO (emergency power off) procedure,

485Equipment

height in rack units (U), 152labeling, 166reusing internally, 596

Error messages, real-time monitoring,531

Escalationestablishing process, 352–353monitoring rate of, 356

Escalation policy, 353, 531–532Escalation procedure, 532ESMTP (extended SMTP), 550

Ethereal, 395Ethernet, 101, 187, 198Ethics, 323

basics, 323–336copyright adherence, 330–332customer usage guidelines, 326–327hiding evidence, 336informed consent, 324issues, 23law enforcement and, 332–335people harming your company, 335privacy and monitoring policy,

336–337privileged-access code of conduct,

327–330professional code of conduct,

324–326something illegal/unethical,

338–340ETR (estimated time to repair), 656ETSI (European Telecommunication

Standards Institute) standard,177–178

Exchange mail server, 107Executing solutions, 375–376Exit process, 287Experience counts, 503Expertise, 508Expiring data, 526Extensibility and servers, 70External audits, 308–309, 317External sites and security, 717

FFacilitator, 930–931Failed disk, mirroring, 83Failover, 86Failures

alerting to, 524corruption of arrays or scrambled

data, 609hot-swap components, 87reporting, 530single points of, 510, 512

Family Educational Rights and PrivacyAct, 323

Family time, 810

970 Index

FAQ (Frequently Asked Questions),256

Fast (100MB) Ethernet, 188, 198FAT, 587FAT32, 587Fault tolerance and decentralization,

510–511FC (fibre channel), 606FCC (Federal Communications

Commission), 330FDDI (Fiber-Distributed Data

Interface), 188Feature creep, 837Features, adding, 21The Feeling Good Handbook (Burns),

806Fiber termination, 202Field offices security team, 312–313File formats, 104File Motel, 622File servers, 121

appliances, 84centralized, 509

File systems fragmentation, 614Filer line of file appliance, 622Files

accidentally deleting, 410–411,621–623

automated checks, 426–428capturing session to, 245listing to be deleted, 410mystery-deletes, 401–402rebuilding, 413

Filesystemsjournaling, 587snapshots of, 622

Filtering email servers, 547FIN packet, 700Fire drills and backups, 643–644Fire suppression in data centers,

149–150Fire-prevention systems, 265Firewalls, 271, 284, 289, 702

email protection, 557general-purpose machines as, 211inbound rules, 123OS-based, 210–211

permitting only outbound email(SMTP) traffic, 123

remote access service, 655–656Firing SAs (system administrators)

access databases, 904–905corporate policies for, 900physical access, 901remote access, 901–902service access, 901–904single authentication database, 905system file changes, 906termination checklist, 900–901

First offer, 802–803First tier of support, 352–353First-class citizens, 45First.last-style email addresses, 545Five-year vision, 864–866Fixing

biggest time-drain, 34–35problems, 373–376real problem, 413same small things time after time,

408things once, 405–412

Flash-cuts, 463–465Flat namespaces, 223Flat network topology, 197Flat physical topology, 212Flexibility, improving, 501Flight director, 478

change completion deadlines,488–489

developing master plan, 481–482mentoring new, 492–493performance level of SA team, 489technique, 473–474

Floor puller game, 183Follow-the-path debugging, 395Follow-through, 28–29, 778–780Formal documents and legal issues, 560Formal training on tools, 400Form-field corruption, 708Formulaic names, 225Four-post racks, 153–154Fragmentation and multiuser systems,

614Frame Relay, 212

Index 971

Free software licenses and copying, 331FreeBSD system, 211Fresh installs, 450–451Front-line support group, 119Front-mountable servers, 153FTP (File Transfer Protocol), 189, 296,

398Full backups, 620, 624, 627–628Full mesh, 212Full redundancy, 86–87, 122Full-disclosure mailing lists, 289Functional group-based topology, 197Functional names, 225–227Functionality and security-sensitive

products, 297Fundamental services, 95, 111Fuzzy match algorithm, 440

GGateways, 106–107, 549General printer architecture policy, 568General-purpose machines, 234Generators, 139–140, 265

backup building circuits, 143distributing, 177failure, 177maintenance, 141

Generic services, 95GET request, 528–529, 691Getting Things Done, 815Gigabit Ethernet, 198Globalization overview, 4Globally flat namespaces, 233GNAC, Inc., 148, 157GNU Stow, 672, 675–677GNU/Cfengine, 237Goal setting, 781–782Goals, 830

nontechnical managers, 836structure to achieve, 821

Golden host, 50Golden master server, 718Good first impressions, 752–755Google, 90

definition of emergencies, 32gmail service, 784IT teams, 747

mass email, 772printer maps, 574updating servers, 463

Google Maps, 721Go-to person, 916–917Graphical programs, 441Graphs and historical monitoring, 527Grouped power cords, 114Groups

mapping onto storage structure,592–593

new manager, 19new members, 18–19

HThe Haggler’s Handbook (Koren and

Goodman), 803Halt key sequence, 121Halt message, 121Handheld token-based system, 291Handles, 232–233Handling paper once, 783–784Happiness, 806–807Happy SAs (system administrators),

777awards wall, 810–811basics, 778–797communication skills, 790–796follow-through, 778–780loving your job, 804–811managing your manager, 811–814negotiation, 798–803organizing, 778professional development, 796–797staying technical, 797time management, 780–790to-do lists and appointment

calendars, 778–780Hard disk controllers, 83Hard disks

blocks, 584cloning, 50–51, 443cyclinders, 584DAD (disk access density), 613defragmenting, 614density, 613discarding, 595

972 Index

Hard disks (continued )drive controller, 585drive protocol, 585fragmentation, 613–614HBA (host bus adapter), 585heads, 584–585increasing size, 613key individual components, 584–585performance, 613platters, 584–585price per gigabyte, 583price per megabyte, 583sectors, 584spindle, 584–585tracks, 584

Hard emotions, 791–792Hard outages, 114Hardware, 81

buying for servers, 69–71cost of, 72–74failure, 597grouped power cords, 114servers, 69

Hardware cards and remote consoleaccess, 81

HavenCo, 133HBA (host bus adapter), 585Head hunters, 875Heating and data centers, 137Hello. World program, 440–442Help, specifying how to get, 351–352Helpdesk, 343

basics, 343–356better advertising for, 358–359call hand-off procedures, 741call-volume ratios, 347centralization, 741classifier role, 368communicating procedures, 344–345corporate culture, 346corporations, 368critically examining metrics, 517customer-to-attendant ratios, 347defining emergency in writing,

353–354defining processes for staff, 352defining scope of support, 348–351

division of labor, 360e-commerce sites, 368emailing new policies, 359escalation procedures, 352–353,

741formal and informal, 344–345friendly face, 346greeters, 367having enough staff, 347home phone number of supervisor,

358identifying top 10 requesters, 357installing new service, 359–360metrics, 347multiple, 741multiyear trends, 356out-of-hours and 24/7 coverage,

357–358out-of-scope technologies, 350–351permitting tickets creation by email,

408portal Web site gateway, 359problems with service, 119recorder, 369–372as referral service, 350reporting problems, 359–360requesting new services, 359–360request-tracking software, 354–356SA (system administrators) teams,

741SAs (system adminstrators),

736–737scripts, 352SLAs (service-level agreements), 32specifying how to get help, 351–352statistics, 354–357time management, 351–352time-to-call completion, 347virtual, 345web site for documentation and

FAQs, 348Helping someone, 804–805HHA (handheld authenticators), 278,

905Hidden infrastructure, 491High consistency, 233–234High-availability data service, 598

Index 973

High-availability databases backups,647–648

High-availability sites, 495–497availability, 497

High-latency links, 101High-latency networks, 102–103High-level management support for

network policies, 280–282Highly critical host maintenance

contracts, 75High-performing salespeople, 363High-port-density network equipment,

168High-reliability data centers, 177–178High-volume list processing, 561–562High-volume list services, 562Hijacked web sites, 703–704HIPAA (Health Insurance Protability

and Accountability Act), 323Hiring SAs (system administrators)

basics, 871–894diversity, 880–881employee retention, 893–894getting company noticed, 894–895identifying people to hire, 871–872interview process, 884–886interview team, 882–883job description, 872–874knowing what you are looking for,

879–880nontechnical interviewing, 891–892persuading them to work for you,

871–872recruiting, 875–877rushing hiring decision, 878selling position, 892–893skill level, 874–875team considerations, 878–882technical interviewing, 886–890timing, 877–878

Hiring System Administrators (Phillipsand LeFebvre), 879

Hiring the person, 873, 876Hiring the skill, 873, 876Historical data

collection, 215–216, 523trending, 493

Historical metamonitoring, 540Historical monitoring, 523–527

data storage service, 601scaling problems, 538

history command, 245Hit-and-run sysadmin, 379Home network routers, 211Home office, 662–663/home/adm/docs directory, 248Homegrown off-site backup storage,

646/home/src directory, 673Horizontal cable management, 158Horizontal scaling, 699–700Hostnames, 62–63, 223Hosts

broadcasting incorrect routinginformation, 208

center-of-the-universe, 122complex routing problems, 209consolidating services onto fewer,

506determining hostname, 62dynamic leases, 62intruders breaking into, 703–704IP addresses, 60–61labeling, 182MAC (media access control) address,

48multihomed, 208multiple servers on one, 697–698names, 228requiring to perform routing, 209securing before going live, 290simple routing, 207–209single-homed, 208starting in known state, 32–33static leases, 62

Hot spares, 547, 587Hot spots, 142Hot-plug components versus hot-swap

components, 88–89Hot-swap components, 87–89HousingMaps, 721How to get help policy, 31, 820How to print document, 573–574How-to docs, 255–256

974 Index

HP OpenView, 367HP-UX, 46, 54HTML (Hypertext Markup Language)

and wikis, 249HTTP (HyperText Transfer Protocol),

189error and status codes, 692–693web based products, 297

HTTP over SSL (Secure Sockets Layer),704–705

HVAC systems, 141–142, 176–177

II statements, 791–792IBM

Clean Desk Policy, 315FDA division, 311Nagano Olympics, 430–431

ICMP (Internet control messageprotocol), 526–527

Ideal data centers, 179–185IDF (intermediate distribution frame),

212–213aligning vertically in building, 199allocating space for, 198–199arranging, 205closet numbers, 200connecting, 203–205connecting cable, 198connecting to another IDF, 198connections with MDF, 199installing jacks, 201–202laying out, 198–199locking, 200numbering, 200punch block, 198remote console access, 200restricted access, 200RJ-45 connectors, 198running fiber, 202security, 200wiring, 198

IDF closets, 201IDS (intrusion detection systems), 299IEEE (Institute of Electrical and

Electronic Engineers), 107IEEE 802.1q VLAN protocols, 212

IEEE 802.1x, 61IETF (Internet Engineering Task Force),

107, 562, 689IETF standards, 214Ignite-UX, 46Illegal or unethical actions, 338–340IMAP (Internet Message Access

Protocol) server, 109IMAP4, 189, 556Implementers, 302, 737Improving system administration

biggest time-drain, 34–35calendaring, 33–34documenting as you go, 34email, 33–34host starting in known state, 32–33power and cooling, 35–36quick fixes, 35quick requests, 29–30simple monitoring, 36time-saving policies, 30–32trouble-ticket system, 28–29

Incident response, 303–307, 319Incident-reporting mechanism, 305Incident-response team, 303–304Incremental backups, 620, 622,

627–628, 633Independent services, 115In-depth attacks, 308Individual file restores, 624Industrial espionage, 267Informal documents and legal issues,

560Informal off-site backup storage, 645Information

malicious alteration, 274protection, 271security, 313–314

Information-protection group, 318Information-protection program, 315Informed consent, 324Infrastructure

maintaining services, 730services, 97standards, 508–509

Infrastructure builder, 917–918Infrastructure teams, 737–739

Index 975

Input, validating, 709Insecurity, 806Insider trading, 337Install room, 55Installation, 43

partially automated, 49–50pervasive monitoring, 535UNIX software, 668well-documented process, 49

Installer, 914Installing new service, 359–360Instant rollback of service conversion,

467–468Integration and security-sensitive

products, 297Integrators, 736Intel, 411Intellectual property, protecting, 310Intelligent queuing mechanisms, 118Interactive web pages, 691–692Intercompany security focus groups,

301Interfaces, labeling, 167Internal auditing, 298–300Internal auditing team, 308Internal mail servers, 123Internal sites publishing model, 716Internal verification, 299Internal web services and security, 704International business sites privacy

laws, 337Internet, 195

gateway and law enforcement, 335mobile phone access, 692security, 271SMTP-based protocol, 550–551transmission of unencrypted

information, 656Internet-based backup systems, 647Interpersonal communication, 376Interpersonal effectiveness, 376Interruption of service, 473Interruptions, handling, 29–30Interview process, 884–886Interview team, 882–883Intranets and privileged information,

704

Intrusion incident-response team, 303Inventory, automated, 238Inventory and spares policy, 593Involvement, 503I/O servers, 70IP addresses, 60–61

dependencies, 121longevity policy, 230–231

IP (intellectual property) manager, 310IP-KVMs, 80–81IRIX RoboInst, 54Irrevocable key, 136iSCSI, 606ISDN (Integrated Services Digital

Network), 196ISO (International Organization for

Standardization) standards, 257ISPs maintenance windows, 475ITIL (Infrastructure Library), 417

JJavaScript, 691–692, 692Job Descriptions for System

Administrators (Darmohray), 874Jobs

advertisement, 872description, 243, 872–874looking for, 19–20protecting, 23–24

Journaling, 587Jukeboxes, 639, 642JumpStart, 46, 48–49, 51, 65, 406

KKerberos authentication system, 105Kernel and packet routing, 210Key escrow, 705Keyboards in data centers, 171Kick-off meetings, 100KickStart, 46Known state, 52, 55KVM switches, 80–81, 486

LL1-A, 81L2 cache, 606–607Lab technician, 919–920

976 Index

Labelingcables, 167–169, 182, 206data centers, 166–169equipment, 166high-port-density network

equipment, 168hosts, 182interfaces, 167keeping up to date, 168network equipment connecting to

WANs, 168network jacks, 200networks, 205–206policy for enforcing standards, 169ports in software, 168printers, 574racks, 160

Labeling devices, 34LAMP (Linux, Apache, MySQL, and

Perl), 697LAMP (Linux, Apache, MySQL, and

PHP), 697LAMP (Linux, Apache, MySQL, and

Python), 697LANs, 188

connections documentation, 207dynamically assigned leases, 60large using VLANs, 212–213network bandwidth, 524not sent routing protocols on, 208star topology, 191–192

Laptops and critical device drivers, 53Large companies

SA (system administrators) team,746

security program, 319Latency

versus bandwidth, 101–103finding problem, 398recording information, 526storage SLA, 596

Law enforcement, working with,332–335

Layers approach, 460–461Layers versus pillars, 460–461LDAP (Lightweight Directory Access

Protocol), 115, 239, 720

LDP (Line Printer Daemon) Protocolover TCP/IP, 569

Leading edge versus reliability,217–218

Leaf node, 193Learning

from carpenters, 410–412from mistakes, 832new skills, 796

Lease times and DHCP, 64–65Legal department, 310–311, 313Legal issues, 560Level 0 backup, 620Level 1 backup, 620Level-focused person, 935Levels, 585Leveraging namespaces, 239License servers, 761Lights-out operation, 147Line-of-sight radio communications,

487Linux Documentation Project, 258Linux system, 211Linux tools, 667LISA (Large Installation System

Administration) conference, 797,848

List of printers, 574List processing, 547

high-volume, 561–562redundancy, 553scaling, 554–555

List servers, 562Live audio and video, streaming, 692Live equipment, 150Load balancers, 89, 554, 700, 702Load balancing print service, 577Load sharing, 87Load testing, 117Loading operating system, 46–54Locally replicated software backups,

683Location numbers, 200Location-based topology, 197Locking, 424–426Log files, rotating, 533Logging, 451, 710

Index 977

Logic bomb, 906Logical networks

maps of, 205–206topology, 195–197

Logical-network topology, 205Logins and name conflicts, 226Log-retention policy, 277Logs, 299

detailed and timestamped, 306storing in nonstandard space, 710

Longevity policy, 230–231Long-term motivators, 804–806Long-term solution, 822–823LOPSA (League of Professional System

Administrators), 72, 324, 796Lose-lose situation, 798Lose-win situation, 798Loving your job

accepting criticism, 807–808asking for help, 808–809bad boss, 807balancing work and personal life,

809–810being motivated, 804–806enjoying what you do, 804great boss, 807happiness, 806–807support structure, 808

Low-latency environment, 102Loyalty, 838Lucent Technologies, 232–233, 457LUDE, 672Lumeta, 151, 477

MMAC (media access control)

address, 48Mac OS X, 237Mac OS X server, 211Machine independence services, 109Machine room, 82, 129Mail delivery systems, 554–555mail global alias, 98Mail relay hosts, 553, 556–557Mail transport systems, 554–555Mail user agents communications, 551Mail-filtering software, 788

Mailing lists, 399, 409, 552, 561–562,788

Mailping, 536Mainframes, 130Maintainer, 915Maintenance, 735–736

generators, 141selecting window for, 443–445UPS, 140–141

Maintenance contracts, 74–78, 731Maintenance patches, 297Maintenance windows, 130

basics, 475–492benefiting company, 474–475communications, 495comprehensive system testing,

489–490corporate culture, 477deadlines for change completion,

488–489developing master plan, 481–482direct console access, 486directing, 478disabling access, 482–483e-commerce sites, 475ensuring mechanics and

coordination, 483–488flight director, 473–474, 478,

492–493handheld radios, 486–488hidden infrastructure, 491high availability for systems, 475high-availabiilty sites, 495–497interruption of service, 473ISPs, 475KVM switches, 486limited service ability, 493–494managing change proposals,

479–481planning, 477postmaintenance communication,

490–491postmortem, 492reducing complexity and making

testing easier, 474redundancy, 496reenabling remote access, 491

978 Index

Maintenance windows (continued )SA group visibility after, 491–492scheduling, 474–476, 495serial console servers, 486shutdown/boot sequence, 483–485testing console servers and tools,

482–483trending historical data, 493undetected problems, 492weekly reminders, 476

Major outage, surviving overview,10–11

Major updates, 420, 422Majordomo mailing lists, 409make account command, 237make command, 236make newuser command, 237Makefiles, 237, 413

automating tasks, 677VPATH facility, 673

Malicious alteration, 274Malware

blocking, 550protection, 284

Managed hosting, 718Management

keeping happy overview, 15security officer, 281security policy issues, 300–314tasks, 797telling you to break the law, 331time-saving policies, 31

Management chain, 733–734Managers

career goals and, 812–813grooming SAs for positions, 813information provided for boss of, 26making success of, 811–812making your needs known to, 812managing, 811–814non-work-related requests, 814raises and, 811time management, 813understanding security job, 282upward delegation, 813–814what system administrators expect

from, 26

Managing quick requests correctly,29–30

Managing risk, 415Managing your manager, 811–814Manual backups, 639, 641Manual installation, 43Manual processes, 46Manual steps and automation,

764–765Mashup applications, 721–722Mass email, 770–773Master images, 50Master plan, 481–482Master station, 538MDA (mail delivery agents), 547MDF (main distribution frame),

198–199, 203–205, 212–213Mean time to attack, 289Measuring, 604Measuring twice, 410–411Mechanics, ensuring, 483–488Media

disasters and, 269off-site backup storage, 644–647

Media servers, 696–697MediaWiki, 253Medium-sized company

SA (system administrators) team,745–746

security program, 318–319Memory and monitoring, 524–525Mentor Graphics, 248, 445Mentoring new flight director,

492–493Mentors, 881–882Mergers overview, 8–9Merging existing namespaces,

226–227Metamonitoring, 539–540Metrics

helpdesks, 347SAs (system administrators), 384security, 317

MIBs, 528Microformats, 692Micromanagement, 855–856Micromanaging, 841

Index 979

MicrosoftActiveDirectory, 64, 237, 332DHCP servers, 60Exchange mail server, 107Kerberos authentication system, 105preventing interoperating with

non-Microsoft Kerberos systems,105

Microsoft Exchange, 551Microsoft IIS, 691Microsoft OSs, 438Microsoft Windows, 410

automating software updates, 54debugging, 396Remote Installation Service, 46

Microsoft Windows NT 4.0, 50MIL-SPEC requirements, 72Minicomputers, 130Mirrored disks

backups, 84break, 599reattached, 599

Mirroring, 83, 585–586, 587,599–600, 792–794

MIS, 312Misdelegator, 379Mobile phones, 170–171, 692Model-based training, 380, 381Modem pools, 664Modems and backward compatibility,

664Modules, 672MONET (multiwavelength optical

network), 188monitor, 930Monitoring, 523

accessibility, 534–535active systems for, 532–534alerting, 215application response time, 537automation, 535availability, 527basics, 523–534capacity, 527–528clogging network links, 538CPU and memory, 524–525CPU usage, 601–602

crashes, 36data storage service, 601–603dependency chains, 539device discovery, 535disk failures, 602documentation, 534–535duplicating, 540email, 337email servers, 552–553email service, 552–553end-to-end testing, 536–537, 561file service operations, 603granular priority system, 538high-volume list services, 562historical, 215–216, 523–524,

525–527individual resource usage, 603I/O local usage, 602lack of usage, 603master station, 538metamonitoring, 539–540multiple variables in SNMP, 528network bandwidth, 524network local interface, 602networking bandwidth usage,

602–603network-interface state transitions,

215networks, 214–215nonredundant network component,

539notification scripts, 602outages, 602performance problems, 524pervasive, 535postmaster address, 553print service, 574–575problems failed to catch, 536RAID for disk failures, 597rate of change, 602real-time, 215, 523–524, 527–534remote probes, 538routing problems, 215scaling problems, 537–539security, 525services, 119setting expectations, 336–337

980 Index

Monitoring (continued )space used/space free, 602spikes or troughs, 601spoolers, 574–575status of printers, 575storage volume utilization, 601storage-access traffic, 601storage-to-server networks, 603tasks, 524web server errors, 698web services, 698–699

Monitoring and privacy policy, 277,318, 321

Monitors in data centers, 171Morale, 838, 855–857Motivation, 502–503Motivators, 804–805Motorola, 316mountd, 397Moving data center overview, 5MPLS (Mail Protocol Label Switching),

187MRTGs (multirouter traffic graphics),

255, 538MS-SMS (Microsoft’s System

Management Service), 668MTA (mail transport agent), 547MTTR (mean time to repair), 73MUA (mail user agent), 547Multicast, 187Multihomed hosts, 208, 210Multimedia files, 692Multimedia servers, 696–697Multiple administrative domains,

219Multiple inexpensive servers,

89–92Multiple servers on one host,

697–698Multiple-star topology, 192, 196Multiply-redundant spoolers, 573Multiuser systems and fragmentation,

614Multiyear maintenance contracts,

800–801My SQL, 238Mystery file deletes, 401–402

NN + 1 redundancy, 85–87Name conflicts, 226–227Name services, 96, 122Name tokens, 545–550Names

aliases, 231corporate culture, 227–228descriptive, 225–226difficult-to-type, 228formulaic, 225, 227functional, 225–227hosts, 228longevity, 231no method for choosing, 225obscuring, 231security implications, 228sequential, 227thematic, 225, 227

Namespace databases, 232Namespace management system, 543Namespaces

abstract or concrete thing, 223access control policy, 229–230adding software packages into, 244attributes, 223backup policies, 230basics, 224–237centralizing into SQL database,

238centralizing management,

236–237change control, 230change procedures, 236changes, 230cleanup, 236–237conflicts, 226consistency policy, 233–234corporate, 543customer-based updating, 239diameter, 232email service, 544–546flat, 223functional aliases, 227further automation, 238globally flat, 233inventory of all systems, 238

Index 981

leveraging, 239longevity policy, 230–231managed formally, 223–224master copies, 237merging existing, 226–227name tokens, 545–550naming policy, 224–228policies, 224–236protecting from modification, 230reuse policy, 235scope policy, 231–233single, global, 232–233thickness, 232unique corporation-wide, 545wide and thick e-commerce, 233

Naming conflicts, 715Naming conventions, 207Naming policy, 224–228Naming standards, 234NAS (network-attached storage),

587–588backups, 600configuration changes of underlying

networks, 610file-sharing services, 605performance, 605

NAS servers, 598, 600NAT (network address translation)

gateways, 702Natural disasters, 131–132, 645NEBS (Network Equipment Building

System) standard, 155,177–178

Negative behavior, 824Negotiations

after making request or offer, 802always refusing first offer, 802–803asking for what you honestly want,

801–802being careful what you say, 803developing positive relationship, 800doing your homework, 800format of meeting, 801information not leaked, 798knowing vendor’s competition, 799multiyear maintenance contracts,

800–801

nebulous requests, 799not revealing strategy to opponent,

802planning, 799power dynamic, 799recognizing negotiating situation,

798–799rehearsing situation, 800silence as negotiating tool, 803variety of techniques, 801working toward win-win situation,

798NetAid, 431NetApp, 121, 622NetApp Filers, 85Network access control, 61Network addressing architectures, 187Network Administrator, 291Network Appliance’s file server, 586Network cables, 167–168Network components outage and

monitoring, 539Network configuration, 57–61, 610Network connectivity policy, 277Network devices, 209–211

automating weekly audit, 529cooling, 201firewalls, 210–211hardware or MAC address, 188hot-swappable interface cards, 88IP (or AppleTalk or DECnet), 188moving packets quickly, 209–210path data travels, 188routers, 209–210software upgrades and configuration

changes, 211switches, 209transport information, 188UPS (uninterruptible power supply),

35Network disk, 668Network equipment

connecting to WANs, 168protected power, 201

Network Information Service, 232Network jacks, 200Network Notes, 690

982 Index

Network policiescentralizing authority, 282–283high-level management support,

280–282Network racks, 204Network router, 84Network row, 204Network services

design of, 196modern computing infrastructures,

739scripted tests, 441

Network vendors, 213–214Network-based backups, 641Network-based software push system,

668Networked off-site backups, 646–647Networking

constants, 219–220TCP/IP, 188–189

Networking devices, 81Networking printers, 568Networks, 187

administrative functions, 89assigned based on physical location,

197bandwidth and local area network,

524basics, 188–217cables, 163centralizing management, 738changes in design, 220clean architecture, 190–191complexity, 190connection to world-wide

governments, 279–280debugging, 190demarcation points, 205direct cabling, 606documentation, 205–207IDF (intermediate distribution

frame), 197–203inconsistent architecture, 196installing jacks, 201–203labeling, 205–206lack of single administrative group,

216–217

leading edge versus reliability,217–218

lunch-related traffic, 215massive, disruptive cleaning, 473MDF (main distribution frame),

203–205modern computing infrastructures,

739monitoring, 214–215multiple administrative domains,

219naming conventions, 207network administrators support,

190network devices, 209–211OSI (Open Systems Interconnection)

model, 188–189overlay networks, 212–213parameter updates, 57–61real-time monitoring, 215running fiber, 202security measures, 272simple host routing, 207–209single administrative domain,

216–217single set of policies and practices,

216solid infrastructure, 287–288standards-based protocols, 214topologies, 191–197tracking software licences, 332unsecured, 289vendor support, 190wiring, 198

Newsletters, 770NFS, 397

badcall, 603caches, 683dependencies outside data centers,

110–111mounting problems tools, 397

NFS server, 112Nine-track tape drives, 649NIS (Network Information Service)

master, 121NNTP, 398Nonconstructive criticism, 808

Index 983

Non-critical server, 74Nonprofit organizations and SA

(system administrators) team, 747Nonstandard protocols, 551Nontechnical interviewing, 891–892Nontechnical manager

analogies for, 835basics, 853–863budgets, 860–862communication, 837, 857–858customer requirements, 836deadlines, 836five-year vision, 864–866goals, 836morale, 855–857one-year plans, 860overriding technical decision, 856priorities, 854–855professional development, 862–863rehearsing executive visits, 858–859requirements tasks, 836–837resources, 854–855single point of contact meetings,

866–868staff meetings, 858–859supporting team, 857technical managers and, 835–837understanding technical staff’s work,

868–869Nonuniform operating system, 46–47Nonverifier, 379Non-work-related requests, 814NTFS, 587Nuclear power plants, 411

OOff state, 42Office location and visibility, 767Office moves, 6–7Off-shoring, 518Off-site backup storage, 644–647Off-site links, 258Off-site records-storage service,

645–646On-call expert, 923One, some, many technique, 56–57,

120

The One Minute Manager (Blanchard),815

The One Minute Sales Person(Johnson), 815

One spooler per building, 573One-day workshops and training

programs, 796, 862One-year plans, 860Online documentation, 206Open architecture services, 104–107Open architectures, 96Open cable management, 158Open file formats, 104, 106Open protocols, 96, 104–106Open source software

licenses and copying, 331security-sensitive products, 296

Open standards, 690Open systems and gateways, 106OpenDirectory, 237OpenSSL, 705Operational requirements services,

100–103Optimization, 604, 607Organizational structures

basics, 727–743examples, 745

Organizationsethics-related policies, 323security policy issues, 300–314

Organizing from the Inside Out(Morgenstern), 815

OS-based firewalls, 210–211OSHA (Occupational Safety and

Health Administration)regulations, 257

OSI (Open Systems Interconnection)model, 188–189

OSPF (Open Shortest Path First), 187OSs (operating systems)

add-ons, 43automated loading, 46–47automating installation, 32–33,

763–764caching algorithms, 701checklists, 32, 53–54client server configuration, 79–80

984 Index

OSs (operating systems) (continued )consistent method of loading,

32–33degrading slowly, 43disk-cloning system, 32inconsistent configuration

problems, 33integrity, 43known state, 52less dependent on hardware, 53life cycle, 41–42loading, 41, 46–54loading files, 43maintaining, 44–65manually loading, 763nonuniformity, 46–47preloaded, 51–53promoting, 45RAID 1, 83reloading from scratch, 52scripts or programs to bring machine

up, 409second-class-citizens, 684–685single-function network

appliances, 79upgrading servers, 435–454vendor loaded, 52verifying software compatibility,

438–439web servers, 79workstations, 41

OTP (one-time password), 278Outages, 382, 597Out-of-hours and 24/7 helpdesk

coverage, 357–358Out-of-scope technologies, 350–351Outsider, 934–935Outsourcing

centralization, 515–518colocation (colo) center, 743printing, 577remote access service, 658–661SA (system administrators) teams,

741–743security, 638, 742

Overhead power bus, 146–147Overlay networks, 212–213

PPackages, 673–675, 677–678

services and, 438source code, 673

Packet routing, 210Pages, 689Paging, 530PAM (pluggable authentication

module), 720Parallel/USB cable connection, 569PARIS (Programmable Automatic

Remote Installation Service), 51Parking spaces for mobile items,

175–176Partially automated installation, 49–50Passive mode, 209Passwords, 273, 528–529, 705Patch cables, 161–163, 203Patch panels, 160–161, 204Patches, 33, 54, 56–57, 161PCL, 569PDA, taking along, 786–787PDUs (power distribution units),

147–149power supplies, 86racks, 151

Peer programming, 447Peer-to-peer print architecture,

572–573Peer-to-peer services, 62Penetration testing, 309Per group spoolers, 573Per project verification, 299Perception, 751–765Performance

changes in, 116–117data storage, 604–608intelligent queuing mechanisms, 118NAS, 605optimizing, 604QoS (quality of service), 118RAID, 604–605RAM, 604remote sites, 118–119SANs, 606services, 116–119spindles, 604

Index 985

Performance review, 834Perimeter security, 272Permanent fixes, 407–409Permanent lease, 60Permissions, 678, 710Personal life, balancing with work,

809–810Personal problems, 805Pervasive monitoring, 535Phone number conversion, 465Phone-screening candidates, 877PHP, 691Physical access, 901Physical issues and scripted tests, 441Physical networks, maps of, 205–206Physical security breaches, 300Physical topology, 212Physical visibility, 767Physical-network conversion, 464Physics, knowledge of, 402Pillars approach, 460–461ping, 397–398Pipelining algorithms, 607Pirated software, 330–332pkginfo package, 438Plaintext and wikis, 249Planning

maintenance windows, 477testing and, 416

Platforms, 44–45controlled by management or by SA

team, 66standards, 508–509

Platters, 584–585Policies, documenting overview, 13Policy conformance, 319Policy enforcer, 923–925Policy navigator, 932Policy writer, 301, 918Polling systems, 525POP (Post Office Protocol) server,

109POP3, 556POPI (Protection of Proprietary

Information) program, 316Port 80, 297Portable serial consoles, 171

portmap traceroute function, 397Positive behavior, 824Positive roles, 914–932Positive visibility, 752–765POST requests, 691–692Postgres, 238Postinstall scripts, 54Postmaintenance communication,

490–491Postmaster address, monitoring, 553Posts, 153–154PostScript, 569Potential security incidents, 304Power

ATS, 139–140available from several sources, 177cleaned, 138data centers, 136–148distributing to racks, 146–148emergency lighting, 143extra electrical capacity, 144–145generators, 139–140loss of, 265maximum load, 143–144overhead power bus, 146–147PDUs (power-distribution units),

147–148providing sufficient, 35–36redundancy, 176–177UPS, 138–141

Power cables, separating from datacables, 166

Power supplies, 85–86PowerUser permissions, 291The Practice of Programming

(Kernighan and Pike), 440, 765Precompiling decisions, 785–787Preloaded operating systems, 51–53Premade patch cables, 203Preparation function, 710Prewiring racks, 160Prewiring trade-offs, 166Price per gigabyte-month, 583Price per megabyte, 583print global alias, 98Print jobs, 572Print server, 121, 577

986 Index

Print serviceaccounting policy, 568–569automatic failover, 577basics, 566–576central funnel architecture, 572–573dedicated clerical support, 578documentation, 573–574general printer architecture policy,

568how to print document, 573–574level of centralization, 566–567list of printers, 574load balancing, 577minimizing waste, 575monitoring, 574–575peer-to-peer print architecture,

572–573printer access policy, 570printer equipment standard,

569–570printer label, 574printer naming policy, 571–572redundant systems, 577system design, 572–573

Print systeminstalling new, 54–55spoolers, 573

Printer abuse, 579Printer access policy, 570Printer label, 574Printer naming policy, 571–572Printers

access to, 570canceling print jobs, 570–571confidentaility, 567consistent tray scheme, 574convenience, 567cost, 567dedicated clerical support, 578environmental issues, 575–576equipment standard, 569–570list of, 574maintenance, 568monitoring status, 575naming, 571–572no standards for, 567nonbusiness use, 579

protocols, 569recommended configuration, 570sharing, 566–567special, 567supplies, 569test print, 575toner cartridges, 569

Printingarchitecture policies, 568centralization, 566–568commodity service, 510duplex, 576/etc/passwd file, 229importance of, 565outsourced, 577printer abuse, 579shredding, 578–579

Prioritiesnontechnical managers, 854–855setting, 24–25technical manager, 820–821,

843–845Prioritizing

problems, 27tasks, 781trouble tickets, 354

Privacy and monitoring policy,336–337

Privacy policies, 337, 544private password, 529Privileged access, 327–330Privileged users, 323Privileged-access code of conduct,

327–330Privileges and web servers, 710Proactive solutions, 76Problem preventer, 915–916Problem reports, tracking, 366Problem statements, 369–372Problem-reporting mechanisms, 304Problem-reporting procedures, 304Problems

architectural decisions, 384–385classifying, 368–369educating customers, 384encapsulating test in script or batch

file, 372

Index 987

finding real, 393fixing, 373–376fixing cause, no symptom, 393–394fixing once, 405–413fixing upstream, 823flexible solutions, 371formal decision tree, 368helping customer save face, 371identifying, 367–373Internet routing, 370knowledge of physics, 402learning about customer’s, 392–393more accurate method to reproduce,

378prioritizing, 27process of elimination, 394reproducing, 372–373short-term solutions, 35skipping steps, 378–380solutions, 373–376successive refinement, 394–395support groups, 369systematic about finding cause,

394–395unreported or not affecting users,

372verifying, 372–373verifying repair, 376–378

Problem-solving, 502Procedures documenting overview,

12–13Process and documentation, 416Process of elimination, 394Processes

centralization, 505change management, 422, 424high confidence in completion,

65–66recording knowledge about, 413

procmail, 784, 788Procrastination, 787Product finder, 920–921Product-development group, 312Production server, 717Products

gluing together several, 846integrating or customizing, 845–846

versus protocols, 104–105security-sensitive purposes, 295–298standardizing on, 509volume purchasing deals, 513

Professional code of conduct,324–326

Professional development, 796–797,862–863

Professional organizations, 796Profiles, managing, 720Program Files directory, 438Programming Pearls (Bentley), 765Projects

design documents for larger, 841finishing overview, 14–15kick-off meetings, 100

Promotions, asking for, 812Proprietary email software, 106Proprietary email system, 107Proprietary file formats, 104Proprietary protocols, 104Prosecution policy, 306Protocols

based on TCP, 397–398embedding communications into,

297limiting on WAN, 191open, 104versus products, 104–105proprietary, 104standards-based, 214Sun RPC-based, 397TCP-based, 398vendor-proprietary, 107, 214

Provisioning new services, 360Proximity badge readers, 135Public information, 274Public networks, 61public password, 529Punch block, 198Purchasing, consolidating, 513–515Push-to-talk features, 488PUT, 528–529

QQA server, 717QoS (quality of service), 118, 187

988 Index

QPS (queries per second), 89–90, 694Quick fixes, 35Quick requests, 29–30

RRack frame, 90Rack unit (U), 152Racks

19-inch racks, 152, 155air circulation, 156boltless, 153cable management, 152, 156–158cage nut, 153cooling system, 151data centers, 150–159depth, 155with doors, 156environment, 159extra floor space, 159first of each group of holes, 152four-post, 153, 154height, 154–155hole size, 153keeping power cables away from

network cables, 151labeling, 160mounting servers, 153–154NEBS (Network Equipment Building

System) compliant, 155numbering holes, 152–153organizing equipment, 151overview, 152–153patch panel, 160–161PDUs (power-distribution units),

151posts, 153–154prewiring, 160rack-mount units, 159rails, 152server wiring, 163shelves, 159specific purpose, 151strength, 158threaded, round holes for bolting

equipment, 153too much prewiring, 163–164two-post, 154

vertical power distribution units, 166width, 155wiring infrastructure, 151

Rack-unit, 90Radical print solutions, 374Radios, 170RADIUS authentication protocol,

232RAID (Redundant Array of

Independent Disks), 87–88,585–587

customizing striping, 611–612file snaphots, 599hardware failure, 83hot spare, 587levels, 585monitoring for disk failures, 597not substitute for backup, 598–599optimizing usage by applications,

611–612performance, 604–605reliability, 597triple-mirror configuration, 600

RAID 0, 585–586, 604–605RAID 1, 83, 585–586, 605RAID 2, 586RAID 3, 586, 605RAID 4, 586, 605RAID 5, 586, 605RAID 10, 586–587, 605RAID mirrors to speed backups, 600RAIDs 6-9, 586Rails, 152Raised floors, 137, 147, 159–160RAM, 604Ramanujan, 228RAS devices, 211Raw storage, 589–590RCS, 237, 453RDF (Resource Description

Framework) Site Summary, 251Reactive solutions, 76–77Reading, 796README file, 248“Ready to eat” systems, 503Real-time availability monitoring,

215

Index 989

Real-time monitoring, 523–524,527–534

acknowledging, 532active monitoring systems, 532–534alert policy, 530alerts, 527, 530–531availability monitoring, 527capacity monitoring, 527–528critical outage, 530–531error messages, 531escalation policy, 531–532escalation procedure, 532flexibility, 528handling problems, 539indicating condition of monitored

item, 538scaling problems, 538SNMP (Simple Network Monitoring

Protocol), 528–529standard mechanisms, 527storage requirements, 527test modules, 528

Reboot test, 427–428Rebuild process, 42–43Rebuilding files, 413Recorder, 369–372Recording requests, 786Recruiting, 875–877Recycling, 575–576RedHat Linux, 46, 54Redirect, 715Redundancy

centralized funnel, 573clients, 553–554data centers, 176–177data integrity, 122data synchronization, 122email service, 553–554full, 86–87high-availability sites, 496–497HVAC systems, 176–177list processing hosts, 553load sharing, 87mail relay hosts, 553maintenance windows, 496n + 1, 86–87physical-network design, 205

power, 176–177spoolers, 568upgrades, 123

Redundant multiple-star topology,193–194

Redundant power supplies, 85–86Redundant servers, 89Redundant site, 268Reference lists, 256–257Reflection, 795–796Refresh period, 467Registry, 410Regression testing, 440Reigning in partner network

connections, 279–280Relational Junction, 702Reliability

choosing affordable amount, 598data centers, 110data storage service, 597–598email service, 123, 546–547grouping, 113–115versus leading-edge networks, 217NAS servers, 598RAID, 597remote access service, 656security and, 273servers, 112–115services, 101, 112–115software depots, 670web hosting, 719web servers, 704

Remote accessaspects not to outsource, 659authentication database, 659connecting to Internet, 653cost analysis and reduction,

663–664directly connecting computer to

network, 653home office, 662–663problems lumped together as, 653reenabling, 491removing, 901–902

Remote access outsourcing companies,658–661

Remote access policy, 277

990 Index

Remote access service, 653acceptable use, 656always-on connections, 656from another company’s site, 656authentication, 661basics, 654–662centralization, 658common customers, 654coverage area, 654customers for trial services, 657email, 654encryption, 656ETR (estimated time to repair), 656firewalls, 655–656helpdesk staff, 657high-speed access, 656home use of, 654–655low-cost, convenient solution,

654–656new technologies, 664–665outsourcing, 658–661perimeter security, 661–662policy, 656reliability, 656requirements, 654–656responsibilities for access, 656security and, 655–656security policies, 656service levels, 656–658short-duration connections, 654in trial phase, 657

Remote console access, 80–83, 200Remote email access, 557Remote Installation Service, 46Remote power management, 147Remote sites, 118–119Removable media, 337Removing roadblocks, 821–823Rensselaer Polytechnic Institute, 238Repair person, 914–915Repeatability, 32Repeaters, 488Replacing services overview, 4–5Replication, 676Reporting problems, 359–360Reproducer role, 372–373Reproducing problems, 372–373

Reputation as can-do person, 760–761Request management, 28–29Request Tracker, 29Requesting new services, 359–360Requests, 759, 786Request-tracking software, 354–356Request-tracking system, 246Resources

security team, 300–303servers, 125

Respecting employees, 838–841Response policy, 305–306Restores, 619

accidental file deletion, 621–623archival backups, 624–625complete restores, 625data-recovery SLA and policy, 626disk failure, 623documentation, 638fire drills for, 644individual file, 624process issues, 637–638reasons for, 621–624security implications, 637–638self-service, 622–623setting expectations with customers,

637speed of, 634tape inventory, 642–643technology changes, 648–649time and capacity planning, 633–635training system administrators, 638types, 624–625, 627

Retention policy of email, 559–560Reuse policy, 235Revenue-generating Internet presence,

742Revision control and automation, 416Revision Control System, 425Revision history, 424–426Rewards, 824–825Rewiring building, 202RFCs, static assignment, 60–61Ring topologies, 192–193, 196Rioting-Mob Technique, 459–460RIP, RIPv2 (Routing Information

Protocol), 207

Index 991

Risk analysis, 262–263Risk manager, 303Risks, 415, 417–418Risk-taking, 262RJ-45 connectors, 198Roaming profiles, 78RoboInst, 46, 54Role accounts, 290–291, 293Rolling upgrade, 123Root access, 327Root account, 291Round-robin DNS name server records,

699–700Routers, 86, 187, 207, 209–211Routine updates, 420, 422Routing, 208Routing hosts, 123Routing protocol, 209Routing protocols, 187RPMs, 54RSS feed, 692RT wiki, 253RTT (round-trip time), 101

SSA (system administration) team

attitude of, 756–758becoming system advocate,

760–765blatant disrespect for people, 756building self-confidence, 22business applications support team,

312business partners relationship, 757centralized models, 732–733centralizing, 507clear directions, 842clerk role, 761coaching, 831–833communicating change to, 418–419consultants, 743–745contractors, 743–745customer support, 735–736,

739–741customers, 756customer-to-SA ratio, 729–730decentralized models, 732–733

deployment of new services, 736designing new service architectures,

736Dilbert check, 879distributed network support, 738division of labor, 759e-commerce sites, 746–747eliminating redundancy, 732fixing quick requests, 759funding models, 730–733goals, 830helpdesk, 741helping customer help himself, 756high support costs, 729hiring considerations, 878–882improving follow-through, 22–23infrastructure teams, 737–739in-person orientation, 755–756large company, 746long-term solution, 822–823maintenance, 735–736maintenance contracts, 731management chain’s influence,

733–735manager keeping track of, 825–827medium-sized company, 745–746morale, 821more customized service, 732nonprofit organizations, 747opportunities for growth on, 881outsourcing, 741–743overstaffing, 728perception of, 756–758personality clashes, 878–879priorities and customer expectations,

758–760promoting from within, 737reduced duplication of services,

732requests against policy, 756resentment toward customers, 757restricting size and growth, 730rewarding, 824–825roles, 735–737, 934–937saying no when appropriate, 756security as cooperative effort,

311–312

992 Index

SA (system administration) team(continued )

senior generalists, 736short-term solution, 822–823sizing, 728–730skill selection, 735–737small company, 745specializing, 745–746standardization, 732strengthening, 849strengths and weaknesses, 732structure, 727–743system clerk, 760town hall meeting, 768–770understaffing, 731universities, 747users, 756venting about customers, 757–758viewed as cost center, 730–731vision for, 830–831written policies to guide, 820

safe-delete programs, 383SAGE (System Administrators’ Guild),

72, 324, 399SANs (storage area networks), 180,

588AoE (ATA over Ethernet), 606backups, 600–601caveats, 603–604cluster file systems, 588components from different vendors,

603configuration of underlying

networks, 610generating snaphots of LUNs, 601moving backup traffic off primary

network, 635moving file traffic off main network,

606performance, 606reducing isolated storage, 588tape backup units, 588

SANS Computer Security IncidentHandling: Step-by-Step booklet,307

Sarbanes-Oxley Act, 323Sarbanes-Oxley compliance, 746

Sarbanes-Oxley governing-practiceregulations, 257

SAs (system administrators)assumer, 379attire, 753basics, 364–380boundaries between areas of

responsibility, 285–286career crisis, 834career goals, 812–813career paths, 833–834Carte Blanche Night, 445checklists, 821closer, 380craft worker, 376deexecutioner, 379dress rehearsal for paper

presentations, 768firing, 899–908fixing problems, 533flexibility, 371good first impression, 752–755greeting customer, 364–367helpdesk, 736–737high-quality handoffs, 381high-stress work life, 855hiring, 20, 871–896hit-and-run sysadmin, 379holistic improvement, 381increased customer familiarity,

381informed consent, 324interaction with customer at

appointment, 753interesting projects, 744, 824involved in hiring process, 760isolation, 27job description, 872–874law enforcement and, 332–335learning from mistakes, 832lunch with customers, 773management expectations, 26management meetings, 766–767meetings with single point of contact,

866–868metrics, 384misdelegator, 379

Index 993

model-based training, 380–381monitoring system, 534morale, 855–857negative roles, 932–934new hire’s first day, 754–755nonverifier, 379ogre, 378outsourcing remote access,

658–659PC delivery, 755physical visibility, 767positive roles, 914–932positive visibility, 755problem identification, 367–373professional development, 862–863promoting to management, 797reproducer role, 372–373selling security to, 314setting priorities for, 734shared responsibilities for machines,

285–286special announcements for major

outages, 382standards, 66stereotypes, 378–380system status web page, 765–766technical development, 833trend analysis, 382–384understanding customers

expectations, 99visibility paradox, 765working alone, 380wrong fixer, 379yelling at people, 753–754

SAS-70 (Statement of AuditingStandards No. 70), 178

ScalingCGI programs, 702challenges, 702–703choosing method, 701–702data flow analysis, 124–125database usage, 702email service, 554–556gradual, 701–702horizontal, 699–700IMAP4, 556importance of, 703

load balancers, 702POP3, 556problems and monitoring, 537–539pulling data from several sources,

702services, 100subsystems and common resources,

702vertical, 699, 700–701web services, 699–703

SCCS (Source Code Control System),237

Schedulingchange management, 419–422change-freeze times, 422, 423maintenance windows, 475–476,

495major updates, 420no changes on Friday, 421routine update, 420sensitive updates, 420, 422

SCM (Software ConfigurationManagement), 67

Scope of responsibility, 350Scope of support, 348–351Scope of work policy, 821Scope policy, 231–233Scope-of-support policy, 348–350script command, 245Scripting languages, 710Scripts

to bring machine up, 409helpdesks, 352OK or FAIL message, 440outputting commands to do task,

763sharing, 411software verification tests, 439–442

Search engines web repository,250–251

Search facility, 250–251SEC (Securities and Exchange

Commission), 329–330Second tier of support, 352–353Second-best situation, 798Second-class-citizens, 684–685Secure connections, 704–706

994 Index

Securing hosts before going live, 290Security, 271

applications, 709–710asking right questions, 273–275authentication, 290–293authorization, 290–293authorization matrix, 293–295automating data access, 710bulk emails, 338certificates, 704–706companies, 314competitive advantage, 314contacts, 316–317containment, 63–64cooperative effort, 311–312data, 271–272data centers, 134–136defeating or finding way around, 285directory traversal, 707–708effectively selling, 313–314email filtering, 284email service, 544, 556–557enabling people to work effectively,

285–286external sites, 717features consistently enabled, 33firewalls, 284form-field corruption, 708hosts determining hostname, 62IDF (intermediate distribution

frame), 200implications of restores, 637–638information, 313–314information protection, 274internal auditing, 298–300internal web services, 704Internet, 271known, standard configurations, 287limiting potential damage, 709logging, 710logs, 299malware protection, 284mean time to attack, 289meeting business needs, 285–287metrics, 317monitoring, 525names, 228

off-site backup storage, 646only as good as weakest link, 283outsourcing, 742passwords, 273permissions and privileges, 710pervasive, 315–316physical breaches, 300process for someone leaving

company, 287projects verification, 299protecting important data, 275–276protecting service availability,

274–275protecting web server application,

706–707protecting web server content,

707–708raising awareness, 316reliability, 273remote access outsourcing

companies, 660remote access service, 655–656remote console, 82–83remote email access, 557secure connections, 704–706secure perimeter, 661–662security-sensitive products, 296selecting right products and vendors,

295–298servers, 97shared development environment,

286–287single administrative domain, 217sites without, 284–285SNMP problems, 529solid infrastructure, 287–288spotlighting bad behavior, 291SQL injection, 708staff disagreeing with management

decisions, 281state of, 284statically assigned IP addresses, 61technologies, 316–317theft of resources, 275through obscurity, 296UNIX, 271validating input, 709

Index 995

vendor-specific, 707VPNs, 284Web, 271web hosting, 719web servers, 703–710web services, 703–710Windows, 271

Security architect, 301–302, 318Security bulletins, 289Security conferences, 316Security consultants, 308–309Security disasters, 268–269Security incidents, 303–307Security industry contacts, 316Security operations staff, 302Security patches, 704Security perimeter, 317Security policies, 271

AUP (acceptable-use policy),276–277

basics, 272–315better technology means less, 278communication policy, 307cooperation from other departments,

276defense in depth, 272disconnection policy, 306–307documenting, 276–283external audits, 308–309HHA (handheld authenticators), 278lack hampering security team,

278–279log-retention policy, 277management and organizational

issues, 300–314monitoring and privacy policy, 277network connectivity policy, 277outside auditing company, 300partner network connections,

279–280perimeter security, 272remote access policy, 277response policy, 305–306technical staff, 283–300without management support,

281–282Security policy council, 282–283

Security professionals, 316Security programs

e-commerce sites, 319–320large companies, 319medium-size company, 318–319organization profiles, 317–321small company, 318universities, 320–321

Security Symposium, 797Security system, 273Security team

advisories, 289auditor, 302benchmarking company, 301business applications support team,

312contacts in industry, 300–301cross-functional teams, 310–313effectively selling security, 313–314field offices, 312–313full-disclosure mailing lists, 289implementer, 302incident response, 303–307incident-response team, 303independent feedback, 308intercompany security focus

groups, 301involved at outset, 311knowing latest attacks, 289legal department, 310points of contact, 304policy writer, 301product-development group, 312reasonable staffing levels, 300resources, 300–303risk manager, 303security architect, 301–302security bulletins, 289security operations staff, 302variety of skills, 301–303

Security-awareness program, 318Security-sensitive logs, 299Security-sensitive products, 295–298Self-help books, 815Self-help desk, 255Self-help systems, 345Self-service restores, 622–623

996 Index

Selling position, 892–893Sendmail, 545Senior generalists, 736Senior management, 308, 313Sensitive updates, 420, 422SEPP, 672Sequential names, 227Sequential reads, 586Serial console concentrators, 80–81Serial console servers, 486Serial consoles, 81Serial port-based devices, 80Serial ports, monitoring, 81Server appliances, 84–85Server computers, 73Server upgrades, 448–449Server virtualization, 506–507Servers

access to, 97buying hardware for, 69–71colocation centers, 71connected to multiple networks,

110controlled introduction, 74–75cooling and ventilation, 71cost, 73, 90cost of hardware, 72–74CPUs, 70data center, 78–79data integrity, 78disposable, 91downtime, 74extensibility, 70front-mountable, 153full redundancy, 122full versus N + 1 redundancy, 86–87growing number of customers, 117hardware, 69heterogeneous environments, 72high availability options, 71high availability requirements, 135high performance throughput, 70homogeneous environments, 72hot-swap components, 87–88hot-swap hardware, 74I/O, 70KVM switches, 80–81

lack of similar configurations on, 506large groups of similar, 74listing contents of directories, 248load balancers, 89load sharing, 87locating in data center, 110location of, 78–79LUN (logical unit number), 588maintenance contracts, 71, 74–78management options, 71MIL-SPEC requirements, 72mirroring boot disks, 83mounting in racks, 153–154MTTR (mean time to repair), 73multiple inexpensive, 89–92name conflicts, 226no side-access needs, 71operating system configuration,

79–80OS configuration, 79–80peak utilization, 117rack mounting, 78–79rack-mountable, 70–71redundant hardware, 74redundant power supplies, 85–86reliability, 110, 112–115reliability and service ability, 84–89remote console access, 80–83required software, 79resources, 125restricting direct login access, 111security, 97separate networks for administrative

functions, 89server appliances, 84–85services, 95, 118simplicity, 97spare parts, 74–78terminals, 80upgrade options, 70upgrading, 435–454UPS (uninterruptible power supply),

35usage patterns, 125vendors, 72versatility, 70wiring, 163

Index 997

Serviceconversions, 457protection, 614

Service access, 901–904Service checklist, 436–438, 453Service conversions

adoption period, 464avoiding, 468–469back-out plan, 465–466basics, 458communication, 461–462dividing into tasks, 460–461doing it all at once, 463–465failure, 466flash-cuts, 463–465future directions for product, 468gradual, 463instant rollback, 467–468invisible change, 457layers versus pillars, 460–461minimizing intrusiveness, 458–460old and new services available

simultaneously, 464physical-network conversion, 464Rioting-Mob Technique, 459–460simultaneously for everyone,

464–465slowly rolling out, 463solid infrastructure in place, 458test group, 463training, 462vendor support, 470without service interruption, 459

Services, 95adding and removing at same time,

450additional requirements, 96administrative interface, 100adversely affecting, 112associated with service-based name,

121authentication and authorization

service, 97average size of data loaded, 125bad first impression, 117basic requirements, 95basics, 96–120

budget, 103business-specific, 95capacity planning, 119cascading failures, 97catch-22 dependencies, 111centralization, 98, 116, 505, 508client systems, 97closed, 104complexity, 107–108consolidating, 506critical, 122customer requirements, 96, 98–100customers relying on, 438data storage, 596–604dataflow analysis for scaling,

124–125dedicated machines, 120–122default responsible entity, 532depending on few components, 113desired features, 101disabling, 450environment, 96, 110–111escalation procedure, 532failover system, 122features wanted in, 98–99first impressions, 120five-year vision, 864–866full redundancy, 122–123function-based names, 109fundamental, 95generic, 95hard outages, 114hardware and software for, 108–109high level of availability, 110independent, 98, 115infrastructure, 97integrated into helpdesk process, 116kick-off meetings, 100latency, 103listing, 453lists of critical servers, 34load testing, 117machine independence, 109machines and software part of, 97mashup applications, 721–722Microsoft Windows, 410modeling transactions, 124

998 Index

Services (continued )monitoring, 103, 119more supportable, 98moving, 109network performance issues, 101network topology, 113–114no customer requirements, 98no direct or indirect customers, 438open architecture, 96, 104–107open protocols, 96operational requirements, 100–103packages and, 438performance, 96, 116–119potential economies of scale, 501protecting availability, 274–275prototyping phase, 657–658providing limited availability,

493–494redundancy, 112reliability, 96, 97, 101, 112–115relying on email, 96relying on network, 96relying on other services, 96–97remote sites, 118–119reorganizing, 501restricted access, 111–112restricting direct login access, 111rolled out to customers, 120scaling, 100server-class machines, 96servers, 118simple text-based protocols, 441simplicity, 107–108, 113single or multiple servers, 115single points of failure, 113SLA (service-level agreement), 99soft outages, 114splitting, 121–122stand-alone machines providing, 96standards, 116talking directly to clients, 62testing, 469tied to global alias, 98tied to IP addresses, 109, 121transaction based, 124trouble tickets, 103tying to machine, 98

upgrade path, 100–101usability trials, 99vendor relations, 108virtual address, 109Web-based, 469

Services Control Panel, 410Shared accounts, 290–292Shared development environment,

286–287Shared directory, 248Shared role accounts, 293Shared voicemail, 292–293Shoe-shining effect, 634Short-term solution, 822–823Shredding, 578–579Shutdown sequence, 485Shutdown/boot sequence, 483–485SIDs (Windows), 223Simple host routing, 207–209Single, global namespaces, 232–233Single administrative domain, 216–217Single authentication database, 905Single points of failure, 510, 512Single-function network appliances, 79Single-homed hosts, 208Sites

assessing overview, 7–8used to launch new attacks, 307virtual connections between, 212without security, 284–285

Skill level, 874–875SLAs (service-level agreements), 32

backup and restore system, 621backups, 625–626monitoring conformance, 525remote access outsourcing

companies, 660services, 99web service, 694

Slow bureaucrats, 789–790Small company

SA (system administrators) team, 745security program, 318

Smart pipelining algorithm, 607SMB (Server Message Block) print

protocol, 569SME (subject matter expert), 374, 375

Index 999

SMS and automating software updates,54

SMTP (Simple Mail Transfer Protocol),104, 189, 398, 548

smtp global alias, 98SMTP server, 109Snake Oil Warning Signs: Encryption

Software to Avoid (Curtin), 316Snake Oil Warning Sings: Encryption

Software to Avoid (Curtin), 559Snapshots of filesystems, 622SNMP (Simple Network Monitoring

Protocol), 528–529SNMP packets, 529SNMPv2, 526SNMPv2 polling, 527SNMPv2 traps, 527Social engineering, 303, 308–309,

333–334Social engineers, 334SOCKS relay, 121Soft emotions, 791–792Soft outages, 114Software

contribution policy, 671–672installation test suite, 440labeling ports, 168management approval for

downloading, 331no longer supported, 439old and new versions on same

machine, 452regression testing, 440reuse policy, 235selecting for support depot, 672single place for customers to look

for, 669tracking licenses, 672upgrade available but works only on

new OS, 439upgrading to release supported on

both OSs, 439verification tests, 439–442verifying compatibility, 438–439

Software depots, 667bug fixes, 670bugs and debugging, 671

building and installing packages, 671commercial software, 684contributing software policy,

671–672customer wants from, 670deletion policy, 671–672different configurations for different

hosts, 682documenting local procedure for

injecting new software packages,672–673

justification for, 669–670librarians, 669local replication, 683managing UNIX symbolic links, 672new versions of package, 670OSs supported, 671packages maintained by particular

person, 671reliability requirements, 670requests for software, 669–670, 672same software on all hosts, 670scope of distribution, 672second-class-citizens, 684–685Solaris, 667–668technical expectations, 670tracking licenses, 672UNIX, 668, 673–679upgrades, 671Windows, 668, 679–682

Software Distributor (SD-UX), 54Software licenses, 332Software piracy, 330–332Software updates, 54–57Solaris

automating software updates, 54JumpStart, 46, 48, 65, 406software depot, 667–668

solution designer, 921Solutions, 373–376

building from scratch, 846–847executing, 375–376expensive, 374proposals, 374radical print, 374radical print solutions, 374selecting, 374–375

1000 Index

Solutions database, 246SONET (synchronous optical

network), 188Source Code Control System, 425SOURCENAME script, 673–674SourceSafe, 425Spam, 703

blocking, 550email service, 549–550

Spammers, 338Spare parts, 74–78

cross-shipped, 77valuable, 175

Spare-parts kit, 77–78Spares, organizing, 174Special applications, 53Specialization and centralization,

508Special-purpose formats, 692Special-purpose machines, 234Spindles, 584–585, 604Splitting

center-of-the-universe host,122

Splitting central machine, 121Splitting services, 121–122Spoolers

monitoring, 574–575print system, 573redundancy, 568

Spot coolers, 146Spreadsheets

service checklist, 436–438Spyware, 284SQL injection, 708SQL lookups, 720SQL (Structured Query Language)

request, 103SSH package, 80SSL (Secure Sockets Layer)

cryptographic certificates, 705Staff

defining processes for, 352Staff meetings

knowledge transfer, 859nontechnical managers,

858–859

Staffing helpdesks, 347Stakeholders, 100, 429

hardware standards, 595signing off on each change, 429

Stalled processesbeing a good listener, 822being good listener, 822communication, 822restarting, 821–823

Standard configurationcustomers involved in, 66

Standard configurationsmultiple, 66–67

Standard protocols, 107, 468Standardization

data storage, 594–596Standardizing on certain phrases,

793–794Standardizing on products, 509Standards-based protocols, 214Star topology, 191–192, 196

multiple stars variant, 192single-point-of-failure problem,

191–192Start-up scripts, 409Static documents, 694–695Static files, 701Static leases

hosts, 62Static web server, 694–695Static web sites

document root, 695status, 397Status messages, 766Stop-gap measures

preventing from becomingpermanent solutions, 50

Storagedocumentation, 247–248

Storage consolidation, 506Storage devices

confusing speed onf, 610other ways of networking, 606

Storage serversallocating on group-by-group

basis, 588serving many groups, 589

Index 1001

Storage SLA, 596–597availability, 596latency, 596response time, 596

Storage standards, 594–596Storage subsystems

discarding, 595Storage-needs assessment, 590–591Streaming, 692Streaming video

latency, 103Streaming-media, 696–697Stress

avoiding, 25Strictly confidential information, 274Striping, 585, 586

customizing, 611–612StudlyCaps, 249SubVersion, 248, 425Subzones, 233Successive refinement, 394–395sudo, 383sudo command, 714sudo program, 329SUID (set user ID) programs, 383Summary statements, 794–795Sun Microsystems, 799Sun OS 5.x

JumpStart, 51Sun RPC-based protocols, 397SunOS 4.x

PARIS (ProgrammableAutomatic Remote InstallationService), 51

unable to automate, 51Supercomputers, 130Superuser account

access from unknown machine,293

Suppliesorganizing, 174

Supportcustomer solutions, 847defining scope of, 348–351first tier of, 352–353how long should average request

take to complete, 349

second tier of, 352–353what is being supported, 348when provided, 348–349who will be supported, 348

Support groupsproblems, 369

Support structure, 808/sw/contrib directory, 678/sw/default/bin directory, 674Switches, 187, 209swlist package, 438Symbolic links

managing, 675Symptoms

fixing, 393–394fixing without fixing root cause,

412System

balancing stress on, 591–592end-to-end understanding,

400–402increasing total reliability, 20

System Administrator’s Code of Ethics,324–3267

System administration, 364accountability for actions, 29as cost center, 734tips for improving, 28–36

System Administrator teamdefining scope of responsibility

policy, 31emergencies, 29handling day-to-day interruptions,

29–30specialization, 29

System Administrator team membertools, 11–12

System advocates, 760–765System boot scripts, 427System clerk, 760system clerk, 918–919System configuration files, 424–426system file changes, 906System files, 428System Management Service, 55–56System software, updating, 54–57System status web page, 765–766

1002 Index

Systemsdiversity in, 512documenting overview, 12–13polling, 525speeding up overview, 16

Systems administratorscoping with big influx, 17keeping happy overview, 16

Systems administrators team, 18

TTape backup units, 588Tape drives, 642

nine-track, 649shoe-shining effect, 634speeds, 634

Tape inventory, 642–643tar files, 673Tasks

automating, 763–764checklists of, 34daily, 785domino effect, 759intrusive, 460layers approach, 460–461monitoring, 524not intrusive, 460order performed, 30outsourcing, 515pillars approach, 460–461prioritizing, 30, 781

TCP, 527, 700TCP connections, 526TCP-based protocols, 397–398,

398tcpdump, 395TCP/IP, 191TCP/IP (Transmission Control

Protocol/Internet Protocol), 187TCP/IP Illustrated, Volume 1

(Stevens), 398TCP/IP networking, 188–189TDD (Test-Driven Development), 442Tech rehearsal, 452Technical development, 833technical interviewing, 886–890Technical lead, 797

Technical library or scrapbook,257–258

Technical manageras bad guy, 828buy-versus-build decision, 845–848clear directions, 842–843coaching, 831–833decisions, 843–848decisions that appear contrary to

direction, 830–831employees, 838–843informing SAs of important events,

840involved with staff and projects, 841listening to employees, 840–841micromanaging, 841positive about abilities and direction,

841–842priorities, 843–845recognition for your

accomplishments, 850respecting employees, 838–841responsibilities, 843role model, 838roles, 843satisfied in role of, 850selling department to senior

management, 849–850strengthening SA team, 849vision leader, 830–831

Technical managersautomated reports, 826basics, 819–848blame for failures, 827brainstorming solutions, 822–823budgets, 834–835bureaucratic tasks, 822career paths, 833–834communicating priorities, 820–821contract negotiations and

bureaucratic tasks, 827–828enforcing company policy, 828–829keeping track of team, 825–827knowledgeable about new

technology, 835meetings with staff, 825–826nontechnical managers and, 835–837

Index 1003

pessimistic estimates, 836recognizing and rewarding successes,

827removing roadblocks, 821–823reports and, 825responsibilities, 820–835rewards, 824–825SLAs, 820soft issues, 822structure to achieve goals, 821supporting role for team, 827–830team morale, 821technical development, 833tracking group metrics, 827written policies to guide SA team,

820–821Technical staff

budgets, 860–862security policies, 283–300

technocrat, 927–928Technologies

security, 316–317Technology platforms, 697technology staller, 932tee command, 395Telecommunications industry

high-reliability data centers, 177–178TELNET, 80, 398Templates

announcing upgrade to customers,445–446

database-driven web sites, 695DHCP systems, 58–60

Temporary fix, 412Temporary fixes

avoiding, 407–409TERM variable, 406Terminal

capture-to-file feature, 245Terminal servers, 171Terminals, 80termination checklist, 900–901Test plan, 417Test print, 575Testing

alert system, 531comprehensive system, 489–490

finding problems, 490server upgrade, 447

Tests integrated into real-timemonitoring system, 451

TFTP (Trivial File Transfer Protocol)server, 59

Theft of intellectual property, 267Theft of resources, 275Thematic names, 225, 227Third-party spying

wireless communication, 530Third-party web hosting, 718–721Ticket system

knowledge base flag, 246Tickets

email creation, 408Time management, 780–790

daily planning, 782–783daily tasks, 785difficulty of, 780–781finding free time, 788goal setting, 781–782handling paper once, 783–784human time wasters, 789interruptions, 780–781managers, 813precompiling decisions, 785–787slow bureaucrats, 789–790staying focused, 785training, 790

Time Management for SystemAdministators (Limoncelli), 815

Time saving policiesdefining emergencies, 31defining scope of SA team’s

responsibility policy, 31how people get help policy, 31

Time server, 121Time-drain

fixing biggest, 34–35Timeouts

data storage, 610Time-saving policies, 30–32

written, 31timing

hiring SAs (system administrators),877–878

1004 Index

Tivoli, 367TLS (Transport Layer Security), 704/tmp directory, 56Token-card authentication server,

121Tom’s dream data center, 179–182Tool chain, 685Tools

better for debugging, 399–400buzzword-compliant, 399centralizing, 116characteristics of good, 397debugging, 395–398ensuring return, 12evaluating, 399evaluation, 400formal training on, 400knowing why it draws conclusion,

396–397NFS mounting tools, 397System Administrator team member,

11–12Tools and supplies

data centers, 173–175Topologies, 191–197

chaos topology, 195flat network topology, 197functional group-based topology,

197location-based topology, 197logical network topology, 195–197multiple-star topology, 192multistar topology, 196redundant multiple-star topology,

193–194ring topologies, 192–193, 196star topology, 191–192, 196

Town hall meetings, 768–770customers, 768–770dress rehearsal for paper

presentations, 768feedback from customers, 769introductions, 769meeting review, 770planning, 768presentations, 768question-and-answer sessions, 768

review, 769show and tell, 769–770welcome, 768

Trac wiki, 253traceroute, 397, 398Tracking changes, 319Tracking problem reports, 366Tracks, 584Training

customers, 462service conversions, 462

Transactionsmodeling, 124successfully completing, 537

Transparent failover, 553–554Traps

SNMP (Simple Network MonitoringProtocol), 528

Trend analysisSAs (System administrators),

382–384Trending historical data, 493Triple-mirror configuration, 600Trojan horse, 671Trouble reports

enlightened attitude toward, 758Trouble tickets

enlightened attitude toward, 758prioritizing, 354

Trouble-ticket system, 28–29documentation, 246

Trouble-tracking software, 366Turning as debugging, 399Two-post posts, 153Two-post racks, 154

UUCE (unsolicited commercial email),

549–550UID

all-accounts usage, 234UID ranges, 234UIDs (UNIX), 223Universal client, 690, 691Universities

acceptable-use policy, 320codes of conduct, 327

Index 1005

constraints, 476monitoring and privacy policy,

321no budget for centralized services,

747–748SA (system administrators) team,

747security programs, 320–321staffing helpdesks, 347

UNIXadd-on packages for, 452–453automounter, 231boot-time scripts, 438calendar command, 419at cmd, 65code control systems, 425crontab files, 438customized version, 52diff command, 377, 440/etc/ethers file, 59/etc/hosts file, 59–60/etc/passwd file, 578history command, 245level 0 backup, 620level 1 backup, 620listing TCP/IP and UDP/IP ports,

438login IDs, 225maintaining revision history,

425–426make command, 236reviewing installed software, 438root account, 291script command, 245security, 271set of UIDs, 223software depot, 668strict permissions on

directories, 43sudo command, 714SUID (set user ID) programs, 383syncing write buffers to disk before

halting system, 608system bot scripts modified by

hand, 427tee command, 395tools, 667

/usr/local/bin, 667/var/log directory, 710Web server Apache, 452wrapper scripts, 671

UNIX Backup and Recovery(Preston), 620

UNIX desktopsconfigured email servers, 547

UNIX kernels, 396UNIX printers

names, 571–572UNIX servers

later users for tests, 442UNIX shells

deleting files, 410–411UNIX software

installation, 668UNIX software depot

archiving installation media, 678area where customers can install

software, 678automating tasks, 677automounter map, 675–677commercial software, 684control over who can add packages,

678defining standard way of specifying

OSs, 677deleting packages, 677/home/src directory, 673managing disk space, 677–678managing symbolic links and

automounter maps, 676–677master file, 677network of hosts, 675–677NFS access, 681obsolete packages, 676packages, 673policies to support older OSs, 676programs in package, 675reliability requirements, 676replication, 676SOURCENAME script, 673–674/sw/contrib directory, 678/sw/default/bin directory, 674symbolic links, 674–675wrappers, 679

1006 Index

UNIX software depotsdifferent configurations for different

hosts, 682local replication, 683NFS caches, 683

UNIX sysemsNFS, 110–111

UNIX system/etc/passwd file, 229/etc/shadow file, 229login IDs, 229/var/adm/CHANGES file, 451

UNIX systemsassembly-line approach to

processing, 395configuring to send email from

command line, 408crontabs, 78debugging, 396distributing printcap information,

572mail-processing utilities, 784Network Information Service, 232no root access for user, 78simple host routing, 207–208sudo program, 329tcpdump, 395/var directory, 78

UNIX workstations, 130UNIX/Linux

filesystem, 587Unknown state, 42Unproductive workplace, 806Unrealistic promises, 503–504unrequested solution person, 922Unsafe workplace, 806Unsecured networks, 289Updates

absolute cutoff conditions, 418authentication DNS, 63back-out plan, 418communication plan, 57differences from installations,

55–56distributed to all hosts, 57dual-boot, 56host already in use, 55

host in usable state, 55host not connected, 56known state, 55lease times aiding in propagating,

64–65live users, 55–56major, 420, 422network parameters, 57–61performing on native network of

host, 55physical access not required, 55routine, 420, 422security-sensitive products, 297sensitive, 420–421, 422system software and applications,

54–57Updating applications, 54–57Updating system software, 54–57Upgrades

advanced planning reducing need,468

automating, 33redundancy, 123

Upgradingapplication servers, 211clones, 443critical DNS server, 453–454

Upgrading serversadding and removing services at

same time, 450announcing upgrade to customers,

445–446basics, 435–449customer dependency check, 437dress rehearsal, 451–452exaggerating time estimates, 444executing tests, 446fresh installs, 450–451installing of old and new versions on

same machine, 452length of time, 444locking out customers, 446–447logging system changes, 451minimal changes from base,

452–453multiple system administrators,

447

Index 1007

review meeting with keyrepresentatives, 437

selecting maintenance window,443–445

service checklist, 436–438tech rehearsal, 452testing your work, 447tests integrated into real-time

monitoring system, 451verification tests, 439–442verifying software compatibility,

438–439when, 444writing back-out plan, 443

UPS (uninterruptible power supply),35, 138–141, 265

cooling, 139environmental requirements,

140–141failure, 177lasting longer than hour, 139maintenance, 140–141notifying staff in case of failure or

other problems, 138power outages, 138switch to bypass, 140trickle-charge batteries, 141

Upward delegation, 813–814URL (uniform resource locator),

690changing, 715inconsistent, 715messy, 715

URL namespaceplanning, 715

Usabilitysecurity-sensitive products,

296–297Usable storage, 589–590USENIX, 399, 848USENIX (Advanced Computing

Systems Association), 796USENIX Annual Technical Conference,

796–797USENIX LISA conference, 562User base

high attrition rate, 18

Users, 756balance between full access and

restricting, 43ethics-related policies, 323

USS (user code of conduct), 326Utilization data, 524

VVariables

SNMP (Simple Network MonitoringProtocol), 528

VAX/VMS operating system, 622vendor liaison, 928–929Vendor loaded operating systems, 52Vendor relations

services, 108Vendor support

networks, 190Vendor-proprietary protocols, 107, 214Vendors

business computers, 70–72configurations tuned for particular

applications, 108home computers, 70–72network, 213–214product lines computers, 70–72proprietary protocols, 104RMA (returned merchandise

authorization), 77security bulletins, 289security-sensitive purposes, 295–298server computers, 70–72support for service conversions, 470

Vendor-specific security, 707Verification tests

automating, 441Hello. World program, 440–442manual, 441–442OK or FAIL message, 440

Verifyingproblem repair, 376–378problems, 372–373

Version control system, 453Versions

storing differences, 425Vertical cable management, 158Vertical scaling, 699, 700–701

1008 Index

Veto power, 505vir shell script, 425Virtual connections between sites,

212Virtual helpdesks, 345

welcoming, 346Virtual hosts, 506–507Virtual machines

defining state, 507migrating onto spare machine, 507rebalancing workload, 507

Virtual servers, 91Virtualization cluster, 507Virus blocking

email service, 549–550Viruses, 284

email system, 557introduced through pirated software,

330web sites, 704

Visibility, 751desk location and, 767newsletters, 770office location and, 767status messages, 766town meetings, 768–770

Visibility paradox, 765Vision leader, 830–831visionary, 929VLAN, 212

large LANs using, 212–213network topology diagrams, 213

Voicemailconfidential information, 292shared, 292–293

Volumes, 587filesystem, 587

VPATH facility, 673VPN service, 664VPNs, 187, 284VT-100 terminal, 80

WW3C (World Wide Web Consortium),

689WAFL file system, 586WAN (wide area network), 102

WAN connectionsdocumentation, 207

WANs, 187, 188limiting protocols, 191redundant multiple-star topology,

194Ring topologies, 193star topology, 191–192

Wattage monitor, 610Web

data formats, 692open standards, 689security, 271special-purpose formats, 692

Web applications, 690managing profiles, 720standard formats for exchanging

data between, 721–722Web browser

system status web page, 766Web browsers, 690, 691

multimedia files, 692Web client, 691Web content, 717

accessing, 689Web council, 711–712

change control, 712–713Web farms

redundant servers, 89Web forms

intruder modification, 708Web hosting, 717

advantages, 718managing profiles, 719–721reliability, 719security, 719third-party, 718–721unified login, 719–721

Web outsourcingadvantages, 718–719disadvantages, 719hosted backups, 719web dashboard, 719

Web pagesdynamically generated, 691HTML or HTML derivitive, 692interactive, 691–692

Index 1009

Web proxieslayers approach, 461

Web repositorysearch engines, 250–251

Web server ApacheUNIX, 452

Web server appliances, 84Web server software

authentication, 720Web servers, 691

adding modules or configurationdirectives, 716

alternative ports, 697–698building manageable generic,

714–718directory traversal, 707–708Horizontal scaling, 699–700letting others run web programs, 716limiting potential damage, 709logging, 698, 710managing profiles, 720monitoring errors, 698multimedia servers, 696–697multiple network interfaces, 698OS (operating system), 79overloaded by requests, 699pages, 689permissions, 710privileges, 710protecting application, 706–707protecting content, 707–708questions to ask about, 714redirect, 715reliability, 704round-robin DNS name server

records, 699–700security, 703–710server-specific information, 699static documents, 694–695validating input, 709vertical scaling, 700–701web-specific vulnerabilities, 707

Web servicearchitectures, 694–698basics, 690–718building blocks, 690–693CGI servers, 695

database-driven web sites, 695–696multimedia servers, 696–697SLAs (service level agreements), 694static web server, 694–695URL (uniform resource locator), 690web servers, 691

Web servicesAJAX, 691–692centralizing, 506content management, 710–714Horizontal scaling, 699–700load balancers, 700monitoring, 698–699multiple servers on one host, 697–698scaling, 699–703security, 703–710vertical scaling, 700–701web client, 691

Web sites, 399, 689basic principles for planning,

715–716building from scratch overview, 3certificates, 704–706CGI programs, 701CGI servers, 695change control, 712–716changes, 713compromised, 704content updates, 712database-driven, 695–696databases, 701deployment process for new releases,

717–718DNS hosting, 717document repository, 248domain registration, 717fixes, 713form-field corruption, 708growing overview, 4hijacked, 703–704HTTP over SSL (Secure Sockets

Layer), 704–705political issue, 713–714publication system, 253secure connections, 704–706separate configuration files, 715setting policy, 693–694

1010 Index

Web sites (continued )SQL injection, 708static, 694–695static files, 701updates, 713updating content, 716viruses, 704visitors, 704web content, 717web hosting, 717web system administrator, 693web team, 711–712webmaster, 693–694

Web system administrator, 693Web team, 711–712Web-based documentation repository,

249–250Web-based request system

provisioning new services, 360Web-based service

surfing web anonymously, 335Web-based Services, 469Webmaster, 693–694, 711, 712Week-long conferences, 796, 862WiFi networks

network access control, 61Wiki Encyclopedia, 252Wiki sites, 692Wikipedia, 252, 258Wikis, 249–250, 252

ease of use, 251enabling comments, 254FAQ (Frequently Asked Questions),

256formatting commands, 249help picking, 250how-to docs, 255–256HTML (Hypertext Markup

Language), 249internal group-specific documents,

255low barrier to entry, 254naming pages, 249off-site links, 258placeholder pages, 249plaintext, 249procedures, 257

reference lists, 256–257requests through ticket system, 255revision control, 254self-help desk, 255source-code control system, 249structure, 254taxonomy, 254technical library or scrapbook,

257–258wiki-specific embedded formatting

tags or commands, 249WikiWikiWeb, 249WikiWords, 249Windows

Administrator account, 291code control systems, 425distribution-server model, 668–669filesystem, 587loading files into various system

directories, 43login scripts, 115network disk, 668network-based software push

system, 668PowerUser permissions, 291security, 271software depot, 668WINS directory, 223

Windows NTautomating installation, 47listing TCP/IP and UDP/IP ports, 438Services console, 438SMB (Server Message Block) print

protocol, 569unique SID (security ID), 51

Windows NT Backup and Restore(Leber), 620

Windows platformsroaming profiles, 78storing data on local machines, 78

Windows software depot, 669commercial software, 684selecting software for, 672

Windows software depots, 679Admin directory, 680–681certain products approved for all

systems, 680–681

Index 1011

directory for each package, 681disk images directory, 680Experimental directory, 680notes about software, 681Preinstalled directory, 680replicating, 681–682self-installed software, 680special installation prohibitions and

controls, 680–681Standard directory, 680version-specific packages, 681

WINS directory, 223Wireless communication

as alerting mechanism, 530third-party spying, 530

Wiringdata centers, 159–166good cable-management

practices, 151higher-quality copper or fiber,

198IDF (intermediate distribution

frame), 198networks, 198payoff for good, 164–165servers, 163

Wiring closet, 197–203Wiring closets

access to, 201floorplan for area served,

200protected power, 201training classes, 200

Workbalancing with personal life,

809–810

Work stoppagesurviving overview, 10–11

Workbenchdata centers, 172–173

Worksationsmaintenance contracts, 74

Workstations, 41automated installation, 43bulk-license popular packages, 331defining, 41disk failure, 78long life cycles, 41maintaining operating systems,

44–65managing operating systems, 41manual installation, 43network configuration, 57–61reinstallation, 43–44spareparts, 74storing data on servers, 78updating system software and

applications, 54–57Worms, 284Wrapper scripts, 671Wrappers, 679Write streams

streamlining, 612

Xxed shell script, 425XML, 692XSRF (Cross-Site Reverse Forgery),

710

YYahoo!, 90