cit 470: advanced network and system administration

CIT 470: Advanced Network and System Administration Slide #1

CIT 470: Advanced Network and System Administration

Workstations and Servers


Topics

1. Machine Lifecycle2. Automated Installs3. Server Hardware4. Services

Machine Lifecycle


Workstation Management


States of Machines

NewA new machine

CleanOS installed, but not yet configured for environment.

ConfiguredConfigured correctly for the operating environment.

UnknownMisconfigured, broken, newly discovered, etc.

Off Retired/surplussed


State Transitions

BuildSet up hardware and install OS.

InitializeConfigure for environment; often part of build.

UpdateInstall new software.

Patch old software.

Change configurations.

Automated Installs


Why Automate Installs?

1. Save time.Boot the computer, then go do something else.

2. Ensure consistency.No chance of entering wrong input during install.

Avoid user requests due to mistakes in config.

What works on one desktop, works on all.

3. Fast system recovery.Rebuild system with auto-install vs. slow tapes.


Trusting the Vendor Installation

Always reload the OS on new machines.– You need to configure the host for your env.– Eventually you’ll reload the OS on a desktop,

leaving you with two platforms to support: the vendor OS install and your OS install.

– Vendors change their OS images from time to time, so systems you bought today have a different OS from systems bought 6 months ago.


Install Types

1. Hard Disk ImagingDuplicate hard disk of installed system.Advantages: fast, simple.Disadvantages: need identical hardware, leads to

many images, all of which must be updated manually when you make a change

2. Scripted InstallsInstaller accepts input from script.Advantages: flexible, systems can be differentDisadvantages: more effort to setup initially


Auto-Install Features

1. UnattendedRequires little or no human interaction.

2. ConcurrentMultiple installs can be performed at once.

3. ScalableNew clients added easily.

4. FlexibleConfigurable to do custom install types.


Auto-Install Components

Boot ComponentMedia (floppy or CD)

Network (PXE)

Network ConfigurationDHCP: IP addresses, netmasks, DNS

Install ConfigurationMedia (floppy or CD)

Network (tftp, ftp, http, NFS)

Install Data and ProgramsNetwork (tftp, ftp, http, NFS)


PXE

Preboot eXecution EnvironmentIntel standard for booting over the network.PXE BIOS loads kernel over network.

ApplicationsDiskless clients (use NFS for root disk.)Booting install program.

How it works1. Asks DHCP server for config (ip, net, tftp.)2. Downloads pxelinux from tftp server.3. Boots pxelinux kernel.4. Kernel uses tftp’d filesystem image or NFS filesystem.


Disk Imaging

1. Setup ftp server.

2. Install OS image on a test client.

3. Verify test client OS.

4. Copy image to server.

5. Boot clients with imaging media.

6. Clients pull image from ftp server.

4. Copy image

1. ftp server

2-3. test client

5. deployment #1

5. deployment #2

6. Pull img

6. Pull img

Clonezilla


g4u


Scripted Install Tools

Red Hat distributions, incl. Centos– Kickstart– Cobbler

Debian distributions, incl. Ubuntu– FAI– Preseed

Mandriva Linux– DrakX

Solaris– Jumpstart



Network Configuration

What’s so bad about manual net settings?– It’s only an IP address and netmask.– What happens if you need to renumber?

Use DHCP instead of manual settings– Make all changes on a single server.– Easy to change settings for entire network.– DHCP can assign static IPs as well as dynamic.

Servers vs. Desktops


How are Servers different?

• 1000s of clients depend on server.• Requires high reliability.• Requires tighter security.• Often expected to last longer.• Investment amortized over many clients,

longer lifetime.


Vendor Product Lines

Home– Cheapest purchase price.– Components change regularly based on cost.

Business– Focuses on Total Cost of Ownership (TCO).– Slower hardware changes, longer lifetime.

Server– Lowest cost per performance metric (nfs, web)– Easy to service rack-mountable chassis.– Higher quality (MIL-SPEC) components.


Server Hardware• More internal space.• More CPU/Memory.

– More / high-end CPUs.– More / faster memory.

• High performance I/O.– PCIe vs PCI– SCSI/FC-AL vs. IDE

• Rack mounted.• Redundancy

– RAID– Hot-swap, hot-spares


Rack Mounting

Efficient space utilization. Simple, rectangular shape measured in RUs. Repair and upgrade while mounted in rack. No side access required.

Requirements Cooling through back, not sides. Drives in front, cables in back. Remote management (serial console, hardware

sensors, VM MUI)


Server Memory

Servers need more RAM than desktops.– x86 supports up to 64GB with PAE.– x86-64 supports 1 PB (1024 TB)

Servers need faster RAM than desktops.– Higher memory speeds.– Multiple DIMMs accessed in parallel.– Larger CPU caches.

http://en.wikipedia.org/wiki/File:Memory_module_DDRAM_20-03-2006.jpg

Server CPUs


Intel Xeon• Up to 8 cores with 2 threads each @ 1.8 to 3.3 GHz• Up to 18 MB L3 cache

AMD Opteron• 4, 6, 8, or 12 cores @ 1.4 to 3.2 GHz• Up to 12 MB L3 cache

IBM Power 7• 4, 6, or 8 cores with 4 threads each @ 3.0 to 4.25 GHz• 4 MB L3 cache per core (up to 32MB for 8-core)

Sun Niagara 3• 16 cores with 8 threads each @ 1.67 GHz• 6 MB L2 cache


Xeon vs Pentium/Core CPUs

Xeon based on Pentium/Core with changes that vary by model:

– Allows more CPUs – Has more cores– Better hyperthreading– Faster/larger CPU caches– Faster/larger RAM support

http://en.wikipedia.org/wiki/File:IntelCorei7Extreme.png


System Buses

Servers need high I/O throughput.– Fast peripherals: SCSI-3, Gigabit ethernet– Often use multiple and/or faster buses.

PCI– Desktop: 32-bit 33 MHz, 133 MB/s– Server: 64-bit 66 MHz, 533 MB/s

PCI-X (backward compatible)– v1.0: 64-bit 133 MHz, 1.06 GB/s– v2.0: 64-bit 533 MHz, 4.3 GB/s

PCI Express (PCIe)– Serial architecture, v3.0 up to 16 GB/s

http://en.wikipedia.org/wiki/File:PCIExpress.jpg


Hardware Redundancy

Disks are most likely component to fail.– Use RAID for disk redundancy.– Cover in detail in Disks lecture.

Power supplies second most likely to fail.– Use redundant power supplies.– Many servers need 2 power supplies

normally.– Need 3 power supplies for redundancy.– Use separate power cord and UPS for each

power supply.

http://en.wikipedia.org/wiki/File:PC-Netzteil_(redundant).jpg


Full and n+1 Redundancy

n+1 Redundancy: One component can fail, but the system is still functional.– Ex: RAID 5, dual NICs with failover

Full Redundancy: Two complete sets of hardware configured with failover mechanism.– Manual: SA switches to 2nd system when notices failure.– Automatic: The second system monitors the first and switches over

automatically on failure.– Load-sharing: Both systems serve users, sharing load, but each has

capacity to handle entire load on its own. When one fails, other automatically handles entire load.


Hot-swap Components

Hot-swap components– Components can be replaced while running.– Need n+1 redundancy for this to be useful.– Don’t need to schedule a downtime.

Issues– Which parts are hot-swappable?– May require a few seconds to reconfigure.– Be sure components are hot-swap, not hot-plug.

http://en.wikipedia.org/wiki/File:SPARCstation20_scsi_cradle_with_drive.jpg


Hot Plug and Hot Spare

Hot Plug– Electrically safe to replace component.– Part may not be recognized until next reboot.– Requires downtime, unlike hot swap.

Hot Spare– Spare component already plugged into system.– System automatically uses hot spare when disk/CPU

board etc. fails.– Provides n+2 redundancy.


Separate Administrative Network

Reliability– Allows access to machines even when network

is down.

Performance– Backups require so much bandwidth that they’re

often done over their own network.

Security– Network security monitoring data and logs sent

across network should be secured.


Maintenance Contracts• All machines eventually break.• Vendors offer variety of maint contracts.• Non-critical: Next-day or 2-day contract.• Clusters: If you have many similar hosts (CPU or web farm),

then on-site spares may be cheaper than maintenance contract.• Controlled Model: Use small # of machine types for all

servers, so you can afford a spares kit.• Critical Host: Same-day response or on-site spares.• Highly Critical: On-site technician + dup machine.


Data Protection• Avoid desktop backups by storing data on servers.

Easy on UNIX, harder on Windows.• Use RAID for server hardware failures.

– Mirror root disk, higher RAID levels for data.– Some servers use 16GB Flash drives for root disk.– Doesn’t protect against software mistakes.

• Server backups– Use specialized admin network to keep load off main

network.– Use specialized tape jukeboxes to fully automate

backups of large data servers (DBs, fileservers).


Keep Servers in Data Center

Data center necessary for server reliability.– Power (enough power, UPS)– Climate control (temperature, humidity)– Fire protection– High-speed network– Physical security

Server Operating Systems


http://en.wikipedia.org/wiki/File:IBM-AIX_logo20080906.png

http://en.wikipedia.org/wiki/File:Solaris_OS_logo.svg

http://en.wikipedia.org/wiki/File:Tux.svg

http://en.wikipedia.org/wiki/File:Openbsd2.svg


Server OS Image

Need greater reliability, security than desktop.– Remove unnecessary OS components.– Configure for best security & performance.

Install and config specialized server software.– Server software: web, db, nfs, dns, ldap, etc.– May need monitoring software too.– Configuration: disk space, networking

Server OS install should be automated too.


Remote Administration

Servers must be accessible remotely.– Allows SA to fix problems quickly at 3am.– Allows SA to work outside machine room.

Remote Administration– Serial console and concentrator (UNIX)– Networked KVM (Windows)– Remote power control.– Important to secure remote admin facilities.


Server Appliances

Dedicated hardware + software– Fileserver (NetApp, Auspex)– Print servers– Routers

Advantages– Performance– Reliability– Easy to setup– Extra capabilities

Disadvantages– Cost


Many Inexpensive Workstations

Why buy server hardware?– Buy two cheap rack-

mount PCs + failover software.

– Works if two PCs cheaper than server.

– Google’s approach with ~450,000 servers.


Blade Servers

• High-density servers on a board.– CPU– Memory– Disk

• Each blade lives in a blade chassis.


Blade Chassis• Blade chassis

provides power, network, remote.

• Typically hot-swappable, hot-spare.

• Racks can only support 1 svr/RU.

• Blades are higher density, but also require more power and cooling.

Services


Servers vs Services

A server is a piece of hardware.

A service is the function that is provided by one or more servers.

http://en.wikipedia.org/wiki/File:Inside_and_Rear_of_Webserver.jpg

http://en.wikipedia.org/wiki/File:WWW_logo_by_Robert_Cailliau.svg


Services• Distinguish structured computing environment

from some standalone PCs.• Large orgs linked through shared services to ease

communication and optimize resources.• Typical environments have many services

– Fundamental: net, DNS, email, auth, printing.– Typical: DHCP, backup, directory, file, license.

• Services often depend on other services– Almost everything depends on DNS.


Providing a Service

A service is more than hardware + software.

A service must be1. Reliable.

2. Scalable.

3. Monitored.

4. Maintained.

5. Supported.


Servers and Services

For a service to be reliable, servers should:– Be as simple as possible.– Have minimum software to run service.– Depend on as few other services as possible.– Depend only on services that are at least as

reliable as the service running on the server.– Have access restricted to SAs.– Be as few as needed for performance +

reliability.


Customer Requirements

Customers are the reason for the service.– How do they intend to use it?– What features do they need?– What features would they like to have?– How critical is the service?– What levels of availability and support are needed?

Service Level Agreement (SLA)– Enumerates services.– Defines level of support.– Commits to response times for problem types.


Operational Requirements

Essential to designing a reliable service– What services does it depend upon?– What other services will depend upon it?– How does it interoperate with other services?– How can it be integrated with auth/dir services?– How does the service scale?– How can the service be upgraded?

• Downtime requirements.• What systems are affected?


Open Architecture

Service should be built around open standards– Check IETF RFCs to see if it’s an open protocol.– Example service: SMTP– Example products: exim, postfix, qmail, sendmail.– Open standards don’t require open source.

Allows vendors to make interoperable products.– Avoids vendor lock-in.– Allows vendor competition (cheaper prices for you.)– Decouples client selection from server selection.– Avoids need for protocol gateways.


Requests for Comments (RFCs)

Documentation for Internet protocols, technologies, and methodologies.

– Standards track RFCs describe Internet standards (TCP, IP, SMTP) and must be approved by IETF.

– Experimental RFCs may become standards.– Best Common Practice RFCs describe how to run

services or use protocols.– Informational RFCs is a catch-all including

proprietary protocols, April Fool’s jokes, etc.

Available from http://www.rfc-editor.org/


Principles for Designing a Reliable Service

Simplicity– The more features, the more bugs.– Simplicity increases reliability, ease of

maintenance.

Vendor Relations– Can be helpful about configuring service.– Let vendors compete for your business.– Stick to vendors who develop for your platform.


Machine Independence

Will eventually move service to new host.– Want to avoid having a downtime.– Want to avoid reconfiguring every desktop.

Use generic DNS alias for machine – Mail server has name romero– DNS alias is smtp

Use virtual IP addresses for non-name svcs– Machine has usual IP address: 192.168.1.54– Virtual: ifconfig eth0:0 192.168.1.5


Dedicated Machines

Put each service on its own machine(s).– If a server crashes, only impacts one service.– Easier to debug if only one service running.– Performance tuning easier with one service.– If you can’t afford a new machine, use a VM.


Environment

Safe environment– Improves reliability: AC, UPS, physical security.– Data center usually provides faster network too.– Only rely on services provided by data center.

Restricted access– Customers should not need to login to servers.– More logins decrease stability, performance.– Even Windows can be stable w/o user logins.



Service components should be tightly coupled.– Other than redundant components.– Share same power source, network.– Reduces service dependencies (single points of

failure.)

Centralize management of service– Managed by one set of SAs.– Support for service by single helpdesk.– Document service.


Performance

Latency vs throughput– Latency is delay before data received.– Throughput is how much data sent per second.– Performance problems typically affects one.– Increasing the other will not solve your problem.

Remote sites– May have high latency to main site.– Do you need secondary servers at remote sites?


Capacity Planning

Estimate capacity from testing.– Test server at 100 qps, 200 qps, until slow.– Identify resources used by each query

• RAM• Disk• Network• CPU

Can service be split onto multiple servers?– Can it be done w/o users noticing?



Monitoring– Availability, problems, performance.– Auto-alert front line support.– Customers shouldn’t discover problems before SA.– Capacity planning: CPU, mem, disk, network, licenses.

Service Rollout– First impressions are difficult to change.– Be ready for support: docs, trained helpdesk.– Use one, some, many technique.


Key PointsDesktop Lifecycle: New, clean, configured, unknown states.

Automated Installs– Why: consistency, fast recovery, saves time.– Install types: imaging vs. scripted.– Components: boot, network, config, data.– Think about how Principles of SA apply.

Servers vs desktops– Requirements and hardware differences.

Redundancy– Full vs n+k redundancy.– Hot plug vs hot spare.

Services– Requirements: service, server, customer, operational.– Machine independence and open architectures.

Performance: Latency vs. throughput.


References1. Mark Burgess, Principles of System and Network Administration,

Wiley, 2000.2. Aeleen Frisch, Essential System Administration, 3rd edition, O’Reilly,

2002.3. R. Evard. "An analysis of unix system configuration." Proceedings of

the 11th Systems Administration conference (LISA), page 179, http://www.usenix.org/publications/library/proceedings/lisa97/full_papers/20.evard/20_html/main.html, 1997

4. Thomas Limoncelli, Christine Hogan, Strata Chalup, The Practice of System and Network Administration, 2nd ed, Limoncelli and Hogan, Addison-Wesley, 2007.

5. Evi Nemeth et al, UNIX System Administration Handbook, 3rd edition, Prentice Hall, 2001.

http://www.usenix.org/publications/library/proceedings/lisa97/full_papers/20.evard/20_html/main.html

http://www.usenix.org/publications/library/proceedings/lisa97/full_papers/20.evard/20_html/main.html

cit 470: advanced network and system administration

Documents