ray pasetes, andy romero css storage services cd/css central services activities coordination...

Ray Pasetes, Andy RomeroRay Pasetes, Andy Romero

CSS Storage ServicesCSS Storage Services

CD/CSSCD/CSSCentral Services Activities Coordination MeetingCentral Services Activities Coordination Meeting

9/19/20069/19/2006

September 2006September 2006 CSS Storage ServicesCSS Storage Services

AgendaAgenda

SAN/NAS servicesSAN/NAS services1.1. PrioritiesPriorities

2.2. SANSAN

3.3. NASNAS

4.4. Strategic DirectionStrategic Direction

Backups - TiBSBackups - TiBS5.5. SupportedSupported

6.6. How it worksHow it works

7.7. StatsStats


Storage Services - PrioritiesStorage Services - Priorities

1.1. ReliabilityReliabilityA.A. Data integrityData integrity

B.B. Service availabilityService availability

C.C. ManageabilityManageability

2.2. CostCost

3.3. PerformancePerformance


Storage ServicesStorage Services

Tier 1 RAID 5 Fibre-Channel

Tier 2 RAID 5/6 SATA

Fabric-A Fabric-B

AFSGlobal Access

<25MB/s

NFS/CIFSCampus Access~200-300MB/s

SCSILocal to FCC

>200MB/s per lun

Backups

IP

FCP

NDMPSAN


AgendaAgenda


2.2. SANSAN

3.3. NASNAS




7.7. StatsStats


Tiered Storage PoolTiered Storage Pool

• Redundant componentsRedundant components

• Tier 1 - RAID 5 Fibre Channel diskTier 1 - RAID 5 Fibre Channel disk– High I/O applicationsHigh I/O applications– Single disk fault tolerantSingle disk fault tolerant

• Tier 2 – RAID6 SATA disk (some Tier 2 – RAID6 SATA disk (some RAID5)RAID5)– Low to medium I/O applicationsLow to medium I/O applications– Double disk fault tolerantDouble disk fault tolerant


Direct SAN Attach - FCPDirect SAN Attach - FCP

• Block-based access (SCSI)Block-based access (SCSI)

• Must be in FCC (500m limit)Must be in FCC (500m limit)

• 2Gbps Fibre-channel network2Gbps Fibre-channel network

• Multi-attach for 24x7 serviceMulti-attach for 24x7 service

• Support for Linux, Solaris, IRIX, Support for Linux, Solaris, IRIX, WindowsWindows


AgendaAgenda


2.2. SANSAN

3.3. NASNAS




7.7. StatsStats


NAS Services - AFSNAS Services - AFS

• Global filesystemGlobal filesystem

• Throughput < 25MB/sThroughput < 25MB/s

• SecureSecure– Communication can be encryptedCommunication can be encrypted– Kerberos authenticationKerberos authentication– Directory ACLsDirectory ACLs

• Limited to 8GB volumesLimited to 8GB volumes


NAS Services – NFS/CIFSNAS Services – NFS/CIFS

• Campus-wide access onlyCampus-wide access only– CIFS off-site via VPN possibleCIFS off-site via VPN possible

• Throughput Throughput – ~200MB/s write~200MB/s write– ~300MB/s read~300MB/s read

• HA clusterHA cluster

• Many virtualization servicesMany virtualization services


NAS Service OverviewNAS Service Overview

• 2 Titan-21 NAS Gateways (Physical)2 Titan-21 NAS Gateways (Physical)• 2 Enterprise Virtual Servers (EVS)2 Enterprise Virtual Servers (EVS)

– Each EVS has its own IP addressEach EVS has its own IP address– Each EVS provides NFS, CIFS and FTP servicesEach EVS provides NFS, CIFS and FTP services– An EVS can run on either Titan (HA Failover An EVS can run on either Titan (HA Failover

Capability)Capability)

• All storage provided by SANAll storage provided by SAN• 4 Gb/s connection to 256 Gb/s LAN core4 Gb/s connection to 256 Gb/s LAN core

– high performance for all customershigh performance for all customers


NAS OverviewNAS Overview

• Storage PoolsStorage Pools– Composed of LUNS (units of physical storage Composed of LUNS (units of physical storage

provided by SAN)provided by SAN)

• File SystemsFile Systems– Multiple file systems reside in a poolMultiple file systems reside in a pool– Different block sizes (large or small files)Different block sizes (large or small files)

• Virtual Volumes (ViVols)Virtual Volumes (ViVols)– Multiple ViVols reside in a poolMultiple ViVols reside in a pool– Key to thin provisioningKey to thin provisioning


NAS Virtualization ServicesNAS Virtualization Services

• Zero downtime capacity upgradesZero downtime capacity upgrades

• Thin ProvisioningThin Provisioning

• SnapshotsSnapshots

• Policy Based Data MigrationPolicy Based Data Migration


Capacity UpgradesCapacity Upgrades

• DAS Capacity UpgradeDAS Capacity Upgrade– Move server to a new rack to fit new Move server to a new rack to fit new

chassischassis– Install new RAID channels to support new Install new RAID channels to support new

RAID chassisRAID chassis– Install new RAID chassisInstall new RAID chassis– Create new file systemsCreate new file systems– Migrate / Rebalance dataMigrate / Rebalance data– Downtime (Lots !!)Downtime (Lots !!)


Capacity UpgradesCapacity Upgrades

• NAS Capacity UpgradeNAS Capacity Upgrade– Increase size of Vivol (best case)Increase size of Vivol (best case)– Add LUNs to storage pool (worst case)Add LUNs to storage pool (worst case)– No need to move serverNo need to move server– No need to install new RAID channelsNo need to install new RAID channels– No need to create new filesystemsNo need to create new filesystems– No need to migrate and rebalance dataNo need to migrate and rebalance data– Downtime (Zero !!)Downtime (Zero !!)


Storage ProvisioningStorage Provisioning

• DAS Style Storage ProvisioningDAS Style Storage Provisioning– Each Project/Dept has its own serverEach Project/Dept has its own server– Each server has its own disksEach server has its own disks– Each server has its own free space poolEach server has its own free space pool– Storage provisioning predictions are often wrongStorage provisioning predictions are often wrong

• Some projects, plenty of available excessSome projects, plenty of available excess(sum of total free space across all servers TB+)(sum of total free space across all servers TB+)

• Others, heavily used…starved for spaceOthers, heavily used…starved for space

• New project = new disk purchaseNew project = new disk purchase

– Sorry …. Sharing not possibleSorry …. Sharing not possible


Storage ProvisioningStorage Provisioning

• NAS/SAN Style Storage Provisioning (Thin)NAS/SAN Style Storage Provisioning (Thin)– All Projects/Services share the sameAll Projects/Services share the same

SAN connected storageSAN connected storage– ViVol parameters can be tuned to over ViVol parameters can be tuned to over

provision physical storageprovision physical storage– New Project, no problemNew Project, no problem

• Get started nowGet started now

• Defer storage purchase until laterDefer storage purchase until later

– Easier to defer and lump storage purchasesEasier to defer and lump storage purchases• Better PricesBetter Prices


Snap ShotsSnap Shots

…\Live-Files…\.snapshot


SNAP ShotsSNAP Shots

• Why snapshots are coolWhy snapshots are cool– User’s can recover their own data User’s can recover their own data

immediatelyimmediately– No need to submit helpdesk ticketNo need to submit helpdesk ticket


Policy Based Data Migration Policy Based Data Migration (PDM)(PDM)

• Ability to automatically migrate data Ability to automatically migrate data from expensive tier 1 to tier 2 diskfrom expensive tier 1 to tier 2 disk

• Does not interrupt usersDoes not interrupt users• Various Migration CriteriaVarious Migration Criteria

– AgeAge– SizeSize– Type (avi, jpg, tiff …..)Type (avi, jpg, tiff …..)

• Various Analysis SchedulesVarious Analysis Schedules


NAS StatusNAS Status

• WindowsWindows– Backups, Printer MigrationsBackups, Printer Migrations– SoonSoon

• CSS Home directoriesCSS Home directories

• PSEEKITS soonPSEEKITS soon

• VMSSTREAMER (data only)VMSSTREAMER (data only)

• CDSERVER (printers first)CDSERVER (printers first)

• PPDSERVER (printers first)PPDSERVER (printers first)

• VMSSERVER1 (printers, www services first)VMSSERVER1 (printers, www services first)

• Others …Others …


NAS StatusNAS Status

• NFSNFS– CMS, OSG, Scientific LinuxCMS, OSG, Scientific Linux– Many customers, Some unexpected growth (…Many customers, Some unexpected growth (…

new NAS on order)new NAS on order)

• Can I use the NAS for my projectCan I use the NAS for my project– YES … It’s a site serviceYES … It’s a site service– CostsCosts

• Small projects …. NoneSmall projects …. None• Large projects …. share portion of disk, tape costs Large projects …. share portion of disk, tape costs

(no management costs)(no management costs)


AgendaAgenda


2.2. SANSAN

3.3. NASNAS




7.7. StatsStats


Virtualization

Storage Pool

Connection

Access

Strategic DirectionStrategic Direction

RAID Storage

Automated Tiering

Thin Provisioning

TCP/IP FCP

Snapshots

NASFile Protocol

NFS/CIFS/FTP~200MB/s

Campus wide

SANBlock Protocol

SCSI-3>200MB/sInside FCC

Thin Provisioning

Automated Tiering

Snapshots

FCP or TCP/IP

Dynamic Resize Dynamic Tuning


Additional InformationAdditional Information

http://computing.fnal.gov/nasanhttp://computing.fnal.gov/nasan

Questions?Questions?

http://computing.fnal.gov/nasan


AgendaAgenda


2.2. SANSAN

3.3. NASNAS




7.7. StatsStats


BackupsBackups

TiBS – True Incremental Backup SystemTiBS – True Incremental Backup System

• Only 1 full network backupOnly 1 full network backup

• All other backups are incrementalAll other backups are incremental– Lightens client workloadLightens client workload

• Use previous backups to create new Use previous backups to create new fullsfulls

• No “per-client” connect costsNo “per-client” connect costs


AgendaAgenda


2.2. SANSAN

3.3. NASNAS




7.7. StatsStats


TiBS – Supported SystemsTiBS – Supported Systems

• ServersServers– IRIX, Linux, Mac OSX, Solaris, WindowsIRIX, Linux, Mac OSX, Solaris, Windows

•Client code must be installedClient code must be installed

– NAS appliances via NFSNAS appliances via NFS


TiBS – UnsupportedTiBS – Unsupported

• Sparse Files (databases)Sparse Files (databases)– Needs to be dumped to flat fileNeeds to be dumped to flat file

• Compressed files as filesystemCompressed files as filesystem

• Swap filesSwap files

• DesktopsDesktops

• Raw devices/block-level backupsRaw devices/block-level backups


AgendaAgenda


2.2. SANSAN

3.3. NASNAS




7.7. StatsStats


TiBS – How it worksTiBS – How it works

Tue

FullFullMonM-T Merge

MonWedM-T-W Merge

Previous FullPrevious Full

New Full Merge

Disk Cache


TiBS – Tape RotationTiBS – Tape Rotation

• Full - 84 days (mirrored)Full - 84 days (mirrored)– Retained 1 yearRetained 1 year

• Monthly – 28 days (mirrored)Monthly – 28 days (mirrored)– Retained 1 yearRetained 1 year

• Weekly – 7 daysWeekly – 7 days– Retained 14 weeksRetained 14 weeks

• Incremental – DailyIncremental – Daily– Retained 56 days**Retained 56 days**


TiBS - RecoverabilityTiBS - Recoverability

• 1 Full tape lost1 Full tape lost– Use other mirror to recoverUse other mirror to recover

• Both Full tapes lostBoth Full tapes lost– Use current monthly and previous full to recreate Use current monthly and previous full to recreate

fullfull

• Both Full tapes and both month tapes lostBoth Full tapes and both month tapes lost– Use current weekly and previous month to Use current weekly and previous month to

recreate new current monthrecreate new current month– Use new current month with previous full to Use new current month with previous full to

recreate new current fullrecreate new current full


TiBS – Actual size of dataTiBS – Actual size of data

• Multiple copies of data on tapeMultiple copies of data on tape– MirroringMirroring– Retention policiesRetention policies– For one year, size ratio is 14:1 For one year, size ratio is 14:1 Tape vs. disk for static data. More for dynamic data.Tape vs. disk for static data. More for dynamic data.


AgendaAgenda


2.2. SANSAN

3.3. NASNAS




7.7. StatsStats


Stats and FactsStats and Facts

• 415 servers being backed up415 servers being backed up– BSS, CMS, CSS, DCG, DIR, DSG, ESE, ESH, EPP, BSS, CMS, CSS, DCG, DIR, DSG, ESE, ESH, EPP,

FESS, MIPP, QCD, REXFESS, MIPP, QCD, REX

• 6770 AFS volumes backed up6770 AFS volumes backed up• ~15 TB of data == 1 full backup~15 TB of data == 1 full backup• Clients can recover from bare metal*Clients can recover from bare metal*• Limited availability for new customersLimited availability for new customers

– Tape resourcesTape resources


More InformationMore Information

http://computing.fnal.gov/site-backupshttp://computing.fnal.gov/site-backups

Questions?Questions?

http://computing.fnal.gov/site-backups

ray pasetes, andy romero css storage services cd/css central services activities coordination...

Documents