insights into tsm/hsm for unix and windowstsm-symposium.oucs.ox.ac.uk/2005/papers/insights into...

IBM Software Group

®

© 2005 IBM Corporation1

Insights into TSM/HSM for UNIX and Windows

Jens-Peter Akelbein ([email protected])IBM Tivoli Storage SW Development

Oxford University TSM Symposium 2005

IBM Software Group | Tivoli software

© 2005 IBM Corporation2 Oxford TSM Symposium 2005 | Insights into TSM/HSM for UNIX and Windows | [email protected]

IBM TotalStorageTaking steps toward an On Demand storage environment

OrchestrationOrchestration

StorageInfrastructureManagement

StorageStorageInfrastructureInfrastructureManagementManagement

HierarchicalStorage

Management

HierarchicalHierarchicalStorageStorage

ManagementManagement

ArchiveManagement

ArchiveArchiveManagementManagement

RecoveryManagement

RecoveryRecoveryManagementManagement

Storage VirtualizationStorage Virtualization

Advanced Copy ServicesAdvanced Copy Services

Hardware InfrastructureHardware InfrastructureTapeTapeDiskDisk Storage NetworkingStorage Networking



Managing too much data on UNIX, Linux, and Windows

The Tivoli Storage Manager product family provides now two HSM productsIBM Tivoli Storage Manager for Space Management

• AIX (JFS, JFS2, GPFS), Linux (GPFS), Solaris (VxFS) HP/UX (JFS)

IBM Tivoli Storage Manager HSM for Windows• Windows 2000 Server, Windows 2000 Advanced Server, Windows 2003 Server,

Windows 2003 Enterprise Server (32 Bit), all NTFS

This session intends to provide some answers for questions like…What differences exist between using HSM on UNIX and Windows?

What technologies and interfaces are used for HSM on UNIX and on Windows?

How did HSM on UNIX evolve over the past years?

… and your questions!



Definition of HSM (the TSM-way)

HSM automatically migrates files from on-line to near-line storage (typically from disk to tape), based on policy settings

Small stub files are retained on disk, appearing as the original files, thus ensuring transparency of HSM for user applications

Stub files contain information pointing to corresponding entry in TSM server DB

Migrated files are recalled automatically back to disk as required

Note: The TSM server has its own type of HSM for migration of files in its storage pool hierarchy.



Usage patterns of File Servers

UNIX File Server

Applications like Engineering, Geophysical, Multimedia, Scientific data, Bioinformatics, Medical imaging

Type of data varies and might not be present in the file name

Maximum size of file up to hundreds of Gigabyte or even Terabyte

Command line usage is common

Windows File Server

Applications like office tools (text, spreadsheets, presentations), Multimedia

Type of data often known by suffix

Maximum size of files up to a few hundred Megabyte or a Gigabyte

GUI usage is common

Please see this more as a pattern than a rule



Migration policy on UNIX

Primary selection criteria are age and/or size where weight factors allow to set

priorities

Exclude small files from becoming eligible at all (minmigfilesize)

Include/exclude statements allow directory and file selection

Set a high threshold to define the maximum allocated file system capacity

Set a pre-migration percentage to free up online storage space faster during

automigration



UNIX



Migration policy on Windows

Configure one or more migration jobsA migration job consists of a path nameInclude/exclude filters can be added for

known file groups or specify file name extensions

File size and age

Jobs can be executed periodically by starting them from the scheduler

…and do all this easily in a GUI !



Configuring a Migration Job

Step 1- Select the root directoryfor the migration job

System administrationtask

Part of the System Administration Client

Fast and efficient in a GUI



Configuring a Migration Job – cont’d

Step 2- Include subdirectories

Step 3- Select migration mode




Step 4- Include/exclude file groups




Step 5-Include/exclude files based on date condition- Absolute or relative- Creation, modification or last access date



Windows



Migration policies: UNIX and Windows

UNIX HSM

Capacity oriented approach(classic HSM)Automigration migrates data when high threshold is reachedFixed amount of data being transferred by a single automigrationAvoids out-of-space conditionPlacement of individual filesdepends on used capacity

Windows HSM

File attribute oriented approach (more like ILM)Placement of individual files is specified exactlyAllows to define jobs for different file paths (often used for e.g. departments/users)Jobs specified in a XML formatScheduled jobs ensure replacing recalled filesDoes not avoid out-of-space condition



Interfaces to the TSM server

TSMServer

File System

File U

TSM HSMClient

UNIX File Server

File System

File W

TSM HSMClient

Windows File Server

TSM APIlibrary

HSMVerbs

TSM Server

Storage Pool(SPACEMGMT)

Storage Pool(Archive)

Unique Object ID

Stub U

File U

differentinterfaces

used internally

Pathname Qualifier(FS/HL/LL)

Stub W

File W

Recommended toset archive retention

forever to keepdata integrity!



Backup integration (backup)

TSMServer

File System TSM HSMClient

UNIX File Server

File System

Windows File Server



TSM BackupClient*

TSM HSMClient

TSM BackupClient

Storage Pool(Backup)

Stub U

File U

File U

File W

Stub W

Stub W

Inline copyknow the link!

(no network traffic)

TSM Server



Backup integration (restore)

TSMServer


UNIX File Server

File System

Windows File Server



TSM BackupClient*

TSM HSMClient

TSM BackupClient

Storage Pool(Backup)File W

Stub W

TSM Server

Stub W

restore

File U

File U

Stub U restore(–migstate=yes)

Recreating uniqueobject ID



Backup integration (versions)

TSMServer


UNIX File Server

File System

Windows File Server



TSM BackupClient*

TSM HSMClient

TSM BackupClient

Storage Pool(Backup)File W

Stub U

File U

File U

Stub W

Stub W

TSM Server

Stub U*

File U*

File U*

copygroup(verexists=2)

Stub W*

File W*

Stub W*

Retains olderversions

U*/W*file U/W contains

now modifiedcontents andgets migrated

again



Backup integration: UNIX and Windows

UNIX File Server

Restore of stubs and complete files through the same interfaceNumber of retained versions controlled by VEREXISTS optionInline copy can avoid network traffic

Windows file Server

Restore of stubs through the B/A clientRestore of migrated files completely through HSM client

Both ways of backup integration allow restoring stubs for faster restore of a file systemOlder versions of file contents are kept on the TSM server

Both products are using one transaction for each file. Fits for medium and large files!

What is a small file? No easy answer, but…LTO tape drives can write not less than a certain amount of data (LTO2: 400KB, LTO3 1,6MB).A single TSM transaction wastes tape space if it does not contain at least such an amount.



Technology used for transparent recall

UNIX HSM (DMAPI)

DMAPI is an open standard defined 1997 by The Open Group; official name: “Data Storage Management (XDSM) API”Goal: Independence of data management applications from underlying file system typesAvoidance of kernel code by applications using DMAPIDMAPI in general provided by file system implementationExamples of file systems supporting DMAPI: XFS for SGI IRIX and Linux, GPFS for IBM AIX and Linux, VERITAS VxFS for Sun Solaris, JFS for HP-UX, JFS2 for AIXStub related to object on TSM server by a unique object ID(renaming the file does not break the pointer)

Windows HSM (Reparse points)

• Using reparse points for intercepting file system calls

• File system filter driver resides in the Windows Kernel; notifies recall application

• Recall application resides in User Space; runs as Windows Service

• Stub related to object on the TSM server by a FS/HL/LL specifier stored in the stub(renaming the file does not break the pointer; but old file name must be used for manual restore)



UNIX HSM scalability3.7

MAXCANDIDATESduration of candidates search

Reconcile and Automigoverlapped

MAXMIGRATORSConcurrent auto-migration

Reconcile and Automigfull concurrency

Candidate searchas own process (scout)

Reconcilew/o orphan check

4.1 4.2 5.1 5.2 5.3

Pro

cess

para

llism

I/Oth

roug

hput

Num

ber

of fi

les

MAXCANDPROCS Daemon controlling parallel FS scans

CANDIDATESINTERVALDaemon controlling parallel FS scans

MINMIGFILESIZE

Streaming Mode

Partial File Recall*

LAN-Free*

*for AIX GPFS; **for Linux GPFS

LAN-Free**Partial File Recall**

Distributed Recallfor GPFS clusters

dsmlsmemory consumptation



TSM/HSM availability and usability5.1 and earlier 5.2 5.3

Usa

bilit

yP

latfo

rmsu

ppor

t Solaris VxFS(3.1)

AIX JFSinitial client

AIX GPFSfor SP/2 (4.1)

AIX GPFSfor pSeries

HP/UX JFSfor PA-RISC

Linux GPFSFor xSeries

AIX JFS2for pSeries

GUI Admin Console*

*not available for all clients

GUI Admin Consolepreview

Non-root user supportfor command line Selective premigration

for command line

No single point of failure during failoverfor GPFS clients

Easy installation on clustersfor GPFS clients

HSM for Windows



References

TSM HSM for UNIX on the Web:http://www-3.ibm.com/software/tivoli/products/storage-mgr-space/

TSM Manuals for UNIX and Windows on the Webhttp://publib.boulder.ibm.com/infocenter/tivihelp/v1r1/index.jsp

Tivoli Field Guide on HSM (from January 2003):http://www-1.ibm.com/support/entdocview.wss?rs=0&uid=swg27002498

DMAPI specification:http://www.opengroup.org/onlinepubs/9657099/

insights into tsm/hsm for unix and windowstsm-symposium.oucs.ox.ac.uk/2005/papers/insights into...

Documents