advanced data sharing technologies

EDUCATION

Advanced Data SharingTechnologies Part 1version 9

Philippe Nicolas, SymantecJonathan Goldick, ONStor

EDUCATION

SNIA Advanced Data Sharing Technologies Tutorial (v9) - Storage Networking World USA - April 3-6th, 2006 2005-2006 Storage Networking Industry Association. All Rights Reserved.

2

Abstract

Advanced Data Sharing Technologies Part 1The What, Why and How of Data Sharing technologies plus block and file-based approachesfor IT Director and Managers, IT/Storage/System Eng., Admins, Architects and TrainersHow to deliver more performance and data accessibility with little to no additional cost? How to take advantage of existing storage infrastructure to provide more value to end-users and the global enterprise? A clear industry direction indicates that Data Sharing architectures and technology can be a good way to achieve these objectives.The first session offers a definition of Data Sharing and a discussion of its benefits with examples linked to the SNIA Shared Storage Model. We cover the main data sharing approaches and describe how they can improve performance, data accessibility, and manageability. This includes Scale-in and Scale-out methods based on block, file and application technologies such as: Cluster Volume Managers, SAN File Systems, Cluster File Systems, Parallel NFS (pNFS), Object-based Storage Devices (OSD) and Global/Parallel File System.

EDUCATION


3

SNIA Legal Notice

The material contained in this tutorial is copyrighted by the SNIA

Member companies and individuals may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced without modification The SNIA must be acknowledged as source of any material used

in the body of any document containing material from these presentations

This presentation is a project of the SNIA Education Committee

EDUCATION


4

Agenda

What is Data Sharing ? Definition

Why Data Sharing ? End User Benefits

How is Data shared ? Block and file-based approaches

EDUCATION

What is Data Sharing ?Why Data Sharing ?

EDUCATION


6

What is Data Sharing ?Definition

Shared access tosame data (value & location) by multiple systems Read/write: changes to data become visible to all servers Read-only access via mechanisms that support shared read/write

access Examples

Read/write access to a shared file is data sharing So is read-only access to a shared file

Clone/Snapshots (read-only or read/write) are not data sharing Changes do not affect original data

Caching is data sharing when changes propagate Changes to cached data must become visible to all Other data changes must become visible via cache

Replication/CDP* is not data sharing because location changes Potential divergence of data

* CDP: Continuous Data Protection

EDUCATION


7

Why Data Sharing ?End User Benefits

Better performance and scalability Larger server can be expen$$ive

Sharing: apply more servers to same problem Scales well for some applications, poorly for others Can avoid replication or cloning

Concurrency and Content access distribution Use same data for more than one application

Administration Consolidated shared resource has lower TCO Data Sharing increases the benefits of Storage/Data

Consolidation

EDUCATION


8

Storage ConsolidationScale-up by Scale-In

FileServer

SharedDisks

Scale-In

StorageNetwork

Data Network - LANNFS/CIFSServer

EDUCATION


9

Performance ImprovementScale-up by Scale-Out

ApplicationServer

DB Engine

Application

Shared Disks

Scale-outStorageNetwork

Shared Storage Software

Cluster Software

EDUCATION


10

Why Data Sharing ?How to apply Data Sharing to do useful things

High Availability Clusters (local & geographic) Scaling applications

Web Servers - Read mostly/load balanced Databases/OLTP/DW - Mostly use direct I/O

Parallel applications and fast failover Systems and Applications Consolidation/Migration Off-host processing

Based on shared file system Can also use by Point-in-Time copy techniques (not related to

our data sharing definition) address both Performance and Availability with no

administration degradation and overhead

EDUCATION


11

Some technologiesand products

IBM AFSIBM AFS

Oracle OPS/RAC

VERITAS CFSVERITAS CFS

PolyServe Matrix Server

HP HP TruClusterTruCluster/CFS/CFS ADIC ADIC StorNextStorNext FSFSNFSNFS

CIFSCIFS

SambaSamba

IBM SANergyIBM SANergy

Sybase MPP

SGI CXFS

Informix XPS

RedhatRedhat GFSGFS

EMC HighRoadEMC HighRoad

RFSRFS

DFSDFS

CodaCoda

Sun QFSSun QFS

PPFSPPFS

DB2DB2

SMBSMB

ISO9660ISO9660

WebNFSTacit Networks Tacit Networks IIsharedshared

Cisco Cisco FileEngineFileEngineDiskSites FilePort

NuviewNuview StorageXStorageX

SUN SAM-FS

Sanbolic Melio FS

WebFSWebFSIBM Storage TankIBM Storage Tank

OpenAFSOpenAFS

IsilonIsilon IQ IQ OneFSOneFS

IBRIX FusionFSONStor STORONStor STOR--FSFSWAFSWAFS

PVFS

OSD

WebNFS

pNFSpNFS

Apple Xsan

LUSTRE

Distributed, Distributed, Cluster orCluster or

SAN File SystemSAN File System

Parallel &Parallel &PartitionedPartitioned

ApplicationsApplications VolumeVolume

FineGround

EDUCATION

How is Data Shared ?

EDUCATION


13

How is data shared ?Approaches and methods

Several levels are possible Share at the volume level (block based) Share at the file or file system level (file, block or object* based) Share at the database or application level (custom)

In all cases, all these methods could occur among like or dissimilar systems (OS), concurrently or serially, directly at the storage or in the network

* For OSD

EDUCATION


14

How is data shared ?Approaches and methods

Traditional/Historical Block level: Volume Management File/File System (FS) level: Local FS (serial data sharing) and

distributed methods with NAS, Samba, AFP (AppleShare), DFS, AFS/OpenAFS, RFS, Coda

App./DB level: custom built-in methods (RDBMS, Email systems)

Advanced/Recent - File/FS level Distributed: WAFS approach (NAS extension)

and Network File Management/Virtualization(NFM), Global FS, SANFS and Cluster FS

Check outSNIA Tutorial:

NAS & iSCSI


NAS & iSCSI


StorageVirtualization



EDUCATION


15

How is data shared ?The SNIA Shared Storage Model

File/record layerFile/record layer

Database(dbms)

File system(FS)

Stor

age

dom

ain

Block layerBlock layer

Storage devices (disks, )Storage devices (disks, )

S

e

r

v

i

c

e

s

S

e

r

v

i

c

e

s

D

i

s

c

o

v

e

r

y

,

m

o

n

i

t

o

r

i

n

g

D

i

s

c

o

v

e

r

y

,

m

o

n

i

t

o

r

i

n

g

R

e

s

o

u

r

c

e

m

g

m

t

,

c

o

n

f

i

g

u

r

a

t

i

o

n

R

e

s

o

u

r

c

e

m

g

m

t

,

c

o

n

f

i

g

u

r

a

t

i

o

n

S

e

c

u

r

i

t

y

,

b

i

l

l

i

n

g

S

e

c

u

r

i

t

y

,

b

i

l

l

i

n

g

R

e

d

u

n

d

a

n

c

y

m

g

m

t

(

b

a

c

k

u

p

,

)

R

e

d

u

n

d

a

n

c

y

m

g

m

t

(

b

a

c

k

u

p

,

)

H

i

g

h

a

v

a

i

l

a

b

i

l

i

t

y

(

f

a

i

l

-

o

v

e

r

,

)

H

i

g

h

a

v

a

i

l

a

b

i

l

i

t

y

(

f

a

i

l

-

o

v

e

r

,

)

C

a

p

a

c

i

t

y

p

l

a

n

n

i

n

g

C

a

p

a

c

i

t

y

p

l

a

n

n

i

n

g

Network

Host

DeviceBlock aggregation

Cluster FS

SharedLVM

ApplicationApplicationlevel

SAN FS

NAS

WAFSGFS

EDUCATION


16

How is data shared ?Volume level

Examples EMC PowerPath Volume Manager (PPVM) HP Shared Logical Volume Manager (SLVM) IBM Logical Volume Manager (LVM) MACROIMPACT SANique Cluster Volume Manager (CVM) REDHAT Logical Volume Manager (LVM) SANBOLIC LaScala VERITAS* [Cluster] Volume Manager (CVM/VxVM)

Data Layout and

Organization

DataPath

LVM LVM

* Merged with Symantec, July 2005

EDUCATION


17


Volume Managers allow data to be shared at a low level (block) without usually a built-in locking mechanism Higher level applications control concurrent accesses to the data as

needed Can combine or divide physical resources (e.g., concatenation,

mirroring, striping) and share the result Volume Managers and the VTOC* problem

Every OS has its own VTOC format Every VM has its own Volume Header and Table Definition

Same VM everywhere and you can share raw volume or same FS Byte orders between processor

Big Endian: Sparc, PA-Risc, Power Little Endian: Intel) Block size on the device and block boundary

could cause troubles Concurrent or Serial access





* Volume Table Of Contents

EDUCATION


18


In-Band Virtualization Out-of-Band Virtualization

ApplicationServers

SharedDisks

StorageNetwork

ApplicationServers

SharedDisks

StorageNetwork

Appliance

Volumecreation

Volumeallocation

Intelligent switch and/or

Appliance

EDUCATION


19


ApplicationServers

StorageNetwork

Shared Volume Manager - Storage Software

HPC App. HPC App. HPC App. HPC App.

Shared Disks

Example: HPC* Application- How ?

Own data format on disk Own lock mechanism

- Benefits Increased throughput More effective use of servers

* High Performance Computing

EDUCATION


20

How is data shared ?File/File System level

Share at File or File System (FS) level

Multiple approaches & levels of maturity Block-based

Local (physical) Disk File System* for serial data sharing Disk based Cluster File System SAN File System or SAN File Sharing System

File-based or Distributed File System NFS/CIFS, WAFS and NFM** Global, Parallel and Distributed FS

* like UFS, HFS, XFS, JFS, VxFS, NTFS, ext2/3** Network File Management (also Network File Virtualization)

EDUCATION


21

File/record layer

Device block-aggregation

Network block-aggregation

Host block-aggregation

H

o

s

t

.

w

i

t

h

L

V

M

a

n

d

s

o

f

t

w

a

r

e

R

A

I

D

1. Direct-attach

NASserver

4. NAS serverHost3. NAS head

NAS head

Host

LANH

o

s

t

.

w

i

t

h

L

V

M

Disk array

SN

2. SN-attachApplication

Cluster FSShared

LVM

Applicationlevel

SAN FS

Cluster FS

Dist. FS

F

i

l

e

/

r

e

c

o

r

d

l

a

y

e

r

B

l

o

c

k

l

a

y

e

r

EDUCATION


22

How is data shared ?File/File System level Local Disk File System

Serial File System sharing on same or dissimilar OS via commonVolume Manager & Physical File System

Good for sequential (not concurrent) data processing and data migration

If OS is different Same Volume Manager avoids VTOC incompatibilities Byte order differences may require meta-data conversion

Intel is Little-endian, most others are Big Endian Examples

Homogeneous OS (common case) Most file systems (and volume managers) support this UFS, HFS, XFS, JFS, VxFS, ext2/3 SDS/SVM, LVM, XVM,

VxVM Sanbolic Kayo, DNF Dynamic Share Heterogeneous OS (need common volume manager)

VERITAS* Storage Foundation with Portable Data Container feature

* Merged with Symantec, July 2005

EDUCATION


23

Import Disk GroupStart VolumeMount File System

How is data shared ?File/File System level Local Disk File System

OS #0

SharedDisks

StorageNetwork

OS #1 OS #2 OS #3


Deport Disk GroupStop VolumeUmount File System

Example: DW* Application

- How ? OS #0 server stores data OS #1 server starts

batches OS #2 server loads data

into the DW OS #3 server backups

data- Benefits

No data multiplication Cost effective for Storage More effective use of

servers No time wasted in

copying data between servers






* Data Warehouse

EDUCATION


24

How is data shared ?File/File System level Cluster File System

Cluster File System (also called Shared Data Cluster) A Cluster FS allow a FS and files to be shared All nodes understand Physical (on-disk) FS structure The File System is mounted by all the nodes Single FS Image

Same data view from all nodes Examples

HP CFS (TruCluster) HP/Cal. Soft. Monster FS IBM GPFS MACROIMPACT SANique CFS POLYSERVE Matrix Server

REDHAT GFS1

SANBOLIC MelioFS VERITAS2 CFS

1 Sistina acquired by RedHat2 Merged with Symantec, July 2005

EDUCATION


25

How is data shared ?File/File System level Cluster FS

SecondHost

Shared Disks

HeartBeatLock Management

StorageNetwork

Cluster File System

Cluster Volume Manager

WebServer

WebServer

WebServer

FirstHost

Cluster

Example: Web Servers Farm How ?

Shared VM/FS Load Balancer in front

Benefits Increased throughput More effective use of servers Failure is transparent SSI/SFSI, High SLAs

Optional Layer

EDUCATION


26

How is data shared ?File/File System level Cluster File System

Asymmetric & Symmetric Implementation Asymmetric uses master node for logging and locking

Lock Mechanism Distributed or Global Lock Management (DLM/GLM)

Different implementation strategies Granularity varies: file, record, byte

Cache Coherence Single File System Image Every modification is seen by all nodes as soon as a

modification in the data sharing domain occurs

Usage Consideration: Concurrent vs Serial data access Concurrent: multiple systems access the data simultaneously Serial: one system at a time uses and access the data

EDUCATION


27

How is data shared ?Advanced methods File/FS Approaches

SAN File System 1 node (Master) or a set of masters

Understand, manage and use metadata on disk Use of file system even if portions of it are inaccessible block addresses distributed to nodes (clients) on request

Other nodes (clients) connection to SAN storage Avoid overhead due to Metadata management access to data directly using blocks addresses sent by Master(s)

Designed to support hundreds or thousands of nodes Mixed role between direct data access with

host based thin software and NAS access

EDUCATION


28

How is data shared ?File/File System level SAN File System

Flexibility of network FS at SAN speed Long-term goal for the industry development for Capacity and

Performance scaling Scaling hundreds of PetaBytes of capacity and tens of GigaBytes/sec

More recent File Server Generation Examples

APPLE Xsan ADIC StorNext FS DataPlow SAN FS

& Nasan FS EMC Celerra HighRoad,

MPFS/MPFSi

HP DirectNFS xNFS (Cal. Soft.) Transoft Fibrenet

IBM TotalStorage SAN FS, SANergy

IBRIX FusionFS SGI CXFS SUN QFS

EDUCATION


29

How is data shared ?File/File System level SAN FS

Data Network - LANFileServer

Example: Multimedia Application How ?

1 big server with NFS/CIFS layer Server and Client SAN FS layer Hundreds of clients

Benefits Increased throughput Consolidate storage, very scalable More effective use of resources

NFS/CIFSServer

StorageNetwork

MetadataServer

Client sw Client sw

File Request

Block list

Data Access

Data and Control Access

App.

App. App.

SharedDisks

EDUCATION


30

How is data shared ?File/File System level SAN FS

How it works ? Asymmetric or Client/Server model Server controls client access, resolves conflicts Thin client software layer handles SAN device and server

interaction

Lock Mechanism Provided by the server at a central location Various granularity: file, record, byte Some implementations use SMB or NFS semantics The server needs to be protected cause it represents a SPOF

Cache Coherency Some implementations deliver cache coherency with traditional

validate/invalidate mechanism, others dont offer cache at all

EDUCATION


31

How is data shared ?File/File System level OSD

OSD = Object-based Storage Device An object comprises

Application data (e.g., file, record) Device-managed metadata (e.g., block allocation) User-accessible attributes (e.g., access times)

Objects have file-like methods for access Open, close, read, write, get/set attributes Commands are authorized

Industry offerings Lustre (www.lustre.org) Bull, CFS, Cray, HP, Scali, SUN Lustre based Panasas

SNIA OSD Working Group OSD as a SCSI command set www.snia.org/tech_activities/workgroups/osd

ID x123Blocks:3,42Length:512


Object-basedStorageDevice


Object-basedStorageDevice

EDUCATION


32


Block Interface

Storage Device

Block I/O Manager

Object Interface

Applications

File SystemUser Component

File SystemStorage Component

System Call Interface

CPUApplications

File SystemUser Component

System Call Interface

CPU

Storage Device

Block I/O Manager

File SystemStorage Component

EDUCATION


33


ManagersMA

NAGEM

ENTEth switchSAN

Clients

Object-based Storage Devices

SECRETSECRETKEYKEY

SECRETSECRETKEYKEY

SECRETSECRETKEYKEY

Access Request

DATA

Intelligent Device

Space ManagementBackup/RecoveryQoS via attributesSecurity

Validate CapabilityValidate CapabilityValidate Capability

EDUCATION


34

How is data shared ?File/File System level Parallel NFS (pNFS)

Now pNFS Goal

NFS Server

Host Net

Storage Net

NFSv4

Storage Servers

ClientClient

Host Net

Storage Net

NFSv4

NFS Server

Storage Servers

Data

Data

Allow NFSv4, data to bypass NFS server No application changes, similar management model

pNFS extensions to NFSv4 communicate data location to clients Clients access data via Fibre Channel, iSCSI, OSD, or even NFS

IETF standardization in progress

EDUCATION


35

How is data shared ?File/File System level SANFS vs CFS

Characteristics& Features SAN FS Cluster FS

Tolerance of Distance(between server and clients)

Important Limited

# of nodes Hundreds Dozens

Heterogeneous OS Yes No

Dedicated Meta-Data Server(s) Required Yes, usually

No cluster assigns functions to nodes

Physical filesystemlayout knowledge

Metadata server only(clients may understand

if same OS)

All nodes (Cluster FS currently requires

same OS)

EDUCATION


36

How is data shared ?File/File System level Parallel File System

Concept/Idea: Data is striped between servers (I/O nodes)

Features Cluster-wide consistent name space User control for file striping across I/O nodes

Asymmetric (master + slave servers + clients) GoogleFS, PVFS*, IBRIX, Panasas (osd)

Symmetric (peer servers + clients) TerraScale, Isilon, Exanet, NetApp (Spinnaker Networks)

* Parallel Virtual File System

EDUCATION


37

How is data shared ?File/File System level Parallel File System

Asymmetric Symmetric

#1#1#0

#2#2

#0

EDUCATION

Conclusion

EDUCATION


39

ConclusionVarious ways to Share Data

Many products and philosophy in the industry OS, disk (local) file system Methods to protect data (locking) Cache coherency mechanisms and semantics Caused by varied objectives and applications

There is no single, simple, efficient data format available on all operating systems !! (sorry) Server and client software needed for Data Sharing Remember VTOC and Byte ordering potential issue

EDUCATION


40

Conclusion to leverage the infrastructure

There are a number of things to consider when choosing a file system or server Will the application work as desired? Will it perform and scale? Does it have the required data management services? Is it secure enough? Is it easy to use and manage?

There is no single solution that is superior in all cases BUT these approaches deliver real applications and business benefits Real measured ROI Performance, Availability and Manageability

EDUCATION


41

Q&A / Feedback

Please send any questions or comments on this presentation to SNIA: [email protected]

Many thanks to the following individuals for their contributions to this tutorial.

SNIA Education Committee

Symantec (Philippe Nicolas) ONStor (Jonathan Goldick) EMC (David Black) CA, Cisco, CNT, Crossroads, EvaluatorGroup, HDS,

HGAI, Inrange, Knowledge Transfer, Microsoft, NationWide, QLogic, Sandia National Laboratories, Seagate, Solution Technology, Sun Microsystems & VERITAS Software

EDUCATION

Advanced Data SharingTechnologies version 9

Philippe Nicolas, SymantecJonathan Goldick, ONStor

advanced data sharing technologies

Documents