supporting application- tailored grid file system sessions...
TRANSCRIPT
![Page 1: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/1.jpg)
Advanced Computing and Information Systems laboratory
Supporting Application-Tailored Grid File System
Sessions with WSRF-Based Services
Ming Zhao, Vineet Chadha, Renato Figueiredo
Advanced Computing and Information SystemsElectrical and Computer Engineering
University of Florida
![Page 2: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/2.jpg)
Advanced Computing and Information Systems laboratory 2
Motivating Example
Curvilinear-grid Hydrodynamics 3D model
Simulating WAvesNearshore model
Regional-scale, curvilinear simulation grid of Tampa Bay
CH3D SWAN
ADCIRC
Every 30 timesteps1.5MB per exchange
Every 30 timesteps1.8MB per exchange
Every timesteps40KB per exchange
ADvanced CIRCulationmodel for coastal waters
Basin-scale, unstructured ADCIRC simulation grid
Tampa Bay
Coastal surge coupled modeling
Shared file system facilitates communication and synchronization between coupled applications
![Page 3: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/3.jpg)
Advanced Computing and Information Systems laboratory 3
Motivating Example
Curvilinear-grid Hydrodynamics 3D model
Simulating WAvesNearshore model
CH3D SWAN
ADCIRC
Every 30 timesteps1.5MB per exchange
Every 30 timesteps1.8MB per exchange
Every timesteps40KB per exchange
ADvanced CIRCulationmodel for coastal waters
Coastal surge coupled modeling
Shared file system facilitates communication and synchronization between coupled applications
WAN
Distributed file systems in wide-area environments?• LAN file systems have shortcomings • WAN file systems not widely deployed
UFL LSU
VIMS
![Page 4: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/4.jpg)
Advanced Computing and Information Systems laboratory 4
Motivating Example
CH3D SWAN
ADCIRC
Coastal surge coupled modeling
Grid Virtual File System (GVFS)• Virtualization, user-level proxy, unmodified kernel NFS• Cross-domain user identity mapping• Performance, security, consistency, reliability enhancements
Dynamic, independent, application-tailored GVFS sessions
WAN
GVFS Session2
UFL LSU
VIMS
GVFS Session1
![Page 5: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/5.jpg)
Advanced Computing and Information Systems laboratory 5
Motivating Example
CH3D
ADCIRC
Coastal surge coupled modeling
How to manage Grid data sessions• Creation, cleanup, isolation, customization …
WANUFL LSU
VIMS
WSRF-based data management services• Interoperability, flexibility, state management
Service
GVFS Session1
GVFS Session2
SWAN
![Page 6: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/6.jpg)
Advanced Computing and Information Systems laboratory 6
Overview
Goal: • Seamless and high-performance data provision for
applications in Grid environments
Challenges: • Application transparency for Grid-enabling of a wide
range of applications• Application-tailored enhancements on performance
and reliability for diverse application needs
Contributions: • WSRF-based data management services• Enabling of application-tailored grid data sessions
![Page 7: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/7.jpg)
Advanced Computing and Information Systems laboratory 7
Outline
Introduction
Architecture
• Data Access: Application-Tailored Sessions
• Control: Data Management Services
Evaluation
Summary
![Page 8: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/8.jpg)
Advanced Computing and Information Systems laboratory 8
Architecture
JobJob
Compute server
JobJob
File server
Proxy
Service
Compute server
data
Proxycache
Proxycache
Proxycache
Proxycache
Proxy
Proxy
Data plane• Proxy-based data sessions
Control plane• Data management services
Service
Service
Service
![Page 9: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/9.jpg)
Advanced Computing and Information Systems laboratory 9
Application-Tailored Data Sessions
Grid data access• Implicit: GVFS proxy RPC interception
• Partial file transfer, block-based disk caching• Configurable cache parameters:
• Capacity, associativity, read/write, write-through/-back• Security mechanisms
• Session-key authentication, encrypted data channel
• Explicit: GridFTP/SFTP• Full file transfer, file-based disk caching• Data accessible through GVFS interface
Cache consistency modelsFault tolerance techniques
![Page 10: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/10.jpg)
Advanced Computing and Information Systems laboratory 10
Cache Consistency ModelsPer-session customization• Overlaid upon native NFS client
polling mechanism• Reconfigurable at run-time
Suitable for various scenarios• Single-client sessions:
• Aggressive read/write caching with write delay
• Multiple-client sessions:• Relaxed polling-based model• Strong callback-based model
cache
kernel
APP Proxy
NFSclient
Systemcalls
RPCs tolocalhost
user
RPCs to server
![Page 11: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/11.jpg)
Advanced Computing and Information Systems laboratory 11
Fault ToleranceCopy-on-write file system• Fail-over client failures• Buffers data modifications on
local stable storage• Application checkpointed with
file system changes consistently
Session redirection• Fail-over server failures• Fault detected by RPC timeout• Subsequent requests redirected
to replica server • Proxy remaps file handles
transparently from kernel
cache
kernel
APP Proxy
NFSclient
Systemcalls
RPCs toloalhost
user
COWsever
shadowfiles
RPCs to serverXRPC redirection
![Page 12: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/12.jpg)
Advanced Computing and Information Systems laboratory 12
Architecture
JobJob
Compute server
JobJob
File server
Proxy
Service
Compute server
data
Proxycache
Proxycache
Proxycache
Proxycache
Proxy
Proxy
Data plane• Proxy-based data sessions
Control plane• Data management services
Service
Service
Service
![Page 13: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/13.jpg)
Advanced Computing and Information Systems laboratory 13
Data Management Services
Service oriented middleware• Creation, customization, management of sessions• File System Service (FSS)• Data Scheduler Service (DSS)• Data Replication Service (DRS)
Built using WS-Resource Framework• Interoperability and state management
Implemented with Perl-based WSRF::Lite• WS-Addressing, WS-ResourceProperties, WS-
ResourceLifetime, WS-BaseFaults, WS-Security etc.
![Page 14: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/14.jpg)
Advanced Computing and Information Systems laboratory 14
File System Service (FSS)Management of GVFS proxies
Customization • Defined in a configuration file• Represented as WS-Resource
Property
Reconfiguration:• By signaling proxy to reload
configuration file
Monitoring:• By signaling proxy to report
accumulated statistics
base_path /home/cachesession_key XXYYZZacache_enabled 1dcache_enabled 1wb_enabled 1inval_enabled 1acache_size 65536acache_asso 8acache_banks 128dcache_size 1048576dcache_asso 16dcache_banks 512inval_min 3inval_max 60
Configuration File
![Page 15: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/15.jpg)
Advanced Computing and Information Systems laboratory 15
Data Scheduler Service (DSS)
Manages Grid data sessions• Interacting with client- and server-side FSS• Session information
• Represented as WS-Resource Property• Stored in MySQL database
Resolves conflicts when scheduling a session• If another session accesses with write caching
• Forces it to write back and disable write caching• If another session has exclusive access
• Denies the new session request
![Page 16: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/16.jpg)
Advanced Computing and Information Systems laboratory 16
Data Replication Service (DRS)
Manages data replication• Replica information represented as WS-Resource
Property, stored in MySQL database
Interacts with DSS for replication and recovery• Replication: requests a session for data transfer• Recovery: provides replica information to client FSS
Supports various consistency schemes• Uses COW to avoid propagation of writes• Active-style or primary-based
![Page 17: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/17.jpg)
Advanced Computing and Information Systems laboratory 17
Example
JobJob
Compute server
JobJob
File server
Proxy
DSS
Compute server
data
Proxycache
Proxycache
Proxycache
Proxycache
Proxy
Proxy
FSS
FSS
FSS
replica
File server
Serviceclient
e.g. In-VIGO
FSSDRS
Data Scheduler Service (DSS)• Session management
File System Service (FSS)• Proxy management
Data Replication Service (DRS)• Replication management
![Page 18: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/18.jpg)
Advanced Computing and Information Systems laboratory 18
Outline
Introduction
Architecture
Evaluation
Summary
![Page 19: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/19.jpg)
Advanced Computing and Information Systems laboratory 19
Weak Consistency: Experiment IBenchmark:• NanoMOS (MATLAB-based 2-
D n-MOSFET simulator)
Scenario:• Software accessed by WAN
users and updated by local administrator:(a) Major: entire MATLAB(b) Minor: one MATLAB toolbox
• NFS vs. GVFS
Observation:• With warm disk cache GVFS
• Filters substantial kernel issued consistency checks
• Delivers performance close to local disk
0
100200
300400
500600
700
Runt
ime
(sec
onds
)
NFS GVFS LOCAL
(a)
Majorupdate
0
100
200
300400
500
600
700
1 2 3 4 5 6 7 8Execution Iterations
Runt
ime
(sec
onds
)
(b)
Minorupdate
![Page 20: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/20.jpg)
Advanced Computing and Information Systems laboratory 20
Weak Consistency: Experiment IIBenchmark: • CH1D (coupled hydrodynamics
simulation and post-processing)
Scenario:• Real-time data accumulated on-
site, and processed off-site• 30 new inputs available before
each run of data processing• NFS vs. GVFS
Observation:• As input dataset grows overhead
caused by consistency checks: • Grows linearly in native NFS, • Stays constant in GVFS
0
10
20
3040
50
60
70
80
1 2 3 4 5 6 7 8 9 10
Execution Iterations
Run
time
(sec
onds
)
NFS GVFS
![Page 21: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/21.jpg)
Advanced Computing and Information Systems laboratory 21
Checkpointing and RecoveryApplication:• Gaussian (computational chemistry tool)
Scenario:• Client (a virtual machine) is checkpointed, continues to
execute and later fails • The program changes the state of the file server irreversibly
– by deleting temporary files after the checkpointing
Observation:• When the VM is resumed to the checkpoint:
• Native NFS: stale file handle error; program aborts• GVFS with COW: program recovered successfully
![Page 22: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/22.jpg)
Advanced Computing and Information Systems laboratory 22
Error Detection and Data Redirection
Application:• SPECseis96 (seismic data processing)
Scenario:• File server fails during the program’s execution
Observation:• Upon native NFS: program fails (aborts or hangs)• Upon GVFS and data replica:
• Proxy detected the error after a RPC timeout• The data request is redirected to the replica within 5 seconds• Program continues successfully and is unaware of the failure
![Page 23: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/23.jpg)
Advanced Computing and Information Systems laboratory 23
Summary
Problem: Application-transparent and application-tailored Grid data access
Solution: WSRF-based data management services for application-tailored Grid file system sessions
Evidence: Experiments based on scientific application execution demonstrate good performance and fault tolerance of GVFS
![Page 24: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/24.jpg)
Advanced Computing and Information Systems laboratory 24
Related Work
Grid data management approaches• GASS, GridFTP
• Explicit transfer via middleware or use of specialized API• Condor, BAD-FS
• On-demand remote data access by interception of system call• Control caching, consistency and fault tolerance to middleware
• LegionFS, Avaki’s Data Grid Access Servers• Access of Grid data based on NFS
WSRF-based Grid middleware• Globus Toolkit 4 based data management middleware• WSRF.NET based (remote job execution grid)• WSRF::Lite based (WEDS)
![Page 25: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/25.jpg)
Advanced Computing and Information Systems laboratory 25
Acknowledgments
In-VIGO team• http://invigo.acis.ufl.edu
Dr. Peter DindaDr. Peter Sheng, SCOOP resources
NSF Middleware InitiativeNSF Research ResourcesIBM Shared University ResearchVMware
Questions?
![Page 26: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/26.jpg)
Advanced Computing and Information Systems laboratory 26
References[FGCS’05] S. Adabala, V. Chadha, P. Chawla, R. Figueiredo, J. Fortes, I. Krsul, A.
Matsunaga, M. Tsugawa, J. Zhang, M. Zhao, L. Zhu, and X. Zhu, “From Virtualized Resources to Virtual Computing Grids: The In-VIGO System”, special issue on Complex Problem-Solving Environments for Grid Computing, Vol 21/6, 2005.
[CC’04 ] R. Figueiredo, N. Kapadia, J. Fortes, “Seamless Access to Decentralized Storage Services in Computational Grids via a Virtual File System”, In Cluster Computing, 2004.
[HPDC’04] M. Zhao, R. Figueiredo, “Distributed File System Support for Virtual Machines in Grid Computing”, In Proceedings of 13th IEEE International Symposium on High Performance Distributed Computing, June 2004.
WSRF::Lite: An Implementation of the Web Services Resource Frameworkhttp://www.sve.man.ac.uk/Research/AtoZ/ILCT
In-VIGO:
In-VIGO prototype can be accessed from http://invigo.acis.ufl.edu; courtesy accounts available.
![Page 27: Supporting Application- Tailored Grid File System Sessions ...zhaom/research/hpdc05-talk.pdfSystem”, In Cluster Computing, 2004. [HPDC’04] M. Zhao, R. Figueiredo, “Distributed](https://reader033.vdocuments.mx/reader033/viewer/2022042307/5ed2fce26e55d1461a0f7dbe/html5/thumbnails/27.jpg)
Advanced Computing and Information Systems laboratory 27
Future Work
Extensive evaluation and performance tuning of service-based middleware
Use of application profiling to assist the customization of Grid data sessions
Fine grained replication management and load balancing schemes