![Page 2: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/2.jpg)
Outline
SDSC and History of SRB–Example Project
Introduction to SRB–Discussion on SRB basics–SRB Clients
Overview of a Data Grid–Infrastructure–Topology
Teragrid Demo–How to use your TG SRB account–How to access Digital Data Collections
![Page 3: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/3.jpg)
Archival Systems
18 PB
15.6 TF
DataStarIBM Power4
4.4 TF
TeraGrid Linux Cluster (IA64)
1.4 PB
Storage Area Network Disk Sun F15K
Disk Server
NetworkingVisualization
Storage and Compute
Resources
Human infrastructure: Experienced multi-
disciplinary staff support a broad spectrum of national
science, engineeringand technology projects
Blue Gene/L(Due 12/04)
2.8/5.7 TF
www.sdsc.edu
![Page 4: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/4.jpg)
Sites Using the SRBCiteSeer, Penn StateCity Univ. of New YorkGeospatial Environment, UCSDDrexel UniversityEOSDIS Distributed Active, NASA GoddardGeorgia TechKentucky State Libraries & ArchivesLibrary of CongressLos Alamos National LabNASA AmesNASA Goddard Space Flight CenterNCSA Grid Computing NIH (NCI Center for Bioinformatics)Penn State UniversityPittsburgh Supercomputing CenterPurdue University. IndianaStanford UniversityTACC, University of TexasTexas A & MUC Santa CruzUCLAUCSD NeuroscienceUniversity of MarylandUniversity of Michigan, CAC department University of New MexicoUniversity of WashingtonUniversity of WisconsinUSCYale University
Academia Sinica, TaiwanASCC, Computing Centre, TaiwanAustralian National UniversityBedford Oceanography,CanadaBioinformatics Institute, SingaporeCSIRO, AustraliaData Storage Institute, SingaporeEGEE, French National CenterGeoForschungsZentrum, GermanyJames Cook University, AustraliaKEK High Energy Physics, JapanMax Planck Institute, NetherlandsParallab, NorwaySouth Australian Advanced ComputingUIB (Parallab) , NorwayUniversity of AmsterdamUniversity of Cambridge, AstronomyUniversity of Cambridge, e-ScienceUniversity of EdinburghUniversity of Genoa, ItalyUniversity of Hong KongUnivrsity of ManchesterUniversity of OsloUniversity of SouthamptonYork Univ (UK)
![Page 5: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/5.jpg)
SDSC SRB Projects (60 million, .5 PB )
Digital Libraries– UCB, Umich, UCSB, Stanford,CDL– NSF NSDL - UCAR / DLESE
NASA Information Power GridAstronomy– National Virtual Observatory – 2MASS Project (2 Micron All Sky Survey)
Particle Physics – Particle Physics Data Grid (DOE)– GriPhyN – SLAC Synchrotron Data Repository
Medicine– Digital Embryo (NLM)
Earth Systems Sciences– ESIPS– LTER
Persistent Archives– NARA– LOC
Neuro Science & Molecular Science– TeleScience/NCMIR, BIRN– SLAC, AfCS, …
![Page 6: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/6.jpg)
Storage Resource Broker (SRB)
A distributed file system (Data Grid)–Client-Server, Server-Server architecture.–Abstracts physical
SRB provides the ability to transparently share data across remote sites.–Heterogeneous Resources–Single sign on–Single logical file hierarchy
![Page 7: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/7.jpg)
What we are familiar with
![Page 8: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/8.jpg)
What we are not familiar with, yet
![Page 9: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/9.jpg)
How do the file systems differ?
Logical Abstraction–Folders are NOT physical–Files do NOT inherit physical location–Everything is potentially distributedAccess Control–Permissions are NOT rwxrwxrwx–Permissions ARE on a object by object basis–Groups and permissions ARE more similar to NTFS
Domains–Geographical / logical grouping of users–Namespace scalability: john@harvard john@mit–Also doubles as groups
![Page 10: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/10.jpg)
Interfaces to theStorage Resource Broker
inQ – Windows ClientScommands – UNIX, DOS Command line ClientJargon – Java API and GUI componentsmySRB – Web ClientMatrix – WSDL, Data Grid WorkflowsC, C++ – C and C++ APIPython – Python APIPerl – Perl API
![Page 11: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/11.jpg)
Common Scommands (75 total)
SinitSenvSpwdSlsScdSgetSputSsh
ScpSmv (logical)Sphymove (physical)SrmSmkdirSrmdirSerrorSchmodSexit
![Page 12: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/12.jpg)
mySRB
![Page 13: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/13.jpg)
BIRN Portal (perl based)
![Page 14: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/14.jpg)
NEEScentral Portal (php based)
![Page 15: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/15.jpg)
Biomedical Informatics Research Network (BIRN)
Major collaboration with SDSC, several of the projects’ Co-Investigators and Co-PIs are at SDSC.
BIRN’s purpose is to provide it’s consortium of neuroscience laboratories the ability to share, compute, and collaborate.
The Storage Resource Broker provides the ability to transparently share data across remote sites.
![Page 16: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/16.jpg)
The BIRN SRB Data Grid
![Page 17: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/17.jpg)
Doing this “Manually”
![Page 18: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/18.jpg)
The BIRN Data Grid
![Page 19: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/19.jpg)
The grid is in the details
![Page 20: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/20.jpg)
File Replication
Sls/home/Demo/SRB-Tutorial/files-2: Doc.txt
Sls -l/home/Demo/SRB-Tutorial/files-2: romanoly 0 z-ucsd-ncmir-nas1 15 2003-07-09-05.15 Doc.txt romanoly 1 z-jhu-cis-nas0 15 2003-07-09-05.16 Doc.txt romanoly 2 z-stanford-lucas-nas 15 2003-07-09-05.16 Doc.txt romanoly 3 z-umn-cmrr-nas0 15 2003-07-09-05.16 Doc.txt romanoly 4 z-uci-bic-nas0 15 2003-07-09-05.17 Doc.txt
![Page 21: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/21.jpg)
Teragrid SRB
–All Teragrid accounts are given a SDSC SRB Teragrid account•The ‘username’ is the same as your SDSC UNIX account name•Your SRB ‘domain’ is ‘teragrid’•You must register your DN string with SDSC’s grid-mapfile or request a SRB password to activate your SRB account. Instructions to do so are here: http://www.sdsc.edu/user_services/srb/account_form.html
–Your ~/.srb/.MdasEnv file OR env variablesmdasCollectionHome '/home/<user-name>.teragrid'mdasDomainName 'teragrid'srbUser '<user-name>'#AUTH_SCHEME 'ENCRYPT1'AUTH_SCHEME 'GSI_AUTH'srbHost 'srb.sdsc.edu'srbPort '7321'defaultResource 'sfs-tape-tgd'SERVER_DN '/C=US/O=NPACI/OU=SDSC/CN=Storage Resource Broker/USERID=srb'
![Page 22: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/22.jpg)
Scommand Features
Command line interface -> SCRIPTING
Available for all of the most popular UNIX flavors and DOS
S-commands are the most flexible and powerful of the clients
They are the fastest, and most reliable
They are multithreaded for big gains in data flow
They are great for scripts, perl wrappers, batch jobs, etc…
Installed man pages via “man [Scommand]”– man Sput
![Page 23: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/23.jpg)
Scommand Notes
Shelp–Gives list of commands with brief summary–“[Scommand] <return>” gives usage info (usually) or try –h flag
Sinit – establishes a sessionSenv – displays connection informationSpwd – display current working directorySexit – ends session
![Page 24: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/24.jpg)
Some Public SRB Collections
Southern California Earthquake Center/home/public.teragrid/SCEC
Two Micron All Sky Survey/home/public.teragrid/2MASS
The Palomar Digital Sky Survey/home/public.teragrid/DPOSS
![Page 25: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/25.jpg)
Watch me do a SRB demo
![Page 26: By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the](https://reader030.vdocuments.mx/reader030/viewer/2022032803/56649e2d5503460f94b1c58e/html5/thumbnails/26.jpg)
Thanks!SRB handles large data and provides the ability to share and collaborate on distributed heterogeneous resources.
Questions?
Teragrid SRB userguide: http://www.sdsc.edu/user_services/srb SRB website:www.sdsc.edu/srbSRB email:[email protected]