genesis ii open source, ogsa implementation genesis ii: easy-to-use, standards based grid middleware...
TRANSCRIPT
Genesis II“Open Source, OGSA Implementation”
Genesis II:Easy-to-use, Standards Based Grid
Middleware
Andrew Grimshaw
Genesis II Team
University of Virginia
“Open Source, OGSA Implementation”
Genesis II“Open Source, OGSA Implementation”
Outline
• Background
• Standards
• Genesis II
• Demo
• Conclusion
Genesis II“Open Source, OGSA Implementation”
Background
• Specifications cannot exist in vacuum– Implementations vet Specifications– Implementation experience shows how spec’s
interact
• Grids have been around for a while, but adoption remains low – why?– Usability– Gap between grid designers and grid users
Genesis II“Open Source, OGSA Implementation”
Why have Grids failed to cross the chasm?
• Applications programmers are exposed to the complexity of the underlying environment– Failures, data/application management, logging, …– Only the enthusiasts want to live on the bleeding edge– Joe six-pack (biologist | chemist | economist | *) just wants to do their
domain research – not do heavy duty hacking– “Activation energy” is too high
• Once they have spent the effort to port there is little leverage – applications written for one software stack will not work on other stacks – often they will not even work on different versions of the same software stack– They must suffer the pain all over again
• Business needs solutions not technology– HTC solutions (e.g., SGE, LSF, Condor) are easy to use – but not really
needed cross-site. – Data sharing solutions are primitive and do not integrate well with the
existing infrastructure
Genesis II“Open Source, OGSA Implementation”
Genesis II
• Users FIRST!– Design the system from the ground up with the overriding
mantra that users come first!– Users don’t want to know about grids– Provide a secure, cohesive system in a production system
available to users today!
• Provide an open source, reference implementation of the OGSA and OGSA-related specifications– Use standards and proto-standards available from the OGF and
OGSA to– Provide feedback into the OGF process on various standards
based on implementation experience
Genesis II“Open Source, OGSA Implementation”
Outline
• Background
• Standards
• Genesis II
• Demo
• Next Steps
Genesis II“Open Source, OGSA Implementation”
Standards Represented in Genesis II
WSDL
SOAP
WS-Addressing
WS-SecurityXACML
SAML
ShibbolethOGF
OGSA
WS-Naming
BES
ByteIO
RNS
JSDL
Genesis II“Open Source, OGSA Implementation”
RNS
• Basic directory service maps strings to EPRs.– Similar to Unix directories (EPR instead of inode)– Can “point” to anything that has an EPR.
Genesis II“Open Source, OGSA Implementation”
Pathnames traverse the graph/usr/bin/gcc
/
bin
usr hosts
B CA
gcc
RNS Directory
Resource
Genesis II“Open Source, OGSA Implementation”
OGSA-ByteIO
Genesis II“Open Source, OGSA Implementation”
ByteIO Properties
Genesis II“Open Source, OGSA Implementation”
JSDL – Job Submission Description Langauge
• XML based - extensible• Widely adopted• Five sections
– Job structure– Job Identity– Job application– Job resource– Job data staging
• Not intended for human consumption
Genesis II“Open Source, OGSA Implementation”
Example JSDL
<JobDefinition><JobDescription><Application><POSIXApplication>
<Executable>gnomad</Executable><Argument>ribo</Argument></POSIXApplication></Application>
<DataStaging> <FileName>ribo.d</FileName> <CreationFlag>overwrite</CreationFlag><DeleteOnTerminate>true</DeleteOnTerminate> <Source><URI>rns:/home/mmm2a/data/ribo.567.x</URI></Source></DataStaging><DataStaging> <FileName>tmapfinal</FileName> <CreationFlag>overwrite</CreationFlag><DeleteOnTerminate>true</DeleteOnTerminate> <Target><URI>rns:/home/mmm2a/res/final.567</URI></Source></DataStaging></JobDescription></JobDefinition>
Genesis II“Open Source, OGSA Implementation”
OGSA-BESBasic Execution Service
• Service interface for starting and managing remote compute jobs
• JSDL as the job specification language• Implementation is not specified
– Queue– Fork/exec– Virtualize– Etc.– Emphasis on Basic
• Comes together in the HPC-BP
Genesis II“Open Source, OGSA Implementation”
WSI-BSP – Basic Security Profile
• Most widely used WS security profile.
• Specifies many things, including how and where identity tokens are passed, their schema, and what they should be called.
• We support username/password, X.509, and SAML.
Genesis II“Open Source, OGSA Implementation”
WS-SecureAddressing
• A profile on WS-Addressing and WS-SecurityPolicy
• Embeds specified URI’s in EPR metadata that correspond to profiled security policies– Tells client the security policy of the refereced
endpoint
• Embeds public key into EPR so that the client can communicate securely with the endpoint.
Genesis II“Open Source, OGSA Implementation”
WS-Naming
• Profile on WS-Addressing EPR• Two components
– Endpoint Identifiers– Endpoint Resolution
• Dynamic rebinding to support traditional distributed system transparencies: migration, failure, replication, etc.
Genesis II“Open Source, OGSA Implementation”
Outline
• Background
• Standards
• Genesis II
• Demo
• Conclusion
Genesis II“Open Source, OGSA Implementation”
Goals
• Provide an easy-to-use production Grid system with compute and data Grid capabilities to various partner groups and research projects,
• Provide a fully functional Grid framework on which further Grid research could be performed,
• To “test drive” the various specifications making their way through various standardization organizations to both vet and better understand those specifications, both in isolation, and together as a whole.
Genesis II“Open Source, OGSA Implementation”
Genesis II Features
• Ease-of-use – focus on user experience
• Standards based – both WS and OGF
• Data and compute grid
• Open Source – Java/Apache Axis
• Robust
• Secure
Genesis II“Open Source, OGSA Implementation”
User’s First
• A large percentage of a grid’s target audience is unable or unwilling to learn new interaction abstractions
• Instead of asking the user to adapt to the grid, we should adapt the grid to the user
Genesis II“Open Source, OGSA Implementation”
User Abstractions
• One of the most ubiquitous user interaction abstractions is the file system– Drag-and-drop– Double Click– /proc filesystem– Named pipes
• RNS, ByteIO, and WS-Naming provide the foundation for building these abstractions
Genesis II“Open Source, OGSA Implementation”
(Most) everything is a file or directory
• Files and directories can be accessed without knowing anything about Grids, Web Services, or anything thanks to FUSE/IFS– FUSE/IFS map the Grid into the file system
• BES resources, queues are directories– “ls” to list the jobs, “cat” a job to see its state– “cp” a JSDL file into the directory -> start the job up– A shell script can start jobs by copying
• Genesis II containers are directories– “ls” to see the services and porttypes
• IDP are files/directories• IS’s are directories
– “cp” a query file to IS, creates result• RDBMS’s will be directories• The user can access all of these services without dealing with Web
Services!!
Genesis II“Open Source, OGSA Implementation”
Genesis II - Data
• Grid shell– Usual commands, ls, cat, etc.
• FTPd for Windows
• FUSE for Linux
• IFS for Windows
• ExportDir tool
• Replicated ExportDir
Genesis II“Open Source, OGSA Implementation”
Why FUSE?
• FUSE is a per-user proxy to the Grid– Permissions and access based on the user’s
credentials
• FUSE maps Grid name space (RNS) into local operating system.
• The Grid looks like a local file system.
Genesis II“Open Source, OGSA Implementation”
Genesis II - Compute
• Grid shell– Run, qsub, qdel,
• BES resources mapped into the file system– All BES resources implement HPC-BP and HPC-FSE
• Genesis queues are workload managers• Both BES containers and queues support the
RNS interface• Applications can be run by copying JSDL files
into containers• JSDL tool• Application Management
Genesis II“Open Source, OGSA Implementation”
Genesis II - Security
• WSI-BSP– SSL/TLS and identity token(s) in SOAP header
• Authentication via – X.509 based
• Uses existing credential infrastructure– Signed SAML assertions (groups realized this way)– Username/token– “login” consists of specifying cert file or authenticating to a WS-
Trust STS • Message integrity via both transport and message level
encryption and signing (configurable)• Authorization via plug-able module
– Currently using access control lists
Genesis II“Open Source, OGSA Implementation”
Demonstration
• ssh onto Linux machine back home
• Windows interface from this machine in Catania
• Connected to XCG – Cross Campus Grid
Genesis II“Open Source, OGSA Implementation”
Steps
• Grid shell– Ls /, /bes-containers, /groups, – Login & cd /home/grimshaw– ls, ls –epr, – Fuse mount
• In Linux, ls grid, cd home/grimshaw– Ls gnomad, pictures, WorkPC– Gnomad demo, pictures demo, – JSDL tool– ls around in the queues– drop a bunch of jobs into a queue– Cd grid/queues, ls around– ExportDirTool from my desktop
Genesis II“Open Source, OGSA Implementation”
Genesis IITake-away messages
• Sufficient body of standards exist with which to build interesting, useful grid systems
• Available under Apache 2 licenses• Very active project!• Information and Download Page
– http://vcgr.cs.virginia.edu/genesisII
• Forum– http://www.cs.virginia.edu/forums/viewforum.php?f=26