ogf introductory overview - ogf 44 at egi conference 2015
TRANSCRIPT
©2015 Open Grid Forum
The Open Grid Forum: History, Introduction and Process
Alan Sill VP of Standards, OGF
Director, NSF Cloud and Autonomic Computing Center Senior Scientist, High Performance Computing Center
Adjunct Professor of Physics Texas Tech University
1
Open Grid Forum 44, May 21-22, 2015 EGI Conference, Lisbon, Portugal
© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015
About the Open Grid Forum:
Open Grid Forum (OGF) is a global organization operating in the areas of cloud, grid and related forms of advanced distributed computing. The OGF community pursues these topics through an open process for development, creation and promotion of relevant specifications and use cases. OGF actively engages partners and participants throughout the international arena through an open forum with open processes to champion architectural blueprints related to cloud and grid computing. The resulting specifications and standards enable pervasive adoption of advanced distributed computing techniques for business and research worldwide.
2
© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015
History and Background
• Began in 2001 as an organization to promote the advancement of distributed computing worldwide.
• Grid Forum --> Global Grid Forum --> GGF + Enterprise Grid Alliance --> formation of OGF in 2005.
• Mandate is to take on all forms of distributed computing and to work to promote cooperation, information exchange, and best practices in use and standardization.
• OGF is best known for a series of important computing, security and network standards that form the basis for major science and business-based distributed computing (BES, GridFTP, DRMAA, JSDL, RNS, GLUE, UR, etc.).
• We also develop cloud, networking and data standards (OCCI, DFDL, WS-Agreement, NSI/NML, etc.) in wide use.
• Cooperative work agreements with other SDOs in place.3
© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015
OGF Standards
OGF has an extensive set of applicable standards related to advanced distributed grid and cloud computing and associated storage management and network operation:
- Managing the Trust Eco-System (CA operations, AuthN/AuthZ) - Job Submission and Workflow Management (JSDL, BES) - Network Management (NSI, NML, NMC, NM) - Federated Identity Management (FedSec-CG) - Virtual Organizations (VOMS) - Secure, fast multi--party data transfer (GridFTP, SRM) - Service Agreements (WS-Agreement, WS-Agreement Negotiation) - Data Format Description (DFDL) - Cloud Computing interfaces (OCCI) - Distributed resource management APIs (DRMAA, SAGA, etc.) - Firewall Traversal (FiTP) - (Many others under development)
4
© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015
• DRMAA: Distributed Resource Management Application API Grid Engine (Univa), Open Grid Scheduler: (open source); TORQUE and related products: Adaptive Computing; PBS Works: Altair Engineering; Gridway: DSA Research; HTCondor: U. of Wisconsin / Red Hat;
• OGSA® Basic Execution Service Version 1.0 and BES HPC Profile: BES++ for LSF/SGE/PBS: Platform Computing; Windows HPC Server 2008: Microsoft Corporation; PBS Works - (client only): Altair Engineering;
• JSDL: Job Submission Description Language (family of specifications): BES++ for LSF/SGE/PBS and Platform LSF: Platform Computing; Windows HPC Server: Microsoft Corporation; PBS Works and PBS/Pro: Altair Engineering; Tivoli Workload Scheduler: IBM Corporation;
• WS-Agreement (family of specifications): ElasticLM License-as-a-Service: ElasticLM; BEinGrid SLA Negotiator, LM-Architecture and Framework: (Multiple partners); BREIN SLA Management Framework: (Multiple partners); WSAG4J, Web Services Agreement for Java (framework implementation): Fraunhofer SCAI.
5
OGF HPC Standards In Use In Industry:
© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015
Starting Point: OGF Documents
7
http://ogf.org/documents
© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015
Public Comment process
8
http://redmine.ogf.org/projects/editor-pubcom/boards/
© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015
OGF Document Types
• Informational: To inform the community about a useful idea or set of ideas.
• Experimental: To inform the community about a useful experiment, testbed or implementation of idea or set of ideas.
• Community Practice: To inform the community of common practice or process, with the objective to influence the community and/or document its current practices.
• Recommendations: To publish a specification, analogous to an Internet Standards track document. Recommendations are initially designated as "proposed," and following further experience and review may become full recommendations.
• Further information including guidance and advice contained in GFD.152 at: http://ogf.org/documents/GFD.152.pdf
9
www.egi.euEGI-InSPIRE RI-261323 www.egi.euEGI-‐InSPIRE RI-‐261323
EGI international presenceStorage Value (yearly increase)
Disk (PB) 235 PB (+69%)
Tape (PB) 176 PB (+32%)
Value (yearly increase)
CPU cores 361,300 across 53 countries (1.44 M job/day)
Standards-based international collaboration
EGI Federated Cloud: A successful standards-based international federated cloud infrastructure
Credit: David Wallom
Chair EGI Federated Cloud Task Force
Members •70 individuals •40 institutions •13 countries
Stakeholders •23 Resource Providers •10 Technology Providers •7 User Communities •4 Liaisons
Technologies •OpenStack •OpenNebula •StratusLab •CloudStack (in evaluation)
•Synnefo •WNoDeS
BSC
CNRS
LMU
OeRC
Masaryk
TUD
IFAE
Cyfronet
100%IT
CESNET
RADICAL
SRCE
DANTE
FZJ
GRNET
GWDG
STFC
SARA
KTH
INFNFCTSG
EGI.eu
Imperial
CESGACETA
IFCA
IGI
IPHC
IN2P3
SZTAKI
IISAS SixSq
Standards •OCCI (control) •OVF (images) •X.509 (authN) •CDMI (storage - under development)
(Updated July 2014)
www.egi.euEGI-InSPIRE RI-261323
Federated Cloud architecture
Open to new members: Join as user, or as an IaaS/PaaS/SaaS service provider:
http://go.egi.eu/cloud
Cloud hypervisor (e.g. OpenStack, OpenNebula, EmotiveCloud, Okeanos…)
Cloud site academic/commercial
Domain specific services in Virtual Machine Images
FedCloud User interfaces
Standards used to enable federation • OCCI: VM Image management • OVF: VM Image format • X509: Authentication • (CDMI: Storage)
FedC
loud
Ope
ration
interfaces
• Information system (BDII) • Monitoring (SAM) • Accounting (APEL) • AAI (Perun)
Virtual organisations
• GLUE2: Resource discovery and Description
• Others in development
Federation monitoring
© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015
Example: Worldwide LHC Computing Grid
13
~450,000 cpu cores ~430 Pb storage
Typical data transfer rate: ~12 GByte/sec
Total worldwide grid capacity: ~2x WLCG across all grids and
VOs
The Role of Standards for Risk Reduction and Inter-operation in XSEDE
XSEDE: The Next Generation of US National Supercomputing
Infrastructure
Cloud and grid standards now power some of the largest academic supercomputing infrastructures in the world!
LSN-MAGIC Meeting February 22, 2012XSEDE Services Layer:
Simple services combined in many ways
15
–Resource Namespace Service 1.1 –OGSA Basic ExecuNon Service –OGSA WSRF BP – metadata and noNficaNon –OGSA-‐ByteIO –GridFTP –JSDL, BES, BES HPC Profile –WS Trust Secure Token Services –WSI BSP for transport of credenNals –… (more than we have room to cover here)
Examples – (not a complete list)
Basic message: XSEDE represents best-of-breed engagement of open computing standards with the US cyberinfrastructure.
Blacklight Shared Memory 4k Xeon cores
Darter 24k cores
Nautilus Visualization Data Analytics
Keeneland CPU/GPGPU
Stampede 460K cores w. Xeon Phi >1000 users Upgrade in 2015
Wrangler Data Analytics
Trestles IO-‐intensive 10k cores 160 GB SSD/Flash
Gordon Data intensive 64 TB memory 300 TB Flash Mem
Open Science Grid High throughput 124 sitesBlue Waters
Leadership
SuperMIC 380 nodes – 1PF (Ivy bridge, Xeon Phi, GPU)
Over 13 million service units/day typically delivered as of 2014 across all XSEDE supercomputing sites (about 3 million core hours/day), totaling about 1.6 billion core hours per year
Yellowstone Geosciences
US National Cyberinfrastructure
Promote an open, robust, collaborative, and innovative
ecosystem
Adopt, create and disseminat
e knowledge
Extend the impact of cyber-
infrastructurePrepare
the current and next
generation
Provide technical
expertise and support services
Collaborate with other CI groups and
projects
FutureGrid*
Maverick Visualization Data Analytics
Comet “Long Tail Science” 47k cores/2 PF High throughput
ACI-‐REF Campus sharing, NSF Cloud (shared)
Grids
Credit: Irene Qualters, US
National Science Foundation
LSN-MAGIC Meeting February 22, 2012Why Open Standards?
• Risk reducMon • Best-‐of-‐breed mix-‐and-‐match • Allows innovaMon/compeMMon at more interesMng layers
• Facilitates interoperaMon with other infrastructures
Andrew Grimshaw
17
Takeaway message• The use of standards permits XSEDE to interoperate with other infrastructures, reduces risks including vendor lock-‐in, and allows us to focus on higher level capabiliMes and less on the mundane
Distributed Across 124 Sites Open Science Grid currently consists of over 124 geographical sites,
operating on a wide variety of computing systems
600k - 800k jobs/day!
© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015 19
OGF Cooperative Agreements In Place as of May 2015
OGF and IEEE: • OGF co-sponsors activities at many IEEE conferences;
pursuing engagements with P2301 & P2302 working groups. OGF and DMTF:
• Joint work register with DMTF on comparison of the OCCI and CIMI specifications.
OGF and ISO/IEC: • OGF has a Category A liaison with ISO/IEC JTC1 SC38 on
Cloud Computing and is working on several joint activities. OGF and SNIA (CDMI):
• OGF has cooperative agreement w/ SNIA with respect to CDMI and has co-hosted 16 community Cloud Interoperability Plugfests so far (series continues at this conference!).
© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015 20
OGF and ETSI: • Cooperative MoU in place; contributing to ETSI CSC effort.
ETSI has joined the Cloud Plugfest series as a co-sponsor. OGF and ITU-T:
• OGF formal liaison in place with ITU-T JCA Cloud group. OGF and TM Forum:
• MoU in place; ongoing cross-SDO work on on End-to-End Management of Cloud Service Agreements, including SLAs.
OGF and CSA: • Cooperative agreement between OGF and CSA in place since
early 2012.
OGF Cooperative Agreements In Place as of May 2015
© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015 21
OGF and OCEAN: • OCEAN has joined the Cloud Plugfest series as a co-sponsor
and as one of the primary sponsors of the previous September 2013 OGF/EGI Cloud Interoperability Week.
OGF and OW2: • OW2 has co-sponsored Cloud Interoperability Plugfests as a
formal co-sponsor and active supporter of open source cloud projects, and supports the OCCIWare project.
OGF and XSEDE: • Extensive XSEDE adoption of OGF standards in production.
OGF and EGI: • Extensive community involvement, interaction and adoption of
OGF specs.
OGF Cooperative Agreements In Place as of May 2015
© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015
Continuing series…
Oriented towards REAL DEVELOPMENT AND TESTING
Past eventsco-sponsored by many open source and standards-related
organizations including: OGF, CSCC, DMTF, SNIA,
OASIS, ETSI, OCEAN and OW2
Long-running developer-oriented
in-person standards and
software testing series
Easy to get involved and join
in events as developers or
project researchers.
Cloud Plugfest 17 at this conference!
http://cloudplugfest.org
Cloud Plugfest Developer Series:
22
More events in planning pipeline.
© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015
Conclusions and Future:
• OGF actively engages many partners and participants throughout the international arena through an open forum with open processes to promote best practices and standards in advanced distributed computing.
• OGF occupies an important role in standards and software development with significant uptake in advanced distributed computing, including cloud, grid, networking and large-scale data processing, transfer and handling.
• OGF documents support a variety of flexible architectures for advanced scientific, community and business uses.
• OGF’s experience has enabled distributed computing built on these architectures to provide more flexible, efficient and utility-like global infrastructures. Join us!
23