d-grid infrastructure - tu dortmund · standard-based open-source toolkit to build private, public...
TRANSCRIPT
![Page 1: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/1.jpg)
D-Grid Infrastructure
Stefan FreitagDortmund University of Technology
6th International Workshop on Grid Computing for Complex Problems
Bratislava, November 2010
![Page 2: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/2.jpg)
Outline
● Projects & Infrastructure● Lessons learned & Future Challenges
![Page 3: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/3.jpg)
2004 2007 20102009 20112008 2012
Gap project
20062005
Integration project
Service layer
Academic projects
Public/ private projects
D-Grid GmbH
Working Groups
Pre-project Phase 1
Phase 2
Phase 3
D-Grid Projects
![Page 4: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/4.jpg)
Gap projects
Service layerprojects
Commercial communityprojects
Academic communityprojects
Integration project
Project structure of D-Grid
![Page 5: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/5.jpg)
Representation (?)
Academic service and resource providers
Academic research communities
D – Grid GmbH
Industrial service and resource providers
Organizational issues
Legal issuesInternational
issues
Public Relation
GAUSS-Allianz WissGrid
Representation structure in D-Grid
![Page 6: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/6.jpg)
Academic projects
Astronomy
Climate Research
High Energy Physics
Medical Science
Humanities
![Page 7: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/7.jpg)
Aircraft Industry
Financial Business
Automobile Industry
Building & Construction Industry
Geospatial Information Systems
Medical Services
Meteorology for regenerative energy industry
Commercial Projects
![Page 8: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/8.jpg)
Commercial Projects
Logistics
Media Technology
Chemical Industry
Numerical Simulation
Medical Engineering
Plasma Technology
Dynamic Service Value Networks
![Page 9: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/9.jpg)
Service Projects
Backup & Recovery
Sustainable Grid Infrastructure
SLA4D-Grid
Networks of Knowledge in the Grid
![Page 10: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/10.jpg)
Gap & Service Projects
VO management
D-MON
DGSI
GapSLC
GIDS
![Page 11: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/11.jpg)
D-Grid Infrastructure
![Page 12: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/12.jpg)
Situation in 2005
High energy physicists● advanced knowledge (CERN experiments)● gLite middleware
Other communities● not really any prior knowledge
How to make all communities happy?● Let them choose!● Community requirements still unclear● Offer a set of commonly used middlewares
![Page 13: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/13.jpg)
Leere Fusszeile
Supported Middlewares
Compute middleware● gLite● UNICORE● Globus Toolkit
Storage middleware● dCache● OGSA-DAI
![Page 14: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/14.jpg)
Leere Fusszeile
Software Stack @ D-Grid Resource
![Page 15: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/15.jpg)
Leere Fusszeile
D-Grid Reference System
● Template for a D-Grid compute and storage resource
● Targeted at small and medium resource centers
● Technical documentation & software packages available to system administrators
● http://dgiref.d-grid.de/wiki/Introduction
● Validation of packages and documentation by installation “from scratch”
● Release Cycle: six months
● Reference system is only a recommendation
● Support available
![Page 16: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/16.jpg)
D-Grid Resources
● > 30.000 CPU cores ● > 5 PByte of storage● In 2009 >= 140 Mio.
CPUh consumed by users
● Accounting ● DGAS/ manually
![Page 17: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/17.jpg)
Usage of D-Grid Resources
![Page 18: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/18.jpg)
Towards Sustainability
![Page 19: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/19.jpg)
Technology is not really a problem!
Focus on existing concepts
● Support for different Grid middlewares
● Virtualization
Bridging the gap to services and applications
● Workflow engines
● Commercial software
Addressing new requirements
● Dynamic virtual organizations
Cooperation between different projects
● Scheduling
![Page 20: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/20.jpg)
Heterogeneous Provider Community
Some large hardware providers play a vital role in D-Grid
● Development of a business model for basic services
Some providers used D-Grid to acquire new hardware
● Investments to support a transitional phase to Grid systems
No known concepts to provide IaaS type of services
● Obstacle for commercial users
● Competition to cloud computing?
![Page 21: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/21.jpg)
There are not enough users!
Prominent figures of the various disciplines have not yet joined● Imbalance between computer science and application science
Most users are not willing to provide the required additional effort. ● Potential lack of focus in the application area
Industrial users are pushed off by bureaucratic obstacles● Use of D-Grid hardware for commercial purpose
![Page 22: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/22.jpg)
Grid computing is a solution, isn’t it?
Funding agencies cannot be wrong!
● More than 100 Mio € in D-Grid
Industry cannot be wrong!
● Enterprise Grids
● Cloud Computing
Science cannot be wrong!
● LHC production Grid
![Page 23: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/23.jpg)
Due to the weather conditions... Cloud Computing!
Photo by: Paolo Malgari
![Page 24: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/24.jpg)
Cloud Computing in D-Grid ?
Situation in 2009● Utilization of compute resources ≥ 140 Mio CPUh
● only around 50% of the available CPUh
½ of D-Grid compute resources idling● Turn them off → GreenIT
● Not really acceptable for a sustainable operation
→ Introduce compute cloud interface as new pillar
Why ?
● Grid is too complex and hence not attractive to SMEs
![Page 25: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/25.jpg)
Attractivity of Cloud Interface I
Cloud● Register once via web. Specify name, credit card
information,... (few minutes)
Grid● Application for X.509 certificate (1-2 days)
Contact Certification authority
Contact Registration authority (f2f required)● Application for VO membership (hours - days)● Updating of the authorization information on the Grid
resources (hours - days)
![Page 26: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/26.jpg)
Attractivity of Cloud Interface II
![Page 27: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/27.jpg)
Cloud Computing in D-Grid
● Extension of the software stack
● What to add?● Platform
virtualization● Compute cloud
middleware
![Page 28: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/28.jpg)
D-Grid Cloud Prototype
Located at Dortmund University of Technology
Services● Compute middlewares
– gLite 3.2
– UNICORE 6
– Globus Toolkit 4.0.8
● Storage middlewares
● dCache 1.9.5
● OGSA-DAI WSRF 2.2
Hardware● 2048 CPU cores, 108 TByte storage
All services deployed in Xen virtual machines
![Page 29: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/29.jpg)
OpenNebula
Statement on homepage● Standard-based open-source toolkit to build private, public and
hybrid clouds. [...] provides flexible architecture, interfaces and components that fit into any existing data center.
Supports Xen, KVM and VMware
Interfaces with Amazon EC2 and ElasticHosts
Other users● CERN (cloud infrastructure consisting of 3,200 cores)
● NIKHEF, Netherlands
![Page 30: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/30.jpg)
Prototyp Setup
![Page 31: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/31.jpg)
Open Issues
Technical level● User management● Authorization● Accounting and Billing● Monitoring● Information system
→ Interfaces to all D-Grid core services
![Page 32: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/32.jpg)
User Management
D-Grid● VO central user management
● Resources query VOMRS servers and generate a mapping
● VO member → local user ID
● Attribute-based authorization
Cloud (OpenNebula)● Local user management,
● No Attribute-based authorization
Can the concept of VOs and Cloud middleware go together?
![Page 33: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/33.jpg)
Lessons learned
![Page 34: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/34.jpg)
Reference System Acceptance
Only a few sites ● Follow the recommended setup● Use the technical documentation
Why should a resource install all middlewares?● System administrators are not qualified to support all
middlewares ● Lack of interest at resource provider level
Migration/ Upgrade path unspecified
![Page 35: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/35.jpg)
Legal Issues – Dual Use
Dual-use technology: a technology which can be used for both peaceful and military aims.
Most industrial countries have export controls on certain types of designated dual-use technologies […] These controls restrict the export of certain commodities and technologies without the permission of the government.
(Wikipedia)
● Is Grid a dual-use technology?
● Export embargo for dual-use technologies to Iran
● Technical solution for a legal requirement
![Page 36: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/36.jpg)
Legal Issues – Liability
With the development of business models, the liability of single VO members and of the whole VO is of interest
What is the legal status of a virtual organization?● Company constituted under Civil law
→ Joint and several liability● In joint and several agreements, the liability for default is
enforceable against all of the signatories as a group[...], or against any one of them as an individual at the choice of the enforcing party (www.businessdictionary.com)
Can business models and virtual organizations co-exist?
![Page 37: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/37.jpg)
Avoid Parallel Universes
Before D-Grid German HEPs used WLCG resources
In 2010 German HEPs still use WLCG resources
● and not a single one uses D-Grid resources
Why?
● Accessibility to compute resources, data and experiment software is limited by the boundaries of a virtual organization
● In WLCG VOs for ALICE, ATLAS, CMS and LHCb were already existing
● In D-Grid a HEP-CG VO was established with the aim to absorb all HEPs independent from their origin
● Even if HEPs joined the HEP-CG VO, they were not able to access the experiment software
![Page 38: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/38.jpg)
Future Challenges
![Page 39: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/39.jpg)
Future Challenge: D-Grid/ NGI-DE
EGI was still EGEE as D-Grid started
With the end of EGEE-III, NGI-DE was established
Since then D-Grid and NGI-DE co-exist in Germany
Merge to only one initiative
● Avoid competition
● Combine endeavors for creating a sustainable infrastructure
● Removal of duplicity of service (GRRS/ GOC DB)
● Integration of D-Grid services (D-Mon)
● Operations concept (1:1 mapping, pool accounts)
![Page 40: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/40.jpg)
New Service: D-Mon
![Page 41: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/41.jpg)
Summary
Projects in D-Grid
D-Grid Infrastructure● Software stack, resources● Prototypic integration of a Cloud middleware
Lessons learned
Legal issues & Future challenges
![Page 42: D-Grid Infrastructure - TU Dortmund · Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components](https://reader033.vdocuments.mx/reader033/viewer/2022042808/5f88cc61f70850533866c986/html5/thumbnails/42.jpg)
Thank you for your attention!
If you found this talk interesting, slides are available at
http://www.slideshare.net/stefanfreitag