introduction to grid technologies in egee
DESCRIPTION
Introduction to Grid Technologies in EGEE. Emanouil Atanassov, Aneta Karaivanova and Todor Gurov Institute for Parallel Processing - BAS. Overview. Evolvement of Grids What is Grid? Grid Services Goals of the EGEE project Building a production Grid for e-Science - PowerPoint PPT PresentationTRANSCRIPT
INFSO-RI-508833
Enabling Grids for E-sciencE
www.eu-egee.org
•iASTRO MC MEETING&WORKSHOP, 27-30, APRIL, 2005,SOFIA, BULGARIA
Introduction to Grid Technologies in EGEE
Emanouil Atanassov,
Aneta Karaivanova and Todor Gurov
Institute for Parallel Processing - BAS
Grid Day Nis 31 Jan 2006 2
Enabling Grids for E-sciencE
INFSO-RI-508833
Overview
• Evolvement of Grids• What is Grid?• Grid Services• Goals of the EGEE project• Building a production Grid for e-Science• Grid applications in EGEE and SEE-GRID• The Grid Challenges
Grid Day Nis 31 Jan 2006 3
Enabling Grids for E-sciencE
INFSO-RI-508833
Evolvement of Grids
Historical perspective• Local Computing
– All computing resources at single site.– People move to resources to work.
• Remote Computing– Resources accessible from distance.– All significant resources still centralized.
• Distributed Computing– Resources geographically distributed.– Specialized access; largely data transfers.
• Grid Computing– Resources and services geographically distributed.– Standard interfaces; transfers of computations and data.
• Web Services and Grid Computing – Grid Services– Industry adopts Grid technology
Grid Day Nis 31 Jan 2006 4
Enabling Grids for E-sciencE
INFSO-RI-508833
What is GRID?
“Coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations” (I.Foster)– Resources are controlled by their owners– The Grid infrastructure provides access to collaborators
A Virtual Organization is:– People from different institutions working to solve a common goal– Sharing distributed processing and data resources
Enabling People to Work Together on Challenging Projects– Science, Engineering, Medicine… - e-Science, e-Health– Public service, commerce… - e-Government, e-Business
The Grid could be the “new age” Internet– ‘[The Grid] intends to make access to computing power, scientific data
repositories and experimental facilities as easy as the Web makes access to information.’, UK PM, 2002
Grid Day Nis 31 Jan 2006 5
Enabling Grids for E-sciencE
INFSO-RI-508833
The GRID vision
• On one hand:– Researchers/employees perform their activities
regardless of geographical location, interact with colleagues, share and access data
• On the other hand:– Scientific instruments and experiments provide huge
amount of data, incl. national databases• And in the middle:
– The Grid: networked data, processing centres and ”grid middleware” as the “glue” of resources.
Grid Day Nis 31 Jan 2006 6
Enabling Grids for E-sciencE
INFSO-RI-508833
Grid Services
• Basic unit of computation – job• Basic unit of storage – file• Information systems – BDII, Globus-mds, R-GMA, file
catalogues, metadata catalogues• Authorization, authentication, accounting (AAA)–
based on PKI (Public key infrastructure)• Every Grid site provides basic Grid services• Advanced Grid Services: MPI jobs, Mass Storage
Facilities accessed via SRM, Fine grained AAA (VOMS, DGAS).
Grid Day Nis 31 Jan 2006 7
Enabling Grids for E-sciencE
INFSO-RI-508833
Grid Services - schema
Grid Day Nis 31 Jan 2006 8
Enabling Grids for E-sciencE
INFSO-RI-508833
Grid Services in gLite
Grid Day Nis 31 Jan 2006 9
Enabling Grids for E-sciencE
INFSO-RI-508833
EGEE Partner Federations
All work in EGEE will be carried out by the 70 partners grouped in 12 federations.
Grid Day Nis 31 Jan 2006 10
Enabling Grids for E-sciencE
INFSO-RI-508833
Goals of the EGEE project
• Goal in one sentence:– Allow scientists from multiple domains to use,
share, and manage geographically distributed resources transparently.
• The EGEE project brings together experts from over 27 countries with the common aim of building on recent advances in Grid technology and developing a service Grid infrastructure, available to scientists 24 hours-a-day.
• The project aims to provide researchers in academia and industry with access to major computing resources, independent of their geographic location. The EGEE project will also focus on attracting a wide range of new users to the Grid.
Grid Day Nis 31 Jan 2006 11
Enabling Grids for E-sciencE
INFSO-RI-508833
Scientific disciplines to run Grid applications
• EGEE aims to establish production quality sustained Grid services – 3000 users from at least 5 disciplines– integrate 50 sites into a common
infrastructure– offer 5 Petabytes (1015) storage
• Demonstrate a viable general process to bring other scientific communities on board Pilot New
Grid Day Nis 31 Jan 2006 12
Enabling Grids for E-sciencE
INFSO-RI-508833
EGEE – building a production Grid for e-Science
• Operations Management Centre (OMC):– At CERN – coordination etc
• Core Infrastructure Centres (CIC)– Manage daily grid operations –
oversight, troubleshooting– Run essential infrastructure services– Provide 2nd level support to ROCs– UK/I, Fr, It, CERN, + Russia (M12)– Taipei also run a CIC
• Regional Operations Centres (ROC)– Act as front-line support for user and
operations issues– Provide local knowledge and
adaptations– One in each region – many distributed
• User Support Centre (GGUS)– In FZK – manage PTS – provide single
point of contact (service desk)– Not foreseen as such in TA, but need is
clear
Grid Day Nis 31 Jan 2006 13
Enabling Grids for E-sciencE
INFSO-RI-508833
Components of a production Grid
• A production Grid consists of stable interoperating Grid sites (Resource centres), which enable access to Grid users from various Virtual Organizations
• Every Grid site provides basic Grid services and follows strict operational procedures.
• Monitoring allows fast detection of problems and their resolution or isolation.
Grid Day Nis 31 Jan 2006 14
Enabling Grids for E-sciencE
INFSO-RI-508833
BG01-IPP setup
UI
- PKI X.509 certificate keys- JDL files
Terminals
enterGrid
enterGrid
enterGrid
enterGrid
UI WN
WN
WN
WN
WN
WNRB/II
CESE
BDII
Grid Day Nis 31 Jan 2006 15
Enabling Grids for E-sciencE
INFSO-RI-508833
Structure of EGEE operations• The grid is flat, but
• Hierarchy of responsibility– Essential to scale the operation
• CICs act as a single Operations Centre– Operational oversight (grid
operator) responsibility
– rotates weekly between CICs
– Report problems to ROC/RC
– ROC is responsible for ensuring problem is resolved
– ROC oversees regional RCs
• ROCs responsible for organising the operations in a region– Coordinate deployment of
middleware, etc
• CERN coordinates sites not associated with a ROC
CIC
CICCIC
CICCIC
CICCIC
CICCIC
CICCIC
RCRC
RCRC RCRC
RCRC
RCRC
ROCROC
RCRC
RCRC
RCRCRCRC
RCRCRCRC
ROCROC
RCRC
RCRC RCRC
RCRC
RCRC
ROCROC
RCRC
RCRC
RCRC
RCRC
ROCROC
OMCOMC
RC = Resource Centre
Grid Day Nis 31 Jan 2006 16
Enabling Grids for E-sciencE
INFSO-RI-508833
Operations monitoring maps
•In LCG-2:
• 137 sites, 34 countries
• >12,000 cpu
• ~5 PB storage
•Includes non-EGEE sites:
• 9 countries, 18 sites
•In LCG-2:
• 137 sites, 34 countries
• >12,000 cpu
• ~5 PB storage
•Includes non-EGEE sites:
• 9 countries, 18 sites
Grid Day Nis 31 Jan 2006 17
Enabling Grids for E-sciencE
INFSO-RI-508833
Selection of Monitoring tools
GIIS Monitor GIIS Monitor graphs Sites Functional Tests
GOC Data BaseScheduled Downtimes Live Job Monitor
GridIce – VO view GridIce – fabric view Certificate Lifetime Monitor
Note: Those thumbnails are links and are clickable.
Grid Day Nis 31 Jan 2006 18
Enabling Grids for E-sciencE
INFSO-RI-508833
Example: LHC at CERN
Grid Day Nis 31 Jan 2006 19
Enabling Grids for E-sciencE
INFSO-RI-508833
CMS LHC Experiment
Grid Day Nis 31 Jan 2006 20
Enabling Grids for E-sciencE
INFSO-RI-508833
Example biomedical app: gPTM3D
• One data set is – DICOM files: 100MB – 1GB– One radiological image: 20MB – 500MB
• Complex interface: optimized graphics and medically-oriented interactions
• Physician interaction is required at and inside all stepsPoorly discriminant data, pathologies, medical windowing
Interaction
RenderExplore Analyse InterpretAcquire
Grid Day Nis 31 Jan 2006 21
Enabling Grids for E-sciencE
INFSO-RI-508833
Figures
Small body
Medium body
Large body
Lungs
Dataset
87MB
210MB
346MB
87MB
Input data
3MB18KB/slice
9.6 MB25KB/slice
15MB22KB/sclice
410KB4KB/slice
Output data
6MB106KB/slice
57MB151KB/slice
86MB131KB/slice
2.3MB24KB/slice
Tasks
169
378
676
95
StandaloneExecution
5min15s1min54s
33min11min5s
18min
36s
EGEEExecution 14 procs.
37s18s
2min30s1min15s
2min03
24s
Grid Day Nis 31 Jan 2006 22
Enabling Grids for E-sciencE
INFSO-RI-508833
Example: The MAGIC Telescope
• Ground based Air Cerenkov Telescope
• Gamma ray: 30 GeV - TeV• LaPalma, Canary Islands
(28° North, 18° West)• 17 m diameter• operation since autumn 2003
(still in commissioning)• Collaborators: IFAE Barcelona, UAB Barcelona,
Humboldt U. Berlin, UC Davis, U. Lodz, UC Madrid, MPI München, INFN / U. Padova, U. Potchefstrom, INFN / U. Siena, Tuorla Observatory, INFN / U. Udine, U. Würzburg, Yerevan Physics Inst., ETH Zürich
Physics Goals: Origin of VHE Gamma raysActive Galactic NucleiSupernova RemnantsUnidentified EGRET sourcesGamma Ray Burst
Grid Day Nis 31 Jan 2006 23
Enabling Grids for E-sciencE
INFSO-RI-508833
~ 10 kmParticleshower
Ground based γ-ray astronomy
~ 1o
Ch
eren
kov
ligh
t
~ 120 m
Gammaray
GLAST (~ 1 m2)
Cherenkov light Image of particle shower in telescope camera
reconstruct: arrival direction, energyreject hadron background
Grid Day Nis 31 Jan 2006 24
Enabling Grids for E-sciencE
INFSO-RI-508833
MAGIC – Hadron rejection
• Based on extensive Monte Carlo Simulation– air shower simulation program CORSIKA– Simulation of hadronic background is very CPU consuming
to simulate the background of one night, 70 CPUs (P4 2GHz) needs to run 19200 days
to simulate the gamma events of one night for a Crab like source takes 288 days.
– At higher energies (> 70 GeV) observations are possible already by On-Off method (This reduces the On-time by a factor of two)
– Lowering the threshold of the MAGIC telescope requires new methods based on Monte Carlo Simulations
Grid Day Nis 31 Jan 2006 25
Enabling Grids for E-sciencE
INFSO-RI-508833
BG application in SEE-GRID VO - SALUTE
The Problem: ultra-fast semiconductor carrier transport
femtosecond relaxation of hot electrons by phonon emission in presence of electric field.
Barker-Ferry equation and Monte Carlo approach• Application in nanotechnologies: innovative results for
GaAs:
collision broadening and memory effects of quantum kinetic model;
Intra-collision field effect: quantum scattering - retarding and accelerating field.
• “NP-hard” problem concerning the evolution time• Parallel and Grid implementation
Grid Day Nis 31 Jan 2006 26
Enabling Grids for E-sciencE
INFSO-RI-508833
Wigner function
800 x 260 points
150 fs
Grid Day Nis 31 Jan 2006 27
Enabling Grids for E-sciencE
INFSO-RI-508833
Energy relaxation process:collisional broadening
Accumulation
From 10 fs up to 250 fs
Grid Day Nis 31 Jan 2006 28
Enabling Grids for E-sciencE
INFSO-RI-508833
BG application in ESR VO – air pollution prediction
• Under development by Tzvetan Ostromsky from IPP• Transition from HPC to Grid computing
Grid Day Nis 31 Jan 2006 29
Enabling Grids for E-sciencE
INFSO-RI-508833
Challenges before new sites
• Install middleware and follow security and middleware upgrades in a timely fashion
• Present valuable resource to the Virtual Organizations that the site supports
• Participate in the various challenges. So far we have seen the HEP and the Biomed VO challenges, and the security challenges
• Participate in innovation efforts – development of middleware and/or grid applications
• Attract new users• The Grid is about people
Grid Day Nis 31 Jan 2006 30
Enabling Grids for E-sciencE
INFSO-RI-508833
BG Grid support centre contact information
Contact persons: • Emanouil Atanassov,
SA1 Activity Leader,
• Aneta Karaivanova,
NA2 Activity Leader,
• Todor Gurov, Alternate EGEE SEE-ROC
and SEE-GRID manager, [email protected]
• Ivan Dimov, EGEE & SEE-GRID Project
manager for BG
•http://www.grid.bas.bg/