EUROPEAN UNION

Status and Current Achievements Status and Current Achievements of PL-Grid Projectof PL-Grid Project

Jacek Kitowski and Łukasz DutkaACK CYFRONET AGH, Cracow, PolandInstitute of Computer Science AGH-UST

in collaboration withPL-Grid Representatives

Michał Turała, Kazimierz Wiatr, Marian Bubak, Tomasz Szepieniec,Marcin Radecki, Alex Kusznir, Zofia Mosurska, Mariusz Sterzel

Piotr Bała, Wojciech Wiślicki, Norbert Meyer, Krzysztof Kurowski, Józef Janyszek, Bartłomiej Balcerek, Jaroslaw Rybicki, Rafał Tylman

Cracow Grid Workshop, CGW 2009October 12-14, 2009

Polish InfrastructurePolish Infrastructurefor Supporting Computational Sciencefor Supporting Computational Science

in the European Research Spacein the European Research Space

National Grid Initiative in Poland in a nutshell Motivation Rationales and Foundations Status – Current Results

PL-Grid Project – Current Results Summary

OutlineOutline

E-Science approach to research Integration activities ongoing in the world

Demand from Community for e-Science ApproachDemand from Community for e-Science Approachin spite of complexity: Computing, Storage, Infrastructurein spite of complexity: Computing, Storage, Infrastructure

E-Science: collaborative research supported by advanced distributed computations Multi-disciplinary, Multi-Site and Multi-National Building with and demanding advances in Computing/Computer Sciences

Goal: to enable better research in all disciplines System-level Science: beyond individual phenomena, components interact

and interrelate, experiments in silico to generate, interpret

and analyse rich data resources• From experiments, observations

and simulations

• Quality management, preservation and reliable evidence

to develop and explore models and simulations• Computation and data at all scales

• Trustworthy, economic, timely and relevant results to enable dynamic distributed collaboration

• Facilitating collaboration with information and resource sharing

• Security, trust, reliability, accountability, manageability and agility

M. Atkinson, e-Science (...), Grid2006 & 2-nd Int.Conf.e-Social Science 2006, National e-Science Centre UKI. Foster, System Level Science and System Level Models, Snowmass, August 1-2, 2007

The Consortium consists of five High Performance Computing Polish Centres representing Communities due to:

Participation in International and National Projects and Collaboration• ~35 international projects FP5, FP6, FP7 on Grids (50% common)

• ~15 Polish projects (50% common)

Needs by Polish Scientific Communities• ~75% publications in 5 Communities

Computational resources to date• Top500 list

European/Worldwide integration Activities

• EGEE I-III, EGI_DS, EGI, e-IRG, PRACE, DEISA, OMII, EU Unit F3 „Research Infrastructure” Experts

National Network Infrastructure ready• (thanks to Pionier National Project)

Rationales behind PL-Grid ConsortiumRationales behind PL-Grid Consortium

GEANT2GEANT2

PL-Grid Foundations – SummaryPL-Grid Foundations – Summary

Motivation E-Science approach to research EGI initiative ongoing in collaboration with NGIs

Milestones: Creation of Polish Grid (PL-Grid) Consortium: http://plgrid.pl

Consortium Agreement signed in January 2007 PL-Grid Project (2009-2012)

Application in Operational Programme Innovative Economy, Activity 2.3 (in Sept. 2008) Get funded March 2, 2009 (via European Structural Funds)

Consortium made up of five largest Polish supercomputing and networking centres (founders) ACK CYFRONET AGH (Cracow) – Coordinator

Polish Infrastructure for Supporting Computational Science in the European Research Space

Response to the needs of Polish scientists and ongoing Grid activities in Poland, other European countries and all over the world

Grid infrastructure (Grid services) PL-Grid

App

licat

ion

App

licat

ion

App

licat

ion

App

licat

ion

Clusters High Performance Computers Data repositories

National Computer Network PIONIER

DomainGrid

Advanced Service Platforms

DomainGrid

DomainGrid

DomainGrid

Assumptions Polish Grid is developing a common base infrastructure – similar to solutions adopted in other

countries. Specialized, domain Grid systems – including services and tools focused on specific types of

applications – will be built upon this infrastructure. These domain Grid systems can be further developed and maintained in the framework of

separate projects. Such an approach should enable efficient use of available financial resources. Creation of a Grid infrastructure fully compatible and interoperable with European and

Worldwide Grids. Plans for HPC and Scalability Computing

enabled. Tighly-coupled activities

PL-Grid Base PointsPL-Grid Base Points

Elements and FunctionalityElements and Functionality

PL-Grid software will comprise: user tools (portals, systems for

applications management and monitoring, result visualization and other purposes, compatible with the lower-layer software used in PL-Grid);

software libraries; virtual organization systems: certificates,

accounting, security, dynamic ; data management systems: metadata

catalogues, replica management, file transfer;

resource management systems: job management, applications, grid services and infrastructure monitoring, license management, local resource management, monitoring.

Users

Nationalcomputernetwork

Grid Application

Programming Interface

Virtual organizations andsecurity systems

Basic Grid services

Gridservices

LCG/gLite(EGEE)

UNICORE(DEISA)

OtherGrids

systems

Gridresources

Distributedcomputational

resources

Grid portals, development tools

Distributed data

repositories

Three Grid structures will be maintained: production, reseach, development/testing.

Operational Center: TasksOperational Center: Tasks

EGIGroup

EGI Testing

Middleware

EGI ProductionMiddleware

Coordination of Operation Management and accounting EGI and DEISA collaboration

(Scalability and HPC Computing) Users’ requirements analysis for

operational issues Running infrastructure for:

Production Developers Research

Future consideration:

Computational Cloud Data Cloud Internal and External

Clouds Virtualization aspects

Operational Center: Current ResultsOperational Center: Current Results

PL-Grid VO (vo.plgrid.pl) fully operational Virtual production resources for PL-Grid users in

4 centres (Gdańsk, Kraków, Poznań, Wrocław) Infrastructure monitoring (failure detection) tools

based on Nagios & EGEE SAM in place Support teams, tools and EGEE/EGI-compliant

operations procedures aimed at achieving high availability of resources established

Knowledge sharing tool for operations problems Problem tracking tool to be released in November Provisional user registration procedure User registration portal in preparation

(1st prototype to be released end of October) Accounting integrated with EGEE

PCSS

Cyfronet

WCSS

Operational Center: NextOperational Center: Next PPlanslans

• User Registration– access to any person registered as researcher in Poland -

www.opi.org.pl– portal being already developed (to be released end of October'09)– access to resources based on

• grants – giving rights to use X CPUhours and TB of disk space

• access services – way of accessing resources in particular computing centre

• PL-Grid Research Infrastructure – “resources on demand” allows to request machines for different purposes– install a grid middleware for testing purposes– for developers to build a testing environment– any other research requiring grid resources– procedures are being defined

• Enable access to UNICORE resources – small amount of dedicated UNICORE machines– integrate UNICORE and gLite – make use of the same resources

Software and Tools Development: In nSoftware and Tools Development: In numbersumbers

Research developers: 20+Main research teams: 8Supercomputing centers

involved : 5Dedicated physical and

virtual machines: 30+Tested infrastructures:

EGEE, DEISANew infrastructures and

extensions: QosCosGrid, GridSpace

Current applications under integration and testing procedures: 10+

Current user tools under integration and testing procedures: 5+

Active links to new user communities in Poland: www.plgrid.pl/ankieta

Software and Tools Development: Next PSoftware and Tools Development: Next Planslans

Closer collaboration with user and developer communities representing various scientific domains (based on feedback received from our questionnaire)

Adaptation of virtualization techniques improving usability, fault tolerance, checkpointing and cluster management

Fancy web GUIs based on Liferay and Vine Toolkit accessing existing and extended e-Infrastructure services

Performance tests of various large scale cross-cluster parallel applications using co-allocation and advance reservation techniques

More analysis of grid management, monitoring and security solutions Common software repository and deployment rules Example services, tools and applications available as virtual boxes with different

configurations Main checkpoints and timeline:

IX 09VI 09 IX 10 IV 11 I 12

We are here

Users’ Support and TrainingUsers’ Support and Training

General task: Running Help-Desk and users’ training and support tasks Supporting how to effectively exploit developed infrastructure including

selection of right applications for users’ problems and seamless running computation on the grid.

Pl-grid is an user oriented infrastructure, support and training team will be close to the users.

At the moment: Support Team is organized Internal training for support team already begun Selection of licensed application required by the users’ community to be

installed on the whole PL-Grid infrastructure First training for the users delivered.

Plans: Many, many trainings for beginners and advanced grid users

PL-Grid Security PL-Grid Security Group:Group: Current ResultsCurrent Results

Defined security guidelines for architecture planning Defined security policy for local sites installations Incident reporting systems review to choose best solution for PL-

Grid GGUS -  Global Grid User Support RTIR - Request Tracking for Incident Response DIHS - Distributed Incident Handling System

Architecture and prototype version of on-line user mapping and credentials distribution system.

PL-Grid Security Group: Next PlansPL-Grid Security Group: Next Plans

Establish NGI-wide CERT Introduce security policies into local sites Develop and deploy local sites security policy conformity monitoring

system check if software (kernel, services) is up to date check listening ports, firewall rules monitor suid/sgid lists and integrity

Develop distributed alert correlation system build on top of network and host sensors.

Summary – ActivitiesSummary – Activities

Short term Establish PL-Grid VO using Partners’ local and EGEE resources Provide resources for covering operational costs Provide resources for keeping international collaboration ongoing

Long term – continuously Software and tools implementation Users’ support and traning Provide, keep and extend the necessary infrastructure Be prepared / work on approaching paradigms and integration development

• HPCaaS and Scalable Computing (Capability and Capacity Computing)

• Clouds Computing (internal-external, computing clouds, data clouds)

• SOA paradigm, knowledge usage …

• „Future Internet” as defined by EC in Workprogramme

http:plgrid.plhttp:plgrid.pl