scalarm: scalable platform for data farming d. król, Ł. dutka, m. wrzeszcz, b. kryza, r. słota...

15
Scalarm: Scalable Platform for Data Farming D. Król , Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013 1

Upload: matilda-williams

Post on 05-Jan-2016

216 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

Scalarm: Scalable Platform for Data FarmingD. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. KitowskiACC Cyfronet AGH

KU KDM, Zakopane, 2013

1

Page 2: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

Agenda

About Us

Problem statement

Solution

Contact

2

Page 3: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

Who are we ?

Computer Systems Group AGH http://www.icsr.agh.edu.pl/

Knowledge in Grids Team http://www.icsr.agh.edu.pl/index.php/knowledge-in-grids-team

Close collaboration with ACC Cyfronet AGH

3

Page 4: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

What are we doing ?

Knowledge supported systems for Virtual OrganizationsFramework for Intelligent Virtual OrganizationsGrid Organizational MemoryX2R

Data monitoring and managementService Level Agreement MonitoringQStorMan

Self-scalable systemsScalarm

4

Page 5: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

A short introduction to data farming 5

Process initializationProcess initialization

Page 6: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

What problem do we address with Scalarm ?

Data farming experiments with an exploratory approach

Parameter space generation with support of design of experiment methods

Accessing heterogeneous computational infrastructure

Self-scalability of the management part

6

Page 7: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

Typical work flow 7

1. Local development

2. A workstation or an institution server 3. National Grid environment

4. Cloud

Page 8: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

Where is the problem ? 8

1. Integrated DevelopmentEnvironment

2. Shell3. Shell + scheduler (gLite, QCG ...)

4. Shell + proprietary software

What about parameter

space ?

What about parameter

space ?

What about fault

tolerence?

What about fault

tolerence?

What about collecting results ?

What about collecting results ?

Page 9: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

What is our solution ? 9

Specify application to run

Execute simulation

Page 10: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

Scalarm overview 10

Self-scalable platform adapting to experiment size and simulation type

Exploratory approach for conducting experiments

Supporting online analysis of experiment partial results

Integrates with clusters, Grids, Clouds

Page 11: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

Use case – Complete platform for data farming

11

Page 12: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

Use case – Simulation-as-a-Service

12

API Execute simulationNative

app

Page 13: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

But, does it scale ? 13

Experiment evaluation:

Experiment size [#simulations]: 100 000, 200 000, 500 000, 1 000 000, 2 000 000

Computational resources [#servers]: 2, 4, 8, 16

#Clients: 240 * (#servers / 2)

Page 14: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

Results 14

Page 15: Scalarm: Scalable Platform for Data Farming D. Król, Ł. Dutka, M. Wrzeszcz, B. Kryza, R. Słota and J. Kitowski ACC Cyfronet AGH KU KDM, Zakopane, 2013

Do you want to now more ?

Talk to us during the conference

Contact us -> [email protected]

Visit a website -> http://www.scalarm.com

15