a grid research toolbox

April 19, 20231

A Grid Research Toolbox

Paris, France

A. Iosup, O. Sonmez, N. Yigitbasi, H. Mohamed, S. Anoep, D.H.J. EpemaPDS Group, ST/EWI, TU Delft

M. JanLRI/INRIA Futurs Paris, INRIA

DGSim

H. Li, L. WoltersLIACS, U. Leiden

I. Raicu, C. Dumitrescu, I. FosterU. Chicago

and Cloud

The FailureTraceArchive

April 19, 20232

A Layered View of the Grid World

• Layer 1: Hardware + OS• Automated• Non-grid (XtreemOS?)

• Layers 2-4: Grid Middleware Stack

• Low Level: file transfers, local resource allocation, etc.• High Level: grid scheduling• Very High Level: application environments (e.g., distributed objects)• Automated/user control• Simple to complex

• Layer 5: Grid Applications• User control• Simple to complex

HW + OS

Grid Low Level MW

Grid High Level MW

Grid Very High Level MW

Grid Applications

Gri

d M

W S

tack

April 19, 20233

Grid Work: Science or Engineering?

• Work on Grid Middleware and Applications

• When is work in grid computing science?• Studying systems to uncover their hidden laws• Designing innovative systems• Proposing novel algorithms• Methodological aspects:

repeatable experiments to verify and extend hypotheses

• When is work in grid computing engineering?• Showing that the system works in a common case, or

in a special case of great importance (e.g., weather prediction)

• When our students can do it (H. Casanova’s argument)

April 19, 20234

Grid Research Problem: We Are Missing Both Data and Tools

• Lack of data• Infrastructure

• number and type of resources, resource availability and failures

• Workloads• arrival process, resource consumption

• …

• Lack of tools• Simulators

• SimGrid, GridSim, MicroGrid, GangSim, OptorGrid, MONARC, …

• Testing tools that operate in real environments• DiPerF, QUAKE/FAIL-FCI

• …

We have problems to solve We have problems to solve in grid computing (as a in grid computing (as a sciencescience)!)!

April 19, 20235

Anecdote: Grids are far from being reliable job execution environments

• 99.999% reliableSmall Cluster

• 5x decrease in failure rate after first year [Schroeder and Gibson, DSN‘06]

Production Cluster

• >10% jobs fail [Iosup et al., CCGrid’06]DAS-2

• 99.99999% reliableServer

• 20-45% failures [Khalili et al., Grid’06]TeraGrid

• 27% failures, 5-10 retries [Dumitrescu et al., GCC’05]Grid3

CERN LCG jobsCERN LCG jobs74.71% successful74.71% successful

25.29% unsuccessful25.29% unsuccessful

Source: dboard-gr.cern.ch, May’07.

So at the moment our students cannot So at the moment our students cannot work in grid computing engineering!work in grid computing engineering!

April 19, 20236

The Anecdote at Scale

• NMI Build-and-Test Environment at U.Wisc.-Madison: 112 hosts, >40 platforms (e.g., X86-32/Solaris/5, X86-64/RH/9)

• Serves >50 grid middleware packages: Condor, Globus, VDT, gLite, GridFTP, RLS, NWS, INCA(-2), APST, NINF-G, BOINC …

Two years of functionality tests (‘04-‘06): Two years of functionality tests (‘04-‘06): over 1:3 runs have at least one failure!over 1:3 runs have at least one failure!

(1)(1) Test or perish!Test or perish!(2)(2) In today’s grids, reliability is In today’s grids, reliability is more important than performance!more important than performance!

A. Iosup, D.H.J.Epema, P. Couvares, A. Karp, M. Livny, Build-and-Test Workloads for Grid Middleware: Problem, Analysis, and Applications, CCGrid, 2007.

April 19, 20237

A Grid Research Toolbox

• Hypothesis: (a) is better than (b).

DGSim

1

2

3

For scenario 1, …

April 19, 20238

Research Questions

Q1: How to exchange grid/cloud data?Q1: How to exchange grid/cloud data?(e.g., Grid/Cloud * Archive)(e.g., Grid/Cloud * Archive)

Q2: What are the characteristics of grids/clouds?Q2: What are the characteristics of grids/clouds? (e.g., infrastructure, workload) (e.g., infrastructure, workload)

Q3: How to test and evaluate grids/clouds?Q3: How to test and evaluate grids/clouds?

April 19, 20239

Outline

1. Introduction and Motivation2. Q1: Exchange Data

1. The Grid Workloads Archive2. The Failure Trace Archive3. The Cloud Workloads Archive (?)

3. Q2: System Characteristics1. Grid Workloads2. Grid Infrastructure

4. Q3: System Testing and Evaluation

Traces in Distributed Systems Research• “My system/method/algorithm is better than yours

(on my carefully crafted workload)” • Unrealistic (trivial): Prove that “prioritize jobs from

users whose name starts with A” is a good scheduling policy

• Realistic? “85% jobs are short”; “10% Writes”; ...• Major problem in Computer Systems research

• Workload Trace = recording of real activity from a (real) system, often as a sequence of jobs / requests submitted by users for execution• Main use: compare and cross-validate new job and

resource management techniques and algorithms• Major problem: real workload traces from several

sourcesAugust 26, 2010

10

April 19, 202311

2.1. The Grid Workloads Archive [1/3]Content

6 traces online

http://gwa.ewi.tudelft.nlhttp://gwa.ewi.tudelft.nl

1.5 yrs >750K >250

A. Iosup, H. Li, M. Jan, S. Anoep, C. Dumitrescu, L. Wolters, D. Epema, The Grid Workloads Archive, FGCS 24, 672—686, 2008.

April 19, 202312

2.1. The Grid Workloads Archive [2/3]Approach: Standard Data Format (GWF)• Goals

• Provide a unitary format for Grid workloads;• Same format in plain text and relational DB (SQLite/SQL92);• To ease adoption, base on the Parallel Workloads Format

(SWF).

• Existing• Identification data: Job/User/Group/Application ID• Time and Status: Sub/Start/Finish Time, Job Status and Exit

code• Request vs. consumption: CPU/Wallclock/Mem

• Added• Job submission site• Job structure: bag-of-tasks, workflows• Extensions: co-allocation, reservations, others possible


April 19, 202313

2.1. The Grid Workloads Archive [3/3]Approach: GWF Example

SubmitWait[s]Run#CPUs Mem [KB]Used Req

#CPUs


April 19, 202314

2.2. The Failure Trace Archive

PresentationTypes of

systems• (Desktop) Grids• DNS servers• HPC Clusters• P2P systems

Stats• 25 traces• 100,000 nodes• Decades of

operation

The FailureTraceArchive

http://fta.inria.frhttp://fta.inria.fr

April 19, 202315

2.2. The Cloud Workloads Archive [1/2]

One Format Fits Them All

• Flat format• Job and Tasks• Summary (20 unique data fields) and Detail (60 fields)

• Categories of information• Shared with GWA, PWA: Time, Disk, Memory, Net• Jobs/Tasks that change resource consumption profile• MapReduce-specific (two-thirds data fields)

15

A. Iosup, R. Griffith, A. Konwinski, M. Zaharia, A. Ghodsi, I. Stoica, Data Format for the Cloud Workloads Archive, v.3, 13/07/10

CWJ CWJD CWT

CWTD

April 19, 202316

2.2. The Cloud Workloads Archive [2/2]

The Cloud Workloads Archive• Looking for invariants• Wr [%] ~40% Total IO, but absolute values

vary

• # Tasks/Job, ratio M:(M+R) Tasks, vary• Understanding workload evolution

Trace ID Total IO [MB]

Rd. [MB] Wr [%] HDFS Wr[MB]

CWA-01 10,934 6,805 38% 1,538

CWA-02 75,546 47,539 37% 8,563

April 19, 202317

Outline





April 19, 202318

3.1. Grid Workloads [1/7]

Analysis Summary: Grid workloads different, e.g., from parallel production envs. (HPC)• Traces: LCG, Grid3, TeraGrid, and DAS

• long traces (6+ months), active environments (500+K jobs per trace, 100s of users), >4 million jobs

• Analysis• System-wide, VO, group, user characteristics• Environment, user evolution• System performance

• Selected findings• Almost no parallel jobs• Top 2-5 groups/users dominate the workloads• Performance problems: high job wait time, high failure rates

A. Iosup, C. Dumitrescu, D.H.J. Epema, H. Li, L. Wolters, How are Real Grids Used? The Analysis of Four Grid Traces and Its Implications, Grid 2006.

April 19, 202319

3.1. Grid Workloads [2/7]Analysis Summary: Grids vs. Parallel Production Systems• Similar CPUTime/Year, 5x larger arrival bursts

GridsParallel Production

Environments (Large clusters,

supercomputers)

LCG clusterdaily peak: 22.5k jobs

A. Iosup, D.H.J. Epema, C. Franke, A. Papaspyrou, L. Schley, B. Song, R. Yahyapour, On Grid Performance Evaluation using Synthetic Workloads, JSSPP’06.

April 19, 202320

Bags-of-Tasks (BoTs)

3.1. Grid Workloads [3/7]More Analysis: Special Workload Components

BoT = set of jobs…

…that start at most Δs after the first job

Time [units]

Parameter Sweep App. = BoT with same binary

Workflows (WFs)

WF = set of jobs with precedence(think Direct Acyclic Graph)

April 19, 202321

• Selected Findings• Batches predominant in grid workloads; up to 96%

CPUTime

• Average batch size (Δ≤120s) is 15-30 (500 max)• 75% of the batches are sized 20 jobs or less


BoTs are predominant in grids

A. Iosup, M. Jan, O. Sonmez, and D.H.J. Epema, The Characteristics and Performance of Groups of Jobs in Grids, Euro-Par, LNCS, vol.4641, pp. 382-393, 2007.

Grid’5000 NorduGrid GLOW (Condor)

Submissions

26k 50k 13k

Jobs 808k (951k)

738k (781k) 205k (216k)

CPU time 193y (651y)

2192y (2443y)

53y (55y)

April 19, 202322

• Traces

• Selected Findings

• Loose coupling• Graph with 3-4 levels• Average WF size is 30/44 jobs• 75%+ WFs are sized 40 jobs or less, 95% are sized 200 jobs

or less


Workflows exist, but they seem small

S. Ostermann, A. Iosup, R. Prodan, D.H.J. Epema, and T. Fahringer. On the Characteristics of Grid Workflows, CoreGRID Integrated Research in Grid Computing (CGIW), 2008.

April 19, 202323

• Adapted to grids: percentage parallel jobs, other values.• Validated with 4 grid and 7 parallel production env. traces


Modeling Grid Workloads: Feitelson adapted

A. Iosup, D.H.J. Epema, T. Tannenbaum, M. Farrellee, and M. Livny. Inter-Operating Grids Through Delegated MatchMaking, ACM/IEEE Conference on High Performance Networking and Computing (SC), pp. 13-21, 2007.

April 19, 202324

• Single arrival process for both BoTs and parallel jobs• Reduce over-fitting and complexity of “Feitelson adapted”

by removing the RunTime-Parallelism correlated model• Validated with 7 grid workloads


Modeling Grid Workloads: adding users, BoTs

A. Iosup, O. Sonmez, S. Anoep, and D.H.J. Epema. The Performance of Bags-of-Tasks in Large-Scale Distributed Systems, HPDC, pp. 97-108, 2008.

April 19, 202325

3.2. Grid Infrastructure [1/5]

Existing resource models and data

• Compute Resources• Commodity clusters [Kee et al., SC’04]

• Desktop grids resource availability [Kondo et al., FCFS’07]

• Network Resources• Structural generators: GT-ITM [Zegura et al., 1997]

• Degree-based generators: BRITE [Medina et al., 2001]

• Storage Resources, other resources• ?

Source: H. CasanovaStatic! Static!

Resource dynamic, evolution, … Resource dynamic, evolution, … NOT consideredNOT considered

April 19, 202326

3.2. Grid Infrastructure [2/5]Resource dynamics in cluster-based grids• Environment: Grid’5000 traces

• jobs 05/2004-11/2006 (30 mo., 950K jobs)• resource availability traces 05/2005-11/2006 (18 mo., 600K

events)

• Resource availability model for multi-cluster grids

Grid-level availability: 70%

A. Iosup, M. Jan, O. Sonmez, and D.H.J. Epema, On the Dynamic Resource Availability in Grids, Grid 2007, Sep 2007.

April 19, 202327

• Correlated failure Maximal set of failures (ordered according to increasing event

time), of time parameter in which for any two successive failures E and F,

where returns the timestamp of the event; = 1-3600s.

3.2. Grid Infrastructure [3/5] Correlated Failures

• Grid-level view• Range: 1-339• Average: 11

• Cluster span• Range: 1-3• Average: 1.06• Failures “stay” within cluster Size of correlated failures

CD

F Average

Grid-level view


April 19, 202328

• Assume no correlation of failure occurrence between clusters

• Which site/cluster? • fs, fraction of failures at cluster s

MTBF MTTR Correl.

• Weibull distribution for IAT• Shape parameter > 1: increasing hazard rate

the longer a node is online, the higher the chances that it will fail


Dynamics Model


April 19, 202329


Evolution Model

A. Iosup, O. Sonmez, and D. Epema, DGSim: Comparing Grid Resource Management Architectures through Trace-Based Simulation, Euro-Par 2008.

April 19, 202330

• Grid workloads very different from those of other systems, e.g., parallel production envs. (large clusters, supercomputers)• Batches of jobs are predominant [Euro-Par’07,HPDC’08]• Almost no parallel jobs [Grid’06]• Workload model [SC’07, HPDC’08]• Clouds? (upcoming)

• Grid resources are not static• Resource dynamics model [Grid’07]• Resource evolution model [EuroPar’08]• Clouds? [CCGrid’11]

• Archives: easy to share traces and associated research

Q1,Q2: What are the characteristics of gridsQ1,Q2: What are the characteristics of grids (e.g., infrastructure, workload)? (e.g., infrastructure, workload)?

http://gwa.ewi.tudelft.nlhttp://gwa.ewi.tudelft.nl

April 19, 202331

Outline





April 19, 202332

4.1. GrenchMark: Testing in LSDCSsAnalyzing, Testing, and Comparing Systems

• Use cases for automatically analyzing, testing, and comparing systems (or middleware)• Functionality testing and system tuning• Performance testing/analysis of applications• Reliability testing of middleware• …

• For grids and clouds, this problem is difficult !• Testing in real environments is difficult/costly/both• Grids/clouds change rapidly• Validity and reproducibility of tests• …

April 19, 202333

4.1. GrenchMark: Testing LSDCSs Architecture Overview

GrenchMark = Grid Benchmark

April 19, 202335

4.1. GrenchMark: Testing LSDCSs Testing a Large-Scale Environment (1/2)

• Testing a 1500-processors Condor environment• Workloads of 1000 jobs, grouped by 2, 10, 20, 50, 100, 200• Test finishes 1h after the last submission• Results

• >150,000 jobs submitted• >100,000 jobs successfully run, >2 yr CPU time in 1

week• 5% jobs failed (much less than other grids’

average) • 25% jobs did not start in time and where cancelled

April 19, 202336

4.1. GrenchMark: Testing LSDCSs Testing a Large-Scale Environment (2/2)• Performance metrics

system-, job-, operational-, application-, and service-level

April 19, 202337

4.1. GrenchMark: Testing in LSDCSs ServMark: Scalable GrenchMark

• Blending DiPerF and GrenchMark.

• Tackles two orthogonal issues: • Multi-sourced testing

(multi-user scenarios, scalability)• Generate and run dynamic test

workloads with complex structure(real-world scenarios, flexibility)

• Adds• Coordination and automation

layers• Fault tolerance module

DiPerF

GrenchMark

ServMark

April 19, 202338

Performance Evaluation of Clouds [1/3]

C-Meter: Cloud-Oriented GrenchMark

Yigitbasi et al.: C-Meter: A Framework for Performance Analysis of Computing Clouds. Proc. of CCGRID 2009

April 19, 202339


Low Performance for Sci.Comp.

• Evaluated the performance of resources from four production, commercial clouds. • GrenchMark for evaluating the performance of cloud

resources• C-Meter for complex workloads

• Four production, commercial IaaS clouds: Amazon Elastic Compute Cloud (EC2), Mosso, Elastic Hosts, and GoGrid.

• Finding: cloud performance low for sci.comp.

S. Ostermann et al., A Performance Analysis of EC2 Cloud Computing Services for Scientific Computing, Cloudcomp 2009, LNICST 34, pp.115–131, 2010.

A. Iosup et al.,Performance Analysis of Cloud Computing Services for Many-Tasks Scientific Computing, IEEE TPDS, vol.22(6), 2011.

April 19, 202340


Cloud Performance Variability• Long-term performance variability of production cloud

services• IaaS:

Amazon Web Services• PaaS:

Google App Engine

• Year-long performance information for nine services• Finding: about half of the cloud services

investigated in this work exhibits yearly and daily patterns; impact of performance variability depends on application.A. Iosup, N. Yigitbasi, and D. Epema, On the Performance Variability of Production Cloud Services, CCGrid 2011.

Amazon S3: GET US HI operations

April 19, 202341

4.2. DGSim: Simulating Multi-Cluster GridsGoal and Challenges

• Simulate various grid resource management architectures• Multi-cluster grids• Grids of grids (THE grid)

• Challenges• Many types of architectures• Generating and replaying grid workloads• Management of simulations

• Many repetitions of a simulation for statistical relevance• Simulations with many parameters• Managing results (e.g., analysis tools)• Enabling collaborative experiments

Two GRM architectures

DGSim

April 19, 202342

4.2. DGSim: Simulating Multi-Cluster GridsOverview

Discrete-EventSimulator

DGSim

April 19, 202343

4.2. DGSim: Simulating Multi-Cluster GridsSimulated Architectures (Sep 2007)

Hybrid hierarchical/ decentralize

d

Decentralized

Hierarchical

Independent

Centralized

DGSimA. Iosup, D.H.J.Epema, T. Tannenbaum, M. Farrellee, M. Livny, Inter-Operating Grids through Delegated MatchMaking, SC, 2007.

April 19, 202344

Q3: How to test and evaluate grids/clouds?Q3: How to test and evaluate grids/clouds?

• GrenchMark+C-Meter: testing large-scale distrib. sys.• Framework• Testing in real environments performance, reliability,

functionality• Uniform process: metrics, workloads• Real tool available

• DGSim: simulating multi-cluster grids • Many types of architectures• Generating and replaying grid workloads• Management of the simulations

grenchmark.st.ewi.tudelft.nlgrenchmark.st.ewi.tudelft.nl

dev.globus.org/wiki/Incubator/ServMarkdev.globus.org/wiki/Incubator/ServMark

April 19, 202345

Take Home Message: Research ToolboxTake Home Message: Research Toolbox

• Understanding how real systems work• Modeling workloads and infrastructure• Compare grids and clouds with other platforms (parallel production

env.,…)

• The Archives: easy to share system traces and associated research • Grid Workloads Archive• Failure Trace Archive• Cloud Workloads Archive (upcoming)

• Testing/Evaluating Grids/Clouds• GrenchMark• ServMark: Scalable GrenchMark• C-Meter: Cloud-oriented GrenchMark• DGSim: Simulating Grids (and Clouds?)

Publications

2006: Grid, CCGrid, JSSPP

2007: SC, Grid, CCGrid, …

2008: HPDC, SC, Grid, …

2009: HPDC, CCGrid, …

2010: HPDC, CCGrid (Best Paper

Award), EuroPar, …

2011: IEEE TPDS, IEEE Internet

Computing, CCGrid, …

April 19, 202346

Thank you for your attention! Questions? Suggestions? Observations?

Alexandru Iosup

[email protected]://www.pds.ewi.tudelft.nl/~iosup/ (or google “iosup”)Parallel and Distributed Systems GroupDelft University of Technology

- http://www.st.ewi.tudelft.nl/~iosup/research.html

- http://www.st.ewi.tudelft.nl/~iosup/research_gaming.html

- http://www.st.ewi.tudelft.nl/~iosup/research_cloud.html

More Info:

Do not hesitate to contact me…

a grid research toolbox

Documents