The Performance of the Czech National GridInfrastructure after Major Reconfiguration of Job

Scheduling System

Dalibor Klusácek Šimon Tóth

Faculty of Informatics, Masaryk University

October 27, 2014

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 1 / 25

Motivation The Czech National Grid

The Czech National Grid

Praha

Liberec

Pardubice

Brno

Olomouc

Ostrava

NIXAMS-IX

SANETACONET

PIONIER

Jihlava

Děčín

Plzeň Karviná

Zlín

České Budějovice

Hradec Králové

CESNETCharles

University

University of West Bohemia

MU Loschmidt Laboratories

University ofSouth Bohemia

CEITEC Masaryk University

CERIT-SC

Institute of Physics ASCR

GÉANT

Inst. of Experiment. Botany ASCR

CERIT-SC

CESNET

Masaryk UniversityFaculty of Informatics

CESNET

CESNET

CESNET

Institute of Org. Chem. and Biochem. ASCR

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 2 / 25

Motivation The Czech National Grid

The Czech National Grid

10 000 CPU cores12GB..6TB RAM per-node1PB permanent storage27PB long-term storage

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 3 / 25

Motivation Motivation

Motivation

workloads evolve throughout timesmall changes no longer sufficientthree major shifts:

1 memory heavy workloads2 increased average job length3 increasingly frequent wide jobs

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 4 / 25

Motivation Motivation

Original Configuration

designed by experts to fit original workloadfull support for very long jobs (up to 30 days)fair allocation of resources across users (long-term)fair wait times across users

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 5 / 25

Reconfiguration Dominant resource fairness

Memory heavy workloads

fairshare fairness modelonly CPU accounted

users with CPU heavy workloads penalizedexploited by some of our users

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 6 / 25

Reconfiguration Dominant resource fairness

Multi-resource fairness model

based on dominant resource model40% of users penalized

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 7 / 25

Reconfiguration Queue reconfiguration

Increase in average job length

originally unclear issueusers experienced long wait timessystem experienced low utilization

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 8 / 25

Reconfiguration Queue reconfiguration

Original Configuration

long (24 hours..30 days)

short (< 6 hours)

normal (6 hours..24 hours)

backfill (< 24 hours)

pro

cess

ing o

rder

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 9 / 25

Reconfiguration Queue reconfiguration

Original Configuration

95% resources, 250 running jobs per-user

100% resources, 300 running jobs per-user

80% resources, single-node1000 running jobs per-user

long (24 hours..30 days)

short (< 6 hours)

normal (6 hours..24 hours)

backfill (< 24 hours)

60% resources, 70 running jobs per-userp

roce

ssin

g o

rder

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 10 / 25

Reconfiguration Queue reconfiguration

Analysis - job arrivals by queue

0

5000

10000

15000

20000

25000

1.1.13

8.1.13

15.1.13

22.1.13

29.1.13

5.2.13

12.2.13

19.2.13

26.2.13

5.3.13

12.3.13

19.3.13

26.3.13

shortlong

backfillnormal

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 11 / 25

Reconfiguration Queue reconfiguration

Analysis - CPU time by queue

200

400

600

800

1000

1200

1400

1600

1800

2000

2200

01.01.13

08.01.13

15.01.13

22.01.13

29.01.13

05.02.13

12.02.13

19.02.13

26.02.13

05.03.13

12.03.13

19.03.13

26.03.13

shortlong

backfillnormal

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 12 / 25

Reconfiguration Queue reconfiguration

Analysis - job runtime in long queue

0

1000

2000

3000

4000

5000

6000

7000

0..1 day 1..2 days 2..4 days 4..7 days 7..14 days 14..30 days

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 13 / 25

Reconfiguration Queue reconfiguration

Goals for new configuration

increase resource pool for mid-range jobs (2-4 days)decrease resource pool for very long jobs (1 week +)

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 14 / 25

Reconfiguration Queue reconfiguration

Simulation

Alea Simulatorhttp://www.fi.muni.cz/~xklusac/alea/

complex job specificationmulti-queue configurations including operational limitshigh-resolution output

historical 5-month workloads

376 722 jobs302 users

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 15 / 25

Reconfiguration Queue reconfiguration

Simulation results

26.5% users with improved average wait time19.2% users with deteriorated average wait timeaverage improvement of wait time by 6.7 hoursaverage deterioration of wait time by 55.7 hours

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 16 / 25

Reconfiguration Queue reconfiguration

Analysis

impact of fairness policy heavily mitigatedfairshare ordering policy overridden by hard priorities

impact of anti-starvation policy amplifiedreservations blocking most resources

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 17 / 25

Reconfiguration Queue reconfiguration

Experimental configuration

100% resc.

2 weeks +

1 day

20% resc. 20% resc.

2 weeks

20% resc.

1 week

2 hours

100% resc. 100% resc.

4 hours

30% resc.

4 days

30% resc.

2 days

virtual fairshare queue

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 18 / 25

Reconfiguration Queue reconfiguration

Simulation results

31.1% users with improved average wait time13.9% users with deteriorated average wait timeaverage improvement of wait time by 7.2 hoursaverage deterioration of wait time by 2.1 hours

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 19 / 25

Reconfiguration Optimizing scheduler

Increasingly frequent wide jobs

parallel jobs are problematic in generalfreeing up resources can cause gaps in utilizationproblems with runtime estimates

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 20 / 25

Reconfiguration Optimizing scheduler

Planning & optimizing scheduler

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 21 / 25

Reconfiguration Optimizing scheduler

Planning & optimizing scheduler

queue-less schedulermaintains a stable scheduleschedule continuously optimized and adjusted

new jobs arrivalpremature job completion

currently managing 45% of grid resources

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 22 / 25

Conclusion Results

Results

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 23 / 25

Conclusion Results

Conclusion

targeting fairness improved overall system performanceuser satisfaction improved as wellrequired relaxation of some policies (e.g. anti-starvation)precise simulation tools are critical

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 24 / 25

Conclusion Results

Summary

multi-resource fairnessnew queue configurationoptimizing scheduler

(Masaryk University) Czech NGI Performance Evaluation October 27, 2014 25 / 25