workload management status of current activity gridpp 13, durham, 6 th july 2005

Workload Management

Status of current activity

GridPP 13, Durham, 6th July 2005

5th July 2005 Workload Management

David Colling, Imperial College London

Activity…

• Scalability testing

• Analysis of current middleware performance

• SGE integration

• GridCC



Scalability Testing

People involved:

Janusz Martyniak, Luke Dikens, Barry MacEvoy, Steve McGough, David Colling



Scalability Testing

• From EDG we knew that it was easy to build a system capable of running 5 jobs concurrently.

• No so easy to build one capable of running 500 jobs or 5000 jobs concurrently.

• The plan was to perform testing to find software bottlenecks and hot spots • Feed the results back to the developers in a “virtuous circle”

Why…



Scalability Testing

The methodology …

• Original plan was to build a testbed across 2 sites (Imperial HEP and LeSC). This was deliverable X.Y

• Take an “engineering” approach. I.e. Submit tests to the testbed and monitor how the different components respond.

• Metrics to be tested to evolve in complexity as the stability grew.



Scalability Testing

What happened…• Decided to join the JRA1 testbed instead of forming

our own. This gave us better access to the developers and much support on other parts of the system that we were not directly testing but which are needed to run the tests e.g. VOMs, RGMA. Also thus made a contribution to wider community. This decision has been praised by Bob Jones and Frederic Hemmer.

• Still decided to run two sites (as per deliverable) as this gave a better testing environment for scalability tests



Scalability Testing

• What happenned …

• We were delayed by the late release of the WMS in EGEE

• However have had two sites in JRA1 testing since immediately after the Athens meeting. The two sites are maintained by JM and LD and they consist of:–Machines: 1 WMS

1 CE2 WNs1 RGMA Server.1 IO Server1 UI

Install: ManualConfig: Site (mostly)Version: R1.1

–Machines: 1 WMS2 CEs (+1)2 WNs (+1)

Install: aptConfig: SiteVersion: R1.1 (+ QF7&8)

Site 1 Site 2



Scalability Testing

To add to these sites…

• SEs • VOMS• Second RGMA server (to complete

split)



Scalability Testing

Actual testing…• Only really started writing scalability tests a couple

of weeks ago• Have defined some basic metrics

– Time to submit as a function of number of jobs for serial submission

– Time to submit for parallel submission– Failure rates as function of active jobs– etc

• Use LB database and system monitoring on WMS node to reconstruct what is going on



Scalability Testing

So, 100 simple jobs submitted sequentially…

• Result preliminary• Example of what we are trying to do• Bypassed known problems especially

cross matching• Summary…



Scalability Testing

28 Success53 Proxy expired (12 hours after the jobs were submitted !)3 Aborted due to reaching retry count16 Ready state

Summary…

In this sample greatest source of failure is CondorC



Scalability testing

Time to RegJob

0

5

10

15

20

25

Seconds from start

Nu

mb

er

of

en

trie

s

All registered in 3 minutes

100 jobs submitted sequentially



Scalability Testing

Time to EnQueued

1

10

100

1000

0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0M

ore

Time (seconds)

Nu

mb

er

of

en

trie

s

Greatest number <5000s(Excel binning)

Long tail of retries



Scalability Testing

Activity on WMS during submission

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

21:07:12 21:21:36 21:36:00 21:50:24 22:04:48 22:19:12 22:33:36

Time

Use

rCP

U u

sag

e

100 jobs submitted sequentially

5 Minutes Still activity 1 hour later

Can plot for individualor groups of processes



Scalability Testing

Future Plans…• Automate testing scripts

• Output directed to web-pages

• Expand metrics as appropriate



Performance of middleware

We access to the job data through the LB databases, so why not have a look?

• People involved Gidon Moont and David Colling




Long tail




Efficiency RunTime (s)

Nu

mb

er o

f en

tries



• Future plans…

• Keep monitoring this across different releases

• Low level activity • Feedback into JRA2




SGE Porting

People involved David McBride, Mona Aggarwal and Owen Maroney



SGE Porting

LCG Integration with Sun Grid Engine (SGE)

• Wish to add LCG as an additional entry point for our existing SGE cluster

• Problem: LCG installation assumes the use of PBS as the cluster management system.

• Solution: replace PBS-specific components with SGE specific components.



SGE PortingPBS-specific components

in LCG(That need replacing)• Globus JobManager

– Already have an existing alternative Globus JobManager for Sun Grid Engine to replace lcgpbs version.

– Implemented in Perl, well understood. – Supports 5.x, 6.x revisions of SGE.– Currently installed, about to enter the first

run of testing as part of an LCG CE installation.



SGE PortingPBS-specific Components in

LCG (That need replacing)

• Information Reporter– Have developed first-pass attempt at an

SGE information reporter. – Again, developed in Perl, small, relatively

straightforward. (Existing PBS code wasn't very clear, but GLUE Schema is public.)

– Installed on site CE, about to enter first run of validation and iterative improvement.



SGE PortingPBS-specific components

in LCG(That need replacing)• Accounting (APEL)

– APEL: Accounting using PBS Event Logs.– SGE does have advanced accounting records but are not

stored in the same format as PBS!– Existing Java-based tooling seems large and complex for

what should be a fairly straightforward task; not obvious where changes could/should be made.

– Refactored version exists in gLite, but would still require new implementation of SGE-specific backend.

– Using updated gLite revision on site may well work, but would introduce manageability issues at upgrade-time.

– Currently wondering whether APEL can simply be replaced with a small perl script(!) Currently looking up for documentation on the APEL/R-GMA reporting interface.



SGE Porting

Community of Interest formed

• Code available from:http://www.lesc.ic.ac.uk/projects/SGE-

LCG.html

• Mailing list

[email protected]

mailto:[email protected]



GridCC

People involved: Marko Krznaric, Janusz Martyniak, Luke

Dickens, John Darlington, Steve McGough, David McBride and David Colling

+ Tiziana & Costas



GridCC

Lot about GridCC at GridPP12 so brief update

• Discussions between GridCC and EGEE (Bob Jones and Frederic Hemmer)

• Agreed to collaborate (e.g. use EGEE CVS) GridCC relies on EGEE

• First release September this year

• Review October this year



GridCC

Bits in red from UK wms activity



Summary

• Activity in 4 areas – testing, – analysis, – SGE port, – GridCC

workload management status of current activity gridpp 13, durham, 6 th july 2005

Documents