custom split-merge support in a hierarchical gridrpc middleware

. . . . . .

Custom Split-Merge Support in a Hierarchical GridRPCMiddleware

Cristian KLEIN, Christian PÉREZ

Avalon / GRAAL, INRIA / LIP, ENS de Lyon

COOP Meeting, 11-12 April 2011, Rennes

Cristian KLEIN (INRIA) DIET Split-Merge COOP, Rennes 1 / 14

. . . . . .

DIET (Rappel)

.What is DIET?..

.. ..

.

.

Distributed Interactive Engineering ToolkitIn the context of COOP: Resource Management System

.Features..

.. ..

.

.

Hierarchical schedulingData managementClient-side schedulingData-flow scheduling


. . . . . .

GridTLSE Use-Cases.Solving AX = B..

.. ..

.

.

A︸︷︷︸m×n

X︸︷︷︸n×p

= B︸︷︷︸m×p

Columns of B can be solved independentlyAX = B1︸︷︷︸

m×p1

AX = B2︸︷︷︸m×p2

· · ·

How to choose p1, p2, . . .?


. . . . . .


.. ..

.

.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

1 10 100 1000Tim

e p

er c

olu

mn

(s)

Number of columns solved simultaneously

Solving AX=B on a single host

Example: let ncols = 100 andsolve 1 column/requestsolve 10 columns/requestsolve 20 columns/request

Problem: How to generate resource-dependent requests in DIET?


. . . . . .


.. ..

.

.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

1 10 100 1000Tim

e p

er c

olu

mn

(s)



Example: let ncols = 100 and multiple hostssolve 1 column/requestsolve 10 columns/requestsolve 20 columns/request



. . . . . .


.. ..

.

.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

1 10 100 1000Tim

e p

er c

olu

mn

(s)



Example: let ncols = 100 and nH = 10solve 1 column/requestsolve 10 columns/requestsolve 20 columns/request



. . . . . .


.. ..

.

.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

1 10 100 1000Tim

e p

er c

olu

mn

(s)



Example: let ncols = 100 and nH = 10solve 1 column/request 10 requests / host t = 7.50 ssolve 10 columns/requestsolve 20 columns/request



. . . . . .


.. ..

.

.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

1 10 100 1000Tim

e p

er c

olu

mn

(s)



Example: let ncols = 100 and nH = 10solve 1 column/request 10 requests / host t = 7.50 ssolve 10 columns/request 1 request / host t = 2.15 ssolve 20 columns/request



. . . . . .


.. ..

.

.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

1 10 100 1000Tim

e p

er c

olu

mn

(s)



Example: let ncols = 100 and nH = 10solve 1 column/request 10 requests / host t = 7.50 ssolve 10 columns/request 1 request / host t = 2.15 ssolve 20 columns/request 5 requests t = 5.00 s



. . . . . .

.. .1 Introduction

.. .2 Solution: Split-Merge Support

.. .3 Example

.. .4 Conclusion


. . . . . .

.. .1 Introduction


.. .3 Example

.. .4 Conclusion


. . . . . .

DIET w/ Client-Side Scheduling


. . . . . .

DIET w/ Client-Side Split-Merge


. . . . . .

.. .1 Introduction


.. .3 Example

.. .4 Conclusion


. . . . . .

Example: Computing the Sum of an Array

.Problem..

.. ..

.

.

“Efficiently” computing the (associative) sum of a given arrayOn heterogeneous resources

.Solution..

.. ..

.

.

SeDs send relative computation power in their estimation vectorsplit() assigns each SeD a sub-vector→ proportional to relative computation power

merge() sums the partial results


. . . . . .

bool CustomSM::split(diet_profile_t *profile, corba_response_t *response,SubRequests &subrequests)

{if (strcmp(profile->pb_name, "sum_vector") != 0) return false;

/* Get original vector */long *vector;size_t vectorLength;diet_vector_get(diet_parameter(profile, 0), &vector, NULL, &vectorLength);

/* Compute total computation power */int numDiscoveredSeDs = response->servers.length();double totalPower = 0;for (int i = 0; i < numDiscoveredSeDs; i++) {

double thisPower = diet_est_get(&response->servers[i].estim, ESTTAG_RELPOWER, 0);totalPower += thisPower;

}

/* Scatter subrequests */size_t subVectorStart = 0;for (int i = 0; i < numDiscoveredSeDs; i++) {

double thisPower = diet_est_get(&response->servers[i].estim, ESTTAG_RELPOWER, 0);char *SeDName = response->servers[i].loc.SeDName;

long subVectorLength = vectorLength * thisPower / totalPower;diet_profile_t* subprofile = diet_profile_alloc(profile->pb_name, 0, 0, 1);diet_vector_set(diet_parameter(subprofile, 0), &vector[subVectorStart]);subVectorStart += subVectorLength;

subrequests.push_back(SubRequests::value_type(subprofile, SeDName));}

return true;}


. . . . . .

bool CustomSM::merge(diet_profile_t *profile, corba_response_t *response,SubRequests &subrequests)

{/* Gather responses */long *out = new long(0);for (SubRequests::iterator it = subrequests.begin();

it != subrequests.end(); it++) {long *subOutValue;

diet_profile_t *subprofile = it->first;diet_scalar_get(diet_parameter(subprofile, 1), &subOutValue, NULL);diet_profile_free(subprofile);

*out += *subOutValue;}diet_scalar_set( diet_parameter( profile, 1 ), out, DIET_VOLATILE, DIET_LONGINT );

return true;}

int main(){

...CustomSM clientScheduler;SpecificClientScheduler::setScheduler(&clientScheduler);...

}


. . . . . .

.. .1 Introduction


.. .3 Example

.. .4 Conclusion


. . . . . .

Conclusion

.DIET Client Split-Merge Support..

.. ..

.

.

We extended the DIET’s client-side schedulingApplication can easily hook their custom split-mergeAllows to adapt DIET requests to available resources(even in the heterogeneous case)

.Future Work..

.. ..

.

.

Apply to GridTLSE use-cases:I What information from SeDs? What algorithm?

More tests, experiments, . . .Study split-merge support in DIET data-flow engine


custom split-merge support in a hierarchical gridrpc middleware

Documents