![Page 1: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/1.jpg)
29th October 2008
Systematic Cooperation in P2P Grids
Cyril Briquet
Doctoral Dissertation in Computing ScienceDepartment of EE & CS (Montefiore Institute)
University of Liège, Belgium
![Page 2: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/2.jpg)
2
Application class: Bags of Tasks
Systematic Cooperation in P2P Grids
● Bag of Task = set of independent computational Tasks
many domains:
● bioinformatics
● computer vision
● data mining
● distributed discrete-event simulation
● GIS, spatial indexing
● medical image processing (tomography)
● protein folding & docking
● search engine crawling & indexation
![Page 3: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/3.jpg)
3
Application class: Iterative Stencil
Systematic Cooperation in P2P Grids
● Iterative Stencil = inter-communicating computational Tasks,
with iterative computations (sync. points)
● system speed = slowest Task
=> load balancing required
● failure of any Task = restart everything, from the start =>
uninterrupted co-allocation required
● typical domains: CFD, electromagnetics
![Page 4: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/4.jpg)
4
Human users + computational Tasks +no money for expensive infrastructure +limited number of desktop computers = ???
Systematic Cooperation in P2P Grids
cluster computing
desktop computing
volunteer computing
Grid computing
● sharing of computing time
● separate organizations
● + fully decentralized and automated... => P2P Grid computing
![Page 5: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/5.jpg)
5
P2P Grids operate in an environment too dynamic for most human users
Systematic Cooperation in P2P Grids
human users and administrators do expectshort response times and a simple interface
complexity of the P2P Grid should be hidden
dynamic peeringrelationships
opportunistic use ofadditional worker nodes
graceful recoveryas worker nodesbecome unavailable
![Page 6: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/6.jpg)
6
Application model = Bag of TasksGrid model = Peer-to-Peer (2-levels)
Systematic Cooperation in P2P Grids
Resource = worker node
(desktop computer)
Peer = controller
(no privileged role,
opaque to other Peers)
![Page 7: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/7.jpg)
7
2 options to run Tasks
Systematic Cooperation in P2P Grids
● send the Task to
one local Resource
● (at peak) submit the Task
to another (supplier) Peer
![Page 8: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/8.jpg)
8
Task execution failures are frequent due to preemption
Systematic Cooperation in P2P Grids
local use => preemption or cancellation => Task execution failure
![Page 9: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/9.jpg)
9
Thesis objectives
Systematic Cooperation in P2P Grids
![Page 10: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/10.jpg)
10
Thesis statement
Systematic Cooperation in P2P Grids
Lightweight Bartering Grid (LBG) middleware
![Page 11: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/11.jpg)
11
Contents
Systematic Cooperation in P2P Grids
● Context & Thesis statement● Scheduling Tasks● Transferring large input data files● Engineering P2P Grid software● Running heavily-communicating Tasks● Conclusion
![Page 12: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/12.jpg)
12
Q: always reciprocate supplying?
Systematic Cooperation in P2P Grids
![Page 13: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/13.jpg)
13
Take what you need,give what you do not need
Systematic Cooperation in P2P Grids
● Network of Favors model (state-of-the-art)
● explains: when to supply, to which Peers
● mitigates free riding
● basic behavior: always supply computing time of idle Resources
even if no (recent) reciprocal consumption
● if several consumers want access to a Resource:
supply to the Peer towards which most indebted
![Page 14: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/14.jpg)
14
Each Peer tracks its own Grid usage
Systematic Cooperation in P2P Grids
● Network of Favors = mechanism for fully decentralized bartering
● each Peer maintains its own accounting of
« debts » of computing time, with each neighbor Peer
![Page 15: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/15.jpg)
15
Bartering based on Network of Favors
Systematic Cooperation in P2P Grids
● no guarantees, but opportunities of sharing when possible
● fully decentralized
● preserves informational opacity between Peers
● can be deployed today (no central banking component)
● existing P2P Grids:
cannot hide Task execution failures to consumer Peers,
because there is no queueing support for Supplying Tasks
![Page 16: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/16.jpg)
16
Scheduling model
Systematic Cooperation in P2P Grids
computations organized (Peer-level) around 2 Task queues:
several “policy decision points” control the flow of Tasks
![Page 17: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/17.jpg)
17
Fault-management classification
Systematic Cooperation in P2P Grids
● fault-tolerance: gracefully adapt to faults after they happened
● fault-avoidance: avoid unreliable Peers
(as a consumer)
● fault-prevention: avoid to cause faults to Tasks of other Peers
(as a supplier)
![Page 18: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/18.jpg)
18
Fault-tolerance mechanisms
Systematic Cooperation in P2P Grids
![Page 19: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/19.jpg)
19
Fault-avoidance mechanisms
Systematic Cooperation in P2P Grids
![Page 20: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/20.jpg)
20
Fault-prevention mechanisms
Systematic Cooperation in P2P Grids
![Page 21: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/21.jpg)
21
Adaptive preemption and cancellation
Systematic Cooperation in P2P Grids
behavior of a supplier Peer at peak, for fault-prevention:
● select for preemption the most recently scheduled Tasks
i.e. who would “suffer” least (PSufferage heuristic)
● mask (preempt) or communicate (cancel) Task execution failure
(cancellation lets consumer select another supplier)
● offer 2nd chance to long-running Tasks,
with a short grace period
![Page 22: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/22.jpg)
22
Contents
Systematic Cooperation in P2P Grids
● Context & Thesis statement● Scheduling Tasks● Transferring large input data files● Engineering P2P Grid software● Running heavily-communicating Tasks● Conclusion
![Page 23: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/23.jpg)
23
Data transfers delay response times
Systematic Cooperation in P2P Grids
● some Bags of Tasks process a large number of large files
e.g. maps
● ... even implicitly
e.g. so-called parameter sweeps
=>
● exploit (temporal, spatial) redundancy between data files
to prevent unnecessary transfer costs
![Page 24: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/24.jpg)
24
Centralized data transfers do not scale
Systematic Cooperation in P2P Grids
![Page 25: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/25.jpg)
25
P2P data transfers (e.g. BitTorrent) exploit orthogonal bandwidth
Systematic Cooperation in P2P Grids
load spread between downloaders => reduced load on data source
supplementary
network links involved
time (N transfers of 1 file) ~ time (1 transfer 1 file)
![Page 26: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/26.jpg)
26
Decentralized data transfer architecture
Systematic Cooperation in P2P Grids
BitTorrent Nodes (= Grid Peers + Resources) exchange data
files transferred
with FTP
if < 50 MB
or # < 2
each Grid Peer runs its own BitTorrent tracker
![Page 27: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/27.jpg)
27
Exploiting Temporal Data Redundancy
Systematic Cooperation in P2P Grids
● Tasks with identical data files scheduled together
(as simultaneously as possible)
● simultaneous transfers are initiated on demand (!)
... to maximize BitTorrent efficiency
![Page 28: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/28.jpg)
28
P2P data transfers not always possible
Systematic Cooperation in P2P Grids
● it may not be possible to schedule concurrently
Tasks depending on identical data files
(e.g. not enough Resources simultaneously available)
● some data files may be required
by multiple Bags of Tasks spread over time
![Page 29: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/29.jpg)
29
Exploiting Spatial Data Redundancy
Systematic Cooperation in P2P Grids
● reuse data files to prevent unnecessary data transfers
distributed caching mechanism (each Resource)
distributed data tracking mechanism (each Peer)
known for its Resources
expected for recent suppliers
● data-aware scheduling to Resources, suppliers
![Page 30: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/30.jpg)
30
256 MB file, 25x4 Tasks, 24 ResourcesBitTorrent vs. FTP, TTG vs. FIFO
Systematic Cooperation in P2P Grids
![Page 31: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/31.jpg)
31
256 MB file, 48 Tasks, 24 Res., BitTorrentvariable redundancy, TTG vs. FIFO
Systematic Cooperation in P2P Grids
![Page 32: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/32.jpg)
32
Implicitly Exploiting Temporal Data Redundancy
Systematic Cooperation in P2P Grids
● each Resource shares data files with BitTorrent
even after they are not required anymore
● side effect of distributed caching:
supplementary number sharing sources
=>
implicit Temporal Tasks Grouping
=>
load removed from the data source with BitTorrent
![Page 33: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/33.jpg)
33
Summary of data redundancy exploitation
Systematic Cooperation in P2P Grids
● BitTorrent (Temporal Task Grouping)
if parallel execution & data transfer both possible
● distributed caching + data-awareness (Spatial Task Grouping)
if parallel execution not possible &
if data available on idle Resources
● BitTorrent + distributed caching (implicit Temporal Task Grouping)
if parallel execution not possible &
if data not available on idle Res. (i.e. available on busy Res.)
![Page 34: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/34.jpg)
34
Contents
Systematic Cooperation in P2P Grids
● Context & Thesis statement● Scheduling Tasks● Transferring large input data files● Engineering P2P Grid software● Running heavily-communicating Tasks● Conclusion
![Page 35: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/35.jpg)
35
Testing P2P Grid software is complex
Systematic Cooperation in P2P Grids
● multiple sources of bugs: large software, scheduling algorithms,
state consistency, network, code execution,
multithreading, data transfers, ...
● difficult to set a P2P Grid into a given state
because P2P Grid = complex, non-dedicated, distributed
● virtualization of messaging
=>
virtualized execution in a controlled environment
![Page 36: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/36.jpg)
36
Virtualization alone is not scalable
Systematic Cooperation in P2P Grids
● 24 hours of virtualized execution = 24 hours
... not temporally-scalable (i.e. execution occurs in real time)
● also virtualize time-consuming operations
i.e. simulate Task execution, timers, multithreading
● discrete-event simulation can enable reproducible evaluations
... but simulation accuracy often limited
![Page 37: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/37.jpg)
37
Code once, deploy twice (Grid Reality And Simulation, M. Quinson)
Systematic Cooperation in P2P Grids
● idea: virtualization + simulation = software engineering tool
● STEP 1: completely virtualize Grid nodes at middleware-level,
i.e. Virtual Machine (e.g. Xen, VMWare), O.S.-level emulation
● STEP 2: then weave simulator code with scheduling algorithms
● massive code reuse between implementations:
first, top-down application of code once, deploy twice
to a complete middleware
![Page 38: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/38.jpg)
38
Communications in the middleware
Systematic Cooperation in P2P Grids
![Page 39: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/39.jpg)
39
Communications in the simulator
Systematic Cooperation in P2P Grids
![Page 40: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/40.jpg)
40
Simulator overview
Systematic Cooperation in P2P Grids
simulation language
input file:
● Grid topology
● synthetic workload
● Peers configuration
output file:
● execution stats
![Page 41: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/41.jpg)
41
Reproducible testing
Systematic Cooperation in P2P Grids
practical benefits:
● issues with live deployment replayed in the simulator
● most of the code tested before going live, at high speed
● simulated algorithms deployed as-is in the middleware
● large-scale parameter sweeps of scheduling policies
![Page 42: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/42.jpg)
42
Self-Bootstrapping
Systematic Cooperation in P2P Grids
● self-bootstrapping = current, stable version of a given system
used to develop next version
● Bag of SimTasks (N simulators embedded into Grid Tasks)
● 1 middleware:
basic policies
● N simulators (SimTasks):
test and evaluate
advanced policies
![Page 43: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/43.jpg)
43
Contents
Systematic Cooperation in P2P Grids
● Context & Thesis statement● Scheduling Tasks● Transferring large input data files● Engineering P2P Grid software● Running heavily-communicating Tasks● Conclusion
![Page 44: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/44.jpg)
44
LaBo GridLattice-Boltzmann computations on a Grid
Systematic Cooperation in P2P Grids
G. Dethier's research,
with Chemical Engineering dept.:
Computational Fluid Dynamics
simulation of flows on a lattice
with Lattice-Boltzmann method
Iterative Stencil application Figure courtesy of Gérard Dethier
![Page 45: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/45.jpg)
45
LBG-SQUARE = LBG x LBG(Lattice-Boltzmann on the Grid x Lightweight Bartering Grid)
Systematic Cooperation in P2P Grids
LaBo Grid LBG-SQUARE
(currently centralized) load balancer
![Page 46: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/46.jpg)
46
Locality-aware co-allocation
Systematic Cooperation in P2P Grids
● how to balance load in a P2P Grid?
● dynamic large-scale co-allocation
● ... thus no advance knowledge of Task schedule
=> no way to mold Tasks before deployment
● load balancing in LaBo Grid performed after scheduling:
dynamic benchmarks
(Gérard Dethier's work on adaption to CPU and network cap.)
=> co-allocation by P2P Grid, locality-awareness by LaBoGrid
![Page 47: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/47.jpg)
47
Fault-tolerance with checkpoint-restart
Systematic Cooperation in P2P Grids
● challenge: decentralized architecture for scalability
=> P2P checkpointing and fault recovery
● distributed checkpoint storage, transfer and reload
(i.e. no centralized checkpoint server)
● nominal operations, checkpoint reload = decentralized
● load (re)balancing = (currently) centralized
● challenge: bursts of Task execution failures (preemption)
=> P2P-aware checkpoint storage
i.e. checkpoints of 1 Task spread to different Peers
![Page 48: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/48.jpg)
48
Contents
Systematic Cooperation in P2P Grids
● Context & Thesis statement● Scheduling Tasks● Transferring large input data files● Engineering P2P Grid software● Running heavily-communicating Tasks● Conclusion
![Page 49: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/49.jpg)
49
Contributions
Systematic Cooperation in P2P Grids
● scheduling model with queueing support,
systematic review of possible policies
(proposal of a new efficient one: adaptive preemption)
● P2P data transfer for P2P Grid computing
● BitTorrent (TTG)
● distributed caching + data-awareness (STG)
● BitTorrent + distributed caching (implicit TTG)
![Page 50: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/50.jpg)
50
Contributions (continued)
Systematic Cooperation in P2P Grids
● software engineering
● first, top-down application to a complete middleware of
code once, deploy twice (Grid Reality And Simulation, M. Quinson)
● reproducible testing
● execution of Iterative Stencils
● LBG-SQUARE (locality-aware co-allocation)
● P2P-aware P2P checkpointing mechanism (fault-tolerance)
![Page 51: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/51.jpg)
51
Perspectives
Systematic Cooperation in P2P Grids
Scheduling
● investigate Task replication as well as reservations
● simulation of data transfers, better simulation of multithreading
● measure system-wide impact of local scheduling choices
Middleware scalability
● improve even more the scalability of data transfers
(CDN-like data replication, adaptive data compression)
● large-scale deployment (Cloud Computing, Volunteer Grid)
![Page 52: Systematic Cooperation in P2P Grids€¦ · 2 Application class: Bags of Tasks Systematic Cooperation in P2P Grids Bag of Task = set of independent computational Tasks many domains:](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c886c7e708231d435e08a/html5/thumbnails/52.jpg)
Thank You.