Scheduling Dynamic Parallelism on Accelerators

Filip Blagojevic, Costin Iancu,Katherine Yelick

Lawrence Berkeley National Laboratory1 Cyclotron Road

Berkeley CA 94720{fblagojevic, cciancu,

kayelick}@lbl.gov

Matthew Curtis-Maury

NetApp, Inc 7301 Kit CreekRd, RTP, NC 27709

mcm@netapp.com

Dimitrios S. Nikolopoulos,Benjamin Rose

Virginia TechBlacksburg, VA 24060

{dsn,bar234}@cs.vt.edu

ABSTRACTResource management on accelerator based systems is com-plicated by the disjoint nature of the main CPU and acceler-ator, which involves separate memory hierarhcies, di!erentdegrees of parallelism, and relatively high cost of communi-cating between them. For applications with irregular par-allelism, where work is dynamically created based on othercomputations, the accelerators may both consume and pro-duce work. To maintain load balance, the accelerators handwork back to the CPU to be scheduled. In this paper weconsider multiple approaches for such scheduling problemsand use the Cell BE system to demonstrate the di!erentschedulers and the trade-o!s between them. Our evaluationis done with both microbenchmarks and two bioinformaticsapplications (PBPI and RAxML). Our baseline approachuses a standard Linux scheduler on the CPU, possibly withmore than one process per CPU. We then consider the ad-dition of cooperative scheduling to the Linux kernel and auser-level work-stealing approach. The two cooperative ap-proaches are able to decrease SPE idle time, by 30% and70%, respectively, relative to the baseline scheduler. In bothcases we believe the changes required to application levelcodes, e.g., a program written with MPI processes that useaccelerator based compute nodes, is reasonable, althoughthe kernel level approach provides more generality and easeof implementation, but often less performance than workstealing approach.

Categories and Subject DescriptorsD.4.1 [Operating Systems]: Process Management — Con-currency, Scheduling, Synchronization; C.1.3 [ProcessorArchitectures]: Other Architecture Styles — Heteroge-neous (hybrid) systems

General TermsPerformance, Design

Copyright 2009 Association for Computing Machinery. ACM acknowl-edges that this contribution was authored or co-authored by an employee,contractor or affiliate of the U.S. Government. As such, the Government re-tains a nonexclusive, royalty-free right to publish or reproduce this article,or to allow others to do so, for Government purposes only.CF’09, May 18–20, 2009, Ischia, Italy.Copyright 2009 ACM 978-1-60558-413-3/09/05 ...$5.00.

KeywordsCooperative Scheduling, Cell BE

1. INTRODUCTIONIn recent years accelerator based parallel architectures

have been resurrected as a viable alternative for system de-sign and there is a large body of work [10, 19, 9, 16, 22,25, 17, 24] indicating that these architectures are able toprovide high performance with very low power consump-tion. The Cell BE is an exponent of such architecture andCell based clusters such as the Roadrunner system at LosAlamos National Laboratory are used in application settingssuch as high performance scientific computing or financialdata analysis.

Accelerator based architectures are asymmetrical, containboth general purpose processors and specialized hardwareand often exhibit di!erent degrees of hardware parallelisminside each hierarchy. The Cell BE contains one PowerPCcore (PPE) with two hardware execution contexts and eightspecialized processors (SPEs). GPU based solutions have avery deep level of parallelism (thousands) and are often inte-grated into systems with generic multi-core processors withlimited parallelism (tens). A common execution model forthese architectures assumes a cooperation between the gen-eral purpose processors and the accelerator hardware: theported applications usually have a “driver” running on themain processor side and o"oad parts of the computation tothe specialized hardware. One of the open research problemsis mapping applications to the two independent executionhierarchies that exhibit di!erent degrees of parallelism andproviding e#cient synchronization and coordination.

Previous work [5, 7] investigates the mapping of paral-lel computation to the Cell BE and shows the importanceof the careful management of the PPE ! SPE interaction:explicitly interacting with the Linux scheduler and yieldingthe PPE whenever processes are waiting for SPE tasks tofinish is able to achieve many-fold application performanceimprovement. Yielding the PPE processors improves the at-tentiveness of the computation to asynchronous events gen-erated by SPEs which results in a higher overall SPE uti-lization, but it is still subject to the implicit Linux kernelscheduling policies. The Linux’s scheduler lack of knowledgeabout the asynchronous nature of the SPE computation islikely to increase the latency of any response to the SPEgenerated events.

161

!"##$%&'()&

*+#,-&.#/012,3#4&

5,62&7#8&'()&

5,62&7#8&.#/012,3#4&

9%2."&-,2,&

!#/012%&

'6:4,;&(()&

!!"# $!"#

Figure 1: Generic structure of code implementingo!oading and PPE!SPE interaction.

In this paper we examine kernel and user-level schedul-ing techniques to minimize the latency of PPE processesresponse to SPE generated events. In both implementationswe use an event-driven cooperative scheduling approach;SPE tasks indicate which PPE process is required to servetheir requests. The kernel level implementation extends theLinux kernel with data structures and system calls whichallow SPEs to request activation of a specific process on thePPE side. For the user-level approach we use shared mem-ory abstractions and a work stealing strategy. We evaluatethe performance of the proposed techniques on the SonyPlayStation3 using micro-benchmarks and two bioinformat-ics applications: PBPI and RAxML. When compared to theperformance of the default Linux scheduling, the results in-dicate that using our kernel modifications we can reduceSPE idle time by up to 30%, while work-stealing providesimprovements of 70%. The reduction of SPE idle time trans-lates directly into application performance improvements:we obtain an average speedup of 5%, with maximum perfor-mance improvements of 10% using kernel scheduling. Usingwork-stealing we get speedups of 12% and 21% respectively.

The principles described in this paper are widely applica-ble and of interest to several classes of software developers onaccelerator based systems: application and framework devel-opers, runtime implementors (e.g. MPI, OpenMP, Charm++,RapidMind) and developers of communication libraries forclusters of accelerator based systems.

2. CELL BE PROGRAMMINGBesides having to address the architectural idiosyncrasies

(alignment restrictions for data transfers, explicitly man-aged local storage), the main challenges in achieving goodperformance on the Cell BE are choosing the right executionmodel and parallelization strategy. The o"oading executionmodel assumes frequent PPE!PPE and PPE!SPE inter-action but little SPE!SPE interaction: there is a driverapplication executing on the PPE that o"oads SPE compu-tations that are independent of each other. This approachhas been shown to provide good performance in practice andmany application studies on the Cell BE employ it. The “of-floading” approach is of particular interest when consideringporting existing parallel applications in a Cell cluster envi-ronment or when using a compiler for parallelizing the Cellcode in the OpenMP fashion. The principles described inthis paper are directly applicable in both situations.

Several software development infrastructures for the Cell

BE, including ours, provide o"oading APIs, e.g. IBM ALF [2],Charm++ O"oading API [1], Map/Reduce for Cell [12],RapidMind [20]; sometimes ad-hoc implementations are pro-vided by developers. We provide a generic o"oading APIfor the Cell BE that allows programmers to select SPEs toperform computations, move data between the local storeand main memory and in particular to allocate and managesynchronization objects in main memory.

For parallel applications designed for execution in a clusterenvironment, partial PPE execution (and oversubscription)is almost required, as communication libraries are not de-signed to run on accelerators. Also, oversubscription with of-floading is an execution model that requires minimal changesfor already existing applications to run on Cell. O"oad-ing execution requires significant PPE!SPE communica-tion. The generic structure of the PPE!SPE code is pre-sented in Figure 1. The asynchronous interaction betweenPPE and SPE is handled in the function wait_for_SPE andvarious mechanisms can be used for its implementation. Inthe rest of this paper we discuss the implementation andperformance trade-o!s of di!erent approaches to implementfast PPE!SPE synchronization and cooperation on the CellBE. We compare asynchronous event handling using IBMCell SDK primitives (callbacks and mailboxes) with kernelextensions for cooperative scheduling and with user level im-plementations for work stealing. Figure 2 illustrates the im-portance of fast synchronization and cooperative schedulingin RAxML (application described in Section 4) running on acluster of Sony PlayStation3 consoles, when the PPE is over-subscribed with 6 MPI processes, each o!-loading on 1 SPE(the Cell processor on the PS3 exposes only 6 SPEs to theapplication). The results show that compared to a schedul-ing policy which is oblivious to PPE ! SPE co-scheduling(labeled as Linux in Figure 2), cooperative scheduling (la-beled as yield-if-not-ready in Figure 2) achieves a perfor-mance improvement of 1.7–2.7". The optimal balance andgranularity of parallelism is not easy to determine in mostapplication settings and high degree of oversubscription islikely to provide the best performance in many scenarios [8,6]. To illustrate benefits of oversubscription, Figure 3 presentsthe performance of RAxML for various configurations of as-signments of SPEs to PPE processes. Note that the perfor-mance of configurations with the highest degree of oversub-scription, that use 6 PPE processes with one SPE assignedto each, is two to three times faster than configurations thatassign 6 SPEs to one PPE process. In this case, mechanismsto increase the responsiveness of PPE processes to asyn-chronous SPE requests are likely to improve performance.

3. HANDLING ASYNCHRONOUS EVENTSAchieving good performance on the Cell BE requires care-

ful orchestration of the PPE! SPE interaction. Co-schedulingof PPE ! SPE parallelism is a technique discussed in [5]that has been shown to significantly improve the perfor-mance of several applications. The main objective of co-scheduling is to maximize SPE utilization, i.e. minimize thewaiting time whenever a thread o!-loaded to an SPE needsto communicate or synchronize with its originating threadon the PPE. This is achieved by increasing the chance thatthe required PPE thread will run as soon as an SPE requestis generated. Figure 4 illustrates how di!erent implementa-tions of the PPE ! SPE synchronization function can havea significant impact on co-scheduling utilization. In Fig-

162

(1,6,1) (2,6,1) (3,6,1) (4,6,1) (5,6,1) (6,6,1) (7,6,1) (8,6,1)0

2000

4000

6000

8000

10000

12000

Exec

utio

n Ti

me

(sec

)

Configuration (Nnode

, Nprocess

, NSPE

)

RAxML Linux

yield!if!not!ready

Figure 2: Performance of yield-if-not-ready policy andthe native Linux scheduler in RAxML. Nnode - numberof nodes, Nprocess - number of processes per node, NSPE

- number of SPEs per process.

RAxML

0

5000

10000

15000

20000

25000

(1,1

,6)

(1,2

,3)

(1,3

,2)

(1,6

,1)

(2,2

,6)

(2,4

,3)

(2,6

,2)

(2,1

2,1

)

(3,3

,6)

(3,6

,3)

(3,9

,2)

(3,1

8,1

)

Configuration (#Nodes, #Processes, #SPE)

Execu

tio

n T

ime

(sec)

DNA Seq Size=1250

DNA Seq Size=2000

DNA Seq Size=5000

Figure 3: Performance improvements with process over-subsription in RAxML. Nnode - number of nodes, Nprocess

- number of processes per node, NSPE - number of SPEsper process.

ure 4(a), PPE threads are busy-waiting for the correspond-ing o!-loaded threads to return results from SPEs. The timequantum allocated to each PPE thread by the OS can causecontinuous mis-scheduling with respect to SPE threads.

Busy-waiting can be combined with explicit yielding inorder to eliminate some of the mis-scheduling. Figure 4(b)illustrates a yield-if-not-ready implementation, where PPEthreads explicitly yield the processor whenever a correspond-ing o!-loaded SPE thread is pending completion. As dis-cussed in the previous section, yield-if-not-ready is able toachieve many-fold application performance improvement. Inthe rest of this paper, for brevity we refer to the yield-if-not-ready policy as YNR.

Although YNR reduces the PPE response time comparedto the busy waiting, the mis-scheduling can still occur, aspresented in Figure 4(b). If p is the total number of activePPE threads, the YNR policy bounds the slack by the timeneeded to context switch across p # 1 PPE threads. Ona Sony PlayStation3, the upper bound on slack when con-sidering 6 active processes (as many as there are availableSPEs) is around 15µs, as determined by the duration of con-text switches using sched_yield() calls under congestion.In the ideal implementation, a PPE thread is run as soon asan SPE request has been generated. Figure 4(c) illustratesthe optimal scheduling situation. In order to achieve the op-timal scheduling, the thread scheduler needs information asto what PPE thread should run to serve the SPE requests.The e#ciency of any cooperative scheduling approach is ulti-mately determined by the system response time – activationof the threads able to handle any outstanding events.

Besides hand coded libraries for fast synchronization us-ing variables in memory, asynchronous PPE! SPE interac-tion can be implemented using two mechanisms provided bythe IBM Cell SDK. A process can register an event handler(callback) to respond to SPE requests. An SPE that trig-gers an event handler will generate a PPE interrupt and itwill block until the handler completes. Processes executingon the PPE side have to observe the interrupt and respondto the SPE request. The latency of this response time is acombination of the software overhead of event handling andthe time for a process to be scheduled on the PPE, giventhe default Linux scheduling policy. In order to decreasethis latency, the synchronization code on the PPE has to beimplemented using YNR. The second signaling mechanismuses interrupt mailboxes. A PPE process can perform block-ing calls to read a mailbox, the process is suspended untildata becomes available. In general, given the fact that in

Linux events are serviced only when a process is scheduledon the CPU, both approaches have to use a YNR schemeand their performance is likely to be lower than a pure YNRscheme using the local store for PPE ! SPE signaling.

Kernel support can be used to implement e#cient cooper-ative scheduling and decrease response latency. We have ex-perimented with the real-time scheduling features availablein Linux and implemented versions of PBPI and RAxMLwhere we force a process to run by increasing its prior-ity. This approach requires a very thorough understand-ing of the application characteristics1 in order to avoid pro-cess starvation. In order to avoid the drawbacks of real-time scheduling we extend the Linux kernel scheduler witha _yield_to() system call and use the main memory forPPE! SPE signaling. For brevity, we refer to this approachas SLED (Slack-Minimizing Event Driven synchronization)and its design is presented in the rest of this section. Ourfinal solution does provide the best performance for micro-benchmarks and applications, and we believe it to be themost intuitive to developers.

3.1 Fast Synchronization Using _yield_to()

The overview of the SLED implementation is illustratedin Figures 5 and 6. The basic data structure for PPE-SPE signaling is a ready_to_run “list”. After o"oading, PPEprocesses explicitly call the SLEDS_wait_for_SPE() function.Each SPE thread upon completing the assigned task writesthe PID of the parent process to the shared ready_to_run datastructure: the presence of a particular PID in any entry in-dicates that the corresponding SPE is waiting for a responsefrom that particular process. Providing cooperative schedul-ing requires two distinct steps: 1) choosing a process to runfrom the list and 2) scheduling the chosen process on a PPEcontext. For the latter, the Linux kernel scheduling codehas to be extended to give priority to a given process.

SLEDS_wait_for_SPE() can be implemented either at user-level or as a system call. The organization of the ready_to_rundata structure determines whether the selection has to bemade inside a critical section. A kernel level implementa-tion will perform this operation inside a critical section bydefault while a user level implementation might have to pro-vide its own mutual exclusion. In order to increase systemresponsiveness, the duration of these critical sections has tobe minimized. In the rest of this section we discuss the de-

1For example the RAxML application uses a master-workerapproach and avoiding live-lock using real-time schedulingrequired a significant e!ort.

163

1

PPE

SPEs

OS quantum

P1

S1

Idle Idle Idle

OS quantum OS quantum

S2

P2 P3 P1

- Mis-scheduling has occured

S3

!"#$%&'(%)*+,-./0%$123"45161

178#9:,;3"45<1

=1

1

yield if not ready

P1 P1 P1P2 P2 P2P3 P3 P3

S1

S2

S3

!"#$%&'(%)*+,-./0%$123"45161

178#9:,;3"45161

1 1-08:<%.#:9<,5=1

1>1

>1

1

S1

P1P1 P1P2 P3 P2 P3 P2 P3

S2

S3

!"#$%&'()%*+,%!-#./012%)345'67383

39:(;<.1+)%=+1<7383

3 3=<)<,>(1<3!-#3,<'=03

333333333330(<;=%)+.,<'=073

3 3?3

?3

(a) (b) (c)

Figure 4: Three cases illustrating the importance of co-scheduling PPE threads and SPE threads. PPE (P) threadspoll shared memory locations directly to detect if a previously o!-loaded SPE (S) thread has completed. Dark intervalsindicate context-switching. Dots mark cases of mis-scheduling.

sign trade-o!s for 1) list organization and management; 2)implementing SLEDS_wait_for_SPE() as a user-level functionor a system call.

Figure 6 presents the implementation of the SLEDS_wait_for_SPE() call; same implementation can be used at user-level aswell as for a system call. As a first step, SLEDS_wait_for_SPEscans the ready_to_run list in order to determine the next pro-cess to run. Depending on the implementation choice, i.e.kernel or user level, the duration of the scan (parameter N)determines the system responsiveness. On a PlayStation3,which allows access to only 6 SPEs, and using the split listimplementation described in Section 3.1.1, the ready_to_runlist contains only 3 entries. We will discuss the influence ofthe parameter N in Section 4.1. The function _yield_to()is a system call which schedules a specific process to run;this is an extension to the standard Linux kernel.

3.1.1 The Synchronization Data Structure: ready_to_runWe have experimented with several data structures for

the implementation of the ready_to_run “list”. A FIFO datastructure would ensure that the process corresponding to theSPE thread which was the first to finish processing wouldbe the first to run on the PPE side; it would also providesome fairness of the SPE allocation and avoid starvation.The downside of a FIFO data structure is that maintainingits consistency requires critical sections which introduce ad-ditional overhead. In order to avoid the locking overheadrequired by FIFO, a data structure with one entry per SPEcan be used, as shown in Figure 5. Consistency is easilymaintained using atomic instructions but the scheduler hasto repeatedly scan the entire list in order to choose the nextcandidate. Scanning the entire list introduces additionaloverhead which influences the system responsiveness. Userlevel implementations are e#cient but there exists the po-tential for unfairness and starvation.

The Linux kernel maintains per CPU running queues,while the approaches described so far provide one data struc-ture shared between all running processes. With this de-sign choice, the implementation has to provide CPU con-text awareness as illustrated by the following scenario. Theo!-loaded task which belongs to the process P1 has finishedprocessing on the SPE side (the PID of the process P1 hasbeen written to the ready_to_run list). Process P1 is boundto CPU1, but the process P2 which is running on CPU2

o!-loads, and initiates the context switch by passing thePID of process P1 to the kernel. Since the context switch

occurred on CPU2 and P1 is bound to run on CPU1, thekernel needs to migrate process P1 to CPU2. This situationcan be solved by explicitly performing process migration in-side our extended kernel scheduling (_yield_to()) or byensuring that processes scanning the ready_to_run list willnever select a process with di!erent CPU a#nity. The situ-ation occurs frequently in practice and both solutions intro-duce additional scheduling overhead. We have experimentedwith both approaches and found the migration solution themost undesirable since it creates an uneven distribution ofprocesses across available CPUs. Depending on the imple-mentation of the next process selection, a#nity checks areperformed either at user level or inside the kernel. Userlevel implementations have to perform two additional sys-tem calls to obtain a#nity information and this leads tosignificant performance degradation when compared to thekernel implementation.

To avoid some of the drawbacks discussed in this sectionwe use an implementation for the ready_to_run data structurethat contains one entry per SPE. The list is statically splitin two halves, each PPE hardware execution context beingresponsible for managing one half; only processes sharingthe execution context are accessing the same ready_to_runlist. In order to eliminate the need for a#nity awareness, theapplication is required to pin its processes to CPUs at jobstart-up time. This approach reduces most of the overheadassociated with the data structure maintenance but it hassome potential for load unbalance.

3.1.2 _yield_to() ImplementationThe standard scheduler used in the Linux kernel, starting

from version 2.6.23, is the Completely Fair Scheduler (CFS).For each process in the system, the CFS records the amountof time that the process has been waiting to be scheduledon the CPU. Based on the amount of time spent waiting inthe run queue and the number of processes in the system,as well as the static priority of the process, each process isassigned a dynamic priority. The dynamic priority of a pro-cess is used to determine when and for how long the processwill be scheduled to run. The data structure used by theCFS for storing the active processes is a red-black tree. Theprocesses are stored in the nodes of the tree, and the processwith the highest dynamic priority (which will be the first torun on the CPU) is stored in the left most node in the tree.The SLED scheduler passes the information from the ready_to_run list to the kernel through the _yield_to() system

164

SPE1

SPE2

SPE3

SPE4

SPE5

SPE6

SPE7

SPE8

1 2

3 4

5 6

7 8

processes

Shared MemoryRegion

PPE schedules processwhich is ready to run

PPE with multiple

Figure 5: Upon completing the assigned tasks, theSPEs send signal to the PPE processes through theready-to-run list. The PPE process which decidesto yield passes the data from the ready-to-run listto the kernel, which in return can schedule the ap-propriate process on the PPE.

1: void SLED_wait_for_SPE(){2: int next, i, j=0;3: if(!ready_to_run[mySPE]) {4: while(next == 0 && j < N){5: i=0;6: j++;7: while(next == 0 && i < 3){8: next = ready_to_run[i];9: i++;10: }11: }12: _yield_to(next);13 }14: }

Figure 6: Outline of SLED wait for SPE. Line 4-11:selection of next process. Line 12: _yield_to() if“local” SPE not ready.

call, which extends the standard kernel scheduler sched_yield()system call by accepting an integer parameter pid which rep-resents the process that should be the next to run. First,the process which should be the next to run is pulled outfrom the tree, and its static priority is increased to the max-imum value. The process is then returned to the runningtree, where it will be stored in the left most node (sinceit has the highest priority). After being returned to thetree, the static priority of the process is decreased to thenormal value. Besides increasing the static priority of theprocess, we also increase the time that the process is sup-posed to run on the CPU. Increasing the CPU time is im-portant, since if a process is artificially scheduled to runmany times, it might exhaust all the CPU time that it wasassigned by the Linux scheduler. In that case, although weare capable of scheduling the process to run on the CPUusing the _yield_to() function, the process will almost im-mediately be switched out by the kernel. Before it exits, the_yield_to() function calls the kernel-level schedule() func-tion which initiates context switching. We have measuredthe overhead in the _yield_to() system call caused by theoperations performed on the running tree and we found it tobe approximately 8% compared to the standard sched_yield()system call.

3.2 User-Level Work-StealingThe SLED scheduler requires kernel modifications to im-

plement the _yield_to() call. In addition, applications re-lying on SLED or YNR have to perform repeated systemcalls and introduce additional overhead in order to achievethe desired scheduling strategy. We investigate an imple-mentation of the same principles using a user level approachbased on work-stealing. In the work-stealing approach aPPE process which is currently running checks the user-level shared work queue and picks the right SPE to respondto, even if that SPE does not belong to the currently run-ning PPE process. To enable the work-stealing execution,we need to provide ”any-to-any”communication between thePPE processes and SPE threads.

On the Cell BE, each SPE local storage is memory mappedinto the address space of the PPE process that had spawnedthe computation. Execution model where ”any-to-any”com-munication (between the PPE processes/threads and SPEs)is allowed, requires a shared address space between the ex-ecution entities on the PPE side. Thread libraries such aspthreads can easily provide the shared memory abstraction,

however our applications of interest have been parallelizedon the PPE side using MPI2 and most MPI implementationsare not thread safe. Furthermore, any existing scientific ap-plications which is MPI based, is likely to be ported to Cellusing the MPI communication. Inserting another (thread)level of parallelism into the application is possible, but re-quires significant programming e!ort.

In the interest of portability and correctness, we had toprovide a shared memory space between all MPI processesrunning on the PPE. In order to achieve this we extendan experimental implementation of the Berkeley UPC [11]compiler, enabling a usage of shared memory across distinctprocesses. This implementation will be made available inthe near future. UPC is a Partitioned Global Address Spacelanguage that provides a shared memory abstraction, uses aSPMD programming model, allows control over data layoutand can easily interoperate with MPI programs.

We use application specific implementations for work steal-ing in both PBPI and RAxML, described in Section 4. Inorder to enable work stealing we had to ensure that all datastructures are allocated in a shared memory region whichis obtained at program start-up. We do provide memoryallocators for the management of this region. The basicabstraction we provide is a shared work queue. Work isdescribed by the parameters required for o"oads and byvariables necessary to identify the state of the computationfor each PPE process. In order to ensure termination, thesecontrol variables are allocated in the shared memory region.

Both applications are written in a recursive style. Sincethe depth of the recursion is not apriori known, computa-tions in the original code are o"oaded only from the termi-nal branch of the recursion. In order to simplify the man-agement of the shared work queue and avoid a full continu-ation passing style implementation, we manually eliminatedrecursion in both applications. The modified applicationsprecompute all the work descriptors and fill in the sharedwork queue. PPE processes repeatedly scan the ready_to_runlist and spawn the next work belonging to the process re-quested by an SPE. Unfolding recursion has the additionalbenefit of increasing the granularity of the o"oaded task.The results reported for the work-stealing implementationuse the same granularities as the original implementation inorder to provide a fair comparison.

2There are Linux processes running on the PPE.

165

0

2

4

6

8

10

12

14

16

18

20

7 14 21 29 36 43 51 58 65 72 80 87 95 102 131

Latency (usec)

Task L

en

gth

(u

sec)

SLED YNR Work-Steal PPE MBOX

Figure 7: Variation of the latency of PPE-SPE re-sponse time for YNR, SLED and work-stealing withincreasing task lengths.

30%

40%

50%

60%

70%

80%

90%

100%

10 100 1000

Task Length (usec)

Util

izatio

n

Work-Steal SLED YNR MBOX

Figure 8: Expected SPE utilization with di"erenttask lengths. Idle time been chosen as the averageSPE idle time across experiments. Work-Steal-3µs,SLED-7µs, YNR-10µs, MBOX-19µs.

4. EXPERIMENTAL SETUPIn all experiments we use the Cell processor embedded in

the Sony PlayStation3 console and the variant of the 2.6.23Linux kernel version specially adapted for it. This is thekernel we have extended with SLED support. We use theIBM Cell SDK3.0 to compile all applications and an exper-imental version of the Berkeley UPC software for the work-stealing implementation. The PS3 allows user level accessto only six SPEs and this represents the upper bound onPPE oversubscription we have explored in all experiments.We investigate the impact of our techniques using micro-benchmarks and two real-world bioinformatics applications:RAxML and PBPI.

4.1 Micro-benchmark ResultsTo understand the implementation trade-o!s we use a

micro-benchmark where multiple MPI processes repeatedlyo"oad tasks to SPEs. The PPE only initiates the o!-loadingof variable length tasks and waits for their completion. Allresults described in this section were obtained with the PPEoversubscribed with 6 MPI processes.

Figure 7 presents the variation of the SPE idle time withincreasing lengths of the o"oaded task. The SPE idle timeis directly determined by the latency of the signaling mecha-nisms employed. Work-stealing exhibits the lowest overheadand the average SPE idle time is around 3µs. With SLEDthe average SPE idle time is around 7µs, while the YNRexperiments show 10µs. These results indicate the upperbounds on the SPE idle time reduction: Work-Steal can re-duce idle time by at most 70% when compared to YNR,while SLED will reduce it by at most 30%. The SLED ex-periments shown correspond to the implementation wherethe selection of the next process is implemented at the userlevel and processes are pinned to CPUs. For comparison, weinclude the overheads due to contention on the PPE side:PPE and MBOX. These represent an upper bound for theSPE idle time when using their associated scheme for signal-ing. PPE presents the overhead (14µs) of repeatedly callingsched_yield() when oversubscribing with 6 processes. Thelower overhead of the YNR scheme is explained by lowerPPE contention. MBOX presents the SPE idle time (19µs)when repeatedly o"oading empty tasks and coordinatingusing register interrupt mailboxes. Synchronization usingcallbacks is an order of magnitude slower.

Figure 8 presents the expected improvements in SPE uti-lization for di!erent task lengths. These improvements arean upper bound on achievable application speedup for anygiven scheme. For Work-Steal, SLED and YNR we haveused the average idle times determined in experiments, 3µs,7µs and 10µs respectively; for MBOX we use 19µs. The ex-pected speedup decreases with the task length and cooper-ative scheduling is most e!ective for short to medium tasks.For example, the expected speedup of Work-Steal comparedto YNR is $ 18% for 30µs tasks and $ 3% for 300µs tasks.The ordering of lines in Figure 8 indicates which implemen-tation strategy will perform better in practice. A compari-son of the SLED and YNR idle times in Figure 7 shows thatthe latter exhibits lower latency for task lengths shorter than$ 15µs. The additional implementation overhead of SLEDcauses this performance degradation for very short“synchro-nization” intervals. This behavior is consistent and YNRoutperforms SLED in micro-benchmarks and applicationsfor these very short tasks.

The performance of SLED is determined by several im-plementation choices and in the rest of this section we willdiscuss the influence of polling length (parameter N in Fig-ure 6) when SLED_wait_for_SPE is implemented at user-levelor kernel level respectively. Figures 9 and 10 present theperformance trends observed when SLED_wait_for_SPE is im-plemented at kernel level. Figure 9 presents the evolution ofperformance with scan length normalized to the best execu-tion time for a given task length; for example a point insidethe (1-1.1) region indicates performance at most 10% worsethan the observed minimum. The shorter the polling inter-val, the closer the performance gets to the observed best.Longer polling inside the kernel decreases the responsive-ness to other processes. Figure 10 presents the evolutionof SPE idle time per o"oad. This is a measure of perfor-mance degradation caused by PPE contention. The overallperformance when polling is implemented at the user levelis relatively insensitive to the duration of the polling inter-val. The average SPE idle time with user level polling is7µs, while kernel level polling shows $ 9µs; this indicatesthat user level polling is preferable to the kernel level. Themicro-benchmark results indicate that a short scan lengthof 50-100 passes over the ready_to_run list is likely to providegood performance regardless of the implementation choice.

166

1

1.05

1.1

1.15

1.2

1.25

1.3

1.35

1.4

Kernel-Level Polling

100 200 300 400 500 600 700 800 900 1000

Number of Scans

14

21

29

36

43

50

57

64

71T

ask L

ength

(usec)

Figure 9: Variation of the relative speed of SLEDwith kernel level polling as a function of task lengthand polling duration. The graph shows the time perexperiment normalized to the shortest time for agiven task length.

2

4

6

8

10

12

14

Response Latency (usec), Kernel-Level Polling

100 200 300 400 500 600 700 800 900 1000

Number of Scans

14

21

29

36

43

50

57

64

71

Task L

ength

(usec)

Figure 10: Variation of SPE idle time as a functionof task length and polling duration. Polling is im-plemented at kernel level.

4.2 Application ResultsThe input sets for both applications contain a number of

species with a given DNA sequence length. The number ofspecies determines the memory footprint of the application,while the length of the input DNA sequence determines theduration of the o!-loaded tasks. For PBPI we use a dataset with 107 species, while for RAxML the set contains 40species. For both applications we vary the length of theDNA sequence from $ 100 to $ 5, 000. Figure 13 shows thedistribution of the duration of the o"oaded tasks for sample“short”, “medium” and “long” DNA sequences. Most tasksin PBPI are shorter than 100µs, while in RAxML long DNAsequences will generate much longer tasks.

Figure 11 presents the speedup for PBPI, achieved by theSLED and work-stealing implementations when comparedto YNR. The x # axis shows the DNA sequence length.For PBPI, work-stealing consistently outperforms YNR andspeedup for the whole workload is 12%, with a maximumspeedup of 21%. YNR outperforms the SLED scheme forsmall task sizes. For task sizes larger than 15µs SLEDachieves an average speedup of 5% for the whole workload,with a maximum of 10%. The performance improvementsfor RAxML are more modest due to the larger task length.The average improvements for the workload are $ 3%, witha maximum of 5%. Work-stealing for RAxML is consistentlyslower than the YNR implementation with an average slow-down of 23% for the workload considered. In RAxML, thesame o!-loaded region can be reached from various parts ofthe code, and also di!erent branches in the code can be takenafter o!-loading completion. To perform work-stealing upono!-loading the processes need to exchange the informationabout branches which will be taken and complete controlover program counters and stacks. Our work migration im-plementation is not a full Continuation Passing Style and weuse only ”partial”migration, i.e. at certain point in the codeeach process will be returned its original data (if the workmigration was performed). We found that the partial work-stealing requires significant synchronization which results insignificant performance degradation.

Figure 12 shows that cooperative scheduling (SLED) im-proves the SPE idle time by $ 20% for both applications.All application performance results are consistent with thetrends reported for micro-benchmarks. The results for both

applications indicate that a judicious implementation of co-operative scheduling is required for best performance. IDLEgraph in Figure 12 represents the SPE idle time – time thatSPEs are waiting for work as a percentage of total execu-tion time. For brevity we report only the general trendsobserved for our design decisions for each implementation.Distributed data structures (ready_to_run list) and processpinning are required for good performance. For the YNRimplementations pinning processes to hardware contexts didnot a!ect performance, this might change in the presence ofapplication level load imbalance. Implementations with ashared data structure require3 a#nity awareness: 1) SLED(unpinned processes) with checking for a#nity at the userlevel produces performance inferior to YNR; and 2) SLEDwith kernel level a#nity checking produces performance in-ferior to SLED and pinning, but better than YNR. For SLEDand pinning the performance is determined by the choice ofimplementing SLED_wait_for_SPE at the user or kernel level.Figure 14 presents the execution time of PBPI with SLEDand polling at the kernel level. While for both PBPI andRAxML performance is relatively insensitive to the pollingduration in all other cases, PBPI exhibits a many-fold per-formance degradation in this case. After each o"oad stagePBPI performs a MPI_Allreduce operation and we believethat spending a longer time inside the kernel for pollingadversely a!ects the progress of the other processes. Wetherefore advocate for implementing synchronization func-tions such as SLED_wait_for_SPE as user-level library calls.

The benefits of cooperative scheduling are observable whenthe PPE hierarchy is oversubscribed and this might not bethe best performing parallelization scheme for a given appli-cation. The parallelization trade-o!s for each application areanalyzed in [6]. Note that extracting loop level parallelismrequired several months of development e!ort for each ap-plication while using task-level parallelism derived directlyfrom the MPI code required less coding e!ort to achievesimilar performance. For RAxML, oversubscription with 6MPI processes produces best performance results when com-pared with more aggressive implementations that use looplevel parallelism. These di!erences range from many-fold totens of percent and our approach will only improve the per-formance. Extending PBPI with loop level parallelism yields

3In order to avoid process migration.

167

PBPI Speedup

-10%

-5%

0%

5%

10%

15%

20%

25%

20

4

44

4

68

4

92

4

11

64

14

04

16

44

18

84

21

24

23

64

26

04

28

44

30

84

33

24

35

64

38

04

40

44

42

84

45

24

47

64

Speedup

DN

A S

eq

uen

ce L

en

gth

SLED-Kernel

SLED-User

Work-Steal

RAxML Speedup

-4%

-3%

-2%

-1%

0%

1%

2%

3%

4%

5%

6%

10

0

20

0

30

0

40

0

50

0

60

0

70

0

80

0

90

0

10

00

11

00

12

00

13

00

14

00

15

00

16

00

17

00

18

00

19

00

20

00

Speedup

DN

A S

eq

uen

ce L

en

gth

SLED User

SLED Kernel

Figure 11: PBPI and RAxML speedup of SLED and work-stealing compared to YNR. Work-stealing forRAxML produces a 23% slowdown for the workload.

PBPI SPE Utilization

-60%

-50%

-40%

-30%

-20%

-10%

0%

10%

20%

30%

40%

20

4

44

4

68

4

92

4

11

64

14

04

16

44

18

84

21

24

23

64

26

04

28

44

30

84

33

24

35

64

38

04

40

44

42

84

45

24

47

64

DNA Sequence Length

SP

E I

dle

Tim

e

Red

ucti

on

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Overall

SP

E I

dle

SLED-Kernel

SLED-User

Work-Steal

IDLE

RAxML SPE Utilization

-30%

-20%

-10%

0%

10%

20%

30%

10

0

20

0

30

0

40

0

50

0

60

0

70

0

80

0

90

0

10

00

11

00

12

00

13

00

14

00

15

00

16

00

17

00

18

00

19

00

20

00

DNA Sequence Length

SP

E I

dle

Tim

e

Red

ucti

on

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Overall

SP

E I

dle

SLED User

SLED Kernel

IDLE

Figure 12: Improvements in SPE utilization in RAxML. IDLE (right-hand y-axis) : time when SPEs are waitingfor work as a percentage of total execution time. SLED-user (left-hand y-axis) represents the reductions of theidle time when compared to the YNR idle time.

an implementation where the best performance is obtainedwith di!erent PPE processes/SPE ratios on a per datasetbasis. The resulting implementation achieves a performanceimprovement that varies from many-fold for very short tasksto at most 30% better for medium and large tasks, whencompared to the implementation with task-level parallelismonly. For short to medium task sizes our scheduling strat-egy recuperates a significant fraction of the performance dif-ferences and alleviates the need for a lengthy developmentprocess to coarsen the loop level parallelism.

5. DISCUSSIONThe results indicate that a cooperative scheduling ap-

proach is able to improve the performance of Cell appli-cations and based on our experience, adding this behaviorto existing codes is not a very time consuming task. Theresults obtained on the Sony PlayStation3 are very positiveand we expect our techniques to have an even higher impacton the Cell blades (QS20, QS21 and QS22) which contain 2PPEs (4 hardware execution contexts) and 16 SPEs and arelikely to exhibit higher contention.

The duration of the busy waiting involved in the selectionof the next process to run determines the performance of thecooperative scheduling approach. The results indicate thatshort polling periods are beneficial to performance, pollingshould be performed outside critical sections and distributeddata structures are required for performance. Polling insidecritical sections can result in many-fold performance reduc-tion (Figure 14) even for an architecture like Cell wherethe degree of active contention for the critical section issmall4. The optimal duration of busy waiting is system de-pendent and our experiments indicate that di!erent granu-larities might be required on the Cell blades. For our SonyPlayStation3 experiments we use 100 list iterations.

4There are only two active hardware contexts.

There are several points of concern related to what typeof applications can benefit from an approach like ours. Thework-stealing approach lowers the overhead per o"oad byavoiding system calls and it is able to respond better toPPE load unbalance at the expense of increased develop-ment time and complexity. Work-stealing without compilersupport for a continuation passing style is also best suitedfor SPMD applications where all processes repeatedly of-fload the same computation on all SPEs. The SLED ap-proach can easily accommodate more irregular applications.Load balance and fairness are of concern: as future workwe will consider providing user level APIs to mark cooper-ative scheduling regions and extending the implementationof _yield_to to penalize non-cooperative processes.

The length of o"oaded tasks is a parameter of concernfor the performance of any cooperative scheduling scheme.This length determines the frequency of the synchroniza-tion events and increasing this length will yield diminish-ing returns for cooperative scheduling. In our experimentsand applications we have observed task lengths up to sev-eral hundred of microseconds. We believe this granularitywill appear very often in applications. Private communica-tions with the authors of [25] revealed medium (less than300µs) granularities for their computational kernels imple-mentations. Analyzing the machine balance of the Cell BEarchitecture provides more support for our assumption. TheCell BE embedded in a PS3 console has a peak rate of 21.03double precision Gflops (230 single precision Gflops) andeach SPE has access to a local store of 256KB. The limitedsize of the local store combined with the fast processing rateallows the SPEs to perform a very large number of opera-tions per data point before exhausting the local store andmaybe requiring PPE/SPE synchronization. A single preci-sion computation can perform one operation per every datapoint in the local store in under 3µs. Algorithms with com-plexity less than O(N

p(N)) are likely to process the whole

168

Task Distribution

0

50000

100000

150000

200000

250000

300000

350000

400000

0-1

0:

10-2

0:

20-3

0:

30-4

0:

40-5

0:

50-6

0:

60-7

0:

70-8

0:

80-9

0:

90-1

00:

100-2

00:

200-3

00:

300-4

00:

400-5

00:

500-6

00:

600-7

00:

700-8

00:

800-9

00:

900-1

000:

1000-1

0000:

Task Length

Nu

mb

er o

f Tasks

RAxML 105

RAxML 1176

RAxML 3063

PBPI 105

PBPI 1176

PBPI 3063

Figure 13: Application sample task distribution for“short”, “medium” and “long” DNA sequences.

PBPI SLED Scanning

0

2

4

6

8

10

12

14

16

40

10

0

16

0

22

0

28

0

34

0

40

0

46

0

52

0

58

0

64

0

70

0

76

0

82

0

88

0

94

0

Number of Scans

Tim

e (

sec)

arch107_350arch107_666arch107_972arch107_1278arch107_1584arch107_1890

Figure 14: PBPI performance with SLED and kernellevel polling.

local store within relatively short time frames. Furthermore,as IBM continues to release enhanced Cell processors (thelatest was announced in June 2008 for the Cell blade QS22),the length of the SPE tasks will only decrease, which will inreturn increase the SPE idle intervals.

The Linux kernel mailing lists contain multiple discus-sion threads related to the implementation and the desir-ability of a yield_to() function system call. The positionof the main kernel developers seems to be that such a func-tion is against the Linux “spirit”. Their position is defendedfor homogeneous multi-processor systems such as worksta-tions or servers that are required to accomodate very diverseworkloads. We believe adding this functionality to kernelsdestined for accelerator based heterogeneous systems is ofpractical value.

6. RELATED WORKThe Cell processor has drawn significant attention in in-

dustry and academia. A large number of evaluation studieshave been conducted to determine suitability of the Cell BEfor scientific computation. As an example we list severalcontributions: Bader et al [3] examine the implementationof list ranking algorithms on Cell; Petrini et al [22] reportedexperiences from porting and optimizing Sweep3D on Cell;the same author presented a study of graph explorationsalgorithms on Cell [23].

Several high-performance programming models and com-pilers have recently emerged as a part of an e!ort to alleviateapplication design di#culty for the Cell BE. CellSs [4] is acompiler and a runtime system which reduces programminge!ort for the Cell processor by enabling usage of a singlecode module, i.e. an optimized SPE executable is automat-ically generated from the main code module. The CellSsruntime system detects data dependencies across SPE tasksand uses the obtained information to e#ciently scheduletasks. Eichenberger et al [14] present compiler techniquestargeting automatic generation of highly optimized code forCell. To spread the parallel code across multiple SPEs theyuse OpenMP thread-level parallelization, which is based onthe o!-loading model. It is likely that both approacheswill experience high volume of data exchange and synchro-nization between the PPE and SPE sides of the processor.Zhao and Kennedy [26] present a dependence-driven compi-lation framework for simultaneous automatic loop-level par-allelization and SIMDization on Cell. Sequoia [15] is a pro-gramming language which enables automatic communica-tion across vertical memory hierarchies. Although it doesnot support a single code module programming on Cell,

Sequoia performs automatic memory management on theSPE side which significantly reduces programming e!ort.CorePy [21] is a runtime system which allows programmersto execute the SPE code from Python applications. All ofthe aforementioned programming models are based on the“o!-loading” approach, where the application is split amongthe PPE and SPEs, and can benefit from e#cient PPE !SPE synchronization. The Charm++ framework providesa Cell o"oading API which requires e#cient PPE-SPE sig-naling. All these e!orts can benefit from e#cient PPE !SPE synchronization using our cooperative scheduling ap-proach. We consider our work to be of special interest toany compiler or programming framework on Cell which canpotentially su!er from high PPE ! SPE synchronizationoverhead. Using cooperative scheduling strategies we targetto reduce the PPE response time when a request is initiatedfrom the SPE side. Consequently, we reduce the idle timeon the SPE side while it is waiting for the PPE reply.

In addition, several studies consider Cell execution mod-els which do not involve heavy PPE! SPE communication.Kudlur et al [18] developed a streaming code generation tem-plate. Their method is capable of automatically mapping astream program to the Cell processor. Another example isthe MapReduce framework [12] which provides support forMapReduce [13] execution model on Cell.

7. CONCLUSIONIn this paper we explore the design and performance trade-

o!s of cooperative scheduling implementations for the accel-erator based systmes. The experimental architecture we useis the Cell Broadband Engine. Programs written for the ac-celerator based systems often contain two disjoint parallelhierarchies executing on hardware with di!erent degrees ofnative parallelism. Achieving good performance requires inmany cases oversubscribing the general purpose core. Forthese cases, mechanisms able to increase the attentivenessof the processes running on the general core to asynchronousrequests incoming from the accelerators will improve perfor-mance. We evaluate two such mechanisms: an integrationof cooperative scheduling in the Linux kernel and a work-stealing user level approach. Both approaches improve ap-plication performance in the presence of oversubscription.The kernel level approach provides more generality and easeof implementation at the expense of performance when com-pared to the work stealing approach. The amount of e!ortrequired to incorporate our techniques into existing appli-cations is not prohibitive and we believe their adoption canincrease programmer productivity by alleviating the need for

169

complicated program transformations to extract loop levelparallelism. Cooperative scheduling can be easily integratedinto MPI and OpenMP runtimes for a performance booston accelerator based system architectures. Furthermore, itis likely that cooperative scheduling will significantly im-prove the communication performance among acceleratorsin a cluster environment since this type of communication iscommonly performed through the general core.

8. REFERENCES[1] Charm++ on the Cell Procesor. Available at

http://charm.cs.uiuc.edu/research/cell/.[2] IBM Accelerated Library Framework for Cell

Programmer’s Guide and API Reference.[3] D. A. Bader, V. Agarwal, and K. Madduri. On the

design and analysis of irregular algorithms on the cellprocessor: A case study of list ranking. In Proc. of the21st International Parallel and Distributed ProcessingSymposium, pages 1–10, 2007.

[4] P. Bellens, J. M. Perez, R. M. Badia, and J. Labarta.Memory - cellss: a programming model for the cell bearchitecture. In Proc. of Supercomputing’2006,page 86, 2006.

[5] F. Blagojevic, M. Curtis-Maury, J.-S. Yeom,S. Schneider, and D. S. Nikolopoulos. SchedulingAsymmetric Parallelism on a PlayStation3 Cluster. InCCGRID ’08: Proceedings of the 2008 Eighth IEEEInternational Symposium on Cluster Computing andthe Grid (CCGRID), pages 146–153, Washington, DC,USA, 2008. IEEE Computer Society.

[6] F. Blagojevic, X. Feng, K. Cameron, and D. S.Nikolopoulos. Modeling Multi-grain Parallelism onHeterogeneous Multi-core Processors: A Case Studywith the Cell BE. In Proc. of the 2008 HiPEACConference, Jan. 2008.

[7] F. Blagojevic, D. Nikolopoulos, A. Stamatakis, andC. Antonopoulos. Dynamic Multi-grain Parallelizationon the Cell Broadband Engine. In Proc. of the 12thACM SIGPLAN Symposium on Principles andPractice of Parallel Programming, pages 90–100, Mar.2007.

[8] F. Blagojevic, A. Stamatakis, C. Antonopoulos, andD. Nikolopoulos. Raxml-cell: Parallel phylogenetictree inference on the cell broadband engine. In Proc.of the 21st International Parallel and DistributedProcessing Symposium, Long Beach, CA, Mar. 2007.

[9] J. Bolz, I. Farmer, E. Grinspun, and P. Schrooder.Sparse Matrix Solvers on the GPU: ConjugateGradients and Multigrid. ACM Trans. Graph.,22(3):917–924, 2003.

[10] M. Charalambous, P. Trancoso, and A. Stamatakis.Initial Experiences Porting a BioinformaticsApplication to a Graphics Processor. In PanhellenicConference on Informatics, pages 415–425, 2005.

[11] W. Chen, D. Bonachea, J. Duell, P. Husbands,C. Iancu, and K. Yelick. A Performance Analysis ofthe Berkeley UPC Compiler. In Proceedings of the17th International Conference on Supercomputing(ICS), June 2003.

[12] M. de Krujif and K. Sankaralingam. MapReduce forthe Cell B.E. Architecture.

[13] J. Dean and S. Ghemawat. MapReduce: SimplifiedData Processing on Large Clusters. pages 137–150.

[14] A. Eichenberger, Z. Sura, A. Wang, T. Zhang,P. Zhao, M. Gschwind, K. O’Brien, K. O’Brien,P. Wu, T. Chen, P. Oden, D. Prener, J. Shepherd, andB. So. Optimizing Compiler for the CELL Processor.In Proc. of the 14th International Conference onParallel Architectures and Compilation Techniques,pages 161–172, Saint Louis, MO, Sept. 2005.

[15] K. Fatahalian, D. R. Horn, T. J. Knight, L. Leem,M. Houston, J. Y. Park, M. Erez, M. Ren, A. Aiken,W. J. Dally, and P. Hanrahan. Memory - sequoia:programming the memory hierarchy. In Proc. ofSupercomputing’2006, page 83, 2006.

[16] N. K. Govindaraju, B. Lloyd, W. Wang, M. Lin, andD. Manocha. Fast Computation of DatabaseOperations Using Graphics Processors. In SIGMOD’04: Proceedings of the 2004 ACM SIGMODinternational conference on Management of data,pages 215–226, New York, NY, USA, 2004. ACM.

[17] M. Kistler, M. Perrone, and F. Petrini. CellMultiprocessor Interconnection Network: Built forSpeed. IEEE Micro, 26(3), May-June 2006. Availablefrom http:// hpc.pnl.gov / people / fabrizio / papers/ ieeemicro-cell.pdf.

[18] M. Kudlur and S. Mahlke. Orchestrating theExecution of Stream Programs on MulticorePlatforms. SIGPLAN Not., 43(6):114–124, 2008.

[19] E. S. Larsen and D. McAllister. Fast Matrix MultipliesUsing Graphics Hardware. In Supercomputing ’01:Proceedings of the 2001 ACM/IEEE conference onSupercomputing (CDROM), pages 55–55, New York,NY, USA, 2001. ACM.

[20] M. Monteyne. RapidMind Multi-core DevelopmentPlatform. Available fromhttp://www.rapidmind.net/case-rapidmind.php.

[21] C. Mueller, B. Martin, and A. Lumsdaine. CorePy:High-Productivity Cell/B.E. Programming. In Proc. ofthe First STI/Georgia Tech Workshop on Software andApplications for the Cell/B.E. Processor, June 2007.

[22] F. Petrini, G. Fossum, M. Kistler, and M. Perrone.Multicore Suprises: Lesson Learned from OptimizingSweep3D on the Cell Broadbend Engine.

[23] F. Petrini, D. Scarpazza, O. Villa, and J. Fernandez.Challenges in Mapping Graph Exploration Algorithmson Advanced Multi-core Processors. In Proc. of the21st International Parallel and Distributed ProcessingSymposium, Long Beach, CA, Mar. 2007.

[24] D. P. Scarpazza, O. Villa, and F. Petrini.Peak-Performance DFA-based String Matching on theCell Processor. In IPDPS, pages 1–8. IEEE, 2007.

[25] S. Williams, J. Shalf, L. Oliker, S. Kamil,P. Husbands, and K. Yelick. The Potential of the CellProcessor for Scientific Computing. ACMInternational Conference on Computing Frontiers,May 3-6 2006.

[26] Y. Zhao and K. Kennedy. Dependence-based CodeGeneration for a Cell Processor. In Proc. of the 19thInternational Workshop on Languages and Compilersfor Parallel Computing, New Orleans, LA, Nov. 2006.

170