designing parallel operating systems via parallel programming

Euro-Par - August 31- September 3, 2004 - Pisa (Italy)

Designing Parallel Operating Systemsvia Parallel Programming


email:[email protected]

Eitan Frachtenberg1, Kei Davis1, Fabrizio Petrini1,

Juan Fernández1,2 and José Carlos Sancho1

1Performance and Architecture Lab (PAL) 2Grupo de Arquitectura y Computación Paralelas (GACOP)

CCS-3 Modeling, Algorithms and Informatics Dpto. Ingeniería y Tecnología de Computadores

Los Alamos National Laboratory, NM 87545, USA Universidad de Murcia, 30071 Murcia, SPAIN

URL: http://www.c3.lanl.gov URL: http://www.ditec.um.es


HARDWAREHARDWARE = = Independent Nodes + High-speed Independent Nodes + High-speed NetworkNetwork

SOFTWARESOFTWARE= = Commodity OS + Parallel Apps + System Commodity OS + Parallel Apps + System SoftwareSoftware

OSOS OSOSOSOS

OSOS

OSOS OSOSOSOS

OSOS

MotivationMotivation

Clusters have been the most successful player in high-performance computing in the last decade


Earth Simulator Earth Simulator 5120 Processors5120 Processors

Thunder (LLNL) Thunder (LLNL) 4096 Processors4096 Processors

ASCI Q (LANL) ASCI Q (LANL) 8192 Processors8192 Processors


Ever-increasing demand for computing capability is driving the construction of ever-larger clusters

Systems are becoming more complex,Systems are becoming more complex,less efficient and less reliableless efficient and less reliable


PROBLEM: parallel software has neither PROBLEM: parallel software has neither evolved nor scaled accordingly to cluster evolved nor scaled accordingly to cluster

sizessizes


SOLUTION: new approach to the design SOLUTION: new approach to the design of parallel software for large-scale of parallel software for large-scale

clustersclusters

Clusters are loosely-coupled systems used for solving inherently tightly-coupled problems

Parallel software keeps all the pieces together

Development of parallel software is a time- and resource- consuming task due to its complexity


GoalsGoals

Target New methodology for the design of parallel

software Simplicity, performance, scalability, reliability Backbone to integrate all nodes into a parallel OS

Vision BSP-like system running MIMD applications

(variable granularity in the order of hundreds of s)

Approach BSP-like global control and coordination of all

system activities Small set of collective communication primitives

for global coordination


Motivation and Goals

Toward a Parallel Operating System

Core Primitives

Parallel Software Design

Case Studies

Concluding remarks

OutlineOutline


Designing a Parallel OS:

Lack of global coordination (loose coupling)

Redundant/missing functionality (complexity)

Toward a Parallel OSToward a Parallel OS

Hardware

CommProtocol 1

CommProtocol 2 . . . Comm

Protocol N

ResourceManagement

ParallelApplication . . . Parallel

File System



Scientific applications are tightly coupled … Data dependencies between nodes They exchange messages very often

… but the processing nodes are “bolted

together” in a loosely coupled fashion

Need for global control and coordination Need for global control and coordination ofof

all the system activities, enforced byall the system activities, enforced byglobal collective communication global collective communication

primitivesprimitives



System-level, global control and coordination of all application and system software activities


Hardware

CommProtocol 1


Protocol N

Global control and coordination

ResourceManagement


File System



Parallel applications use point-to-point and

collective communication

System software tasks are either collective

operations or can be cast in terms of them

Parallel applications and system Parallel applications and system software can be built atop the same software can be built atop the same

communication primitivescommunication primitives



Least common denominator of system and application software Core Primitives


Hardware

CommProtocol 1


Protocol NCore Primitives

Global control and coordination

ResourceManagement


File System




Core Primitives


Case Studies

Concluding remarks

OutlineOutline


Parallel software built atop three primitives Xfer-And-Signal

– Transfer block of data to a set of nodes– Optionally signal local/remote event upon completion

Test-Event– Poll local event

Compare-And-Write– Compare global variable on a set of nodes– Optionally write global variable on the same set of nodes

Core PrimitivesCore Primitives


Parallel software built atop three primitives Xfer-And-Signal (QsNet):

– Node S transfers block of data to nodes D1, D2, D3 and D4

S D1 D2D4D3



Parallel software built atop three primitives Xfer-And-Signal (QsNet):

– Node S transfers block of data to nodes D1, D2, D3 and D4

– Events triggered at source and destinations

S D1 D2D4D3

SourceEvent

DestinationEvents



Parallel software built atop three primitives Compare-And-Write (QsNet):

– Node S compares variable V on nodes D1, D2, D3 and D4

S D1 D2D4D3

•Is V {, , >} to Value?



Parallel software built atop three primitives Compare-And-Write (QsNet):

– Node S compares variable V on nodes D1, D2, D3 and D4

– Partial results are combined in the switches

S D1 D2D4D3





Core Primitives


Case Studies

Concluding remarks

OutlineOutline


•Global Strobe•(time slice starts)

•Global Strobe•(time slice ends)

Task 1

Task 2

•Global•Synchronization


Tim

e S

lice

(h

un

dre

ds

of s

)Toward a Parallel OSToward a Parallel OS

Global control/coordination of all system activities

Task 3


Using the core primitives… Global control and coordination

– Strobe sent at regular intervals (time slices) Compare-And-Write + Xfer-And-Signal (Master) Test-Event (Slaves)

– All system activities are tightly coupled– Global information is required to schedule resources, global

synchronization facilitates the task but it is not enough

Global resource scheduling– Exchange of requirements/restrictions

Xfer-And-Signal + Test-Event– Resource scheduling

Parallel Software DesignParallel Software Design


Characteristic Workstation Cluster

Job Launching OS Scripts atop the OS

SYSTEMSYSTEM

SOFTWARESOFTWARE

Job Scheduling Timeshared by OSBatch queued or gang scheduled by middleware

CommunicationStandard IPC and shared memory

MPI

StorageStandard file system

Custom parallel file system

Debuggability Standard toolsCustom parallel debugging tools

Fault Tolerance Little or noneApplication-specified checkpointing


Applications System calls Rely on System Software


Characteristic

Requirement Solution

Job Launching

Data Dissemination

Flow Control

Termination Detection

Xfer-And-Signal

Compare-And-Write

Compare-And-Write

Job SchedulingHeartbeat

Context Switch

Xfer-And-Signal

Prioritized messages/multiple rails

Communication

PUT

GET

Barrier

Broadcast

Xfer-And-Signal

Xfer-And-Signal

Compare-And-Write

Compare-And-Write + Xfer-And-Signal

Using the core primitives…



Can we really buildsystem software using

this new approach?

Parallel Software Design Parallel Software Design



Introduction

Core Primitives


Case Studies

Concluding remarks

OutlineOutline


Experimental Setup

Characteristic

Crescendo Cluster Wolverine Cluster

Nodes 32 x Dell 1550 64 AlphaServer ES40

CPUs/Node 2 x 1GHz Pentium-III 4 x 833MHz EV68

Memory/Node 1 GB 8 GB

Network Cards QM-400 Elan3 QM-400 Elan3

OS RH 7.3 + QsNet kernel RH 7.1 + QsNet kernel

Software

Qsnetlibs v1.5.0-0 +

Intel C/Fortran Compiler 5.0.1

Qsnetlibs v1.5.0-0 +

Compaq´s C Compiler

Case StudiesCase Studies


STORM (Scalable TOol for Resource Management)– Architecture:

Set of dæmons running on the management/compute nodes Built atop the three core primitives BSP-like behavior: management activities are synchronized

and scheduled every few hundreds of microseconds

– Functionality: Job Launching Job Scheduling (FCFS, gang scheduling and others)

New scheduling algorithms can be “plugged in” Resource Accounting



Job Launching: send/execute/check for completion

40 times faster than the best reported result!!!



BCS-MPI (Buffered CoScheduled MPI)– Architecture

Set of cooperative threads running in the NIC Built atop the three core primitives BSP-like behavior: communications are synchronized and

scheduled every few hundreds of microseconds– Functionality:

Subset of the MPI standard– Paves the way to provide:

Traffic segregation Deterministic replay of user applications System-level fault tolerance



SWEEP3D and SAGE Performance (IA32)– Production-level MPI versus BCS-MPI


0.5% SPEEDUP 2% SPEEDUP



Introduction

Core Primitives


Case Studies

Concluding remarks

OutlineOutline


Methodology for designing parallel software Coordination of all system and application software activities

in a BSP-like fashion Parallel applications and system software built atop a basic

set of collective primitives for global coordination Backbone to integrate all nodes into a parallel OS

Promising preliminary results demonstrate that

this approach is indeed feasible

Concluding RemarksConcluding Remarks


Kernel-level implementation User-level solution is already working

Deterministic replay of MPI programs Ordered resource scheduling may enforce

reproducibility

Transparent fault tolerance Global coordination simplifies the state of the

machine

Future WorkFuture Work




email:[email protected]

Eitan Frachtenberg1, Kei Davis1, Fabrizio Petrini1,

Juan Fernández1,2 and José Carlos Sancho1

1Performance and Architecture Lab (PAL) 2Grupo de Arquitectura y Computación Paralelas (GACOP)

CCS-3 Modeling, Algorithms and Informatics Dpto. Ingeniería y Tecnología de Computadores

Los Alamos National Laboratory, NM 87545, USA Universidad de Murcia, 30071 Murcia, SPAIN

URL: http://www.c3.lanl.gov URL: http://www.ditec.um.es


Characteristic

Requirement Solution

StorageMetadata Transfer

File Data TransferXfer-And-Signal

Debuggability

Debug Data Transfer

Debug Synchronization

Xfer-And-Signal

Compare-And-Write

Fault Tolerance

Fault Detection

Checkpointing Synchronization

Checkpointing Data Transfer

Compare-And-Write

Compare-And-Write

Xfer-And-Signal

Using the core primitives…



Job Scheduling: gang scheduling

Very small time slices: RESPONSIVENESS !!!



•Global Strobe•(time slice starts)

•Global Strobe•(time slice ends)

Exchange of comm requirements

Communication scheduling

Real transmission



Tim

e S

lice

(h

un

dre

ds

of s

)Toward a Parallel OSToward a Parallel OS

BCS-MPI: real-time communication scheduling



BCS-MPI: real-time communication scheduling

designing parallel operating systems via parallel programming

Documents