introduction basics mpi interface topologies message passing...

Debugging andEvaluating ParallelPrograms

Assignment

Evaluating ParallelPrograms

CS/IT 451 – Parallel Processing

Lecture 2 – Message Passing Computing

Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

1 / 64

Message PassingInterface


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

2 / 64

• Google group – get me your email if youare not a member yet


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

3 / 64

Message Passing Paradigm

• Principles– Partitioned address space– Support only explicit parallelization

• Implications– Each data element belongs to one partition

in the address space– Encourages locality of access– All interactions require cooperation of two

or more processes


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

4 / 64

Quiz 2.1

• Can you use message passingparallelization if you do not know howto decompose your problem intoparallel parts?

No.


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

5 / 64

Quiz 2.2

• Can you use shared memoryparallelization efficiently if you do notknow how to decompose your probleminto parallel parts?

No. Well maybe if you use aparallelizing compiler, but itprobably won’t be very efficient.


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

6 / 64

Message Passing Paradigm

• Structure of Message Passing Programs– Synchronous– Asynchronous– Loosely synchronous– Single program multiple data


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

7 / 64

Building Blocks: Send/Receive

• Generic operation definitions• Send

• Receivereceive(void *recvbuf, int nelems, int source)

send(void *sendbuf, int nelems, int dest)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

8 / 64


• Simple Example

P0

A=100;send(&A, 1, 1);A=0;

P1

receive(&B, 1, 0);Printf(“%d\n”,B);


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

9 / 64


• Message Passing Operations– Blocking non-buffered

• Idling• Deadlocks

– Blocking buffered• Deadlocks

– Non-blocking• Caveats


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

10 / 64

Quiz 2.3

• Show a simple example that leads todeadlock if you have blocking sends?

P0 P1send (&A,1,1); send(&A,1,0);receive(&b,1,1); receive(&b,1,0);


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

11 / 64

MPI• Standard library for message passing• C or Fortran libraries• More than 125 routines• 6 basic routines

– MPI_Init– MPI_Finalize– MPI_Comm_size– MPI_Comm_rank– MPI_Send– MPI_Receive


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

12 / 64

MPI: Starting and Terminating• MPI_Init

– Called prior to any calls to any other MPIroutines

• MPI_Finalize

– Called after last other MPI call to clean-up and terminate the MPI environment

MPI_Finalize()

MPI_Init(int *argc, char ***argv)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

13 / 64

Quiz 2.4

• What happens if you call MPI_Sendbefore you call MPI_Initialize or afteryou call MPI_Finalize?

You have created an error andthe implementation can doanything – it is an error.


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

14 / 64

MPI: Communicators• Communication Domain

– A set of processes that are allowed tocommunicate with each other.

– Each process can belong to multiplecommunication domains.

– type MPI_COMM• MPI_COMM_WORLD

– Includes all processes


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

15 / 64

Quiz 2.5

• Can an MPI process belong to 0, 1, ormany communicators?

1 or many; each processalways belongs toMPI_COMM_WORLD


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

16 / 64

MPI: Information Collection• MPI_Comm_size

– Returns (in the size variable) the numberof processes in the communicator comm.

• MPI_Comm_rank

– Returns (in the rank variable) theidentifier of the calling process.

int MPI_Comm_size(MPI_Comm comm, int *size)

int MPI_Comm_rank(MPI_Comm comm, int *rank)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

17 / 64

MPI: Creating a communicatorMPI_Group MPI_GROUP_WORLD, subgroup;int ranks[] = {2, 4, 6, 8};MPI_Comm the_comm;...MPI_Init(&argc, &argv);MPI_Comm_group(MPI_COMM_WORLD,

&MPI_GROUP_WORLD);MPI_Group_incl(MPI_GROUP_WORLD, 4, ranks, &subgroup); /*

local */MPI_Group_rank(subgroup, &me); /* local */MPI_Comm_create(MPI_COMM_WORLD, subgroup,

&the_comm);


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

18 / 64

MPI: Send & Receive• Send

• Receive

MPI_Send(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)

MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

19 / 64

MPI: Information Collection• Hello World Program

#include <mpi.h>main (int argc, char *argv[]){ int npes, myrank; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD,&npes); MPI_Comm_rank(MPI_COMM_WORLD, &myrank); printf(“From process %d out of %d, Hello\n”, myrank, npes); MPI_Finalize();}


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

20 / 64

MPI: Status & Count• Status

• Countint MPI_Get_count(MPI_Status *status, MPI_Datatype datatype, int *count);

typedef struct MPI_Status { int MPI_SOURCE; int MPI_TAG; int MPI_ERROR; };


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

21 / 64

Why does MPI not guaranteebuffering?

• MPI does not guarantee to bufferarbitrary messages because memory is afinite resource on all computers. Thus, allcomputers will fail under sufficientlyadverse communication loads. Differentcomputers at different times are capableof providing differing amounts ofbuffering, so if a program relies onbuffering it may fail under certainconditions, but work correctly under otherconditions. This is clearly undesirable.


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

22 / 64

Why does MPI not guaranteebuffering?

• Given that no message passing systemcan guarantee that messages will bebuffered as required under allcircumstances, it might be asked whyMPI does not guarantee a minimumamount of memory available forbuffering. One major problem is that it isnot obvious how to specify the amount ofbuffer space that is available, nor is iteasy to estimate how much buffer spaceis consumed by a particular program.


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

23 / 64

Quiz 2.6

• List three MPI datatypes.

See next slide.


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

24 / 64

MPI: DatatypesMPI Datatype C DatatypeMPI_CHAR Signed charMPI_SHORT Signed short intMPI_INT Signed intMPI_LONG Signed long intMPI_UNSIGNED_CHAR Unsigned charMPI_UNSIGNED_SHORT Unsigned short intMPI_UNSIGNED Unsigned intMPI_UNSIGNED_LONG Unsigned long intMPI_FLOAT FloatMPI_DOUBLE DoubleMPI_LONG_DOUBLE Long doubleMPI_BYTEMPI_PACKED


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

25 / 64

Slightly more complicated example#include "mpi.h"main( argc, argv )int argc;char **argv;{ char message[20]; int myrank; MPI_Status status; MPI_Init( &argc, &argv ); MPI_Comm_rank( MPI_COMM_WORLD, &myrank ); if (myrank == 0) /* code for process zero */ { strcpy(message,"Hello, there"); MPI_Send(message, strlen(message), MPI_CHAR, 1, 99, MPI_COMM_WORLD); } else /* code for process one */ { MPI_Recv(message, 20, MPI_CHAR, 0, 99, MPI_COMM_WORLD, &status); printf("received :%s:\n", message); } MPI_Finalize();}

Message send/receive;Extend this to N processors.


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

26 / 64

Quiz 2.7

• Can MPI_Send and MPI_Recv, asdescribed, lead to deadlock?

Yes, we will see anexample next…


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

27 / 64

MPI: Deadlockint a[10], b[10], myrank;MPI_Status status;…MPI_Comm_rank(MPI_COMM_WORLD, &myrank);if (myrank == 0) { MPI_Send(a, 10, MPI_INT, 1, 1, MPI_COMM_WORLD); MPI_Send(b, 10, MPI_INT, 1, 2, MPI_COMM_WORLD); }else if (myrank == 1){ MPI_Recv(b, 10, MPI_INT, 0, 2, MPI_COMM_WORLD, &status); MPI_Recv(a, 10, MPI_INT, 0, 1, MPI_COMM_WORLD, &status); }…


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

29 / 64

MPI: Deadlockint a[10], b[10], npes, myrank;MPI_Status status;…MPI_Comm_size(MPI_COMM_WORLD, &npes);MPI_Comm_rank(MPI_COMM_WORLD, &myrank);MPI_Send(a, 10, MPI_INT, (myrank+1)%npes, 1, MPI_COMM_WORLD); MPI_Recv(b, 10, MPI_INT, 1, 1, (myrank-1+npes)%npes, 1, MPI_COMM_WORLD, &status); …


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

31 / 64

MPI: Deadlock⇒Safeint a[10], b[10], npes, myrank;MPI_Status status;…MPI_Comm_size(MPI_COMM_WORLD, &npes);MPI_Comm_rank(MPI_COMM_WORLD, &myrank);If (myrank%2 == 1) { MPI_Send(a, 10, MPI_INT, (myrank+1)%npes, 1, MPI_COMM_WORLD); MPI_Recv(b, 10, MPI_INT, 1, 1, (myrank-1+npes)%npes, 1, MPI_COMM_WORLD, &status); }else { MPI_Recv(b, 10, MPI_INT, 1, 1, (myrank-1+npes)%npes, 1, MPI_COMM_WORLD, &status); MPI_Send(a, 10, MPI_INT, (myrank+1)%npes, 1, MPI_COMM_WORLD); }…


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

32 / 64

MPI: SendrecvMPI_Sendrecv(void *sendbuf, int sendcount, MPI_Datatype senddatatype, int dest, int sendtag,

void *recvbuf, int recvcount, MPI_Datatype recvdatatype, int source, int recvtag, MPI_Comm comm, MPI_Status *status);


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

34 / 64

MPI: Deadlock⇒Safeint a[10], b[10], npes, myrank;MPI_Status status;…MPI_Comm_size(MPI_COMM_WORLD, &npes);MPI_Comm_rank(MPI_COMM_WORLD, &myrank);MPI_Sendrecv(a, 10, MPI_INT, (myrank+1)%npes, 1, b, 10, MPI_INT, 1, 1, (myrank-1+npes)%npes, 1, MPI_COMM_WORLD, &status);…


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

35 / 64

MPI: Sendrecv_replace

MPI_Sendrecv_replace(void *sendbuf, int count, MPI_Datatype datatype, int dest, int sendtag, int source, int recvtag, MPI_Comm comm, MPI_Status *status);


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

37 / 64

MPI: Deadlock⇒Simple Safeint a[10], npes, myrank;MPI_Status status;…MPI_Comm_size(MPI_COMM_WORLD, &npes);MPI_Comm_rank(MPI_COMM_WORLD,&myrank);MPI_Sendrecv_replace(a, 10, MPI_INT, (myrank+1)%npes, 1, (myrank-1+npes)%npes, 1, MPI_COMM_WORLD, &status);…


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

38 / 64

Topologies and Embedding

• Topology• Goodness of Mapping• Communication Domains & Ranks• Embedding


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

39 / 64

Cartesian Topologies• Virtual process topologies• Cartesian topologies

• Process naming

int MPI_Cart_create(MPI_Comm comm_old, int ndims, int *dims, int *periods, int reorder, MPI_Comm *comm_cart)

int MPI_Cart_rank(MPI_Comm comm_cart, int *coords, int *rank)int MPI_Cart_coords(MPI_Comm comm_cart, int rank, int maxdims, int *coords)int MPI_Cart_shift(MPI_Comm comm_cart, int dir,int s_step, int *rank_source, int *rank_dest)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

41 / 64

Overlapping Computation andCommunication

• Non-blocking Communicationint MPI_Isend(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request *request)

int MPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Request *request)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

43 / 64

Overlapping Computation andCommunication

• Non-blocking Communicationint MPI_Test(MPI_Request *request, int *flag, MPI_Status *status)

int MPI_Wait(MPI_Request *request, MPI_Status *status)

int MPI_Request_free(MPI_Request request)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

45 / 64

Avoiding Deadlock• Using non-blocking communication

int a[10], b[10], npes, myrank;MPI_Status status;MPI_Request requests[2];…MPI_Comm_rank(MPI_COMM_WORLD, &myrank);If (myrank%2 == 0) { MPI_ISend(a, 10, MPI_INT, 1, 1, MPI_COMM_WORLD); MPI_ISend(b, 10, MPI_INT, 1, 2, MPI_COMM_WORLD); }else { MPI_IRecv(b, 10, MPI_INT, 0, 2, &requests[0], MPI_COMM_WORLD); MPI_IRecv(a, 10, MPI_INT, 0, 1, &requests[1], MPI_COMM_WORLD); } …


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

46 / 64

Collective Operations

• CollectiveCommunication/Synchronization– Barrier

int MPI_Barrier(MPI_COMM comm)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

48 / 64


• Collective Communication– Broadcast

int MPI_Bcast(void *buf, int count, MPI_Datatype datatype, int source, MPI_Comm comm)

MPI_Bcast MPI_Bcast MPI_Bcast


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

50 / 64

Collective Operations• Collective Communication

– Reduction

int MPI_Reduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op,int target, MPI_Comm comm)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

52 / 64

Predefined Reduction OperationsOperation Meaning DatatypesMPI_MAX Maximum C int or floatMPI_MIN Minimum C int or floatMPI_SUM Sum C int or floatMPI_PROD Product C int or floatMPI_LAND Logical AND C intMPI_BAND Bitwise AND C int or byteMPI_LOR Logical OR C intMPI_BOR Bitwise OR C int or byteMPI_LXOR Logical XOR C intMPI_BXOR Bitwise XOR C int or byteMPI_MAXLOC Max value location Data pairsMPI_MINLOC Min value location Data pairs


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

53 / 64


• Collective Communication– Prefix

int MPI_Scan(void *sendbuf, void *recvbuf, int count, MPI_Datatype, MPI_Op op, MPI_Comm comm)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

54 / 64


• Collective Communication– Scatter

int MPI_Scatter(void *sendbuf, int sendcnt,MPI_Datatype sendtype, void *recvbuf, int recvcnt,MPI_Datatype recvtype, int source, MPI_Commcomm)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

57 / 64


– Gatherint MPI_Gather(void *sendbuf, int sendcount, MPI_Datatype senddatatype, void *recvbuf, int recvcount, MPI_Datatype recvdatatype, int target, MPI_Comm comm)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

58 / 64


– Allgatherint MPI_Allgather(void *sendbuf, int sendcount, MPI_Datatype senddatatype, void *recvbuf, int recvcount, MPI_Datatype recvdatatype, MPI_Comm comm)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

59 / 64


• Collective Communication– Gather Vector

int MPI_Gatherv(void *sendbuf, int sendcount, MPI_Datatype senddatatype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvdatatype, int target, MPI_Comm comm)

int MPI_Allgatherv(void *sendbuf, int sendcount, MPI_Datatype senddatatype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvdatatype, MPI_Comm comm)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

60 / 64


• Collective Communication

int MPI_Scatter(void *sendbuf, void sendcount, MPI_Datatype senddatatype, void *recvbuf, int recvcount, MPI_Datatype recvdatatype, int source, MPI_Comm comm)

int MPI_Scatterv(void *sendbuf, void *sendcounts,int *displs, MPI_Datatype senddatatype, void*recvbuf, int recvcount, MPI_Datatype recvdatatype,int source, MPI_Comm comm)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

61 / 64


• Collective Communication– All-to-All

int MPI_Alltoall(void *sendbuf, void sendcount, MPI_Datatype senddatatype, void *recvbuf, int recvcount, MPI_Datatype recvdatatype, MPI_Comm comm)

int MPI_Alltoallv(void *sendbuf, void *sendcounts, int *sdispls, MPI_Datatype senddatatype, void *recvbuf, int *recvcounts, int *rdispls, MPI_Datatype recvdatatype, MPI_Comm comm)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

62 / 64


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

64 / 64 http://www.mpi-forum.org/docs/mpi-11-html/node64.html#Node64


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

65 / 64 http://www.mpi-forum.org/docs/mpi-11-html/node64.html#Node64


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

65 / 64

Communicators

• Splitting general communicators

int MPI_Comm_split(MPI_Comm comm, int color,int key, MPI_Comm *newcomm)

3


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

66 / 64

Communicators• Splitting Cartesian communicators

int MPI_Cart_sub(MPI_Comm comm_cart, int *keep_dims, MPI_Comm *comm_subcart)

1

2

3


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

67 / 64

Evaluating Parallel Programs

• Parallel Execution Time– tp = tcomp + tcomm

• Communication Time– tcomm = tstartup + ntdata

– Or for all communication…q messages– tcomm = q(tstartup + ntdata)


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

68 / 64

Timing in MPI• Inside the code

– Fortran – thread time• dtime• etime

– C• times()

– C – wallclock time• MPI_wtime


Assignment




Introduction

Topologies

Overlapping C&C

Basics

Collective Comms.

Communicators

MPI Interface

69 / 64

Analyzing and DebuggingParallel Programs

introduction basics mpi interface topologies message passing...

Documents