1

　

Lecture 4: Distributed-memory Computing with PVM/MPI

2

Cluster Computing

What is cluster computing Use a number of work stations or PCs to construct a virtual parallel

computer.

Comparing with parallel computers, cluster processing is cheap. Performance is worse than parallel computers, but it can be largely

improved by using high performance network.

3

Global/Cloud Computing

What is global (GRID) computing Use all the resources on internet to construct a super distributed parallel virtual computer

Intranet +

Internet

Super parallelcomputers

PC clusterUser

High performance PC

4

Implement of cluster processing

Standard software for cluster processing 　 PVM (Parallel Virtual Machine) 　 MPI (Message Passing Interface)

History of PVM

1989 　 PVM1.0 released by Oak Ridge National Laboratory.

1991 　 PVM2.0 released by University of Tennessee.

1993 　 PVM3.0 released.

1994 　 Published with free manual

2001PVM 3.4.3 released with C and Fortran.

PVM: A Users’ Guide and Tutorial for Networked Parallel Computing http://www.netlib.org/pvm3/book/pvm-book.html

5

Implement of cluster processing

History of MPI1993-1994 　 Specification (MP-I) was decided by the

experts from 40 companies.

1995 　 Specification of MPI-2 was decided.

2001 　 MPI-1.1 based implementation was released.

It is used as default environment in many distributed parallel processing systems.

6

Common in PVM and MPI

　 Communication is based on 1 to 1 message passing.

　 They are designed for not only work stations and PCs but also parallel computers.

　 Corresponding to many OS and Languages.

（ OS: UNIX, Windows95/NT, Languages: C, Fortran, JAVA) 　 Free 　 Standard software in distributed and parallel computing.

send(5, message） receive(3)

Processor ID: ３ Processor ID: ５

7

Construction of PVMSoftware of PVM

Daemon pvmd used for the communication between

processors. Console pvm used for constructing virtual parallel computer

（ xpvm is the console combined with GUI) Library of functions such as pvm_send and pvm_receive for message sending and

receiving.

On any computer, one can use pvm or xpvm to construct a virtual parallel computer, and then make and execute programs.

A virtual parallel computer

pvmd pvmd

pvmdpvmd

(opty1) (optima)

(opty2) (opty3)

（ Constructed by connected 4 computers ）

8

Start of PVM

(1) On any computer, use command pvm or xpvm to start pvmd.

pvmd

(opty1) (optima)

(opty2) (opty3)

On optima, use pvm or xpvm to start pvmd .

9

Start of PVM

(2) 　 On console pvm add computers to construct a virtual parallel computer. (At the this time, pvmd is started in these computers.)

pvmd

(opty1) (optima)

(opty2) (opty3)

pvmd

pvmdpvmd

On optimaadd opty1, opty2, opty3 to virtual parallel computer by pvmd.

10

Execution of PVM Program execution on PVM

(1) 　 When the program which contains communication functions of pvm is executed, the specified programs will be executed by pvmd

pvmd pvmd

pvmdpvmd

(opty1) (optima)

(opty2) (opty3)

prog1

prog1 prog1

prog1

When program prog1 is executed on optimawhich contains communication functions, the process (in this case prog 1) will be started in all processors by pvmd.

11

Execution of PVM

(2) Once the execution of a program started, communication and processing inside the program will be done automatically until the end.

pvmd pvmd

pvmdpvmd

(opty1) (optima)

(opty2) (opty3)

prog1

prog1 prog1

prog1

Processing is held with the communication between processors.

12

Using PVM Using command pvm

chen[11]% pvmpvm> add opty1 opty2 opty3 （ add opty1, opty2, opty3 to the virtual computer ）3 successful HOST DTID opty1 80000 opty2 c0000 opty3 100000pvm> conf 　 (Show the configuration of the virtual parallel computer)4 hosts, 1 data format HOST DTID ARCH SPEED DSIG optima 40000 X86SOL2 1000 0x00408841 opty1 80000 X86SOL2 1000 0x00408841 opty2 c0000 X86SOL2 1000 0x00408841 opty3 100000 X86SOL2 1000 0x00408841pvm> spawn edm4 　（ Execute program edm4 on the virtual computer)1 successfult80001pvm> halt 　　 （ Halt of the virtual computer ）Terminated

chen[12]%

XPVM Screen Shot

14

Using XPVM

Using xpvm (X application)

15

PVM programming – Example 1 (Greeting) 1. Example (hello.c, hello_other.c)(hello.c) #include "pvm3.h" main() { int tid, answer; pvm_spawn("hello_other",

(char**)0,0,"",1, &tid); printf("Hello, Dr.t%x\n",tid); pvm_recv(tid,1); pvm_upkint(&answer,1,1); printf("Fine, thank you,

Dr. t%x", answer); pvm_exit(); }

(hello_other.c) #include "pvm3.h" main() { int mytid, ptid; mytid = pvm_mytid(); ptid = pvm_parent(); printf("Good morning, Prof. t%x.\n",ptid); printf("How are you?\n"); pvm_initsend(PvmDataDefault); pvm_pkint(&mytid,1,1); pvm_send(ptid, 1); pvm_exit(); }

16

PVM Programming – Example 1(Greeting)

Start the process hello

Hello, Dr. t80004

（ Receive the message by pvm_recv ） Fine, thank you, Dr. t80004

（ Process hello_other is started ） 　Hello, Prof. t4000a.

　How are you?

（ Send message by pvm_send ）

pvm_spawn

　 (Process hello_other is started on other computers by pvm_spawn.

　 Started process has its task ID.)

pvm_send

pvm_recv（ Finish the process by pvm_exit ）

（ Finish the process by pvm_exit ）

17

2. An Example -- Numerical Integral

Integral t

sdxxfI )( can be calculated approximately by I= )(

1

0

N

iixfh ,

where N

sth

and )10( )5.0( Nihisxi (Fig. 1). In this

approximate calculation, the larger the N is the more accurate the I is.

The ratio of the circumference dxx 1

0

214 can be calculated

approximately by

1

0

214N

iixh , where

Nh

1 and

)10( )5.0( Nihixi .

s t ix

)( ixf

Fig. 1 Numerical integral

PVM programming – Example 2 (Calculate π)

1

0

214 x

Equation of unit circle:Area of unit circle: π or

122 yx

1

0

214 x

18

/* Calculating PI by integral (Mater Program) */#include <stdio.h> /* For standard I/O functions¡ */#include "pvm3.h" /* For stand pvm functions */

main(argc,argv)int argc;char **argv;{ int intervals; /* Number of intervals */ int ctid[2]; /* task id (subtasks) */ int n; /* Number of intervals for one subtask */ double s,t; /* Interval of tasks */ double result1,result2; /* Results of subtasks */

if(argc<=1){ /* Finish if no any parameter */ printf("Usage:pi_dual intervals\n"); pvm_exit(); exit(0); }

sscanf(argv[1],"%d",&intervals); /* Get the first parameter and put it into intervals */ if(pvm_spawn("pi_slave",(char **) 0,0,"",2,ctid)!=2){ /* Start two subtasks for executing slave program pi_slave¡ Save the subtask ids into arry ctid */ printf("can't start pi_slave\n"); /* Fail in starting the subtasks */ pvm_exit(); exit(0); }

n=intervals/2; /* Interval for one subtask */ s=0;t=0.5; /* First subtask responsible for interval [0,0.5] */ pvm_initsend(0); /* Prepare for data sending */ pvm_pkint(&n, 1 ,1); /* Put the interval into the sending buffer */ pvm_pkdouble(&s, 1, 1); /* Put the value of s into the sending buffer */ pvm_pkdouble(&t, 1 ,1); /* Put the value of t into the sending buffer */ pvm_send(ctid[0], 1); /* Send the content of the buffer to the first subtask */

s=0.5;t=1; /* Secong subtask responsible for interval [0.5,1] */ pvm_initsend(0); /* Prepare for data sending */ pvm_pkint(&n, 1 ,1); /* Put the interval into the sending buffer */ pvm_pkdouble(&s, 1, 1); /* Put the value of s into the sending buffe */ pvm_pkdouble(&t, 1 ,1); /* Put the value of t into the sending buffe */ pvm_send(ctid[1], 1); /* Send the content of the buffer to the second subtask */

pvm_recv(-1, 1); /* Waiting for receiving the data from the subtasks, and put the data into the receiving buffer when received */ pvm_upkdouble(&result1,1,1); /* Put the received value (the result from the first subtask) into result1*/ pvm_recv(-1, 1); /* Waiting for receiving the data from the subtasks, and put the data into the receiving buffer when received */ pvm_upkdouble(&result2,1,1); /*Put the received value (the result from the second subtask)into result2*/ printf("Intervals=%d, PI=%1.15f",intervals,result1+result2); /* Show the result */ pvm_exit(); /* End */ exit(0);}

Master Program (pi_dual.c)

PVM programming – Example 2 (Calculate π)

19

/* Program for calculating PI by integral (slave program) */#include <math.h> /* Using numerical fuctions (sqrt, etc.) */#include "pvm3.h" /* Using pvm functions */

main(){ int ptid; /* Master task id */ int n; /* Number of intercals */ double s,t; /* Interval of integral */ int i; /* Variable for loop */ double h; /* Width of one interval */ double sum=0; /* Accumulated value of the area. Initial value=0 */ double x; /* x coordinate used at present */ ptid = pvm_parent(); /* put master task id into ptid */ pvm_recv(ptid, 1); /* Wating for receiving the data from master task, and put the data into the receiving buffer when received */ pvm_upkint(&n, 1, 1); /*Put the interval assigned for one task into n */ pvm_upkdouble(&s, 1 ,1); /*Put the left endpoint of the interval to s*/ pvm_upkdouble(&t, 1 ,1); /* Put the right endpoint into t */ h=(t-s)/n; /* Width of small intervals */ for (i=0;i<n;++i){ /* Repeat the loop n times */ x=(i+0.5)*h+s; /* Midpoint of x coordinate */ sum+=(4*sqrt(1-x*x))*h; /* Calculate the area of one small interval and add it to sum. */ } pvm_initsend(0); /* Prepare for sending data */ pvm_pkdouble(&sum, 1, 1); /* Put sum into the sending buffer */ pvm_send(ptid, 1); /* Send the content of the sending buffer into the master process */ pvm_exit(); /* End */ exit(0);}

Slave Program (pi_slave.c)

PVM programming – Example 2 (Calculate π)

20

MPI programming – Example 1(Greeting)

master

slave

21

MPI programming – Example 2 (Calculate π)#include "mpi.h" #include <stdio.h> #include <math.h> int main( int argc, char *argv[] ) { int n, myid, numprocs, i; double PI25DT = 3.141592653589793238462643; double mypi, pi, h, sum, x; MPI_Init(&argc,&argv); MPI_Comm_size(MPI_COMM_WORLD,&numprocs); MPI_Comm_rank(MPI_COMM_WORLD,&myid); while (1) { if (myid == 0) { printf("Enter the number of intervals: (0 quits) "); scanf("%d",&n); } MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD); if (n == 0) break; else { h = 1.0 / (double) n; sum = 0.0; for (i = myid + 1; i <= n; i += numprocs) { x = h * ((double)i - 0.5); sum += (4.0 / (1.0 + x*x)); } mypi = h * sum; MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); if (myid == 0) printf("pi is approximately %.16f, Error is %.16f\n", pi, fabs(pi - PI25DT)); } } MPI_Finalize(); return 0; }

one task

reduction function for parallel summation

22

MPI programming – Reduction

PVM? MPI? PVM is easy to use, especially on a network of workstations. Its message

passing API is relatively simple MPI is a standard, has a steeper learning curve and doesn’t have a

standard way to start tasks MPICH does have an “mpirun” command

If building a new scalable, production code, should use MPI (widely supported now)

If experimenting with message passing, are interested in dynamics, use PVM.

MPI has a very rich messaging interface and designed for efficiency http://www-unix.mcs.anl.gov/mpi

PVM has a simple messaging interface + Process control, Interoperability, Dynamics: http://www.epm.ornl.gov/pvm

Perform comparably when on Ethernet MPI outperforms when on MPP Both are still popular, but MPI is an accepted community standard with

many support chains.

Exercise

24

(1) Read the PVM program pi_dual.c and pi_slave.c which are used for calculating. Compile pi_dual.c and pi_slave.c. Learn to use xpvm.

(2) Revise program pi_dual.c to pi_single.c such that it uses only one pi_slave task. Find N (the number of the divided intervals) such that the running time of pi_single is 30 sec, 60 sec, 90 sec, and 120 sec, respectively.

(3) Compare the running time of pi_dual and pi_single using the different values of N obtained in (2). Notice that the ideal speed-up rate is

(4) Revise program pi_dual.c to pi_multi.c such that it can use any number of slave tasks. For example,

Pvm> spawn -> pi_multi 200000000 4

starts 4 slave tasks which do calculation in intervals [0,0.25],[0.25,0.5],[0.5,0.75],[0.75,1], respectively, where each interval is divided to 50000000 smaller intervals.

(5) For each value of N obtained in (2), get the running time pi_multi when the number of slave tasks is 1,2,4,8,16, respectively, find the speed-up rate (notice that the ideal rate is, investigate the change in the speed-up rates, and discuss the reasons.

(6) Consider how to balance the work load among the host computers. (Hint: according to the work load of each computer, the time needed for running pi_slave is different. Assign more slave tasks to the computers which has lower work load. Design the PVM program to realize your idea, and discuss how much the running time and the speed-up rate are improved.

(7) Change the above PVM programs into the MPI programs.