parallel computing in matlab

Parallel Computing in Matlab

2

PCT

• Parallel Computing Toolbox

• Offload work from one MATLAB session (the client) to other MATLAB sessions (the workers).

• Run as many as eight MATLAB workers (R2010b) on your local machine in addition to your MATLAB client session.

3

MDCS

• MATLAB Distributed Computing Server

• Run as many MATLAB workers on a remote cluster of computers as your licensing allows.

• Run workers on your client machine if you want to run more than eight local workers (R2010b).

4

MDCS installing

5

Typical Use Cases

• Parallel for-Loops– Many iterations– Long iterations

• Batch Jobs

• Large Data Sets

6

Parfor

• Parallel for-loop

• Has the same basic concept with “for”.

• Parfor body is executed on the MATLAB client and workers.

• The necessary data on which parfor operates is sent from the client to workers, and the results are sent back to the client and pieced together.

• MATLAB workers evaluate iterations in no particular order, and independently of each other.

7

Parfor

A = zeros(1024, 1);for i = 1:1024 A(i) = sin(i*2*pi/1024);endplot(A)

A = zeros(1024, 1);matlabpool open local 4parfor i = 1:1024 A(i) = sin(i*2*pi/1024);endmatlabpool closeplot(A)

parallelization

8

Timing

A = zeros(n, 1);matlabpool open local 8ticparfor i = 1:n A(i) = sin(i);endtoc

A = zeros(n, 1);ticfor i = 1:n A(i) = sin(i);endtoc

n for parfor10000 0.003158 0.040542

1000000 0.080678 0.221070

100000000 23.161180 14.230125

9

When to Use Parfor?

Each loop must be independent of other loops.

Lots of iterations of simple calculations.

orLong iterations.

Small number of simple calculations.

10

Classification of Variables

broadcast variable

sliced input variable

loop variable

reduction variable

sliced output variable

temporary variable

11

More Notes

d = 0; i = 0;parfor i = 1:4 b = i; d = i*2; A(i)= d;end

A [2,4,6,8]d 8

i 4

b 4

A [2,4,6,8]d 0

i 0

b /

d = 0; i = 0;for i = 1:4 b = i; d = i*2; A(i)= d;end

12

More Notes

C = 0;for i = 1:m for j = i:n

C = C + i * j; endend

How to parallelize?

13

Parfor: Estimating an Integral

14

Parfor: Estimating an Integral

function q = quad_fun( m, n, x1, x2, y1, y2 ) q = 0.0; u = (x2 - x1)/m; v = (y2 - y1)/n;

for i = 1:m x = x1 + u * i; for j = 1:n y = y1 + v * j; fx = x^2 + y^2; q = q + u * v * fx; end endend

15

Parfor: Estimating an Integral

• Computation complexity: O(m*n)

• Each iteration is independent of other iterations.

• We can replace “for” with “parfor”, for either loop index i or loop index j.

16

Parfor: Estimating an Integral

(m, n) 1 + 0 1 + 1 1 + 2 1 + 3 1 + 4(100, 100) 0.005 0.255 0.087 0.101 0.114

(1000, 1000) 0.035 0.066 0.046 0.045 0.053

(10000, 10000) 3.123 3.123 1.626 1.143 0.883

(100000, 100000) 308.282 309.926 157.393 108.819 85.185

ticA = quad_fun(m,n,0,3,0,3);toc

function q = quad_fun( m, n, x1, x2, y1, y2 ) q = 0.0; u = (x2 - x1)/m; v = (y2 - y1)/n;

parfor i = 1:m x = x1 + u * i; for j = 1:n y = y1 + v * j; fx = x^2 + y^2; q = q + u * v * fx; end endend

17

Parfor: Estimating an Integral

(m, n) 1 + 0 1 + 1 1 + 2 1 + 3 1 + 4(100, 100) 0.005 1.754 1.975 2.126 2.612

(1000, 1000) 0.035 13.146 15.286 18.661 22.313(10000, 10000) 3.123 113.368 139.568 178.425 220.155

(100000, 100000) 308.282

function q = quad_fun( m, n, x1, x2, y1, y2 ) q = 0.0; u = (x2 - x1)/m; v = (y2 - y1)/n;

for i = 1:m x = x1 + u * i;

parfor j = 1:n y = y1 + v * j; fx = x^2 + y^2; q = q + u * v * fx; end endend

ticA = quad_fun(m,n,0,3,0,3);toc

18

SPMD

• SPMD: Single Program Multiple Data.

– SPMD command is like a very simplified version of MPI.

– The spmd statement lets you define a block of code to run simultaneously on multiple labs, each lab can have different, unique data for that code.

– Labs can communicate directly via messages, they meet at synchronization points.

– The client program can examine or modify data on any lab.

19

SPMD Statement

20

SPMD Statement

21

SPMD

• MATLAB sets up the requested number of labs, each with a copy of the program. Each lab “knows" it's a lab, and has access to two special functions:– numlabs(), the number of labs;– labindex(), a unique identifier between 1 and numlabs().

22

SPMD

23

Distributed Arrays

• Distributed()– You can create a distributed array in the MATLAB client, and its data is

stored on the labs of the open MATLAB pool. A distributed array is distributed in one dimension, along the last nonsingleton dimension, and as evenly as possible along that dimension among the labs. You cannot control the details of distribution when creating a distributed array.

24

Distributed Arrays

• Codistributed()– You can create a codistributed array by executing on the labs

themselves, either inside an spmd statement, in pmode, or inside a parallel job. When creating a codistributed array, you can control all aspects of distribution, including dimensions and partitions.

25

Distributed Arrays

• Codistributed()– You can create a codistributed array by executing on the labs

themselves, either inside an spmd statement, in pmode, or inside a parallel job. When creating a codistributed array, you can control all aspects of distribution, including dimensions and partitions.

26

Example: Trapezoid

1 1( ) ( ) ( ) ( )2 2

b

af x dx f a f b b a

1 21 1( ) ( ) ( ) ( )2 2 1

b

na

b af x dx f x f x f xn

27

Example: Trapezoid

• To simplify things, we assume interval is [0, 1] , and we'll let each lab define a and b to mean the ends of its subinterval. If we have 4 labs, then lab number 3 will be assigned [ ½, ¾].

28

Example: Trapezoid

29

Pmode

• pmode lets you work interactively with a parallel job running simultaneously on several labs.

• Commands you type at the pmode prompt in the Parallel Command Window are executed on all labs at the same time.

• Each lab executes the commands in its own workspace on its own variables.

pmode spmd

Parallel computing synchronously

Each lab has a desktop No desktop for labs

Can’t freely interleave serialand parallel work

Can freely interleave serialand parallel work

30

Pmode

31

Pmode

• labindex() and numlabs() still work;

• Variables only have the same name, they are independent of each other.

32

Pmode

• Aggregate the array segments into a coherent array.

codist = codistributor1d(2, [2 2 2 2], [3 8])whole = codistributed.build(segment, codist)

33

Pmode

• Aggregate the array segments into a coherent array.

whole = whole + 1000section = getLocalPart(whole)

34

Pmode

• Aggregate the array segments into a coherent array

combined = gather(whole)

35

Pmode

• How to change distribution?

distobj = codistributor1d()I = eye(6, distobj)getLocalPart(I)

distobj = codistributor1d(1);I = redistribute(I, distobj)getLocalPart(I)

36

GPU Computing

• Capabilities– Transferring data between the MATLAB workspace and the GPU– Evaluating built-in functions on the GPU– Running MATLAB code on the GPU– Creating kernels from PTX files for execution on the GPU– Choosing one of multiple GPU cards to use

• Requirements– NVIDIA CUDA-enabled device with compute capability of 1.3 or greater– NVIDIA CUDA device driver 3.1 or greater– NVIDIA CUDA Toolkit 3.1 (recommended) for compiling PTX files

37

GPU Computing

• Transferring data between workspace and GPU

• Creating GPU data

N = 6;M = magic(N);G = gpuArray(M);M2 = gather(G);

38

GPU Computing

• Executing code on the GPU

– You can transfer or create data on the GPU, and use the resulting GPUArray as input to enhanced built-in functions that support them.

– You can run your own MATLAB function file on a GPU.

• If any of arg1 and arg2 is a GPUArray, the function executes on the GPU and return a GPUArray

• If none of the input arguments is GPUArray, then arrayfun executes in CPU.• Only element-wise operations are supported.

result = arrayfun(@myFunction, arg1, arg2);

39

Review

• What is the typical use cases of parallel Matlab?

• When to use parfor?

• What’s the difference between worker(parfor) and lab(spmd)?

• What’s the difference between spmd and pmode?

• How to build distributed array?

• How to use GPU for Matlab parallel computing?