parallel matlab on hpc - usc · parallel matlab on hpc ... • using graphical user interface (gui)...

3/3/17

1

Parallel MATLAB on HPC

AvalonJohnson,JimiChui,CesarSulandErinShaw

USCCenterforHigh-PerformanceCompu6ng(HPC)HPCResearchandEduca6onFacilita6on

USC HPC§  Founda;onforcomputa;onalresearchatUSC

•  Sharedresourcethatisfreelyavailabletofaculty,researchersandgraduatestudents

•  Worldclasssupercompu;ngcenter-  Currentlyranked12thfastestacademiccluster

§  Research&Educa;onFacilita;on•  Drop-intoofficehoursorrequestaconsulta;on•  AKendanewusermee;ngorpar;cipateinaworkshop

-  Seeh6ps://hpc.usc.edu/educa;on-  Seeh6ps://hpc.usc.edu/officehours-  [email protected]

3/3/17

2

Outline§  Introduc;on

•  Prerequisites,setupworkspaceandlaunchMATLAB•  Introduc;ontoparallelMATLAB

§  RunMATLABonsinglenodew/ParallelCompu;ngToolbox(PCT)•  Usinggraphicaluserinterface(GUI)•  Usingcommandlineinterface(CLI)•  UsingCLIwithbatchparalleljobexecu;on

§  RunMATLABonmul;plenodesw/DistributedCompu;ngServer(DCS)•  Usingcommandlineinterface(CLI)

§  Beyondparfor•  labIndex,Composite,andBroadcast&Receive•  Prac;ceusingPBS

Outline§  Introduc6on





3/3/17

3

Prerequisites§  YouwillneedX11DisplayForwardingtousetheGUI

•  OnWindows:Set‘EnableX11forwarding’onyourclient(thisisusuallythedefault)

•  OnMacs:DownloadandinstallXquartz(www.xquartz.org)

§  Login

§  Run‘xeyes’programtoverifythatdisplayisworking

$ ssh -X {USCNetId}@hpc-login3.usc.edu

$ xeyes #now move cursor$ ^C #control-c to quit

Workspace setup§  Alwaysworkinyourprojectdirectory!

§  Createamatlabworkshopdirectory&copyworkshopfilestoit

§  Youshouldnowhavethefollowingfiles

$ mkdir matlab$ cd matlab$ cp /home/rcf-proj/workshop/matlab/* .

$ cd /home/rcf-proj/{project}/{user}

$ lsbroadcastReceive.mEs;matePi.mjobScript.mlabIndex.msubmit.pbssubmitTasks.m

3/3/17

4

Request an interactive compute node

-l procs=8Use“procs”insteadoftypical“nodes=x:ppn=x”DistributesMATLABprocessesacrossnodesmostefficiently

-l software=MPCT+1 Checkout1MATLABPCTlicenseforinterac;veapplica;on(client)Eachcoreneedsalicense–MATLABwillrequestmorelicensesperourconfigura;on

-l advres=HPCCWorkshop.xx –A workshopUsenodesreservedinadvanceforthisworkshopOmittheselinesaXerworkshop

NOTE:Whenyourunrealjobs,ifyouhaveaccesstoacondoqueue,besuretoincludethe–q<queue_name>op;on

$ qsub -I l walltime=02:00:00 –l procs=8 –l software=MPCT+1 -A workshop -l ‘advres=HPCWorkshop.xx’

[csul@hpc-login3 ~]$ qsub I -l 'walltime=1:00:00’-l 'procs=8’ –l 'software=MPCT+1’ qsub: waiting for job 14526397.hpc-pbs.hpcc.usc.edu to start qsub: job 14526397.hpc-pbs.hpcc.usc.edu ready ---------------------------------------- Begin PBS Prologue Tue Apr 5 13:33:13 PDT 2016 Job ID: 14526397.hpc-pbs.hpcc.usc.edu Username: csul Group: ucsadmin Project: default Name: STDIN Queue: quick Shared Access: no All Cores: no Has MIC: no Nodes: hpc2289 TMPDIR: /tmp/14526397.hpc-pbs.hpcc.usc.edu End PBS Prologue Tue Apr 5 13:33:13 PDT 2016 ----------------------------------------

Receive an interactive compute node

3/3/17

5

Launch MATLAB in second window

Thisisyouroriginalhpc-loginshell

When‘qsub’succeeds,acomputenodeshellwillopeninthiswindow

DonotrunMATLABwithGUIinthisshell,itwillbeveryslow;runinnewterminalwindowinstead

Usethiswindowtoedityourprogramandmonitoryourjob

Openasecondterminalwindowandlogintoheadnode(with–X) $ ssh –X hpc-login3.usc.edu

ThenlogintocomputenodeandrunMATLABfromhere $ ssh –X hpcxxxx $ source /usr/usc/matlab/

R2016a/setup.sh $ matlab&

Outline§  Introduc6on

•  Prerequisites,setupworkspaceandlaunchMATLAB•  Introduc6ontoparallelMATLAB




3/3/17

6

How MATLAB supports parallelization§  Clusterobjects

parallel.cluster.Local( interactwithclusteronlocalmachine)parallel.cluster.Torque(interactwithclusterrunningTorque)parallel.cluster.Mpiexec(interactwithclusterusingmpiexec)

§  ParallelCompu;ngToolbox(USClicensed)parpool+parfor

§  DistributedCompu;ngServer(USClicensed)batchparfor,spmd(singleprogram,mul;pledata)jobs,spmdMPIcommunica;on

MATLAB’s parallel machinery Loginnode

3/3/17

7

MATLAB program: EstimatePi.m% Calculate the value of Pi using a Monte Carlo simulation max=1e9; tic; n=0; parfor i = 1:max

x=rand; y=rand; if (x^2 + y^2 < 1.0)

n=n+1; end

end elapsedTime = toc; pi = (4.0 * n / max);

MATLAB’s parfor loop

3/3/17

8



§  RunMATLABonsinglenodew/ParallelCompu6ngToolbox(PCT)•  Usinggraphicaluserinterface(GUI)•  Usingcommandlineinterface(CLI)•  UsingCLIwithbatchparalleljobexecu;on



Run on single node with PCT (GUI)MATLAB’s Parallel Computing Toolbox (MPCT)

§  Insecondterminalwindow

x

§  TherearethreestepsforconfiguringMATLAB’sPCT1.Setclusterprofile(SingleNodeProfile_GUI)

TellMATLABwhichPBSresourcesareavailable

2.SetparallelpoolpreferencesTellMATLABwhichclusterprofileandhowmanyworkerstouseforthisrun

3.Startparallelpoolandexecuteprogram

§ WhenMATLABready,setpathtoyourmatlabdirectory

$ ssh -X hpcxxxx$ source /usr/usc/matlab/R2016a/setup.sh #add matlab to path$ matlab

3/3/17

9

1. Set cluster profile (GUI)

3/3/17

10

3/3/17

11

§  Set“Descrip;on”(op;onal)§  Set“NumWorkers”

#workers=#cores(procs)§  Set“JobStorageLoca;on”

whereresultswillgo§  Closewindow

2. Set parallel pool preferences (GUI)

§  AccessPreferencespanelfromtoptaborlower-lepbuKon§  Set“DefaultCluster”

selectaprofiletouse§  Set“Preferrednumberofworkersinaparallelpool”

#workers=#cores–1(i.e.,leaveonecoreforMATLAB)§  Closewindow

3/3/17

12

3. Start parallel pool (GUI)§  Select“Startparallelpool”

•  Parallelbarswillblinkun;lready•  Hoveroverforrunstatus

§  In“CurrentFolder”window•  Selectscript(es;matePI.m)•  OpensinEditorwindow

§  Runprogram(greentriangle)•  Type$toptoseerunningprocesses

§  ExitMATLAB



§  RunMATLABonsinglenodew/ParallelCompu6ngToolbox(PCT)•  Usinggraphicaluserinterface(GUI)•  Usingcommandlineinterface(CLI)•  UsingCLIwithbatchparalleljobexecu;on



3/3/17

13

CLI: Run on single node with PCT§  Next,we’lltakethesamethreestepsusingcommandline

•  Remember,weares;llrunninginterac;velyonacomputenode

§  StartMATLABwithoutGUI

$ source /usr/usc/matlab/R2016a/setup.sh$ matlab –nodesktop -nodisplay

MATLAB is selecting SOFTWARE OPENGL rendering. < M A T L A B (R) > Copyright 1984-2015 The MathWorks, Inc. R2015a (8.5.0.197613) 64-bit (glnxa64) February 12, 2015 To get started, type one of these: helpwin, helpdesk, or demo. For product information, visit www.mathworks.com.

Academic License >>

1. Set cluster profile (CLI)§  Createclusterprofileforsingle-nodeparalleljob

•  TellsPCTwhichPBSresourcesareavailable

•  Setjobworkdirectory(singlequotesrequired)

•  Setnumberofworkerstoqsubprocsvalue(PCTmax=12)

>> MPCTprofile = parallel.cluster.Local

>> MPCTprofile.JobStorageLocation = '/home/rcfproj/{project}/{user}/{matlab_dir}'

>> MPCTprofile.NumWorkers = 8

3/3/17

14

Set cluster profile§  Youcansaveyourprofileandreuseit

•  Thisprofileissavedin.matlab,andisre-usable•  Ifyouhavealreadycreatedthis,next;meyourun,insteadofalltheformercommands,justsay:

§  Atthispointwecaneither•  RunEs;matePi.mscriptinterac;velyintheforegroundorrunitinthebackgroundasabatchjob

>> MPCTprofile.saveAsProfile('SingleNodeProfile')

>> MPCTprofile = parcluster('SingleNodeProfile')

2. Set parallel pool and execute program%Start parpool (specify cluster profile and workers)>> parpool(MPCTprofile,7)

Starting parpool ….connected to 7 workers. %Run program >> estimatePi

ans = Elapsed time = …...

%Check values >> pi

pi = 3.1416

%Always clean up >> delete(gcp(‘nocreate’))

3/3/17

15



§  RunMATLABonsinglenodew/ParallelCompu6ngToolbox(PCT)•  Usinggraphicaluserinterface(GUI)•  Usingcommandlineinterface(CLI)•  UsingCLIwithbatchparalleljobexecu6on



MATLAB batch parallel job execution§  YoucanspawnjobsfromwithinMATLAB

•  Abatchjobwillruninthebackground§  MATLABhasitsownschedulerforbatchjobs

•  Forexample,ifyouhadtwoindependenttasks

§  RunabatchjobusingPCT>> job = batch(MPCTprofile, 'EstimatePi', ’pool', 7)

3/3/17

16

MATLAB batch parallel job execution§  Toviewjobstate

§  Toviewtheresults

§  Typevariablenamestoseeresults

>> job %view state of job

>> pi pi = 3.1416

>> elapsedTime elapsedTime = 1.9440

>> load(job)




§  RunMATLABonmul6plenodesw/DistributedCompu6ngServer(DCS)•  Usingcommandlineinterface(CLI)


3/3/17

17

Run on multiple nodes (GUI)MATLAB’s Distributed Computing Server (MDCS)

§  Requesttwonodes

§  Opennewwindow

f

§  TherearethreestepsforMATLABMDCSconfigura;on1.Setclusterprofile(Mul;NodeProfile_GUI)

Setupresourcetemplate:TellMATLABhowtopassresourcestoPBS

2.SetparallelpoolpreferencesTellMATLABwhichclusterprofileandhowmanyworkerstouseforthisrun

3.Startparallelpoolandexecuteprogram

$ ssh –X {user_id}:hpc-login3.usc.edu$ ssh -X hpcxxxx$ matlab

qsub -I l walltime=01:00:00 –l nodes=2:ppn=8 –l software=MPCT+1 -A workshop -l ‘advres=HPCWorkshop.xx’

§  OpenMainMenu=>Environment=>Parallel=>ManageClusterProfiles§  ChooseTorque§  Renameto“Mul;NodeProfile_GUI”§  Edit

1. Set cluster profile

3/3/17

18

Set resource template

§  SetJobStorageLoca;on§  SetNumWorkers§  SetResourceTemplate

-lprocs=^N^-sopware=MDCS+^N^-mabe–M{user_id}@usc.edu

2. Set parallel pool preferences

§  AccessPreferencespanelfromtoptaborlower-lepbuKon§  Set“DefaultCluster”§  Set“Preferrednumberofworkersinpool”

#workers=#cores–1§  Closewindow

3/3/17

19

3. Start parallel pool§  Select“Startparallelpool”

•  Parallelbarswillblinkun;lready•  Selectscript(Es;matePI.m)andrun

§  Howtocheckyourmul;nodeprocess?•  $myqueue

-  Onejobisourstdinjob(interac;veshell)-  Onejobismatlab’swhichhasrequested15cores(andreceivedtwonodes)andiswai;ng

-  Youcansshtoanyoneofthesenodesandrun$top

§  ExitMATLAB




§  RunMATLABonmul6plenodesw/DistributedCompu6ngServer(DCS)•  Usingcommandlineinterface(CLI)


3/3/17

20

Run on multiple nodes (CLI)§  Next,we’lltakethesestepsusingthecommandline§  StartMATLABwithoutGUI

$ source /usr/usc/matlab/R2016a/setup.sh$ matlab –nodesktop -nodisplay

MATLAB is selecting SOFTWARE OPENGL rendering. < M A T L A B (R) > Copyright 1984-2015 The MathWorks, Inc. R2015a (8.5.0.197613) 64-bit (glnxa64) February 12, 2015 To get started, type one of these: helpwin, helpdesk, or demo. For product information, visit www.mathworks.com.

Academic License >>

Setting MDCS profiles

§  CreateclusterprofileforMATLAB’sDCS•  TellsMATLABhowtotalktomul;plenodes

§  Saveprofile,andreuselater

>> MDCSprofile = parallel.cluster.Torque >> MDCSprofile.JobStorageLocation =

'/home/rcfproj/{project}/{user}/{matlab_dir}'

>> MDCSprofile.saveAsProfile(’MultiNodeProfile')

>> MPCTprofile = parcluster(’MultiNodeProfile')

3/3/17

21


§  SetResourceTemplate

§  Note:Thereisnowall;mespecified§  Nwillbedeterminedbyuslater

>> MDCSprofile.ResourceTemplate=' -l procs=^N^ -l software=MDCS+^N^ -m abe -M {user_id}@usc.edu'


§  MATLABalsoneedstoknow•  Howtologintocomputenodes(ssh)•  Howtotransferfiles(scp)

§  Saveyourprofile

>> MDCSprofile.RshCommand='ssh' >> MDCSprofile.RcpCommand='scp'

>> MDCSprofile.saveAsProfile('MultiNodeProfile')

3/3/17

22

Using MDCS profiles§  Usingyourpreviouslysavedprofile

§  Setmaximumwall;me

§  Ifyouhaveawall;meyouuseopenyoucansaveandrestoretheprofile

>> MPCTprofile = parcluster('OneHr_MultiNodeProfile ')

>> MDCSprofile.SubmitArguments=’ l walltime=1:00:00'

>> MDCSprofile.saveAsProfile('OneHr_MultiNodeProfile')

>> MDCSprofile = parcluster('MultiNodeProfile')

Launching an MDCS job§  Howmanycorestorequest?

•  Mostnodesinmainqueuehaveeither8or16corespernodesomul;plesof8(8,16,24,32…)or16(16,32,48…)arebest

•  Makesiteasierforjobschedulertofindyourresources,jobstartssooner

§  Launchabatchjob

•  16cores,1forMATLABplus15workers•  MATLABhandlesqsubdetails•  NOTE:YoucancloseyourMATLABsessionandinterac;vesessionatthispointifyouaresavingresultstofile(butdon’tdofortutorial)

•  Check$myqueue

>> job = batch(MDCSprofile,'EstimatePi', 'parpool', 15)

3/3/17

23

Multinode job: Checking status>> jobs=findJob(MDCSprofile) jobs= 8x1 Job array: ID Type State FinishTime Username Tasks -------------------------------------------------------------- 1 5 independent running csul 1 2 6 pool finished Apr 18 11:01:49 csul 7 3 7 pool finished Apr 22 15:05:07 csul 3 4 8 pool finished May 27 10:04:36 csul 32 5 9 pool finished May 27 10:04:27 csul 32 6 10 pool finished May 27 10:35:18 csul 32 7 11 pool finished May 27 10:36:23 csul 32 8 12 pool finished May 27 11:39:00 csul 32

Multinode job: Retrieving results

>> load(jobs(12)) >> pi pi =

3.1416 >> elapsedTime elapsedTime =

42.3374

3/3/17

24





§  Beyondparfor•  labIndex,Composite,andBroadcast&Receive•  Prac6ceusingPBS

Practice using PBS§  Thereareprac;cescriptsin/home/rcf-proj/workshop/matlab

•  Copythemtoyourmatlabworkshopdirectory

§  Normallyyouwoulddothefollowing1.  CreateaPBSscript2.  Submitabatchjobfromheadnode(e.g.,qsubsubmit.pbs)

§  Youcantrythisnow-  Exitcomputenode(youshouldbeonheadnode)-  Thescriptssubmit.pbsandjobScript.mshowhowtosubmitEs;matePito

thequeue

3/3/17

25

Submit EstimatePi.m to queue

#!/bin/bash#PBS-lnodes=1:ppn=1#Launchmatlab,workerswillruninanotherjob#PBS-Nmatlab_launchersource/usr/usc/matlab/R2016a/setup.shcd$PBS_O_WORKDIRmatlab-nodisplay-nosplash-nodesktop-r"jobScript"

parpool('Mul;NodeProfile_GUI',7)%Increaseworkersasdesired%ReplacewithotherexamplesEs;matePi%IncreasesizeofproblemasdesiredpielapsedTimedelete(gcp('nocreate'))

submit.pbs jobScript.m

Beyond parfor§  parforisthemostbasicwaythatMATLABcan

parallelizeyourcode§  Some;mesyou'llneedtohavemorecontrolover

whatgetsrunoneachworker§  Herearesomeexamplestogiveyouanideaofthe

capabili;esofMATLAB’sParallelCompu;ngToolboxandDistributedCompu;ngServer

3/3/17

26

Beyond parfor§  labIndex§  Composite§  Broadcast&receive§  Lookatthescriptsyoucopiedfrom/home/rcf-proj/

workshop/matlab§  Seeh6p://www.mathworks.com/help/distcomp/execute-

simultaneously-on-mul;ple-data-sets.html?requestedDomain=www.mathworks.com

submitTasks.m %Demotasksubmission%Thisscriptshowyouhowtocreatejobandtask%andsubmitthem%loadthedefaultclusterprofilec=parcluster;%Youcanspecifyadifferentprofileifyoulike%c=parcluster('Mul;NodeProfile_CLI')%createsubmitargumentsifyouwanttorunthisasamul;nodejob%c.SubmitArguments='-lwall;me=00:30:00-E';%createjobj=c.createJob;a=rand(4);

%createtwotasksj.createTask(@sin,1,{a});j.createTask(@cos,1,{a});%runbothtasksatthesame;mej.submit;%wait;lljobfinishbeforefetchingresultsj.wait('finished');pause(20);%fetchoutputresult=j.fetchOutputs;f=[result{:}];%savedatasave('submitTasks.out','f','-ascii');

3/3/17

27

labIndex.m %Demoofparpool,Compositeandlabindex

%ThisscriptdemosparpoolandComposit,labindex%itmustberunasamul;-nodejob%loadthedefaultclusterprofile%c=parcluster;%Youcanspecifyadifferentprofileifyoulikec=parcluster('Mul;NodeProfile_CLI')%createsubmitargumentsc.SubmitArguments='-lwall;me=00:30:00-E';%request3workersc.parpool(3)

%computeprocessa=Composite();spmdtemp=labindex*ones(10);a=temp*temp;end

%saveworkerresultsd=[a{:}];Save(’labIndex.out','d','-ascii');%cleanupdelete(gcp('nocreate'))

broadcastReceive.m %Demolabindexandmpicommunica6on

%demolabindexandmpicommunica;on,send,receive,broadcast%Thisscriptmustberunasamul;-nodejob%loadthedefaultclusterprofile%c=parcluster;%Youcanspecifyadifferentprofileifyoulikec=parcluster('DefaultMul;NodeProfile')%createsubmitargumentsc.SubmitArguments='-lwall;me=00:30:00-E';c.parpool(3)

%Setupa'singleprogram,mul;pledata'environmentspmdswitchlabindex%Iflabindexis1,1:5toscreen%iflabindexis2,receivedatafrom1onwhattoprinttoscreen%otherwiseprint11:15toscreencase1A=1:5;labSend(A,2,199);%sendtoworker2case2A=labReceive(1,199);%receivefromworker1otherwiseA=11:15;%createhisownendendA{:}1/3 2/3

3/3/17

28

broadcastReceive.m spmd%iflabindexis1,print6:10toscreen%otherwisereceivedatafrom1onwhattoprinttoscreeniflabindex==1A=labBroadcast(1,6:10);%broadcasttoallelseA=labBroadcast(1);%receivebroadcastendendA{:}delete(gcp('nocreate')) 3/3

End Notes§  MATLABdocumenta;on

h6p://www.mathworks.com/help/

§  MATLABlicenses•  Toviewavailability,typeoncommandline/auto/usc/matlab/R2016a/etc/glnxa64/lmu;[email protected]|grepMATLAB

•  ToviewgraphicallyhKps://hpc-monitoring.usc.edu/dashboards/queues.html

parallel matlab on hpc - usc · parallel matlab on hpc ... • using graphical user interface (gui)...

Documents