makefile::parallel - dependency specification language
TRANSCRIPT
![Page 1: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/1.jpg)
Makefile::Parallel
Dependency Specification Language
![Page 3: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/3.jpg)
"Well done is quickly done"Ceaser Augustus
![Page 4: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/4.jpg)
![Page 5: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/5.jpg)
![Page 6: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/6.jpg)
~ 100 64-bits cores~ 100Gb RAM~ 4TB storageMyrinet 10GbLinux (CentOS)
SeARCH Cluster
![Page 7: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/7.jpg)
How to run those processes?
3 options!
![Page 8: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/8.jpg)
Solution 1
while(! end) { run process $p while(! finished $p) { sleep n }
mark $p as done}
![Page 9: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/9.jpg)
Solution 2
#!/bin/foo
run p1run p2 result1run p3 result2run p4 result1 result2
![Page 10: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/10.jpg)
Solution 3
Makefile::Parallel
:-)
![Page 11: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/11.jpg)
M::P - Design Goals
Formal specification to describe process dependenciesReuse a well known language syntax (don't reinvent the wheel)Embed other languages to reuse their expressive powerMake it easy to run and maintainGenerate profiling dataHandle and recover from errors Run on different parallel / distributed computaing platformsHave fun and profit :-)
![Page 12: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/12.jpg)
M::P - The language
Started as a Makefile subsetEvolved with our own sugar syntaxSupport for parametric rules Perl
Parse::YAPP
![Page 13: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/13.jpg)
![Page 14: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/14.jpg)
M::P - Simple example
prepare: (5:00) mkdir OutputData p <- sub{ print "$_\n" for (3..10) } run$p: prepare (20:00:00) [2] runMyProgram -p $p InputData > OutputData/run.$p cleanup: run$p (5:00) for a in @p; do rm -f OutputData/run.${a}.tmp; done
![Page 15: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/15.jpg)
M::P - Simple exampleprepare: (5:00) mkdir OutputData p <- sub{ print "$_\n" for (3..10) } run$p: prepare (20:00:00) [2] runMyProgram -p $p InputData > OutputData/run.$p cleanup: run$p (5:00) for a in @p; do rm -f OutputData/run.${a}.tmp; done
![Page 16: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/16.jpg)
M::P - Simple exampleprepare: (5:00) mkdir OutputData p <- sub{ print "$_\n" for (3..10) } run$p: prepare (20:00:00) [2] runMyProgram -p $p InputData > OutputData/run.$p cleanup: run$p (5:00) for a in @p; do rm -f OutputData/run.${a}.tmp; done
![Page 17: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/17.jpg)
M::P - Scheduler
Try to run the processes in parallelMust be extensiveShould generate logs and visual profiling data
![Page 18: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/18.jpg)
PBS - Portable Batch System
Job Scheduling for Clusters
TorqueOpenPBS Common API
qsubqstatqdeltracejob
![Page 19: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/19.jpg)
M::P Scheduler - Main Loop
do { launch processes with fulfilled deps collect ended processes for each proc in ended_processes if proc has variables to expand calculate variables manipulate the dependency graph save journal sleep 10 seconds} while(!all processes completed)
![Page 20: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/20.jpg)
M::P GetOpts
-local=(n) Schedule on the local machine-pbs Schedule on a PBS capable cluster-continue Recover from the last error-dump Dumps the AST of the parsed specification-clean Remove temporary files-debug (no need to debug)
![Page 21: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/21.jpg)
M::P Profiling
Runtime logging: [...]2006/12/12 10:49:22 The job "ipfp005" is ready to run. Launching2006/12/12 10:49:22 Launched "ipfp005" (23996)2006/12/12 10:49:52 Process 23996 (ipfp005) has terminated [30s]2006/12/12 10:49:52 The job "postipfp005" is ready to run. Launching2006/12/12 10:49:52 Launched "postipfp005" (23997)2006/12/12 10:50:02 Process 23997 (postipfp005) has terminated [10s][...]
![Page 22: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/22.jpg)
M::P Profiling
Final report: ID Start Time End Time Elapsedcodify 2006-12-12T10:41:10 2006-12-12T10:49:11 8m 1sngramsA 2006-12-12T10:49:11 2006-12-12T11:07:46 18m 34sngramsB 2006-12-12T10:49:11 2006-12-12T11:05:44 16m 33sinitmat001 2006-12-12T10:49:11 2006-12-12T10:50:12 1minitmat002 2006-12-12T10:49:11 2006-12-12T10:50:43 1m 31sinitmat003 2006-12-12T10:49:11 2006-12-12T10:51:03 1m 51s[...]
![Page 23: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/23.jpg)
M::P Profiling
![Page 24: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/24.jpg)
M::P Error Handling
![Page 25: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/25.jpg)
M::P Error handling
![Page 26: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/26.jpg)
M::P Real World Usage
NLP process
~ 100 lines of Makefile::Parallel syntax~ 4Gb of textDesktop P4 3Ghz == 1 weekSeARCH cluster == 12 hours
![Page 27: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/27.jpg)
M::P Real World Usage
![Page 28: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/28.jpg)
M::P About the module
use Clone qw(clone);use Cwd;use Data::Dumper;use Digest::MD5;use GraphViz;use Log::Log4perl;use Parse::Yapp;use Proc::Reliable;use Proc::Simple;use Time::HiRes qw(gettimeofday tv_interval);use Time::Interval;use Time::Piece::ISO;
![Page 29: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/29.jpg)
M::P About the module
![Page 30: Makefile::Parallel - Dependency specification language](https://reader035.vdocuments.mx/reader035/viewer/2022081800/5592b60e1a28ab770d8b4747/html5/thumbnails/30.jpg)
M::P New features
0.4- support for multiple parametric variables on each rule- new bugs added
0.5- fixed a bunch of bugs- new bugs added
0.6- fixed a small (but important!) bug- ran out of new ideas