integrative parallel programming in hpc

Download Integrative Parallel Programming in HPC

Post on 28-Nov-2014

277 views

Category:

Science

1 download

Embed Size (px)

DESCRIPTION

Lecture in the Georgia Tech "Hot CSE" seminar, 20140922

TRANSCRIPT

  • 1. Integrative Parallel Programming in HPC Victor Eijkhout 2014/09/22
  • 2. Introduction Motivating example Type system Demonstration Other applications Tasks and processes Task execution Research Conclusion GA Tech | 2014/09/22| 2
  • 3. Introduction GA Tech | 2014/09/22| 3
  • 4. My aims for a new parallel programming system 1. There are many types of parallelism ) Uniform treatment of parallelism 2. Data movement is more important than computation ) While acknowledging the realities of hardware 3. CS theory seems to ignore HPC-type of parallelism ) Strongly theory based IMP: Integrative Model for Parallelism GA Tech | 2014/09/22| 4
  • 5. Design of a programming system One needs to distinguish: Programming model How does it look in code Execution model How is it actually executed Data model How is data placed and moved about Three dierent vocabularies! GA Tech | 2014/09/22| 5
  • 6. Programming model Sequential semantics [A]n HPF program may be understood (and debugged) using sequential semantics, a deterministic world that we are comfortable with. Once again, as in traditional programming, the programmer works with a single address space, treating an array as a single, monolithic object, regardless of how it may be distributed across the memories of a parallel machine. (Nikhil 1993) As opposed to [H]umans are quickly overwhelmed by concurrency and
  • 7. nd it much more dicult to reason about concurrent than sequential code. Even careful people miss possible interleavings among even simple collections of partially ordered operations. (Sutter and Larus 2005) GA Tech | 2014/09/22| 6
  • 8. Programming model Sequential semantics is close to the mathematics of the problem. Note: sequential semantics in the programming model does not mean BSP synchronization in the execution. Also note: sequential semantics is subtly dierent from SPMD (but at least SPMD puts you in the asynchronous mindset) GA Tech | 2014/09/22| 7
  • 9. Execution model Virtual machine: data ow. Data ow expresses the essential dependencies in an algorithm. Data ow applies to multiple parallelism models. But it would be a mistake to program data ow explicitly. GA Tech | 2014/09/22| 8
  • 10. Data model Distribution: mapping from processors to data. (note: traditionally the other way around) Needed (and missing from existing systems such as UPC, HPF): distributions need to be
  • 11. rst-class objects: ) we want an algebra of distributions algorithms need to be expressed in distributions GA Tech | 2014/09/22| 9
  • 12. Integrative Model for Parallelism (IMP) Theoretical model for describing parallelism Library (or maybe language) for describing operations on parallel data Minimal, yet sucient, speci
  • 13. cation of parallel aspects Many aspects are formally derived (often as
  • 14. rst-class objects), including messages and task dependencies. ) Specify what, not how ) Improve programmer productivity, code quality, eciency and robustness GA Tech | 2014/09/22| 10
  • 15. Motivating example GA Tech | 2014/09/22| 11
  • 16. 1D example: 3-pt averaging Data parallel calculation: yi = f (xi1; xi ; xi+1) Each point has a dependency on three points, some on other processing elements GA Tech | 2014/09/22| 12
  • 17. ;
  • 18. ; distributions Distribution: processor-to-elements mapping distribution: data assignment on input distribution: data assignment on output
  • 19. distribution: `local data' assignment )
  • 20. is dynamically de
  • 21. ned from the algorithm GA Tech | 2014/09/22| 13
  • 22. Data ow We get a dependency structure: Interpretation: Tasks: local task graph Message passing: messages Note: this structure follows from the distributions of the algorithm, it is not programmed. GA Tech | 2014/09/22| 14
  • 23. Algorithms in the Integrative Model Kernel: mapping between two distributed objects An algorithm consists of Kernels Each kernel consists of independent operations/tasks Traditional elements of parallel programming are derived from the kernel speci
  • 24. cation. GA Tech | 2014/09/22| 15
  • 25. Type system GA Tech | 2014/09/22| 16
  • 26. Generalized data parallelism Functions f : Realk ! Real applied to arrays y = f (x): yi = f x(If (i)) This de
  • 27. nes function If : N ! 2N for instance If = fi ; i 1; i + 1g. GA Tech | 2014/09/22| 17
  • 28. Distributions Distribution is (non-disjoint, non-unique) mapping from processors to sets of indices: d : P ! 2N Distributed data: x(d) : p7! fxi : i 2 d(p)g Operations on distributions: g : N ! N ) g(d) : p7! fg(i) : i 2 d(p)g GA Tech | 2014/09/22| 18
  • 29. Algorithms in terms of distributions If d is a distribution, and (funky notation) x y x + y; x y x y the motivating example becomes: y(d) = x(d) + x(d 1) + x(d 1) and the
  • 30. distribution is
  • 31. = d [ d 1 [ d 1 To reiterate: the
  • 32. distribution comes from the structure of the algorithm GA Tech | 2014/09/22| 19
  • 33. Transformations of distributions How do you go from the to
  • 34.

Recommended

View more >