developing parallel applications with the eclipse parallel...

BSC 2012 © 2012 IBM Corporation

Developing Parallel Applications with the

Eclipse Parallel Tools Platform

Greg Watson IBM STG

[email protected]

BSC 2012 © 2012 IBM Corporation 2

Parallel Tools Platform Enabling Parallel Application Development

parallel tools pla*orm (ptp) Tools to assist new

breed of programmers to develop parallel

programs

Best practice tools for experienced parallel

programmers

Improve parallel tools and the productivity of

tool developers

Leverage Eclipse ecosystem and community for

development and support

Provide focal point for parallel tool development for a broad

range of architectures


PTP Timeline

2005

2006

2008

2012

Eclipse 3.1

Callisto 3.2

Europa 3.3

Ganymede 3.4

Galileo 3.5

Helios 3.6

Indigo 3.7

PTP 5.0 June ’11

PTP 4.0 June ’10

PTP 3.0 Nov ’09

PTP 2.0 Mar ’08

PTP 1.0 Mar ’06

PTP 2.1 Nov ’08

PTP 1.1 Feb ’07

Creation Apr ’05

2010

2007

2009

18 committers from 11 different organizations

Nearly 65,000 downloads since Feb 24 “Eclipse for Parallel Application Developers”

SR1 Sep ’10

SR2 Feb ’11

2011

2013

PTP 6.0 June ’12

Juno 4.2

PTP 6.0 is part of the Eclipse “Juno” simultaneous release of over 70

Eclipse projects

SR1 Sep ’11

SR2 Feb ’12


PTP Application Development Cycle

4

Application Execution

Application Debugging

Coding & Static Analysis

Dynamic & Performance Analysis


  Eclipse provides a wide variety of coding assistance tools –  Project management, Editing and formatting, Navigation, Advanced

searching, Refactoring, Version control   C/C++ Development Tools (CDT)

–  Standard (Makefile) and managed builders, Support for arbitrary toolchains, Visual debugging using GDB, High level views (outline view, call hierarchy, type hierarchy, include browser), Refactorings

  Parallel Tools Platform (PTP) –  Fortran, New project wizards (MPI, OpenMP) Content Assist, Hover help,

Built-in API descriptions (MPI, OpenMP, LAPI, UPC), Location of parallel “artifacts” in code (MPI, OpenMP, PAMI, and UPC), Barrier analysis, Deadlock detection

5



  Assistance tools to increase productivity of parallel programmers

–  New project wizards (MPI, OpenMP)

–  Content Assist (command/API completion), hover help, built-in API help descriptions in an html help “view” (MPI, OpenMP, LAPI, UPC)

–  Location of parallel “artifacts” in code: MPI, OpenMP, LAPI APIs, and UPC

6

Coding & Static Analysis Parallel Language Development Tools (PLDT)


  Static analysis tools   Provides “advanced” error checking   Not always 100% accurate   MPI analysis

–  Barrier deadlock detection

  OpenMP analysis –  Concurrency analysis –  Common OpenMP problems

7

Coding & Static Analysis Parallel Language Development Tools (2)


Fortran Development Tools   Photran project

–  http://eclipse.org/photran –  Tech lead at UIUC, Jeff Overbey –  More UIUC students are contributing –  Merged with PTP in 2009 –  Photran 8.0 released in Jun ’12

with PTP 6.0 / Eclipse Juno   Photran features:

–  Supports Fortran 77-2008 –  Syntax-highlighting editor –  GUI interface to gdb –  Makefile-based compilation –  Compiler error extraction –  Outline view –  Open declaration –  Fortran refactorings –  C preprocessor support

8


Fortran Refactoring

9

Image courtesy Jeff Overbey, UIUC

Some samples:   Rename

–  Change name of every use of a variable, function, etc. –  Only proceeds if new name will be legal

  Extract procedure –  Moves statements into a new subroutine, replacing

statements with a call –  Local variables are passed as arguments

  Introduce implicit none –  Adds an ‘implicit none’ statement –  Adds explicit variable declarations

  Photran 7.0: 31 refactorings



10






Open MPI

11

Application Execution Launching & Monitoring

PBS/Torque

SLURM

MPICH2

  Improves visibility into target system   Single point of interface for launching and

control   Manages interaction with different runtime

systems and job schedulers

LoadLeveler

PE


  Extensible framework for launching & monitoring – System and node status information

– Job status (e.g. position in queue) & application status

– Job submission & control

– Debugger launch   Configuration files to support different resource managers

– Job schedulers (LoadLeveler, PBS, Torque, SLURM, GridEngine) –  Interactive runtimes (e.g. PE, Open MPI, MPICH2, MVAPICH)

– Systems (AIX, Linux, Power, x86, BG/Q, Cray)   Local or remote system support

– Command-line tools executed locally or via ssh connection

12

Application Execution Target Configuration Framework



13






Application Debugging PTP Parallel Debugger

  Mid-scale integrated debugger   Tightly integrated with Eclipse   Supports debugging multiple jobs

simultaneously   Utilizes backend debugger (e.g. gdb) for

low level operations   Targeted at SPMD programming models   Supports mixed MPI & thread

debugging   Single process and group operations   Platform for building new debugging

paradigms


  Scalable debugger using multicast reduction network   Integrated with PTP and launched using target configurations   Supports basic debug dommand   Uses gdb on backend

15

Application Debugging IParallel Debugger Architecture

process process

process

gdb

sdm

gdb gdb

process process

process

gdb

sdm

gdb gdb

process process

process

gdb

sdm

gdb gdb

sdm master



16






Dynamic & Performance Analysis Dynamic Analysis Tools

17

  Perform analysis on the running application using external tools

  Generate results that must be brought back into Eclipse as part of the development workflow

  May require external tool for visualization or other purposes


Dynamic & Performance Analysis Tools Integrated with PTP

  Tuning and Analysis Utilities (TAU) –  Instrumentation and transparent re-build of application

executable –  Execution of profiled application and collect performance data –  Performance data visible in UI

–  Launches paraprof visualization client from Eclipse   Graphical Explorer of MPI Programs (GEM)

–  Formal Dynamic Verification of MPI Applications

–  Detects all deadlocks, assert violations, MPI object leaks, and default safety properties

–  Matches sends and receives –  Allows post-verification review of highlighted bugs

–  Works with a variety of MPI implementations

18


Conclusion

19

  Complexity of peta-scale application development is becoming clearer to developers

–  IDEs are starting to be seen as key to improving HPC developer productivity

–  IDEs are becoming more accepted in the HPC community

  Eclipse is now a very mature platform –  Has more users than VS –  Hundreds of commercial products based on Eclipse

  PTP is the only available platform on which to build an HPC development environment

–  PTP developer community is starting to reach critical mass –  Beginning to see a diverse user base forming

  It’s Free!


Online Information

  Information about PTP –  Main web site for downloads, documentation, etc.

–  http://eclipse.org/ptp

–  Developers’ wiki for designs, planning, meetings, etc. –  http://wiki.eclipse.org/PTP

–  Articles and other documents –  http://wiki.eclipse.org/PTP/articles

20


Community

  PTP Mailing lists –  Major announcements (new releases, etc.) - low volume

–  http://dev.eclipse.org/mailman/listinfo/ptp-announce –  User discussion and queries - medium volume

–  http://dev.eclipse.org/mailman/listinfo/ptp-user –  Developer discussions - high volume

–  http://dev.eclipse.org/mailman/listinfo/ptp-dev

  Meetings –  Monthly developer conference calls

–  Announced on ptp-dev list –  Monthly user conference calls

–  Announced on ptp-user list –  Annual developer/user meeting

–  Tentatively scheduled for Sept 18/19 in Chicago

21


Demo

22


Thank You

23

developing parallel applications with the eclipse parallel...

Documents