development environments for hpc: the view from...

Development Environments for HPC: The View from NCSA

Jay Alameda National Center for Supercomputing Applications, University of Illinois at Urbana-Champaign

DEHPC ‘15

San Francisco, CA 18 October 2015

Acknowledgements ª  Portions of this material are supported by or based upon work supported by the

Defense Advanced Research Projects Agency (DARPA) under its Agreement No. HR0011-07-9-0002, the United States Department of Energy under Contract No. DE-FG02-06ER25752, the Blue Waters sustained petascale computing project, which is supported by the National Science Foundation under award number OCI 07-25070, and the SI2-SSI Productive and Accessible Development Workbench for HPC Applications, which is supported by the National Science Foundation under award number OCI 1047956

ª  The SI2-SSI team is lead by Jay Alameda (NCSA), Greg Watson (IBM), Steven Brandt (LSU), and Allen Malony (U Oregon). Team members and senior personnel include Beth Tibbitts (IBM), Ralph Johnson (U Illinois), Chris Navarro (NCSA), Sameer Shende (U Oregon), Wyatt Spear (U Oregon), Brian Jewett (U Illinois), Galen Arnold (NCSA), and Rui Liu (NCSA)

ª  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation

1

Outline

ª  Overview of Eclipse and the Parallel Tools Platform (PTP)

ª  Motivation for Workbench for High Performance Computing (WHPC)

ª  Advantages of Eclipse for HPC Software Developers

ª  Sustainability, Investment, Collaboration: Building Community

ª  Challenges with DEHPC

ª  Opportunities (for discussion)

2

Eclipse

ª Integrated development environment (IDE) ª Edit code, compile, run, debug without leaving Eclipse ª Graphical user interface

3

Eclipse

4

Eclipse

ª Integrated development environment (IDE) ª Edit code, compile, run, debug without leaving Eclipse ª Graphical user interface

ª Plug-ins add new functionality to Eclipse

ª Support languages: C/C++, Fortran, Python, … ª Support version control systems: Git, Subversion, … ª Support issue tracking systems: Bugzilla, Jira, … ª Support HPC development (?!)

5

Parallel Tools Platform (PTP)

ª PTP: a set of plug-ins that extend Eclipse to support HPC development ª Write code on your laptop

ª Build it on a remote HPC system using its compilers

ª Run it on the remote HPC system (qsub)

ª Debug it on the remote system (MPI debugger)

ª Tune it for performance on the remote system

ª …all inside Eclipse! 6

7

Eclipse Parallel Tools Platform (PTP) Coding & Analysis

Code completion

8

Launching & Monitoring

Eclipse Parallel Tools Platform (PTP) Coding & Analysis

9

Launching & Monitoring

Eclipse Parallel Tools Platform (PTP)

Debugging

Coding & Analysis

10

Launching & Monitoring

Eclipse Parallel Tools Platform (PTP)

Debugging

Coding & Analysis

Performance Tuning

11

Launching & Monitoring

Eclipse Parallel Tools Platform (PTP)

Debugging

Coding & Analysis

Performance Tuning

Motivation for Workbench for High Performance Computing (WHPC)

ª  Context ª NSF 1047956: SI2-SSI: A Productive and Accessible

Development Workbench for HPC Applications Using the Eclipse Parallel Tools Platform

ª  Stable, portable platform for tool development ª  Focus on tool functionality, manage rapid evolution of HPC

platforms ª  Encourage consistent tool look and feel ª  Support for HPC application development practices

ª  Why Parallel Tools Platform? ª High potential to meet needs of a WHPC. ª  Target next generation of HPC developers growing up with

IDEs (Eclipse, Visual Studio, ...) ª Need to cultivate community of users!

12

Improvements

ª Work within Eclipse release cycle ª Major (API-breaking) improvements with coordinated

June release ª Last major release Eclipse 4.5 “Mars” released

June 2015 ª Minor enhancements and bug-fixes with two

coordinated service releases in September and February ª Eclipse 4.5 SR1 Released September, 2015.

ª Foci of improvements ª Improve usability ª Improve productivity

13

Consider 2 possible types of users of Eclipse Parallel Tools … ª Science code users/modelers

ª Need to build science code ª May need to modify science code (and rebuild)

ª Software specialists enabling modeling projects ª Lots of software engineering concerns

ª Next set of slides address some of those concerns.

14

Science code users/modelers

ª Some of the challenges ª Complex codes (eg WRF) ª Codes + HPC architectures can be daunting ª Adding user code not always easy

ª WRF: from http://wrf-model.org/PRESENTATIONS/2000_04_18_Klemp/sld007.htm

15

Navigating Computational Science Codes

16

Code navigation

Syntax-aware editing (navigate to program units and declarations)

Code Outline

Eclipse aiding in a typical code workflow…

ª May want to add a model output variable ª Eclipse PTP makes it easy to navigate source,

make changes ª Drive remote builds on HPC resources

ª Make, autotools…

ª Run – can generate a run configuration for particular system, batch environment

17

Software Specialists enabling modeling projects

ª Need a wider array of software engineering tools ª Source repository ª Issue tracking ª Documentation ª Performance tuning…

ª Eclipse Parallel Tools can help with many of these concerns

18

Source Code Control: “Team” Features

ª Eclipse supports integration with multiple version control systems (VCS) ª CVS, SVN, Git, and others ª Collectively known as “Team” services

ª Many features are common across VCS ª Compare/merge ª History ª Check-in/check-out

ª Some differences ª Version numbers ª Branching

19

Issue Tracking

ª Mylyn Bridge ª Tracks tasks, links to source and bug repositories

20

Connections to Jira, bugzilla, …

Eclipse Documentation

ª Eclipse Help System – built in and standalone (http://help.eclipse.org)

21

Performance Tuning: PTP TAU plug-ins http://www.cs.uoregon.edu/research/tau

ª  TAU (Tuning and Analysis Utilities) ª  First implementation of External Tools Framework (ETFw) ª  Eclipse plug-ins wrap TAU functions, make them available

from Eclipse ª  Full GUI support for the TAU command line interface ª  Performance analysis integrated with development

environment

22

Sustainability, Investment, Collaboration: Building Community

ª Stakeholders/investors ª Funding agencies à innovative capability ª HPC vendors

ª contribute to open source foundational software ª basis for proprietary or open-source add-ons

ª Supercomputing centers ª user support and training ª xml documents for batch system and local policy

integration

23

Sustainability, Investment, Collaboration: Building Community II ª Stakeholders/investors continued

ª Supercomputing integrating organizations (eg, XSEDE): ª user support ª single sign-on support (?) ª training ª education support

ª Tool providers ª xml documents for integration of command line

tools ª extra value-add from tool-specific plugins?

ª Anyone else?

24

Challenges for DEHPC

ª Budgetary pressure on supercomputing centers and integrating organizations

ª Challenges successfully competing proposals in open calls (eg, NSF SI2). ª Innovation, impact are paramount here

ª User community is large, disperse (eg lots of downloads of PTP, hard to connect to user community)

25

Opportunities (for discussion)

ª Can we build a community to support a DEHPC? ª PIs proposing to funding agencies for innovative

additions? ª Vendor contributions to DEHPC? ª Supercomputing centers and integrating

organizations support? ª Eg, User support, training, configuration (xml)

documents

26

Eclipse Parallel Tools Platform

www.eclipse.org/ptp

27