visualisation services at scd/hartree center193.62.125.70/ciuk-2016/martinturner.pdf ·...

CIUK December 2016

Visualisation Services at SCD/Hartree Center

Aim to have the human back into the Experimental-Simulation-

Computational-Loop in order to more efficiently use STFC

imaging facilities and HPC cycles

Martin Turner

Erica Yang, Srikanth Nagella

Ron Fowler, Evgueni Ovtchinnikov

Emma Tattershall, Barry Searle

Rob Allan,

George Leaver, Simon Hood

Tom Christy, David Hughes

+ thanks to many others

http://www.ccpi.ac.uk/

Collaborative Computational Project in Tomographic Imaging has been funded for

the next phase 2015-2020 and will use a range of computational HPC facilities.

Visualisation Matters

� Sites @ STFC

� Laying Fibre

� Use Cases

� Adding GPUs

� Adding login Nodes

� CCPi links

Viglen Cluster uses etc…

TSB Space Applications Catapult

University of Reading

Visualisation Matters Example

SCD contracted Dr Lee

Margetts and Ms Louise

Lever at the University of

Manchester to help add

transient thermal modelling

capability to the open source

parallel finite element

software ParaFEM. This

enabled Dr Llion Evans at

the Culham Centre for

Fusion Energy and his team

to use it to study plasma

facing wall components.





























Visualisation Matters:Creating a discussion space

Visualisation Matters:Discovering insight

Visualisation Matters:Commercial Example

Software demonstration by Thomas Salvini from

construqtive.com that turned our curved wall into an ipad-

mini controlled dashboard for selected and changeable

visual analytics output. Initial test with the Clarity3 software

in its vanilla format.

Use Case: Hartree development“having an argument”

Different users using ParaView etc.;

� multi-modality and stereoscopic display for Matthieu

Chavent (University of Leeds Researcher in Molecular

Visualisation), Biophysical Modelling, Biological

Complexity

Use Case: The human-in-the-computational-loop: still

important for supercomputing

Training and steering example: Stefano Rolfo and Charles Moulinec presenting

and running CFD code in the Hartree training room, and then in the Hartree

Visualisation Facility.

IBM Blue Gene/Q are highly ranked on the green

computing list (STFC system was #30 on the November

2014 Top500 Green List at 2,178 mflops per watt) the

computational cycles are not free and still limited

to a certain size of cores.

Use Case: Software for the Future: integrating ParaView and OpenFOAM

One of the most useful aspects of ParaView is its ability to run in client-server mode so

that the data of a simulation can remain on the parallel file system it was generated on

and only the required visualisation geometry is sent to the client. This avoids the need to

transfer many GigaBytes of data to the client, which can be slow and inconvenient.

iCSF (interactive) vs CSF (batch)

� iCSF - run interactively

– A backend node becomesyour workstation

– Start application GUI

• E.g., MATLAB, Rstudio, Stata, SAS

� CSF - run in batch

– No direct access tobackend nodes

– Batch system runs jobson backends without GUIs Login

node

Jobqueue

/home

/scratch

Backend batch nodes

= GPU

home shared

RDS (Isilon)

Loginnode

/home

Backend interactive nodes

What is the CSF?

� Computational Shared Facility

� A general purpose batch compute cluster

Loginnode

Job

queue

/scratch

home

Backend compute nodes

Connect to the login

node remotely.

No direct access

to backend

compute nodes.

Two main storage

areas visible to all

nodes.

100s of compute

nodes (~9300 cores)

GPU compute nodes

Compute nodes

connected together

by local network for

parallel apps & files.

All work (“jobs”) submitted

to batch system's job queue

18home shared

RDS (Isilon)

Hardware: Compute Nodes

� CSF has a variety of compute nodes

– We’ll cover which to use, why and how later

19

Intel: 2 x 6-core Xeon

(Westmere)

64 & 96 GB RAM (one with 512GB)

GigE or InfiniBand networking64 / 96GB


(Sandybridge)

64 & 256 GB RAM

GigE networking64/256GB


(Ivybridge)

64 GB RAM

GigE networking64 GB

AMD: 4 x 8-core Opteron

(Magny-Cours)

64 GB RAM

InfiniBand networking64 GB

AMD: 4 x 16-core Opteron

(Bulldozer)

128 GB RAM

InfiniBand networking128 GB

128 GB


(Haswell)

128 GB RAM

InfiniBand networking128GB

What is the iCSF?

� interactive Computational Shared Facility

� Non-batch compute cluster

Loginnode

/scratch

home

Backend compute nodes

Connect to the login

node remotely.

Two main storage

areas visible to all

nodes.

interactive compute

nodes shared by

multiple users.

20home shared

RDS (Isilon)

login node forwards

you to a backend

node.

256GB 256GB 2TB

64GB

Part of the Computationally

Intensive Research Ecosystem

21

Similarities of iCSF and CSF

� Both are a collection of servers (compute nodes) linked

together to form two large systems

� Operating system on both is the same – Scientific Linux.

– Need to use the command line a little on iCSF,

– CSF more command-line heavy

� They share the same home directory (similar to your P-

Drive - a private area for your files)

� Run work on CSF in batch, analyse/Viz in a GUI on iCSF

� To use both you must be part of a group/faculty that has

contributed hardware

Differences between iCSF and CSF

� iCSF (aka incline - interactive computational linux env)– The ‘i’ means interactive. No batch. Run apps straightaway.– Has a login node, but you are automatically routed to a compute node.– Suitable for GUI applications - start your app and use as you would on a

desktop.– Ideal for code development and testing.– Instant access to compute resources.– Typically running one job/app at a time (possibly a few together)

� CSF (computational shared facility)– Accessed and used via a login node. No direct access to the compute nodes.

– All work must be submitted to a batch system.

– Very command-line based.

– Hand your work over to the system, go for a coffee or on holiday/conference, come back and collect results.

– Large scratch filesystem so that running jobs don’t run out of disk space

Starting apps on the iCSF

� You need to load a modulefile to use the installed software. Run at inclineNN prompt, EG:

module load apps/gcc/rstudio/0.98.983module load apps/binapps/stata/14

module load apps/binapps/matlab/R2014amodule load apps/binapps/sas/9.4

� Then start it (the & lets you keep using the terminal):rstudio &

xstata

matlab &

sas &

� You should see a familiar looking GUI/window

� The software pages tell you which modulefile you need:– http://ri.itservices.manchester.ac.uk/icsf/software/

0. Your Windows desktop 2. X2GO connected to nyx3,4(gives you a Linux desktop to work from)

1. X2GO

4. Log in to iCSF from nyx3 desktop5. Run matlab on iCSF(windows appear on nyx3 desktop)

6. X2GO(matlab keeps running!)

3. Start a terminal on nyx3 desktop Hartree Centre ADD Login Nodes

Two Quadro 5000

65GB 32 core : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz

NVIDIA-SMI 361.45 Driver Version: 361.45.11

There are 2 extra login nodes currently,

� one running EnginFrame and DCV and the other

� for general visualisation use, e.g. with Virtual GL, VisIt, ParaView, etc.

NICEDesktop Cloud Visualization (DCV)

End users:

• Intuitive, application-centric Web interface to start, control and re-connect to a session

• Single sign-on for batch and interactive applications

• All data transfers from and to the remote visualization farm are handled by EnginFrame

• Built-in collaboration, to share sessions with other users

• The load and usage of the visualization cluster is monitored in the browser

• The solution also delivers significant added-value for the system administrators:

• No need of SSH / SCP / FTP on the client machine• Easy integration into identity services, Single Sign-On (SSO), Enterprise portals• Automated data life cycle management• Built-in user session sharing, to facilitate support• Interactive sessions are load balanced by a scheduler (LSF, GridEngine or Torque/MOAB) to achieve

optimal performance and resource usage• Better control and use of application licenses• Monitor, control and manage users’ idle sessions

NICE (DCV)

� Coupled with these third party remote visualization engines (which specialize in delivering high frame-rates for 3D graphics), the NICE offering for Remote Visualization solves the issues of user authentication, dynamic session allocation, session management and data transfers.

� When using NICE DCV, the scene geometry and graphics state are rendered on a central server, and pixels are sent to one or more remote displays.

� This approach requires the server to be equipped with one or more GPUs, which are used for the OpenGL rendering, while the client software can run on "thin" devices.

Remote visualisation on HartreeA Constelcom Ltd. project

Initial results using ParaView client-

server and TurboVNC/VirtualGL

Hartree iDataPlex

� Fast access to data on GPFS file system

� High performance graphics nodes each with:

– 2 nVidia quadro 6000 cards

– 64 GBytes memory

– Access to X11 server for remote rendering

� Security access – only ssh access allowed

– Use ssh port forwarding

ParaView X11/GLX forwarding

• Requires client support for GLX extensions

• Does not user remote GPU

• Slow response due to verbose X11 protocol

• Lack of X11-extensions may crash program

• Unusable on slow connections

% ssh –X user@gfxlogin8

$ paraview &

$

ParaView client server

� Client paraview; pvserver on remote

� Communication via ssh port forward

� Rendering: simple on client, complex on remote

# client connects to server over tunnel

% paraview –url=cs://localhost

Advantages:

• Fast GUI response

• Large memory on

server

• Parallel processing

# forward port & start remote servers% ssh –L11111:localhost:11111 gfxlogin8$ DISPLAY=:0 mpirun –n 4 pvserver …

ParaView turboVNC/VirtualGL

� All processing on remote, any VNC client on local

� VirtualGL to exploit remote GPU

# port forward, start vncserver

% ssh –L5902:localhost:5902 gfxlogin8

$ vncserver :2

# run vnc client locally

% vncviewer :2

# in vncviewer shell

$ vglrun paraview &

# optional

$ DISPLAY=:0 mpiexec –n 4 pvserver

Advantages:

• Good on slow links

• Clients on Ipad,

Android

Diamond Light Source

ISIS Neutron Source

Central Laser Facility

STFC Facilities

Visualisation

& Computing

Use Case: IMAT beamline: linking the human-visualisation to the facility

imaging capture process

The IMAT (Imaging and Materials Science & Engineering) beamline, due for use in 2016-

7, is going to be the first neutron imaging instrument at ISIS that will offer unique time-of-

flight tomography-driven diffraction techniques.

This will capitalise on the latest image reconstruction procedures and event mode data

collection schemes involving a strong visualisation component. ParaView has been tested

and will incorporate fast parallel image reconstruction algorithms with on-the-fly image

processing and visualisation to inform and guide experiments.

SCARF Configuration

The hardware for

SCARF is purchased

on a yearly basis,

with each new

addition added to the

same set of

computational job

queues so as to give

the users a seamless

experience.

Backend VM nodes

Horizon VMware software which has integrated support for

NVIDIA Grid K2.

� Target VMs are Windows only.

� Users have to download a horizon client and …

� will be able to get the VM desktop to their PC/Laptops.

ULTRA is underpinned

by an array of high-end

computing

technologies, including

high speed data

acquisition, high

throughput data

transfer, cluster

computing, parallel

rendering and remote

visualisation to enable

end-to-end fast parallel

reconstruction

workflows for facility

users.

Summary

� Connect to your data storage

� Add memory to login nodes

� … then lots of cores

� … and a GPU card

� Have graphics facilities capable of 50-100GB

� Learn how to be intelligent >2TB data is okay

� There are solutions with GPUs

� … and login nodes make a great location

Contacts

http://tinyurl.com/STFCVis

� [email protected]



visualisation services at scd/hartree center193.62.125.70/ciuk-2016/martinturner.pdf ·...

Documents