high performance computing (planing-usage-application)
TRANSCRIPT
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
1/41
High Performance Computing:Planing, usage, and Application
Yudi Rosandi
Workshop on High Performance Computing:from design to application
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
2/41
Outline
Planing
Hardware components
Infrastructures
Operating system
Usage
Basic knowledgeCluster Application
Simulation, research and development
Data processing, forecasting
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
3/41
Meet some Beowulf clusters....
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
4/41
Meet some Beowulf clusters....
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
5/41
Meet some Beowulf clusters....
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
6/41
Meet some Beowulf clusters....
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
7/41
Planing
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
8/41
Cluster
collection of computers
interconected one and another via network
working together on a single programor interdependent different programs
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
9/41
Planing of a cluster
What is the purpose?
How much budget do we have?
What is the expected outcome?
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
10/41
High availibility cluster
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
11/41
Load Balancing
multi-user environtment
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
12/41
Compute cluster (HPC)
single parallel program run in many procs
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
13/41
Budget...LOW
build from scratch
chalengingprecious experience
troublesome but interresting
freedom
Pitfallsmost problems comes from networking
cable mesh
high maintenance cost
high power consumtion(depends on hardware selection)
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
14/41
Budget...HIGH
high-end cluster
blade serverready built racks
What we get
production ready
integrated high-speed lan (infiny-band)good cooling system
low power consumption
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
15/41
HPC Outcome
Research
General sciencesfast publication
many ready to use programs
ability to solve complex problems
insight to unknown aspects in science
Computer science
parallel programming
computer administration
networking
user managementbenchmarking and optimization
Low research cost!
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
16/41
HPC Outcome
Production
fast computationfast decission making
increasing services
high availability
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
17/41
Hardware components
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
18/41
What kind of hardware do we need?
Anything...
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
19/41
Hardware requirements
Computing nodes
fast computerfast network interface
sufficient memory
local scratch storage
Storage serverdedicated, reliable
large storage
sufficient memory (+8Gb)
high-speed network connection(channel bonding, etc)
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
20/41
Hardware requirements
Head server(s):reliable, not necessarily fast
dhcp
nis
login
queuing
note:
these servers may be separated individual servers,combined servers in one machine, or virtualized(e.g. Xen)
Login server may need special consideration!
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
21/41
Common clusterfor high performance computing
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
22/41
Needed infrastructure
Room temperature control
Stabil electricityPower consumption
networking backbone
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
23/41
Operating system
Things to consider
stabilityreliability
managability
accessibility
security
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
24/41
Operating system: LINUX
Why????
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
25/41
Operating system: LINUX
stability (OK)
reliability (OK)managability (OK)
accessibility (OK)
security (OK)
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
26/41
Cluster Usage
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
27/41
Basic knowledge to usea cluster computing facility
know how to:make remote connection
use a shell (csh,sh,bash,etc)
file-system: locate directories, file permission
submit/manage programswrite a parallel program
don't do:
hacking
running malicious programrobbing memory
spamming file-system
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
28/41
Know your limits!Respect other users!
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
29/41
Remote login
Accesible from internet
static addressdedicated
Secured internet connection
key authentication
firewall
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
30/41
Remote login
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
31/41
Remote login
Files for ssh connection (under ~/.ssh/)
id_rsa (@client)identity file, must be kept 'only' inthe connecting computerSECRET! don't transfer via email, etc!
id_rsa.pub
public key, to be distributed to allservers
known_hosts (@client)
list of host keys of known servers
authorized_keys, authorized_keys2 (@server)
list of public keys of clients/user those allowed to connect.
config (@client)
local user specific configuration
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
32/41
command to create id keys:ssh-keygen
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
33/41
Shell
elements
variablesprogram controls
redirection
pipes
customizing shellmake your self convenient
personal commands, path, etc
basic scripting knowledge
creating job-scriptbuilding embarrassingly parallel programs
automating processes
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
34/41
Shell: bash
Standard in Linux
CompleteFlexible
Advanced
Demo/example....
declaring variable
simple program controls
redirection and pipes
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
35/41
Application
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
36/41
Animation movies!(@ pixar)
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
37/41
Animation movies!(@ pixar)
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
38/41
Material science (@ TU-KL)
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
39/41
Material science (@ TU-KL)
a glance at computing center
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
40/41
Application
incomplete examples...
Research:modelling
nano-sciences
complex-system
Industryimage rendering
data processing
Many others...
-
8/3/2019 High Performance Computing (Planing-Usage-Application)
41/41
Thank you