software engineering @ fom institute amolf marco konijnenburg sig se @ nikhef 14 november 2013 kite...
TRANSCRIPT
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
Marco Konijnenburg
SIG SE @ Nikhef
14 november 2013
KITE
A down to earth approach to cloud computing
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
2
Contents
• Context
• Predecessor
• Goals
• Design
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
3
Processing large datasets
• Datasets are over 10 GB in size
• Collection of thousands of spectra
• Each spectrum can be processed individually
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
4
Compute
• Use general purpose compute clusters
• Use dedicated compute cluster
• Splittable compute jobs
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
5
Cloud
• HPC –Cloud Calligo @ SARA – Open Nebula implementation
• Configurable systems– Mix various systems (like different OS-es)
• Start and stop systems at will– Using VM’s
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
6
COMMIT
Job workersJob office (headnode)
Calligo
Firewall
Proxy
Internet
ToPoS
J ob administratorJob administrator
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
7
Job submission application• Connect to headnode
• Specify script , data and storage for MS processing
• Specify number of tokens (job parts)
• Submit job– Receive email of job
status– View progress
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
8
Cloud UI
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
9
Improve
• Easier interface for non computer scientists– Start and stop preconfigured group of VM’s
• Interface from desktop app to web app– Multi platform– Always up to date– Needs a login to remember user settings
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
10
Overview
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
11
Kite log in
• Use existing techniques
– Browser login, SSO
– Using Shibboleth to connect to SURFConext
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
12
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
13
KRON
• Kite relay for Open Nebula
• Only command and data passing
• Reports to Kite when (re)started
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
14
KRON
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
15
KRON (expanded)
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
16
Communication
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
17
Issues & Wishes• XML RPC requires extra log in, not
SURFConext’ed
• Within one group VM’s ‘see’ each other
• Data in and out
• Scale it up to multiple clouds, possibly multiple types
• Scale it up to users from other institutes
Soft
ware
Engin
eeri
ng @
FO
M Inst
itute
AM
OLF
18
The end
Thank you for your attention
Contact:
MARCO.KONIJNENBURG
HTTP://WWW.LINKEDIN.COM/IN/MARCOKONIJNENBURG