hpc on aws - siemensmdx2.plm.automation.siemens.com/sites/default/files/presentation/... ·...
TRANSCRIPT
1
Hiroshi Kobayashi, Dev./Lab. IT SystemHGST Japan, Ltd.
Jun 3, 2015
HPC on AWS
2
HPC on AWS
HPC = High Performance Computing
AWS = Amazon Web Service
3
Agenda
• HGST
• Why choose Cloud?
• Performance
• Flexibility
• What’s Next…
• Summary
4
HGST Company Profile
Founded in 2003 through the combination of the hard drive businesses of IBM, the inventor of the hard drive, and Hitachi, Ltd (“Hitachi”)
Acquired by Western Digital in 2012
Headquartered in San Jose, California
Approximately 38,000 employees worldwide
More than 4,700 active worldwide patents
Develops innovative, advanced hard disk drives, enterprise-class solid state drives, external storage solutions and services
Delivers intelligent storage devices that tightly integrate hardware and software to maximize solution performance
5
Broadening Lineup of Storage SolutionsRECENT INNOVATIONS
Solid State Storage Solutions HGST Storage Software
Ultrastar®
SN100 Series NVMe PCIe
Ultrastar®
SSD800MH.B,SSD1600MM &SSD1600MRSAS SSD
FlashMAX® III PCIe
HGST ViridentSolutions
HGSTViridentSpace
HGST10TB SMR HDD
HGST Ultrastar® He8
HDD Storage Solutionswith HelioSeal™ Technology
Active ArchivePlatform
Petabyte-scale Data CenterStorage Solutions
6
HGST Active Archive System
Our first fully integrated system with 4.7PB raw capacity per rack!
Complete scale-out object storage system for cloud data centers
4.7PBraw capacity per rack
Optimized for active archive workloads
BreakthroughTCO
BeatsWhite BoxEconomics
Highest DensityImproves Data
Center Efficiency
Lowest Power per TB with Fast Data
Access
Scales to Exabytes of Capacity
7
Market Leadership
8
Agenda
• HGST
• Why choose Cloud?
• Performance
• Flexibility
• What’s Next…
• Summary
9
Why choose Cloud?
• Background₋ A few years ago, HPC implementation project was started.
Project team investigated several cloud HPC services except for AWS. But those did not satisfy HGST’s requirement.
₋ CIO Steve Phillpott recommended AWS for HPC. He had much experience of HPC on AWS at life-science industry.
₋ Through several Proof of Concept projects, began to understand Pros/Cons of On-premise and Cloud HPC.
• Key factors are…₋ Scalability, Data transfer, Remote Visualization
₋ Commercial Application, Cost…
10
Agenda
• HGST
• Why choose Cloud?
• Performance
• Flexibility
• What’s Next…
• Summary
11
Scalability
• CD-adapco provided the benchmark data on their cluster.• C3 provide significant improvement to the scalability• C3 is 1.81x faster than CR1• Still behind to physical cluster with InfiniBand
1.81x faster
1.70x slower
※1 EN = Enhanced Networking
※2 placement group enable
※3 evaluated by elapse time
※4 only 200steps
12
Remote Visualization
• Result data is too huge to download• Transferring huge data is NOT a option• Require Remote Visualization for huge result data
Server – Client Mode
Remote Desktop Console
AWS graphic
server
Client
Users
Consume server side
GPU resource and license
Remote access
via RDC/VNC
AWS file
server
Client
Users
Consume server
side license
Consume client side
GPU resource
Not good performance…Slower responseSlower rendering
Great performance!!!Almost same performance as
local workstation with high-end graphic card
G2
13
Data Collaboration
• Transferring huge data is NOT a option• Even 48TB of d2.8xlarge may not be sufficient
for long term / huge data repository• High cost for re-computing of large scale model
ClientUsers
ClusterMaster
Computing
Nodes
Shared storage
S3 bucket
job submission
small data back to client
AWS Simple
Storage Service
(S3)
14
Performance
• Scalability₋ C3.8xlarge improved the scalability dramatically
₋ Higher scalability is better
• Remote Visualization₋ Star-CCM+ is ready
₋ Other applications are NOT ready
• Data Collaboration₋ No need to struggle with the storage capacity and durability
• AWS can support whole process of simulation works!!!
15
Agenda
• HGST
• Why choose AWS for HPC?
• Performance
• Flexibility
• What’s Next…
• Summary
16
Hybrid HPC Architecture
• Local + Cloud = Hybrid HPC environment
• AWS + Cycle Computing http://www.cyclecomputing.com/
HGST
Virtual Private Cloud
AWS
Cluster
MasterComputing
Nodes
ClientUsers
Shared
Storage
data I/Oattached
Local Cluster
S3 bucket
Auto ScaleOut / In
Fixed Capability
17
Shape Compute To Match Work To Be Done
• All Jobs Run In Parallel on AWS 1.67x Throughput Improvement
Time
Before:
Shared Cluster Computer
512 core 512core 512core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
Today:
AWS EC2 CC2 Cluster
(Max Total 512 core)
512corewaiting
256 core 256 core
128 core 128 core
waiting
waiting
64 core
64 core
64 core
64 core
64 core
64 core
64 core
64 core
18
Shape Compute To Match Work To Be Done(Cont.)
19
Shape Storage To Match Work To Be Done
• No need to struggle with the storage capacity and durability!!!
ClientUsers
ClusterMaster
Computing
Nodes
Shared storage
S3 bucket
job submission
small data back to client
∞
20
Shape Cost To Match Work To Be Done
• Workload is NOT constant
• Server Reservation Discount = Reserved Instances (RI)
• Analyzing workload Utilizing RI Optimizing cost
21
Agenda
• HGST
• Why choose Cloud?
• Performance
• Flexibility
• What’s Next…
• Summary
22
What’s next for Cloud HPC…
• Computing Performance₋ More scalability, like InfiniBand
• Remote Visualization₋ Higher performance than RDC-TCP/IP
₋ PC over IP®? NICE DCV®? Star-CCM+ is ready!!!
• Commercial Application License₋ End User License Agreement (EULA)
₋ Hybrid License Server
₋ Consumption Based License Power On Demand!!!
Local
License
Server
23
Agenda
• HGST
• Why choose Cloud?
• Performance
• Flexibility
• What’s Next…
• Summary
24
Summary
• At this moment, HPC on AWS is NOT perfect₋ Scalability, Remote Visualization except for Star-CCM+
• HPC on AWS has extremely high flexibility₋ Hybrid HPC, Shape Compute/Storage/Cost To Match Work To
Be Done
• Flexibility will drive to responding to the changing business model
• Benefit of HPC on AWS should be verified with each applications based on its characteristic
• Required collaboration with application venders
25
Helping the World Harnessthe Power of Data withSmarter Storage Solutions