standard building blocks for grid and hpc clusters · 2006-09-06 · standard building blocks for...

Standard Building Blocks for Grid and HPC Clusters

LAURENCE LIEW [email protected]

Director, Open Source Grid Development Centre

Platform Computing, August 2006

© Platform Computing Inc. 2003-20062

Agenda

Platform Computing

HPC History & Today

HPC in the Future

A Standard?

Q&A

1

2

3

4

5


Platform Overview

� World-leading Grid software solution provider� Industry-leading partnerships with Dell, HP, IBM, S GI, SAS, …� Global Presence -- clients, offices, service, suppor t, training � 1,600 Fortune 2000 customers around the world

� Growing profitably every year since inception� 14 years experience with mature, proven production solutions � Dedicated vertical industry teams and expertise � Strong commitment to standards (OGSA, GGF etc.)

• Core Product Team: 120 Employees in Development26 Ph.D’s; Many Masters Degrees

• Support Team: 35 people10 Phd’s

• 3 Development Centres: Canada, China and SingaporeSingapore -> Open Source Grid Development Centre (US$5M investment)


Industry adoption and technical innovation over 14 years

1992• Aerospace

• Telco OEM’s

1992• Aerospace

• Telco OEM’s

Over 1,000 production grids deployed

0 to 1,600 clients0 to 1,600 clients

1993• Gov’tResearch

• Universities

1993• Gov’tResearch

• Universities

1994• Semiconductor

1994• Semiconductor

1995• Pharmaceutical

• Chemical

1995• Pharmaceutical

• Chemical

1996• Oil and Gas• Computer

Graphics

1996• Oil and Gas• Computer

Graphics

1997• Automotive

• Software (ISV’s)

1997• Automotive

• Software (ISV’s)

1998• Defense

• Telco services

1998• Defense

• Telco services

1999• Life

Sciences

1999• Life

Sciences

2000• Banking

and Financial

2000• Banking

and Financial

2001• Systems integrators

2001• Systems integrators

2002• Data mining

services

2002• Data mining

services

1992 - Batch workload management

1992 - Batch workload management

Platform Industry client

Platform grid technology

1996 – Multiple clusters

1996 – Multiple clusters

1997 – Reporting

1999 – License pre-emption

1999 – License pre-emption

2000 - Analytics2000 - Analytics

2001 – On-line workload management

2001 – On-line workload management

2005• Telco

2005• Telco

2005 – VM workload

management

2005 – VM workload

management

2002 – Jobflow automation

2002 – Jobflow automation


Applications Silos Today

The Need is for Variable Resources to Meet Variable Business Demand The Need is for Variable

Decouple the applications from resources

The Result is IT Agility


Platform Grid Architecture for SOI/SOA

ApplicationApplication Application Application Application

Enterprise Applications

Results: Model architecture that enables Enterprise Grid

SOA

SOI

Resource Orchestration Platform Enterprise Grid Orchestrator™ (EGO)

Platform Enterprise Grid Orchestrator Developers Edition

Platform Symphony™

Platform LSF®Family of Products

Platform Virtualization Orchestrator ™

3rd Party Integrations

Application Orchestration


Multi-Cluster

Platform LSF Family Products Overview

License Scheduler

Macrovision FLEXlm

Application License Optimization

Reporting

Web Application

Job Submission Script

Workload Management Cluster

Superior Workload Management

System Manageability

Cluster

Analytics Integration with Leading Applications

INTEGRATED APPLICATIONS FROM VENDORS

Abaqus

IBM

Ansys

LSTC

Fluent

MSC


HPC in our lives: Some common examples

Improve Business PerformanceIn a world of increasing pressure to deliver faster , better and cheaper, being first and fastest is critical to profitabilit y and competitiveness.

Fundamentally, HPC provides two core benefitsThroughput: Get more done in the same time� Example: Computer chip verification test suites ~ 1 Million jobs per day

Optimization: Do more with Less ~ Do the Important Things First

Free Thinkers From Constraint ~ remove boundariesImagine the impact of curing a life-threatening disease by saving valuable time on the road to discovery. Imagine how safe our roads would be if all cars could be quality and safety tested repeatedly at a fraction of the cost of physical prototype testing.modeling the fluid flow and combustion in a car engine to maximize fuel efficiency optimizing the scheduling of work on production lines to maximize utilization accessing huge databases for on-line transaction processing and data miningof the banks' ATM networks Hollywood, video games and animationLong running batch applications: end of day accounts or billing


Competitive and Financial Benefits of HPC

“It has been a continuous stream of revenue to our bottom line, giving us the ability to look into other development areas.”

“It drives innovation, R&D effectiveness, and productivity.”

“Our intent is to migrate to a management policy of disposable computing”

“The time to market would prohibit our

business from existing.”

“The ROI is returned within one year of the purchase.”

Reduced Costs

Business Survival

Return on Investment

Increased Competitiveness

“… saved us hundredsof thousands of dollars in hardware costs, improved our computingperformance, and gave us a robust system…”

Sources: Platform 2003-4; Council for Competitiveness HPC User’s Summary


History of HPC

In the past, “Supercomputers” ruled the day

The “supercomputers” in the 1980's used proprietary, custom-designed hardware (CPU, memory, disk and interconnect) to achieve high performance levels

Commonly referred to as “Capability Computers”

Very expensive, single vendor and proprietary

Systems were frequently “lashed together” in a loosely coupled cluster and shared

Workload schedulers kept the supercomputer fully utilized (>90%)

HPC was constrained to the “privileged few” – wealthy and savvy typically used for grand challenge problem solving

Solutions were “built” not bought

Supercomputing technology primarily evolved through significant and expensive R&D

Engineered performance: “the better mouse trap”


HPC Today

We have evolved from Supercomputers to “Supercomputing”

Solutions are commonly bought, not built

Clusters of computers are common and moving beyond academic, research and science

Today, HPC is the domain of the capable and enabled

Price/performance is now attractive and compelling (3:1 ratio over SMP)

The market is driven by Commoditization and Standardization


HPC Linux Clusters

Common today: Beowulf Style

clusters.

Multi-Cluster: a cluster of clusters interconnected


Why Should I Care about Commoditization and Standardization?

The effects of Commoditization change the market; the forces arecontinually at work and everyone will benefit

Customers can buy bundled, pre-integrated clusters from several vendors at attractive prices today

These solutions are built on a combination of industry standard and commodity hardware components

Today these solutions are increasingly built with low cost, best of breed, open source software stacks

Differentiation now lies with the whole product solution and is centered around support and services, not engineered performance

Less about the “better mouse trap” more about “fewer mice”


The Linux Paradigm Shift

HPC YESTERDAY

ProprietySoftware

ProprietyHardware

ExpensiveClosedSingle Source

HPC NEXT GENERATION

ProprietySW

Best of BreedOpen Source

Linux

Superior ValueStandards - BasedBest of Breed functionalitySingle responsibility

HPC TODAY

WLM

CM

SM

OS

Linux

Misleadingly inexpensiveOpenMultiple Vendors

Dec

oupl

edS

oftw

are

Com

pone

nts

In the past, large UNIX servers were clustered together with systems management provided by IHVs - The stack was proprietary


Vertical Solutions: application Prototyping / Sizing- Energy/Petroleum - Life Science

- Automotives – Manufacturing and Design

- Custom application benchmarks- Standard benchmarks- Performance studies

Resource Monitoring / ManagementResource dynamic allocationCheckpoint restarting and

Job redistributing

Compilers and math libraryPerformance tools

- MPI analyzer / profiler- Debugger

- Performance analyzer and optimizer

MPI 2.0 / Fault Tolerant MPIMPICH, MPICH-GM, MPI/LAM, PVM

Interconnect Technologies- FE, GbE, 10GE… (RDMA)- Myrinet, Quadrics, Scali

- InfinibandManagement Hardware

Interconnects Hardware

Interconnect Protocols

Operating Systems

Middleware / API

ClusterHardwareSoftware

Monitoring &Management

Application

Node Monitoring & Management

Benchmark

Development Tools

Workload Management

Platform Hardware

ClusterInstallation

ClusterFile System

- Reliable PVFS- GFS , GPFS …

- Storage Cluster Solutions

IA-32, IA64 (Processor / Platform) comparison

Standard rack mounted, blade and brick servers / workstations

Cluster monitoring Load analysis and

BalancingRemote accessWeb-based GUI

Cluster monitoringDistributed System

Performance Monitoring Workload analysis and

BalancingRemote accessWeb-based GUI

Remote installation / configurationPXE support

System ImagerLinuxBIOS

Cluster Functionality: the Building Blocks


HPC in the Future

The Near Future:

“Personal Supercomputing”

The domain of the average, non-savvy

What’s really happening? New Market Disruption

…enables a new class of users to acquire and use th e technology for a growing set of problems

Customer Expectations: The Requirements have shifted

Provide an end-to-end solution

Reduce business risk and consistency

Single point of contact for sales, services, support

Provide customers a “single throat to choke”

Simplify purchase, deployment and management

Reduce the complexity and make clusters easy to deploy and use – over and over again


A Bold Statement

HPC technologies will be adopted by mainstream IT customers as part of some larger “whole product”.

…as part of a larger Middleware footprint

…an ERP Application Suite

…or as part of standard Linux and Windows Clusters

The industry needs to place HPC products in a manner that matches buying patterns

What is needed: a standardized, consistent, replicable approach


What the Detractors Say

“It’s too hard”

“Why?”

“Not possible”

“Huh?”

Drop by www.platform.com/Rocks and see what we have been able to do.

standard building blocks for grid and hpc clusters · 2006-09-06 · standard building blocks for...

Documents