Alabama Supercomputer Authority

Internet and Technology for Education

2

High Performance Computing

University of Alabama

University of SouthAlabama

University of Alabamain Huntsville

Alabama A&M University

University of Alabamaat Birmingham

Auburn University

Athens State UniversityAuburn University -Montgomery

Bevill State CollegeJacksonville State University

Troy UniversityTuskegee University

University of West AlabamaUniversity of Montevallo

U.S. Air ForceU.S. Army

NASAIntel Corporation

Operon BiotechnologiesTime Domain

Alabama State University

3

• Materials Science

• Computational Fluid Dynamics

• Computer Science

• Medical

• Social Science

• Education

• Electromagnetics

• Computational Chemistry

• Structural Dynamics

• Physics

• Earth Science

Alabama Supercomputer Applications

4

Compilers and Programming

Compilers GNU C/C++ Fortran 77 (All) Intel C/C++ Fortran 77/90/95 (All) Portland Group Fortran 77/90/HP (XD1 Only) Pathscale Fortran 77/90 (XD1 only)

Parallel Programming OpenMP MPI Pthreads Java threads Math libraries; ACML, SLATEC, MKL, SCSL, IMSL.

5

Memory Architectures

Distributed Memory-A CPU can access its memory only– Example : A cluster made from Single CPU PC’s – Both the Cray XD1 and the SGI Altix 350 can run distributed memory programs.

Shared Memory-Multiple CPUs can access the same memory– Example: Dual CPU PC– The Cray XD1 can run shared memory programs on the two CPUs in a given node.– The SGI Altix 350 can run shared memory programs on the 16 CPUs in the compute

nodes.

Hybrid – combines both approaches– Cray XD1 Example: Form a cluster out of Dual CPU Nodes (mostly distributed)– SGI Altix Example: Form a Cluster out of 16 CPU Nodes (mostly shared)

(A Node = 1 copy of the operating system)

6

• 216 Itanium 2 Processors

- 1186 GFLOPS Peak

• Shared Memory Architecture

- NUMAlink, Inifiniband, fiber channel and gigabit ethernet data networks.

• Memory - 1040 Gigabytes

• Disk Storage - 10.8 Terabytes

SGI Altix Supercomputer

7

Altix Login 6P

Metadata 2P

Scratch Disks

Compute 16P 1.4 GHz

Compute 16P 1.5 GHz

FC Switch

Compute 72,72P

1.6 GHz

Ethernet Switch

Home Directories Applications Queue Server

License and Password Server

Firewall

Backup Copy

AREN Internet and Internet2SGI Altix

8

• 132 AMD Opteron Processor System- 580 GFLOPS Peak

• Distributed Memory, Rapid Array Network.

• 6 Xilinx Virtex 4 LX160 FPGAs

Cray XD1 Supercomputer

• Memory - 244 Gigabytes

Nodes with 2,4,8 Gigabytes

• Disk Storage

- 3.5 Terabytes

9

The AMD Opteron Processor

Dedicated Memory Bus

64KB

64KB

1 MB

Up to 19.2 GB/s I/O

Native 32 & 64 bit x86 compatibility

10

Balanced Interconnect

Removing the Removing the communications bottleneckcommunications bottleneck

Removing the Removing the communications bottleneckcommunications bottleneck

Processor I/O Interconnect

GigaBytes GFLOPS GigaBytes per Second

CrayXD1

Memory

Xeon Server

6.4GB/sDDR 400

8 GB/s

5.3 GB/sDDR 333

0.25 GB/sGigE

1 GB/sPCI-X

11

• 168 Opteron Processors

- 1008 GFLOPS Peak

• Shared/Distributed Memory Architecture

- Inifiniband, and gigabit ethernet data networks.

• Memory - 1312 Gigabytes

• Disk Storage - 21.5 Terabytes

Dense Memory Cluster

12

Performance Comparison

DMC SGI Altix Cray XD1

# CPUs 168 216 132

CPU Type Opteron Itanium 2 Opteron

SPECFP/cpu 2117 1931 - 2207 1553

Relative power 1.73 2.24 1

Memory (GB) 1312 1040 shared 240 dist.

Disk (TB) 18 1.1 5.3

Storage (TB) 3.5 10.8 3.5

GFLOP 1008 1186 580

Clock (GHz) 3.0 1.4, 1.5, 1.6 2.2

13

Torque/MOAB queue system

User commands

• qsub – run a job

• qstat – see status

• checkjob – see job information

Queue Server Node

• Server – keeps track of queues and jobs

• Scheduler – chooses when/where to run jobs

Compute node

* MOM daemon – runs jobs and reports available CPUs/memory

MOAB runs jobs to ensure maximum utilization of the system, without over-subscription, and ensures jobs get the requested amount of memory/CPUs.

14

AREN Backbone

• Highspeed Backbone

• 1.5-100 Mb to Clients, driven by their requirements. 10Mb typical.

• Multiple Internet Access Points (GigE): 1.6 Gbps

ASA has the networking infrastructure to move very large datasets via high bandwidth.

TuscaloosaTuscaloosa

BirminghamBirmingham

HuntsvilleHuntsville

MobileMobile

FlorenceFlorence

TroyTroy

500 MbsInternet

1 Gbs1 Gbs155 Mbs155 Mbs

100 Mbs100 Mbs

50 Mbs50 Mbs

500 MbsInternet

Auburn

100 MbsInternet

Dothan

500 Mbs Internet

155 Mbs155 Mbs

10 Mbs10 Mbs

10 Mbs10 Mbs

155 Mbs155 Mbs

155 Mbs155 Mbs

155 Mbs10 Gbs Sep.155 Mbs

10 Gbs Sep. JacksonvilleJacksonville

MontgomeryMontgomery

165 Mbs165 Mbs10 GbsATL Sep.

15

Statewide Education NetworkAlabama Research & Education Network (AREN)

Public Libraries

Postsecondary Colleges

Central Offices

UniversitiesK-12 SchoolsK-12 Schools

SDE

Internet

ASA’s high speed network is connected to the Supercomputer Center, Internet, and Internet2.

16

Statewide Network: A Reality!

T156 Kb

19942008

17

Network Design and Engineering

– Internet Provider

– LAN, MAN, WAN

– Network Consulting, Analysis, Troubleshooting

– Network Security

Network Solutions and Services

Network Installation

– Network Equipment

– Integration and Test

Network Operations

– Network Monitoring

– Network Operations Centers

– Helpdesk

– Enterprise Management

– Internet Usage Statistics for Clients

Custom Development

– Linux, NT, DNS, Proxy, Content Filter / Firewall, Mail, Web, etc.

– Server Hosting

18

• Point to Point (DS1, DS3)

• Frame Relay• ATM (DS1, DS3, OC-3,

OC-12)• SONET• xDSL• Metro Ethernet• Wireless

Network Technologies

Ethernet 10/100/1000 Fiber Channel (Storage Area

Network) FDDI HiPPI (800, 1600, GSN) ATM (LANE, MPOA, MPLS)

TCP/IP IPv6 IPX SNA Legacy Protocols, e.g.,

AppleTalk

UNIX (Linux, SCO, BSDi) Windows XP, Server 2003,

Vista Mac OS X Cisco IOS Novell

WAN Technologies Protocols

LAN Technologies Network Operating Systems

19

Internet Services

• Virtual Private Networks

• Web Services– Internet– Intranet

• Security - Firewalls, Content Filtering, Intrusion Detection, Vulnerability Assessment

• Servers - Web, Email, Collaboration, DNS

• Quality of Service Configuration

• DNS Hosting and Management

20

Alabama Virtual Library (AVL) – Over 614,000 Average Monthly Searches;

464,093 AVL Cards Issued through 9/03

Technology Support through Partnerships

Alabama Learning Exchange (ALEX) – Custom Software Development, Technical Support, and Server Hosting

Content Filtering – Available For All AREN Internet Clients Through Statewide Telemate License

Data System Applications – Web-based Portals for Data Analysis & Reporting, Including Secure Data Transmission

21

AREN Benefits

•State Agency That is a Proven Internet Service Provider for Education in Alabama Since 1994

•24x7 Network Operations – Network Is Monitored and Maintained By Networking Experts

•Excellent Source for Technology Consulting, Assistance, and Training

•Custom System Development

•Internet2 – Very High Speed Connection of State Network Education Institutions to this Resource Dedicated to Research and Academics

22

Computer Facility and Operations Center

Excellent Facility for Expansion and New Services such as Backup Operations for State Government Data Centers and

Computing on Demand Resources

State of the Art Computer Facility in Cummings Research Park with Capacity for Expansion and Security

Computer room floor space of 3,065 square feet (50% open) with an additional 22,000 square feet of office, storage, and meeting space

Multiple levels of physical security to the building, office area, and the computer room are provided by controlled access magnetic keys

24 x 7 Network and Computing Operations, Monitoring, and Helpdesk

Staffed by Network Engineers, Systems Analysts, System Developers, and Technicians

23

ASA Clients include: Research Universities Regional Universities Community Colleges K-12 Systems and Schools State Government Agencies

Alabama Department of Transportation Alabama Department of Finance – Information Services Division

Public Libraries Federal Agencies Local Governments Private Industry (e.g., Intel)

ASA has Partnerships with: State Department of Education Department of Postsecondary Education Alabama Public Library Service Alabama Virtual Library The University of Alabama Gulf Central Gigapop/Internet2/State Regional Optical Network NASA

Diverse Clients and Partnerships

24

Summary

The Alabama Supercomputer Authority provides a number of services for academic, government and economic development within the state of Alabama.

High Performance Computing facilities State wide network Support of distance learning initiatives Web and email hosting Custom development Disaster recover hosting Computer security services Alabama Virtual Library

25

AlabamaSupercomputer

Authority

State of Alabama Leader and Trusted Partner for Technology