Minimalist’s Linux Cluster

Changyoung Choi, Jeonghyun Kim, Seyong Kim

Department of PhysicsSejong University

Reported in Lattice’99

1. Usual Linux cluster

• Easy to build, use and maintain• Off-The-Shelf everything!

– Ordinary PC’s– Ordinary network components– Free OS (Linux)– Free parallel programming environment

(MPI)

Room for improvement

• Occupies too much space• Consumes too much power• Not all the components in an ordinary

PC is necessary for computation– Possible further reduction in cost!– Possible reduction in power consumption

• Supporting full bisection bandwidth is costly!

Conditions

• No custom-made hardwares– If it takes too long to develop ->

no advantage– If there is, should be recyclable

• Should be efficient enough• Should be do-able with small manpower

2. Construction(hardware)

• Node – ATX size standard mother board– Intel Pentium IV-2.4 GHz– 512 Mbytes DDR SDRAM– 100 Mbps ethernet card (3+1)– 180W power

• Cage for ATX size mother board• 1 front end server

CPU Intel P-4,2.4GHz

\236,500(=$198)x36

motherboard

ASUS P4PE \203,500(=$170)x36

memory 512MB PC2700 DDR

\112,200(=$93.5)x36

NIC 3+1 realtek \53,000(=$44)x36

power Sun ATX 180W \25,300(=$21)x36

cable \10,000(=$8)x36

cabinet 2000x910x750

\880,000(=$733)

cabinet accessory \365,200(=$304)

Cost per node $534.5

Total cost $534.5x36+$1037 = $20,279

Price Includes Tax

Software

• WOW Linux 7.1 (equiv. RedHat 7.1)• Linux Terminal Server Project (LTSP)

– Remote booting (DHCP + TFTP)– Diskless node system

• 2-D mesh network configuration– Script developed in-house

• MPI programming environment– MPICH/LAM

Setup

• Front end wait• F-end send DHCP

info• send kernel using

TFTP service• send client file

system by NFS

• Node 1 power-up• NIC listens to DHCP

server• IP setup complete

and download kernel• setup file system on

memory and copy NFS mounted file system to memory

Setup(continued)

• Node 1 wait• Node 1 send DHCP

info• Node 1 send kernel

image• Node 1 act as a

pipe line between front end and node 2 for NFS

• Node 2 power-up• NIC listens to node 1

DHCP server• IP setup complete and

download kernel from node 1

• setup file system on memory and copy NFS mounted file system to memory

Parallel boot

3. Performance

• 1-D ring full QCD code– Not suitable for 2-D mesh structure

• 8^3 x 512 lattice

8^3 x 32 lattice testedon 1-node

1187 sec, CG = 181.2 -> 369 MFLOPS

8^3 x 512 lattice testedon 4-node

5522 sec, CG = 391.4 -> 1.24 GFLOPS

8^3 x 512 lattice testedon 8-node

3029 sec, CG = 390.9 -> 2.25 GFLOPS

4. Discussion

• recyclable– cabinet takes standard motherboard size

-> replacing node is enough for a upgrade

• occupies smaller foot-print, consumes less power

• need to fine-tune• need to modify the code for 1-D ring

to 2-D mesh