wiki:NorbertLimulus

The Norbert Limulus Cluster

Norbert Design

Norbert was designed to use totally off-the-shelf parts. There is no custom case work required. Of course the hardware could be placed in a smaller volume with a custom case design, but that would require extra work and our goal is to create an easy to assemble system.

Norbert basically consists of a head node and three diskless worker nodes. All nodes have an extra PCIe GigE card for compute traffic. The head node has a DVD, a hard drive, and an extra network card to connect to the LAN. Each node has a single socket  Gigbyte GA-G33M-DS2R motherboard with a 2.33 GHz Core2 Duo (E6550) processor, and one GByte of RAM (upgraded to 2GB in 2008). There are a total of eight cores in Norbert.

Norbert Pictures

Norbert measures 26 inches (66 cm) across, 16 (41 cm) inches deep, and 25 (64 cm) inches high. I placed the cases on a inexpensive wire shelf cart so it can be moved easily.

The following is a front view. The two GigE switches are on the top. Units can be removed without too much trouble, however, there is little need to take anything off the shelves. The cases have build in temperature sensors (blue lights on the front). The head node is located on the top left (it has a DVD drive).

limulus-front-500x414.jpg

The following a picture of the back. The two switches can be seen on the top. Of course there is a need for some cabling clean up (I plan on making custom length cables). There is a powerstrip on bottom middle of the shelf. The single powerstrip cord is the only power connection needed.

limulus-back-500x375.jpg

Hardware Manifest

ComponentAmountTypeUnit Price2007
Total Price
2008
Total Price
Processor4Core Duo E6550 2.3 GHz$179$716$680
Motherboard4Gigabyte GA-G33M-DS2R$135$540$540
Memory4Kingston ValueRAM 1GB (2 x 512MB) DDR2 800$64$256$168*
Cluster Network Card4Intel PRO/1000 PT PCIe X1$50$200$160
LAN Network Card1Intel PRO/1000 MT PCI$50$50$30
Case4Apevia X-QPACK-NW-AL/420 MicroATX Case 420W PS$70$280$280
Hard Drive1Seagate SATA ST3250824SV 250GB 7200$80$80$80
DVD Drive1Samsung SATA SH-S183L $30$30$30
GigE Switches2SMC SMC8505T$63$126$100
Cables83 ft Cat6$3$24$24

September 2007 Total: $2302 (US dollars)
September 2008 Total: $2092 (US dollars)*

  • includes RAM upgrade price from 1GB/node to 2GB/node

Performance

HPL 2007: 47.71 GFLOPS before RAM upgrade (1 GB per node) Full HPL Results
HPL 2008: 53.44 GFLOPS after RAM upgrade (2 GB per node) Full HPL Results

Price to Performance

September 2007: $48.25/GFLOP
September 2008: $39.15/GFLOP

Power

Note: all measurements were made "at the wall" using a Kill-O-Watt meter.

Node Idle and Load Power

The following table summarizes the idle and loaded power draw (in Watts) measured for each node and processor (E6550=Core2 Dual 2.33GHz, Q6600=Core2 Quad 2.4GHz). All cores were busy running the single copies of the NAS parallel benchmark. The highest Watts valued was used.

IdleLoad Components
Head Node E6550 78 128 2GB RAM, 250 GB HDD, 2 extra NICs, Nvidia GF-6200 256MB PCI video
Head Node Q660086 154 2GB RAM, 250 GB HDD, 2 extra NICs, NVidia GF-6200 256MB PCI video
Worker Node E6550 57 1042GB RAM, 1 Extra NIC
Worker Node Q6600 63 125 2GB RAM, 1 Extra NIC

Of course, if the nodes are not being used, they are manually powered down to stand-by mode (5Watts) and then restarted with WOL (wake on lan). I would like to integrate this into the scheduler.

Power Load Running HPL

When it is fully loaded running the HPL benchmark Norbert (four E6550's) draws between 371-483 Watts. The number varied as the benchmark progressed which I assumed is an indication of how modern processors are managing power consumption based on load. Even with the maximum power draw, the power performance ratio is a comfortable 10-11 Watts per GFLOPS.