wiki:NorbertLimulus

Version 14 (modified by admin, 15 years ago) (diff)

--

The Norbert Limulus Cluster

Norbert Design

Norbert was designed to use totally off-the-shelf parts. There is no custom case work required. Of course the hardware could be placed in a smaller volume with a custom case design, but that would require extra work and our goal is to create an easy to assemble system.

Norbert basically consists of a head node and three diskless worker nodes. All nodes have an extra PCIe GigE card for compute traffic. The head node has a DVD, a hard drive, and an extra network card to connect to the LAN. Each node has a single socket  Gigbyte GA-G33M-DS2R motherboard with a 2.33 GHz Core2 Duo (E6550) processor, and one GByte of RAM (upgraded to 2GB in 2008). There are a total of eight cores in Norbert.

Norbert Pictures

Norbert measures 26 inches (66 cm) across, 16 (41 cm) inches deep, and 25 (64 cm) inches high. I placed the cases on a inexpensive wire shelf cart so it can be moved easily.

The following is a front view. The two GigE switches are on the top. Units can be removed without too much trouble, however, there is little need to take anything off the shelves. The cases have build in temperature sensors (blue lights on the front). The head node is located on the top left (it has a DVD drive).

limulus-front-500x414.jpg

The following a picture of the back. The two switches can be seen on the top. Of course there is a need for some cabling clean up (I plan on making custom length cables). There is a powerstrip on bottom middle of the shelf. The single powerstrip cord is the only power connection needed.

limulus-back-500x375.jpg

Hardware Manifest

ComponentAmountTypeUnit Price2007
Total Price
2008
Total Price
Processor4Core Duo E6550 2.3 GHz$179$716$680
Motherboard4Gigabyte GA-G33M-DS2R$135$540$540
Memory4Kingston ValueRAM 1GB (2 x 512MB) DDR2 800$64$256$168*
Cluster Network Card4Intel PRO/1000 PT PCIe X1$50$200$160
LAN Network Card1Intel PRO/1000 MT PCI$50$50$30
Case4Apevia X-QPACK-NW-AL/420 MicroATX Case 420W PS$70$280$280
Hard Drive1Seagate SATA ST3250824SV 250GB 7200$80$80$80
DVD Drive1Samsung SATA SH-S183L $30$30$30
GigE Switches2SMC SMC8505T$63$126$100
Cables83 ft Cat6$3$24$24

September 2007 Total: $2302 (US dollars)
September 2008 Total: $2092 (US dollars)*

  • includes RAM upgrade price from 1GB/node to 2GB/node

Performance

HPL 2007: 47.71 GFLOPS before RAM upgrade (1 GB per node) Full HPL Results
HPL 2008: 53.44 GFLOPS after RAM upgrade (2 GB per node) Full HPL Results

Price to Performance

September 2007: $48.25/GFLOP
September 2008: $39.15/GFLOP

Power

Note: all measurements were made "at the wall" using a Kill-O-Watt meter.

Idle Power

While idling, the head node draws about 75 Watts and each of the three compute nodes draw 62 Watts for a total of 261 Watts. Of course, if the nodes are not being used, they are manually powered down to stand-by mode (5Watts) and then restarted with WOL (wake on lan). I would like to integrate this into the scheduler.

Compute Load

When it is fully loaded running the HPL benchmark Norbert draws between 371-483 Watts. The number varied as the benchmark progressed which I assumed is an indication of how modern processors are managing power consumption based on load. Even with the maximum power draw, the power performance ratio is a comfortable 10-11 Watts per GFLOPS.