The Norbert Limulus Cluster
Norbert was designed to use totally off-the-shelf parts. There is no custom case work required. Of course the hardware could be placed in a smaller volume with a custom case design, but that would require extra work and our goal is to create an easy to assemble system.
Norbert basically consists of a head node and three diskless worker nodes. All nodes have an extra PCIe GigE card for compute traffic. The head node has a DVD, a hard drive, and an extra network card to connect to the LAN. Each node has a single socket Gigbyte GA-G33M-DS2R motherboard with a 2.33 GHz Core2 Duo (E6550) processor, and one GByte of RAM (upgraded to 2GB in 2008). There are a total of eight cores in Norbert.
Norbert measures 26 inches (66 cm) across, 16 (41 cm) inches deep, and 25 (64 cm) inches high. I placed the cases on a inexpensive wire shelf cart so it can be moved easily.
The following is a front view. The two GigE switches are on the top. Units can be removed without too much trouble, however, there is little need to take anything off the shelves. The cases have build in temperature sensors (blue lights on the front). The head node is located on the top left (it has a DVD drive).
The following a picture of the back. The two switches can be seen on the top. Of course there is a need for some cabling clean up (I plan on making custom length cables). There is a powerstrip on bottom middle of the shelf. The single powerstrip cord is the only power connection needed.
|Component||Amount||Type||Unit Price||2007 |
|Processor||4||Core Duo E6550 2.3 GHz||$179||$716||$680|
|Memory||4||Kingston ValueRAM 1GB (2 x 512MB) DDR2 800||$64||$256||$168*|
|Cluster Network Card||4||Intel PRO/1000 PT PCIe X1||$50||$200||$160|
|LAN Network Card||1||Intel PRO/1000 MT PCI||$50||$50||$30|
|Case||4||Apevia X-QPACK-NW-AL/420 MicroATX Case 420W PS||$70||$280||$280|
|Hard Drive||1||Seagate SATA ST3250824SV 250GB 7200||$80||$80||$80|
|DVD Drive||1||Samsung SATA SH-S183L||$30||$30||$30|
|GigE Switches||2||SMC SMC8505T||$63||$126||$100|
|Cables||8||3 ft Cat6||$3||$24||$24|
September 2007 Total: $2302 (US dollars)
September 2008 Total: $2092 (US dollars)*
- includes RAM upgrade price from 1GB/node to 2GB/node
Price to Performance
September 2007: $48.25/GFLOP
September 2008: $39.15/GFLOP
Note: all measurements were made "at the wall" using a Kill-O-Watt meter.
Node Idle and Load Power
The following table summarizes the idle and loaded power draw (in Watts) measured for each node and processor (E6550=Core2 Dual 2.33GHz, Q6600=Core2 Quad 2.4GHz). All cores were busy running the single copies of the NAS parallel benchmark. The highest Watts valued was used.
|Head Node E6550||78||128||2GB RAM, 250 GB HDD, 2 extra NICs, Nvidia GF-6200 256MB PCI video|
|Head Node Q6600||86||154||2GB RAM, 250 GB HDD, 2 extra NICs, NVidia GF-6200 256MB PCI video|
|Worker Node E6550||57||104||2GB RAM, 1 Extra NIC|
|Worker Node Q6600||63||125||2GB RAM, 1 Extra NIC|
Of course, if the nodes are not being used, they are manually powered down to stand-by mode (5Watts) and then restarted with WOL (wake on lan). I would like to integrate this into the scheduler.
Power Load Running HPL
When it is fully loaded running the HPL benchmark Norbert (four E6550's) draws between 371-483 Watts. The number varied as the benchmark progressed which I assumed is an indication of how modern processors are managing power consumption based on load. Even with the maximum power draw, the power performance ratio is a comfortable 10-11 Watts per GFLOPS.