| | 31 | |
| | 32 | === Switchless 10 GigE === |
| | 33 | |
| | 34 | These tests are very preliminary using old hardware and untuned software. |
| | 35 | |
| | 36 | The first issue is how to add 10GigE without a switch. Create a four node loop using [http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge#Bridge_priority Ethernet Bridge]. A simple loop will require a Spanning Tree Protocol, but that cuts a link. So replace the cut link with secondary network. Then 4 of 6 routes are 0 hop and 2 of 6 are 1 hop. |
| | 37 | |
| | 38 | Hardware: Head node is Sandy Bridge, i5-2400S with 4 GB RAM, The nodes are single dual core E6550, with 4 GB RAM. 10Gig NICS are [http://www.chelsio.com/adapters/t4_unified_wire_adapters/ Chelsio T420-SO-CR] |
| | 39 | |
| | 40 | Software: [http://open-mx.gforge.inria.fr/ open-mx] |
| | 41 | |
| | 42 | |
| | 43 | The following are the best case "0 hop" for omx_perf (note: MTU is 1500 and no other tweaking. |
| | 44 | |
| | 45 | {{{ |
| | 46 | length 0: 6.992 us 0.00 MB/s 0.00 MiB/s |
| | 47 | length 1: 6.495 us 0.15 MB/s 0.15 MiB/s |
| | 48 | length 2: 6.460 us 0.31 MB/s 0.30 MiB/s |
| | 49 | length 4: 6.575 us 0.61 MB/s 0.58 MiB/s |
| | 50 | length 8: 6.481 us 1.23 MB/s 1.18 MiB/s |
| | 51 | length 16: 6.541 us 2.45 MB/s 2.33 MiB/s |
| | 52 | length 32: 6.441 us 4.97 MB/s 4.74 MiB/s |
| | 53 | length 64: 7.045 us 9.08 MB/s 8.66 MiB/s |
| | 54 | length 128: 7.293 us 17.55 MB/s 16.74 MiB/s |
| | 55 | length 256: 8.629 us 29.67 MB/s 28.29 MiB/s |
| | 56 | length 512: 9.286 us 55.14 MB/s 52.59 MiB/s |
| | 57 | length 1024: 10.649 us 96.15 MB/s 91.70 MiB/s |
| | 58 | length 2048: 13.121 us 156.08 MB/s 148.85 MiB/s |
| | 59 | length 4096: 17.434 us 234.94 MB/s 224.05 MiB/s |
| | 60 | length 8192: 21.409 us 382.64 MB/s 364.92 MiB/s |
| | 61 | length 16384: 31.486 us 520.36 MB/s 496.25 MiB/s |
| | 62 | length 32768: 49.448 us 662.68 MB/s 631.98 MiB/s |
| | 63 | length 65536: 90.665 us 722.83 MB/s 689.35 MiB/s |
| | 64 | length 131072: 150.501 us 870.90 MB/s 830.56 MiB/s |
| | 65 | length 262144: 266.566 us 983.41 MB/s 937.85 MiB/s |
| | 66 | length 524288: 500.195 us 1048.17 MB/s 999.61 MiB/s |
| | 67 | length 1048576: 997.626 us 1051.07 MB/s 1002.38 MiB/s |
| | 68 | length 2097152: 2055.265 us 1020.38 MB/s 973.11 MiB/s |
| | 69 | length 4194304: 4249.680 us 986.97 MB/s 941.25 MiB/s |
| | 70 | }}} |
| | 71 | |
| | 72 | The worst case numbers (1 hop) are below: |
| | 73 | |
| | 74 | {{{ |
| | 75 | length 0: 14.197 us 0.00 MB/s 0.00 MiB/s |
| | 76 | length 1: 14.280 us 0.07 MB/s 0.07 MiB/s |
| | 77 | length 2: 13.970 us 0.14 MB/s 0.14 MiB/s |
| | 78 | length 4: 13.838 us 0.29 MB/s 0.28 MiB/s |
| | 79 | length 8: 13.538 us 0.59 MB/s 0.56 MiB/s |
| | 80 | length 16: 13.689 us 1.17 MB/s 1.11 MiB/s |
| | 81 | length 32: 13.636 us 2.35 MB/s 2.24 MiB/s |
| | 82 | length 64: 14.697 us 4.35 MB/s 4.15 MiB/s |
| | 83 | length 128: 15.761 us 8.12 MB/s 7.74 MiB/s |
| | 84 | length 256: 17.786 us 14.39 MB/s 13.73 MiB/s |
| | 85 | length 512: 19.116 us 26.78 MB/s 25.54 MiB/s |
| | 86 | length 1024: 21.683 us 47.22 MB/s 45.04 MiB/s |
| | 87 | length 2048: 25.549 us 80.16 MB/s 76.44 MiB/s |
| | 88 | length 4096: 28.822 us 142.11 MB/s 135.53 MiB/s |
| | 89 | length 8192: 38.929 us 210.43 MB/s 200.68 MiB/s |
| | 90 | length 16384: 52.026 us 314.92 MB/s 300.33 MiB/s |
| | 91 | length 32768: 76.272 us 429.62 MB/s 409.72 MiB/s |
| | 92 | length 65536: 144.706 us 452.89 MB/s 431.91 MiB/s |
| | 93 | length 131072: 224.688 us 583.35 MB/s 556.33 MiB/s |
| | 94 | length 262144: 384.179 us 682.35 MB/s 650.74 MiB/s |
| | 95 | length 524288: 704.347 us 744.36 MB/s 709.88 MiB/s |
| | 96 | length 1048576: 1390.370 us 754.17 MB/s 719.23 MiB/s |
| | 97 | length 2097152: 2910.845 us 720.46 MB/s 687.09 MiB/s |
| | 98 | length 4194304: 6007.359 us 698.19 MB/s 665.85 MiB/s |
| | 99 | }}} |
