The VIA-inspired networks may get all the attention, but they aren’t the only ones available. Myrinet, from Myricom, is second in popularity only to Ethernet. It has been around for a long time and has a lot of community support. It is also quite cheap, by HPC standards.
Then there is the SCI (Scalable Coherent Interface) standard, produced primarily by Dolphin. SCI is unique in that the topology is specified to be a ring. Thus, the user can build a ring of rings to create a torus. The direct connection (no router) makes this an extremely low latency solution.
SCI also does not require the pinning of memory for RDMA; all remote memory is directly accessible by the remote process’s virtual address. If the page has been swapped to disk, the NIC will hold the data until the page comes back. Protection occurs through swapping process ids during job startup. Thus, protection is for the whole process, not select regions of memory.
This scheme makes RDMA faster, easier and more scalable because there is no need to exchange permission keys. It also, unfortunately, limits the applications of RDMA to more “static” settings. The lack of permission keys prohibit the use of this network in systems where processes come and go.
A third, newer network is QsNet, from Quadrics. As with SCI, there is no pinning for RDMA, all remote memory is accessible. QsNet also offers hardware support for collective communication, thereby giving greater scalability for operations such as one-to-many broadcasts and many-to-one reductions.
Myrinet, SCI and QsNet are the main competitors to Ethernet and InfiniBand. It remains to be seen how viable these technologies will be in an increasingly commoditized market place.
