The past two weeks or so have seen a lot of activity in comments and email about HPC interconnects, most particularly about the relative merits of 10 GbE and InfiniBand.
Earlier this month I pointed to Doug Eadline’s article at Linux Mag arguing that 10GbE will see increasing adoption in HPC
There are some things to consider, but in general, “Good Old Ethernet” is taking its next big jump. I am convinced that over the next year there will be a significant up-tick in 10 GigE HPC clusters.
In a move that we just love here at insideHPC world HQ (yes, my living room), reader Patrick Chevaux responded back and gave us permission to publish his thoughts:
I agree with the title of your article (10GE ready for HPC).
However I want to make a short comment about the following paragraph from Doug Eadline :
“My 10 GigE prediction is based on the following rule of thumb, Speed, Simplicity, Cost, pick an two. I believe 10 GigE will win because of simplicity and cost. IB is already faster and has better latency and if you need this level of performance you are not even looking at Ethernet direction. The joy of clustering is that one size does not fit all and you can build your cluster around your needs.”
InfiniBand has won the HPC niche market exactly because of the 3 parameters cited above.
As we all know InfiniBand has always been and still is ahead of Ethernet in terms of “speed” (throughput or goodput and latency). One could also argue about the inherent features like flow control and separate lanes which provide better traffic handling than good old Ethernet. With the open IB architecture and a number of ongoing developments being done within the IBTA (InfiniBand Trade Association, www.infinibandta.org), the speed advantage is going to continue if not widen up. If not convinced look at the IB roadmap with QDR 8x, EDR and HDR coming soon.
Simplicity really means getting commercial hardware and software products that are proven, easy to use and provide the required high performance without having to design complex software schemes.
Ethernet is definitely easy to use with well-known integrated hardware and software stacks. The real problem is the use of old, inefficient protocols such as IP (V4), TCP, UDP and the like. As we all know performance of the IP stack is terrible and improving it is a major software burden.
InfiniBand on the other hand, shares the same basic hardware and software characteristics, silicon is widely available and cheap, drivers are integrated in operating systems, and user s can rely on high level programs like MPI to get very good interprocess performance
Cost is the key point for most users. And here, InfiniBand has won the battle with very little fight. HPC specialists and system Integrators, like ClusterVision, who used to rely on Gigabit Ethernet, Myrinet or Quadrics have quickly moved into selling only InfiniBand HPC interconnects, starting with InfiniBand SDR, moving to DDR in 2007 and now standardizing on InfiniBand QDR because of lower cost.
So, indeed the OpenFabrics Alliance has been doing the right thing in migrating all the InfiniBand “goodies” like : RDMA, User-mode I/O (zero-copy, kernel bypass), high throughput and low latency to Ethernet. Yet, InfiniBand is the undisputed leader in the Top500 list.
I see problems for future “data center” Ethernet in not having a proper hop-by-hop flow control scheme and not having an efficient traffic separation scheme like InfiniBand’s virtual lanes.
Finally I wish the 2 worlds will eventually converge :
- InfiniBand and Ethernet PHYs are so similar that it would be stupid not to merge
- InfiniBand has demonstrated that
- cheap high performance host adapters can be made
- cheap high performance switches can be made
I dream of 10GE, 40GE, 100GE, etc.. products providing the best of both worlds (high perf, low cost).
Patrick, thanks for taking the time to write in. And if you’ve got thoughts on this topic, please drop us a line or leave a comment on the blog. We love to hear back from you.