I received this question in reference to an article from a few months ago. My paper was about functionality instead of mere performance, though my comments regarding RDMA-based overhead should hint at how poor InfiniBand is for some applications. Many of the benchmarks out there assume that the memory region is being reused and that the protection tags can be cached, which isn’t the case when there are numerous communication partners in the system.
As for 10 Gig E, vendors typically offload TCP onto the card, which takes care of most issues when communicating over the Internet Protocol. The real question is whether 10 Gig E can match InfiniBand for IP-based communication. I believe it already can.
It is certainly possible to tweak an IB app to run faster by using uDAPL in place of Sockets, provided there are few communication partners. Oracle RAC does this by restricting communication to selected pre-determined pairs; that is, there is no free-for-all that one typically finds in open client / server architectures.
Most customers would be served equally well with Ethernet. The reason I’m pushing that network is that it is much more commodity than InfiniBand. And indeed, we now see that vendors are pushing a hybrid solution, such as iWARP, Myri-10G, and QsTenG. That is, vendors with experience in high-performance computing are building on Ethernet and pushing it for enterprise markets, in addition to their traditional technical markets. The overall goal isn’t performance (though they certainly are achieving that) but rather price.
