Comments on: Primer on InfiniBand for cluster buyers

By: John West

John West — Thu, 17 Sep 2009 14:38:19 +0000

Hey Jeff – that just means that you’re finding my site useful, so I’ll happily host the argument. By the way, I moved your link to your original post.

Oh, I responded to your post on your blog, by the way. I appreciate the time that you (and Brice) took in responding, and I yield to you both.

By: Jeff Squyres

Jeff Squyres — Wed, 16 Sep 2009 23:20:14 +0000

Brice’s point is valid (see http://blogs.cisco.com/ciscotalk/performance/comments/lies_damn_lies_and_statistics/). Beginners who read the HPCprojects article and will likely conclude “Ethernet = high latency.” That’s like saying “General Electric = Appliances” — but even though GE may be *best* known for its appliances, it has something like 12 or 15 other core businesses (jet engines, anyone?). So, too, Ethernet may be best known for TCP, but there are many other (significantly more efficient, lower latency, …etc.) transports available. They’re just a kernel module and userspace library away (no different than OpenFabrics or InfiniBand).

IMHO: To someone who is familiar with the market, this “primer” is pretty thinly veiled marketing. To a newbie, it is structured to guide them to the opinion that IB is the solution they need.

Just my $0.02 (and to be clear, just like I said in my blog entry — I’m just as biased as anyone else ).

(sorry you get the brunt of this, John — there doesn’t appear to be a way to make comments on the HPCprojects site )

By: John West

Tue, 15 Sep 2009 20:04:43 +0000

Brice – You are drawing a distinction between Ethernet as a substrate and TCP/IP that lays on top of Ethernet, and yes you are right that they are different. The primer (which was written at HPC Projects, not by me, just to be clear since you used the personal pronoun in your comment) addresses the common view of Ethernet+TCP/IP, which is the view that someone coming fresh to InfiniBand and/or building one of their first clusters is likely to bring to the table. Someone who understands the distinction you are making likely wouldn’t need the primer in the first place.

By: Brice

Brice — Tue, 15 Sep 2009 15:39:10 +0000

You are totally confusing things. You can do IB-like protocols over Ethernet. Mellanox does this with RDMAoE and the performance is almost the same. Myricom has even been doing MXoE for 4 years now (the performance was exactly the same there since there’s no MTU problem or encapsulation overhead like RDMAoE does).
The problem with Ethernet is not in Ethernet, it’s either in the above TCP/IP stacks, or in the fact that most Ethernet NICs do not offer advanced features such as RDMA. But Ethernet itself does not have the drawbacks you’re claming when compared to IB.