Gilad is the VP of Technical Marketing @ Mellanox. That’s all.

You can run MX over 10G link layer to get lower latency, but this require MX on both sides,

As opposed to running IB on one-side? What is on the other side?

and still, it will be higher than the 2u stated above (the best Eth switches are 600ns per switch hop,

You should try a Fujitsu switch. They are less than 450 ns.

and MX itself is not lower than 2us…,

Latest NICs, latest CPUs, just under 2 us…

by the way Cisco switch latency is around 3us per hop…. definitely not good for MPI….).

You said it was fine when you were selling SDR NICs…

When you run 8 jobs at the same time, you will still get the 1us per job, but with 10G the latency will increase with job count. Bottom line, InfiniBand is still the best performance interconnect, and probably will stay like that in the next years.

Best micro-benchmark performance, perhaps. Let’s talk alltoall on a large fabric.

Scott

But once 10GbE over CAT takes hold (and it is a question of when, not if, imho), it’ll happen very quickly, because 10GbaseT is a natural evolution of the overwhelmingly ubiquitous network technology whose use is driven by so many things outside HPC.

Actually, I’m going to change my point slightly, because I think 10GbE dominance is also dependent on when Intel & Broadcom really decide to get into the market, and when Dell decides it’s time for 10GbE.

Also, in my experience, there’s an innate resistance to 10GbaseT, among the 10GbE startups, precisely because it will drive prices way down.

The real question, in my mind, is that when 10GbE does become ubiquitous, will it still be considered a high speed interconnect technology? Will Mellanox, Myricom and other have moved on to the next great thing? I hope so.

I’ve been waiting for almost 5 years for inexpensive 10GigE. Every year, the vendors keep saying, “it’s here, it’s here!” and the costs just are dropping fast enough.

I lived through the GigE price drop and that was fairly easy to see coming. But I just can’t see inexpensive 10GigE coming. The NICs are still too expensive and the switch costs are just too high (As I mentioned before, looking at 24-port switch prices for 10GigE is misleading at best. Building multi-tiered Ethernet switches from 24-port switches will just kill performance).

So I’m hoping someone comes along and sprinkles magic pixie dust on 10GigE and the prices drop to an acceptable level. Until then I just don’t see it being competitive to IB in HPC.

BTW – in the company I work for, we are seeing a resurgence in IB on the enterprise side because of the performance for systems with lots of VM’s and the fact that 10GigE is just not coming down in price like people would like.

And no-one should underestimate the importance of being able to run your interconnect over CAT-6/7, if only for the reason that you can cut your own cables with CAT, as opposed to having to buy expensive fibre/cx-4 cabling, that cannot easily be cut to length nor repaired in the field. Hell, cables are the single main reason the blade business exists.

Remember, we’ve seen this before with regular GbE. One minute, each NIC is $700, and the switches are hugely expensive, and the next the NICs are embedded on servers, and we’re buying 24port switches from Dell at less than $200/port. Quanta is already building low cost 24-port 10GbE switches for their OEMs.

Those who forget their history are doomed to repeat it.

- Price – switch prices depends on the switch configuration and the margin that the vendor take. You can find 24 port IB DDR switches from $3000 to $4000 to $5000. The statement on IB 24 port DDR switch in $16K seems to be unreal (else it is a golden switch….). The 10G 24 port switches are in the range of $16k-$24k, so if you look on price, IB is still much cheaper. Even IB QDR is cheaper than that.

- Performance – 10G is 10Gb/s real data rate, IB DDR is 16Gb/s and QDR is 32Gb/s. Latency – if you use standard IB vs standard 10G, it is 1us on IB vs 8-10us with 10G. You can run MX over 10G link layer to get lower latency, but this require MX on both sides, and still, it will be higher than the 2u stated above (the best Eth switches are 600ns per switch hop, and MX itself is not lower than 2us…, by the way Cisco switch latency is around 3us per hop…. definitely not good for MPI….). When you run 8 jobs at the same time, you will still get the 1us per job, but with 10G the latency will increase with job count. Bottom line, InfiniBand is still the best performance interconnect, and probably will stay like that in the next years.

- Woven – unlike 10G NICs, you can find a variety of IB adapters, from SDR to QDR, and from PCIe x4 Gen1 to PCIe x8 Gen2. There are multiple flavors, each with different latency capabilities as well. If you take the lowest performance DDR card (and the cheapest of course), and run it in a PCIe x4 interface, you will get lower bandwidth than 10G…. marketing tricks to show that 10G is not worse than IB.

- RDMA – not all application uses RDMA, but some does for good reasons. Not many people know that you get zero copy with InfiniBand if you are using RDMA OR Send/Receive. The different between RDMA and Send/Receive on IB is in the CPU overhead on the remote side (with RDMA the remote CPU is not innovated in the data transaction).

- 10GBaseT – this was the promise of 10G – use the “same” cables as 1G. Good motivation, the cables will be cheaper, but the switch cost much, much, much more. Do your own math. If you need the cat6 cables, go with the 10GBaseT, but the switch/NICs will be more expensive than the SFP+ ones.

- Cables – most of the installation will go SFP+ for 10G, CX4 for IB DDR and QSFP for IB QDR. at the end of the day, the cable cost is the same for both technologies

Enough for now …..

It is the case today that 10GbE is a more expensive to deploy than IB. Therefore, it is incumbent on the DC manager to decide between deployment costs, TCO and manageability.

You are confusing Ethernet and TCP/IP over Ethernet. With MX over Ethernet and a good low-latency switch, you can get 2 us.

Running HPL over 10GbE (NetEffect NICs + Woven Switch), and HPL over 4X DDR produced results that were indistinguishable from each other, at a reasonable node count.

That’s not to say that Infiniband isn’t faster than 10GbE, but it does suggest that when you take everything into account (PCI-E bus, driver overhead/efficiency, etc…), with real or near-real applications, using moderate packet sizes, the actual difference between the two is minimal for most real-world cases.

Of course, that should change somewhat as PCI-Express 2.0 and QDR Infiniband gain market traction, but I’m guessing that in real applications, unless you’re running something like Amber, it won’t make much difference, performance wise.