I would also like to point out that many good transports have come and gone due to many different reasons. The market would say that since QLogic and Mellonox are the only manufactuors of infiniband ASICS that it might not have much of a future. If it weren’t for the national labs, infiniband would have died years ago. Reference your top 50 largest machines, 50% are in the US and of those 60% are federally funded. Without that gravy train, there would be no infiniband. Watch to the coming budget to dictate their survival. It is an uphill battle to convince me that legislation doesn’t shift economies. Hold the glasses down, the tablecloth is about to be pulled.

I agree that infiniband has a very low latency and outstanding performance but it is nowhere near the sub-100nano that the BlueGene machines can pull off. If scientific computing is the job, then use a proprietary interconnect that out performs infiniband and is custom built for the program’s needs. There is no need to keep this “middle” open technology alive other than for pet projects. Any cost/benefit review proves the TCO model.

What we have here are computer people who know little of business and business people who know little of computing. The truth will not be found in either camp. The only wisdom I can offer is, “We’ll see” as we move down the roadmap. Is it the general consensus that in the economy will continue to develop separate modes of communication? Do we not see the righting on the wall? If the path of transports was to be developed in parallel, then we would still have wide spread ATM, x.25, Banyon, Novel, etc… Ethernet has converged everything from power, Audio/video, Telephone, and now Fibre Channel. Infiniband is in the sights and it will do to it what it did to ATM. Throw enough cash at something and you will find a solution. The tone from OFED in Sonoma to the HPC Forum is, “Get ready” Channel I/O is going to be a layer on Ethernet. It will all be converged. Slim L2 RDMA functionality will be a vLAN configuration with TRILL enabled. 20x electrical lanes running 10GigBaud is already pushing 200Gb/sec to a CFP. The CMOS wafer is almost at 28nm. Size is not a factor, heat is not a factor, fiber is now cheaper than copper with attached optics, and now we are seeing well under 5W per port on 10Gig. It is moving too quickly for Infiniband and their captive IBTA to stay ahead.

If you read this far and are not furious, then you have already come to the last stage of truth: wide spread acceptance. This business goal and technology direction should be self-evident to you. If it is not, then you have been cooped up in your camp too long and are starting to get weird. Step outside and look around.

Comments welcome-

Perhaps this explains why there’s little pickup of verbs interfaces in the enterprise: development time his a more important impact than on HPC were applications and middleware lives on for decades.

Anyway, back to HPC. Since these verbs are indeed so low-level, one needs to consider the impact of how they evolve and are implemented by the InfiniBand community. Here is the life-cycle of how OpenFabrics adopts new features when, say, Vendor A has to address a shortcoming in the existing interface or simply evolve the interface for a newer generation of hardware:
1. Vendor A implements new feature through a verb which, because of the low-level nature of the API, requires modifications to Vendor A’s entire stack (software down to hardware).
2. Vendor A proposes that OpenFabrics adopt the verb so the community can take advantage of the said new feature.
3. OpenFabrics adopts the extension and it is rolled out in the next release.
4. Vendors B and C scramble to *emulate* the verb in time so they too can tout Open Fabrics support.

Here’s the problem: Vendor A is the *only* InfiniBand vendor to extend and natively implement new verbs in their silicon. Therefore, InfiniBand *is* Vendor A while others are me too implementations thereof.

Cisco realized they could only lose when evolution of InfiniBand became so tied to vendor A’s silicon so they dropped InfiniBand. Plus, there are already dozens of companies taking the said silicon and customizing it for a (demanding) HPC community. This makes for tough competition on details that don’t have to be dealt with in the Ethernet world where customers are low touch and requires less specialization.

Now Vendor A wants to bring its vision of the world to Ethernet. It will be interesting to see how that goes. Ethernet has much more silicon providers and has always flourished within bounds that are more loosely defined than InfiniBand’s. Of course, the pace of adoption is much slower in the Ethernet world but this shouldn’t come as a surprise since there are many silicon providers to set the rules. One has to wonder how the Ethernet community as a whole would swallow the large set of silicon-dependent nuts and bolts called “OpenFabrics verbs”.

On the flip side, the HPC community could benefit from Vendor A leveraging its Ethernet side of the business to help sustain its niche investments in evolving InfiniBand. We already know that we have valiant armies of programmers ready to take on that challenge.

“Simplicity really means getting commercial hardware and software products that are proven, easy to use and provide the required high performance without having to design complex software schemes.”

I refer to slide 5 in my presentation at the Sonoma OpenFabrics conference this year:

http://www.openfabrics.org/archives/spring2009sonoma/wednesday/panel1/panel1.zip

It’s a simple graph of lines of code in Open MPI for each different network transport. If you don’t want to open up the zip file, I’ll copy the numbers here:

* MX: 1,210 and 2,331 (Open MPI has 2 different “flavors” of MX support)
* Shared memory: 2,671
* TCP: 4,159
* OpenFabrics: 18,627

Yes, that’s right — OpenFabrics requires 18 *THOUSAND* lines of code in Open MPI — TCP is less than a quarter of that. MX is less than half again of that. Note that the 18K number doesn’t even include any of the memory registration code that is only used by OpenFabrics (not TCP, shared memory, MX, …etc.). So 18K is actually below the real number.

Therefore, I do believe that Doug’s statement is correct: Speed, Simplicity, Cost, pick any two. “Simplicity”, as defined by Patrick, is definitely NOT met by the API used by Infiniband (i.e., OpenFabrics verbs).

(sorry, this just raises a favorite rallying point of mine — the OpenFabrics verbs API is actually *extremely* low level and quite difficult to write to for user-level applications. FWIW: it’s been (accurately, IMHO) remarked that OF verbs looks and feels much more like a kernel-level interface than a user-level interface)