Comments on: Special Feature: Collective Performance in InfiniBand

By: Video: PSM – A Major InfiniBand OFED Enhancement | insideHPC.com

Video: PSM – A Major InfiniBand OFED Enhancement | insideHPC.com — Mon, 28 Mar 2011 12:24:05 +0000

[...] Yaworski is the author of a guest feature posted here recently on PSM and InfiniBand collectives. [...]

By: Greg Lindahl

Greg Lindahl — Fri, 11 Mar 2011 18:47:11 +0000

Matt, PSM is *optional*. You can also use QLogic HCAs as a normal InfiniBand adaptor. The only difference is that PSM has a lot higher performance for small packets and collective operations. You can not achieve this higher performance using stock InfiniBand.

The standard that all HPC people care about the most is MPI. The most popular MPI implementations support PSM.

By: Matt

Matt — Thu, 10 Mar 2011 21:50:08 +0000

So the comparison between PSM and Myrinet makes sense now. When people finally learn that proprietary is the wrong way???

By: Greg Lindahl

Greg Lindahl — Thu, 03 Mar 2011 22:13:52 +0000

PSM is the library needed to implement MPI using InfiniPath’s (now TruScale’s) InfiniBand extension. It wasn’t exposed to outside users for the first year or so that InfiniPath was shipping. Last I looked, PSM was used to enable a bunch of MPI implementations, including OpenMPI, Platform MPI, and MVAPICH (part of OFED).

Christian Bell (now of Myricom) gets credit for making PSM a useful API, we did a terrible job of designing the API before he showed up.

By: Chris

Chris — Thu, 03 Mar 2011 20:48:31 +0000

Greg, is PSM the InfiniPath API? I was at an early PathScale customer (Australian National) years ago. From what I remember at the time, PathScale only offered MPI to customers initially and then rolled-out verbs support (back when OpenFabrics was known as “OpenIB”) just before the QLogic merger. Has PSM always been offered for customer use?

By: Greg Lindahl

Greg Lindahl — Thu, 03 Mar 2011 19:36:40 +0000

I’m confused that people are confused about PSM. PSM has always been a part of PathScale/QLogic’s InfiniBand offering, and PSM has been supported by all the major MPI implementations for several years now. PSM is how InfiniBand should have been designed in the first place.

By: Ben

Ben — Wed, 02 Mar 2011 21:14:47 +0000

From what I have learned after checking around, there are no plans from Voltaire/Mellanox to adopt PSM but to stick with the IB standard verbs specification. As you said Chris, PSM looks more like Myricom MX, and probably will stay like that.

Joe – I have looked at the Mellanox web site but did not see the document that you referred to. Can you point to the URL?

By: Chris

Chris — Wed, 02 Mar 2011 19:07:32 +0000

Digging around the inter-webs, Performance Scaled Messaging (PSM) is indeed in OFED, but only QLogic appears to support it. It looks similar to Myricom’s MX library.

It’s a shame most 10 GigE vendors aren’t as obsessive over software as Myricom. Do other InfiniBand vendors plan to support PSM?

If there isn’t wider support for PSM, then non-MPI users will probably just stick to verbs. (And if the goal is really just to have TCP, most customers will stick with vanilla 10 Gig E. Keep in mind who your *real* competitor is before even bothering with something as superficial as performance.)

By: UncleKracker

UncleKracker — Tue, 01 Mar 2011 23:24:17 +0000

This discussion on PSM is giving me PMS

By: Ben

Ben — Tue, 01 Mar 2011 23:14:38 +0000

Yes, you are right, we can argue forever, but as long as the data is presented correctly, I will not have any issues. I did review the data on the Fluent web site, and it is full of different platforms and setting. However, I encourage you to visit the same page, since there are other InfiniBand numbers (I assume that what is not marked as TrueScale is either from Voltaire or Mellanox) which are better than the numbers you listed in your article, so I assume that you have carefully picked not the best numbers listed on the page. I have also noticed that with the Aircraft benchmark, QLogic did NOT scale beyond 16 nodes – the performance at 16 nodes was 9573.4 and the performance at 32 nodes was 3712.1 (higher is better) – so the performance actually dropped from 16 nodes to 32 nodes, while all other Linux based solution did scale. This info you did not capture in your nice article. So I guess that if I want to build more than 16 nodes with InfiniBand, I should shop somewhere else.

By: Joe Yaworski

Joe Yaworski — Tue, 01 Mar 2011 22:33:31 +0000

Ben, please do not get upset at this article beause of the work that Voltaire did in testing and analyzing the collective performance of Mellanox technology. I would suggest that you discuss your concerns with the Voltaire personnel that did this testing and came to these conclusions. I would hope that the Voltaire results are factual, and they are not “marketing work at best”.

As for open standards, PSM is as open as Verbs. Both are published in OFED and require a specific adapter architecture to run.

If you had gone to the FLUENT benchmark site (link provided above), you will find that the performance of systems with TrueScale IB, single rail to single rail IB implementations, are best in class at 16-nodes+ for FLUENT 12.1. In fact, in most cases, these systems with TrueScale IB also provide better performance than dual rail IB (non-TrueScale) implementations. The reason for this, in part, is due to the native collective performance of TrueScale with PSM that was covered in this article.

Ben we could go on forever commenting about this. The conclusion is that the performance results (Voltaire and ANSYS) are published and accessable to everyone.

By: Ben

Ben — Tue, 01 Mar 2011 20:05:13 +0000

It is wonderful to see how you use the previous Voltaire marketing against the Mellanox solution to bash the Mellanox offloading …  marketing work at its best…. I would not use those statements if you want to write something meaningful. It really reduces from your credibility. By the way, ORNL did publish much higher node count results – check it out.

Now to the serious discussion – using light or lighter software stacks is not something new – Cray has their own version, as well as IBM. It is no more than creating proprietary software interfaces that connects to the hardware interface of the NIC. Verbs interface can work great on the Mellanox/Voltaire solutions and bad on QLogic, and PSM can work great on QLogic and bad on Mellanox. The question is what is proprietary and what is not, what is part of an open specification and what is not, and might go the same way Quadrics went.

By the way, you did not answer my Fluent questions – are you comparing the same platforms, same OS etc? saying that the your numbers are on Fluent web site does not help to understand if you are comparing apples to shoes… please also note that Eddy benchmark is no longer important to us – it is too small and no one uses it.

By: Joe Yaworski

Joe Yaworski — Tue, 01 Mar 2011 19:30:38 +0000

The ORNL research is very, very limited in scale, given that their research was done on just 8 nodes. Otherwise, the information would have been used. The collectives performance numbers at scale came directly from the Mellanox/Voltaire white paper on FCA. The TrueScale collective performance numbers come from QLogic’s runs on ~2000 node Westmere cluster at LLNL. As for the benefits of “offloading” here is a direct quote from that same Mellanox/Voltaire white paper. “A common approach to this challenge improves the implementation of MPI collective operations by using intelligent or programmable network interfaces to offload the burden of communication activities from the host processor(s) within the NIC. Such implementations have shown significant improvement for micro-benchmarks that isolate collective communication performance, but these results have not translated to significant increases in performance for real applications.”

BTW, the TrueScale Fluent numbers are official results published on the ANSYS FLUENT benchmarks site (http://www.ansys.com/Support/Platform+Support/Benchmarks+Overview), which gives the platform information. The Mellanox & Voltiare FLUENT come from the Mellanox/Volatire whitepaper and there are also results published on the ANSYS FLUENT site.

By: Joe Yaworski

Joe Yaworski — Tue, 01 Mar 2011 18:36:40 +0000

The Convention-IB and Voltaire FCA mentioned in the article are Verbs based. QLogic TrueScale IB Collective results are PSM based. PSM is a streamlined, lightweight, message based interface for MPI. As a result, PSM performance with MPI and in particular Collectives, does not require any special acceleration hardware or code. This also means that PSM’s native/inbuilt acceleration is available to all major MPI’s and for all collectives functions. PSM is a new capability in OFED that the QLogic TrueScale IB is designed to take advantage of.

By: Ben

Ben — Tue, 01 Mar 2011 18:32:15 +0000

This article is no more than comparing apples to shoes. ORNL already demonstrated flat latency using the adapter based collectives offloads, and there was definitely no bottlenecks reported in their publications. Therefore numbers presented here are very questionable. The Fluent benchmarks lack any platform information, setup information and version information therefore not credible at all. This article is no more that a marketing propaganda, and I would refer to some papers published by credible organization for real comparison.

One big thing missing here for example is the purpose of offloading the collective communications – to take advantage of non blocking collectives and achieve nearly optimum communication overlap. I would suggest the author to do some homework by readying some of the publications made in the past by Quadrics and more recently by ORNL to learn more on the subject.

By: Chris

Chris — Tue, 01 Mar 2011 18:14:14 +0000

I too would like an explanation of just what this is. What does QLogic have that other vendors don’t? The marketing speak above says that this is standard and doesn’t require a special adaptor, so why doesn’t this work on all vendors’ products?

Also, the notion of using the switch to accelerate collective communication is something that Quadrics implemented many many years ago.

By: Jeff Squyres

Jeff Squyres — Tue, 01 Mar 2011 17:11:00 +0000

Are these new verbs API function calls?