OFA to Streamline Verbs Interface

Print Friendly, PDF & Email

In this special guest feature, our insideHPC Performance Guru reports from the OpenFabrics Developers Workshop 2014.

Bill D'Amico

Bill D’Amico

I was at the Open Fabrics developers conference in Monterey, California on March 30 and April 1. While there were no major announcements of new products or technologies made at the conference, the main take away is that OpenFabrics Alliance members are working to streamline and make the verbs interface more efficient. Since this is an open source effort this will take some time, but the consensus is that there is too much overhead both in the depth of the call stack and in the size of associated data structures for verbs to scale well in the exascale era.

Both Paul Grun and Sean Hefty addressed the need to streamline Verbs in back to back talks. Applications do not need the full weight of the Verbs libraries. OFA has constituted a working group to develop a new framework for access to communication fabric that centers on the applications, not on the hardware. The idea is that the framework will be small extensible API like software interfaces. They won’t be tied to a specific hardware like the Verbs API (which parallels the Verbs semantics as defined in the InfiniBand architecture specification

Here’s a slide on the existing Verbs API – take aways are that Queue Pairs have a lot of heavy lifting to do (because it has to handle every possible Verbs activity), the model does not fit underlying hardware interfaces particularly well for non InfiniBand hardware.


A lighter weight API for each application needed interface could look more like this:


The green box is the interfaces that are visible to the application, purple box is implemented in an optimized library provided by the hardware vendor. The smaller, extensible interfaces in the green box allow the application to only use the services that application needs. If properly implemented the libfabrics shared library (purple box) will not have to provision all the data structures needed by the Verbs API. This means much less heap space usage by applications, freeing that memory for use by computations. Current Verbs implementations can use upwards of half a gigabyte of memory per node just for data structures!

The working group is having regular conference calls on Tuesday mornings at 9:00 AM pacific time. To join, sign up for the mailing list. This working group is operating using established open source processes. Participation by technical professionals is welcome.

You can check out all the OFA talks and presentations in our OpenFabrics Workshop Video Gallery.