The paper “A Critique of RDMA” presents a good argument against remote direct memory access in high-performance networks. An overview of the technology is present in an earlier post on the Virtual Interface Architecture (VIA).
The VIA model requires an enormous amount of overhead for the initial communication because the software layer must pin the pages and swap the permission keys. This may be permissible in applications that match with remote memory access (such as partitioned global address space languages) because of various caching techniques. Other applications, like remote procedure calls, do not map well to remote memory access and thus do not permit caching.
It should be noted though that QsNet does not have pinning as a kernel patch performs DMA transfers once the page has loaded into memory on its own. This method is not particularly portable as each version of the kernel must be modified slightly differently, which makes software maintenance difficult for IT managers. Also of note is that Blue Gene does not have virtual memory, and therefore ensures that pages are always in memory. Both IBM’s and Quadrics’s approaches are not commodity, however, and are therefore expensive.
Given that most software in networked systems will require explicit notification of message receipt, the remote memory access model is not particularly adequate, and thus the RDMA constructs are not particularly useful. (As a side note, partitioned global address space languages are best targeted for ccNUMA machines, though these are even more expensive.) The aforementioned paper basically states that network vendors would be better off pushing constructs for programmed IO, such as message matching on the NIC. This indeed appears to be the best approach of all as it maps well to Sockets.
