What is iWARP?

The Internet Wide Area RDMA Protocol (iWARP) is an Internet Engineering Task Force (IETF) update of the RDMA Consortium’s RDMA over TCP standard. This later standard is zero-copy transmission over legacy TCP.

Because a kernel implementation of the TCP stack is a tremendous bottleneck, a few vendors now implement TCP in hardware. As simple data loses are rare in tightly coupled network environments, the error-correction mechanisms of TCP may be performed by software while the more frequently performed communications are handled strictly by logic embedded on the NIC. This additional hardware is known as the TCP offload engine (TOE).

TOE itself does not prevent copying on the receive side, and must be combined with RDMA hardware for zero-copy results. The RDMA / TCP specification is a set of different wire protocols intended to be implemented in hardware (though it seems feasible to emulate it in software for compatibility but without the performance benefits).

The main component is the Data Direct Protocol (DDP), which permits the actual zero-copy transmission. DDP itself does not perform the transmission; TCP does.

However, TCP does not respect message boundaries; it sends data as a sequence of bytes without regard to protocol data units (PDU). In this regard, DDP itself may be better suited for SCTP, and indeed a focus of the IETF has been to standardize RDMA over SCTP. To run DDP over TCP requires a tweak known as marker PDU aligned (MPA) framing so as to guarantee boundaries of messages.

Furthermore, DDP is not intended be to accessed directly. Instead, a separate RDMA protocol (RDMAP) provides the services to read and write data. Therefore, the entire RDMA over TCP specification is really RDMAP over DDP over MPA over TCP. All of these protocols are intended to be implemented in hardware.

Like InfiniBand, iWARP does not have a standard programming interface, only a set of verbs. Unlike IBA, iWARP only has reliable connected communication as this is the only service that TCP and SCTP provide. The iWARP specification also omits many of the special features of IBA, such as atomic remote operations.

In all, iWARP is the basics of InfiniBand applied to Ethernet. This gives it tremendous capabilities for both legacy software and next generation applications.

Word of caution: “iWarp” is also the name of a supercomputing project started at Carnegie-Mellon University in the late 1980′s. This work covered what now might be called “system on a chip.” Be careful when looking for iWARP online as many search results refer to the CMU system, which has nothing to do with the IETF standard.

Resource Links: