Analyst David Floyer over at Wikibon has posted a new report on the cost effectiveness of InfiniBand switches for large computer networks. What I found interesting is his contention that the design of IB host channel adapters has become an application performance bottleneck thanks to the comparable processing power of today’s x86 chips:
The solution that best-of-breed InfiniBand architectures such as TrueScale employ is to execute the communications protocol in the server using a “lightweight” protocol that minimizes the memory and CPU cycle overhead on the server. Thus a Nehalem-based server using a host-based design could process approximately 5 times more messages than the same server with a tradition HCA-offload design. As the number of nodes in a HPC cluster increase, the message rate per server increases. Wikibon believes that for the vast majority of large-scale HPC applications, a host-based design will become the de facto standard over the next two years.
The report concludes that, for most supercomputing workloads, “Optimized IB configurations” could your save you as much as $1,117 per server. And that’s money you could spend on enhancing other parts of your computing infrastructure.