Sign up for our newsletter and get the latest HPC news and analysis.
Send me information from insideHPC:

HDR InfiniBand Technology Reshapes the World of High-Performance and Machine Learning Platforms

In this special guest feature, Gilad Shainer, Gerald Lotto, Scot Schultz from Mellanox write that the coming wave of 200 Gb/s HDR InfiniBand will reshape the world of high performance computing.

hdrThe newest InfiniBand technology, 200Gb/s HDR InfiniBand, was announced at SC16 (November 2016). The prior generation, 100Gb/s EDR technology, was announced late 2014, indicating a two-year cadence between interconnect generations, which demonstrates the ever-increasing need for higher data throughput. It is important to mention that InfiniBand, as an industry standard, guarantees full backward and forward compatibility. Also, as a complete Software Defined Network technology, it enables full flexibility to create unlimited system configurations and topologies, from non-blocking and oversubscribed fat tree to mesh, hypercube, enhanced hypercube, Dragonfly, 3D Torus, 6D Torus and more.

The announcement of HDR InfiniBand included the three required network elements to achieve full end-to-end implementation of the new technology: ConnectX-6 host channel adapters, Quantum switches and the LinkX family of 200Gb/s cables. The newest generations of InfiniBand bring the game changing capabilities of In-Network Computing and In-Network Memory to further enhance the new paradigm of Data-Centric data centers – for High-Performance Computing, Machine Learning, Cloud, Web2.0, Big Data, Financial Services and more – dramatically increasing network scalability and introducing new accelerations for storage platforms and data center security.

Data-Centric computing is based on an interconnect environment that can act as a co-processor, enabling synergistic offload opportunities for protocols and computation such as data aggregation and reduction operations. These operations are critical to overcoming latency bottlenecks in High-Performance Computing, Machine Learning and other applications (Figure 1). The SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) technology that was introduced in previous generation of the network adapters has been enhanced in ConnectX-6 especially for the needs of Machine Learning applications, adding support for aggregation and reduction of larger data sets typical of such applications.

Figure 1 – CPU-Centric versus Data-Centric data center architecture

Figure 1 – CPU-Centric versus Data-Centric data center architecture

There are multiple advantages offered by the SHARP technology, including the aggregation of data on the network fabric itself, which achieves magnitude of order performance improvements; reducing the amount of information that must be moved across the data center; and removing the processing overhead that would be required from the CPU.

MPI Tag Matching is another capability supported by ConnectX-6 that further reduces MPI latency and CPU overheads. Offloading of MPI Tag Matching enables the migration of the entire MPI rendezvous protocol from the software to the interconnect hardware. As a result, nearly 70% of MPI operations can be performed on the interconnect hardware instead of by software implementation.

The net result is that HDR InfiniBand interconnect solutions offer more than just a doubling of bandwidth to 200Gb/s. Real application accelerations of up to 10X have been demonstrated for workloads in many different disciplines, including weather and manufacturing simulations, material science and molecular modeling, computational fluid dynamics and, of course, cognitive computing and Machine Learning. Figure 2 summarizes the HDR InfiniBand offloading and In-Network Computing engines.

Figure 2 – HDR InfiniBand offloading and In-Network Computing engines

Figure 2 – HDR InfiniBand offloading and In-Network Computing engines

This generation of InfiniBand also introduces a novel innovation for lowering the cost of network design and network topologies. HDR100, a new capability for 100Gb/s data speed, has been announced, which enables a 4-lane 200Gb/s port to be split into two distinct 2-lane 100Gb/s ports. In so doing, the HDR switch provides double density radix for 100Gb/s data speeds, with up to 80 ports of HDR100. The new HDR100 protocol is supported by ConnectX-6 InfiniBand adapters and Quantum InfiniBand switch systems, and is both backward and future compatible with any InfiniBand technology
or speed.

Figure 3 – CAPEX and OPEX savings with HDR100 versus OmniPath. 1536-node system size is used as an example, as it is optimized for OmniPath switch radix. HDR100 provides 3X real estate savings, 4X cable savings and 2X power savings.

Figure 3 – CAPEX and OPEX savings with HDR100 versus OmniPath. 1536-node system size is used as an example, as it is optimized for OmniPath switch radix. HDR100 provides 3X real estate savings, 4X cable savings and 2X power savings.

While performance is the top priority for most HPC practitioners, cost is always a consideration as well. HDR100 is a game changer for data center fabric designs. The effective switch radix of 80 ports in a 1U form factor means higher densities, less real estate, and fewer network components, saving both power and cost. For example, a simple two-tier non-blocking CLOS fabric design using Quantum could support 400 nodes at 100Gb/s with only 15 1U switches and half the cables that would be required to build the same connectivity with 24 Omni-Path switches. The largest Quantum director-class switch can support up to 1600 nodes with non-blocking 100Gb/s HDR100 connectivity in a 24kW configuration, offering significant savings in space, power, cooling, cabling, overall cost, and complexity over Omni-Path (Figure 3). HDR100 enables data center managers to build up to an 80-node cluster with a single 1U switch, up to 3600 nodes with only two layers of switches in a fat-tree network topology, and up to 128,000-nodes with only three layers of switches, nearly 5X higher scalability than competitive products.

Another technological advancement aimed at lowering TCO for environments that do not need full HDR bandwidth is Multi-Host technology. With ConnectX-6, up to eight separate hosts (servers) can share a single adapter, resulting in significant cost reductions not only for network adapters, but for cables and switch ports as well. Another capability supported by the Multi-Host architecture is Socket Direct to bypass QPI-bus reliance for network access. This can reduce latencies by about 20% and reduce CPU utilization by up to 50%.

InfiniBand continues its leadership in the world of interconnect technology from generation to generation with key advancements in technology, capabilities, and efficiency, enabling the path to Exascale and beyond. With the changing landscape of HPC, spilling over into the areas of Machine Learning and cognitive computing, Web 2.0 and the Internet of Things (IoT), Cloud computing, and Hyperscale supporting business intelligence and analytics, the need for extreme performance has never been higher. HDR 200Gb/s network technology is aimed to deliver the data speeds required today, and to provide a competitive and cost/performance advantage to its users.

Sign up for our insideHPC Newsletter

Leave a Comment


Resource Links: