Sign up for our newsletter and get the latest HPC news and analysis.
Send me information from insideHPC:

Intel Omni-Path Architecture: End-to-End Fabric Solution Designed for HPC and Enterprise

This is the third of four articles on Intel Omni-Path Architecture (Intel OPA) next-generation fabric for high-performance computing (HPC) solutions. Three other articles cover what an HPC application needs, Intel OPA performance, and Intel OPA’s impact on the Top500 list.

Intel is no stranger to interconnect technologies and High Performance Computing (HPC). The company has been producing Ethernet controllers and involved in HPC for over 20 years. But, Intel Omni-Path Architecture immerses Intel in an entire fabric architecture for HPC, rather than just a single component.

Intel OPA is the fabric that ties together an entire HPC system, integrating host fabric interfaces (HFIs), edge switches, director-class switches, optics and cables, and software—all from Intel. “For a new technology, Intel OPA is very mature in terms of performance and stability,” said Joe Yaworski, High Performance Fabric Marketing Director at Intel, “because we leveraged established InfiniBand and Aries technologies that we acquired, and then enhanced them with Intel technologies and software.” Here’s an overview of Intel OPA components.

Joe Yaworski: 'For a new technology, Intel OPA is very mature in terms of performance and stability.' #datacentersClick To Tweet
Intel OPA

Intel OPA Host Fabric Interface (HFI)—Evolving from the Intel True Scale Fabric adapters, Intel OPA’s HFI is designed specifically and optimized for the needs of HPC.

Intel OPA Host Fabric Interface (HFI)—Evolving from the Intel True Scale Fabric adapters, Intel OPA’s HFI is designed specifically and optimized for the needs of HPC. It uses an advanced connectionless design to deliver scalable performance for large clusters, especially when running MPI codes. Leveraging an enhanced, next-generation version of the Performance Scaled Messaging (PSM) library, Intel OPA HFIs offer very high message injection rates with low latency compared to InfiniBand* verbs. Additionally, the Intel OPA HFI is intelligent, using the best method available to get application traffic onto the wire with minimum latency. It either establishes an RDMA channel for large loads, or leverages the high performance of the Intel Xeon processor in the node to handle the communications processing for smaller messages. Intel OPA dynamically selects what is best for the application.

Intel OPA Edge Switches—Designed around Intel’s 48-radix switch silicon, Intel OPA condenses fabric communications at the rack level into a much smaller footprint, enabling larger islands of nodes with full bandwidth compared to the 36-port switches used by other technologies. The more condensed Edge Switch reduces overall switch count, making more room for compute nodes and helping to reduce the overall cost of a cluster solution because fewer switches are needed. But that’s just the hardware benefit. In the software, Intel OPA uses Traffic Flow Optimization to keep high-priority data efficiently moving through the switch, rather than making it wait for lower priority data streams. Plus, Intel OPA’s Packet Integrity Protection assumes data delivery is successful unless told otherwise, checking a packet CRC at each switch hop and requesting a resend only when needed, instead of waiting for an error to propagate to an endpoint. These technologies enhance operation in the fabric, delivering low-latency, scalable, and predictable performance under load.

Intel OPA Director-class Switches—Leveraging the same 48-radix silicon, Intel OPA Director-class Switches offer dense form factor designs capable of supporting as many as 768 100 Gbps ports, and they can scale up to 153.6 Tbps bandwidth with 300 ns switch latency. Like all of Intel OPA, the switches are optimized for HPC’s needs for high message rates and throughput.

Management Software—Configuring, tuning, and managing the fabric allows the integrator and the system administrator to optimize fabric performance, troubleshoot issues, and monitor activities across the network. The Intel OPA Fabric Suite of software provides comprehensive control of administrative functions using a GUI-based interface. Today’s open source application software written for existing OpenFabrics Alliance* interfaces will run on Intel OPA with no code changes required. Best of all, the software from Intel is licensed at no cost and contributed to open source to the HPC community.

Intel OPA is an end-to-end fabric with all the components expected of a comprehensive high-performance network, designed to work together seamlessly for the workloads that enable today’s deep discoveries and insight. As an HPC fabric solution, it has had considerable impact in the industry since its introduction. That’s what we’ll look at in the next article.

More information is available on the Intel OPA web site.

Leave a Comment


Resource Links: