The Cray XC series is a distributed memory system capable of sustained performance in the petaflop range. Creating a large server farm with fast CPUs doesn’t map well to applications that require storage connectivity, as most do, or socket to socket communication within the overall system. Thus, a flexible and high speed networking solution is critical to the overall performance of the computing system.
The Cray XC network design addresses a number of challenges for a range of installations. These include:
- Systems can be configured to match bandwidth requirements and cost constraints by varying the number of optical connections
- Address space of general purpose processors is extended to efficiently access all the physical memory in the system
- Suite of communication and synchronization mechanisms: block transfer engines for asynchronous pipelined memory copies, atomic memory operations, barriers, source and destination-side synchronization, message delivery doorbells and collective combining trees
- Network protocol designed for efficient transfer of 8- to 64-byte messages at low overhead; combination of small packet sizes and a rich set of packet-by-packet adaptive routing algorithms ensure high utilization under heavy load
To accomplish these challenges, Cray designed the Aries interconnect that demonstrates substantial improvements on standard network performance metrics. The Aries interconnect uses a design that incorporates four high performance network controllers and a high radix router.
As part of the Cray networking environment for the Cray XC30, the Aries interconnect and related software stack have been optimized for parallel computing environments. The Cray software stack includes a high performance MPI implementation as well as the Cray SHMEM API. Leading edge innovative applications require very high global bandwidth that is delivered through the Cray Dragonfly design. This whitepaper discusses, in technical detail, fat-trees and why HPC implementations use them. The Aires system-on-a-chip is discussed in great detail, giving the readers an excellent insight as to its design and use cases. Each Aires chip provides networking to four nodes on a Cray SC30 compute blade.
The performance of this interconnect have been measured using a variety of tests and are detailed in full in this technical whitepaper. The work at Cray has resulted in a single state-of-the-art component – the Aries network ASIC – that powers the communication network of the Cray XC series supercomputer. The Dragonfly network used by the Cray XC series provides cost-effective, scalable global bandwidth. This design together with the Cray Linux Environment and advanced RAS features allows Cray XC systems to scale to millions of cores. The Cray XC programming environment enables efficient use of these resources on the most demanding HPC applications.
Download the Cray XC Series Network whitepaper now !