In this video from the 2013 Open Fabrics Developer Workshop, Robert Woodruff from Intel presents: OFS for the Intel Xeon Phi.
In this video from the 2013 Open Fabrics Developer Workshop, Sean Hefty from Intel presents: InfiniBand Scalable Subnet Administration – An OFA Project.
You can check out more OFA videos at our Open Fabrics Workshop Video Gallery.
In this video from the 2013 Open Fabrics Developer Workshop, Paul Grun from Cray presents: Evolving OFS for the Modern World – A Challenge to Attendees.
In this podcast, the program co-chairs of the 2013 Hot Inteconnects Conference discuss how the annual symposium covers cross-cutting issues spanning computer systems, networking technologies, and communication protocols for high-performance interconnection networks.
- Madeleine Glick (APIC Corporation)
- Torsten Hoefler (ETH Zurich)
- Fabrizio Petrini (IBM TJ Watson)
Hot Interconnects takes place in San Jose, California on August 21-23. The conference Call for Papers has been issued, with abstracts due April 26, 2013.
In this video from the Lustre User Group 2013 conference, Eric Barton from Intel presents: Lustre – Fast Forward to Exascale.
Back in July 2012, Whamcloud was awarded the Storage and I/O Research & Development subcontract for the Department of Energy’s FastForward program. Shortly afterward, the company was acquired by Intel. The two-year contract scope includes key R&D necessary for a new object storage paradigm for HPC exascale computing, and the developed technology will also address next-generation storage mechanisms required by the Big Data market.
The subcontract incorporates application I/O expertise from the HDF Group, system I/O and I/O aggregation expertise from EMC Corporation, object storage expertise from DDN, and scale testing facilities from Cray, teamed with file system, architecture, and project management skills from Whamcloud. All components developed in the project will be open sourced and benefit the entire Lustre community.
Over at The Register, Timothy Prickett Morgan writes that a GE presentation at the recent GPU Technology Conference discussed the benefits of Remote Direct Memory Access (RDMA) for InfiniBand and its companion GPUDirect method of linking GPU memories to each other across InfiniBand networks.
On plain old CPUs, RDMA allows CPUs running in one node to reach out through an InfiniBand network and directly read data from another node’s main memory, or push data to that node’s memory without having to go through the operating system kernel and the CPU memory controller. If you prefer 10 Gigabit Ethernet links instead, there is an RDMA over Converged Ethernet, or RoCE, wrapper that lets RDMA run on top of Ethernet – as the name suggests. With GPUDirect, which is something that InfiniBand server adapter and switch maker Mellanox Technologies has been crafting with Nvidia for many years, the idea is much the same. Rather than having a GPU go back to the CPU and out over the network to get data that has been chewed on by another GPU, just let the GPUs talk directly to each other over InfiniBand (or Ethernet with RoCE) and get the CPU out of the loop.
Read the Full Story.
This week Mellanox announced that its end-to-end FDR InfiniBand technology is powering the Stampede supercomputer at the TACC. As the most powerful supercomputing system in the NSF XSEDE program, the 10 Petaflop Stampede system integrates thousands of Dell servers and Intel Xeon Phi coprocessors with Mellanox FDR 56Gb/s InfiniBand SwitchX based switches and ConnectX-3 adapter cards.
The InfiniBand network was easy to deploy and delivers incredible application performance on a consistent basis,” said Tommy Minyard , director of Advanced Computing Systems, TACC. “Utilizing Mellanox FDR 56Gb/s InfiniBand provides us with extremely scalable, high performance — a critical element as Stampede is designed to support hundreds of computationally- and data-intensive science applications from around the United States and the world.”
Stampede supports national scientific research into weather forecasting, climate modeling, drug discovery and energy exploration and production. Read the Full Story.
Over at the MPI Blog, Jeff Squyres writes that the distance-from-home analogy is good way to help explain application latency.
So when you send a message to a peer (e.g,. MPI_SEND to another MPI process), consider with whom your communicating: are they next door, in the next subdivision, or in the next city? That gives you an idea of the magnitude of the cost of communicating with them. But let’s add another dimension here: caches and RAM. Data locality is a major factor in performance, and is frequently under-appreciated.
Read the Full Story.
It has been a while since the folks from the EXTOLL project in Germany announced their venture to develop an ultra-low latency interconnect technology for supercomputing. With ISC’13 coming up, I caught up with R. Mondrian Nuessle from Extoll to discuss their plans for the technology and their exhibit at ISC.
insideHPC: What is the EXTOLL interconnect and who is the target user of this technology?
Mondrian Nuessle: The EXTOLL interconnect technology was specifically developed for High Performance Computing. It aims at minimizing the communication overhead between nodes by optimizing the whole communication stack from the physical layer all the way up to the application interfaces like MPI.
insideHPC: How does EXTOLL differ from commodity technologies currently available out there?
Mondrian Nuessle: EXTOLL technology tops commodity technologies by virtually all metrics relevant for HPC including latency, message rate, and bandwidth. Users’ benefit depends on the particular application, but typically a speed up by a factor of 2 will be experienced. This is achieved by an ultra low latency of 600ns, a message rate of more than 100 million messages per second and a bandwidth of 120 Gb/s per link. Each host adapter features 6 bi-directional links of 120Gb/s each, as well as an integrated low-latency message router.
To form an EXTOLL network, EXTOLL adapters are plugged directly together forming for example a 3D torus topology. Thus, the EXTOLL interconnect technology is designed to be a direct network rendering external switches obsolete. This alone will allow customers to realize significant OPEX and CAPEX savings. The EXTOLL interconnect also implements a lot of different technologies to optimally support HPC work loads. Amongst them are low-latency messaging services, high-bandwidth bulk transfers, hardware implemented barriers and multicast, deterministic and adaptive routing, a large amount of reliability features and many more. In summary, the EXTOLL technology is optimized for HPC from the start with no trade-offs! This enables customers to close the gap between commodity clusters and dedicated MPP HPC systems. So in one sentence one can say, by using EXTOLL technology users will get the features, performance and benefits of MPPs for the price tag of commodity clusters.
insideHPC: Does your software stack support MPI? Will your software be open source?
Mondrian Nuessle: Yes, of course. The EXTOLL software stack supports MPI as a “premiere citizen”. From an OS perspective, EXTOLL will focus on Linux first. Linux kernel drivers as well as the low-level API libraries and the MPI integration will be released as open-source. One of the first MPI distributions that will be supported is OpenMPI.
But the EXTOLL software is not uniquely focused on MPI. Support for other communication middlewares and runtimes is under development. An example is GASNET. TCP/IP transport service will be available, too.
insideHPC: Are you still in the prototype stage or is the technology currently available?
Mondrian Nuessle: The EXTOLL ASIC is just in the tape-out stage. First silicon will be available around mid of 2013. Prototypes are based on FPGAs and are fully functional. These prototypes including the beta software stack are out in the field and show performance that is comparable to leading commodity products in many regards, although the raw punch of the FPGA is at least a factor of 4 less than the targeted ASIC technology.
insideHPC: What will you be showcasing at your booth during ISC’13?
Mondrian Nuessle: First of all we will be demonstrating the EXTOLL interconnect with industry standard servers in cooperation with Thomas Krenn AG and NVIDIA. One other thing we will be showing is EXTOLL’s direct GPU-to-GPU communication. One GPU directly communicates with and accesses the memory of a second GPU via the EXTOLL network without involving the host CPUs. This dramatically improves Inter-GPU communication, with savings in energy and time. This new technique is in particular useful with recent Nvidia features like Dynamic Parallelism and GPUDirect RDMA. It addresses the increasing use of accelerators in HPC.
We will also be presenting our 12x active optical cables (AOC). This cable features an electrical connector that can be plugged directly into any electrical connector of EXTOLL cards. Depending on the length of links, users can choose to use EXTOLL AOC or electrical cabling. Moreover, EXTOLL is used within the EU funded FP7 Project DEEP for the BOOSTER interconnect and first BOOSTER node hardware will be presented in cooperation with Eurotech at the Eurotech booth and at the booth of the Jülich Supercomputing Center (JSC).
insideHPC: Why is ISC’13 an important event for you as you commercialize this company?
Mondrian Nuessle: The best way to commercialize a new product or even a company is to be at the right place at the right time. ISC is definitely among the “hot” places for HPC. There is a perfect opportunity to meet trade partners, get their personal feedback, initialize/continue negotiations and become aware of upcoming developments. While SC is the premier venue for the US market, ISC is inevitable to talk to European custmers and partners. And for EXTOLL as a German company, we are especially happy to be able to attend this event in Germany.
In this video from the HPC Advisory Council Switzerland Conference, Dan Waxman from Mellanox provides a hands-on training for InfiniBand entitled: Hands-on Training: Know Your Cluster Bottlenecks and Maximize Performance.
Depending on the application of the user’s system, it may be necessary to modify the default configuration of the network adapters and the system/chipset configuration. This slide deck describes common tuning parameters, settings & procedures that can improve performance of the network adapter. Different Server & NIC vendors may have different recommendations for the values to be set – but the general tuning approach should be similar. For the hands-on demo we will utilize Mellanox ConnectX adapters – thus we will implement the recommended settings issued by Mellanox.
Revolutionary Thinking, Evolutionary Technology – How Intel’s Bet on Fabric Integration will Enable Exaflops
In this special guest feature from The Exascale Report, Mike Bernhardt writes that Intel is placing a big bet on fabric integration on its journey to Exascale.
We already know exascale-class systems will be far too expensive to make them commercially available. And we’ve heard several years of discussion on the staggering power requirements an exaFLOPS system would require. So, is anyone doing anything creative to get past these barriers?
With a clever brand most of us marketing types can really appreciate, Intel’s True Scale Fabric represents an architectural change that can potentially benefit Cloud Computing, Big Data, HPC, and establish a path toward exascale.
The architectural change Intel is going for here is to bring the processor and the controller closer together. There’s more to it of course including some specialized hardware and software as one would expect, but the net effect according to Intel will be a reduction of power consumption and the density of the servers.
The move in this direction did not happen overnight. Intel has been working on this strategy for quite some time. Intel’s portfolio of assets to support this integration of storage and network controllers with Intel’s processors has been significantly enhanced with its acquisition of HPC interconnect technology from Cray, the acquisition of the Ethernet switching company, Fulcrum
Microsystems, and pulling in the InfiniBand assets of QLogic.
Joe Yaworski, Intel’s Fabric Product Marketing Manager, recently presented five key points to emphasize Intel’s wisdom in moving in this direction:
- Datacenter (HPC & Cloud) growth requires new innovations to meet the growing demand and performance requirements
- Fabrics are becoming the next bottleneck to an unrelenting need for data in cloud and HPC workloads
- Fabric integration will be required to address the growing need for bandwidth, scalability, power and system density
- Intel is uniquely positioned with its acquisitions of – Cray interconnect group, QLogic InfiniBand program and products and Fulcrum assets to meet the need with fabrics technology innovation and CPU platform integration in the future
- One of the solutions for the future may be moving the fabric controller closer to the CPU. This will provide the potential for meeting high bandwidth and performance goals, while at the same time delivering the highest possible energy efficiency
For The Exascale Report, we do not consider this article in the category of a product announcement as any products based on such an integrated fabric are still in the future. We do see this as an important path to be explored, and we will continue to monitor and report on this topic.
In this video from the HPC Advisory Council Switzerland Conferenc, Zhiqi Tao from Intel (formerly of Whamcloud) presents: A Generic Methodology for Optimizing an HPC Storage System.
Designing a large scale, high performance storage system presents significant challenges. This paper describes a step-by-step approach to designing a storage system and presents a design methodology based on an iterative approach that applies at both the component level and the overall system level. The paper includes a detailed case study in which a Lustre storage system is designed using the approach and methodology presented.
In related news, the Lustre User Group conference will take place in San Diego April 16-18.
In this video from the HPC Advisory Council Switzerland Conference, D.K. Panda from Ohio State University presents: Accelerating Big Data with Hadoop (HDFS, MapReduce and HBase) and Memcached. Download the slides (PDF).
In this video from the HPC Advisory Council Switzerland Conference, Colin Bridger presents: Mellanox: The Foundation for Scalable Computing.