Intel’s solution-building HPC framework relies on best-of-breed components, including Intel® Solutions for Lustre* Software
This is the sixth in a series of articles on “6 Things You Should Know About Lustre”. See links below to find other topics covering Lustre in Enterprise, the Cloud, Financial Services, and next-generation storage.
With terabytes to petabytes of data to move—in massively large single-file data sets or millions of small files—users of today’s High Performance Computing (HPC) systems are more concerned about the efficiency and speed of data movement than ever before. Moore’s Law has allowed processor designers and manufacturers over the years to take the bottlenecks out of compute, moving the choke points to other areas of the system—storage, memory, and communications (fabric). HPC integrators and operators have battled for years with building balance into these massive, complex systems to optimize the overall performance.
The recently revealed Intel® Scalable System Framework (Intel® SSF) is an approach to enable builders and operators to more easily architect and integrate flexible, performant, and balanced HPC solutions for their customers. Intel has assembled the best of breed components, from Intel-enhanced, open source community-based software to power-efficient and high-performance hardware technologies of their own, into a framework on which next-generation HPC is being constructed by leading OEMs and integrators in the industry.
Lustre, proven for its throughput in research labs for years, is now among the top few, leading parallel file systems capable of meeting the needs of the HPC community—both in research and enterprise. Some of the most recent HPC installations with Lustre are seeing hundreds of gigabytes[1] to terabytes[2] per second of throughput. The company has incorporated Intel® Solutions for Lustre Software as part of the Intel SSF because it provides the performance to move data and minimize storage bottlenecks. Lustre is also open source based, and already enjoys a wide foundation of deployments in research around the world, while gaining significant traction in enterprise HPC. Intel’s version of Lustre delivers a high-performance storage solution in the Intel SSF that next-generation HPC needs to move toward the era of Exascale.
Performance alone, however, is not enough to become part of a total solution HPC framework in today’s IT environments. Reliability, manageability, and flexibility are top of mind for enterprise HPC, where Total Cost of Ownership is right up there in importance with performance.
Today’s Lustre has undergone many years of evolution by contributions from the developer community and additional tools added by Intel engineers for the Intel versions to make it best of breed for not only traditional HPC workloads, but for visualization, machine learning, and Big Data Analytics. The mature features these enhancements provide, such as High Availability (HA), Hierarchical Storage Management (HSM), reliability, and adaptability for Hadoop, are now part of core Lustre capabilities. Plus, Intel has added the Intel® Manager for Lustre in their own enterprise version, which simplifies and accelerates Lustre deployment and management. Innovative security and replication features are among the forefront of continuing Lustre enhancements.
Key to the goals for Intel SSF is to deliver balance and move data closer to the compute cores. CPU manufacturers have proven over the years that getting data closer to the cores improves overall efficiency and performance. That’s why we’re seeing more and more integration into the processor silicon of functions that used to be outboard, such as memory and graphics. Intel is taking it a step further and integrating part of the fabric into the processor to accelerate data movement to remove communications bottlenecks. Intel SSF is also working on innovations in memory and SSDs, integrated at the system level, all to help move more data to the compute complex faster.
But data begins in the file system, which is why Lustre and Intel Solutions for Lustre software, are critical to enabling Intel’s new framework for next-generation HPC solutions.
Read the rest of the “6 Things You Should Know About Lustre” series:
#2 Lustre at the Core of HPC and Big Data Convergence
#3 Lustre Accelerates the Convergence of Big Data and HPC in Financial Services
#4 Setting a Path for the Next-Generation of High-Performance Computing Architecture
#5 Lustre and Persistent Storage
[1] San Diego Supercomputing Center’s Data Oasis based on Intel® Enterprise Edition for Lustre* software reports up to 300 GB/s (http://www.sdsc.edu/services/hpc/parallel_file_systems.html).
[2] Spider II at Oak Ridge Leadership Computing Facility (OLCF) deliver 1 TB/s (http://users.nccs.gov/~yk7/papers/SC14-SOP-Spider.pdf)
