This is the second in a four-part series about Intel® Scalable System Framework. Other contributions cover Intel® Omni-Path Architecture fabric, HPC software and the OpenHPC community, and Intel® Xeon Phi™ processor.
Intel has been working on a new HPC design philosophy for HPC systems called Intel® Scalable System Framework (Intel® SSF), an approach designed to enable sustained, balanced performance in HPC as the community pushes towards the Exascale computing era. Central to Intel SSF performance is the Lustre* scalable, parallel file system (PFS). Intel® Enterprise Edition for Lustre software (Intel® EE for Lustre software) is the Intel distribution of the well-known PFS, which is used by the majority of the fastest supercomputers around the world.
Lustre is key to Intel SSF as a major part of the architecture’s memory-storage hierarchy. The file system, once considered only for national laboratories and academia, and used traditionally as an extremely fast scratch file system, is now being deployed as persistent storage in enterprise HPC installations around the world. The most recent release of Lustre adds important reliability and security features, which enterprises need for their mission-critical workloads that depend on the system’s (and Lustre’s) robustness and performance. For example, the Bank of Italy uses Intel EE for Lustre software in a converged architecture that bridges both their financial scientific HPC cluster and Windows* users across the organization.
“We have a significant investment in Lustre,” says Brent Gorda, General Manager of Intel’s High Performance Data Division, the group that manages Lustre development. “Company developers continue to add enhancements to the open source code base which was just released as Lustre 2.8 (open source version), as well as continuously improving Intel EE for Lustre software.” In April we released Lustre 3.0, considerably improving security and reliability features from the previous version.”
Lustre version 2.9 (open source version) is scheduled for general availability later this year, which includes further enhancements.
Lustre developers focus on maintaining and improving its performance as the fastest file system on the planet. But, as seen at the recent Lustre User Group (LUG) 2016 gathering of developers and users, new and ongoing Lustre projects are enhancing features, such as extending Lustre support for the ZFS file system, adding more security hardening, improving performance with Big Data analytics, and LNET support. New security features include client-side SELinux* support for policy enforcement, Kerberos* encryption, and Kerberos authentication of nodes and users on the Meta Data Servers (MDS). Further hardening continues to be worked on.
While Lustre is often considered complex software, the Intel® Manager for Lustre, part of Intel EE for Lustre software, continues to be enhanced with features that ease Lustre deployment and management. And Lustre training is now available through learn.intel.com, offering courses that can help new users gain an understanding of what Lustre is and how it works.
In addition to code contributions, Intel recently opened five new Intel® Parallel Computing Centers (Intel® PCC) that will specialize in improving the performance and ease of use of Intel® Solutions for Lustre software.
These centers include:
• Johannes Gutenberg University, focusing on Lustre QoS (Quality of Service)
• Lawrence Berkeley National Laboratory, focusing on optimized Big Data Analytics
• University of Hamburg, focusing on enhanced adaptive compression in Lustre
• University of California Santa Cruz, focusing on automatic tuning and contention management for Lustre
• GSI Helmholtz Centre for Heavy Ion Research, focusing on a unifying Lustre benchmark suite
Many current projects on Lustre are using the resources of these centers, including work on Apache* Spark* running with Lustre at Lawrence Berkeley National Laboratory, and performance enhancements through contention management at University of California at Santa Cruz.
For more information on LUG 2016, see http://opensfs.org/events/lug-2016/. For more information about Intel Enterprise Edition for Lustre and Intel Solutions for Lustre software, visit here.
