Since its beginnings in 1999 as a project at Carnegie Mellon University, Lustre, the high performance parallel file system, has come a long, long way. Designed and always focusing on performance and scalability, it is now part of nearly every High Performance Computing (HPC) cluster on the Top500.org’s list of fastest computers in the world—present in 70 percent of the top 100 and nine out of the top ten. That’s an achievement for any developer—or community of developers, in the case of Lustre—to be proud of.
Get the insideHPC Special Report to Lustre – Click here to download.
Along the journey to be part of the top ten systems, Lustre has evolved, a necessity driven by both performance needs and enterprise interests and demands. While government and research institutions were the original users of Lustre, it has made serious inroads into enterprise markets, such as climate and meteorology, financial services, engineering and design, media, oil and gas, and pharmaceuticals. And now, with the democratization of HPC, enterprise interest continues to increase to use Lustre as a persistent file system. Thus, while still focusing on performance and scalability, the Lustre developer community is responding with features and enhancements that meet enterprise IT expectations. The latest version of Lustre has added many of these features.
CTOs and IT departments have been talking about convergence of scientific computing, office automation and productivity, and High Performance Data Analytics (often called Big Data) for some time now. It takes a high-performance file system to simultaneously meet the needs of all these. Organizations are looking at and deploying Lustre as that file system, enabling convergence across corporate, scientific computing, and analytics.
The cloud is becoming a cornerstone to technical and enterprise computing. Adding parallel file system functionality to cloud computing delivers the fullness of HPC as a Service. Customers can take advantage of this functionality via the Amazon Web Services marketplace. Intel’s Cloud Edition rounds out a trilogy of Lustre distributions from Intel, including the Intel® Enterprise Edition for Lustre software and Intel® Foundation Edition for Lustre software. As the leading contributor to Lustre development, Intel remains committed to the Lustre Community.
Lustre is a complex set of codes that takes time to enhance and expand with new capabilities. Thus, the Lustre community of developers remains committed to keep it not only as the fastest and most scalable, open source parallel file system available, but to also add the rich enhancements the user community desires.
In this series of articles, InsideHPC presents a ‘cluster’ of topics on Lustre, covering a wide range of topics of interest to user and developer community alike. Or you can download these articles in this handy PDF.
High-Performance Lustre* Storage Solution Helps Enable the Intel® Scalable System Framework
Lustre* Accelerates the Convergence of Big Data and HPC in Financial Services
Lustre* at the Core of HPC and Big Data Convergence
Commercial Grade Lustre File Systems
Lustre: This is Not Your Grandmother’s (or Grandfather’s) Parallel File System
Setting a Path for the Next-Generation of High-Performance Computing Architecture
Deploying Hadoop on Lustre Storage: Lessons Learned and Best Practices
Get the insideHPC Special Report to Lustre – Click here to download
