Fifteen years ago, the HPC community began developing a super fast and highly scalable parallel file system in response to the limitations of NFS. Called Lustre, it is the dominant file system technology now in use at half of the top 100 supercomputer installations worldwide (top500.org).
This third article is an editorial series that explores high performance storage and the benefits of Lustre solution for HPC.
No longer confined to Big Science, Lustre has become increasingly more attractive for use commercially where sustained high I/O bandwidth is needed to process large streams of data. Lustre has proven it can support more than a terabyte per second I/O throughput, tens of thousands of client systems, and petabytes of storage.
Unlike NFS, where all I/O goes through a single node, Lustre stripes data across multiple data servers and saves meta-data (file names, owner, permissions) over separate servers. This allows the system to optimize for different workloads and demands, and permit access to files from multiple clients independently. By configuring redundant servers and enabling failover storage, Lustre offers a file system that is highly reliable with no single point of failure.
(Click on image to enlarge view)
Lustre itself is open source. The source code is freely available, and is maintained and developed within the HPC community. Two consortia of leading industry, research, and academic organizations, Open Scalable File Systems (OpenSFS.org) and the European Open File System (EOFS.eu), are charged with the ongoing stewardship and coordinated development of Lustre, independent of any single platform or vendor hardware.
Next week’s article will look at Commercial Grade Lustre File Systems. If you prefer you can download the complete guide to Lustre Solutions in a PDF from the insideHPC White Paper Library courtesy of Intel.
