The Lustre file system is a open source, parallel file system that supports the requirements of leadership class HPC and Enterprise environments worldwide. Lustre provides a POSIX compliant interface and scales to thousands of clients, petabytes of storage, and has demonstrated over a terabyte per second of sustained I/O bandwidth. Many of the largest and most powerful supercomputers on Earth today are powered by the Lustre file system, including over 60% of the TOP100 sites.
The recent release of a commercial version of the Lustre* parallel file system was big news for business data centers facing ever expanding data analysis and storage demands. Now, Lustre, the predominant high-performing file system installed in most of the supercomputer installations around the world, could be deployed to business customers in a hardened, tested, easy to manage and fully supported distribution.
“Sharing a common code base with other Open ZFS platforms has given ZFS on Linux the opportunity to rapidly implement features available on other Open ZFS platforms. At present, Illumos is the reference platform in the Open ZFS community and despite its ZFS driver having hundreds of features, ZoL is only behind on about 18 of them.”
“Clearly, given the amount of code and years of investment in applications, we are not going to be able to flip the switch and presto chango have everyone run on REST systems. It is likely going to take the rest of the decade and then some to be able to switch over, but in the mean time we have a scaling issue.”
“At Cray, we are a big user and investor in Lustre. Because Lustre is such a great fit for HPC, we deploy it with almost all of our systems. We even sell and deliver Lustre storage independent of Cray compute systems. But Lustre is not (yet) the perfect solution for distributed and parallel-I/O, so Cray invests a lot of time and resources into improving, testing, and honing it. We collaborate with the open-source Lustre community on those enhancements and development. In fact, Cray is a leader in the Lustre community through our involvement in OpenSFS.”
As an example of what you can do with key-value storage and how simple it can be, Seagate has created a new storage drive called Kinetic that you address using REST-like commands such as get, put, and delete. A simple open-source library allows you to then develop IO libraries so that applications can perform IO to/from the drives. Some object storage solutions such as Swift have already been ported to use the Kinetic drives. Ceph is also developing a version that can use Kinetic drives. Other object based storage systems such as Lustre and Gluster could theoretically use this technology as well.”
“HPC cluster performance is often degraded because more and more data and larger files overwhelm limited hard drive capacity. But if you use Amazon Web Services (AWS), such bottlenecks may be a thing of the past. Intel, in collaboration with AWS, offers a Cloud Edition for Lustre Software that allows customers to use the power of the worlds’ most popular HPC storage system to increase scalability and performance.”