Sponsored Post
This is the first in a series of articles on “6 Things You Should Know About Lustre”. Other topics will cover Lustre in Big Data, the Cloud, enterprise HPC, and next-generation storage.
Managing enterprise data is not getting any easier. “A lot of enterprises today are leaving data lying on the floor,” says Brent Gorda, General Manager of Intel’s High Performance Data Division. “They are simply overwhelmed with so much of it, they don’t know what to do with it.” They are looking for alternatives to their existing storage solutions.
Many organizations have turned to NFS, but as their data sets grow in size—and NFS servers grow in numbers—NFS becomes too complex to manage. Additionally, it slows down productivity. Users have to first know where their data is before accessing it. As enterprises reach the limits of NFS and look for alternatives, Lustre*—the parallel file system—offers a compelling solution.
Lustre* is not just for the national labs any longer. It was born out of serving up data extremely fast to the world’s most powerful HPC clusters using parallel I/O to improve performance and scalability. But, it was not designed around enterprise needs.
With the help of the Lustre developer community (of which Intel is a key contributor), OpenSFS* (the organization that manages Lustre releases), and Intel’s own enabling enhancements in its Intel® Enterprise Edition for Lustre software (Intel® EE for Lustre), Lustre is capable of supporting enterprise endeavors and meeting enterprise IT needs. Here are five reasons why Lustre is enterprise-ready.
- Lustre is easily managed
One of the biggest challenges with gaining traction in the enterprise has been the misperception that Lustre is difficult to work with. Thus, Intel added Intel® Manager for Lustre (IML) to Intel EE for Lustre. IML is a web-based interface that allows IT to quickly create a Lustre file system built on industry-standard best practices. This gives enterprise IT the ability to stand up–in minutes–a high-performance file system on a trusted hardware base that meets their requirements. Besides simplifying installation, IML provides a unified, consistent view of the storage systems’ configuration, monitoring, and overall management.
- Lustre supports availability/reliability and capacity
Lustre is a parallel file system designed for performance and efficiency. But enterprise needs go beyond those. They require a large measure of reliability, availability, and extended capacity. Thus, part of the enterprise model for Intel EE for Lustre includes high availability hardware building blocks and Hierarchical Storage Management (HSM) support built into Intel EE for Lustre. With these services, companies can balance a storage infrastructure that provides performance, high availability components, and capacity with HSM to offload aged data onto slower, low cost archive storage, but keep it available if the data is needed.
- Lustre is becoming more adaptable
Companies may have data on other file systems across the organization, such as Windows* PCs and servers and NFS servers. Since this data is valuable, the Lustre community is integrating these protocols—leveraging the POSIX compliance of Lustre to create scalable, parallel storage gateways that export data from Lustre via NFS, and via the SMB protocol using Samba for Windows sharing. These enhancements will enable organizations to leverage their wide variety of data and enable them to more readily turn it into actionable knowledge.
- Lustre is adding security
Enterprises are more security conscious than ever; they expect security capabilities to be an integral part of their overall infrastructure. The Lustre community is diligently working on tools based on SELinux for fine-grained access control for applications. Additionally, the community is incorporating Kerberos* to authenticate nodes and for encryption of data in transit across the network. These, and other work being done will help Lustre meet enterprise security expectations.
- Lustre is flexible
Lustre was originally designed around massive data sets comprising terabytes of data stored in relatively few files. Meta data servers did not need to be as responsive as the storage servers. But with enterprise applications, some data sets comprise an incredibly large number of files that contain a few gigabytes or less of data. Because accesses to all these files start with the meta data, the meta data servers can become a bottleneck. The latest version of Lustre included a significant enhancement to the performance of the meta data system, from about 60,000 file creates per second to over one million (over 10X improvement), making Lustre much more performant for smaller file sizes used in the enterprise.
The Lustre developer community has made significant enhancements to Lustre 2.5.x. to make it enterprise-class. Companies can follow continuing developments at http://opensfs.org.
Find out more about Intel solutions for Lustre.
