This week Whamcloud announced that the company is extending its working relationship with EMC Corporation to provide deeper integration between the Parallel Log-structured File System (PLFS) and Lustre. To get the full story, I caught up with Percy Tzelnic from EMC’s CTO Office and the SVP Fast Data Group.
insideHPC: What is the nature of the collaboration with EMC that you announced this week?
Percy Tzelnic: The collaboration between EMC and Whamcloud is designed to help Lustre improve metadata handling and small file I/O performance, leveraging PLFS, which is open source software whose development started at Los Alamos National Lab and continues as a joint project with EMC. In the extreme HPC domain (large supercomputing clusters), PLFS also reduces checkpoint time by a large order of magnitude.
insideHPC: EMC is a big player in the enterprise storage space. Does this collaboration indicate a growing interest in Lustre on the commercial computing side of the fence?
Percy Tzelnic: EMC has been very active in the HPC domain. A large portion of commercial HPC has been very successfully addressed with EMC Isilon, as well as VNX as block storage for file systems such as StorNext and even Lustre. The current Fast Data initiative is focused on both extreme, as well as commercial HPC. EMC is definitely interested in commercial HPC, and Lustre has emerged in some of the verticals as another valuable tool, enabling VNX block to be deployed in very high bandwidth and low latency use cases.
insideHPC: The news release mentions that this extension of an existing collaboration with EMC. What have been the results so far?
Percy Tzelnic: EMC has been working with Whamcloud for over a year. This collaboration helped with the development of a Lustre appliance that includes EMC VNX technology, in a partnership with a third party. This appliance is being currently test marketed and elicited a high level of interest.
insideHPC: How will this collaboration help with the goal of getting the Lustre client included in the upstream Linux kernel?
Percy Tzelnic: EMC is interested in making Lustre readily consumable by our commercial HPC customers. EMC is collaborating with the Linux community and Whamcloud to enable the availability of Lustre client software as part of the standard Linux distribution.
insideHPC: With tighter integration with PLFS, will Lustre enable much faster checkpoint/restarts for large file systems?
Percy Tzelnic: Yes, PLFS improves checkpoint performance by several orders of magnitude in a Lustre filesystem environment. This is a specific use-case, mostly present in extreme HPC — but PLFS also addresses certain commercial HPC applications needs when deployed with Lustre.