This week the Joint Institute for Computational Sciences at ORNL announced a partnership with WARP Mechanics to study next generation object storage technologies for use in HPC environments. In this video from LUG 2014, Josh Judd from Warp Mechanics presents: Practical Applications of Lustre/ZFS Hybrid Systems.”
“LLNL’s largest supercomputer is paired with a 55-petabyte file system, known as Grove, that stores vast quantities of simulation data. Grove must transfer information to and from Sequoia at a minimum of 500 gigabytes per second. To support Grove’s high storage capacity and bandwidth requirements, LC software developers have engaged in a multi-year project to replace much of Grove’s Lustre foundation with ZFS. “
Josh Judd from Warp Mechanics describes how the company delivers Lustre Over ZFS on Linux. “No single technology solves all problems faced in today’s complex world. WARP Mechanics’ philosophy is to customize the many and varied systems into the exact set of solutions required to address the problems. WARP Mechanics leverages tried–and-true technologies from the most advanced systems and removes the complexity, delivering customized turnkey solutions.”
With the release of Lustre 2.4 support for ZFS Lustre servers has arrived. Historically, Lustre has only supported ext4/ldiskfs servers and while those servers have performed well they do suffer from a number of well known limitations. Extending Lustre to use a next generation filesystem like ZFS allows it to achieve greater levels of scalability and introduces new functionality.