The MSST 2016 conference kicks off next week with a focus on massive storage systems and technology. One of the issues of focus at the meeting will be what might replace the aging POSIX namespace for long term storage of Big Data? The MarFS open source project from Los Alamos National Laboratory may just be the ticket. To learn more, we caught up with David Bonnie from Los Alamos National Laboratory.
insideHPC: What is the problem that MarFS is intended to solve?
David Bonnie: As we continue to scale system memory footprint, it becomes more and more challenging to scale the long-term storage systems with it. Scaling tape access for bandwidth becomes increasingly challenging and expensive when single files are in the many terabytes to petabyte range. Object-based scale out systems can handle the bandwidth requirements we have, but are also not ideal to store very large files as objects. MarFS sidesteps this while still leveraging the large pool of object storage systems already in existence by striping large files across many objects.
insideHPC: MarFS is described as a “Near-POSIX File System using Scale Out Commercial/Cloud for Data and Many POSIX File Systems for Metadata (SPOOFS).” That sounds like you wanted to get away from POSIX, but you couldn’t quite get there. Is that accurate?
David Bonnie: Well, not quite. We wanted to get away from the complexity of POSIX for data, yet retain the parts of POSIX that people are used to (metadata manipulation). By divorcing ourselves from the complications of ensuring a completely POSIX data flow, we can massively simplify the data movement and storage mechanisms. MarFS lets us retain the parts of POSIX that users appreciate for data management (chown, chmod, rename, mv, etc) without inheriting the complexity of managing POSIX semantics for data manipulation. By treating the data as essentially immutable, we can leverage the very simple PUT/GET/DELETE semantics of “cloudy” data storage systems to scale out storage with ease.
insideHPC: Why build something like MarFS rather than adapt existing file systems?
David Bonnie: I wouldn’t call MarFS a true file system. It’s an object storage system that masquerades as a file system except when writing data. By restricting how you can write, we gain a lot of flexibility in storage, but we retain the ability to read and manipulate the layout of the data. Existing solutions that marry file and object semantics tend to go through a lot of hoops to make objects feel like files when writing; this breaks down at the extremes of scale that we work with.
insideHPC: Is MarFS open source? Do you think vendors will be able to leverage this technology for commercial offerings?
David Bonnie: MarFS and the associated tools to work with it are hosted on GitHub:
https://github.com/mar-file-system/marfs
https://github.com/pftool/pftool
We encourage vendors to utilize any part of this design that fulfills a commercial need!
insideHPC: What will be the key take-aways from your talk at the MSST Conference?
David Bonnie: Essentially, objects are nice, but people know and love hierarchical file systems. Further, legacy applications can run on top of MarFS without modification if they require read-only access, and very little modification if they require write access. Each part of the MarFS system has been designed to be scalable to the extreme: the data storage, the metadata engine, the management tools, etc. From single large files in the petabyte range to billions of files in a directory, MarFS should not only handle it, but it is designed to do so from the ground up.
David Bonnie will give a talk on MarFS at the MSST conference, which takes place May 2-6 in Santa Clara. Register now.
