David Bonnie from LANL presented this talk at the MSST 2017 Mass Storage Conference. MarFS is a Near-POSIX File System using cloud storage for data and many POSIX file systems for metadata. “With MarFS in production at LANL since fall 2016, we have gained new insights, learned lessons, and expanded our future plans. We’ll discuss the various hurdles required to deploy such an ambitious system with minimal manpower. Further, we’ll delve into the challenges, triumphs, and defeats on the road to a new tier of inexpensive scalable storage.”
MarFS – A Scalable Near-POSIX File System over Cloud Objects
October 28, 2016 by
Gary Grider from LANL presented this talk at the Storage Developer Conference. “MarFS is a Near-POSIX File System using cloud storage for data and many POSIX file systems for metadata. Extreme HPC environments require that MarFS scale a POSIX namespace metadata to trillions of files and billions of files in a single directory while storing the data in efficient massively parallel ways in industry standard erasure protected cloud style object stores.”
MarsFS – A Near-POSIX Namespace Leveraging Scalable Object Storage
May 10, 2016 by
David Bonnie from LANL presented this talk at the 2016 MSST Conference. “As we continue to scale system memory footprint, it becomes more and more challenging to scale the long-term storage systems with it. Scaling tape access for bandwidth becomes increasingly challenging and expensive when single files are in the many terabytes to petabyte range. Object-based scale out systems can handle the bandwidth requirements we have, but are also not ideal to store very large files as objects. MarFS sidesteps this while still leveraging the large pool of object storage systems already in existence by striping large files across many objects.”
Interview: Moving Beyond POSIX with the new MarFS Object Storage Project
April 26, 2016 by
“We wanted to get away from the complexity of POSIX for data, yet retain the parts of POSIX that people are used to (metadata manipulation). By divorcing ourselves from the complications of ensuring a completely POSIX data flow, we can massively simplify the data movement and storage mechanisms. MarFS lets us retain the parts of POSIX that users appreciate for data management (chown, chmod, rename, mv, etc) without inheriting the complexity of managing POSIX semantics for data manipulation. By treating the data as essentially immutable, we can leverage the very simple PUT/GET/DELETE semantics of “cloudy” data storage systems to scale out storage with ease.”
Gary Grider Presents: HPC Storage and IO Trends and Workflows
April 6, 2016 by
“Trends in computer memory/storage technology are in flux perhaps more so now than in the last two decades. Economic analysis of HPC storage hierarchies has led to new tiers of storage being added to the next fleet of supercomputers including Burst Buffers or In-System Solid State Storage and Campaign Storage. This talk will cover the background that brought us these new storage tiers and postulate what the economic crystal ball looks like for the coming decade. Further it will suggest methods of leveraging HPC workflow studies to inform the continued evolution of the HPC storage hierarchy.”
