In this video from LUG 2015 in Denver, Rick Wagner from SDSC presents: SDSC’s Data Oasis Gen II: ZFS, 40GbE, and Replication.
The second generation of SDSC’s Data Oasis Lustre storage is coming online to support Comet, a new XSEDE cluster targeted at the long tail of science. The servers have been designed with Lustre on ZFS in mind, and also update the network to use bonded 40GbE interfaces. The raw storage totals 7.7 PB and are again based on commodity hardware provided by Aeon Computing, maintaining our focus on cost. Most of work in preparing Comet’s storage has focused on performance, and using this design we have achieved 7.5 GB/s per server sustain read bandwidth over a bridged Ethernet to IB fabric. I will describe the work required to get there, include the selective application of patches to the Lustre code, ZFS tuning, and the interplay of NUMA architecture with Lustre SMP node affinity and CPU partitioning. The second major feature of Data Oasis is the reuse of the first generation servers as Durable Storage, i.e., a backup file system for the second generation hardware. This is being implemented using Robinhood, along with some custom software to facilitate the initial transition from one generation of hardware to the next. All of the aforementioned is in progress, and will be updated according to our experiences as the deployment of Comet proceeds.”