As data analytics becomes more mission critical, hardware and software need to evolve to handle both historical data (batch) and real time streaming data. This combined ability to manage different types of data is critical for a wide range of organizations.
“With the current Lustre Performance Monitoring Tool (LMT) no longer in active development, and the current version incompatible with DNE based Lustre 2.5 deployments, there is a critical need for a new set of tools delivering the same basic Lustre performance metrics with the added ability to work compatibly with contemporary releases of Lustre.”
“The Cray-Seagate partnership is helping expand the boundaries of what’s possible in large-scale, data-intensive computing, far beyond what we could have imagined just 10 years ago. This continued innovation using the Lustre open file system is helping assist data-intense applications critical to advancements in important industries around the world.”
“Large scale HPC IO is usually done either with a file per process or to a single shared file. Single shared file IO does not scale well in Lustre compared to file per process. This presentation from Cray’s Patrick Farrell will give details, examine the reasons for this, and explore existing and potential solutions. Group locks and a new feature, lock ahead, will be discussed in the context of strided IO.”
“The second generation of SDSC’s Data Oasis Lustre storage is coming online to support Comet, a new XSEDE cluster targeted at the long tail of science. The servers have been designed with Lustre on ZFS in mind, and also update the network to use bonded 40GbE interfaces. The raw storage totals 7.7 PB and are again based on commodity hardware provided by Aeon Computing, maintaining our focus on cost.”
In this video from LUG 2015 in Denver, Jason Goodman from Cray presents: Cray’s Storage History and Outlook – Lustre+. “As a leader in open systems and parallel file systems, Cray builds on open source Lustre to unlock any industry-standard x86 Linux compute cluster using InfiniBand or 10/40 GbE utilizing proven Cray storage architectures.”