The following sections of this paper will describe the Lustre File System, the Dell Storage for HPC with Intel EE for Lustre solution followed by performance analysis and conclusions. Appendix A: Benchmark Command Reference
Lustre is a complex set of codes that takes time
to enhance and expand with new capabilities. Thus, the Lustre community of developers remains committed to keep it not only as the fastest and most scalable, open source parallel file system available, but to also add the rich enhancements the user community desires.
Finding oil and gas has always been a tricky proposition, given that reserves are primarily hidden underground, and often as not, under the ocean as well. The costs involved in acquiring rights to a site, drilling the wells, and operating them are considerable and has driven the industry to adopt advanced technologies for locating the most promising sites. As a consequence, oil and gas exploration today is essentially an exercise in scientific visualization and modeling, employing some of most advanced computational technologies available.
Today, Lustre File System is based entirely on Linux and is using kernel- based server modules to deliver the expected performance. Lustre can support many types of clients and runs on almost any modern hardware. Scalability is one of the most important features of Lustre File System and can be used to create a single namespace of what appears to be almost limitless capacity.
Even the largest HPC clusters can experience degradation due to poor I/O performance. This occurs as massive amounts of data and increasingly large individual files combine limited disk drive hardware capacity to cause significant bottlenecks. Lustre is an open source parallel file system that improves the overall scalability and performance of HPC clusters. It provides cluster client nodes with shared access to file system data in parallel, greatly increasing throughout and performance. Lustre is the most widely used HPC storage system in the world-with parallel storage capabilities utilized by over 50% of HPC deployments-and can scale to tens of thousands of clients.
In this Guide, we take a look at what an HPC solution like Lustre can deliver for a broad community of business and commercial organizations struggling with the challenge of big data and demanding storage growth.