Today DDN Japan announced that the University of Tokyo and the Joint Center for Advanced High Performance Computing (JCAHPC) has selected DDN’s burst buffer solution “IME14K” for their new Reedbush supercomputer. ” “Many problems in science and research today are located at the intersections of HPC and Big Data, and storage and I/O are increasingly important components of any large compute infrastructure.”
The DDN User Group meeting is returning to Salt Lake City for SC16. The meeting takes place Monday, Nov. 14 from 2pm-6pm at the Radisson Hotel near the convention center.
“EXAScaler 3.0 raises the bar for Lustre performance and management,” said Laura Shepard, senior director of products and vertical markets, DDN. “As the world’s most experienced Lustre provider, DDN leverages input from a broad installed base and the Lustre community to deliver the most advanced Lustre solutions to our customers around the globe.”
Gary Paek from Intel presented this talk at the HPC User Forum in Austin. “Traditional high performance computing is hitting a performance wall. With data volumes exploding and workloads becoming increasingly complex, the need for a breakthrough in HPC performance is clear. Intel Scalable System Framework provides that breakthrough. Designed to work for small clusters to the world’s largest supercomputers, Intel SSF provides scalability and balance for both compute- and data intensive applications, as well as machine learning and visualization. The design moves everything closer to the processor to improve bandwidth, reduce latency and allow you to spend more time processing and less time waiting.”
Coming to SC16? Intel is hosting a one-day Lustre training session on Friday November 18th, 2016 at the Sheraton in Salt Lake City. “You can expect the top Lustre experts to spend the day with on the training topics that you care about.”
The following sections of this paper will describe the Lustre File System, the Dell Storage for HPC with Intel EE for Lustre solution followed by performance analysis and conclusions. Appendix A: Benchmark Command Reference
Lustre is a complex set of codes that takes time
to enhance and expand with new capabilities. Thus, the Lustre community of developers remains committed to keep it not only as the fastest and most scalable, open source parallel file system available, but to also add the rich enhancements the user community desires.
Finding oil and gas has always been a tricky proposition, given that reserves are primarily hidden underground, and often as not, under the ocean as well. The costs involved in acquiring rights to a site, drilling the wells, and operating them are considerable and has driven the industry to adopt advanced technologies for locating the most promising sites. As a consequence, oil and gas exploration today is essentially an exercise in scientific visualization and modeling, employing some of most advanced computational technologies available.
This paper clearly demonstrates that once optimized for large I/O throughput the Dell MD3460 / Intel Enterprise Edition Lustre (IEEL) solution provides storage density and performance characteristics that are very well aligned to the requirements of the mid-to-high end research storage market. After the throughput tuning had been applied the I/O performance of the Dell storage brick doubled, producing single brick IOR client performance maxima of 4.5GB/s R/W. Single rack configurations can thus be implemented that provide 2.1 PB of usable storage and 36 GB/s R/W performance. A capacity optimized configuration is also illustrated providing a solution with a cost reduction of ~35% relative to the performance optimized solution. These bulk performance and density metrics place the Dell / IEEL solution at the high end of the solution space but within the commodity IT supply chain model. This will provide the price performance step change that the scientific, technical and medical research computing communities need to help close the demand vs. budget gap that has emerged due to huge growth in demand seen within the research community for both storage capacity and performance. This marks a turning point in commoditization of research storage solutions echoing the commodity revolution that was seen in research computing market with the advent of HPC clusters. Many large scale HPC customers are finding it difficult to architect HPC and data analysis system with the required capacity, performance and cost parameters. Commodity high end parallel files system as described in this paper dramatically improve this situation.
Today, Lustre File System is based entirely on Linux and is using kernel- based server modules to deliver the expected performance. Lustre can support many types of clients and runs on almost any modern hardware. Scalability is one of the most important features of Lustre File System and can be used to create a single namespace of what appears to be almost limitless capacity.