MailChimp Developer

Sign up for our newsletter and get the latest HPC news and analysis.
Send me information from insideHPC:


Video: Growth of Lustre Adoption and Intel’s Continued Commitment

Brent Gorda, Intel

“We are now working with over 100 channel partners globally. You can get access to Intel Lustre from almost everyone who sells storage or compute worldwide. We’re expanding this to include software partners, cloud partners. We want to create the best product possible out of this open source technology, and make it available economically to the channel partner, and enable you to go after these hugely expanding markets of cloud and big data, while not giving up on HPC.”

EOFS Elects HP’s Frank Baetke as Chairman

Frank Baetke, HP

Frank Baetke was recently elected by the EOFS Board to serve as the new chairman of the European Open File System organization.

Video: Intelligent Cache Hinting in Lustre

1573196

“Data caching can provide increased performance when using a mix of high and low performance storage, but traditional replacement algorithms like LRU may evict important data in multi-tenant environments, or in situations where the cache is “cold”. By tagging and prioritizing data within the storage system, we can create a more intelligent mechanism that avoids many of the problems inherent to traditional caching. Methods for prioritizing data and passing this information through the filesystem will be discussed, as well as a performance analysis of small file IO in Lustre with cache hinting, and possible future enhancements.”

Video: Understanding Hadoop Performance on Lustre

Hadoop Cluster

“In this talk, Seagate presents details on its efforts and achievements around improving Hadoop performance on Lustre including a summary on why and how HDFS and Lustre are different and how those differences affect Hadoop performance on Lustre compared to HDFS, Hadoop ecosystem benchmarks and best practices on HDFS and Lustre, Seagate’s open-source efforts to enhance performance of Lustre within “diskless” compute nodes involving core Hadoop source code modification (and the unexpected results), and general takeaways ways on running Hadoop on Lustre more rapidly.”

Video: Lustre + Linux – Putting the House in Order

tux

In this video from LUG 2015 in Denver, James Simmons from ORNL presents: Lustre + Linux – Putting the House in Order. “In the last year great strides have been made to sync up the lustre Intel branch to what is upstream. We present what that current state is as well as what is left for the intel branch to bring this to completion.”

Video: Monitoring a Heterogeneous Lustre Environment with Splunk

lug

“Monitoring a large Lustre site, running multiple generations of Lustre filesystems can be a challenge. Some equipment offer vendor specific monitoring interfaces while others, built on open source Lustre, have minimal monitoring capabilities. This talk will report on our operational experience using a homegrown python module to collect data from each filesystem. We will discuss in detail how the data is visualized centrally in Splunk and cross-referenced with users workload to analyze and troubleshoot our environment.”

Deploying Hadoop on Lustre Storage: Lessons Learned and Best Practices

Hadoop Cluster

In this video from LUG 2015 in Denver, J.Mario Gallegos from Dell presents: Deploying Hadoop on Lustre Storage: Lessons Learned and Best Practices. “Merging of strengths of both technologies to solve big data problems permits harvesting the power of HPC clusters on very fast storage.”

Video: Current Status of ZFS as Backend File System for Lustre

zfs-linux

“Intel supports users, system integrators, and OEMs using ZFS with Intel Lustre. In this presentation, we summarize the results of proof-of-concept (PoC) on a variety of the ZFS configurations. We cover sequential and metadata performance, data Integrity, manageability, availability and reliability. The work identifies the areas where development should be focused in order to fill gap in performance or functionality and encourage system administrator to integrate this technology with the existing high availability framework like Pacemaker/Corosync. We also cover the most important tunables for ZFS in combination with Lustre and the most notable metrics for Lustre and ZFS.”

Video: Lustre Network (LNET) Router Configuration and Tuning

fragalla

“n this session, Seagate covers configuration guidelines and tuning of LNET Routing from InfiniBand to Ethernet using Lustre 2.1 through 2.6 server/clients as well demonstrating performance results by means of a synthetic benchmark called IOR. In addition, this presentation includes the topic of LNET Router failure and recovery during I/O as well as what environments can expect during these failure events.”

Introduction to the Lustre File System

lustre-logo2

Although there are a number of truly huge implementations of Lustre today, the community is still far from reaching the maximum configurations that the Lustre architecture is designed for. Inside the Lustre File System describes the basics of how the Lustre File System operates with descriptions of the newest features.