Sign up for our newsletter and get the latest HPC news and analysis.

Progress Report on Efficient Integration of Lustre and Hadoop/YARN

In this video from LUG 2014, Weikuan Yu from Auburn University and Omkar Kulkarni from Intel present: Progress Report on Efficient Integration of Lustre and Hadoop/YARN.

Using Hadoop with Lustre provides several benefits, including: Lustre is a real parallel file system, which enables temporary or intermediate data to be stored in parallel on multiple nodes reducing the load on single nodes. In addition, Lustre has its own network protocol, which is more efficient for bulk data transfer than the HTTP protocol. Additionally, because Lustre is a shared file system, each client sees the same file system image, so hardlinks can be used to avoid data transfer between nodes.

goal

Download the slides (PDF) * See more talks from LUG 2014

Resource Links: