To most computer users, saving a Microsoft Word file to disk or copying a file to a USB drive is as close as they will get to seeing I/O in action, but the needs and requirements in a High Performance Computing (HPC) environment are quite different. Hundreds to thousands of servers will be writing their files to hundreds to thousands of individual disk drives. To accomplish this with performance and scalability requires the use of the Lustre High Performance Parallel File System. All users should read this whitepaper, which describes the Lustre file system, or Lustre 101 for short.
The fact that 75 % of the fastest HPC systems in the world use Lustre, demonstrates an on-going belief in the future of Lustre and its continued development. Lustre contains a number of components, that together, allow for efficient and fast storage and retrieval of large amounts of data. Lustre is based on Linux and uses kernel-based server modules to obtain the high performance I/O that these users require. In terms of capacity, the theoretical limits on the Lustre file system allows for storage of over 512 Petabytes (PB) in total. That is equivalent to 1,000,000of the 1 TB drives in your latest laptop.
Lustre separates the actual content of the files and the metadata describing the file and this allows data to grow to very high numbers while maintaining high performance. To obtain such high performance when reading or write a file, Lustre can stripe the data over a number of physical drives and optimizes the placement of the data onto drives with more available capacity to create a more balanced system. Lustre works with a variety of network entities, such as the latest Ethernet and IB networks, both of which are available in modern HPC data centers.
The whitepaper, Inside the Lustre File System from Seagate describes the inner workings of Lustre in a way that is easy to understand, yet is technical enough for many users and systems administrators. Lustre is a mature and stable file system that through its developers has consistently been able to respond to the needs of organizations that require high performance throughput and expanding capacity. Lustre 101 is a must read for developers, implementers and system administrators who require the utmost in file system performance for large scale computing. Download the whitepaper now.
