This guide to lustre from the editors of insideHPC explores high performance storage solutions in an exploding commercial data universe. Each week we’ll release a section of this guide or you can download the complete PDF from the insideHPC White Paper Library courtesy of Intel.
Overview
The recent release of a commercial version of the Lustre* parallel file system was big news for business data centers facing ever expanding data analysis and storage demands. Now, Lustre, the predominant high-performing file system installed in most of the supercomputer installations around the world, could be deployed to business customers in a hardened, tested, easy to manage and fully supported distribution.
Proven to scale up to extreme storage capacities measured in tens or even hundreds of petabytes accessible to tens of thousands of clients, Lustre offers high throughput with high availability that is vendor-neutral for hardware, servers, and network platforms.
In this Guide, we take a look at what an HPC solution like Lustre can deliver for a broad community of business and commercial organizations struggling with the challenge of big data and demanding storage growth.
Data – The Next Big Challenge
For a long time, the big challenge was squeezing as many compute cycles as possible out of silicon chips so they could go on solving the really gigantic problems in science and engineering faster than was ever thought possible. Now, by clustering computers to work together on problems, scientists are free to conceive even larger and more complex real-world problems to compute, and data to analyze.
The new challenge is keeping those very fast compute engines fed with a constant and rich stream of data.
Until recently, this data challenge was limited to scientists at the national labs doing large scale government-sponsored supercomputing research in astrophysics, weather prediction and climate modeling, or in data-intense industries such as aerospace, oil and gas exploration, and defense.
Today, detailed analyses over extreme volumes of data has become a major challenge for commercial enterprises solving equally complex business computing problems, from predicting financial risk and bringing drugs to market faster, to detecting patterns of fraud over millions of internet transactions. Enterprise data centers have discovered that their conventional hardware/software configurations and approaches are not nearly adequate to handle this explosion of data-intensive computing. Businesses are finding that those High Performance Computing (HPC) solutions developed in the labs need to be repurposed to create solutions for performing complex commercial computing tasks for the world of High Performance Data Analytics (HPDA).
Over the next few weeks this editorial series will explore the topics of:
- Attaining High Performance Scalable Storage
- What is Lustre?
- Commercial Grade Lustre File Systems
- Is Lustre Scalable, Affordable and Reliable?
- Unleashing the Power of Parallel Storage
If you prefer you can download the complete guide in a PDF from the insideHPC White Paper Library courtesy of Intel.
* Other names and brands may be claimed as the property of others.
