A look at what Panasas is calling Dynamic Data Acceleration (DDA) and how it dramatically improves HPC performance in a mixed-workload environment.
By Curtis Anderson, Senior Software Architect – Panasas, Inc.
The majority of HPC sites run a much wider range of applications on their HPC cluster now than they did even just four years ago. Each of those applications applies a different pattern of file operations to the storage subsystem, a “mixed workload”.
HPC storage subsystems have traditionally struggled with adjusting to these changing environments, delivering sub-par and dramatically inconsistent performance. A new HPC parallel file system technology is looking to solve this by automatically adapting to changing file sizes and workloads.
Modern Storage Performance Can Be Challenging
Your compute nodes and your applications are sitting idle if they’re waiting for storage.
All-flash solutions sound like the right answer, until you get the quote and realize how many compute nodes you could buy with that.
Traditional tiering based upon a file’s temperature sounds complex but OK, until you realize all those PBs of HDDs are not contributing to your performance, instead they’re buried in a layer your application cannot access. Tiering is double the complexity to manage and has lower overall reliability compared to an integrated single-tier solution.
Hybrid storage solutions that include HDDs as well as SSDs in a single tier are cost-effective, but can they deliver the performance you need?
Dynamic Data Acceleration: Automatically Adapting to Diverse and Changing Workloads Without Manual Intervention
Dynamic Data Acceleration is a new, proprietary software feature of the Panasas PanFS® parallel file system that utilizes a carefully architected combination of technologies to get the most out of all the storage devices in the subsystem.
In this integrated system, NVMe SSDs store metadata, SSDs store small files, and large files are stored on low-cost, high-bandwidth HDDs. By dynamically managing the placement of files on SSD and HDD, based upon the size of the files, PanFS with Dynamic Data Acceleration delivers twice the performance per HDD than any other hybrid HPC parallel file system. Each type of device is used only for the workloads it excels at supporting, maximizing the efficiency of all storage media in a seamless, total-performance system that automatically adapts to changing file sizes and workloads.
Dynamic Data Acceleration is a feature of the PanFS parallel file system running on the Panasas ActiveStor® Ultra HPC turn-key appliance. ActiveStor Ultra Storage Nodes are dedicated to storing data and metadata while Director Nodes are dedicated to storage cluster management. It is a single-tier architecture that is more resilient and manageable compared to the complexity and manual intervention associated with tiered HPC storage systems.
Dynamic Data Acceleration uses these storage devices in each ActiveStor Ultra Storage Node:
- An extremely low latency NVDIMM (a storage class memory device) is used to capture the user data and metadata being written by the application so the application can continue as quickly as possible.
- The 32GB of DRAM in each ActiveStor Ultra is used as an extremely low latency cache of the most recently read or written data and metadata.
- A very low latency NVMe SSD contains a database that stores and retrieves all metadata.
- Quick and high bandwidth SSDs are used to store small files.
- HDDs provide high bandwidth storage if they are never asked to store anything small and only asked to do large sequential transfers. We only store large files on our low-cost-per-TB HDDs.
Change is a Constant, Adaptability without Intervention is Required
Inconsistent performance and lack of adaptability in the face of change has been a major headache for both application users and storage administrators. PanFS with Dynamic Data Acceleration eliminates this headache by providing HPC and enterprise IT organizations with a high-performance storage solution that keeps up with their pace of change.
For more information on Dynamic Data Acceleration and the PanFS parallel file system download the White Paper.
