Sign up for our newsletter and get the latest HPC news and analysis.
Send me information from insideHPC:


On the Long Path to Exascale, Storage is Already Working Smarter

Sponsored Post on Exascale Storage

As the countdown to Exascale continues, Exascale-like storage problems are already showing up in today’s massively parallel, heterogeneous HPC systems. Historically, storage and I/O have kept pace with growing system demands, but, because of the limitations of spinning media and the cost of solid state storage technologies, storage performance improvements have come at a disproportionately higher cost and lower efficiency than their compute counterparts. With current storage challenges demanding a solution now, a smarter approach has surfaced, opening a new paradigm for storage and I/O.

Historical Approaches

Conventional approaches of scaling performance and capacity on spinning and Solid State media have proven to be expensive and highly inefficient. On spinning disk, the number of drives is directly correlated with I/O delivery, so you simply add more spindles to increase performance. Straightforward as it may seem, these high performance drives are more expensive, lower in capacity and force users into an extremely inefficient growth path. Alternatively, Solid State media offers superior performance in fewer drives, but again only delivers low capacity and expensive scaling due to its significantly higher cost per gigabyte.

With both approaches, the desired I/O is achieved, but neither allows the user to optimize cost and capacity. They are tied and this constraint usually means the user is either paying a lot for capacity they don’t need (which is also expensive to power and cool), or achieving the performance they need, but at a very high cost and with little capacity.

Back in 1999, when VMware® launched the virtualization revolution by decoupling the physical server from the logical server, they created a new compute provisioning paradigm that forever changed the data center. VMware allowed users to run multiple jobs on a single virtualized system. This approach maximized compute resources, effectively slashing the cost of individual computing operations.

Introducing IME, the Smart Method

Much like the business and architectural transformation that resulted from VMware’s innovations, DataDirect Networks (DDN) has addressed the long-standing challenges associated with the overprovisioning of storage by decoupling I/O performance from capacity. The solution, known as Infinite Memory Engine™ (IME), is a highly transactional, resilient and reliable “burst buffer cache” and I/O accelerator for HPC and Big Data applications.

Storage Exascale

IME is composed of client software resident on compute nodes, and server software for the I/O servers that aggregate and virtualize disparate compute or I/O server resident SSDs. This creates a single pool of extremely low latency, high performance, non-volatile memory-based storage to become a new fast data tier.

Not only does IME intelligently decouple storage performance from spinning disk storage capacity, it also:

  • Significantly accelerates applications by moving I/O right next to compute resources to reduce latency, delivering 50% faster performance than all flash arrays
  • Greatly reduces cluster idle time through intelligent, forward looking I/O provisioning
  • Breaks down network bottlenecks for more efficient data center operation
  • Reduces power consumption, increases data center density and lowers system cost

Typical Big Data and HPC applications addressed by IME include analytics, financial services, scientific computing and research, life sciences/genomics, oil and gas, and many more.

Unparalleled Data Center Efficiency

IME brings numerous benefits to the data center.

For example, IME:

  • Boosts data center efficiency by dramatically reducing hardware, power, floor space and the number of components to manage and maintain
  • Provides massive application acceleration by returning wasted processing cycles to compute that were previously managing storage activities or waiting for I/O from spinning disk, greatly increasing compute ROI
  • Is compute and storage hardware agnostic as this software-defined storage scales limitlessly and protects data via distributed erasure coding in this NVM fast data tier

Working Smarter Always Leads to Lower TCO

With IME, DDN has addressed a storage problem that has been unresolved ever since the introduction of disk-based storage. IME allows data centers to run a greater number of complex simulations faster and with less hardware. Large datasets can be moved out of HDD storage and into memory quickly and efficiently. Then, data can be moved back to HDD storage once processing is complete much more efficiently with unique algorithms that align small and large writes into streams, enabling users to implement the largest, economical HDDs to hold capacity. Workload performance is optimized to reduce time to insight and discovery. Cost savings of up to 80% can be realized while achieving infinite scalability and highly efficient I/O performance.

DDN’s IME solution transforms storage from a bottleneck to becoming a major contributor to a smoothly functioning IT infrastructure that supports the organization’s most ambitious HPC and big data and performance-intensive applications.

And looking to the future, IME has taken its place as one more step on enabling the road to Exascale.

DDN is a leading provider of high performance storage systems. For massively scalable high performance storage contact DataDirect Networks, www.ddn.com or call: 1-800-837-2298

Resource Links: