Full-spectrum HPC Scheduling

Print Friendly, PDF & Email

Photo: Shutterstock/By SeventyFour

In this Sponsored Post, our friends over at Altair explain how a meta-scheduler, or hierarchical scheduler, can be thought of as a private or team-based scheduler that uses shared underlying resources.

High-performance computing (HPC) used to be easy to define. It was a data center full of big-iron supercomputing power owned by government, university, or large enterprise. Traditional HPC workloads in areas including weather prediction, crash testing, and failure analysis are well-understood, and HPC hardware and applications have largely evolved alongside one another.

Increasingly complex workloads

As technology has evolved and become available anywhere and anytime in the cloud, HPC is accessible to a broader audience. It’s also becoming increasingly complex as it converges with technologies like machine learning (ML) and artificial intelligence (AI).

ML kernels, while built on established algorithms, are still in their infancy when applied to real-world problems. ML is a complex set of interdependent tasks with extraordinary data sets. These tasks may also be run incrementally in response to algorithm parameter changes or modest changes to the data sets.

With today’s complex and varied workloads, it’s not “one size fits all” for computing power — or for the technology that orchestrates how jobs will run on it.

Lengthy, highly parallel jobs vs. quick, high-throughput jobs

Traditional HPC jobs are normally long and highly parallel, and they are also well-understood. A weather simulation will be rerun every 4 hours or so and the codes change only occasionally.

In contrast, the semiconductor design space has a wide range of runtimes. Short jobs might take less than 5 minutes while longer jobs can run for a week or two. Runtimes can also vary widely for even small change inputs. Additionally, there is a significant mix of memory and core requirements. While the bulk of logical simulation jobs scale up to 4 cores, some design checks may be parallel to several hundred cores.

Workloads for meta-scheduling

A meta-scheduler, or hierarchical scheduler, can be thought of as a private or team-based scheduler that uses shared underlying resources. Difficult workloads and special workloads are good candidates for meta-scheduling. Examples include sets of several hundred thousand short jobs, jobs with complex dependencies, and workflows that are continually introspected for status.

These workloads put a large load on a shared scheduler and may be classed as “bad” when in fact they may be in themselves efficient but very hard for a scheduler to process. The accommodation for these workloads might be separate infrastructure — itself highly inefficient — or a “don’t do that!” response from the queue admin.

How a meta-scheduler works

Meta-scheduling can be handled in different ways. Altair’s meta-scheduling model, used by Altair Acclerator™ Plus, is demand transfer, a process during which the upper-level scheduler expresses a demand on the base or shared scheduler, which responds with resources as they become available. The upper-level scheduler directly connects to those resources, bypassing the base scheduler, for job dispatch. This is not a job forwarding model. Jobs are retained by the upper-level scheduler so any job introspection (for job status, for example) is handled by the upper-level scheduler. The base scheduler is offloaded from both queries and, in the case of short jobs, from the individual dispatch load.

A private scheduler can dispatch several hundred jobs per second. Multiple scheduler instances can be used, and each has its own dispatch capacity. Meta-scheduling can readily achieve 6-10x improved throughput over the base scheduler for large numbers of similar jobs — without the need for additional compute capacity.

Diverse workloads on a single HPC infrastructure

Private schedulers are fast, but complex policies still need to be handled by the base scheduler. Highly parallel jobs and wide jobs are also best handled by the base scheduler. Hierarchical scheduling provides a blend of both fast and smart scheduling using the same shared resources. No partitioning is required, and that’s a significant performance win.

Meta-schedulers like Accelerator Plus enable greater throughput, better license and resource utilization, and more flexible scheduler usage models — plus millisecond dispatch latency, which yields the most benefits for short jobs. Accelerator Plus can be layered on top of Altair Accelerator™ for EDA jobs and on Altair PBS Professional™ for more traditional HPC workload scheduling.

Hierarchical scheduling is especially useful in the semiconductor industry for workloads including design verification regressions, library QA, physical design flows with complex dependencies and data handoffs, library characterization, and software unit testing. In data science, it can be employed to expedite jobs such as backtesting, parameter sweeps, labeling, and incremental training.

Learn more about hierarchical scheduling and Altair Accelerator Plus.