Today Altair released version 13 of PBS Professional workload manager. To learn more, we caught up with Bill Nitzberg, CTO for Altair PBS Pro.
insideHPC: Altair is referring to PBS Professional as their biggest release in the 20-year history of the product. Was that the plan from the beginning, or did it grow into something much larger?
Bill Nitzberg: The foundation of 13.0 was defined at the outset to replace the underlying communication architecture of PBS Pro to support Exascale-sized systems. Plenty, by itself, for a major release. To accommodate such a fundamental change, we switched our release process from time-based to feature-based and also added a formal Beta test phase, both of which meant a longer release cycle.
Separately, a few years ago, we started a very customer-centric “innovation process” – prototyping, iterating and co-developing new features directly with customers. Our goal is to have at least three or more happy beta customers for a feature before declaring it ready for general availability, and we had a lot of features mature during the longer 13.0 release cycle.
So, architecting for exascale and adding tons of beta-driven capabilities: These two factors combined to make 13.0 “our biggest release ever.”
insideHPC: With the increasing complexity of today’s nodes and clusters, is the job for PBS Pro getting harder to do well?
Bill Nitzberg: For a time, it looked like HPC was converging on Intel-based clusters connected by a high-speed network, and the only complexity would be the network (and perhaps the different memory sizes on the nodes). The reality today is that HPC is getting more and more complex, with ARM and POWER joining the CPU race; with accelerators, co-processors, and GPU-based networking; and with more layers in the memory hierarchy than ever, including SSDs for burst buffer capabilities. Scheduling all these pieces is certainly more complex than just scheduling Intel-based CPUs connected by a network!
At its core, however, PBS Pro is a resource-based scheduler, and, to PBS Pro, all these “complexities” are just resources. So, for PBS Pro, the job really isn’t any harder. However, it is more computationally expensive. For that reason, we continue to pay a lot of attention to raw scheduling speed (e.g., profiling real customer workloads), and make major improvements in every release (13.0 included).
insideHPC: From the outside PBS Pro 13 looks to be all about performance. Would you agree?
Bill Nitzberg: Well… PBS Pro 13 brings huge improvements in speed, scale, and resilience – we believe that’s what makes PBS Pro 13 ready for exascale.
But… 13.0 includes a lot more: big enhancements in the plugin framework (including infrastructure supporting cgroups), enhanced scheduling policy controls (including an expanded job sort formula and preemption controls), and major upgrades in usability (including the ability to create custom resources directly from qmgr), and that’s not all!
insideHPC: What were you able to do “under the hood” to make PBS Pro faster?
Bill Nitzberg: When PBS Pro was originally designed more than 20 years ago, we built our own PBS-to-PBS, scalable communication layer. As the TCP/IP stack was pretty immature (both fragile and not so speedy), we build a reliable packet protocol based on UDP (called “RPP”). Back then, individual computers were very expensive, so trading a little reduction in speed for an increase in reliability was important. RPP has served PBS Pro amazingly well, but it was reaching its limits, so we embarked on a multi-year project to completely replace it.
Three fundamental changes have taken place in the industry since we developed RPP: system (cluster) sizes have gotten much larger, individual computers have become much less expensive, and “the web” changed the fabric of the internet leading to great improvements in the TCP/IP stack (which now easily handles tens of thousands of connections on a single socket).
The new PBS-PBS communication layer (“TPP”) leverages these fundamental changes. Our “TPP” has been built from the ground up; it uses multi-threaded (non-blocking) messaging for improved speed, employs persistent connections to reduce latency, and is fully fault-resilient to ensure node/link failures do not impact operation.
In addition, PBS Pro 13 defaults to a high-throughput scheduler mode where asynchronous job start overlaps scheduling work and job launch, increasing throughput, especially for small jobs.
insideHPC: We are still a long way from an exascale machine. How does PBS Pro move us closer to exascale computing?
Bill Nitzberg: The industry needs to accomplish a lot in the coming years to deliver a working, useful exascale machine. PBS Pro is only one piece of the puzzle… but it’s an important piece. Job scheduling and workload management are core capabilities – a “must have” for every HPC system – ensuring HPC goals are met by enforcing site-specific use policies, enabling users to focus on science and engineering rather than IT, and optimizing utilization (of hardware, licenses, and power) to minimize waste.
Exascale infrastructure requires advances in speed, scale, resilience, and power management. We’re confident the underlying architecture of PBS Pro 13 is the right foundation in all four of these areas. However, with another 10-100x increase to go, I’m sure there is plenty of work left for everyone.
One more thing: although we’re saying PBS Pro 13 is “architected for exascale,” making PBS Pro faster and more resilient really helps everyone – the small, the medium, and the really big.
insideHPC: In your mind, what is the one thing that sets PBS Pro apart from the competition?
Bill Nitzberg: Altair. I can’t say it any better than some of our customers, e.g,
For me the most important decision factor of the workload manager is the vendor…” –Weizmann Institute
Altair actually exceeded my expectation. I’ve worked with several companies… and haven’t got someone who is so committed.” –CHPC
There is no arguing that Altair is a unique company in the HPC space. Altair makes HPC middleware, develops HPC applications, and actually uses HPC for real engineering and design work. Bringing together users, applications developers and middleware developers under one roof provides a fantastic, positive feedback loop. It’s why we’ve made “Altair knows HPC” an unofficial tagline for the PBS Works business.
Of course, PBS Pro also offers some truly unique benefits over the competition. Our plugin framework, allowing customers to modify and extend PBS Pro on-the-fly, without source, is the most comprehensive. Our scheduling priority formula offers superior flexibility, allowing any arbitrary formula to express policy (including exceptions), versus a table-driven or coefficient-driven approach. And our focus on security is unmatched – PBS Pro is the only workload manager to achieve a globally recognized EAL3+ security certification and now we also support “MLS” with SELinux.
Altair will demonstrate PBS Professional 13.0 at booth #1110 during the ISC High Performance conference in Frankfurt, Germany.