Since its announcement at SC15 in November, the OpenHPC community has made important strides toward its mission of creating and supporting a flexible open source HPC software stack that simplifies deploying and managing HPC systems. In just a few short months, the open source community hosted at The Linux Foundation has had many productive working group discussions, installed a Technical Steering Committee (TSC) and Governing Board, and even provided releases of the initial software stack based on early community feedback. The initial software stack includes over 60 packages, including tools and libraries, as well as provisioning, a job scheduler and more. The complete list is available on the project GitHub page.
Governance structure now in place
On June 16, OpenHPC announced it had established technical leadership and the formation of working groups to advance the project’s mission and accelerate code contributions. Like any open source community, the long-term success of OpenHPC will depend on the expertise and enthusiasm of participants and their ability to create momentum for the project. Early participation in OpenHPC has been robust with experts from a diverse group of stakeholders filling technical and governance positions. Balazs Gerofi of RIKEN, a TSC member, said his initial experience with the community has been encouraging. “I was surprised to see how fast OpenHPC evolved and how enthusiastic the vendors and labs were to participate. It seemed like right out of the gate, there was already a repository set up where one could pull tested packages from,” Gerofi explained.
The OpenHPC Governing Board, which is composed of platinum directors and an elected representative of silver members, is tasked with big-picture direction for the community, including budgetary oversight, establishing IP policies, marketing, and long-term roadmap guidance. It includes Alan Clark (SUSE), Binh Chu (Dell), Figen Ulgen (Intel) and Sunny Sundstrom (HPE). Karl Schultz (Intel) is the technical director. The strong cross-industry representation on both the TSC and Governing Board will help ensure that the community considers the wide-ranging technical needs of different types of HPC sites.
The TSC is responsible for technical direction and managing code contributions for the community. It will manage the majority of day-to-day efforts and contains contributors from across 18 leading national computing labs, universities and industry organizations. TSC positions include project lead, integration testing lead, component development representative, end-user site representative, integration testing lead, and component maintainers. In the near future, the TSC will begin releasing new resources for the community, including an overview for how the community will obtain new components along with a publishing road map.
Providing a flexible set of building blocks
The OpenHPC community is keenly aware that to support innovation on medium and high-end HPC systems, flexibility will be critical. Yet there has been some confusion about whether the OpenHPC stack is an all-or-nothing solution.
According to Karl Schulz, Principal Engineer for the Intel Enterprise and High Performance Computing Group, and project lead for OpenHPC, the community is meant to be a mid-stream building block type of repository that integrates and tests third-party software and then makes it available as a distribution. Sites can pick and choose components that are useful to them and use them in their own environments with tools that may not be available in OpenHPC.
“We document how to do a system install from end to end, starting with bare metal. But if you don’t want to use the OpenHPC provisioning system, there’s not a requirement with other downstream software that the development environment is tied to the provisioning system,” explained Schulz. Obviously, there are some dependencies between all of these different software packages. Not every component can be peeled away, but many of them can. By design, that’s part of OpenHPC; it’s intended to be a flexible gathering location for lots of software.”
He adds that the long-term intent is to expand options for key functional packages over time. “We recognize that different packages have different strong points, but initially we just had to pick options that had reasonable traction within the community,” he explained.
Contributors step up to advance OpenHPC mission
The OpenHPC Governing Board and TSC members bring an array of perspectives on the best way to create a software stack capable of supporting the widely varying needs of different types of HPC sites, but they are united by the goals of reducing inefficiencies and accelerating innovation in HPC wherever possible.
Scott Misage, VP and GM, High Performance Computing for HPE Servers, and a member of the Governance Board, believes the timing was right for the OpenHPC community launch because closer collaboration among lab and industry stakeholders has become essential to faster innovation. “Being a leader in today’s HPC world means collaborating with other vendors, educational and governmental institutions, developers, and users, to make contributions that expand the potential of HPC, and we think that OpenHPC can be instrumental in expanding and improving this collaboration. As a leading HPC system supplier, we also believe we have a lot to offer this large, diverse community from a leadership perspective and that it was important to step up and participate,” said Misage.
Piush Patel of Altair said that his company got involved in OpenHPC because the community’s goals align very closely with the direction it has set for some of its HPC product offerings. He notes that Altair’s product design group faces many of the challenges that OpenHPC is looking to address on a daily basis: “Altair’s product development group consists of more than 700 engineers who leverage HPC to run advanced engineering simulations that help them design innovative products and solutions on behalf of our customers. Therefore, we have had considerable first-hand experience with the challenges and complexities of HPC…We realized we had an opportunity to work with the OpenHPC community to simplify many aspects of HPC,” explained Patel. As part of its contributions, Altair has not only assigned a leading technologist to participate on the TSC, it is also creating an open-source licensed version of its leading workload manager, PBS Professional, which will be included as an upstream project to OpenHPC.
Balazs Gerofi of RIKEN said that because open source software has proven to be an accelerator of innovation, RIKEN didn’t hesitate to get involved in OpenHPC from early on. “RIKEN believes that open source software plays a very important role in supercomputing, we develop several of our own open source software components targeting large-scale supercomputers and we were very excited to be part of an effort that is driven by the same mentality… We believe that the OpenHPC project could be highly beneficial for further innovation in the field of supercomputing” explained Gerofi.
Schulz explained that after seeding the project, Intel is excited to see the community-wide efforts taking shape. “From the beginning Intel knew that participation from diverse stakeholders would be key to its success, which is why the project is hosted at The Linux Foundation, a neutral forum for open source collaboration. The HPC community response has been stronger than expected, and it’s been heartwarming to see the number of people who are eager to participate…Personally, I came to Intel from a top academic lab where I constantly wrestled with the challenges that OpenHPC is trying to address. That’s why I’m so confident that despite the fact that every lab might do things slightly differently, there is a huge amount of collective wisdom with these packages that we can finally tap to minimize repeated efforts,” Schulz said.
A promising start
According to Schulz, early feedback from people who have “kicked the OpenHPC tires” have been mostly positive. He notes that 100% of users will never be happy, but most of the people he’s talked to say that the current approach to OpenHPC makes sense, even if they might have done some things differently. “With a challenge as complex as this, simply deciding on a binary distribution route will lead to a number of decisions that will disappoint some people,” explained Schulz. “Think of something as trivial as the install pack. Where do you install all of this software? Different sites have different conventions… That’s why we have tried to do things in a generic way. We know we won’t bat 100%, but most users will understand the reasoning behind our decisions.”
Adoption encouraged, feedback and input appreciated
While most of the TSC and Governing Board roles have been filled by early project supporters, that doesn’t mean others can’t get involved. Schulz said that there is no requirement to be part of the formal governance structure to be part of the project; as an open source project, all are welcome to participate. “People can make suggestions and even contribute code right of the box. We appreciate that.,” he explained. Other OpenHPC members echoed his sentiment about the importance of getting involved. Gerofi said that it remains to be seen is how effectively the OpenHPC community will be able to ensure all the components work together, adding “I think there is lot of space for participation there.” And Misage pointed out how HPC technology is now the base for a lot of science shaping our world, so involvement from leading labs, developers and users is critical: “We only are as strong as our community. The power of the community-driven software model has been proven in Linux, in Big Data, and in dozens of other projects and markets. Get involved in OpenHPC and help us all maximize its potential.”