In this special guest feature, Wolfgang Gentzsch from The UberCloud writes that we’ve never been so close to ubiquitous computing for researchers and engineers.
Wolfgang Gentzsch is President of The UberCloud.
High-performance computing (HPC) continues to progress, but the next big step toward ubiquitous HPC is coming from software container technology based on Docker, facilitating software packaging and porting, ease of access and use, service stack automation and self-service, and simplifying software maintenance and support.
In 2013, Docker Linux Containers saw the light of day. The key practical difference between Docker and Virtual Machines is that Docker is a Linux-based system that makes use of a userspace interface for the Linux kernel containment features. Another difference is that rather than being a self-contained system in its own right, a Docker container shares the Linux kernel with the operating system running the host machine. It also shares the kernel with other containers that are running on the host machine. That makes Docker containers extremely lightweight, and well suited for HPC, in principle. Still it took us about a year to develop – based on micro-service Docker container technology – the ‘macro-service’ production-ready counterpart for HPC, plus enhancing and testing it with a dozen of HPC applications and with complex engineering workflows, on about a dozen different single- and multi-node cloud provider resources. These high-performance interactive software containers, whether they be on-premise, on public or on private clouds, bring a number of core benefits to the otherwise traditional HPC environments with the goal to make HPC widely available, ubiquitous:
- Packageability: Bundle applications together with libraries and configuration files: A container image bundles the needed libraries and tools as well as the application code and the necessary configuration for these components to work together seamlessly. There is no need to install software or tools on the host compute environment, since the ready-to-run container image has all the required components.
- Portability: Build container images once, deploy them rapidly in various infrastructures: Having a single container image makes it easy for the workload to be rapidly deployed and moved from host to host, between development and production environments, and to other computing facilities easily. The container allows the end user to select the appropriate environment such as a public cloud, a private cloud, or an on-premise server.
- Accessibility: Bundle tools such as SSH into the container for easy access: The container is setup to provide easy access via tools such as VNC for remote desktop sharing. In addition, containers running on computing nodes enable both end-users and administrators to have a consistent implementation regardless of the underlying compute environment.
- Usability: Provide familiar user interfaces and user tools with the application: The container has only the required components to run the application. By eliminating other tools and middleware, the work environment is simplified and the usability is improved. The ability to provide a full featured desktop increases usability (especially for pre and post processing steps) and reduces training needs.
- Security: We understand that you need to know your data is secure: Engineers care about the safety of their designs. With UberCloud HPC containers you run in a private compute environment. We enable encryption for data transfers, remote access, and data storage. Your cloud resources are dedicated to you, and they are not shared. This way, potential security threats from root access are completely eliminated. And when your processing is finished, your container environment is deleted.
We only use professionally managed cloud infrastructures, with strong physical security controls including biometric entry authentication and armed guards. You have fine grained access control which gives you the flexibility to define your security requirements. Your IT team tells us what firewall rules and authentication methods they need. We implement them. Done.
Figure 1: Why are HPC software containers becoming more and more popular? (numbers are from high to low priority)
During the past six years we have successfully built HPC application containers for SIMULIA Abaqus, GROMACS, ANSYS, Flowvision, LS-Dyna, Siemens/STAR, COMSOL, LAMMPS, NAMD, NICE, Numeca FINE/Marine, FINE/Turbo, OpenFOAM, PSPP, Red Cedar HEEDS, Scilab, Transvalor, Matlab, and more. These application containers are now running on cloud resources from Advania, Amazon AWS, Atos/Bull, CPU 24/7, Google, Microsoft Azure, Opin Kerfi, OzenCloud, and others. Before we used our application containers, our HPC cloud projects took on average about 3 months. Since we started to apply containerization in January 2015 (with HPC cloud project #143), our users’ cloud projects take only 3 days on average, which speaks for itself !
Together with recent advances in engineering application software and HPC hardware technologies, the advent of lightweight pervasive, packageable, portable, scalable, interactive, easy to access and use HPC software containers running seamlessly on workstations, servers, and any cloud, is bringing us ever closer to what Intel – in 2012 already – called the democratization of high performance technical computing. We are arriving at the age of Ubiquitous Computing where computing “technology recedes into the background of our lives” and High Performance Computing will be in the hands of every engineer.
Wolfgang Gentzsch is an industry executive consultant for high performance, technical and cloud computing. He was the Chairman of the Intl. ISC Cloud Conference Series, an Advisor to the EU projects DEISA and EUDAT, directed the German D-Grid Initiative, was a Director of the Open Grid Forum, Managing Director of the North Carolina Supercomputer Center (MCNC), and a member of the US President’s Council on Science & Technology PCAST. In the 90s Wolfgang founded several HPC companies, including Gridware (which developed the distributed resource management software Grid Engine) acquired by Sun where he became Sun’s Senior Director of Grid Computing.
If you are interested in more details and in real-life engineering case studies, You can download the full article and the free UberCloud Compendium of 21 HPC container-based engineering case studies.
Sign up for our insideHPC Newsletter
