Delivering on the Promise of Hybrid Multi Cloud for HPC

Sponsored Post

The high-performance computing (HPC) community has generally been slow to adopt cloud compared to other industry sectors for several well documented reasons:

  • Compromised performance due to virtualised compute instances
  • Applications availability and difficulty with licensing
  • The complexities of setting up an HPC environment within a public cloud framework
  • Requirements for low latency interconnect and fast, parallel file systems
  • Concerns over security

However, today many of these barriers to adoption have been removed or mitigated and according to industry analysts such as Hyperion and Intersect360 Research HPC cloud is now the fastest growing segment of the HPC market.

Recent surveys of major HPC facilities report that over 90% of all HPC centers expect to use cloud services for at least part of their HPC service provision by 2026 [1].  It is crucial to note though, that this is generally supplemental to on-premises resources rather than replacing them completely.

Some market verticals, such as Life Sciences, were early adopters of cloud and are leading the charge, although it is now true that all sectors are growing, moving increasing portions of their workload the cloud to:

  • Reduce the size and associated costs of on-premises facilities
  • Move non-time critical jobs to the cloud releasing on-premises resource for more time critical works
  • Evaluate new technologies without the high cost of capital purchases
  • Burst workloads to manage peaks and troughs

Atos OneCloud is an industry wide initiative to help organizations migrate workloads to the cloud, finding the right balance points that work for them.  Core features of OneCloud include:

  1. Industry specific Consultancy services to develop Cloud business solutions at scale
  2. Multi-Cloud orchestration across Private and Public, and across all major Public Cloud providers to maximize application migration and portability, lower operational costs, and ensure Cloud interoperability
  3. Highly standardized and automated management framework and architecture
  4. Next generation Private & Sovereign Cloud platform, ready to be deployed and managed in any data center, to ease the migration to the Cloud while ensuring compliance requirements are met around data sovereignty and security
  5. Cloud Application Development and Modernization / Re-platforming, including strong DevSecOps, allowing customers to accelerate business-critical applications’ time to market
  6. Cloud Artificial Intelligence & Machine Learning to enhance business processes, create new solutions and monetize enterprise data
  7. World-leading Bare Metal solutions to support non-virtualized business critical applications adjacent to the Cloud, to increase the breadth of consumable Cloud services
  8. World-class Cloud Edge & Far Edge solutions combined with new 5G connectivity solutions ensuring secure and local processing and optimized bandwidth consumption
  9. Cybersecurity supervision services, allowing to operate and respond to Cloud native security controls and anticipate threats in a prescriptive vs predictive mode
  10. Decarbonization offerings guaranteeing year on year carbon footprint reduction of Cloud infrastructure, data and applications.

What does Atos OneCloud mean for HPC?

There are three key service components required to deliver a comprehensive user experience for HPC in the cloud:

  • Simple point and click interface and API batch service end point for HPC provisioning, including bursting containerized workloads that can be expanded on demand on the target public cloud platform
  • Job submission to cloud via API or traditional schedulers such as SLURM or LSF gateways with the ability to track usage and integrate this with an automated charge back mechanism
  • Fully converged digital transformation of HPC with Kubernetes to support unified, multi cloud deployments

Atos’s recent acquisition of Nimbix [2] supplements its existing HPC cloud software environment and in doing so has created the industry’s most comprehensive hybrid HPC cloud portfolio.   Nimbix JARVICE XE was the world’s first container native hybrid/multi-cloud HPC platform, enabling scale-out and accelerated HPC and AI workflows (SaaS and PaaS) globally on any Kubernetes compatible infrastructure.

The Smart HPC Web Portal (XCS) gives users and system administrators alike secure and direct access to all resources and applications. Once a user is logged on, he or she can access a complete work environment, customized for their job role and applications. From there, they can load and manage data, set parameters for the simulation, run the calculation, track its progress, and then proceed to post-processing and visualization of results.

Atos’s exclusive 3D visualization module, XRV lets you use post-process modules directly on the server: only the encrypted, compressed images are transferred to the user workstation. Exchanging less data means greater security, more fluidity, and less use of infrastructure resources (networks, user workstations…), as well as being more convenient for users and offering more opportunities for increased collaboration.

Nimbix JARVICE XE brings industry-leading Supercomputing Cloud technology into the data center to support advanced computing workflows while maintaining the option to easily burst to any public cloud for additional resources. JARVICE XE uniquely delivers accelerated applications and workflows that take advantage of diverse infrastructure including InfiniBand, GPUs, and FPGAs on dedicated Kubernetes infrastructure. Specifically, it provides the:

  • Ability to convert SLURM jobs into low level “Pod” scheduler jobs
  • Flexibility to deploy scalable HPC jobs through containers
  • Tenant isolation so that no clients share the same resources
  • Nimbix HyperHub applications catalogue to help the user define workflows

 

Nimbix HyperHub provides a multi-cloud open marketplace for HPC and AI applications.

JARVICE XE enables customers to easily burst from on-premises solutions to public cloud systems from Google, AWS, and Azure and others or to manage internal systems as private clouds.

Cloud HPC Services

Of course, enabling access to resources is only one part of the equation.  To provide a complete solution requires access to consultancy and professional services offerings to maximize the cost effectiveness of the resources.

Through our global HPC Cloud Center of Competency and Center of Excellence in Advanced Computing, Atos is also able to offer clients the following services:

  • Cluster Configuration services: Design and deploy an HPC cluster in the Cloud
  • Workflow construction Build an Optimized HPC workflow which can be deployed on public cloud or private facilities
  • HPCaaS portal customization: Build a customized GUI for your key application in the cloud
  • HPC application porting and optimization: port and optimize your application to cloud based HPC platform
  • HPC Container Development: Create, build, and deploy containerized workflows
  • HPC application benchmarking: Benchmark your application on different cloud platforms to optimize your costs
  • HPC application analysis: understand the bottlenecks in your HPC applications
  • HPC Training services: Learn about HPC technologies and applications

Atos can also offer additional bespoke services for specific customer domains such as the following services:

  • CAE Simulation Integration: design and run CAE environment for product development/R&D
  • Accelerated Genome sequence analysis: either on-prem or cloud based

Conclusion

HPC cloud usage is expanding across all industry sectors with a strong desire from customers for a flexible hybrid model that can be deployed in a multi-cloud scenario.

However, problems persist in the simplified deployment of cloud for HPC and the management of workflows, applications and secure data movement.

With the acquisition of Nimbix and the combination of Nimbix JARVICE XE with existing tools and frameworks, Atos has created the industry’s most comprehensive portfolio of cloud HPC products and services with which to address the full range of envisaged HPC usage scenarios.

[1] Big Compute: 2021 State of Cloud HPC Report, HPC Practitioner Survey

[2] Atos Press Release July 27th, 2021:  Atos strengthens its leading position in HPC cloud enablement and acquires leading global HPC cloud platform provider Nimbix

About the Author

Andy Grant
VP, Large Strategic HPC Deals
HPC, AI & Quantum, Atos