In this video from HPCKP’19, Christian Kniep from QNIB Solutions presents: Containerized Convergence of Big Data and Big Compute.
Since the early days of High Performance Computing operations and usage of such systems was highly dependent on the vendor, the discipline, the diversity of use-cases and the community it was embedded in. The use-cases and communities of Big Data, Big Compute are converging, due to commoditization in hard- (x86) and software (Linux), the growing importance of big enterprise IT and the advent of HPC characteristics in AI/ML workloads. With the introduction of containers as distribution artifact and Kubernetes as orchestration substrate, this convergence might experienced it’s final push to take place. This talk will dissect the convergence by refreshing the audiences’ memory on what containerization is about, segueing into why AI/ML workloads are leading to fully fledged HPC applications eventually and how this will inform the way forward. In conclusion Christian will discuss the three main challenges `Hardware Access`, `Data Access` and `Distributed Computing` in container technology and how they can be tackled by the power of open source, while focusing on the first.
Christian Kniep recently joined AWS. With a 10 year journey rooted in the HPC parts of the german automotive industry, Christian started to support CAE applications and VR installations. After getting bored with the small pieces, he became the InfiniBand go-to-guy while operating a 4000 node crash-test cluster and pivoted to the R&D department of Bulls BXI interconnect. When told at a conference that HPC can not learn anything from the emerging Cloud and BigData companies, he became curious and is now leading the containerization effort at Playstation Now. In 2017 Christian joined Docker Inc to help push the adoption forward and be part of the innovation instead of an external bystander. Christian likes to explore new emerging trends by containerizing them first and seek application in the nebulous world of IT.
