Sign up for our newsletter and get the latest big data news and analysis.

Interactive Supercomputing with Jupyter and DASK

Anderson Banihirwe from NCAR gave this talk at SciPy 2019. “This talk demonstrates how to use Dask and Jupyter on large high-performance computing (HPC) systems to scale and accelerate large interactive data analysis tasks — effectively turning HPC systems into interactive big-data platforms. We will introduce dask-jobqueue which allows users to seamlessly deploy and scale dask on HPC clusters that use a variety of job queuing systems such as PBS, Slurm, SGE, and LSF. We will also introduce dask-mpi, a Python package that makes deploying Dask easy from within a distributed MPI environment.”