Anderson Banihirwe from NCAR gave this talk at SciPy 2019. “This talk demonstrates how to use Dask and Jupyter on large high-performance computing (HPC) systems to scale and accelerate large interactive data analysis tasks — effectively turning HPC systems into interactive big-data platforms. We will introduce dask-jobqueue which allows users to seamlessly deploy and scale dask on HPC clusters that use a variety of job queuing systems such as PBS, Slurm, SGE, and LSF. We will also introduce dask-mpi, a Python package that makes deploying Dask easy from within a distributed MPI environment.”
MIT Lincoln Laboratory Takes the Mystery Out of Supercomputing
January 18, 2017 by
“Many supercomputer users, like the big DOE labs, are implementing these next generation systems. They are now engaged in significant code modernization efforts to adapt their key present and future applications to the new processing paradigm, and to bring their internal and external users up to speed. For some in the HPC community, this creates unanticipated challenges along with great opportunities.”
