Xiaoyi Lu from Ohio State University
In this video from the 2018 OpenFabrics Workshop, Xiaoyi Lu from OSU presents: High-Performance Big Data Analytics with RDMA over NVM and NVMe-SSD.
“The convergence of Big Data and HPC has been pushing the innovation of accelerating Big Data analytics and management on modern HPC clusters. Recent studies have shown that the performance of Apache Hadoop, Spark, and Memcached can be significantly improved by leveraging the high-performance networking technologies, such as Remote Direct Memory Access (RDMA). Most of these studies are based on `DRAM+RDMA’ schemes. On the other hand, Non-Volatile Memory (NVM) and NVMe-SSD technologies can support RDMA access with low-latency, high-throughput, and persistence on HPC clusters. NVMs and NVMe-SSDs provide the opportunity to build novel high-performance and QoS-aware communication and I/O subsystems for data-intensive applications. In this talk, we propose new communication and I/O schemes for these data analytics stacks, which are designed with RDMA over NVM and NVMe-SSD. Our studies show that the proposed designs can significantly improve the communication, I/O, and application performance for Big Data analytics and management middleware, such as Hadoop, Spark, Memcached, etc. In addition, we will also discuss how to design QoS-aware schemes in these frameworks with NVMe-SSD.”
Xiaoyi Lu is a Research Scientist in the Department of Computer Science and Engineering at the Ohio State University. He is currently working with Prof. Dhabaleswar K. (DK) Panda in the Network Based Computing Lab. His research interests include High Performance Interconnects and Protocols, Big Data Computing (Hadoop/Spark Ecosystem), Parallel Computing (MPI/PGAS), Virtualization, and Cloud Computing.
