SC16 Salt Lake City, UT

DS6. High Performance File System and I/O Middleware Design for Big Data on HPC Clusters

Student: Nusrat Islam (Ohio State University)
Advisor: Dhabaleswar K. Panda (Ohio State University)
Abstract: HDFS is the primary storage engine for Hadoop MapReduce, Spark, and HBase. HDFS, along with these Big Data middleware, is increasingly being used in HPC platforms for scientific applications. Modern HPC clusters are equipped with high performance interconnects (e.g. InfiniBand), heterogeneous storage devices, and parallel file systems. But HDFS cannot fully leverage these resources of modern HPC clusters. In this thesis, an RDMA (Remote Direct Memory Access)-Enhanced HDFS design is proposed that maximizes overlapping among different stages of HDFS operations. Data placement policies for HDFS are also devised to efficiently utilize the heterogeneous storage media, such as RAM Disk, SSD, HDD, and Parallel File System. A key-value store-based burst-buffer system to integrate Hadoop with Lustre has also been presented. Finally, advanced designs to exploit the byte-addressability of (Non-Volatile Memory) NVM for HDFS are proposed. Co-designs with MapReduce, Spark, and HBase offer significant performance benefits for the respective middleware and applications.

Summary: pdf
Presentation: pdf
Poster: pdf

Doctoral Showcase Index