Burst Buffers: Early Experiences and Outlook

Authors: Dr. Deborah Bard (Lawrence Berkeley National Laboratory)

BP Abstract: The long-awaited Burst Buffer technology is now being deployed on major supercomputing systems, including new machines at NERSC, LANL, ANL, and KAUST. In this BOF, we discuss early experience with Burst Buffers from both a systems and a user’s perspective, including challenges faced and perspectives for future development. Short presentations from early adopters will be followed by general discussion with the audience. We hope that this BOF will attract interest and participation from end-users and software/hardware developers.

Long Description: The goal of this BOF is to bring together those in the supercomputing community with an interest in Burst Buffer technology. Burst Buffers are a layer of NVRAM-based storage that have been long proposed as a solution to the problem of bursty I/O in busy HPC systems. This year has seen the first deployments of a Burst Buffer storage layer in supercomputer systems, and this will be the first BOF dedicated to discussion of the technology. SC16 is a very timely venue to discuss the first real-life experiences with this technology, and to understand the issues that have arisen in the diverse applications for which Burst Buffers are being used. With the Burst Buffer concept in its infancy, this is also an important opportunity to influence future development and deployment plans. Many HPC systems are considering deploying Burst Buffers, and this topic is expected to be of general interest to the wider HPC community in both scientific computing and industry, as evidenced by a slew of recent papers on Burst Buffer concepts and testing, and the award of the 2016 Cray User Group Best Paper to NERSC’s Burst Buffer paper. Experts from the DoE National Lab computing centers at NERSC, LANL and ANL, and from the supercomputing center at KAUST, will briefly present their experiences with their early deployment of Burst Buffers from the point of view of both systems specialists and users. This will be followed by general vendor-agnostic discussion with the audience on the following topics: How users actually use a Burst Buffer How Burst Buffers fit into a future storage hierarchy Pros and cons of NVRAM-based storage in HPC systems Will flash storage replace spinning disk in HPC systems ? The future of custom Burst Buffer software support - for example, on-demand filesystems, caching etc. This discussion will be complemented by a live interactive survey, polling the audience on the following questions: What motivated you to use a Burst Buffer (or would motivate you in the future)? What applications have been run on the Burst Buffer? What size jobs have run using a Burst Buffer - in terms of compute nodes and Burst Buffer nodes? What issues have been faced in using the Burst Buffer technology? The outcome of this BOF will be a document that contains a summary of the survey results and a summary of early experiences using Burst Buffers. We will also create a Burst Buffer User Group and mailing list which is open to all (in particular welcoming users beyond the major HPC sites), and a corresponding website where Burst Buffer enthusiasts can share experiences, best practices and example use cases of using the Burst Buffer. This will also serve as a place to gather requirements for new systems

Birds of a Feather Index