An Ephemeral Burst-Buffer File System for Scientific Applications
SessionFile Systems and I/O
Session ChairJohn Bent
Event Type
Paper
Data Analytics
File Systems
I/O
Intermediate
Storage
TimeThursday, November 17th1:30pm - 2pm
Location355-D
DescriptionBurst buffers are becoming an indispensable hardware resource on
large-scale supercomputers to buffer the bursty I/Os from scientific
applications. However, there is a lack of software solutions for burst buffers
to be efficiently shared by applications within a
batch-submitted job and recycled across different batch jobs.
In addition, burst buffers need to cope with a variety of
challenging I/O patterns from data-intensive scientific applications.
In this study, we have designed an ephemeral Burst-buffer based File System (BurstFS) that
supports scalable and efficient aggregation of I/O bandwidth from burst buffers
while having the same life cycle as a batch-submitted job. BurstFS features
several techniques including scalable metadata indexing, I/O delegation,
and service-side read clustering and pipelining.
Through extensive tuning and analysis, we have validated
that BurstFS has accomplished our design objectives, with linear scalability
in terms of aggregated I/O bandwidth for parallel writes and reads.
large-scale supercomputers to buffer the bursty I/Os from scientific
applications. However, there is a lack of software solutions for burst buffers
to be efficiently shared by applications within a
batch-submitted job and recycled across different batch jobs.
In addition, burst buffers need to cope with a variety of
challenging I/O patterns from data-intensive scientific applications.
In this study, we have designed an ephemeral Burst-buffer based File System (BurstFS) that
supports scalable and efficient aggregation of I/O bandwidth from burst buffers
while having the same life cycle as a batch-submitted job. BurstFS features
several techniques including scalable metadata indexing, I/O delegation,
and service-side read clustering and pipelining.
Through extensive tuning and analysis, we have validated
that BurstFS has accomplished our design objectives, with linear scalability
in terms of aggregated I/O bandwidth for parallel writes and reads.
