SC16 Salt Lake City, UT

87. A Cross-Layer Solution in Scientific Workflow System for Tackling Data Movement Challenge

Authors: Dong Dai (Texas Tech University)Robert Ross (Argonne National Laboratory)Dounia Khaldi (Stony Brook University)Yonghong Yan (Oakland University)Dorier Matthieu (Argonne National Laboratory)Neda Tavakoli (Texas Tech University)Yong Chen (Texas Tech University)

Abstract: Scientific applications running in HPC environments are more complex and more data-intensive nowadays. Workflow systems are typically used to manage such complexity. Traditionally, scientific workflow systems work with parallel file systems. As such, the data need to be transferred between compute nodes and storage systems, which introduces a significant performance bottleneck on I/O operations. One promising solution to tackle this challenge is to exploit the data locality in HPC storage hierarchy. Several recent studies have been done regarding building a shared storage system, utilizing compute node resources, to serve HPC workflows with locality, such as Hercules and WOSS etc. However, in this research, we argue that providing a compute-node side storage system is not sufficient to fully exploit data locality. A cross-layer solution together with storage, compiler, and runtime is necessary. We take Swift/T, a workflow system for data-intensive applications, as a prototype platform to demonstrate our solution.

Poster: pdf
Two-page extended abstract: pdf

Poster Index