87. A Cross-Layer Solution in Scientific Workflow System for Tackling Data Movement Challenge
Event Type
Poster
LocationLower Lobby Concourse
DescriptionScientific applications running in HPC environments are more complex and more data-intensive nowadays. Workflow systems are typically used to manage such complexity. Traditionally, scientific workflow systems work with parallel file systems. As such, the data need to be transferred between compute nodes and storage systems, which introduces a significant performance bottleneck on I/O operations. One promising solution to tackle this challenge is to exploit the data locality in HPC storage hierarchy. Several recent studies have been done regarding building a shared storage system, utilizing compute node resources, to serve HPC workflows with locality, such as Hercules and WOSS etc. However, in this research, we argue that providing a compute-node side storage system is not sufficient to fully exploit data locality. A cross-layer solution together with storage, compiler, and runtime is necessary. We take Swift/T, a workflow system for data-intensive applications, as a prototype platform to demonstrate our solution.
Archive