Server-Side Log Data Analytics for I/O Workload Characterization and Coordination on Large Shared Storage Systems
SessionFile Systems and I/O
Session ChairJohn Bent
Authors
Event Type
Paper
Data Analytics
File Systems
I/O
Intermediate
Storage
TimeThursday, November 17th2pm - 2:30pm
Location355-D
DescriptionInter-application I/O contention and performance interference has been recognized as a severe problem. In this work, we demonstrate, through measurement from the world’s No. 2 supercomputer, that high I/O variance co-exists with the fact that individual storage units remain under-utilized for the majority of the time. This motivates us to propose AID, a system that performs automatic application I/O characterization and I/O-aware job scheduling. AID analyzes existing I/O traffic and batch job history logs, without any application-related prior knowledge or user/developer’s involvement. It identifies the small set of I/O-intensive candidates among parallel applications running on a supercomputer, and subsequently mines their I/O patterns, using more detailed per-I/O-node traffic logs. Based on such auto-extracted information, AID provides online I/O-aware scheduling recommendations to steer I/O-intensive applications away from heavy ongoing I/O activities. We evaluate AID on the same supercomputer, using both real applications and our own pseudo-applications.
