Designing MPI Library with On-Demand Paging (ODP) of InfiniBand: Challenges and Benefits
SessionSystems and Networks II
Session ChairKoji Inoue
Event Type
Paper
Advanced
Architectures
Intermediate
Networks
System Software
TimeWednesday, November 16th11am - 11:30am
Location355-E
DescriptionExisting InfiniBand drivers require the communication buffers to be “pinned” in physical memory during communication. Most runtimes leave these buffers “pinned” until the end of the run. Such situation limits the swappable memory space for applications. To address these concerns, Mellanox has recently introduced the On-Demand Paging (ODP) feature for InfiniBand. With ODP, communication buffers are paged in when they are needed by the HCA and paged out when the OS needs to swap them. This paper presents a thorough analysis of ODP and studies its performance characteristics. With these studies, we propose novel designs of ODP-aware MPI communication protocols. To the best of our knowledge, this is the first work to study and analyze the ODP feature and design an ODP-aware MPI library. Performance evaluations with applications show that ODP-aware designs can reduce the size of pin-down buffers by 11X without performance degradation compared with the pin-down scheme.
