SC16 Salt Lake City, UT

Lustre Community BOF: Lustre Deployments for the Next 5 Years


Authors: Mr. Stephen Simms (Indiana University)

BP Abstract: Lustre is the leading open-source file system for HPC. Since 2011 Lustre has transitioned from a single vendor focus to a community developed file system with world-wide contributors. Lustre is now more widely used and in more mission-critical installations than ever. Lustre currently supports many HPC infrastructures scientific research and financial services, oil and gas, advanced manufacturing, and visual effects. At the Lustre BOF, the worldwide community of developers, administrators, and solution providers will gather to discuss recent developments, such as Progressive File Layouts (PFL) and the ability to scale metadata servers (Distributed Namespace), new challenges, and corresponding opportunities.

Long Description: Lustre is the leading open source file system for HPC. Since 2011 Lustre has transitioned from a single vendor focus to a community developed file system with contributors from around the world. As a result, Lustre is now more widely used and in more mission-critical installations than ever. Lustre currently supports many HPC infrastructures. At this year’s Lustre Community BOF the worldwide community of Lustre developers, administrators, and solution providers will gather to discuss new challenges and corresponding opportunities. Used on all the top 10 systems and over 70% of the TOP100 systems on the June 2016 TOP500 list, Lustre is one of the most widely adopted technologies in HPC. From the Tianhe-2 system at the National Supercomputer Center in, China, the Titan system at Oak Ridge National Laboratory, the Sequoia system at Lawrence Livermore National Laboratory, to the K computer at RIKEN Advanced Institute for Computer Science in Kobe, Japan, and more. Lustre is also widely used across mid and small-scale HPC systems with continued adoption attributable to the stability of the Lustre file system as it has matured. Vital to the technology is the community that continues to drive Lustre forward. As Lustre development has evolved into a true open source model, the end-users, developers, and solutions providers have come together through the worldwide OpenSFS community and the European Open File System community. This community development model has resulted in significant new features, improved stability and broader adoption. At the 2016 Lustre community BOF we will focus on the recent feature developments and discuss how they will shape the next 5 years of Lustre deployment around the world. Progressive file layouts (PFL) and the ability to scale metadata servers (Distributed Namespace) are among such game-changing features. From financial services, oil and gas, advanced manufacturing, to basic sciences, Lustre is at the heart of many HPC infrastructures and must continue to evolve to support new use cases. We will explore these use cases and discuss the Lustre roadmap for meeting the requirements they present. The session will include a town hall meeting open to all BOF participants. OpenSFS and the EOFS Cooperative will provide initial discussion topics focusing on a brief set of challenges. The list of challenges will be compiled by the OpenSFS and EOFS by polling the community before the BOF. These challenges will then be posed to our attendants by the BOF session leaders to initiate an open floor discussion for their view of how Lustre needs to evolve to meet these requirements. The Lustre Community BOF has been held at previous SC events and has been well attended and enthusiastically received. SC presents a rare opportunity for the worldwide community of Lustre users, administrators, developers, and solution providers to meet and discuss how to make Lustre more successful. Attendance is usually very good with 100 – 120 attendees. A written report summarizing the Lustre Community BOF will be developed and will be made available to the Lustre community at lustre.opensfs.org

Conference Presentation: pdf


Birds of a Feather Index