SC16 Salt Lake City, UT

On-Demand Infrastructure for Data Analytics and Storage


Authors: Dr. Franck Cappello (Argonne National Laboratory)

BP Abstract: Extreme-scale simulations and experiments can generate much more data than can be stored and analyzed at a single site. The scientific community needs significant improvements on data access for complex and accurate analyses. The BOF will provide a forum for groups to discuss approaches, successes, and challenges in distributed computation, storage and analysis. The focus is on developing this community by leading by example, and speakers will report on the SC16 “on-demand infrastructure for scientific data analytics and storage” experiment. The expected outcome is more groups experimenting with their own distributed computing/data projects. A report will summarize the BOF outcomes.

Long Description: Data size and throughput is becoming one of the main limiting factors of extreme-scale simulations and experiments. With current system computational capabilities, extreme-scale scientific simulations and experiments can generate much more data than can be stored at a single site. The scientific community needs significant improvements on data access to perform complex and accurate analyses, while avoiding severe data reduction methods. Also, often a single site cannot simultaneously satisfy both computing and data analytics requirements. Finally, extreme-scale simulations and experiments tend to push toward a model where a group runs simulations that are then analyzed by many other groups. The BOF will provide a forum for groups to discuss approaches, successes, and challenges in distributed computation, storage and analysis. The focus of the BOF is to develop the community of researchers, engineers and practitioners in domain sciences and computer science (infrastructure, networking, transfer, workflow, storage, data integrity, etc.) already active or interested on this topic. The objective is to lead the BOF and the discussion by example. In particular, speakers will report on the SC16 “on-demand infrastructure for scientific data analytics and storage” experiment. The expected outcome is more groups experimenting with their own distributed computing/data projects. The use case behind the SC16 experiment is a 1 trillion particles cosmology simulation, generating 500 snapshots and producing 20PB of data. Users would like to track small features in time. For that, they need to analyze all snapshots. Unfortunately, there is no supercomputing center having enough file system capabilities to store all the data. Such simulation is important for the cosmology community and the results would be shared by several research groups. Other communities have analogous needs. One example is the high energy physics community with the Large Hadron Collider and its dedicated data analytics infrastructure. Another example is the genomics community, which has its own data banks. As a starting point of discussion, speakers will introduce and discuss the concept of “on-demand infrastructure for scientific data analytics and storage” connecting data production sites, data centers for storage, and data analysis centers. This concept relies on technologies needed for the convergence of HPC and big data, such as virtualized resources, resource reservation, software deployment, user group management, policy management, etc. It combines some grid principles as well as some cloud principles: Infrastructure as a Service, elasticity, and virtualization. A team of about 15 researchers and staff from Argonne, UIUC, UIC, DDN, and including SCinet leaders is preparing the experiment. Speakers will report on challenges and findings in terms technology readiness, practicality and open issues. The BOF will engage the audience with a Q&A and discussion period devoted to community building. The objective of the BOF is to bring more people to extend this project, and leverage what was accomplished to develop new projects on “on-demand infrastructure for scientific data analytics and storage”. The expected outcome is more groups experimenting with their own distributed computing/data projects. The BOF will also produce a report summarizing the discussion and outcomes.

Conference Presentation: pdf


Birds of a Feather Index