Replication and reproducibility of experimental computer science results in peer-reviewed paper is gaining relevance in the HPC community. SC, the leading conference in the field, wants to promote and support replication and reproducibility through a new initiative that aims to integrate aspects of past technical papers into the Student Cluster Competition (SCC). The SCC is excited to announce “A parallel connectivity algorithm for de Bruijn graphs in metagenomic applications” as the winning paper for the inaugural reproducibility initiative. This paper and accompanying application will be reproduced in the SCC at SC16. This is the first time that students have been challenged to reproduce a paper rather than run prescribed data sets. Although they are doing similar tasks from previous competitions, they are seeing it from an entirely new perspective, as a component to the scientific process. “We want students to understand, early in their careers, the important role reproducibility plays in research.” explains the SCC Chair Stephen Harrell
The students will be undertaking replication which is a first step towards reproducibility. “Replication is a pre-requisite for the success of any project that intends to build on the results of a published paper. […] Without the assurance that comes from replication, there is a high risk that you will make false assumptions about the results in a published paper, leading to incorrect comparisons or failure of your own experiments that attempt to build on those published results.” says Mike Heroux, Senior Scientist at Sandia National Laboratories and author of the Transactions on Mathematical Software Journal’s Replicated Computational Results Initiative.
The importance of reproducibility is underscored by the changing scholarly records that more frequently include data, code and software. These changes support reuse of data and software for new scientific discovery. The SC community must develop best practices to link data and software to scholarly records. This initiative taken with the SCC is a first step toward developing a set of best practices.
We would like to congratulate the author of the reproducibility proposal Chirag Jain. Chirag will be helping the SCC committee to create a challenging competition task for the students and judge the students’ work. He will also receive a certificate of recognition from SIGHPC for his work on this initiative.
The original paper is available from the ACM Digital Library at http://dx.doi.org/10.1145/2807591.2807619