This page provides answers to several questions that may arise during the effort to prepare and to submit a paper to SC16. However, we recognize that we cannot possibly cover all questions here. If your specific question is omitted from this list, please feel free to send it to papers@info.supercomputing.org. We will try to answer your question promptly and if appropriate will consider adding it to those addressed here.
Q. What are the main review criteria for acceptance?
A. We will focus on originality, technical soundness, presentation quality, timeliness, impact, and relevance to SC. These criteria will be applied uniformly across the nine topical areas (Algorithms; Applications; Architectures and Networks; Clouds and Distributed Computing; Data Analytics, Visualization and Storage; Performance Measurement, Modeling, and Tools; Programming Systems; State of the Practice; and System Software) to drive the acceptance of contributions that measurably improve upon the state of the art along dimensions that are relevant for SC.
Q. Is it mandatory for authors to select the primary area of their contribution?
A. Yes. Authors must indicate the primary area of contribution from the nine topical areas. We understand that contributions may straddle more than one area and in such cases, we encourage authors to indicate a secondary area of contribution.
Q: Will there be best paper awards?
A. Yes. The best paper (BP) and best student paper (BSP) candidates are selected during the review meeting in June and announced together with the notifications after the meeting. BP and BSP candidates are marked in the conference program. The BP and BSP winners are selected at the conference by an ad-hoc committee and announced at the award ceremony on Thursday.
Q: What constitutes a State-of-the-Practice (SOP) paper?
A: A SOP paper can describe a first-of-its-kind technology or methodology or can capture a unique perspective (or experience) on issues, challenge, and solutions for dealing with aspects of unprecedented scale and complexity, particularly the experiences and knowledge that can be generalized to wide ranges of systems and usages. As stated in the Call for Papers, concrete case studies within a conceptual framework (i.e., experiential topics) would likely serve as the basis for submitted papers, but how the experience generalizes to wider applicability should be explored.
Q: I am concerned that if the standards for SOP are the same as for the regular papers, the SOP papers will be rejected for not being sufficiently academically rigorous. How will you handle this?
A: Although SOP papers will be reviewed under the same rigorous academic peer review process as the papers in other areas, e.g., careful reviews and face-to-face discussions by anonymous reviewers, the acceptance criteria will be tailored to value the new and generalizable insights as gained from experiences with particular HPC machines/operations/applications/benchmarks, overall analysis of the status quo of a particular metric of the entire field or historical reviews of the progress of the field.
Such types of papers are actually common in other academic disciplines, including branches of computer science. For example, software engineering highly values the “experience papers” of particular frameworks and methodologies; human-computer interaction embodies numerous works on analysis of human behavior given a particular interface; social science is about collecting data on social phenomenon and providing meaningful insights based on their statistical analysis.
Q: What is a rebuttal?
A: The rebuttal is for addressing factual errors in the reviews and for answering specific questions posed by reviewers. There will be an opportunity to upload a rebuttal to address factual errors and specific questions in the reviews via the online submission system during a rebuttal period. Then, authors may upload up to 750 words of text in the system before the rebuttal deadline. The rebuttals will be read by the referees and factored into the discussion leading up to the acceptance decisions made at the Technical Papers Committee meeting.
Q: Should I write a rebuttal?
A: The authors of any paper may upload a rebuttal. The choice of whether to submit one and how much time to spend on it is up to the authors of each paper. As a general guideline, submitting a rebuttal is a good idea, if nothing else, to acknowledge the efforts of the reviewers and to indicate how the paper will evolve as a result of their constructive feedback. Rebuttals are also useful to address any errors that the reviews contain or specific questions than can be answered with short textual descriptions.
Q: What should be included in the rebuttal?
A: The rebuttal is for addressing factual errors in the reviews and for answering specific questions posed by reviewers. The rebuttal can also help clarify the merits and novelty of the paper with respect to prior work, if the authors feel that the reviewers misunderstood the paper’s contributions and scope. It is very limited in length, and must be self-contained. It cannot, for instance, contain URLs to external pages.
Q: Now that I’ve read the reviews of my paper, I see much better how to organize it so it will be clear to the reader. Can I do this reorganization and upload the new version during the rebuttal period?
A: No. The rebuttal period is only for addressing factual errors in the reviews, not for getting revised text into the review process. The committee members will have only a short time in which to read and act on your rebuttal, and it must be short and to the point. Hence, it will be limited to 750 words of text. However, you may succinctly state that you will correct grammatical errors or follow suggestions to reorganize the presentation order of your paper.
Q: Between paper submission and the rebuttal period, we’ve gotten some really cool new results for our paper. Can I upload those results during the rebuttal period? I’m sure that they will make the reviewers realize the importance of our approach.
A: No. The rebuttal period is for addressing factual errors in the reviews, not for getting new results into the review process.
Q: What if a reviewer clearly didn’t read my paper carefully enough? What if the reviewer seems to lack basic knowledge of the area on which the paper focuses? How should the rebuttal address these issues?
A: We’ve all received reviews that made us angry, particularly on first reading. The rebuttal period is short and doesn’t allow for the cooling-off period that authors have before they write a response to a journal review. As a result, authors need to be particularly careful to address only factual errors or reviewer questions in the rebuttals rather than letting their emotions show through.
Please don’t say: “If reviewer X had just taken the time to read my paper carefully, he would have realized that our algorithm was rotation invariant.” Instead say: “Unfortunately, Section #4 must not have been as clear as we had hoped because Reviewer X did not understand that our algorithm was rotation invariant and he was therefore skeptical about the general applicability of our approach. This revised version of the second paragraph in Section 4 should clear up this confusion.”
Remember that your rebuttal gets sent to all reviewers; you do not want to offend or to alienate them. In particular, you want the reviewers to come out of the rebuttal process sufficiently enthused about your paper to champion it at the committee meeting, and if the paper is accepted and needs to be revised, then you want them to feel sufficiently comfortable with you as an author that they are willing to “shepherd” the paper through the revision process.
Q: I uploaded a rebuttal, but got no feedback. How can I be sure the reviewers received and actually read my rebuttal?
A: If you can view your rebuttal comments in the online review system, so can your reviewers. Rest assured that rebuttal information is considered and can be very helpful in the selection process.
Q: I understand that SC16 will apply a plagiarism test program to submissions. What constitutes plagiarism? Can I plagiarize my own work?
A: Please see the ACM website for more information on identifying plagiarism: http://www.acm.org/publications/policies/plagiarism_policy
Authors should submit new, original work that represents a significant advance from even their own prior publications.
Q: What are the SC16 guidelines for Conflicts of Interest (COI)?
A: A potential conflict of interest occurs when a person is involved in making a decision that 1) could result in that person, a close associate of that person, or that person’s company or institution receiving significant financial gain, such as a contract or grant, or 2) could result in that person, or a close associate of that person, receiving significant professional recognition, such as an award or the selection of a paper, work, exhibit, or other type of submitted presentation.
Technical Papers Committee members will be given the opportunity to list the potential conflicts during review process. Technical Papers Committee chairs and area chairs will make every effort to avoid assignments that have a potential COI.
For SC16, we consider a conflict of interest to exist with:
- Your Ph.D. advisors, post-doctoral advisors, Ph.D. students, and post-doctoral advisees forever;
- Family relations by blood or marriage, or equivalent (e.g., a partner), forever;
- People with whom you collaborated in the past five years — collaborators include:
- Co-authors on an accepted/rejected/pending research paper;
- Co-PIs on an accepted/pending grant;
- Those who fund your research;
- Researchers whom you fund; or
- Researchers with whom you are actively collaborating;
- People who were employed by, or a student at, your primary institution(s) in the past five years, or people who are active candidates for employment at your primary institution(s); and
- Close personal friends or others with whom you believe a conflict of interest exists.
Note that “service” collaborations, such as writing a DOE, NSF, or DARPA report, or serving on a program committee, do not inherently create a COI.
Other situations can create COIs and you should contact the Technical Papers Chairs for questions or clarification on any of these issues.
Q: What is the process followed by the reproducibility initiative at SC16?
A. Authors of SC16 paper submissions who are interested in participating in the reproducibility initiative at SC17 should select the reproducibility checkbox when submitting their paper in Linklings. If the paper is accepted for the SC16 technical program, the authors will be contacted by the Student Cluster Competition (SCC) Committee and invited to submit a reproducibility proposal to further investigate whether the code(s) and test(s) in their SC16 paper can be adapted to become an application for the SCC at SC17. Authors of selected papers will work with the SCC Committee to transform their code into a suitable application and will be assisted in selecting a proper license. The application will be part of the tests that students at the SCC at SC17 will run in their competition. Authors will be recognized at SC17 with a SIGHPC certificate of appreciation.
A reproducibility proposal includes:
- A copy of the software or a pointer to an archive of the software
- Careful documentation of the process used to produce results
- Any input data set required to initialize, calibrate or guide the simulation
For more information: http://sc16.supercomputing.org/studentssc/student-cluster-competition/call-for-scc-application-proposal/
Q. If my paper is selected for the reproducibility initiative at SC17, what material and commitment will I be required to provide to the SCC Committee?
A. The authors of the associated paper will assist the committee and answer student questions throughout the year regarding their application and paper. In addition, one of the paper authors must agree to serve as the application expert for the competition at SC17 and will help to judge the student teams on the application.
Q. If the SCC Committee is not able to reproduce the work in my SC16 paper, will my paper be removed from the SC16 program and proceedings?
A. No, the lack of reproducibility in preparation for SC17 will NOT rescind the acceptance of the paper for SC16.
Q. How many papers will be considered and finally selected among those with a checked reproducibility box in Linklings?
A. All the papers with a checked reproducibility box in Linklings that submit reproducibility proposals will be considered, but only between one and three papers will be selected to reproduce their work.
Q. What criteria will be use for the selection of the final paper(s)?
A. The paper(s) for the SCC will be selected based on the availability of the code, the commitment of the authors to assist in the preparation and assessment process of the SCC benchmark, and the suitability of the code for the type of clusters built by the student teams.
Important SC16 Information
Web Submissions: https://submissions.supercomputing.org/
Email Contact: papers@info.supercomputing.org
