Title: SIGCOMM reviewing Q&A
Host: Yiting Xia (Max Planck Institute for Informatics)
Panelists: Minlan Yu (Harvard University), Vyas Sekar (Carnegie Mellon University), Philip Brighten Godfrey (University of Illinois Urbana-Champaign), Sujata Banerjee (VMware Research Group), Ratul Mahajan (University of Washington)
Scribe: Xing Fang (Xiamen University), Jinghui JIang (Xiamen University)
Introduction
Effective paper reviewing is crucial for the SIGCOMM community’s growth. To enhance transparency and inclusivity, this session will have TPC chairs engage with the community to understand authors’ needs and provide insights into the review process. This will empower participants with confidence, foster open dialogue, and promote a culture of transparency and collaboration, ensuring the vitality of SIGCOMM research.
The Review Process:
Before the formal exchange began, in order to let the participants better understand the review process of SIGCOMM, TPC first gave a brief introduction to the review process.
Before the Reviews
Before the review officially begins, there are some preparations to be completed. It is necessary to invite enough reviewers to ensure full coverage of expertise areas and ensure that each paper can be assigned to a sufficient number of reviewers. In addition, in the preparation stage, TPC also needs to handle the conflict between reviewers and papers.
Two rounds of reviews
The review process of SIGCOMM has two rounds of reviews. In the first round, each paper is assigned to three reviewers in related fields according to the review preferences. According to the review opinions of the first round, papers that receive two positive comments or borderlines will enter the second round. Depending on the actual situation, articles that do not enter the second round may sometimes receive early rejection notices.
In the second round of review, one or two reviewers will be assigned to the paper based on the number of reviewers in the first round to ensure that the paper has enough reviewers. Sometimes, some papers will invite external experts to review.
Online discussion and Pc meeting
In the online discussion, all reviewers can read other people’s reviews. At this stage, Reviewers discuss what they think about the paper (accept, pc-discuss, or reject) and the reasons.
Some papers are pre-accepted before the meeting. At the PC meeting stage, the discussion order is based on reviewer availability, topics, scores, and randomness. Every reviewer chimes in their views of the paper and ultimately decides whether the paper is accepted or rejected.
Final steps
In the final step, all papers will be assigned shepherds. Shepherds will assist authors in revising their papers. Assignment of shepherds is based on pc discussion, expertise, and shepherding loads. Unlike previous years, this year, SIGCOMM published reviews for most of the accepted papers.
Questions
Q1: Should there be a standardized process for early notification of rejections, or should it be left to the discretion of the PC chairs?
Vyas : Emphasizes both the advantages and disadvantages of early notifications, noting that while they provide quick feedback and reduce waiting time for authors, they also add logistical challenges and risk premature rejections.
Minlan : Stresses the importance of making the right decision over providing fast feedback, emphasizing that every paper deserves careful consideration. Rushing the review process, especially with late reviewer feedback, can be detrimental, and it is crucial to prioritize thorough evaluations
Brighten: Believes that faster answers are generally beneficial but acknowledges the difficulty in standardizing early notifications due to varying review processes. The community needs to be flexible and understanding of the timing and inherent imperfections in the process.
Ratul : Opposes early notifications, arguing that they lead to an increase in hastily resubmitted, unpolished papers. A thorough review process is essential to ensure that only well-prepared papers are accepted, and waiting for comprehensive feedback results in more robust submissions.
Sujata : Recognizing the benefits of early notifications, such as timely feedback, emphasizes the need for a disciplined review process to ensure it is implemented effectively. The autonomy given to PC chairs allows for innovative and tailored approaches, balancing speed and quality based on community input and conference needs.
Q2: What kind of flexibility do PC chairs have in the review process?
Sujata: PC chairs have significant flexibility, allowing them to adapt procedures based on community input and evolving needs. While certain standards are followed annually, chairs can implement new ideas and adjust processes. However, maintaining timely reviews requires discipline, which can be challenging to guarantee.
Q3: What are the implications and future directions for open reviews?
Vyas: Open reviews lack a standard definition, ranging from full public visibility to controlled transparency. It’s crucial to maintain high-quality reviews through qualified program committees. A balance between public transparency and quality control is necessary, with caution against potential pitfalls like low-quality public feedback
Minlan: The current experiment involves a significant number of authors participating in public reviews. The next steps will be based on community feedback regarding the usefulness of this transparency. The aim is to focus on technical aspects of reviews, avoiding personal opinions, and ensuring the transparency initiative benefits the review process without causing harm.
Ratul: The main goal of open reviews is to increase the program committee’s accountability by clearly explaining the reasons behind paper acceptance. Different levels of transparency can be implemented, with a focus on scrutinizing the review process and providing justifications for decisions. Future improvements will depend on the feedback and response from the community.
Q4: Does posting manuscripts on arXiv before submitting them to a conference affect the review process?
Brighten: Posting manuscripts on arXiv before conference submission is generally fine and beneficial, as it allows authors to share their work and get feedback. It is unreasonable to expect authors to refrain from discussing their work for extended periods. While reviewers should not actively search for author identities, if they accidentally discover it, they should inform the chairs to handle it appropriately.
Q5: How are papers discussed and decided upon in PC meetings?
Sujata : PC chairs are responsible for ensuring that papers are reviewed appropriately, but they cannot read all the papers themselves. They might read some papers requiring more scrutiny due to disagreements among reviewers. In the PC meeting, PC chairs ensure high review quality and defend ratings based on reviewer comments
Vyas: PC chairs are aware of debatable papers and may selectively read them. They help highlight key points during discussions and encourage convergence on decisions by facilitating dialogue among reviewers.
Brighten: In PC meetings, reviewers of the paper, typically 5-6 people, discuss their views. Champions of the paper present positive aspects, while detractors highlight fundamental problems. The PC chair and committee weigh the pros and cons. Strong champions can sway decisions, and in cases of significant controversy, additional expert opinions may be sought. Generally, the PC meeting helps converge decisions through balanced discussions.
Minlan: The discussion is mainly led by reviewers of the paper. Other PC members, who haven’t read the paper, should not lobby or give opinions without context. The focus is on ensuring informed discussions based on detailed reviews.
Ratul: PC chairs influence decisions by setting and reinforcing discussion ground rules. The number of papers brought up for discussion affects the dynamics, as knowing acceptance rates can change the nature of discussions. Effective PC chairs can significantly impact the outcomes by managing these factors.
Q6: What are the expectations and rules around shepherding and the resubmission process?
Minlan: The advisory resubmission process is designed to keep requirements minimal, focusing on essential changes that could shift a paper from rejection to acceptance. This might include one additional experiment to demonstrate a system’s effectiveness. Different conferences may implement this procedure differently.
Brighten : Shepherding should be specific and focused, addressing only the key issues necessary to make the paper acceptable. The process should involve clear, narrow tasks rather than extensive revisions.
Q7: How much time do PC members use, and what do they do?
Vyas : Each paper typically gets at least three reviews in the first round. PC members usually handle about 25 papers each, with review rounds taking around 1.5 months. This workload distribution ensures thorough evaluations and efficient time management.
Q8:How are papers with divided reviews (e.g., one positive, one negative) handled?
Ratul: Divided opinions are common and beneficial as they lead to thorough discussions. Reviewers make independent judgments and then engage in debate, educating each other and reaching a consensus. Different perspectives, including technical disagreements, are addressed during these discussions.
Q9: How should students approach submitting papers to avoid divided opinions?
Minlan : Divided opinions are beneficial for fostering thorough discussions. Students should focus on producing solid work and not worry about avoiding differing opinions. Serious technical flaws should be addressed, while differences in personal taste lead to healthy debates, enhancing the paper’s evaluation.
Vyas : Students should have two objectives: produce strong, well-researched work and embrace diverse opinions. Having a strong voice and inviting constructive feedback is more important than trying to avoid extreme scores. Good work will naturally improve the chances of acceptance.
Q10: Is there any difference in focus between academic papers and industry papers?
Sujata : There are differences in reviewing criteria for research track papers and experience track papers. Experience track papers often come from industry and focus less on novelty and more on practical experiences, such as scaling services or validating community myths. Both types of papers have different criteria and are reviewed accordingly, with experience papers emphasizing real-world application and outcomes.
Q11: Should we only accept very well-done, solid papers, or how do we promote risky and relatively immature work?
Brighten : Advocates for accepting more risky work and shifting review culture to focus on a paper’s contributions rather than its shortcomings. Suggests that this approach would help support innovative research that may have more unanswered questions.
Vyas : Notes that PC chairs can influence the acceptance of risky papers by selecting program committee members and guiding discussions. Encourages examining public reviews of accepted papers to understand how risky work is evaluated and accepted, and to consider this when championing such papers.
Ratul : Proposes that the solution to accepting imperfect but promising papers is to increase the number of accepted papers overall. Believes that accepting technically correct papers, even with some imperfections, allows for a broader range of research, including riskier work.
Minlan : Emphasizes the importance of PC members advocating for risky papers they find valuable. Appreciates reviewers who champion such papers, as this can lead to the acceptance of more diverse and innovative work. Encourages a culture where risky ideas are given a chance.
Sujata : Recalls that there used to be a category for vision papers, which projected future directions without requiring experimental results. Suggests reinstating this category to encourage submission of innovative but unproven ideas.
Personal thoughts
It is interesting that Ratul mentioned how the acceptance rate of a conference can significantly influence the Program Committee’s (PC) decisions. When the acceptance rate target is known, it can affect the stringency of reviews, potentially shifting focus from the paper’s merit to meeting quotas. Adopting a flexible acceptance rate based on submission quality each year could help ensure deserving research is recognized.
Brighten’s point about focusing on a paper’s strengths rather than its shortcomings is also insightful. This approach fosters a more positive review culture and encourages innovation. However, it might overlook significant flaws, leading to the acceptance of less robust papers. Balancing strengths with a critique of weaknesses is essential for maintaining research quality.