Reviewer Instructions for Abstracts

Introduction

Thank you for agreeing to review papers for the NLDL Abstract Track, which aims to provide a platform for interesting ideas that may not yet be mature enough for a full paper in the field of machine learning. We value your expertise and commitment to ensuring the quality of the abstracts. In this guideline, we outline key considerations and criteria for evaluating submissions.

Please note that the Abstract Track is designed to give authors a space to discuss ongoing research, share interesting ideas, or present existing work that has been published elsewhere. In the first two cases, we prioritize the correctness and soundness of the proposals over the extensiveness of the results. In the latter case, we suggest to accept published papers in venues that are of interest to the ML community.  In this case, the authors should have disclosed that the paper is already published and where.

Review

The review form requires you to state:

• Summary of the abstract: You should provide a concise description of the abstract.  In a couple sentences, you have to state what the abstract is about, what it is proposing, and the main results of it (if any).  Provide a factual representation of the abstract and the author claims.
  The objective of this summary is two-fold.  One one hand, the authors can see if the submission was understood as intended, and address any factual errors.  On the other, the Area Chair can use this information to obtain a general picture of the submission and its value for a discussion.

• Rating: You should provide a recommendation based on the options below. In general, the correct ideas will be accepted, while the incorrect and out-of-scope ones will be rejected.

  • 2: Correct, definitely interesting ideas for the ML community

  • 1: Incorrect, the ideas are flawed and unsound

  • 0: Out of scope, the ideas are out of scope for the ML community (may or not be correct)

• Justification: You should provide a concise description of why you rate the abstract the way you did.  You should summarize and raise the main points that made you lean towards your rating.  You should justify with summaries the main relevant points, and how they outweigh the others.  For example, if you rated the abstract as "Incorrect", you should raised and summarize the main issues of the abstract and describe how they are flawed.
  The goal of this description is for the Area Chair to understand your rationale behind the evaluation of the paper. 

Post-review Discussion

There is no discussion between the authors and the reviewers for the Abstract Track.  The post-review discussion will happen during the final decision period, and it will be used by the Area Chairs (ACs) to obtain additional information if needed.  The decisions are final.

General Review Criteria

1. Coherence and Correctness: The primary criterion for acceptance is the clarity and correctness of the ideas present in the abstract. Ensure that the ideas are logical and that the presented methodology, experiments, and results (if any) are sound. Note that the Abstract Track encourages the submission of ongoing ideas.  Thus, we expect incomplete work to be submitted, and shouldn't be penalized.

2. Published Works: We encourage the authors of relevant works to the ML community to come and discuss their published works as well.  The Abstract Track will serve as a platform to broaden the discussion and dissemination of these results. Please ensure that the authors of the abstract are a subset the authors of the original published work.

3. Out of Scope Works: The main rejection criteria should be the relevance of the work.  If the abstract is out of scope to the ML community, the abstract should be rejected, despite its correctness.

Reviewer's Summary and Recommendations

In your review, please provide a concise summary of your evaluation.  While not required, constructive feedback is essential to help authors improve their work, and you are encouraged to provide it to the authors.

Please remember that our primary goal is to provide an engaging discussion platform for the Abstract Track. Given the low stakes, we aim to encourage more interesting ideas and err on the side of allowing discussions of potentially flawed ideas rather than limiting them. Your expertise and thorough evaluation are invaluable in achieving this goal.

Thank you for your dedication to advancing the field of machine learning.

FAQ

About the review process

• Q. I found that the submission is not anonymized correctly.  What should I do?
  A. The Abstract Track is single blind.  This means that the identity of the reviewers is concealed but not the authors.  So, it is OK for the paper not to be anonymized.

• Q. Will there be a rebuttal?
  A. No.  There is no rebuttal phase.  Reviews are final.  

• Q. Will there be a discussion phase between the authors and reviewers?
  A. No.  There is no rebuttal phase.  Reviews are final.  

• Q. Since there is no rebuttal, is my job over after submitting the reviews?
  A. Yes, and No. While there is no rebuttal or author discussion, the Area Chair may contact your through OpenReview to ask questions if needed.  So, please keep paying attention to the discussion in case you are needed.

About LLMs

• Q. What is the LLM Policy for reviewers?
  A. Reviewers may use any device, including an LLM, to polish their review wording, but must vouch for, and be responsible for, the accuracy of the review. It is a significant act of reviewer misconduct to allow an LLM to see a submission. PCs interpret showing a submission to an LLM as a deliberate reviewer violation of confidentiality. The PCs reserve the right to report reviewer misconduct to other future machine learning and related conferences. These conferences then may take actions, e.g., there was a recent PAMI-TC vote that CVPR reviewer misconduct may lead to a 2-year submission ban.

• Q. How will the LLM policy be implemented?
  A. An author may complain to their AC that a summary (and/or other parts of the review) have been prepared by an LLM that has seen the paper. Such a complaint would need to be supported by an example summary (or other part of the review) prepared by the author giving the paper to an LLM. If this matches the reviewer’s comments sufficiently, ACs will pass the complaint on to PCs who are then entitled, but not required, to act. Complaints must be submitted on a separate confidential comment. PCs strongly discourage frivolous complaints. Authors should be aware that a complaint to an AC about a review prepared by an LLM without reasonable evidence in support of that complaint, is wasting the ACs time.