TMLR guidelines for reviewers

This document provides instructions for reviewers. Before reading this section, make sure to also read and be familiar with the Editorial Policies, the Ethics Guidelines, and the Code of Conduct.

Reviewing Responsibilities

As a reviewer, you are expected to hold the following responsibilities.

Double blind. The TMLR review process is meant to be double blind. The identities of the authors of a submission will be kept from the reviewers, and vice versa. You are expected not to take any actions that would violate this double blind state. This includes not actively seeking to find out the identity of the authors of a submission you are reviewing (e.g. by searching for related presentations online or preprints on arXiv). Generally, we suggest you do searches on related work after doing a preliminary read of the submission. If you accidentally discover the identity of the authors, and believe it may influence your judgment, then contact the Action Editor (AE) so they may determine whether you should be unassigned from the submission.

Open Reviewing. As soon as all reviews are submitted for a submission, they will become publicly visible (but anonymous). The authors' response to your review and subsequent discussion will also be immediately visible. You are expected to actively engage with the authors until the issues you have raised for discussion have been sufficiently explored. During this process, you are expected to be respectful and abide by our Code of Conduct.

Annual Quota. Reviews are assigned on a rolling basis, as submissions are received. TMLR will not assign you more than six reviews per year (unless you choose to review additional papers). On the other hand, you are expected to accept all requests for review of submissions that fall within your expertise and quota. Acceptable exceptions are 1) if you have an unsubmitted review for another TMLR submission or 2) situations where exceptional personal circumstances (e.g. vacation, health problems) render you incapable of fully performing your reviewing duties.

Review deadline. TMLR is committed to a reviewing process with fast turnaround. Therefore, reviews must be submitted within 2 weeks of their assignment by the AE. An exception to this expectation can only be made for submissions with more than twelve pages of main content. The subsequent discussion with the authors may vary in length, with the AE determining the point at which they are ready to submit a decision. Two weeks after all reviewers have submitted their review and the discussion with the authors has started (and no later than 1 month), each reviewer will then be allowed to submit a final decision recommendation to the AE.

Conflict of interest. You are responsible for making sure your affiliation and education/career history information is up to date in your OpenReview profile, so we can properly enforce our conflict of interest policy. We consider in conflict any individual or institution with which you have shared a collaboration within the past three years. After being assigned a submission, if you suspect or believe that you have a conflict of interest with any of the authors, consult with the AE for next steps.

Submission Acceptance Standard

The Editorial Policies page provides a brief description of TMLR's acceptance criteria, which we expand further here.

Acceptance of a submission to TMLR should be based on positive answers to the following two questions.

Are the claims made in the submission supported by accurate, convincing and clear evidence?

This is the most important criteria. This implies assessing the technical soundness as well as the clarity of the narrative and arguments presented.

Any gap between claims and evidence should be addressed by the authors. Often, this will lead reviewers to ask the authors to provide more evidence by running more experiments. However, this is not the only way to address such concerns. Another is simply for the authors to adjust (reduce) their claims.

Would some individuals in TMLR's audience be interested in the findings of this paper?

This is arguably the most subjective criteria, and therefore needs to be treated carefully. Generally, a reviewer that is unsure as to whether a submission satisfies this criteria should assume that it does.

Crucially, it should not be used as a reason to reject work that isn't considered “significant” or “impactful” because it isn't achieving a new state-of-the-art on some benchmark. Nor should it form the basis for rejecting work on a method considered not “novel enough”, as novelty of the studied method is not a necessary criteria for acceptance. We explicitly avoid these terms (“significant”, “impactful”, “novel”), and focus instead on the notion of “interest”. If the authors make it clear that there is something to be learned by some researchers in their area from their work, then the criteria of interest is considered satisfied. TMLR instead relies on certifications (such as “Featured” and “Outstanding”) to provide annotations on submissions that pertain to (more speculative) assertions on significance or potential for impact.

Here's an example on how to use the criteria above. A machine learning class report that re-runs the experiments of a published paper has educational value to the students involved. But if it doesn't surface generalizable insights, it is unlikely to be of interest to (even a subset of) the TMLR audience, and so could be rejected based on this criteria. On the other hand, a proper reproducibility report that systematically studies the robustness or generalizability of a published method and lays out actionable lessons for its audience could satisfy this criteria.

Review Format

A review should have the following content.

Summary of contributions Brief description, in the reviewer's words, of the contributions and new knowledge presented by the submission.

Strengths and weaknesses List of the strong aspects of the submission as well as weaker elements (if any) that you think require attention from the authors.

Requested changes List of proposed adjustments to the submission, specifying for each whether they are critical to securing your recommendation for acceptance or would simply strengthen the work in your view.

Broader impact concerns Brief description of any concerns on the ethical implications of the work that would require adding a Broader Impact Statement (if one is not present) or that are not sufficiently addressed in the Broader Impact Statement section (if one is present).

Immediately after submitting your review, the crucial period of discussion with the authors starts . You should engage with the authors by responding to their requests for clarifications, acknowledging the changes they make to their submission and signal to them as soon as you are willing to recommend acceptance.

Official Recommendation

Once two weeks have passed since all reviews have been submitted, reviewers will be able to submit their official recommendations to the AE . Specifically, they will be asked for the following.

Decision recommendation (accept, leaning accept, leaning reject or reject) Whether or not you recommend accepting the submission, based on your initial assessment and the discussion with the authors that followed.

Certification recommendations Certifications are meant to highlight particularly notable accepted submissions. Notably, it is through certifications that we make room for more speculative/editorial judgement on the significance and potential for impact of accepted submissions. Certification selection is the responsibility of the AE, however you are asked to submit your recommendation.

The AE will monitor the discussion and consult the reviewers' feedback and recommendations. When they will be ready to take a decision on the submission, they will also evaluate and rate your review.