Instructions for Reviewers

Introduction:

Thank you for agreeing to review for ACL 2019! With over 2800 submissions and over 2000 members in the program committee, this is a massive effort. We are determined to ensure that all the submissions undergo a fair and thorough review, and have written these instructions to support you in the process. Your reviews will be essential in:

  • providing expert feedback that authors can use to improve their paper (either for the final submission to this conference or for future work).
  • providing opinions to the senior program committee about whether the paper should be accepted for the conference and why

Unlike several recent conferences, we have opted for a simple, streamlined review form (similar to that employed in EMNLP 2018 - please see below) that should make your job easier as it is less time consuming to complete. However, please do ensure to carefully address all the points in the form.

Like NAACL 2019, ACL 2019 will not have an author response period. That means that you should read the papers assigned to you very carefully, and also ensure to be available during the discussion period. When your Area Chair initiates discussion among reviewers, please take an active part in it and share your view. This is important to make sure that the author’s point of view is represented and that any mistakes are caught before being sent to the authors.

The review form:

As with the EMNLP2018 review form, there are three main sections: 1) an in-depth review which is meant for both authors and the program committee, 2) a section for author feedback that includes details of how the authors can improve their work, regardless of whether the paper is accepted for this conference, and 3) a section that includes any confidential comments for the committee.

Download the review form in pdf.

1. In-depth review: this section has 5 parts.

  • The core review. This is the most important part. It should include your view of the main contributions the paper is intended to make and how well it succeeds at making these contributions. From your point of view, what are the significant strong and weak parts of the paper and the work it describes? This could be a 2 paragraph (or longer) essay and/or a bullet list. Remember to describe how the work advances the state of knowledge in computational linguistics and/or highlight why it fails to make a sufficient contribution.
  • Reasons to accept: please briefly summarize from your core review the main reasons why this paper should be accepted for the conference, and how the ACL community would benefit from it. You may refer back to your review to provide more context and details.
  • Reasons to reject: please briefly summarize the possible risks or harm that might come from having this paper published and presented in something close to its current form. What are parts that would need to be improved in order to advance the state of knowledge?
  • Overall recommendation. Here you are asked to synthesize the above and come up with your own recommendation for the paper. We have used the traditional 5 point scale, rather than the 6 point scale to avoid confusion. Like EMNLP 2018, we also allow you to select half points, if you are in between these broad categories or to be able to give more nuance for borderline papers (e.g. 2.5 or 3.5 rather than 3). These numbers are just a concise way of expressing your overall opinion and relative importance of the factors mentioned above. Decisions will be made not just on the scores and certainly not on average scores, but will also take into account the whole review above, reviewer discussion and Area Chair metareviews and recommendations. However it is important to align your recommendation with the reasoning given above, so that authors will be able to understand the motivation for the recommendations and how decision were arrived at.
  • Reviewer confidence. This section should be used to inform the committee and authors how confident you are about your recommendation, taking into account your own expertise and familiarity with this area and the paper’s contents.

2. Additional Feedback for the authors: This is the place to provide the authors with more about what you liked or found problematic in the submission, and how they can improve the paper for the final version (or a future paper). Major issues that are relevant for deciding whether the paper should be accepted should be provided in earlier sections, but other points that would be helpful for the author should go here. Although we will not have an author response period, you can raise questions that you thought should be addressed in the paper here, so the author scan consider how to address these in future writings. There are separate parts for references, typos and smaller points.

3. Confidential information. This section has four parts, and will not be shared with the authors.

  • Recommendation for Presentation Type: if you think the paper would be more naturally presented in an oral or poster form, please indicate that here (or state no preference). Presentation type is not meant to be an indication of the quality of the paper, and limited space and time means that it may not be possible to fulfil all recommendations.
  • Recommendation for Awards: please indicate here if you think this paper should be considered for an award (fewer than 1% of all submitted papers)
  • Justification for Award Recommendation: If you answered yes to B, please indicate why you think this paper deserves consideration for an award.
  • Confidential Comments: if you have something to say just to the committee (e.g. something that might reveal your identity or that of others in the committee, or questions or strong comments that may not be suitable for the author), please indicate that here.

General Guidelines for ACL reviewing:

Please take a balanced approach when reviewing the papers. One objective is to have a solid technical program; hence, it is important to be thorough. On the other hand, we also want a broad and interesting program - so please do not be too picky. ACL should only accept high- quality papers, both long and short, describing a complete piece of work, while also try to keep an open mind. (Please note that a short paper is not a shortened long paper. Instead short papers should have a point that can be made in a few pages and present a focused contribution.) You probably received papers which are not up to your personal standards. Some may still have scientific or technical merit, and could be interesting to others.

We suggest that you consult some of the excellent advice from experienced reviewers and conference organizers on the web. In particular:

Confidentiality:

Reviewers must keep the submitted materials confidential throughout the reviewing and author response processes. In particular, don’t share or talk about the contents of submissions with anyone outside the committee or for any reason other than assessing its suitability to be presented and providing feedback.

Supplementary materials:

Supplementary materials are allowed as a stand-alone document uploaded as a supplementary file. Supplementary materials are supplementary, and you have no obligation to read them. You should treat them like other citations in submissions that may be helpful in understanding background or details beyond the scope of the paper itself.

Secondary Reviewers:

As in most conferences, you may solicit help from others. However, after the secondary reviewer is finished, please rewrite the review in your own words, and adjust the scores accordingly. You should be able to convincingly represent the opinions as YOUR OWN, and not those of secondary reviewer.