| layout | title | permalink |
|---|---|---|
page |
For Action Editors |
/aes |
An action editor (AE) is a person who oversees a team of reviewers in reviewing the paper, and in the end provides a meta-review summarizing the review process.
Each month, after the submission deadline, action editors who are available that month will be automatically assigned submissions. The action editor will first confirm they have no conflict of interest with each submission. Then, they will quickly check the submission for any major violations in formatting or other factors and desk reject the submission if it is in violation. The action editor will then be presented with a list of reviewers automatically suggested. The action editor will choose reviewers, using this automatic suggestion as a base. This will take place within 7 days of the paper submission deadline.
Each month, starting around 24 days after the submission deadline, the action editor will remind any reviewer who has not yet completed their review.
Each month, between 30 and 35 days after the submission deadline, the action editor will read the reviews, discuss with the reviewers if necessary, and:
- Summarize the review results in a meta-review (see the meta-review form and review form).
- Perform a compliance check, checking the paper meets formatting and other requirements.
You can see more information about the reviewing schedule on the Dates and Venues page.
AEs serve for an initial term of one year, renewable. We will have a centralized system that keeps track of “review balance”, how much reviewing people have done, and tries to distribute it equitably. In addition, some people prefer to handle many papers at a time as opposed to a few every month.
Action editors may indicate that they are unavailable for a period of time. To do so, please complete this form.
Interested in reviewing for ACL Rolling Review?
<iframe width="640px" height= "480px" src= "https://forms.office.com/Pages/ResponsePage.aspx?id=9028kaqAQ0OMdrEjlJf7WXcpizsAwgRHnquEMJ-wjU5UOVAyS0NFOVZLUlZYWEZEM043QVhZS1ZOOS4u&embed=true" frameborder= "0" marginwidth= "0" marginheight= "0" style= "border: none; max-width:100%; max-height:100vh" allowfullscreen webkitallowfullscreen mozallowfullscreen msallowfullscreen> </iframe>We are also recruiting volunteer emergency action editors:
<iframe width="640px" height= "480px" src= "https://forms.office.com/Pages/ResponsePage.aspx?id=9028kaqAQ0OMdrEjlJf7WXcpizsAwgRHnquEMJ-wjU5UMFlINUlXRldFSEZZS0pMTzdCV0tMTEVNSS4u&embed=true" frameborder= "0" marginwidth= "0" marginheight= "0" style= "border: none; max-width:100%; max-height:100vh" allowfullscreen webkitallowfullscreen mozallowfullscreen msallowfullscreen> </iframe>This FAQ is intended to help reviewers and action editors navigate OpenReview and the review process. Please direct inquiries to: support@aclrollingreview.org
You may also find this guide helpful.
Q: What is the normal review timeline?
A:
See the dates and venues page.
The most important deadline is the review deadline, which is the 15th of the month following paper submission, and the meta-review deadline, which is the 20th of the month following paper submission. There is a Google calendar here that you can import into your own calendar to help you keep track.
Q: What is the timeline for emergency reviews?
A:
Sometimes it is necessary to have an emergency review because an originally-assigned reviewer was not able to perform the review.
In this case, we still hope that we can maintain the original timeline (finishing reviews one month after the paper was submitted), but depending on the timing it may not be possible for the reviewer to finish a high-quality review by this deadline.
In this case, please consult with the reviewer and decide a feasible timeline, keeping in mind that the authors of the paper will be eagerly awaiting their review feedback.
Q: What if I'll be unavailable? A: Reviewers may choose to mark themselves as unavailable for some period. To do so, please complete this form. Please do so before the 15th of the first month in which you will be unavailable.
Q: How can I set a maximum load for myself? A: Use this form. Please complete it before the 15th of the month in which you first want this maximum load.
Q: How can I communicate with reviewers?
A:
You may communicate with reviewers using the discussion forum or by sending them email directly (reviewers and AEs are not blind to each-other).
To send a note via the discussion forum in OpenReview, you can use a button labeled "Official Comment" at the top of the paper forum.
Click on it to open a message form. You can select to exclusively message certain sub-groups, e.g. only reviewer 2.
Q: Who contacts reviewers to kick off reviewing?
A:
The editors-in-chief email all reviewers to kick off reviewing.
Q: Who sends reminders to reviewers?
A:
You may remind reviewers by posting in a paper's Discussion Forum.
Q: How do I communicate to the editors in chief?
A:
Email editors@aclrollingreview.org.
Q: What do I do if I'm unavailable in a particular month?
A:
Fill out this form to indicate your unavailability for one or more months.
Q: How do I check and modify reviewer assignments?
A:
At the beginning of the review period, from about the 17th of the month to the 22nd of the month, you will be asked to check and potentially modify the assignments of the reviewers to papers that you are in charge of.
-
Checking Initial Assignments (up to the 22nd): When you get an email notifying you of being in charge of a paper, please log in to check the assignments. At the top of the Area Chairs Console there is an item "Reviewer Browser: Modify Reviewer Assignments". Click the link to access the Reviewer Browser, which shows reviewer suggestions and tracks the status of invitations. Here is how to modify assignments using the "Edge Browser": https://openreview.net/faq#question-edge-browser-AC. It is important to select reviewers with expertise appropriate to the paper. Some suggestions are listed based on an automatic score, but this score may not always be reliable, so feel free to make modifications if they are not reliable. Click on a reviewer's name to see their profile and publications; if they are a good match and do not have too many assignments already, you can unassign one of the currrent reviewers and add a new reviewer. Once you are happy with the assignment (even if you make no modifications), there is nothing more to do. A notification will be sent out on the 23rd of the month notifying your selected reviewers that they have been assigned papers and that they should commence reviewing.
-
Modifying Assignments (23rd and beyond): At some point after reviewers have been assigned you may have to add new reviewers, either because a reviewer contacts you regarding not being a good match for the paper, or because they are unresponsive. If you need to do this, we would suggest you either think of reviewers who you think are appropriate, or use the reviewer browser and sorting by "Affinity Score" to find reviewers who are appropriate. Because the newly assigned reviewer will have a somewhat compressed reviewing timeline (since they were not assigned right at the beginning of the reviewing process), it would be good to reach out to them via email and confirm their availability before assigning them. Here is how to modify assignments using the "Edge Browser": https://openreview.net/faq#question-edge-browser-AC.
Q: How many reviewers are there per paper?
A:
Four reviewers should be assigned per paper at the beginning of the review process.
Each paper must get at least three reviews (four is also acceptable) before the AE writes the meta-review and the reviews are released to the authors.
Q: Does automatic review assignment check if the reviewer is an action editor?
A:
We hope that action editors will also be able to serve as reviewers in some cases, as action editors are often the most experienced members of our community who can provide high-quality reviews on some topics. However, when we assign papers we will attempt to balance your load so the overall load of action-edited and reviewed papers is manageable. That being said, you should never be assigned as reviewer of a paper where you are action editor, so if this happens please contact us to notify us of the issue, then unassign yourself and assign a different reviewer.
Q: In what order are alternative reviewers for a submission displayed (eg alphabetical, by match score, random)?
A:
You can choose several orders. If you choose "affinity score" they will be sorted by the matching score given by an ML model (specifically, as of September this is the SPECTRE-MFR model provided by OpenReview).
Q: Can we see the number of submissions a reviewer is already assigned?
A:
Yes, look for the number next to "assignments" by their name. Please do consider this when making new assignments.
Q: Can we see author information?
A:
No, we are managing a two-way anonymized process up through action editors. It is not possible for you to assign a reviewer who has a COI that is automatically identifiable.
Q: What can I do to make sure I get good papers as AE?
A:
Please fill out your OpenReview profile with (1) a link to your DBLP profile and click "import papers from DBLP", and (2) a link to your Semantic Scholar profile. We import papers from these two sources, so having both filled in will give us the fullest view of your publication record for automatic paper assignment.
Q: Does a reviewer showing up at the top of the list in green mean that they’ve accepted the assignment?
A: Reviewers are assigned, not invited. If a reviewer cannot review your paper, they will contact you via email or via the discussion forum in OpenReview.
Q: What is the compliance checklist?
A:
- Is the content of this paper appropriate to a *ACL venue?
- Is the paper in English?
- Does this paper use the ARR template?
- Does this paper adhere to length constraints for the paper type (long: 8, short: 4, in both cases exclusive of references and appendices)?
- Is the submission (including paper and any supplemental materials) anonymized?
- Is this paper not already published elsewhere?
Q: One of my papers doesn't follow the compliance checklist, what should I do?
A:
The paper should be desk-rejected, please get in touch with the editors in chief.
Q: One of my assigned submissions is more than four pages but less than eight pages in length. What should I do?
A:
Treat it as a long paper submission.
Q: Can the ethics statement extend into a 5th/9th page?
A:
Yes.
Q: How do I actually desk reject a submission?
A:
In OpenReview, you should see a button labeled "Desk Reject".
Q: How do I enter a meta-review?
A:
Click on the "official meta-review" button in the system.
Q: Why can I not press the "meta-review" button?
A:
The meta-review button may not be clickable if not all reviews are submitted yet. In this case, please try to encourage the reviewers to submit their reviews, or recruit a new emergency reviewer.
Q: How do I flag a submission for ethics review?
A:
Please email the editors-in-chief of ARR.
Q: Is the list of ARR action editors published somewhere?
A:
Yes, see here.
Q: Are there multiple action editors per paper?
A: No
Q: I got 4 papers. Should I expect this load in the future?
A:
We aim for the average monthly load in any given month to be 1-2 papers, but in certain periods (close to large conferences), it's possible that the load may be higher, and in slow periods the load may be lower.
Q: Am I expected to accept all invitations? or should I select a subset that I am most interested in?
A:
You are free to decline to review papers, but keep in mind that ARR will be the sole reviewing process for ACL/NAACL in 2022, so a) you won't get separate review requests from conference organizers, and b) the volume should come down in the following months.
Q: Why did I get an invitation to be AE of something later than the normal time?
A:
It is possible that an EICs may have sent an emergency action editor request for a paper where the original AE was not able to handle it or we could not get in contact. For these papers the deadlines for reviews and meta-reviews can be extended flexibly as necessary (although we obviously hope the paper can be done sooner rather than later).
Q: I got a review invitation that I’d like to decline, but when I click the decline button, I get the following error message on Openreview: "Wrong key, please refer back to the recruitment email." What do I have to do to be able to accept/decline invitations? Is this a known problem?"
A:
This is a known issue as the link (as opposed to the link text) uses %-encoding for umlauts (or even any non-ASCII characters). Openreview currently send emails in plain text and they are planning to send them in HTML format with all the links already translated. We hope that will solve the problem.
Q: When I click the accept button in the invitation email to serve as an "action editor", a message returned from the ARR system says "It seems like you already accepted an invitation to serve as a Reviewer for ARR 2021 ............". What can I do?
A: This seems to be a technical problem with OpenReview. We are working with OpenReview to resolve this.