This post is meant to clarify what Early Reject entails and define the differences between ‘Desk Reject’ (DR) and ‘Quick Reject’ (QR) and what it means for the CHI reviewing process and the community at large.
Early Reject means that a paper that has been submitted to the CHI conference will not be sent to external reviewers and will be judged internally by the committee. The process is designed to reduce the huge burden on external reviewers in the community. CHI currently receives around 4,000 papers. Each paper requires four reviews to be written: two internal by the committee (the Primary Associate Chair (1AC) meta review and the Secondary Associate Chair (2AC) review) and two by external reviewers who are recruited by committee members. This produces the need for around 16,000 reviews - a number increasing by approximately 14% each year. This is unsustainable and in parallel this year we have seen large increases in the numbers of papers submitted to other SIGCHI sponsored conferences including UIST and CSCW.
The Early reject process is still somewhat controversial - and needs to be fair and transparent. We detail here both the criteria and process for what it takes for a paper to be early rejected.
Desk Rejects (DR’s)
First; easier to understand, and less controversial, are those papers that will never have a chance of getting into the conference no matter how many external reviews they get. These can be determined mostly mechanically and fairly quickly.
Here are some of the criteria by which a paper will be desk rejected:
- Incomplete submissions
- Not written in English
- Obviously not a conference paper (patent disclosure, popular press article, a complete book, etc.)
- Clearly out of scope for the conference (e.g. formal methods for interstellar microcontrollers)
- The paper lacks anonymization: either leaving the author names in the paper or having an acknowledgements section that reveals authors or the institution (e.g. specific supporting grant information); if the acknowledgements thank a particular person for helping them with their paper, or thank their participants, that should be okay in most cases.
- Something is so broken in the paper that makes it impossible to review.
- Notice: There is NO STRICT paper length for CHI2021, so paper length is not, by itself, grounds for desk rejection.
- Papers SHOULD be submitted using the standard submission templates which are now single-column. Papers will NOT be desk-rejected THIS year if a paper is submitted using an improper style, but may be rejected in the future.
Here is the process for a Desk Rejection:
Once the Primary Associate Chair (1AC) determines that a paper is a candidate for desk rejection, they will confer with the Secondary AC (2AC) for the paper. Both will communicate with the two Subcommittee Chairs (SCs) with their proposal (considering all conflict of interests from the SCs). If ALL four members of the committee (ACs and SCs) agree that the paper warrants a desk rejection, a SHORT (a couple of lines) review with the justification for the desk rejection will be written by the 1AC in the Precision Conference System and the notification will be sent to the authors by the early rejection notification day.
Quick Rejects (QR’s)
The purpose of the quick reject process (QR) is the same as with the desk reject process (DR) - to save the load on the community, but these are NOT as easily determined by a few straightforward criteria, and thus care needs to be taken for a full justification of why this paper has not been reviewed by external reviewers. The burden on the ACs is particularly strong since they need to enter a complete and compelling review and the paper is still evaluated by both ACs and both SCs (conflicts permitting).
Such papers need to be within the qualification of the ACs assigned to them, and agreement between the ACs and SCs must be reached that they can not see the paper being accepted under any circumstances.
The criteria for a QR:
- HCI contribution is much too small given the length of the submitted paper (we expect that papers would on average be between 4-12 pages in the old two column format and 6 to 18 pages in the new one column format). This criterion includes papers that are far too long, as if someone submitted a book manuscript. Since there is no strict paper length, this is a subjective criterion, however if authors attempt to submit papers that are clearly inappropriate for a conference submission (extended book chapters, unedited reports), then a paper may be quick rejected to prevent the undue burden on external reviewers.
- Grossly insufficient detail to replicate the apparatus or the experiment
- Grossly insufficient data to validate the analysis to support the claim
- Grossly insufficient literature review to contextualize and/or evaluate the proposed novelty/contribution to HCI in particular
- Paper is very sloppy: lots of typos, missing references, formatting issues (including large white spaces)
The quick reject process entails considerable work for both ACs and SCs. Each paper will still have 4 distinct people that will have looked at any paper that is a potential QR, and all must agree that the paper would not have a chance of acceptance into the conference. (This is not an arbitrary decision that can be made by a single AC). The 1AC must first confer with the 2AC; come to an agreement and convince both SCs that the paper should be quick rejected. The paper will not require 2 external reviewers and thus decrease the overall load on the community. At this point, the 1AC must write up a sufficient and constructive review detailing why this paper is being quick rejected which must be acknowledged by the 2AC. This review will be checked by both SCs and the authors will be notified of the quick reject status on notification day for early rejects.
CHI is not alone in adopting a quick reject process to help manage the increasing review burden while maintaining fairness in the process adopted.
Papers Chairs
Pernille Bjørn, University of Copenhagen
Steven Drucker, Microsoft Research