This blog post is about the current scores of submissions to the Papers Track for SIGCHI 2021 after the completion of the first round of reviews.It is patterned almost directly after a similar post made for the 2019 conference with some small updates for current figures (Our thanks to those authors) (https://chi2019.acm.org/2018/11/17/chi-2019-reviews-and-rebuttals/).
You will have recently received your first indications of how your submissions might fare in December’s program committee (PC) meeting. Most papers will have received four reviews. Two reviews have been provided by external reviewers. One of the PC members, the paper’s ‘2AC’ has also provided a review of your submission. The final review is a meta review provided by the paper’s ‘1AC’, another member of the committee.
There were 2844 complete submissions, a decrease of 281 submissions or down ~10% from CHI2020. Of these 2844 submissions, 3 were withdrawn, 106 were Quick Rejected (QR), 42 were Desk Rejected (DR) leaving 2693 papers or 94%. 34 DRs were for anonymization reasons, 3 for plagiarism, 2 for scope inappropriate for CHI, and the other 3 based on extensive portions of the paper written in a non-standard (English) language. Please see the post on QR and DR to find more details about the criteria for the Early Rejection process.
The lowest score a reviewer can assign a submission is one, and five is the highest. Reviewers also self-report their expertise as it relates to a given submission. This is rated on a scale from one (least knowledgeable) to four (most knowledgeable). Together, these scores give you an idea of how your paper has fared.
Each submission has an ‘Overall Score’, which is the mean of the scores given by the ACs and reviewers of a submission. The mean of these overall scores is currently 2.556 (SD=0.67). If your paper has an overall score of 3.6 or better, it is currently in the top 10% of submissions by score. Only 81 papers scored a mean of 4.0 or better (3%). The distribution of scores is illustrated in Figure 1. Overall, scores are on par with 2018 and 2020, with this year’s mean of 2.556 being almost exactly the same as the mean in 2018 and 2020 and slightly better than in 2019 (2.48).
In other words, overall reviewers lean toward rejecting most papers, with around a quarter of papers receiving a neutral or positive response from reviewers. Experienced submitters to CHI will recognise the distribution of scores. It is quite similar each year. If this is your first time submitting to CHI, then the score distribution should help you contextualize the ratings your submissions have received. The main takeaway is that a little more than half (1511, 56.1%) of all submissions score ≥2.0 and <3.0.
|Year||Count of papers||After QR/DR/W||Mean score||Score for 25.7% accept||Score for 90% rank|
A huge amount of effort goes into reviews. The latest data we have shows there are 11,240 reviews in the system. These reviews comprise 41,991,192 characters or about 8,398,238 words. The longest was about 5000 words. The shortest was about 30 words. There were about 2500 reviews of over 1000 words, and the mean length was 747 words (SD=462). The distribution of review length is illustrated in Figure 2.
We also had a look at whether there was a relationship between overall score (i.e., mean score) and mean review length. (We use means because an individual only has nine score choices and the plot is a little dull, then.) We plot this relationship in Figure 3. Note that submissions with very high scores and submissions with very low scores tend to receive the shortest reviews, on average.
Finally, some comments on page length data since for the first year, there was no page limit enforced. Authors were encouraged to submit papers of a length commensurate with their contributions.
|Year||Count of papers||Mean Word count||Median Word count||STDDEV|
Table 2: Mean and Median Word counts between 2020 and 2021 (an increase in 12.8% for the mean and 9.5% for Median).
This is the raw comparison, but there were 6 papers (out of 2844 that were above 16000 words). Of these, 4 are in Understanding People, 1 is in Assets, and 1 is in Games.
Excluding those, we have:
Overall, there were 150 papers in 2021 that were greater than the maximum length of papers in 2021 (about 5% of the submissions).
People often wonder whether it is ‘worth’ submitting a rebuttal at or below a certain score. In previous years, the overall mean for scores does not move much after rebuttals, but this does not mean that individual scores do not move. They can and they do. If the ACs for one of your submissions have encouraged you to write a rebuttal then it might make sense to do so. If nothing else, a rebuttal gives you the chance to consider the perceived limitations of your submission so that you can fix them, whether for CHI 2021 or another venue.
A number of members of our community have provided views on how to write a rebuttal. Here are a few that were suggested in last year’s blog:
- Writing rebuttals by Niklas Elmqvist, University of Maryland, College Park
- Writing CHI Rebuttals by Gene Golovchinsky
- SIGCHI Rebuttals: Some Suggestions How to Write Them by Albrecht Schmidt
- A CHI Rebuttal by Simone O’Callaghan
- How to Write SIGCHI Rebuttals by Hyunyoung Song
This list isn’t exhaustive.