Some Stats about CHI 2021 Papers

  • November 30, 2020

This blog post is about the current scores of submissions to the Papers Track for SIGCHI 2021 after the completion of the first round of reviews.It is patterned almost directly after a similar post made for the 2019 conference with some small updates for current figures (Our thanks to those authors) (https://chi2019.acm.org/2018/11/17/chi-2019-reviews-and-rebuttals/).

 

You will have recently received your first indications of how your submissions might fare in December’s program committee (PC) meeting. Most papers will have received four reviews. Two reviews have been provided by external reviewers. One of the PC members, the paper’s ‘2AC’ has also provided a review of your submission. The final review is a meta review provided by the paper’s ‘1AC’, another member of the committee.

 

There were 2844 complete submissions, a decrease of 281 submissions or down ~10% from CHI2020. Of these 2844 submissions, 3 were withdrawn, 106 were Quick Rejected (QR), 42 were Desk Rejected (DR) leaving 2693 papers or 94%. 34 DRs were for anonymization reasons,  3 for plagiarism, 2 for scope inappropriate for CHI, and the other 3 based on extensive portions of the paper written in a non-standard (English) language.  Please see the post on QR and DR to find more details about the criteria for the Early Rejection process.

 

The lowest score a reviewer can assign a submission is one, and five is the highest. Reviewers also self-report their expertise as it relates to a given submission. This is rated on a scale from one (least knowledgeable) to four (most knowledgeable). Together, these scores give you an idea of how your paper has fared.

 

Each submission has an ‘Overall Score’, which is the mean of the scores given by the ACs and reviewers of a submission. The mean of these overall scores is currently 2.556 (SD=0.67). If your paper has an overall score of 3.6 or better, it is currently in the top 10% of submissions by score. Only 81 papers scored a mean of 4.0 or better (3%). The distribution of scores is illustrated in Figure 1. Overall, scores are on par with 2018 and 2020, with this year’s mean of 2.556 being almost exactly the same as the mean in 2018 and 2020 and slightly better than in 2019 (2.48). 

 

In other words, overall reviewers lean toward rejecting most papers, with around a quarter of papers receiving a neutral or positive response from reviewers. Experienced submitters to CHI will recognise the distribution of scores. It is quite similar each year. If this is your first time submitting to CHI, then the score distribution should help you contextualize the ratings your submissions have received. The main takeaway is that a little more than half (1511, 56.1%) of all submissions score ≥2.0 and <3.0.

 

Overall Scores:

Year Count of papers After QR/DR/W Mean score Score for 25.7% accept Score for 90% rank
2021 2844 2693 2.556 3.0 3.6

 

 

Figure 1: Histogram of mean scores for the submissions to the Papers track. Marked is the current mean score, 2.556; the score needed to be in the top 25.7% (acceptance rate for CHI2019).

 

 

A huge amount of effort goes into reviews. The latest data we have shows there are 11,240 reviews in the system. These reviews comprise 41,991,192 characters or about 8,398,238 words. The longest was about 5000 words. The shortest was about 30 words. There were about 2500 reviews of over 1000 words, and the mean length was 747 words (SD=462). The distribution of review length is illustrated in Figure 2.

 

Figure 2: Histogram of mean review lengths (in characters) for submissions to the Papers Track. The longest review was about 5000 words while the shortest 30.

 

 

We also had a look at whether there was a relationship between overall score (i.e., mean score) and mean review length. (We use means because an individual only has nine score choices and the plot is a little dull, then.) We plot this relationship in Figure 3. Note that submissions with very high scores and submissions with very low scores tend to receive the shortest reviews, on average.

 

 

Figure 3. Mean character count in reviews against mean score for all submissions. Note that reviews around the mean submission score receive the longest reviews.

 

 

Finally, some comments on page length data since for the first year, there was no page limit enforced. Authors were encouraged to submit papers of a length commensurate with their contributions. 

 

 

Overall Scores:

Year Count of papers Mean Word count Median Word count STDDEV
2020 3126 6600 7081 2046
2021 2844 7450 7755 2816

Table 2: Mean and Median Word counts between 2020 and 2021 (an increase in 12.8% for the mean and 9.5% for Median).

 

Figure 4: Word counts between 2020 and 2021

 

This is the raw comparison, but there were 6 papers (out of 2844 that were above 16000 words). Of these, 4 are in Understanding People, 1 is in Assets, and 1 is in Games.

Excluding those, we have:

Figure 5: Word counts between 2020 and 2021 exclude 6 long papers

 

Overall, there were 150 papers in 2021 that were greater than the maximum length of  papers in 2021 (about 5% of the submissions).

 

People often wonder whether it is ‘worth’ submitting a rebuttal at or below a certain score. In previous years, the overall mean for scores does not move much after rebuttals, but this does not mean that individual scores do not move. They can and they do. If the ACs for one of your submissions have encouraged you to write a rebuttal then it might make sense to do so. If nothing else, a rebuttal gives you the chance to consider the perceived limitations of your submission so that you can fix them, whether for CHI 2021 or another venue.

 

A number of members of our community have provided views on how to write a rebuttal. Here are a few that were suggested in last year’s blog:

This list isn’t exhaustive.

HOME >  Blog >  Some Stats about CHI 2021 Papers
pagetop