This page provides guidelines for reviewers responsible for assessing submissions to CHI.

Key points:

• Your primary criterion for judging a paper is: Does this submission provide a strong contribution to the field of HCI?  Remember that there are many ways a paper can make a contribution to HCI, and you should review the paper appropriately. See “Contributions to CHI” for details.
• Reviewers rate each Paper using a 5-point scale; your written appraisal must support your numeric ranking.
• A high-quality review is typically about a page of written text; very short reviews are frustrating for authors and hurt the review process. Always put yourself in the author’s position: what level of detailed feedback would you like to see for your own work?
• Also remember to write detailed reviews even if you like the paper - short positive reviews without justification often have difficult conditions during the PC meeting.

Contributions

The primary criterion for the evaluation of all papers is the submission’s contribution to HCI. In all cases, a CHI paper must break new ground and make an original research contribution. However, it is important to recognize that there are many ways for which a paper can make a contribution to HCI, and you should review the paper appropriately. Please see Selecting a Subcommittee for a list of some of the types of contributions a paper can make to HCI, and Guide to a Successful Submission for the associated criteria that you can use to assess this type of contribution.

Paper Lengths

Papers of different lengths are reviewed within the same rigorous review process and at the highest level are judged by very similar criteria (i.e., does this paper provide a strong contribution to the field of HCI?). However, it is important as a reviewer to realize that the type of content that is appropriate for a shorter paper is somewhat different than for a longer paper. A shorter paper should present brief and focused research contributions that are noteworthy but may not be as comprehensive or provide the same depth of results as a long paper.

As something new for CHI2021, Papers do not have a page limit, but paper length should be commensurate with their contribution (see below).

It should be noted that, in the past, authors were often encouraged to submit a maximum-length paper of up to 10 pages. For CHI 2021, authors are encouraged to submit a paper of length proportional to its contribution and thus there are no arbitrary maximum (or minimum) length papers, however clarity of writing is considered vital to a high-quality submission.

If your research contribution requires only 4, 7, or 9 pages (plus references), please submit a paper of that length. Reviewers will be instructed to weigh the contribution of a submission relative to its length. Papers should be succinct, but thorough in presenting the work. Typical papers will be 8-10 pages long (plus references) but papers can be shorter (e.g. 4-5 pages) if the contribution is smaller. Shorter, more focused papers are encouraged and will be reviewed like any other paper. Papers whose lengths are incommensurate with their contributions will be rejected. Papers may be perceived as too long if they are repetitive or verbose, or too short if they omit important details, neglect relevant prior art, or tamper with formatting rules to save on page count.

Prior Publication

Content appearing at CHI should be new and groundbreaking. Therefore, material that has been previously published in widely disseminated archival publications should not be republished unless the work has been significantly revised. Guidelines for determining “significance” of a revision are stated in the ACM Policy on Pre-Publication Evaluation and the ACM Policy on Prior Publication and Simultaneous Submissions. Roughly, a significant revision would contain more than 25% new content material (i.e., material that offers new insights, new results, etc.) and significantly amplify or clarify the original material. These are subjective measures left to the interpretation and judgment of the reviewers and committee members – authors are advised to revise well beyond the policy guidelines.

An exception is for work that has previously been presented or published in a language other than English. Such work may be translated and published in English at CHI. The original author should typically also be the author (or co-author) of the English translation, and it should be made clear in your submission’s abstract that this is a translation.

Also note that non-archival venues, such as workshop presentations, posters, and CHI’s own Late-Breaking Work do not count as prior publications. Furthermore, a CHI paper should not be rejected on the grounds that it overlaps with work developed independently that was published after the CHI submission was made, during the review period. In other words, work that an author couldn’t have known about shouldn’t count against him or her.

Replicating Work

The policy on prior publication refers only to re-publication of one’s own work; this does not preclude publication of work that replicates other researchers’ work. Novelty is highly valued at CHI, but constructive replication can also be a significant contribution to human-computer interaction, and a new interpretation or evaluation of previously-published ideas can make a good CHI paper. For future replications to be possible, however, submitted work must include sufficient information. Efforts to include complete, well-organized supplementary material facilitating replication, such as software, analysis code and data, should be rewarded.

Transparency

Lack of transparency in the way research results are reported can be a ground to doubt the contribution. See the “Transparency” section in the Guide to a Successful Submission for a discussion of transparency in different contribution types.

Subcommittees

To improve the reviewing process, the CHI program committee is divided into approximately a dozen topic areas divided into approximately two dozen subcommittees. Each subcommittee is responsible for a topic area within HCI (see Selecting a Subcommittee for details). Each subcommittee is chaired by two Subcommittee Chairs (SCs), who invited the relevant Associate Chairs (ACs) who are knowledgeable in the topics covered by the subcommittee. As specialists in this topic area, the primary responsibility of the AC is to recruit excellent reviewers (such as you) for each submission.

However, as a reviewer, you should not judge the paper by how well it fits the subcommittee theme(s). Many papers will not cleanly fit into a particular subcommittee for a variety of reasons, and we do not want to penalize authors for this. Remember, the subcommittee organization is there only to try to improve reviewer matches and to better handle the volume of submissions. If you have a paper that does not fit the subcommittee theme, evaluate it as best you can with respect to the paper’s own quality. Any topic is valid, as long as it fits within the interests of a reasonable fraction of the overall CHI audience. The primary criterion for review is the submission’s contribution to HCI.

For more information about the overall CHI review process, see CHI Papers Review Process.

References

We highly recommend Ken Hinckley’s thoughtful piece on what excellent reviewing is. If we had any way to enforce this, we would make it “required reading” for CHI reviewers and ACs.

Even with great guidelines like these that we can all agree on, the debate about what makes a good CHI paper has been going on as long as the CHI conference has existed. If you are interested, the papers below touch upon this debate and contain references to additional papers that concern it.

• Greenberg, S. and Buxton, B. 2008. Usability evaluation considered harmful (some of the time). In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems. CHI ’08. ACM, 111-120. DOI=http://doi.acm.org/10.1145/1357054.1357074
• Olsen, D. R. 2007. Evaluating user interface systems research. In Proceedings of the 20th Annual ACM Symposium on User interface Software and Technology. UIST ’07. ACM, 251-258. DOI=http://doi.acm.org/10.1145/1294211.1294256
• Dourish, P. 2006. Implications for design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’06, ACM, 541-550. DOI=http://doi.acm.org/10.1145/1124772.1124855
• Newman, W. 1994. A preliminary analysis of the products of HCI research, using pro forma abstracts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating interdependence (Boston, Massachusetts, United States, April 24 – 28, 1994). ACM, New York, NY, 278-284. DOI=http://doi.acm.org/10.1145/191666.191766
• Daniel Reed and Ed H. Chi. 2012. Online privacy; replicating research results. Commun. ACM 55, 10 (October 2012), 8-9.