Trust in Decision

  • January 5, 2021

CHI trust and transparency in one community

 

Our goal as Papers Chairs is to create an exciting CHI program that appeals to people with diverse interests. Although we, the Papers Chairs, are honoured to take on this task, it is not without its challenges; we hope that by illuminating the decision process, our CHI community (authors, reviewers, Subcommittee Chairs - SCs, and Associate Chairs - ACs) can reach a common understanding and trust.

 

Size and Diversity of CHI

 

ACM SIGCHI is the one main conference venue where all the SIGCHI specialized conferences and diverse topics with Human-Computer Interaction research come together and interact. Due to the large number of papers and diversity of topics at CHI, the subcommittee structure (detailed below) is a way to divide the evaluation of papers into topic areas and disciplines, thus allowing papers to be judged by appropriate experts in the respective areas. At times, even these subcommittees become so large that it is necessary to split them into 2 (or even 3) groups (splits), reducing the number of papers that need to be discussed in each subcommittee during the PC meeting. Specifically, we try to keep the number of papers per subcommittee to a maximum of 200 papers. For CHI2021, we had 21 different subcommittees (16 areas where 5 of the areas were further divided into splits).  

 

Our responsibility as Papers Chairs (PCs), is to ensure that SIGCHI is more unified than 21 separate sub conferences. We do so by carefully choosing appropriate subcommittee chairs (SCs), who, in turn, choose their constituent associate chairs (ACs). In addition, our role requires responding to individual questions from ACs, SCs, and authors; and keeping all the SCs informed of those answers so they can act in concert. 

 

Subcommittee Chairs (SCs) are chosen based upon their academic merits, overall experience, participation in the review process at previous SIGCHI (and other) conferences, and their ability to facilitate fair discussions of papers. We trust the SCs to be fair and unbiased in their assessment of papers and selection of their Associate Chairs (ACs). Each pair of SCs chooses the constituent makeup of their subcommittee by selecting appropriate ACs (12-25 depending on the subcommittee). 

 

Associate Chairs (ACs), in turn, are tasked with the recruitment of external reviewers and reading, reviewing, and summarizing multiple reviews. The external reviewers are chosen for particular papers based on their knowledge of the specialized topic discussed in the submission. As a team, the ACs drive the discussions and decision making both prior to and during the PC meeting. 

 

Our reviewing process takes the common practice where the reviewers do not know the identity of the authors and the authors do not know the identity of the reviewers. Conflict of interest concerns are carefully handled throughout the evaluation process. All SCs and ACs must declare the institutional conflicts as well as designate from the entire list of submitted authors with whom they are conflicted. Furthermore, when assigning papers to ACs, SCs are aware of the identity of the authors of papers in order to make doubly sure that those papers are assigned to unconflicted ACs. Furthermore, the primary AC is aware of the identity of the authors so that they can recruit external reviewers who are not conflicted. The other AC and the external reviewers are not aware of the identity of the authors when writing the review or discussing the reviews before the committee meeting.

 

The Selection Process

 

Subcommittee Chairs (SCs) oversee the assignment of papers, structure the discussion, handle confidentiality issues, and oversee the review processes on individual papers. It is the role of SCs to ensure that papers are treated fairly - and, during the PC meeting, discussed as if the authors were sitting next to the committee and listening in on the discussion. It is not the SCs who decide which papers to accept; this is ultimately the associate chairs (ACs) that make up the subcommittee. Each paper will be read by at least 2 ACs (in some cases more) and at least 2 external reviews and discussed during the papers committee meeting. The SCs are the facilitators of these discussions. 

 

Creating a common ethos

 

A particular challenge for coherence across the entire conference is keeping the subcommittees involved with each other and reaching a common ethos. 

 

In addition to the typical problems of keeping all the subcommittees in sync, this year there were unprecedented circumstances due to the ongoing global pandemic - COVID19.  There were new challenges raised by a completely virtual program committee meeting. We had little opportunity for SCs to interact and share joint concerns for our unified conference. We also had significant challenges holding one large synchronous meeting across widely disparate time-zones. These challenges will continue and need to be addressed by future PCs, SCs, and ACs since having all ~600 people come together to meet in person was already prohibitively expensive and detrimental to the environment.

 

Acceptance Rates

 

The overall acceptance rate [in the last ten years] for CHI has been consistently within the range of 20-27%. Historically, in the 1980’s the acceptance rate has been as low as 16% and as high as 39%!  

 

There have historically been different acceptance rates in different subcommittees and splits of subcommittees, which also varies from year to year. These can range from below 15% or be much higher. Again, there is no specific target percentage rate that any committee is expected to achieve (or not surpass). Each paper needs to be judged on its own merits and be found worthy, but we, as Papers Chairs encourage an attitude of accepting work that may be interesting to the community, allowing it to drive the field forward, and create interesting discussions. In the words of one pair of SCs:

 

"The consensus [on the decision process] was that PCs and SCs should not veto or artificially lower [or raise] the recommended outcomes of the reviewers and ACs--the subject matter experts who closely read the papers offered constructive feedback, reviewed the rebuttals, and discussed them."

 

Our goal (the entire papers committee) is that the sum of the accepted CHI papers creates a program that we all care about and look forward to attending. 

 

Sincerely

Steven Drucker and Pernille Bjørn

CHI2021 Papers chairs

 

HOME >  Blog >  Trust in Decision
pagetop