Peer-review is a major part of the paper selection process for CHI, and high-quality reviews made by experts are important. However, giving a large number of paper submissions to CHI we also know that many will be reviewing for the first time, and this is why we decided to write this blogpost for senior reviewers to reflect upon their practices and junior reviewers to gain insights on core aspects of reviewing.
Peer-review is a process where experts within the field are confidentially reading other researcher's work and providing recommendations and feedback about the work. CHI is a diverse field and NO ONE is an expert in all fields of CHI - and there are different criteria for what makes an excellent CHI paper within specific sub-disciplines (mostly represented as different subcommittees on the program committee). However, there are still some general instructions for reviewing - and recruiting reviewers.
Writing a good review
When you accept an invitation to review a paper, you should consider it with confidentiality and for fair treatment. Approach the paper as you would want others to approach your own paper. Start by skimming the paper - read the title, abstract, introduction, and conclusion - so you get an idea about what the paper is trying to do. Consider what the appropriate criteria for evaluating this type of paper should be.
Then read the complete paper, the method, the result, and the discussion and evaluate whether you understand what the paper wants to achieve, how the authors approach this and consider the contributions. Consider the related work and see how this paper builds upon and/or contrasts with other papers already published in the field. Ask yourself who would want to know about this work and who would appreciate this kind of contribution, how significant this contribution is, and how well this paper demonstrates the contribution.
- Be kind and respectful. Write the review as if the paper author is sitting in front of you
- Very short reviews (2-3 lines) are problematic (especially remember this if you write a positive review), because when an AC is sitting at the PC meeting and only has “this paper should be accepted” without any academic argument - the review often ends up being dismissed. Nearly as bad is a very short negative review without sufficient justification for why the paper should not be accepted.
- Be aware that some authors do not have access to a native English speaker. A paper should not be rejected based solely on poor grammar or word usage. If the work is sound, recommendations for final editing can be made.
- Reviews should be several paragraphs long - and can often be structured like the following:
- Paragraph 1 describing what the paper is about, demonstrating to the authors that you have read and understood what they were trying to do
- Paragraph 2 identifying positive points on the paper, starting with something like: “I appreciate that the authors did X, Y, and Z in their submission”
- Paragraph 3 clearly identifying your concerns with the paper, what is missing, and considering the importance with respect to the overall contribution of the paper (remember that contributions can take different forms). Be descriptive and respectful. Start something like: “I would have liked it if the authors would have..“
- Paragraph 4 constructive and clear suggestions explaining why you’re making your recommendation. Especially if you recommend rejection, start something like: “In future revisions, the author could consider…”
- Be precise, concrete, clear, and constructive - and write in a respectful tone
Recruiting appropriate reviewers (only relevant for ACs)
All papers are reviewed by one Associate Chair (AC2) and two recruited reviewers external to the program committee. It is another Associate Chair (AC1) who recruits the two external reviewers. Identifying good reviewers is about considering the collective expertise of the ‘reviewing-team’ for the specific paper. Consider diversity in the reviewing team of the paper - don’t recruit two junior reviewers - try to recruit at least 1 senior reviewer; don’t recruit reviewers from the same institution - try to recruit reviewers from different institutions and different geographical areas. You want diverse expert opinions.
- Choose your reviewers wisely, remembering that you will be using their recommendation during the PC meeting to accept or reject a paper.
- Read/Skim the paper
- What is the topic, method, and contribution?
- Who would be fair reviewers for this paper?
- Who is the paper referencing? Could they be good reviewers? (Note to authors: since reviewers are often taken from your references, be aware of this, always treat referenced authors with appropriate respect).
- Often the best reviewer is not the person sitting next to you in a hallway
- PCS suggests names, but you need to look them up before you pick them (consider clicking the dblp link next to the reviewer's name to learn more about them). Often the suggestions in PCS are not the most appropriate reviewers and should be used only as rough starting points when you can’t find appropriate reviewers by other means.
- Consider diversity in geography, gender, experience, ethnicity, labs, institutions etc.
- Consider complementary skills (technical, big picture, application area, theory, etc.)
- Be aware of hawks and doves
- Junior reviewers tend to be very harsh (remember no paper is perfect, and it’s much easier to tear down a paper than it is to find the important elements)
- Some reviewers are famous for being super critical
- Some reviewers are super nice
Ken Hinckley (a CHI Academy member, test of time award winner, and senior editor for TOCHI for several years) penned an essay on reviewing, that while specific to one particular domain, has good advice for any committee member (or author) for the CHI conference. Please take a look here: http://mobilehci.acm.org/2015/download/ExcellenceInReviewsforHCICommunity.pdf