ACM UbiComp 2015 Policy on Anonymization in the Review Process

Background: Anonymization is sometimes used in academic review processes in a number of ways. First, reviewer identities may be kept from the authors; this allows reviewers to write critical reviews without the authors identifying them. Second, reviewer identities may be kept from other reviewers; this can help more junior reviewers share their frank assessment of the paper without worrying that they are contradicting more senior people who they perhaps normally would not contradict. Third, author identities may be kept from reviewers; this can help avoid bias (or unintentional bias, or just perceived bias) towards authors (or their institutions) by reviewers.

In the text below, “reviewer” includes both “PC member” and “external reviewers” (who are not PC members).

None of the policies below override the conflict policy – in no case will a reviewer identity be shared with anyone who has a conflict with a paper.

UbiComp has the following policies on anonymization:

  • Reviewer identities will always be kept confidential from authors and conflicted PC members/PC chairs. The only exception to this is during a “shepherding” process, where a PC member is asked to help the authors through direct communication to make changes to a paper. This process MAY be done using non-anonymized means (e.g. email) if the relevant PC member agrees.
  • External reviewer identities will be kept confidential from external reviewers. PC member identities MAY be kept confidential from external reviewers (but this is often not possible, e.g. due to reviewing system limitations and because the external reviewer obviously knows who invited them to review).
  • Reviewer names WILL NOT be kept confidential from PC members (except those conflicted). This is because (a) typically both PC members assigned to a UbiComp paper allocate reviewers, so it helps them to know the identity of the other reviewers so they can solicit reviews from a broad range of viewpoints (e.g. not unwittingly inviting another member of the same institution to review), and (b) during the PC meeting, it sometimes helps to know who a review is by, and (c) it enables excellent reviewers to receive some recognition since PC members will see their identities. Note that typically only PC members assigned to a paper will see reviewer identities during the review phase; only during the PC meeting phase is it necessary for all PC members to see reviewer identities.
  • Author identities WILL NOT be hidden from PC members reviewing the paper, or (at the PC meeting) from all PC members. The former is so that PC members can assign non-conflicted external reviewers, and it is important because the external reviewers themselves often cannot tell if there is a conflict if the paper is anonymized. The latter is for conflict detection at the PC meeting – by showing the author list of a paper prior to its discussion, it is easier to reliably exclude conflicted people.
  • During the review process, the authorship of the paper should be de-emphasised so that focus is on the paper content only. This includes (a) requiring authors to leave blank the author list area of a paper’s first page, so that it is not part of reviewers’ first impressions of a paper, (b) after the reviewer assignment phase is complete, if possible, turning off the option for PC members to see author identities, and (c) informing all reviewers that author and institution identities must not be used to influence review scores or any other aspect of the review process.
  • Authors MAY fully anonymize their paper (i.e. not just the normal author list area, but the full paper) if they wish. If this is done, external reviewers will not find out who the authors are. This is accomplished by e.g. anonymizing faces or names in figures, and anonymizing all self-references by referencing the prior work neutrally (e.g. "In previous work, Smith et al. found..." rather than "In previous work, we found..."). The reason this is optional is that some papers are hard to fully anonymize (e.g. because the systems they refer to are well-known to the community), and in some cases anonymization can be detrimental to fair review (e.g. for a study about a well-known system, it makes a big difference to the perceived contribution if the authors are those who built the system or not)
  • Authors MUST NOT remove relevant self-references from a paper for anonymization reasons, e.g. by saying “blank for blind review”. Papers using "blank for blind review" may be rejected simply because it is impossible to evaluate the contribution of a paper if other related papers are not accessible. Authors who wish to be anonymous to external reviewers should reference prior work neutrally(e.g. "In previous work, Smith et al. found..." rather than "In previous work, we found..."). In the case of simultaneous related (but not identical) submissions, authors may remove author names from the citation, but if so they must provide the PC chairs with an anonymized copy of the other paper. Please also refer to the Policy on Novelty.