This information here is to provide guidance to reviewers for VRST 2021. Specifically those who will perform full reviews like the secondary (2AC) and external reviewers.
The primary (1AC) will examine all full reviews for quality. If the primary find any reviews of poor quality (e.g., lacking details, missing reasons for rejection, etc.), they will ask reviewers to update their reviews. In extreme cases, 1AC may remove poorly written or incomplete reviews and find replacement reviewers.
Our goal is to ensure the highest possible reviewing standards for VRST 2021 either full paper (~6700 words, single column) and short paper (~3300 words, single column). Please read these guidelines before reviewing and reach out to us if you are unclear about anything or need more information (Slack or email). We also set up a Slack workspace where you can ask questions about the reviewing process or questions about your role as an IPC member etc. The below information is a guideline for reviewing the paper regardless if it is a full paper or a short paper.
In terms of the actual review, Ken Hinckley offers great advice on reviewing:
Hinckley, K. (2016). So You’re a Program Committee Member Now: On Excellence in Reviews and Meta-Reviews and Championing Submitted Work That Has Merit.
The good folks at ISMAR’21 have summarized major elements from Ken’s paper, added insights from advice by Steve Mann and Mark Bernstein, and made them available on the ISMAR 2021 website. We present a few of those points with minor changes below. Please see the link for a longer list.
- Read papers with care and sympathy. Many hours of work — in some cases, years of work — have gone into research and writing this paper. Try reviewing as you would like others to review your own paper.
- Short and/or content-free reviews that read like unsubstantiated opinions are insufficient and will be rejected by 1AC.
- State specifically why the paper is “great”, “mediocre” or “bad”.
- Clearly describe on what grounds the paper should be accepted (or rejected). Keep in mind that this may be someone’s first paper. Your constructive feedback will be appreciated.
- Explicitly and clearly discuss the weaknesses and limitations in a positive and constructive manner. Specifically, don’t be insulting – be positive.
- Your review should be a critique of what the authors have done and not what they should have done.
- Do not reject a paper because of anything that can be fixed/addressed easily as authors will have the opportunity to do so for their camera ready version.
- Avoid the fallacy of novelty. Specifically, do not simply reject papers because they replicated experiments.
- User studies are not required or appropriate for all papers. While the authors need to support their claims with evidence, that form of evidence can vary from paper to paper. In short, the work needs to have a validation but not necessarily with a user study.
- Systems papers should be reviewed holistically, taking into account the limitations imposed by COVID-19. There are many different ways of evaluating a systems paper. Please do not reject the paper simply because it is built using existing well-known techniques if it accomplishes new functionality. In such situations, judge the novelty and significance of the new functionality. Here are some references for how to evaluate different types of research:
Thank you for your support and work to ensure the highest-quality VRST reviews. If you need more information, please do not hesitate to contact us: papers@vrst2021.org