Lab Guide to Reviewing Papers


Reviewing papers is a great way to learn about new research and how to write better papers (by putting yourself in the position of a reviewer). The comments below are intended to be helpful, I'm very open to other suggestions!


Reviewing papers is a great way to learn about new research and how to write better papers (by putting yourself in the position of a reviewer). The comments below are intended to be helpful, I'm very open to other suggestions!

If you're new to this don't worry. the best way to learn is by doing.

When you get a paper to review, first check that you have enough time to complete the review by the deadline given. try to download it asap to make sure it prints properly and to insure that the paper isn't completely beyond your comprehension. Also check that you don't have a conflict of interest (if you are on the same campus with a co-author or have written a paper with a co-author in the past 5 years). If so notify the assigning person immediately.

If you are helping me with a review, please prepare a plain text version of your review and email it to me well ahead of the deadline so I can go over it with you before I finalize. This is especially important if you are new to reviewing.

It can take from 2 to 5 hours to do a proper review of a conference paper, and 5-10 hours or more to do a proper review of a jounal paper. Usually it takes 3 readings of a paper before you're ready to review it.

1st reading: read quickly without taking notes to get the overall aim and goals of the paper. don't worry if you don't understand all the details!

2nd reading: read carefully and mark with questions, notes. Use the internet and library to look up possible related papers that may not have been cited.

Draft your review (see below).

3rd reading: read once more carefully to make sure you haven't missed anything.

---------------

Your Review:

The first part of the review will be a series of bullet points, typically choices ranging from Excellent to Poor for:

Originality of concept(s):
Relevance of applications:
Technical soundness:
Thoroughness of results:
Importance of results:
Clarity of presentation:


There may also be a request to suggest a Session Title for the paper or if it should be considered for a Best Paper Award. Until you've had extensive experience and can put papers into context of the field, this is very hard, so please give your best estimates and I'll go over it after I read your draft.

The key part of your review is the Advice to Authors:

This is a body of text that includes 4 parts:


  • 1) A summary of the paper's main contributions in your own words (be careful not to simply restate the Abstract)

  • 2) Your evaluation of how well the paper reviews Related Work. For this part check Google, CiteSeer, or the library using keywords from the paper. You can suggest that they reference specific related papers you find from the last 2-3 years.

  • 3) Your evaluation of strengths and weaknesses of the paper. Be careful to offer constructive suggestions about how they might address each of the latter.

  • 4) suggestions on things to fix, for example figures that are unclear or typos (note: for conference papers you don't need to proofread in detail, that can take up enormous amounts of time).

  • 5) A short summary of your overall evaluation and recommendation, for example: "This paper introduces an interesting new model for holding deformable parts and is acceptable for the conference but would be improved with more experimental results."
The body of your Advice to Authors should be at least 100 words (200-300 is more typical), and longer in a review of a Journal paper.

Under Advice to the Editor/Program Committee, you can usually leave that blank unless you have something confidential you need to convey, for example if you find evidence that the paper includes plagiarism etc.

Note again that the tone of your review should be constructive and supportive. You are an anonymous reviewer, but you should view the process as though you are reviewing the paper for a friend or colleague. If the paper is weak the authors will appreciate and benefit from your advice and suggestions.

It is much easier to write a positive review than a negative review.

Ken Goldberg, Oct 2004
goldberg [at] berkeley [dot] edu
The Task of the Referee, by Alan Smith:
http://www.computer.org/portal/cms_docs_transactions/transactions/tpami/freecontent/taskoftheferee.pdf