Navigation

“Peer review is broken. But let’s do it as effectively and as conscientiously as possible.” — Rosy Hosking, CommLab

“A thoughtful, well-presented evaluation of a manuscript, with tangible suggestions for improvement and a recommendation that is supported by the comments, is the most valuable contribution that you can make as a reviewer, and such a review is greatly appreciated by both the authors of the manuscript and the editors of the journal.” — ACS Reviewer Lab

Criteria for success

A successful peer review:

  1. Contains a brief summary of the entire manuscript. Show the editors and authors what you think the main claims of the paper are, and your assessment of its impact on the field. What did the authors try to show and what did they try to claim?
  2. Clearly directs the editor on the path forward. Should this paper be accepted, rejected, or revised?
  3. Identifies any major (internal inconsistencies, missing data, etc.) concerns, and clearly locates them within the document. Why do you think that the direction specified is correct? What were the issues you identified that led you to that decision?
  4. Lists (if appropriate — i.e. if you are suggesting revision or acceptance) minor concerns to help the authors make the paper watertight (typographical errors, grammatical errors, missing references, unclear explanations of methodology, etc.).
  5. Explain how the arguments can be better defended through analysis, experiments, etc.
  6. Is reasonable within the original manuscript scope; does not suggest modifications that would require excessive time or expense, or that could instead be addressed by adjusting the manuscript’s claims.

Structure Diagram

A typical peer review is 1-2 pages long. You can divide your content roughly as follows:

Identify your purpose

The purpose of your pre-publication peer review is two-fold:

  1. Clearly explain to the editors and authors any issues that you have identified with the science being reported (within reason), requiring evaluation of…
    1. Scientific integrity (which can be handled with editorial office assistance)
    2. Quality of data collection methods and data analysis
    3. Veracity of conclusions presented in the manuscript
  1. Determine match between the proposed submission and the journal scope (subject matter and potential impact). For example, a paper that holds significance only for a particular subfield of chemical engineering is not appropriate for a broad multidisciplinary journal. Determining match is usually done in partnership with the editor, who can answer questions of journal scope.

Analyze your audience

The audience for your peer review work is unusual compared to most other kinds of communication you will undertake as a scientist. Your primary audience is the journal editor, who will use your feedback to make a decision to accept or reject the manuscript. Your secondary audience is the author, who will use your suggestions to make improvements to the manuscript. Typically, you will be known to the journal editors, but anonymous to the authors of the manuscript. For this reason, it is important that you balance your review between these two parties.

The editors are most interested in hearing your critical feedback on the science that is presented, and whether there are any claims that need to be adjusted. The editors need to know:

  • Your areas of expertise within the manuscript
  • The paper’s significance to your particular field

To help you, most journals willhave guidelines for reviewers to follow, which can be found on the journal’s website (e.g., Cell Guidelines).

The authors are interested in:

  • Understanding what aspects of their logic are not easily understood
  • Other layers of experimentation or discussion that would be necessary to support claims
  • Any additional information they would need to convince you in their arguments

Skills

Format Your Document in a Standard Way

Peer review feedback is most easily digested and understood by both editors and authors when it arrives in a clear, logical format. Most commonly the format is (1) Summary, (2) Decision, (3) Major Concerns, and (4) Minor Concerns (see also Structure Diagram above).

There is also often a multiple choice form to “rate” the paper on a number of criteria. This numerical scoring guide may be used by editors to weigh the manuscript against other submissions; think of it mostly as a checklist of topics to cover in your review.

Summary

The summary grounds the remainder of your review. You need to demonstrate that you have read and understood the manuscript, which helps the authors understand what other readers are understanding to be the manuscript’s main claims. This is also an opportunity to demonstrate your own expertise and critical thinking, which makes a positive impression on the editors who often may be important people in your field.

It is helpful to use the following guidelines:

  • Start with a one-sentence description of the paper’s main point, followed by several sentences summarizing specific important findings that lead to the paper’s logical conclusion.
  • Then, highlight the significance of the important findings that were shown in the paper.
  • Conclude with the reviewer’s overall opinion of what the manuscript does and does not do well.

Decision

Your decision must be clearly stated to aid the interpretation of the rest of your comments (see Criteria for Success). Do this either as part of the concluding sentence in the summary paragraph, or as a separate sentence after the summary. In general, you try to categorize within the following framework:

  • Accept with no revisions
  • Accept with minor revisions
  • Accept with major revisions
  • Reject

Some journals will have specific rules or different wording, so make sure you understand what your options are.

Most reviews also contain the option to provide confidential comments to the editor, which can be used to provide the editor with more detail on the decision. In extreme cases, this can also be where concerns about plagiarism, data manipulation, or other ethical issues can be raised.

The Decision area is also where you can state which aspects of complex manuscripts you feel you have the expertise to comment on.

Major Concerns (where relevant)

Depending on the journal that you are reviewing for, there might be criteria for significance, novelty, industrial relevance, or other field-specific criteria that need to be accounted for in your major concerns. Major concerns, if they are serious, typically lead to decisions that are either “reject” or “accept with major revisions.”

Major concerns include…

  • issues with the arguments presented in the paper that are not internally consistent,
  • or present arguments that go against significant understanding in the field, without the necessary data to back it up.
  • a lack of key experimental or computational data that are vital to justify the claims made in the paper.
  • Examples: a study that reports the identity of an unexpected peak in a GC-MS spectrum without accounting for common interferences, or claims pertaining to human health when all the data presented is in a model organism or in vitro.

One of the most important aspects of providing a review with major concerns is your ability to cite resolutions. For example…

  • If you think that someone’s argument is going against the laws of thermodynamics, what data would they need to show you to convince you otherwise?
  • What types of new statistical analysis would you need to see to believe the claims being made about the clinical trials presented in this work?
  • Are there additional control experiments that are needed to show that this catalyst is actually promoting the reaction along the pathway suggested?

Minor Concerns (optional)

Minor concerns are primarily issues that are raised that would improve the clarity of the message, but don’t impact the logic of the argument. Most commonly these are…

  • Grammatical errors within the manuscript
  • Typographical errors
  • Missing references
  • Insufficient background or methods information (e.g., an introduction section with only five references)
  • Insufficient or possibly extraneous detail
  • Unclear or poorly worded explanations (e.g., a paragraph in the discussion section that seems to contradict other parts of the paper)
  • Possible options for improving the readability of any graphics (e.g., incorrect labels on a figure)

While minor concerns are not always present in the case of reviews with many major concerns, they are almost always included in the case of manuscripts where the decision is an accept or accept with minor revisions.

Offer revisions that are reasonable and in scope

Think about the feasibility of the experiments you suggest to address your concerns. Are you suggesting 3 years’ more work that could form the basis for a whole other publication? If you are suggesting vast amounts of animal work or sequencing, then are the experiments going to be prohibitively expensive? If the paper would stand without this next layer of experimentation, then think seriously about the real value of these additional experiments. One of the major issues with scientific publishing is the length of time taken to get to the finish line. Don’t muddy the water for fellow authors unnecessarily!

As an alternative to more experiments, does the author need to adjust their claims to fit the extent of their evidence rather than the other way round? If they did that, would this still be a good paper for the journal you are reviewing for?

Structure your comments in a way that makes sense to the audience

Formatting choices:

  • Separate each of your concerns clearly with line breaks (or numbering) and organize them in the order they appear in the manuscript.
  • Quote directly from the text and bold or italicize relevant phrases to illustrate your points
  • Include page and line/paragraph numbers for easy reference.

Style/Concision:

  • Keep your comments as brief as possible by simply stating the issue and your suggestion for fixing it in a few sentences or less.

Offer feedback that is constructive and professional

Be unbiased and professional.

Although the identities of the authors are sometimes kept anonymous during the review process (this is rare in chemical and biological research), research communities are typically small and you may try to “guess” who the author is based on the methodology used or the writing style. Regardless, it is important to remain unbiased and professional in your review. Do not assume anything about the paper based on your perception of, for example, the author’s status or the impact their results may have on your own research. If you feel that this might be an issue for you, you must inform the editor that there is a conflict of interest and you should not review this manuscript.

Be polite and diplomatic.

Receiving critical feedback, even when constructive, can be difficult and possibly emotional for the authors. Since you are not anonymous to the editors, being unnecessarily harsh in your feedback will reflect badly on you in the end. Use similar language to what you would use when discussing research at a conference, or when talking with your advisor in a meeting. Manuscript peer review is a good way to practice these “soft” skills which are important yet often neglected in the science community.

Additional resources about effective peer reviewing

This article was written by Mike Orella (MIT Chem E Comm Lab); edited by Mica Smith (MIT Chem E Comm Lab) and Rosy Hosking (Broad Comm Lab)