Checklists

Editors’ checklists

The BMJ's editors don't routinely use checklists for critical appraisal, but these are the kind of questions we ask ourselves when reading papers:

Critical appraisal questions

• What is the paper about?
• Why was the study done?
• What type of study was done?
• Was it primary research (experiment, RCT, cohort, case-control, cross-sectional, longitudinal, case report/series)?
• Was it secondary research (overview, systematic review, meta-analysis, decision analysis, guidelines development, economic analysis)?
• Was the design appropriate (for study on treatment, diagnosis, screening, prognosis, or causation)?
• Was the study ethical?
• Is the design right (see table below)?

Does this treatment work?	systematic review, RCT
How good is a diagnostic test?	(prospective) cohort study
Should we screen?	RCT
What causes this disease?	RCT, prospective cohort study, case control study (rare diseases)
What did people think or do?	cohort study, cross-sectional survey, qualitative study

Qualitative research checklist

These are the questions that BMJ editors should consider when appraising papers presenting original qualitative research (although we don't routinely use a checklist for this):

• Was the research question clearly defined?
• Overall, did the researcher make explicit in the account the theoretical framework and methods used at every stage or the research?
• Was the context clearly described?
• Was the sampling strategy clearly described and justified?
• Was the sampling strategy theoretically comprehensive to ensure the generalisability of the conceptual analysis (diverse range of individuals and settings, for example)?
• How was the fieldwork undertaken? Was it described in detail?
• Could the evidence (fieldwork notes, interview transcripts, recordings, documentary analysis, etc) could be inspected independently by others: if relevant, could the process of transcription be independently inspected?
• Were the procedures for data analysis clearly described and theoretically justified? Did they relate to the original research questions? How were themes and concepts identified from the data?
• Was the analysis repeated by more than one researcher to ensure reliability?
• Did the investigator make use of quantitative evidence to test qualitative conclusions where appropriate?
• Did the investigator give evidence of seeking out observations that might have contradicted or modified the analysis?
• Was sufficient of the original evidence presented systematically in the written account to satisfy the sceptical reader of the relation between the interpretation and the evidence (for example, were quotations numbered and sources given)?

Peer reviewers’ checklists

BMJ peer reviewers do not have to fill in standard appraisal forms. But we do ask all of them to consider this general guidance:

General guidance for BMJ peer reviewers

The manuscript is a confidential document. Please do not discuss this even with the author.

The BMJ now has a system of open peer review. This means that you will be asked to sign your report on any paper we send you. It does not mean that authors should contact you directly; we will continue to ask them to direct any queries through us. Openness also means that we ask reviewers and authors to declare any competing interest that might relate to papers considered by the BMJ.

As a reviewer you will be advising the editors, who make the final decision (aided by an editorial "hanging committee" for some papers). We will let you know our decision. We will pass on your signed report to the author; please do not make any comments that you do not wish the author to see. Even if we do not accept a paper we would like to pass on constructive comments that might help the author to improve it.

Please give detailed comments (with references, whenever possible) that will both help the editors to make a decision on the paper and the authors to improve it.

For all papers:
Is the paper important? Will the paper add enough to existing knowledge? Does the paper read well and make sense?

For research papers please comment on:

• Originality — does the work add enough to what is already in the published literature? If so, what does it add? If not, please cite relevant references.
• Importance of the work to general readers — does this work matter to clinicians, patients, teachers, or policymakers? Is a general journal the right place for it?
• Scientific reliability
• Research question — clearly defined and appropriately answered?
• Overall design of study — adequate?
• Participants studied — adequately described and their conditions defined?
• Methods — adequately described? For randomised trials: CONSORT style? Ethical?
• Results — answer the research question? Credible? Well presented?
• Interpretation and conclusions — warranted by and sufficiently derived from/focused on the data? Message clear?
• References — up to date and relevant? Any glaring omissions?
• Abstract/summary/key messages/This Week in BMJ — reflect accurately what the paper says.

Not all of these points will be relevant for non-research papers. Please use your discretion about the above list when reporting on other types of paper.

Some types of paper need more specific appraisal, and we may ask reviewers to use one or more of these special checklists :

• general statistical checklist 
• checklist for statistical assessment of randomised controlled trials
• checklists for health economics papers
• checklist for appraising clinical management guidelines
• checklist for lessons of the week
• checklist for drug points 

Statisticians’ checklists

General statistical checklist 

The BMJ asks statistical reviewers to use this checklist when appraising original research papers:

Design features of the study

• Was the objective of the study sufficiently described?
• Was an appropriate study design used to achieve the objective?
• Was there a satisfactory statement given of source of subjects?
• Was a pre-study calculation of required sample size reported?

Conduct of study

• Was a satisfactory response rate achieved?

Analysis and presentation

• Was there a statement adequately describing or referencing all statistical procedures used?
• Were the statistical analyses used appropriate?
• Was the presentation of statistical material satisfactory?
• Were the confidence intervals given for the main results?
• Was the conclusion drawn from the statistical analysis justified?

Recommendation on paper

• Is the paper of acceptable statistical standard for publication?

Checklist for statistical assessment of randomised controlled trials

Reports of randomised controlled trials must conform to the CONSORT statement. BMJ statistical reviewers also use the following checklist to assess such papers:

Design features

• Objective of the trial sufficiently described?
• Satisfactory statement of diagnostic criteria for entry to the trial?
• Satisfactory statement of the source of participants?
• Concurrent (not historical) controls used?
• Interventions well defined?
• Random allocation to intervention used?
• Method of randomisation described?
• Acceptably short delay from allocation to start of intervention?
• Potential degree of blindness used?
• Satisfactory statement of criteria for outcome measures?
• Outcome measures appropriate?
• Pre-study calculation of sample size reported?
• Duration of post-intervention follow up stated?

Conduct of trial

• Intervention and control groups comparable in relevant measures?
• High proportion of participants followed up?
• High proportion of participants complete intervention?
• Were participants who dropped out from intervention and control groups described adequately?
• Adverse effects of intervention reported?

Analysis and presentation

• All statistical procedures adequately described or referenced?
• Statistical analyses appropriate?
• Prognostic factors adequately considered?
• Presentation of statistical material satisfactory?
• Confidence intervals given for the main results?
• Conclusions drawn from the statistical analysis justified?

Recommendation on paper

• Is the paper of acceptable statistical standard for publication?
• If "No" , could it become acceptable with suitable revision?

Checklists for health economics papers

We now require that any paper presenting original studies on health economics should conform with the guidelines published in the BMJ in 1996, and we ask authors to complete the checklist shown below. For a full explanation, see Drummond and Jefferson, BMJ 3 August 1996; 313:275. The completed checklist is then seen by editors and peer reviewers. 

Authors must provide a completed checklist, indicating on which page of the paper each item is dealt with:

Study design
(1) The research question is stated
(2) The economic importance of the research question is stated
(3) The viewpoint(s) of the analysis are clearly stated and justified
(4) The rationale for choosing the alternative programmes or interventions compared is stated
(5) The alternatives being compared are clearly described
(6) The form of economic evaluation used is stated
(7) The choice of form of economic evaluation is justified in relation to the questions addressed

Data collection
(8) The source(s) of effectiveness estimates used are stated
(9) Details of the design and results of effectiveness study are given (if based on a single study)
(10) Details of the method of synthesis or meta-analysis of estimates are given (if based on an overview of a number of effectiveness studies)
(11) The primary outcome measure(s) for the economic evaluation are clearly stated
(12) Methods to value health states and other benefits are stated
(13) Details of the subjects from whom valuations were obtained are given
(14) Productivity changes (if included) are reported separately
(15) The relevance of productivity changes to the study question is discussed
(16) Quantities of resources are reported separately from their unit costs
(17) Methods for the estimation of quantities and unit costs are described
(18) Currency and price data are recorded
(19) Details of currency of price adjustments for inflation or currency conversion are given
(20) Details of any model used are given
(21) The choice of model used and the key parameters on which it is based are justified

Analysis and interpretation of results
(22) Time horizon of costs and benefits is stated
(23) The discount rate(s) is stated
(24) The choice of rate(s) is justified
(25) An explanation is given if costs or benefits are not discounted
(26) Details of statistical tests and confidence intervals are given for stochastic data
(27) The approach to sensitivity analysis is given
(28) The choice of variables for sensitivity analysis is justified
(29) The ranges over which the variables are varied are stated
(30) Relevant alternatives are compared
(31) Incremental analysis is reported
(32) Major outcomes are presented in a disaggregated as well as aggregated form
(33) The answer to the study question is given
(34) Conclusions follow from the data reported
(35) Conclusions are accompanied by the appropriate caveats

Authors may enter N/A if an item on the checklist is not appropriate, but this is only acceptable for items 9,10, 12-15, 20, 21, 23-29, and 31.

Editor’s checklist for health economics papers

BMJ editors should also consider these more general points about such papers:

• Is the research question important?
• Is the economic importance of the question stated?
• Is the topic of interest to BMJ readers?
• Is there enough economic detail to allow peer review?
• If the economic content is sound, would we want to publish the paper?
• Is there a reasonable chance that the economic content is sound?

Checklist for appraising clinical management guidelines

The BMJ asks reviewers to use this checklist, which is based on material published by three main sources - the US Agency for Health Care Policy and Research, the NHS Management Executive, and the North of England Guidelines Group:

Authors of papers presenting clinical management guidelines should ensure that:

• patient population to which guidelines apply is clearly described

• condition(s) to be detected, treated, or prevented is/are clearly defined
• circumstances (clinical/non-clinical) in which exceptions might be made in using the guidelines are clearly described

• methods for taking into account patient preferences in using the guidelines are clearly described

• guidelines have been reviewed by independent experts or peers

• guidelines have been piloted or pre-tested, with adequate discussion of how such results have been used

• main topics covered by the guidelines are presented clearly, with an accurate summary

• methods used to identify and select evidence are described clearly

• sources of information used in developing guidelines are referenced adequately

• methods used to assess strength of evidence are adequate and are described clearly

• method used to synthesise evidence (eg for meta-analysis or decision analysis) are clearly described

• methods used to reach expert or group consensus, and strength of consensus are clearly described

• those who developed the guideline eg individuals, interest groups are described clearly and are appropriate eg sufficiently multidisciplinary

• methods for taking into account potential biases/competing interests among guideline developers are clearly described and adequate

• methods used to seek views of interested parties not on the panel are described

• each major recommendation can be found easily

• recommendations are consistent with each other

• recommendations collectively cover all clinically relevant circumstances, or guideline explains why they do not

• qualitative and quantitative information is given about the health benefits and potential harms/risks of recommendations eg additional life expectancy, quality adjusted life years

• adequate qualitative and quantitative information is given about the costs of recommendations

• other guidelines dealing with the same topic are mentioned, and differences discussed

• levels or categories of evidence are expressed thus:

Ia evidence from meta-analysis of randomised controlled trials

Ib evidence from at least one randomised controlled trial

IIa evidence from at least one controlled study without randomisation

IIb evidence from at least one other type of quasi-experimental study

III evidence from non-experimental descriptive studies, such as comparative studies, correlation studies, and case control studies

IV evidence from expert committee reports or opinions or clinical experience of respected authorities, or both

• strength of recommendations is expressed thus:

A directly based on category I evidence

B directly based on category II evidence or extrapolated recommendation from category I evidence

C directly based on category III evidence or extrapolated recommendation from category I or II evidence

D directly based on category IV evidence or extrapolated recommendation from category I, II, or III evidence

• date for scheduled review or expiry of the guidelines is given

Checklist for lessons of the week

We ask reviewers to consider these points when appraising such papers:

Does the paper answer the following questions:

1. How common is the condition (Should be neither so rare as not really to matter, nor so common that everyone should be well aware of the problem)?
2. How commonly is the condition missed?
3. How serious is it if missed?
4. Will this report contribute to preventing missed cases?
5. Is it a good read?

The ideal lesson of the week would relate to a common condition, which is relatively commonly missed, serious if missed and where the report will help prevent further missed cases. The following exclusion criteria may also be useful:

1. "Cock ups" and delayed diagnosis with no significant clinical consequences.
2. Cases where everything possible was done but it ended in disaster anyway.
3. Finger pointing ("someone else messed it up and we sorted it out").
4. Single cases of unusual presentations of very rare illnesses.

Checklist for drug points 

We ask reviewers to check that these reports of adverse drug events include:

• Age and sex of patients
• Suspected drug and all drugs currently being taken with start, stop, and restart dates; and outcome. Generally something more than simple coincidence in time is required: rechallenge with the patient’s informed consent, or immunological investigations may tip the balance of probabilities.
• Prior experience with drug or adverse reactions to related drugs
• Other diseases and environmental factors and their timing
• Ancillary information from pharmaceutical company and regulatory agency
• All published reports
• Other factors relevant to verify specific types of adverse drug reactions (for example, blood concentration in overdose, baseline laboratory data, ethnic group)

Technical editor's checklist

When the BMJ accepts a paper provisionally and asks the authors to revise it, a technical editor completes a checklist covering any points of house style that need attention. The checklist below is for original research papers: slightly different checklists are used for other types of article: 

When returning your paper after revision please supply the information requested below.

• Source of funding
• Summary of 150 words for "This Week in BMJ"
• Structured abstract of 250 words
• Key messages box
• Opening summary of 75-100 words
• Address for each author and one position held at time of study
• Author for correspondence
• Abbreviations should not be used and should be spelt out in full each time
• Actual numbers of patients or subjects, as well as percentages, within the text and tables
• All values in SI units (except blood pressure in mm Hg)
• The numbers from which histograms were drawn. If these are percentages please also provide the actual numbers. (We generally convert histograms into tables, but even if we leave them as histograms the data from which they were drawn are helpful.)
• This article is too long as a short report. Please reduce it to within 600 words with one table or figure and at most five references
• Double spacing (not 1.5 spacing) for text and references; margins 3 cm or wider
• References must be set out in Vancouver style (see Uniform requirements for manuscripts submitted to biomedical journals. International Committee of Medical Journal Editors. Med Educ. 1999; 33(1):66-78):
• surnames and initials of all authors (or of only the first six)
• title of the article or chapters
• page numbers of each article or chapter
• editors of books
• publisher of each book
• place of publication of books
• year of publication of books
• title of the journal in full
• volume number of the journal
• has the reference been published or accepted for publication? If not please cite in text and renumber other references