• Check for retractions
• Critically appraise the quality of the remaining studies
• Record the data from this step in PRISMA flow diagram

Retraction is a mechanism for correcting the literature when an article has errors or flaws so serious that it should not be relied upon. It's a means for correcting the literature – part of a healthy science ecosystem!
A retraction doesn't necessarily mean intentional wrongdoing on the part of the authors; it can just be honest error.

Ways to check for retractions:

• Retraction Watch database: search by DOI or PubMed ID
  • Zotero and EndNote have partnered with Retraction Watch to identify retracted articles in your citation manager library.
• Terms to watch for in abstracts and article records: retraction, retracted, erratum, errata, “expression of concern”, comments, author responses to comments, "Update in: [article title]”.
• Search for retraction information on this set of articles in databases using filters like "Retracted Publication" in PubMed or filters called any of the "terms to watch for" above.

After the full-text screening, you will have a much smaller pool of results. Before you start to extract and analyze your data, you will need to to assess risk of bias in your results; sometimes referred to as critical assessment or quality appraisal. This step is an important part of why systematic reviews and meta-analysis are considered to be a high level of evidence.

According to the Institute of Medicine standards, appraisal for a systematic review requires the authors to:

• 3.6.1 Systematically assess the risk of bias, using predefined criteria
• 3.6.2 Assess the relevance of the study’s populations, interventions, and outcome measures
• 3.6.3 Assess the fidelity of the implementation of interventions

About bias and appraisal in general:

• Cochrane Handbook for Systematic Reviews of Interventions - Chapter 7: Considering bias and conflicts of interest among the included studies
• AHRQ - Assessing the Risk of Bias in Systematic Reviews of Health Care Interventions (slides)

Different types of studies have different appraisal criteria. There are appraisal support tools to assist you with many of them.

Collections of appraisal checklists for a variety of study types:

• JBI Critical Appraisal Tools: These critical appraisal tools from the Joanna Briggs Institute assist in assessing the trustworthiness, relevance and results of published papers.
• CASP Checklists: The Critical Appraisals Skills Programme (CASP) has appraisal checklists designed for use with Systematic Reviews, Randomized Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule.
• CEMB Critical Appraisal Tools: This site from the Oxford Centre for Evidence-Based Medicine contains useful tools and downloads for the critical appraisal of different types of medical evidence. Appraisal sheets are provided together with several helpful examples.

Appraisal checklists for specific study types:

• Randomized clinical trials
  • Cochrane Handbook for Systematic Reviews of Interventions - Chapter 8: Assessing risk of bias in a randomized trial
  • PEDro Scale:This scale was designed to score randomized controlled trial for internal validity and sufficient statistical information. Please review the notes at the bottom of the downloadable PDF
• Cohort, Case-Control, and Cross-sectional studies
  • Cochrane ROBINS-I Tool
  • The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses
  • STROBE Checklist
  • Tool to Assess Risk of Bias in Case Control Studies from the CLARITY Group at McMaster University
• Diagnostic Accuracy tests
  • AHRQ - Assessing Risk of Bias as a Domain of Quality in Medical Test Studies
  • QUADAS-2 - from Population Health Sciences at the University of Bristol Medical School
• Mixed Methods:
  • McGill Mixed Methods Appraisal Tool (MMAT) 2018 User Guide
• Systematic Reviews:
  • AMSTAR Checklist
• Clinical Practice Guidelines
  • National Guideline Clearinghouse Extent of Adherence to Trustworthy Standards (NEATS) Instrument
  • AGREE-II Appraisal of Guidelines for Research and Evaluation

As with journal articles included in reviews, grey literature must be appraised before incorporating it into a review.

• Unpublished studies and clinical trials: appraise using the same tools as published studies and trials.
• Other types of grey literature: AACODS checklist
  • provides guidance for evaluating Authority, Accuracy, Coverage, Objectivity, Date, Significance in the material.
  • developed by Jess Tyndall, Medical Librarian and Head of the Gus Fraenkel Medical Library at Flinders University as an evaluation and Critical Appraisal Tool specifically for use with grey literature sources
  • can be applied to materials across formats and disciplines

There can be a lag time of 1-5 years between publication and retraction. If something about an article still seems off, you can:

• Check for information about the article on listervs, electronic bulletin boards on the topic, colleagues, journal comments, altmetrics (Twitter, blogs, news--though use careful judgement about these!)
• Apply REAPPRAISED checklist

   

Guides to critically appraising systematic reviews and meta-analyses are a great resource for learning what needs to be present in a high-quality SR.
Try appraising your own review before submitting it for publication!

Guides:

• How to Read a Systematic Review and Meta-analysis and Apply the Results to Patient Care: Users’ Guides to the Medical Literature; Murad et al., 2014
• AMSTAR checklist - The AMSTAR checklist can be used to evaluate the methodological quality of an extant systematic review or as a guide when creating a systematic review. Short for "A MeaSurement Tool to Assess systematic Reviews."
• Critical Appraisal Skills Programme Systematic Review Checklist