Now that you have finished screening, you need to critically appraise your selected studies for quality and risk of bias. It is an important process in the review as it assesses your included studies for trustworthiness, value and relevance. Depending on who you are drafting your review for [Cochrane, JBI] there could be existing guidelines and standards that you need to follow when critically appraising the studies. The types of studies included in the review will also dictate which checklist could be used for the appraisal process. Check our list of Appraisal Tools for more information.
Things to consider when appraising the studies:
- How well was the study conducted?
- Is the study easy to understand?
- Check for bias including, selection, performance, attrition, detection and reporting biases
Reporting levels of evidence is also an important part of critical appraisal.
Meta Analysis - systematic review that uses quantitative methods to synthesize and summarize the results.
Systematic Review - summary of the medical literature that uses explicit methods to perform a comprehensive literature search and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid studies.
Randomised Controlled Trial - Participants are randomly allocated into an experimental group or a control group and followed over time for the variables/outcomes of interest.
Cohort Study - Involves identification of two groups [cohorts] of patients, one which received the exposure of interest, and one which did not, and following these cohorts forward for the outcome of interest.
Case Control Study - study which involves identifying patients who have the outcome of interest [cases] and patients without the same outcome [controls], and looking back to see if they had the exposure of interest.
Case Series - report on a series of patients with an outcome of interest. No control group is involved. [Definitions from Centre for Evidence Based Medicine]
Systematic review and bias tools
The below critical appraisal tools [CATs] can be used to assess the selected studies:
JBI checklists
CATs for a range of study types: case control, case reports, cohort, diagnostic test , economic evaluations, prevalence, quasi-experimental, RCTs, SRs and more.
ROBIS: Tool to assess risk of bias in systematic reviews
Assesses the risk of bias in systematic reviews [rather than studies] in 3 phases: [1] assess relevance [optional], [2] identify concerns with the review process and [3] judge risk of bias.
Cochrane Handbook - Chapter 8: Risk of Bias
This chapter outlines how to critically appraise randomised trails using of the revised version of the Cochrane Risk of Bias tool.
CASP
Tools for systematic reviews, randomised controlled trials, cohort studies, and case control studies.
QUADAS-2
QUADAS-2 is a quality assessment tool for diagnostic accuracy studies that consists of 4 phases: [1] patient selection, [2] index text [3] reference standard and [4] flow and timing.
-
AMSTAR
Assessment of Multiple Systematic Reviews is a 37-item assessment tool used to assess the methodological quality of systematic reviews.
Reporting tools
PRISMA
PRISMA is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. PRISMA primarily focuses on the reporting of reviews evaluating the effects of interventions, but can also be used as a basis for reporting systematic reviews with objectives other than evaluating interventions.
Next you need to extract necessary data from your included studies. This will help you assess the validity of the included studies and will make analysis, synthesis and interpretation easier. There are guides that outline how to conduct appropriate data extraction and depending on what type of review you are conducting and where you are publishing you might need to follow specific guidelines for data extraction.
Things to remember when extracting data:
- It is best practice to develop a data extraction worksheet or workbook as part of your protocol. This should include the key concepts of the review that you are interested in. Developing a data workbook early in the process will save time when you reach this step.
- Your data extraction workbook should be comprehensive enough that you should not have to keep returning to your included studies.
- Only extract data that is relevant to your systematic review question.
If you have screened your review in Covidence it has data extraction capabilities. Please see Introduction video below:
Guidelines & Standards
- Cochrane Handbook: Chapter 5 - Collecting Data - This chapter details data collection and management requirements for authors conducting health science SRs. Section 5.5 specifically looks at the extraction process. The relevant MECIR are included.
- PRISMA Preferred Reporting Items for Systematic Reviews - Item 9 of the PRISMA 2020 Checklist addresses data extraction and is described further in the Explanation and Elaboration document.
Once you have extracted your data, you need to synthesise your data. This involves bringing together your findings from the included studies to describe and explain what your systematic review has found.
Meta-analysis
One way to synthesise data is by using meta-analysis. This approach requires the studies to be similar or contain qualitative data. A meta-analysis is a statistical combination of the results from each included study to create one large study. A forest plot or meta-view is an excellent way to present the synthesised data.
Meta-synthesis [Narrative or descriptive synthesis]
If the studies are not similar you will need to synthesise the data using a narrative or descriptive synthesis. This type of synthesis is more subjective and provides an overall view of the included studies. As narrative or descriptive synthesis has no standard process, you could use a guide or framework to consider quality, quantity, consistency, impact and applicability of the evidence.
Forest plots and other methods
Tables and figures can help help to present included studies and their findings in a clear way. A ‘Findings Summary’ table provides key information regarding the quality of evidence, the effect and outcomes of the included studies. Forest plots are often used to illustrate results of individual studies.