Systematic reviews, scoping reviews and meta-analysis

A protocol statement makes clear if a protocol already exists and makes note of its registration number, if applicable. If using an existing protocol in the literature, be sure to acknowledge and make explicit reference to it. The explicit use of another protocol helps to make research advances clear when a researcher is using a similar methodological protocol as a previous study. 

If the protocol is new, it may need to be registered to facilitate future research in the same area. Scoping review protocols are often registered with the Center for Open Science, FigShare or at the systematic review register of JBI. Systematic review protocols are often registered with Cochrane, PROSPERO or Campbell.

Example 7: Acknowlediging an existing protocol

"The protocol for this review is available at Fitzpatrick & Burns, 2017" (Fitzpatrick & Burns, 2019, Types of studies section).

Example 8: Acknowledging a new protocol

"A scoping review protocol was created to guide the process and is available from the corresponding author upon request. This paper adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for scoping reviews" (Dol et al., 2019, Overview section).

Variations on this language are possible. The important thing is that if a previous protocol is being followed, it should be acknowledged formally. Some reviews link directly to the protocol.

This states the eligibility criteria of the literature sought. This can be done using the PPC (Population, Concept, and Context) or PICO (Population/Patient/Problem, Intervention, Comparison, and Outcome) framework or in other ways. Mention the types of evidence included, along with your justification for any limits you apply to the source type. Eligibility criteria might also be mentioned briefly, and the details relegated to an appendix or table.

Example 9: Eligibility criteria for included studies

"Articles were determined eligible for inclusion if they discussed the use of social media by health researchers, including but not limited to use of social media for recruitment, data mining, social media initiatives or campaigns, hashtag communities, and journal clubs. Articles could be from health researchers at any stage of their career (trainee to faculty member) and across any types of health research (policy, services, outcomes, medical, and basic)" (Dol et al., 2019, Inclusion and exclusion section).

This describes the databases you used and includes coverage dates.

Example 10: Introducing databases searched and dates

"A range of sources were used to ensure a comprehensive coverage of the literature. An initial search was conducted in August 2017, repeated and finalized in November 2019. The search made use of the following databases: …" (Wanko Keutchafo et al., 2020, Database searching section).

Example 11: Search strategy with database details and dates

"We also searched the following trial registries on 11 March 2019: World Health Organization International Clinical Trials Registry Platform (WHO ICTRP); US National Institutes of Health Ongoing Trials Register ClinicalTrials.gov). We used the sensitivity and precision‐maximising filter for retrieving randomised trials from MEDLINE and Embase as recommended in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins, 2011), which we adapted for the other databases" (Merriel et al., 2019, Electronic searches section).

This section outlines your search criteria and the limits placed on the search for your study. Some reviews include a full search strategy as an appendix or figure.

Example 12: Search strategy as an appendix

"We include a database search log in an online appendix to this review. This log contains, for each database that was searched, the terms, phrases, and Boolean operators that were used to identify relevant studies; fields that were searched; and restrictions or filters that were used. The log also includes comments on the search strategy used for each database to describe any database-specific procedures that were used to identify studies. Finally, the log indicates the number of records that were retrieved from each database along with the number of full-text studies that were downloaded from this pool" (Fitzpatrick & Burns, 2019, Electronic searches section).

Example 13: Search strategy embedded in text

"Our general/starting-point search terms for this meta-analysis include those used by Cooper et al. (2003), augmented by terms used in pertinent research published after that meta-analysis. The basic form of the search terms is: “year-round school*” or “year-round education” or (school AND (“alternative calendar” or “modified calendar” or “balanced calendar”) or (“year-round calendar” AND school). We modified the precise terms, phrases, and Boolean operators to take advantage of the search features, index terms identified in the resource’s thesaurus, and tools within each of 22 specific search/retrieval resource" (Fitzpatrick & Burns, 2019, Electronic searches section).

This section outlines how the sources were selected for your study, including the criteria used and how disputes were resolved.

Example 14: Source selection and dispute resolution

"The titles were reviewed against the eligibility criteria by EW. This initial search was monitored, exported into EndNote X9 reference manager, for abstract and full text screening by EW. The duplicated studies were deleted, followed by independent reviewing of the abstracts by EW and JK. Studies deemed ‘unclear’ were advanced to the subsequent screening stage. Assistance from the study university library services was requested when full texts could not be retrieved from the databases and five full texts were provided" (Wanko Keutchafo et al., 2020, Study selection section).

Sometimes referred to as ‘data extraction’, this section outlines how the data was charted from the sources of evidence included, and whether it was done independently or in duplicate. It also notes any processes of confirmation that you might have used. Importantly, this section visually presents the how the data was selected.

Example 15: Data extraction by author

"Microsoft Excel was utilized for this stage. The collected data points were author(s), title, publication year, country of first author’s affiliated university, research setting, purpose, participant demographics, research methods, measures, interventions, key findings, and limitations. The authors, participants, measures, interventions, and findings are summarized in Table 1. Aggregate data have been presented in the results section. As the first author compiled the data for Table 1, 12 articles were removed from the study because they were not primary research (n = 3), measured stereotypy and not SF (n = 5), did not isolate PA from other activities (n = 1), did not include a PA component (n = 2), or did not include a SF component (n = 1). The discarded articles were approved by authors two and five before the analysis was complete" (Reinders et al., 2019, Chart the data section).

Example 16: Data extraction using a form

"Two review authors (of AM, JF, and KB) independently extracted data from each study onto a data collection form based upon the Cochrane Effective Practice and Organisation of Care (EPOC) Group data collection checklist (EPOC 2013). Review authors (of AM, JF, and KB) piloted the form and ensured that it was fit for purpose and that there was consistency of approach. We refined the form as we progressed in the data extraction process by adding further fields or categories to the existing fields" (Merriel et al., 2019, Data extraction section).

This section explains what data was obtained from the studies reviewed. It needs to include a summary of the results based on the inclusion criteria, for example PCC, if you are using it.

Example 17: Data points related to inclusion criteria

"Extracted data included bibliographic details, country and setting, aim/objective, study design, targeted population, nurses’ nonverbal strategies used while communicating with older adults, older adults’ interpretation of nurses’ nonverbal behaviors, and relevant outcomes of interest" (Wanko Keutchafo et al., 2020, Data extraction section).

Example 18: Data points in relation to research question and inclusion criteria

"We extracted the student outcome data needed for calculating the effect size(s) from each study. To consider our second research question, we recorded two independent variables of interest: calendar structure and the duration of summer break. For each study, we recorded standard information on the study and report itself. This included the report author, year of publication or release, published/unpublished status, and the matching protocol used to identify the comparison school(s). For the treatment schools examined, this included the state in which the schools were located, years of student testing data included, and the type of score used for the outcome measurement. We also recorded sample/student characteristics associated with each estimate" (Fitzpatrick & Burns, 2019, Student outcomes section).

Scoping reviews do not require critical appraisal of the evidence included, as the purpose is generally to map what evidence exists, regardless of the quality. Some may include this step if it is relevant to the objectives, however. 

If you include critical appraisal, include also a rationale of how the appraisal aligns with the objectives; the approach used, such as tools or checklists; the process, number of reviewers and how disputes were resolved; and how the findings from the appraisal were used. This section is sometimes titled ‘Quality Appraisal’ and consists of a statement showing how the evidence sources were treated and discarded as appropriate. This is based on the exclusion criteria. A scoring system might be used in this section. 

Example 19: Quality appraisal in a scoping review

"Of the 22 included studies, 16 studies underwent methodological quality assessment using the MMAT version 2018 [reference omitted]. The remaining six [references omitted] were excluded from the quality appraisal because they were not primary studies. The 16 studies which underwent methodological quality assessment showed high methodological quality and scored between 80 and 100%. Of these studies, 15 studies [references omitted] scored 100%, and one [references omitted] scored 80%" (Wanko Keutchafo et al., 2020, Quality of evidence section).

Systematic reviews must include quality appraisal and a risk of bias assessment. These are often conducted using standard tools, which must also be described. State who assessed each study and how disputes were solved. Explain any system used to rank the studies. Any other potential source of bias needs to be explained as well as the relative risk and how this was resolved.

Example 20: Risk of bias assessment in a systematic review

"We used the EPOC 'Risk of bias' tool to assess the risk of bias (EPOC, 2015). The areas of bias addressed by the tool cover the domains outlined in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins, 2011). Two review authors (of AM, JF, and KB) independently assessed the risk of bias of each included study, and assessment was compared and reconciled, if necessary, with the help of an arbitrator. We categorised each study as having low, high, or unclear risk of bias using the EPOC ‘Risk of bias’ tool (EPOC, 2015). Any disagreements were resolved by discussion or by consulting the senior review author" (Merriel et al., 2019, Assessment of risk of bias section).

Systematic reviews and meta-analyses will have additional sections on measures of the treatment effect, which examines the effect estimates and confidence intervals of included studies.

Example 21: Effect estimates and confidence intervals

"From each study we collected the outcomes relevant to this review, regardless of whether they were the primary outcome for each individual study or not. We extracted the effect estimate and confidence intervals of the intervention from the data provided in the publication. We were unable to perform a meta‐analysis due to the heterogeneity of outcomes reported in the included studies. We presented a structured synthesis of the results as reported by the authors" (Merriel et al., 2019, Measures of treatment section).

Other typical sections in systematic reviews and meta-analyses can include assessments of heterogeneity, reporting bias, units of analysis, certainty assessments and missing data.

Example 22: Management of missing data

"Studies that did not report all data necessary to calculate an effect size were handled in one of three ways. First, authors were contacted in order to seek supplemental information to allow for standard calculations. For a subset of studies whose authors could not provide additional data, the N and mean but not SD figures were provided. However, SDs can be imputed for effect size calculations with continuous outcomes" (Fitzpatrick & Burns, 2019, Dealing with missing data section).

Example 23: Assessment of heterogeneity

"Due to the nature of this review, we expected significant statistical heterogeneity between studies. In addition, it was difficult to anticipate a priori the sources of heterogeneity. We therefore extracted all important sources of heterogeneity in the data abstraction form, which included methodological and contextual aspects of the included studies" (Merriel et al., 2019, Assessment of heterogeneity section).

In a systematic review, the implications and recommendations of your study are often included at the end of the discussion, acting as the conclusion. 

In contrast, a scoping review usually has the conclusion as a separate section. Your conclusions need to match objectives and questions you set out for the review, summarising what you found out.

Example 30: Conclusion in a scoping review

"In summary, this scoping review provides insight into the relationship between PA and SF in young people with ASD. From the current literature, PA may be related to the social interactions and behaviors of young people with ASD. This review has summarized the relevant literature regarding PA and SF and suggests future directions for research. It has become evident that PA is a viable intervention option to target some of the primary concerns associated with ASD. Further, interventions educating young people with ASD about how to engage in PA may enhance quality of life through increased PA participation and diversified social relationships" (Reinders et al., 2019, Conclusion section).

If you plan to publish your review in a journal, ensure you check their author guidelines for the required referencing style. For example, the JBI Evidence Synthesis journal requires references in Vancouver, while other journals may have their own style. 

It is a good idea to use EndNote to manage your references, as it is easy and quick to alter the reference style in a document. Read about how EndNote can assist you as a higher degree research student. 

It is encouraged now to make data freely available for others to use. This aids research integrity, accountability, and reproducibility. For a review, this may include the data extracted from studies, the full search strategy, a list of excluded sources, clean datasets in a form able to be re-analysed, metadata or analytic software code. 

Ensure you check the copyright restrictions first, because records exported from databases, for example, may include copyright material and are therefore unable to be shared. These files can be appended to the review text or shared on a site such as FigShare and linked, as shown in the example below.

Example 31: Supplementary material appended to an article

"The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2019.00120/full#supplementary-material" (Reinders et al., 2019, Supplementary material section).