How do I choose a meta analysis topic?

Meta-Analytic Methodology for Basic Research: A Practical Guide

• 1Faculty of Dentistry, McGill University, Montreal, QC, Canada
• 2Shriners Hospital for Children-Canada, Montreal, QC, Canada

Basic life science literature is rich with information, however methodically quantitative attempts to organize this information are rare. Unlike clinical research, where consolidation efforts are facilitated by systematic review and meta-analysis, the basic sciences seldom use such rigorous quantitative methods. The goal of this study is to present a brief theoretical foundation, computational resources and workflow outline along with a working example for performing systematic or rapid reviews of basic research followed by meta-analysis. Conventional meta-analytic techniques are extended to accommodate methods and practices found in basic research. Emphasis is placed on handling heterogeneity that is inherently prevalent in studies that use diverse experimental designs and models. We introduce MetaLab, a meta-analytic toolbox developed in MATLAB R2016b which implements the methods described in this methodology and is provided for researchers and statisticians at Git repository [//github.com/NMikolajewicz/MetaLab]. Through the course of the manuscript, a rapid review of intracellular ATP concentrations in osteoblasts is used as an example to demonstrate workflow, intermediate and final outcomes of basic research meta-analyses. In addition, the features pertaining to larger datasets are illustrated with a systematic review of mechanically-stimulated ATP release kinetics in mammalian cells. We discuss the criteria required to ensure outcome validity, as well as exploratory methods to identify influential experimental and biological factors. Thus, meta-analyses provide informed estimates for biological outcomes and the range of their variability, which are critical for the hypothesis generation and evidence-driven design of translational studies, as well as development of computational models.

Introduction

Evidence-based medical practice aims to consolidate best research evidence with clinical and patient expertise. Systematic reviews and meta-analyses are essential tools for synthesizing evidence needed to inform clinical decision making and policy. Systematic reviews summarize available literature using specific search parameters followed by critical appraisal and logical synthesis of multiple primary studies [Gopalakrishnan and Ganeshkumar, 2013]. Meta-analysis refers to the statistical analysis of the data from independent primary studies focused on the same question, which aims to generate a quantitative estimate of the studied phenomenon, for example, the effectiveness of the intervention [Gopalakrishnan and Ganeshkumar, 2013]. In clinical research, systematic reviews and meta-analyses are a critical part of evidence-based medicine. However, in basic science, attempts to evaluate prior literature in such rigorous and quantitative manner are rare, and narrative reviews are prevalent. The goal of this manuscript is to provide a brief theoretical foundation, computational resources and workflow outline for performing a systematic or rapid review followed by a meta-analysis of basic research studies.

Meta-analyses can be a challenging undertaking, requiring tedious screening and statistical understanding. There are several guides available that outline how to undertake a meta-analysis in clinical research [Higgins and Green, 2011]. Software packages supporting clinical meta-analyses include the Excel plugins MetaXL [Barendregt and Doi, 2009] and Mix 2.0 [Bax, 2016], Revman [Cochrane Collaboration, 2011], Comprehensive Meta-Analysis Software [CMA [Borenstein et al., 2005]], JASP [JASP Team, 2018] and MetaFOR library for R [Viechtbauer, 2010]. While these packages can be adapted to basic science projects, difficulties may arise due to specific features of basic science studies, such as large and complex datasets and heterogeneity in experimental methodology. To address these limitations, we developed a software package aimed to facilitate meta-analyses of basic research, MetaLab in MATLAB R2016b, with an intuitive graphical interface that permits users with limited statistical and coding background to proceed with a meta-analytic project. We organized MetaLab into six modules [Figure 1], each focused on different stages of the meta-analytic process, including graphical-data extraction, model parameter estimation, quantification and exploration of heterogeneity, data-synthesis, and meta-regression.

In the present manuscript, we describe each step of the meta-analytic process with emphasis on specific considerations made when conducting a review of basic research. The complete workflow of parameter estimation using MetaLab is demonstrated for evaluation of intracellular ATP content in osteoblasts [OB [ATP]ic dataset] based on a rapid literature review. In addition, the features pertaining to larger datasets are explored with the ATP release kinetics from mechanically-stimulated mammalian cells [ATP release dataset] obtained as a result of a systematic review in our prior work [Mikolajewicz et al., 2018].

MetaLab can be freely accessed at Git repository [//github.com/NMikolajewicz/MetaLab], and a detailed documentation of how to use MetaLab together with a working example is available in the Supporting materials.

Validity of Evidence in the Basic Sciences

To evaluate the translational potential of basic research, the validity of evidence must first be assessed, usually by examining the approach taken to collect and evaluate the data. Studies in the basic sciences are broadly grouped as hypothesis-generating and hypothesis-driven. The former tend to be small-sampled proof-of-principle studies and are typically exploratory and less valid than the latter. An argument can even be made that studies that report novel findings fall into this group as well, since their findings remain subject to external validation prior to being accepted by the broader scientific community. Alternatively, hypothesis-driven studies build upon what is known or strongly suggested by earlier work. These studies can also validate prior experimental findings with incremental contributions. Although such studies are often overlooked and even dismissed due to a lack of substantial novelty, their role in external validation of prior work is critical for establishing the translational potential of findings.

Another dimension to the validity of evidence in the basic sciences is the selection of experimental model. The human condition is near-impossible to recapitulate in a laboratory setting, therefore experimental models [e.g., cell lines, primary cells, animal models] are used to mimic the phenomenon of interest, albeit imperfectly. For these reasons, the best quality evidence comes from evaluating the performance of several independent experimental models. This is accomplished through systematic approaches that consolidate evidence from multiple studies, thereby filtering the signal from the noise and allowing for side-by-side comparison. While systematic reviews can be conducted to accomplish a qualitative comparison, meta-analytic approaches employ statistical methods which enable hypothesis generation and testing. When a meta-analysis in the basic sciences is hypothesis-driven, it can be used to evaluate the translational potential of a given outcome and provide recommendations for subsequent translational- and clinical-studies. Alternatively, if meta-analytic hypothesis testing is inconclusive, or exploratory analyses are conducted to examine sources of inconsistency between studies, novel hypotheses can be generated, and subsequently tested experimentally. Figure 2 summarizes this proposed framework.

Steps in Quantitative Literature Review

All meta-analytic efforts prescribe to a similar workflow, outlined as follows:

1] Formulate research question

Define primary and secondary objectives

Determine breadth of question

2] Identify relevant literature

Construct search strategy: rapid or systematic search

Screen studies and determine eligibility

3] Extract and consolidate study-level data

Extract data from relevant studies

Collect relevant study-level characteristics and experi-mental covariates

Evaluate quality of studies

Estimate model parameters for complex relation-ships [optional]

4] Data appraisal and preparation

Compute appropriate outcome measure

Evaluate extent of between-study inconsistency [heterogeneity]

Perform relevant data transformations

Select meta-analytic model

5] Synthesize study-level data into summary measure

Pool data and calculate summary measure and confidence interval

6] Exploratory analyses

Explore potential sources of heterogeneity [ex. biological or experimental]

Subgroup and meta-regression analyses

7] Knowledge synthesis

Interpret findings

Provide recommendations for future work

Meta-Analysis Methodology

Search and Selection Strategies

The first stage of any review involves formulating a primary objective in the form of a research question or hypothesis. Reviewers must explicitly define the objective of the review before starting the project, which serves to reduce the risk of data dredging, where reviewers later assign meaning to significant findings. Secondary objectives may also be defined; however, precaution must be taken as the search strategies formulated for the primary objective may not entirely encompass the body of work required to address the secondary objective. Depending on the purpose of a review, reviewers may choose to undertake a rapid or systematic review. While the meta-analytic methodology is similar for systematic and rapid reviews, the scope of literature assessed tends to be significantly narrower for rapid reviews permitting the project to proceed faster.

Systematic Review and Meta-Analysis

Systematic reviews involve comprehensive search strategies that enable reviewers to identify all relevant studies on a defined topic [DeLuca et al., 2008]. Meta-analytic methods then permit reviewers to quantitatively appraise and synthesize outcomes across studies to obtain information on statistical significance and relevance. Systematic reviews of basic research data have the potential of producing information-rich databases which allow extensive secondary analysis. To comprehensively examine the pool of available information, search criteria must be sensitive enough not to miss relevant studies. Key terms and concepts that are expressed as synonymous keywords and index terms, such as Medical Subject Headings [MeSH], must be combined using Boolean operators AND, OR and NOT [Ecker and Skelly, 2010]. Truncations, wildcards, and proximity operators can also help refine a search strategy by including spelling variations and different wordings of the same concept [Ecker and Skelly, 2010]. Search strategies can be validated using a selection of expected relevant studies. If the search strategy fails to retrieve even one of the selected studies, the search strategy requires further optimization. This process is iterated, updating the search strategy in each iterative step until the search strategy performs at a satisfactory level [Finfgeld-Connett and Johnson, 2013]. A comprehensive search is expected to return a large number of studies, many of which are not relevant to the topic, commonly resulting in a specificity of 50] [Gavaghan et al., 2000; Higgins et al., 2003]. Additionally, the Qtotal statistic is not a measure of the magnitude of heterogeneity due to its inherent dependence on the number of studies. To address this limitation, H2 heterogeneity statistics was developed as the relative excess in Qtotal over degrees of freedom df:

H2 is independent of the number of studies in the meta-analysis and is indicative of the magnitude of heterogeneity [Higgins and Thompson, 2002]. For values