Article Text

Download PDFPDF

Meta-analysis: mistake or milestone in medicine?
  1. Farid Foroutan1,2,
  2. Gordon Guyatt2,
  3. Ana Carolina Alba1,
  4. Heather Ross1
  1. 1 Ted Rogers Centre of Excellence in Heart Function, Peter Munk Cardiac Centre, University Health Network, Toronto, Ontario, Canada
  2. 2 Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada
  1. Correspondence to Farid Foroutan, Ted Rogers Centre of Excellence in Heart Function, Peter Munk Cardiac Centre, University Health Network, Toronto, ON M5G 2C4, Canada; farid.foroutan{at}

Statistics from

Since 1981, Evidence-Based Medicine (EBM) has evolved into a framework for partnering with patients to resolve or cope with health problems, taking full advantage of available research evidence.1 The modern practice of EBM relies on three fundamental principles1: (1) optimal clinical decision making requires a systematic review of the best available evidence; meta-analysis sometimes, though not always, strengthens associations, (2) using sophisticated framework for judging the quality or trustworthiness of evidence and (3) consideration of trade-offs between benefits, risks, burden and costs, on the basis of patients’ values and preferences.

Despite EBM’s wide acceptance over the past 30 years, scepticism remains. Specifically, sceptics of EBM’s first principle believe that our knowledge of cardiovascular medicine would be no different with or without the utilisation of systematic reviews and meta-analyses for ascertainment of evidence.2 Such beliefs are not unsupported on social media platforms such as Twitter, by means of tweets and retweets, over 270 000 knowledge users have expressed scepticism against systematic reviews and meta-analyses.3 In this editorial, we present a meta-analyst’s perspective, providing evidence to support EBM’s first principle regarding the use of systematic reviews and meta-analyses to better inform the cardiology audience.

Advantages of systematic reviews and meta-analyses

For any well-formulated research question, systematic reviews and meta-analyses can identify all the available relevant evidence (systematic search), critically appraise the risk of bias of the identified evidence, combine the results to increase statistical power (meta-analysis when appropriate), identify sources of variation across studies (subgroup analyses) and rate the overall quality of the evidence (certainty in the results).

Increased statistical power and precise estimates

History demonstrates the advantage of systematic reviews and meta-analyses over narrative reviews. In 1992, Antman et al 4 compared expert recommendations for management of patients with myocardial infarction (MI) with the best evidence available at the time. The authors conducted a retrospective cumulative meta-analysis (performing a new meta-analysis when the results of a new clinical trial are published). They showed that if a meta-analysis assessing the efficacy of thrombolytic therapy on reduction of mortality post-MI had been performed on the first 10 trials, it would have shown a statistically significant reduction in mortality. By the publication of the 10th trial, the majority of narrative reviews and textbooks, however, either recommended against or did not mention the use of thrombolytic therapies. The clear reduction in mortality was not obvious at the time because no meta-analysis was done. Failure to show the benefit of thrombolytics, which would have been clear and evident through meta-analysis, resulted in a plethora of randomised controlled trials (RCTs) enrolling >40 000 patients, half of whom did not receive life-saving therapy. It was only after the publication of the 70th RCT that textbooks and narrative reviews consistently recommend the use of thrombolytic therapy for MI.4 This delay resulted in exposure of patients to inadequate treatment and significant resource waste, when the benefit of thrombolytic therapies could have been appreciated by meta-analyses at a much earlier stage.

Another example is the history behind the use of lidocaine as prophylactic therapy post-MI. Randomised trials failed to show benefit for the use of lidocaine post-MI. During 1970–1978, however, almost all recommendations supported the use of lidocaine, despite the point estimates of trials suggesting an increased mortality risk. From 1970 to 1988, 15 RCTs assessed the efficacy of lidocaine for prevention of death post-MI, and only after publication of the 14th and 15th RCTs did expert recommendation start to change, in 1985, from supporting to recommending against the use of lidocaine.4 These examples highlight the limitation of narrative reviews; since the advent of meta-analysis, one can find no such egregious errors in recommendations in the face of compelling evidence.

Transparent assessment of bias and confidence on the available evidence

In contrast to narrative reviews, rigorous systematic reviews carefully document issues related to risk of bias of the individual studies. For instance, systematic reviews of randomised trials will carefully document concealment of randomization, blinding of patients, clinicians, data collectors and adjudicators of outcome. Moreover, they will conduct this assessment in duplicate to ensure accuracy. Few, if any traditional narrative reviews, conducted such comprehensive assessments of risk of bias or included safeguards against inaccurate judgements. Systematic reviews and meta-analyses address ‘certainty’ or confidence in their results, using tools such as the Grading of Recommendations Assessment, Development and Evaluation (GRADE).5 GRADE is highly recognised, recommended and endorsed by over 100 healthcare societies globally to assess certainty in the evidence. Certainty in the evidence can be decreased due to risk of bias at study level, imprecision in pooled effect estimate, inconsistency across study results, indirectness and publication bias and can be increased due to large effect, dose–response relation and residual confounding against the effect. By using GRADE, authors of systematic reviews and meta-analyses remain transparent to the issues inherent to the available evidence on their research question of interest. Traditional narrative reviews often neglected issues of risk of bias and never applied systematic and replicable approaches.

Addressing reasons for differences in effect estimates

Rigorous systematic reviews will investigate potential explanations for heterogeneity (inconsistent results) across the primary studies. A priori hypotheses will address issues of risk of bias, differences in patients enrolled, interventions administered and how outcomes were measured. Meta-analyses allow application of statistical approaches–rather than intuitive judgements open to the biases of the reviewer–to determine if apparent explanations of differences could be explained by chance. Finally, rigorous reviews apply a set of carefully developed criteria to determine if apparent explanations (ie, the identification of subgroup effects) are more likely to be accurate or spurious.6 This careful attention to explaining heterogeneity deals with objections to systematic reviews and meta-analyses placing equal value on studies with different characteristics (eg, low and high risks of bias studies or different therapy doses or study duration) identifying the best and most accurate available evidence.

Examples of recent meta-analysis impacting guideline recommendations

The meticulous and reproducible methodology of systematic reviews and meta-analysis has generated dramatic conclusions that the cardiology community had not inferred from individual studies. For example, although guidelines make strong recommendation for use of implantable cardioverter defibrillator (ICD) in patients with systolic dysfunction and ischaemic cardiomyopathy, the use of ICDs in non-ischaemic cardiomyopathy remains controversial. A recently published, but underpowered, RCT showed no significant reduction in mortality in non-ischaemic cardiomyopathy.7 However, after conducting a meta-analysis, it was evident that the use of ICDs significantly reduced the risk of mortality in patients with systolic dysfunction and non-ischaemic cardiomyopathy8 (figure 1).

Figure 1

Random-effects meta-analysis evaluating the effect of ICD therapy on all-cause mortality in patients with non-ischaemic cardiomyopathy. Image from Alba et al. 8 ICD, implantable cardioverter defibrillator.

Two systematic reviews and meta-analyses (one for efficacy9 and one for prognosis10) provided the evidence for a guideline panel to make recommendations for use of transcatheter aortic valve implantation (TAVI) and surgical aortic valve replacement (SAVR).11 The meta-analysis assessing the efficacy of these two therapies showed that TAVI, originally designed for patients at high risk for SAVR, can be extended to low-surgical to moderate-surgical risk patients.10 The prognostic review showed that SAVR valves are durable up to 10 years postimplantation, whereas durability of TAVI valves is highly uncertain. This resulted in strong recommendation for use of TAVI in patients older than 85 years and strong recommendation for SAVR in patients under 65 years.11

Systematic reviews and meta-analyses can provide important new evidence by narrowing CIs. For example, the use of aspirin for the primary prevention of cardiovascular disease in men has been evaluated by five RCTs,12 three of which showed a reduction in risk of MI with use of aspirin. The pooled effect estimate of the meta-analysis also suggested a reduction in risk of MI but showed more precise estimates (the meta-analysis showed a pooled 14% to 46% relative risk reduction, whereas the two largest trials suggested 1% to 53% risk reduction). Patients with a 10% risk over a decade may view the evidence differently if their benefit may be as low as 1/1000 fewer MI over a decade (the lower boundary of the CI from the trials) versus 14/1000 (the lower boundary of the CI from the meta-analysis). Thus, narrowing CIs is important.

With the advent of new pharmaceutical agents for treatment of cardiovascular diseases, patients and clinicians are provided with many choices for therapy. Randomised clinical trials may not always be available comparing all medications to one another. Questions may arise with regard to the efficacy of each drug in comparison with the others. Network meta-analysis (NMA), which addresses multiple treatments for the same condition and identifies superior and inferior alternatives, uses both direct and indirect evidence to generate best estimates of effect. For instance, an NMA by Psaty et al assessed efficacy and safety of antihypertensive therapies as first-line agents for prevention of cardiovascular disease morbidity and mortality.13 Of alpha-blockers, ACE inhibitors, beta-blockers, angiotensin receptor blockers, calcium channel blockers and low-dose diuretics, the authors identified low-dose diuretics as the most effective at preventing all adverse cardiovascular outcomes (coronary heart disease, congestive heart failure, stroke, cardiovascular disease events and cardiovascular disease mortality). In the future, as the number of therapies for the same cardiovascular diseases increases, NMA is likely to play an increasingly important role.

Not all that glitters is gold

Not all systematic reviews and meta-analyses are rigorously conducted. Despite the well-established methodology, methodologically flawed meta-analyses are frequently published. This issue is not specific to meta-analyses, but one that is generalizable to all study designs including randomised trials and observational studies. The flaw isn’t within the study type but rather the investigators who fail to adhere to optimal standards. Authors may make a wide variety of mistakes, including inappropriately presenting strong conclusions despite a small number of events. Readers should be sceptical of large effects with few events and small-sample sizes. Indeed, the GRADE guidance makes exactly this point.14


In conclusion, systematic reviews and meta-analyses are powerful and informative. They should not be condemned as a form of medical fake news.2 There will always be poorly conducted randomised trials of little use (and potentially misleading)—the same is true of meta-analysis. When faced with a meta-analysis, clinicians should look at its methods and results critically—and use the guidance available to help them in doing so.1 There is no doubt, however, that optimal clinical care requires knowledge of the best available evidence, and systematic summaries are required to be confident that clinicians indeed have access to that evidence. With rigorous systematic summaries, we can be confident as we partner with patients to ensure that they achieve the best possible outcomes.



  • Contributors All authors contributed to the drafting and revising of the submitted manuscript.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient consent Not required.

  • Provenance and peer review Commissioned; internally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.