Advantages of systematic reviews and meta-analyses

For any well-formulated research question, systematic reviews and meta-analyses can identify all the available relevant evidence (systematic search), critically appraise the risk of bias of the identified evidence, combine the results to increase statistical power (meta-analysis when appropriate), identify sources of variation across studies (subgroup analyses) and rate the overall quality of the evidence (certainty in the results).

Increased statistical power and precise estimates

History demonstrates the advantage of systematic reviews and meta-analyses over narrative reviews. In 1992, Antman et al 4 compared expert recommendations for management of patients with myocardial infarction (MI) with the best evidence available at the time. The authors conducted a retrospective cumulative meta-analysis (performing a new meta-analysis when the results of a new clinical trial are published). They showed that if a meta-analysis assessing the efficacy of thrombolytic therapy on reduction of mortality post-MI had been performed on the first 10 trials, it would have shown a statistically significant reduction in mortality. By the publication of the 10th trial, the majority of narrative reviews and textbooks, however, either recommended against or did not mention the use of thrombolytic therapies. The clear reduction in mortality was not obvious at the time because no meta-analysis was done. Failure to show the benefit of thrombolytics, which would have been clear and evident through meta-analysis, resulted in a plethora of randomised controlled trials (RCTs) enrolling >40 000 patients, half of whom did not receive life-saving therapy. It was only after the publication of the 70th RCT that textbooks and narrative reviews consistently recommend the use of thrombolytic therapy for MI.4 This delay resulted in exposure of patients to inadequate treatment and significant resource waste, when the benefit of thrombolytic therapies could have been appreciated by meta-analyses at a much earlier stage.

Another example is the history behind the use of lidocaine as prophylactic therapy post-MI. Randomised trials failed to show benefit for the use of lidocaine post-MI. During 1970–1978, however, almost all recommendations supported the use of lidocaine, despite the point estimates of trials suggesting an increased mortality risk. From 1970 to 1988, 15 RCTs assessed the efficacy of lidocaine for prevention of death post-MI, and only after publication of the 14th and 15th RCTs did expert recommendation start to change, in 1985, from supporting to recommending against the use of lidocaine.4 These examples highlight the limitation of narrative reviews; since the advent of meta-analysis, one can find no such egregious errors in recommendations in the face of compelling evidence.

Transparent assessment of bias and confidence on the available evidence

In contrast to narrative reviews, rigorous systematic reviews carefully document issues related to risk of bias of the individual studies. For instance, systematic reviews of randomised trials will carefully document concealment of randomization, blinding of patients, clinicians, data collectors and adjudicators of outcome. Moreover, they will conduct this assessment in duplicate to ensure accuracy. Few, if any traditional narrative reviews, conducted such comprehensive assessments of risk of bias or included safeguards against inaccurate judgements. Systematic reviews and meta-analyses address ‘certainty’ or confidence in their results, using tools such as the Grading of Recommendations Assessment, Development and Evaluation (GRADE).5 GRADE is highly recognised, recommended and endorsed by over 100 healthcare societies globally to assess certainty in the evidence. Certainty in the evidence can be decreased due to risk of bias at study level, imprecision in pooled effect estimate, inconsistency across study results, indirectness and publication bias and can be increased due to large effect, dose–response relation and residual confounding against the effect. By using GRADE, authors of systematic reviews and meta-analyses remain transparent to the issues inherent to the available evidence on their research question of interest. Traditional narrative reviews often neglected issues of risk of bias and never applied systematic and replicable approaches.

Addressing reasons for differences in effect estimates

Rigorous systematic reviews will investigate potential explanations for heterogeneity (inconsistent results) across the primary studies. A priori hypotheses will address issues of risk of bias, differences in patients enrolled, interventions administered and how outcomes were measured. Meta-analyses allow application of statistical approaches–rather than intuitive judgements open to the biases of the reviewer–to determine if apparent explanations of differences could be explained by chance. Finally, rigorous reviews apply a set of carefully developed criteria to determine if apparent explanations (ie, the identification of subgroup effects) are more likely to be accurate or spurious.6 This careful attention to explaining heterogeneity deals with objections to systematic reviews and meta-analyses placing equal value on studies with different characteristics (eg, low and high risks of bias studies or different therapy doses or study duration) identifying the best and most accurate available evidence.

Examples of recent meta-analysis impacting guideline recommendations

The meticulous and reproducible methodology of systematic reviews and meta-analysis has generated dramatic conclusions that the cardiology community had not inferred from individual studies. For example, although guidelines make strong recommendation for use of implantable cardioverter defibrillator (ICD) in patients with systolic dysfunction and ischaemic cardiomyopathy, the use of ICDs in non-ischaemic cardiomyopathy remains controversial. A recently published, but underpowered, RCT showed no significant reduction in mortality in non-ischaemic cardiomyopathy.7 However, after conducting a meta-analysis, it was evident that the use of ICDs significantly reduced the risk of mortality in patients with systolic dysfunction and non-ischaemic cardiomyopathy8 (figure 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Random-effects meta-analysis evaluating the effect of ICD therapy on all-cause mortality in patients with non-ischaemic cardiomyopathy. Image from Alba et al. 8 ICD, implantable cardioverter defibrillator.

Two systematic reviews and meta-analyses (one for efficacy9 and one for prognosis10) provided the evidence for a guideline panel to make recommendations for use of transcatheter aortic valve implantation (TAVI) and surgical aortic valve replacement (SAVR).11 The meta-analysis assessing the efficacy of these two therapies showed that TAVI, originally designed for patients at high risk for SAVR, can be extended to low-surgical to moderate-surgical risk patients.10 The prognostic review showed that SAVR valves are durable up to 10 years postimplantation, whereas durability of TAVI valves is highly uncertain. This resulted in strong recommendation for use of TAVI in patients older than 85 years and strong recommendation for SAVR in patients under 65 years.11

Systematic reviews and meta-analyses can provide important new evidence by narrowing CIs. For example, the use of aspirin for the primary prevention of cardiovascular disease in men has been evaluated by five RCTs,12 three of which showed a reduction in risk of MI with use of aspirin. The pooled effect estimate of the meta-analysis also suggested a reduction in risk of MI but showed more precise estimates (the meta-analysis showed a pooled 14% to 46% relative risk reduction, whereas the two largest trials suggested 1% to 53% risk reduction). Patients with a 10% risk over a decade may view the evidence differently if their benefit may be as low as 1/1000 fewer MI over a decade (the lower boundary of the CI from the trials) versus 14/1000 (the lower boundary of the CI from the meta-analysis). Thus, narrowing CIs is important.

With the advent of new pharmaceutical agents for treatment of cardiovascular diseases, patients and clinicians are provided with many choices for therapy. Randomised clinical trials may not always be available comparing all medications to one another. Questions may arise with regard to the efficacy of each drug in comparison with the others. Network meta-analysis (NMA), which addresses multiple treatments for the same condition and identifies superior and inferior alternatives, uses both direct and indirect evidence to generate best estimates of effect. For instance, an NMA by Psaty et al assessed efficacy and safety of antihypertensive therapies as first-line agents for prevention of cardiovascular disease morbidity and mortality.13 Of alpha-blockers, ACE inhibitors, beta-blockers, angiotensin receptor blockers, calcium channel blockers and low-dose diuretics, the authors identified low-dose diuretics as the most effective at preventing all adverse cardiovascular outcomes (coronary heart disease, congestive heart failure, stroke, cardiovascular disease events and cardiovascular disease mortality). In the future, as the number of therapies for the same cardiovascular diseases increases, NMA is likely to play an increasingly important role.

Not all that glitters is gold

Not all systematic reviews and meta-analyses are rigorously conducted. Despite the well-established methodology, methodologically flawed meta-analyses are frequently published. This issue is not specific to meta-analyses, but one that is generalizable to all study designs including randomised trials and observational studies. The flaw isn’t within the study type but rather the investigators who fail to adhere to optimal standards. Authors may make a wide variety of mistakes, including inappropriately presenting strong conclusions despite a small number of events. Readers should be sceptical of large effects with few events and small-sample sizes. Indeed, the GRADE guidance makes exactly this point.14