Article Text

PDF

Development and validation of a Bayesian index for predicting major adverse cardiac events with percutaneous transluminal coronary angioplasty

Abstract

OBJECTIVE To create a risk model for predicting major adverse complicating events of percutaneous transluminal coronary angioplasty (PTCA), and to test the accuracy of the model on a prospective cohort of patients

SETTING Tertiary cardiac centre

METHODS Available software can predict probabilities of events using Bayes's theorem. To establish the accuracy of these predictive tools, a Bayes table was created to evaluate major adverse complicating events (MACE)—death, emergency coronary artery bypass grafting (CABG), or Q wave infarct occurring during the in-patient episode—on the first 1500 patients in the department PTCA database (development group); the predictive value of this model was then tested with the subsequent 1000 patients (evaluation group). The following probabilities were assessed to determine their association with MACE: age, sex, left ventricular function, American Heart Association lesion morphology classification, cardiogenic shock, previous CABG, diabetes, hypertension, multivessel PTCA.

MAIN OUTCOME MEASURES To establish the discriminatory ability of the predictive index, calibration plots and receiver operating characteristic (ROC) curves were obtained to compare the development and evaluation groups.

RESULTS The ROC curve plotted to determine the discriminatory value of the Bayesian table created from the development group (n = 1500) in predicting MACE in the evaluation group (n = 1000) showed a moderately predictive area under the curve of 0.76 (SEM 0.07). This predictive accuracy was confirmed with separately constructed calibration plots.

CONCLUSIONS Accurate predictions of MACE can be identified in populations undergoing percutaneous intervention. The database used allows operators to obtain consent from patients appropriately from their own experience rather than from other published data. If a national PTCA database existed along similar lines, individual operators and interventional centres could compare themselves with nationally available data.

  • percutaneous transluminal coronary angioplasty
  • Bayesian risk
  • outcome prediction

Statistics from Altmetric.com

There is a low but significant incidence of major adverse cardiac events (MACE)—acute myocardial infarction, emergency coronary artery bypass grafting (CABG), and death—in patients undergoing percutaneous transluminal coronary angioplasty (PTCA). The current evaluation of risk for these major complications arising from PTCA is based on studies of PTCA performed over a decade ago in single high volume centres.1-3 These data are probably no longer accurate or applicable to UK centres, nor do they necessarily apply to individual operators or to their case mix. For example, cardiac centres which routinely undertake PTCA in high risk cases, such as cardiogenic shock, may have significantly higher MACE rates compared with centres with a more conservative approach. National publication of MACE rates will inevitably reflect differences between major UK centres, which—in the absence of risk stratification—purchasers of healthcare and the public may be tempted to interpret at face value. With increasing pressure for the medical profession to be more accountable, it seems appropriate to evaluate methods of PTCA risk assessment which reflect current practice, specific centres, case mix, and individual practitioners.

Our objectives in this study were: first, to determine a risk model for predicting MACE from our own database; and second, to test the accuracy of this model on a prospective cohort of patients.

Methods

Since January 1995, we have collected data prospectively on all patients undergoing PTCA at our institution (Patient Analysis and Tracking System (PATS), Dendrite Clinical Systems Reading, UK). The clinical characteristics for multiple relevant clinical variables were entered onto the database at the time of the procedure. The software available with the PATS database can be used to predict the probability of specific events using Bayes's theorem. In order to validate the predictive accuracy of our variables, data from two groups of patients were analysed. A Bayes table was created to evaluate MACE on our first 1500 procedures (development group). The following clinical variables were assessed to determine their influence on MACE: age, sex, left ventricular function (defined by the ejection fraction on left ventricular angiography: good, > 50%; moderate, 30–49%; poor, < 30%), American Heart Association classification of lesion morphology, clinical presentation (stable angina, unstable angina, primary angioplasty for acute myocardial infarction, failed thrombolysis, cardiogenic shock), previous CABG, diabetes mellitus, unequivocal hypertension, renal failure (plasma creatinine > 200 μmol/l at time of PTCA), and multivessel PTCA.

The outcome measures of interest were defined as one or more of the following:

  • death occurring at any time during the hospital period;

  • Q wave myocardial infarction, confirmed by the development of pathological Q waves in the relevant leads of a 12 lead ECG, associated with a rise in serum creatine kinase MB to at least twice normal during the hospital stay; if patients were treated during the evolution of a myocardial infarct (PAMI, failed thrombolysis), this was not considered to be a Q wave infarct complicating the PTCA procedure;

  • emergency CABG as a result of complications arising from the PTCA procedure.

The predictive value of this model was then tested with the subsequent 1000 procedures (evaluation group) on the PTCA database. In order to establish the discriminatory ability of the predictive index, calibration plots and receiver operating characteristic (ROC) curves were constructed.

STATISTICAL ANALYSIS

The methods used to develop this model are based on Bayes's theorem, which proposes that future outcome can be predicted from known outcomes (appendix ).4 5 The method of prediction uses an algorithm which computes the probability that a given outcome will occur for a patient with certain risk factors. The algorithm is based on the odds ratio measured on factors similar to the given patient. There are two relevant points to consider when using these methods.

First, the method assumes independent risk factors and therefore there is a tendency to overestimate risk in a patient with multiple risk factors. However, Bayes's theorem has been used extensively when exploring the influence of multiple risk factors, where it is accepted there is a significant degree of covariance. Previous work has minimised this problem by carefully grouping factors into clinically related categories. This has enabled a damping factor to be incorporated within the software to accommodate this and it has been shown that this significantly improves the accuracy between the actual and expected outcomes.6

Second, patients with missing data are assumed to reflect the same variable distribution as the other patients in the cohort. This means the whole sample can be used, eliminating any reduction in power if significant numbers were omitted.

DEVELOPMENT OF A BAYES TABLE

In order to take advantage of the non-linear effect of continuous variables (for example, age) on outcomes, multiple probability tables are constructed for different categorical ranges. The model divides the continuous variables into clinically relevant categories, computes separate probabilities for each independently, and combines those probabilities into one table.

Table 1 shows the influence of various clinical variables on the incidence of MACE. The contribution column indicates the difference between the calculated MACE, if this was the only known risk factor, and the average for all patients. For example, a patient of 77 years would have a contribution of 2.90% plus an expected MACE rate of 5.8%. In the presence of multiple risk factors, the result is not the sum of the contributions but what is computed by the Bayesian equation when all the odds ratios are considered.

Table 1

Bayes table to determine the influence of clinical variables on the incidence of major adverse complicating events (MACE) in patients undergoing percutaneous transluminal coronary angioplasty in 1500 consecutive procedures (development group)

VALIDATION METHODS

Receiver operating characteristic curve

To assess the ability of the model to predict MACE for individual patients, an ROC curve was computed. This was achieved by comparing the expected and observed incidence of MACE from the two groups and converting the risk per cent for each patient to a yes/no (true/false) answer. The area under the curve is presented as a proportion of 1—that is, an ROC of 1.0 would indicate a perfect system for prediction, and that of 0.5 would be no better than the toss of a coin.

Calibration plot

As a second independent means of measuring the predictive accuracy of this model, Bayesian risk groups were computed for the development group. This was achieved by calculating the expected probability of MACE within the development group, subdividing them into different risk groups (0–2%, 2–5%, etc incidence of MACE), and then plotting the predicted event rate against the actual event rate in the evaluation group (calibration plot).

Results

DEVELOPMENT GROUP

The first 1500 procedures were taken from 1 January 1995 until 31 December 1996. Within this group, a 5.8% incidence of MACE was recorded: death, 1.7%; emergency CABG, 2.1%; Q wave myocardial infarction, 2.0%. A Bayes table was created to determine the prevalence of possible risk factors associated with MACE (table 1).

PREDICTORS OF MAJOR COMPLICATIONS IN THE DEVELOPMENT GROUP

Table 1 shows the distribution of clinical variables in relation to MACE, followed by a calculated probability of a specific factor influencing a MACE event (weighted probability).

The following clinical variables were found to be predictors of MACE, ranked in order of importance: presentation with cardiogenic shock, ejection fraction < 30%, thrombolysis given within the last 24 hours, renal impairment (creatinine > 200 μmol/l), and age > 76 years. All other factors including diabetes mellitus and hypertension had little influence on the prevalence of MACE (see table 1).

VALIDATION OF THE DISCRIMINATORY ABILITY OF THE DEVELOPMENT GROUP BY APPLICATION TO AN EVALUATION GROUP

To validate the predictive ability of our weighted risk indices we tested them in our second group of 1000 PTCA procedures (evaluation group) taken from 1 January 1997 to 1 March 1998. The two groups were well matched, other than the incidence of abciximab administration which was greater in the evaluation group (see table2).

Table 2

Characteristics of the development and evaluation groups

The actual prevalence of MACE for this second group was determined and compared with our predicted prevalence of events.

ROC CURVE AND CALIBRATION PLOT TO DETERMINE THE DISCRIMINATORY VALUE OF THE PREDICTIVE TOOLS

We found that the area under the curve of our ROC was moderately predictive of events, at 0.76 (SEM 0.07) (fig 1). To establish further the accuracy of these predictive tools, a calibration plot was calculated comparing the predicted MACE rate in different risk groups created from the development group with the same risk groups created from the evaluation data. Figure 2 shows an accurate assessment of risk in the low risk groups, but it was less accurate in patients with greater risk, probably because of the smaller numbers of patients involved.

Figure 1

Receiver operating characteristic (ROC) curve plotted to determine the discriminatory value of the Bayesian table created from the development group (n = 1500) in predicting major adverse cardiac events (MACE) in the evaluation group (n = 1000). The area under the ROC curve is a moderately predictive 0.76 (SEM 0.07).

Figure 2

Bayes's risk groups were calculated from the development group and plotted against the incidence of major adverse cardiac events (MACE). These were compared against the actual event rate of MACE using the same risk groups created from the evaluation group. This calibration plot shows an accurate assessment of risk in the low risk groups, but it was less accurate for patients with greater risk.

Discussion

Highlighting the potential risks to patients undergoing invasive coronary procedures is an increasingly important component of good clinical practice. At present, consent procedures in patients undergoing PTCA in the UK appear to be haphazard.7 This may be because the perceived incidence of MACE is relatively low, and because many centres feel that more specific and detailed consent is unnecessary. Certainly our data support the very small risk of MACE in low risk patients undergoing single vessel PTCA. However, recent medicolegal events have shown that issues over informed consent are now paramount in the public domain and need to be accurately and openly clarified by the medical profession, not least by practitioners of PTCA.

Available software now exists for cardiology departments to collect detailed prospective data about their interventional experience. The groups used to evaluate the accuracy of the predictive model were well matched. There was a small but increased use of abciximab in the evaluation group which may have influenced the incidence of MACE; however, the numbers within the cohort were too small for this to be likely. Our data have shown that predictors of MACE can be identified from unique populations of angioplasty patients, and that these factors can be weighted accordingly. By doing so, consent can be tailored to an individual patient and individual operators rather than using wholesale, often transatlantic, quotations of risk which may be inaccurate and too sanguine for departments with varying case mix.

The software package available with the PATS database allows a Bayes table to be created from individual centres' prospective data. This table predicts the probability of any one event occurring to an individual, based on probabilities acquired from past experience. However, the incidence of MACE in overall angioplasty practice is low, and it might be suggested that this makes it more difficult to predict complications accurately. However, when we tested the accuracy of our Bayes table in predicting MACE in our first 1500 patients against our subsequent 1000 patients, using calibration plots and ROC curves, it proved to have an acceptable discriminatory value, which compares favourably with other published models of risk prediction for PTCA.8 The tools were less discriminatory for patients undergoing high risk procedures, mainly because the prevalence of these cases is comparatively low. As our database develops, it is likely that the discriminatory accuracy in this important group of patients will improve.

The Bayes technique applied in this study is essentially a univariable method. However, Bayes's theorem has been used extensively when exploring the influence of multiple risk factors, where it is accepted there is a significant degree of covariance. Other methods of assessing the influence of different risk factors on angioplasty outcome have used logistic regression, and Cox analyses have been published previously.8 Our results compare favourably with these.

At present it is not mandatory to use supporting data to detail the potential risks attached to a PTCA procedure. Is it acceptable to tell a patient that this procedure has an “increased risk,” or is it more appropriate for a cardiologist to say “in our hands, the risk of MACE for this procedure is y% compared with an overall risk of x% for all comers”? There is no doubt in our mind that the latter view will increasingly predominate.

There are further advantages in keeping detailed prospective data about angioplasty practice. Interventional cardiology is a rapidly developing field and new technology is becoming available at an ever increasing rate. In the setting of an interventional database, an operator can assess a new device in the knowledge that it can be fully evaluated against standard practice.

Most of the data collected on angioplasty practice and complications in this country do not cater for the case mix of patients passing through the different catheter laboratories. Data tailored to specific risk allow centres to be compared on an even playing field, and would be more appropriate and appealing to purchasing and regulatory bodies. Indeed, if a national database existed along similar lines, individual operators and interventional centres could compare themselves with nationally available data.

Appendix

The method used for applying Bayes's theorem was developed during the second world war.

Let S indicate survival and D indicate death. The prior odds of surviving are p(S)/p(D), where p(S) = 1 − p(D).

If one supposes using n predictive features with observed values s1,s2, . . .,sn and considers them as a single finding, Bayes's theorem may be written as “the posterior odds are the likelihood ratio multiplied by the prior odds.” Thus:

Embedded Image

Clearly, as the Bayesian method assumes independent risk factors, there is a tendency to overestimate risk for patients with multiple correlated risk factors. Previous work has minimised this problem by careful grouping of clinically related factors and considering them as one factor, to dampen any overestimate of risk that multiple risk factors entail.6

References

View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.