Article Text

Download PDFPDF

Validity of the GRACE (Global Registry of Acute Coronary Events) acute coronary syndrome prediction model for six month post-discharge death in an independent data set
  1. P J Bradshaw,
  2. D T Ko*,
  3. A M Newman,
  4. L R Donovan,
  5. J V Tu
  1. Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada
  1. Correspondence to:
    Dr Pamela Bradshaw
    School of Population Health, University of Western Australia, M431 Clifton Street Campus, Nedlands, WA 6009, Australia; pamela{at}sph.uwa.edu.au

Abstract

Objective: To determine the validity of the GRACE (Global Registry of Acute Coronary Events) prediction model for death six months after discharge in all forms of acute coronary syndrome in an independent dataset of a community based cohort of patients with acute myocardial infarction (AMI).

Design: Independent validation study based on clinical data collected retrospectively for a clinical trial in a community based population and record linkage to administrative databases.

Setting: Study conducted among patients from the EFFECT (enhanced feedback for effective cardiac treatment) study from Ontario, Canada.

Patients: Randomly selected men and women hospitalised for AMI between 1999 and 2001.

Main outcome measure: Discriminatory capacity and calibration of the GRACE prediction model for death within six months of hospital discharge in the contemporaneous EFFECT AMI study population.

Results: Post-discharge crude mortality at six months for the EFFECT study patients with AMI was 7.0%. The discriminatory capacity of the GRACE model was good overall (C statistic 0.80) and for patients with ST segment elevation AMI (STEMI) (0.81) and non-STEMI (0.78). Observed and predicted deaths corresponded well in each stratum of risk at six months, although the risk was underestimated by up to 30% in the higher range of scores among patients with non-STEMI.

Conclusions: In an independent validation the GRACE risk model had good discriminatory capacity for predicting post-discharge death at six months and was generally well calibrated, suggesting that it is suitable for clinical use in general populations.

  • ACS, acute coronary syndrome
  • AMI, acute myocardial infarction
  • EFFECT, enhanced feedback for effective cardiac treatment
  • GRACE, Global Registry of Acute Coronary Events
  • PCI, percutaneous catheter based intervention
  • STEMI, ST segment elevation acute myocardial infarction
  • acute coronary syndrome
  • risk model
  • acute myocardial infarction
  • mortality
  • population study
View Full Text

Statistics from Altmetric.com

Patients who have been hospitalised for acute myocardial infarction (AMI) remain at increased risk for cardiovascular death in the year after discharge. In a cohort of 1299 patients Froom et al1 found the risk for ischaemic events, including death, to be greatest in the first few weeks after AMI, declining rapidly up to 10 weeks and remaining in a steady state thereafter. Similarly, the period for increased risk for death among patients after a percutaneous catheter based intervention (PCI) complicated by a rise in cardiac enzymes is up to four months.2 Risk scores can assist in identifying patients at increased risk for death within six months of discharge, for both patients with ST segment elevation AMI (STEMI) and patients with non-STEMI.3

The GRACE (Global Registry of Acute Coronary Events) study collected information from patients admitted with an acute coronary syndrome (ACS) to 94 hospitals in 14 countries in North and South America, Europe and the United Kingdom, and Australia and New Zealand. Overall 32% of patients were classified as having STEMI, 27% non-STEMI and 41% unstable angina.4 The data were collected between 1999 and 2002. The GRACE model for calculating the risk for all cause mortality at six months after discharge from hospital among patients across the spectrum of ACS was developed and validated in cohorts from the GRACE registry.5 The GRACE ACS risk model has also been published as an online risk calculator and in downloadable versions for hand-held devices (http://www.outcomes-umassmed.org/grace/acs_risk.cfm). The risk model, based on information available during the hospital stay, has not yet been tested in an independent AMI population.

METHODS

The EFFECT (enhanced feedback for effective cardiac treatment) study has been described previously.6 Briefly, EFFECT AMI is a cluster, randomised trial to determine whether early versus late feedback of hospital adherence to evidence based performance indicators improves the quality of AMI care.

Study population

For phase I of the EFFECT study all patients admitted to hospitals in the province of Ontario, Canada, during the fiscal years 1999/2000 and 2000/01 with a most responsible diagnosis of AMI (International classification of diseases, ninth revision, code 410), and who had not been admitted for AMI in the year prior, were identified from the Canadian Institute for Health Information’s Discharge Abstract Database. From this population a target sample of 125 patients from each site was randomly selected for each acute care hospital in the province that treated a minimum of 30 AMI cases per annum. Of 104 eligible acute care hospitals in Ontario, 103 (99%) from 85 corporations participated in the EFFECT study, making this a truly population based study. Early and late feedback groups were randomly assigned within hospital type. Hospitals were classified as small (fewer than 50 beds), community or teaching, as designated by the Ontario Joint Policy and Planning Committee.7 The study data were abstracted from patients’ hospital charts and, in addition to their use for the clinical trial, provided a representative community sample for outcome studies.

Validation of an AMI case before data abstraction required that two of the three following variables be present: characteristic ECG changes, pain of assumed ischaemic origin and raised cardiac enzymes. Patients with AMI as an in-hospital complication were excluded. Record linkages based on unique, encrypted identification provided vital status to 12 months and additional data on in-hospital procedures and discharge status for patients transferred from the admitting hospital.

Ethics approval

The study protocol was approved by the research ethics boards of each of the 85 participating hospital corporations in Ontario who gave permission to access the patients’ charts for data abstraction. Administrative data were linked under existing agreements with provincial and national agencies.

Calculation of risk scores

Risk scores were calculated from the following variables and weighted according to the GRACE model: age, history of congestive heart failure, history of myocardial infarction, heart rate and systolic blood pressure on presentation, ST segment depression, initial serum creatinine, cardiac enzymes raised above the upper limit of normal for that laboratory and in-hospital PCI (appendix 1).5 The outcome variable was all cause mortality within six months from the day of discharge alive from hospital for the index AMI. The predictive capacity of the model for post-discharge death to 12 months was also tested.

Statistical analysis

Summary statistics were used to describe the characteristics of the EFFECT study patients at baseline and at discharge for those surviving hospitalisation. The risk scores were calculated and categorised into risk groups defined by the predetermined cut points in the published risk calculator. The χ2 test for trend was used to determine the association between GRACE risk score groups and mortality to six months. The probability of death at the midpoint of each GRACE predetermined range was used as the expected proportion of deaths for comparison with that observed in the EFFECT cohort.

Receiver operating characteristic curves were constructed to estimate the discriminatory capacity of the model in all patients and in subgroups determined by age, diagnosis and reperfusion status, with the scores used as a continuous variable. The C index statistic and 95% confidence interval (CI) are reported for both six and 12 month mortality. The analyses were repeated after exclusion of patients whose total length of hospital stay exceeded 30 days and 90 days to determine whether an extended length of stay affected the results.

The model’s calibration was tested by plotting the observed percentage of deaths at six months against that predicted. The component variables of the GRACE risk model and the total risk score were entered into separate logistic regression models to test their association with the outcome. The Hosmer–Lemeshow statistic from the regression modelling was used as an indicator of goodness of fit for the score as a predictor variable.

The data were analysed with SPSS V.12.0 (SPSS Inc, Chicago, Illinois, USA).

RESULTS

Baseline

Most of the 11 510 patients in the whole EFFECT AMI cohort (80%) were treated at community hospitals, with 7% of patients from small hospitals and 13% from teaching hospitals. Crude in-hospital mortality was 11.0% overall, being 8.9%, 11.1% and 11.5% at small, community and teaching hospitals, respectively.

Patients with STEMI constituted 49.5% of patients, of whom 59% were treated with reperfusion therapy, mostly thrombolysis; the remainder of the EFFECT cohort were patients with non-STEMI.

After discharge

The GRACE risk score was calculated for 9713 (94.8%) of the 10 242 of EFFECT AMI study patients who were discharged alive after the index admission. The variables with the highest proportion of missing data were history of congestive heart failure (4.1%), initial serum creatinine (2.4%) and history of AMI (1.4%); the others were < 1.0%. Risk scores were not calculated for these patients.

Table 1 shows the characteristics of the patients who survived to discharge and for the whole EFFECT cohort at baseline.

Patient characteristics

The AMI patients in EFFECT were older, more likely to be in Killip class II-IV, and to be in cardiac arrest on arrival at hospital than the patients from the full spectrum of ACS enrolled in the GRACE study (table 1). A smaller proportion of EFFECT patients had a history of previous AMI or congestive heart failure, or had undergone CABG or PCI, possibly related to the selection of only those who had not had an AMI in the previous year. The EFFECT patients, from Ontario, were more likely than the patients on the international registry to have smoked tobacco but not to have a history of hypertension or hyperlipidaemia.

Mortality

The post-discharge crude mortality at six months for EFFECT study patients with AMI (7.0%) was higher than for the full spectrum of patients with ACS in GRACE (4.8%). It was 5.2% and 8.8% for patients with STEMI and non-STEMI, respectively, in the EFFECT study. Exclusion of patients whose length of stay exceeded 90 days (n  =  25) did not change overall mortality. Crude mortality was 6.8% when patients hospitalised for > 30 days (n  =  253) were excluded.

Increases in risk score and observed deaths corresponded well in the EFFECT population at both six and 12 months (fig 1).

Distribution of risk scores

The distribution of risk scores was different for patients with STEMI and with non-STEMI. In keeping with the higher crude mortality, the higher risk groups had a greater proportion of patients with non-STEMI than with STEMI (fig 2).

Discrimination

The discriminatory capacity of the model was good overall (C statistic 0.80, 95% CI 0.78 to 0.81) and for both patients with STEMI (C statistic 0.81, 95% CI 0.78 to 0.84) and patients with non-STEMI (0.78, 95% CI 0.76 to 0.80). The model performed equally well for patients with STEMI who had been given reperfusion therapy on admission (C statistic 0.78, 95% CI 0.72 to 0.83) and those who had not (0.78, 95% CI 0.75 to 0.81). Discrimination was poorer when patients were grouped by age, with a C statistic of 0.72 (95% CI 0.68 to 0.81) for those aged over 65 years versus 0.74 (95% CI 0.70 to 0.78) for those aged ⩽ 65 years. The model’s ability to discriminate between patients with AMI at risk of death within six months was as good as or better than that for the GRACE development and validation cohorts (table 2).

The discriminatory capacity of the model did not change when patients with an extended length of hospital stay were excluded from the analysis. The discriminatory capacity for death within 12 months of discharge for EFFECT patients was the same as for six months, being 0.81 (95% CI 0.79 to 0.84) for patients with STEMI and 0.78 (95% CI 0.77 to 0.80) for patients with non-STEMI.

Calibration

Each of the component variables of the GRACE risk score, with the exception of ST segment depression, had significant univariate association with six month post-discharge mortality. ST depression may be less important when the cohort does not include patients with unstable angina. In the multivariate models ST segment depression, performance of an in-hospital PCI (2.3% for EFFECT patients, 27% in the GRACE development cohort) and a previous AMI were not independent predictors of the outcome for EFFECT patients with AMI. Almost all patients (97%) had raised cardiac enzymes, so that variable did not contribute to the model but was retained in the risk score calculation to allow for comparison with the GRACE cohort. The goodness of fit estimated in the regression models is provided, but not the other outputs, as they are not the focus of this validation study.

Statistical goodness of fit of the model with component variables was shown for patients with STEMI (Hosmer–Lemeshow p  =  0.12) but not overall (p  =  0.002), nor for patients with non-STEMI (p  =  0.04). The fit was better when the risk score was used as the single explanatory variable for both STEMI (p  =  0.41) and non-STEMI groups (p  =  0.06) but still was not significant for the whole EFFECT cohort (p  =  0.012).

Although results of the statistical goodness of fit test were variable, the calibration of observed against expected deaths was good overall. Correspondence was better, however, for STEMI than for non-STEMI groups, for whom the risk of death within six months was underestimated in the upper ranges of risk score (fig 3). Mortality was overestimated for patients with the highest scores but the numbers were small, being < 1% of both STEMI and non-STEMI groups.

DISCUSSION

The GRACE risk model for mortality within six months of hospital discharge performed as well in the EFFECT AMI cohort as in the original GRACE ACS population, and this was the case for patients with both STEMI and non-STEMI. Furthermore, the discriminatory capacity of the model for EFFECT patients with STEMI exceeded that of patients with STEMI in the GRACE development and validation cohorts. The model held good for stratification of the risk for mortality to 12 months after discharge.

The good performance of the GRACE ACS model among STEMI patients in the EFFECT cohort (C statistic 0.81) contrasts with the more variable discriminatory capacity for patients with STEMI seen in the GRACE development and validation cohorts (0.71 and 0.76, respectively). Although not discussed by the authors,5 the C statistic for the GRACE cohort patients with STEMI deserves comment, as 0.71 in the large development group can be considered to be only fair. The variation does not seem related to the numbers of patients or to the proportion in each diagnostic group within ACS, as the C statistic for patients with non-STEMI was consistent across the cohorts. It may be that the unequal distribution of risk across the spectrum of ACS affects the performance of the model for diagnostic groups within ACS. The consistency of the model’s performance for patients with non-STEMI, compared with the relatively poor performance and inconsistency for patients with STEMI and unstable angina, suggests that the model may not be as robust across the spectrum of ACS as the developers hoped.

The greater than predicted proportion of deaths within six months of discharge in some risk groups of patients with non-STEMI in the EFFECT cohort may be related to the different performance of the model in different risk groups, to unaccounted differences in the GRACE and EFFECT populations, or to differences in the care given to patients in hospital or after discharge between Ontario and other countries on the international registry.

Many risk studies are developed and validated only in the original dataset. Testing in a completely independent dataset provides a rigorous test of the utility of the model and should be undertaken before the model is recommended for widespread use.

Conclusions

The GRACE risk model showed good discriminatory capacity for predicting six month death after discharge among patients with STEMI and non-STEMI. It stratified patients well, and there was good correspondence between the proportion of deaths observed in the EFFECT cohort and that predicted for each risk group according to the published risk nomogram.

APPENDIX 1

TableA1

GRACE (Global Registry of Acute Coronary Events) risk calculator for six month post-discharge mortality

Table 1

 Characteristics of patients in GRACE (Global Registry of Acute Coronary Events) at baseline and of patients with acute myocardial infarction (AMI) in the EFFECT (enhanced feedback for effective cardiac treatment) study on admission to and at discharge alive from hospitals in Ontario, 1999–2001

Table 2

 Performance of the GRACE risk model assessed by the C statistic in ST segment elevation acute myocardial infarction (STEMI) and non-STEMI groups in the EFFECT (enhanced feedback for effective cardiac treatment) acute myocardial infarction (AMI) cohort and in the original GRACE study

Figure 1

 Six and 12 month mortality after hospital discharge in patients with acute myocardial infarction in the EFFECT (enhanced feedback for effective cardiac treatment) trial by GRACE (Global Registry of Acute Coronary Events) risk score group. STEMI, ST segment elevation acute myocardial infarction.

Figure 2

 Distribution of GRACE scores by risk group in the EFFECT (enhanced feedback for effective cardiac treatment) study patients with ST segment elevation acute myocardial infarction (STEMI) and non-STEMI.

Figure 3

 Observed six month mortality after hospital discharge for EFFECT (enhanced feedback for effective cardiac treatment) patients with ST segment elevation acute myocardial infarction (STEMI) and non-STEMI versus predicted mortality (GRACE).

Acknowledgments

The EFFECT study was conducted under the auspices of the Canadian Cardiovascular Outcomes Research Team (CCORT). CCORT is funded by grants from the Canadian Institutes of Health Research and the Heart and Stroke Foundation of Canada. Dr Tu is supported by a Canada Research Chair in Health Services Research.

REFERENCES

View Abstract

Footnotes

  • Published Online First 30 December 2005

  • * Also the Division of Cardiology, Schulich Heart Centre and the Department of Medicine, Sunnybrook and Women’s College Health Sciences Centre, Toronto, Ontario, Canada

  • Also the Division of General Internal Medicine and Clinical Epidemiology and Health Care Research Program, Sunnybrook and Women’s College Health Sciences Centre, and the Department of Health Policy, Management and Evaluation, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada

  • Competing interests: none declared

  • Ethics approval: the study protocol was approved by the research ethics boards of each of the 85 participating hospital corporations in Ontario, who gave permission to access the patients’ charts for data abstraction. The linkages to administrative data were conducted under existing agreements with provincial and national agencies.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.