Patients who have been hospitalised for acute myocardial infarction (AMI) remain at increased risk for cardiovascular death in the year after discharge. In a cohort of 1299 patients Froom et al¹ found the risk for ischaemic events, including death, to be greatest in the first few weeks after AMI, declining rapidly up to 10 weeks and remaining in a steady state thereafter. Similarly, the period for increased risk for death among patients after a percutaneous catheter based intervention (PCI) complicated by a rise in cardiac enzymes is up to four months.² Risk scores can assist in identifying patients at increased risk for death within six months of discharge, for both patients with ST segment elevation AMI (STEMI) and patients with non-STEMI.³

The GRACE (Global Registry of Acute Coronary Events) study collected information from patients admitted with an acute coronary syndrome (ACS) to 94 hospitals in 14 countries in North and South America, Europe and the United Kingdom, and Australia and New Zealand. Overall 32% of patients were classified as having STEMI, 27% non-STEMI and 41% unstable angina.⁴ The data were collected between 1999 and 2002. The GRACE model for calculating the risk for all cause mortality at six months after discharge from hospital among patients across the spectrum of ACS was developed and validated in cohorts from the GRACE registry.⁵ The GRACE ACS risk model has also been published as an online risk calculator and in downloadable versions for hand-held devices (http://www.outcomes-umassmed.org/grace/acs_risk.cfm). The risk model, based on information available during the hospital stay, has not yet been tested in an independent AMI population.

METHODS

The EFFECT (enhanced feedback for effective cardiac treatment) study has been described previously.⁶ Briefly, EFFECT AMI is a cluster, randomised trial to determine whether early versus late feedback of hospital adherence to evidence based performance indicators improves the quality of AMI care.

Study population

For phase I of the EFFECT study all patients admitted to hospitals in the province of Ontario, Canada, during the fiscal years 1999/2000 and 2000/01 with a most responsible diagnosis of AMI (International classification of diseases, ninth revision, code 410), and who had not been admitted for AMI in the year prior, were identified from the Canadian Institute for Health Information’s Discharge Abstract Database. From this population a target sample of 125 patients from each site was randomly selected for each acute care hospital in the province that treated a minimum of 30 AMI cases per annum. Of 104 eligible acute care hospitals in Ontario, 103 (99%) from 85 corporations participated in the EFFECT study, making this a truly population based study. Early and late feedback groups were randomly assigned within hospital type. Hospitals were classified as small (fewer than 50 beds), community or teaching, as designated by the Ontario Joint Policy and Planning Committee.⁷ The study data were abstracted from patients’ hospital charts and, in addition to their use for the clinical trial, provided a representative community sample for outcome studies.

Validation of an AMI case before data abstraction required that two of the three following variables be present: characteristic ECG changes, pain of assumed ischaemic origin and raised cardiac enzymes. Patients with AMI as an in-hospital complication were excluded. Record linkages based on unique, encrypted identification provided vital status to 12 months and additional data on in-hospital procedures and discharge status for patients transferred from the admitting hospital.

Ethics approval

The study protocol was approved by the research ethics boards of each of the 85 participating hospital corporations in Ontario who gave permission to access the patients’ charts for data abstraction. Administrative data were linked under existing agreements with provincial and national agencies.

Calculation of risk scores

Risk scores were calculated from the following variables and weighted according to the GRACE model: age, history of congestive heart failure, history of myocardial infarction, heart rate and systolic blood pressure on presentation, ST segment depression, initial serum creatinine, cardiac enzymes raised above the upper limit of normal for that laboratory and in-hospital PCI (appendix 1).⁵ The outcome variable was all cause mortality within six months from the day of discharge alive from hospital for the index AMI. The predictive capacity of the model for post-discharge death to 12 months was also tested.

Statistical analysis

Summary statistics were used to describe the characteristics of the EFFECT study patients at baseline and at discharge for those surviving hospitalisation. The risk scores were calculated and categorised into risk groups defined by the predetermined cut points in the published risk calculator. The χ² test for trend was used to determine the association between GRACE risk score groups and mortality to six months. The probability of death at the midpoint of each GRACE predetermined range was used as the expected proportion of deaths for comparison with that observed in the EFFECT cohort.

Receiver operating characteristic curves were constructed to estimate the discriminatory capacity of the model in all patients and in subgroups determined by age, diagnosis and reperfusion status, with the scores used as a continuous variable. The C index statistic and 95% confidence interval (CI) are reported for both six and 12 month mortality. The analyses were repeated after exclusion of patients whose total length of hospital stay exceeded 30 days and 90 days to determine whether an extended length of stay affected the results.

The model’s calibration was tested by plotting the observed percentage of deaths at six months against that predicted. The component variables of the GRACE risk model and the total risk score were entered into separate logistic regression models to test their association with the outcome. The Hosmer–Lemeshow statistic from the regression modelling was used as an indicator of goodness of fit for the score as a predictor variable.

The data were analysed with SPSS V.12.0 (SPSS Inc, Chicago, Illinois, USA).

RESULTS

Baseline

Most of the 11 510 patients in the whole EFFECT AMI cohort (80%) were treated at community hospitals, with 7% of patients from small hospitals and 13% from teaching hospitals. Crude in-hospital mortality was 11.0% overall, being 8.9%, 11.1% and 11.5% at small, community and teaching hospitals, respectively.

Patients with STEMI constituted 49.5% of patients, of whom 59% were treated with reperfusion therapy, mostly thrombolysis; the remainder of the EFFECT cohort were patients with non-STEMI.

After discharge

The GRACE risk score was calculated for 9713 (94.8%) of the 10 242 of EFFECT AMI study patients who were discharged alive after the index admission. The variables with the highest proportion of missing data were history of congestive heart failure (4.1%), initial serum creatinine (2.4%) and history of AMI (1.4%); the others were < 1.0%. Risk scores were not calculated for these patients.

Table 1 shows the characteristics of the patients who survived to discharge and for the whole EFFECT cohort at baseline.

Patient characteristics

The AMI patients in EFFECT were older, more likely to be in Killip class II-IV, and to be in cardiac arrest on arrival at hospital than the patients from the full spectrum of ACS enrolled in the GRACE study (table 1). A smaller proportion of EFFECT patients had a history of previous AMI or congestive heart failure, or had undergone CABG or PCI, possibly related to the selection of only those who had not had an AMI in the previous year. The EFFECT patients, from Ontario, were more likely than the patients on the international registry to have smoked tobacco but not to have a history of hypertension or hyperlipidaemia.

Mortality

The post-discharge crude mortality at six months for EFFECT study patients with AMI (7.0%) was higher than for the full spectrum of patients with ACS in GRACE (4.8%). It was 5.2% and 8.8% for patients with STEMI and non-STEMI, respectively, in the EFFECT study. Exclusion of patients whose length of stay exceeded 90 days (n  =  25) did not change overall mortality. Crude mortality was 6.8% when patients hospitalised for > 30 days (n  =  253) were excluded.

Increases in risk score and observed deaths corresponded well in the EFFECT population at both six and 12 months (fig 1).

Distribution of risk scores

The distribution of risk scores was different for patients with STEMI and with non-STEMI. In keeping with the higher crude mortality, the higher risk groups had a greater proportion of patients with non-STEMI than with STEMI (fig 2).

Discrimination

The discriminatory capacity of the model was good overall (C statistic 0.80, 95% CI 0.78 to 0.81) and for both patients with STEMI (C statistic 0.81, 95% CI 0.78 to 0.84) and patients with non-STEMI (0.78, 95% CI 0.76 to 0.80). The model performed equally well for patients with STEMI who had been given reperfusion therapy on admission (C statistic 0.78, 95% CI 0.72 to 0.83) and those who had not (0.78, 95% CI 0.75 to 0.81). Discrimination was poorer when patients were grouped by age, with a C statistic of 0.72 (95% CI 0.68 to 0.81) for those aged over 65 years versus 0.74 (95% CI 0.70 to 0.78) for those aged ⩽ 65 years. The model’s ability to discriminate between patients with AMI at risk of death within six months was as good as or better than that for the GRACE development and validation cohorts (table 2).

The discriminatory capacity of the model did not change when patients with an extended length of hospital stay were excluded from the analysis. The discriminatory capacity for death within 12 months of discharge for EFFECT patients was the same as for six months, being 0.81 (95% CI 0.79 to 0.84) for patients with STEMI and 0.78 (95% CI 0.77 to 0.80) for patients with non-STEMI.

Calibration

Each of the component variables of the GRACE risk score, with the exception of ST segment depression, had significant univariate association with six month post-discharge mortality. ST depression may be less important when the cohort does not include patients with unstable angina. In the multivariate models ST segment depression, performance of an in-hospital PCI (2.3% for EFFECT patients, 27% in the GRACE development cohort) and a previous AMI were not independent predictors of the outcome for EFFECT patients with AMI. Almost all patients (97%) had raised cardiac enzymes, so that variable did not contribute to the model but was retained in the risk score calculation to allow for comparison with the GRACE cohort. The goodness of fit estimated in the regression models is provided, but not the other outputs, as they are not the focus of this validation study.

Statistical goodness of fit of the model with component variables was shown for patients with STEMI (Hosmer–Lemeshow p  =  0.12) but not overall (p  =  0.002), nor for patients with non-STEMI (p  =  0.04). The fit was better when the risk score was used as the single explanatory variable for both STEMI (p  =  0.41) and non-STEMI groups (p  =  0.06) but still was not significant for the whole EFFECT cohort (p  =  0.012).

Although results of the statistical goodness of fit test were variable, the calibration of observed against expected deaths was good overall. Correspondence was better, however, for STEMI than for non-STEMI groups, for whom the risk of death within six months was underestimated in the upper ranges of risk score (fig 3). Mortality was overestimated for patients with the highest scores but the numbers were small, being < 1% of both STEMI and non-STEMI groups.

DISCUSSION

The GRACE risk model for mortality within six months of hospital discharge performed as well in the EFFECT AMI cohort as in the original GRACE ACS population, and this was the case for patients with both STEMI and non-STEMI. Furthermore, the discriminatory capacity of the model for EFFECT patients with STEMI exceeded that of patients with STEMI in the GRACE development and validation cohorts. The model held good for stratification of the risk for mortality to 12 months after discharge.

The good performance of the GRACE ACS model among STEMI patients in the EFFECT cohort (C statistic 0.81) contrasts with the more variable discriminatory capacity for patients with STEMI seen in the GRACE development and validation cohorts (0.71 and 0.76, respectively). Although not discussed by the authors,⁵ the C statistic for the GRACE cohort patients with STEMI deserves comment, as 0.71 in the large development group can be considered to be only fair. The variation does not seem related to the numbers of patients or to the proportion in each diagnostic group within ACS, as the C statistic for patients with non-STEMI was consistent across the cohorts. It may be that the unequal distribution of risk across the spectrum of ACS affects the performance of the model for diagnostic groups within ACS. The consistency of the model’s performance for patients with non-STEMI, compared with the relatively poor performance and inconsistency for patients with STEMI and unstable angina, suggests that the model may not be as robust across the spectrum of ACS as the developers hoped.

The greater than predicted proportion of deaths within six months of discharge in some risk groups of patients with non-STEMI in the EFFECT cohort may be related to the different performance of the model in different risk groups, to unaccounted differences in the GRACE and EFFECT populations, or to differences in the care given to patients in hospital or after discharge between Ontario and other countries on the international registry.

Many risk studies are developed and validated only in the original dataset. Testing in a completely independent dataset provides a rigorous test of the utility of the model and should be undertaken before the model is recommended for widespread use.

Conclusions

The GRACE risk model showed good discriminatory capacity for predicting six month death after discharge among patients with STEMI and non-STEMI. It stratified patients well, and there was good correspondence between the proportion of deaths observed in the EFFECT cohort and that predicted for each risk group according to the published risk nomogram.

APPENDIX 1