Article Text
PDF
Abstract
Background Standardised mortality ratios (SMR) are often used to depict cardiovascular care. Data missingness, data quality, temporal variation and case-mix can, however, complicate the assessment of clinical performance.
Objectives To study Primary Care Trust (PCT) 30-day SMRs for STEMI and NSTEMI whilst considering the impact of missing data for age, sex and IMD score.
Design Observational study using data from the Myocardial Ischaemia National Audit Project (MINAP) database to generate PCT SMR maps and funnel plots for England, 2004–2007.
Patients 217,157 patients: 40.4% STEMI and 59.6% NSTEMI.
Results 95% CI 30-day unadjusted mortality: STEMI 5.8% to 6.2%; NSTEMI 6.6% to 6.9%; relative risk, 95% CI 1.14, 1.10 to 1.19. Median (IQR) data missingess by PCT for composite of age, sex and IMD score was 1.4% (0.7% to 2.2%). For STEMI and NSTEMI statistically significant predictors of mortality were mean age (STEMI: P<0.001; NSTEMI: P<0.001), proportion of females (STEMI: P<0.001; NSTEMI: P<0.001) and proportion of missing ages (STEMI: P=0.02; NSTEMI: P<0.001). Proportion of missing sex also predicted 30-day mortality for NSTEMI (P=0.01). Maps of SMRs demonstrated substantial mortality variation, but no evidence of North / South divide. There were significant correlations between STEMI and NSTEMI observed (R2 0.72) and standardised mortality (R2 0.49) rates. PCT data aggregation gave an acceptable model fit in terms of deviance explained. For STEMI there were 33 (21.7%) regions below the 99.8% lower limit of the associated performance funnel plot, and 28 (18.4%) for NSTEMI; the inclusion of missing data did not affect the distribution of SMRs.
Conclusions The proportion of missing data was associated with 30-day mortality for STEMI and NSTEMI, however it did not influence the distribution of PCTs within the funnel plots. There was considerable variation in mortality not attributable to key patient-specific factors, supporting the notion of regional-dependent variation in STEMI and NSTEMI care.
- Epidemiology
- NSTEMI
- STEMI
Statistics from Altmetric.com
Standardised mortality ratios (SMR) are often used to depict clinical performance and guide safer health care, with some organisations linking payment to achievement.1–3 Others, however, suggest that SMR are not a good indicator of quality of care because they do not separate preventable from inevitable deaths.4 Moreover, data missingness, poor data quality, temporal variation in mortality rates and the case-mix fallacy (which assumes that case-mix adjustment leads to unbiased comparisons) are also cause for concern when using SMR to assess clinical performance.5–7 Nonetheless, mortality is a readily available and frequently used measure of clinical care.8
For example, in England, SMR derived from administrative data demonstrate a reduction in mortality from coronary heart disease.9 They also reveal considerable regional variation in death rates. When limited resources may be redirected and underperforming organisations identified for further investigation, valid representation of regional performance is essential.10 Consequently, the derivation of SMR that may be used for clinical performance should consider any limitations of mortality adjustment.
In England and Wales standards of acute myocardial infarction (AMI) care are monitored through the Myocardial Ischaemia National Audit Project (MINAP),11 an extensive multicentre clinical database collecting prospective data on patients admitted with an acute coronary syndrome (ACS) to 216 acute hospitals in England and 17 acute hospitals in Wales. We aimed to study primary care trust (PCT) SMR as a proxy for hospital-level care for AMI and therefore considered data missingness, data quality, temporal variation in mortality and regional-level patient factors in the modelling strategy.
Methods
Study design
Our analyses were of data from MINAP, which was established in 1998 to meet the audit requirements of the national service framework for coronary heart disease.12 Details of MINAP and its potential for research have previously been published.13–15 Data for ACS patients are collected prospectively at each acute hospital by a secure electronic system, developed by the Central Cardiac Audit Database (CCAD), electronically encrypted and transferred on-line to a central database.16 CCAD is part of the National Clinical Audit Support Programme,17 which is part of the NHS Information Centre for Health and Social Care.18 MINAP is overseen by a multiprofessional steering group representing the stakeholders and is based at the National Institute for Clinical Outcomes Research at University College London.19 Each patient entry offers details of the patient journey, including the method and timing of admission, inpatient investigations, treatment and date of all-cause death from linkage to the UK statistics authority using a unique NHS number.
Cohort description
The data studied comprised 217 157 patients with a diagnosis of ST-elevation myocardial infarction (STEMI) or non-ST-elevation myocardial infarction (NSTEMI) resident in England and admitted to hospital between 1 January 2004 and 31 December 2007. The consensus document of the Joint European Society of Cardiology/American College of Cardiology20 was used as the diagnostic standard for AMI, and provided the basis for categorisation into STEMI. There were 87 781 (40.4%) patients with a discharge diagnosis of STEMI and 129 376 (59.6%) with a discharge diagnosis of NSTEMI.
Four complete years were used to mitigate biases due to seasonality, which is known to exist in ACS admissions and mortality.5 Only English hospitals were considered due to differences in care processes between England and Wales that may complicate direct comparisons of variation between the two countries.21
Statistical methods
We adjusted 30-day all-cause mortality for PCT-level patient factors age, sex and the index of multiple deprivation (IMD) score.6 22 23 Modelling 30-day mortality at the patient level (using a hierarchical model) provided low explanatory power with the deviance explained being only 2% (deviance is one quality of fit statistic for a model, the lower the deviance the worse the fit). In contrast, modelling at the (aggregated) level of the PCT using PCT average age, PCT proportion of women, PCT mean of the IMD scores for the patients' residences and the PCT proportion of missingness of these covariates increased the deviance explained to over 60%. Furthermore, modelling at this level ensured that the influence of hospital coders was distributed over several PCTs.
A binomial linear model24 was used to adjust for the regionally aggregated case mix and model the proportion of deaths in each region with R statistical software version 2.10.1 (http://www.r-project.org/). The model gave the expected 30-day mortality rate for each of the regions adjusted for mean age, proportion of women and deprivation for STEMI and NSTEMI. The predicted (fitted values) from the model provided the expected number of deaths in each of the regions based upon the region's mean patient profile and the incompleteness of these data.
The SMR for each region was calculated as the ratio of the observed and expected deaths. Visual representation of regional SMR by the number of admissions (PCT mortality performance) was undertaken using funnel plots.25 These were plots of the SMR against the appropriate number of admission with exact 99.8% confidence limits (corresponding to plus or minus 3 SD for normal models) provided by the model. Geographical representation was undertaken using maps (MapInfo Pro software).
Ethical consideration
The investigators had access to data in which patient identity was protected. A trusted third party, with approval of the National Information Governance Board, undertook linkage with national registries.
Results
Age, sex and IMD
A greater proportion of male patients was recorded (64%). The mean age (SD) of the cohort was 69 (14) years with STEMI patients presenting younger than NSTEMI (66 (14) years vs 72 (13) years, respectively). The median (IQR) IMD score was 18.6 (10.5 to 32.1) and was similar for STEMI, 18.2 (10.3 to 31.8) and NSTEMI, 18.3 (10.4 to 31.8), and for men, 18.5 (10.5 to 31.2) and women, 18.7 (10.6 to 32.5).
Missing data
Figure 1 shows the proportion of missing data for a composite of age, sex and IMD score plotted by PCT (2004–7). Data missingness for these variables improved between 2004 and 2007: from 0.4% in 2004 to 0.1% in 2007 for age, 1.1% to 0.2% for sex and 1.8% to 1.4% for IMD (table 1). The median (IQR) age, sex and IMD score missingness composite was 1.4% (0.7% to 2.2%). The median (IQR) missingness for age, sex and IMD was 0.16% (0% to 0.33%), 0.21% (0% to 0.52%) and 1.32% (0.61% to 2.28%), respectively. One PCT had an overall missingness of 91% and was largely due to the absence of IMD score data, although 95% of regions had less than 7% missingness. Three PCT had no submitted data for STEMI deaths and one PCT had no submitted data relating to NSTEMI cases. When hospitals below the lower quartile and above the upper quartile of missingness were compared, there was no significant difference in STEMI SMR, 95% CI (0.80, 0.57 to 1.03 and 0.89, 0.69 to 1.09, respectively) and NSTEMI SMR, 95% CI (0.80, 0.62 to 0.97 and 0.91, 0.73 to 1.09, respectively).
Geographical variation of overall proportion of missing data in age, sex and index of multiple deprivation score.
Proportion of missing data by year
Thirty day mortality rates
The 30-day unadjusted mortality rate for STEMI was 6.0% (95% exact binomial CI 5.8% to 6.2%) and the equivalent rate for NSTEMI was significantly higher at 6.7% (95% exact binomial CI 6.6% to 6.9%); RR 1.14, 95% CI 1.10 to 1.19. The 30-day SMR for STEMI and NSTEMI are shown in figure 2, and demonstrate no evidence of a north/south divide as shown for coronary heart disease.9
Standardised mortality ratios (SMR) for ST-elevation myocardial infarction (STEMI) (left) or non ST-elevation myocardial infarction (NSTEMI) (right), 2004–7.
Comparison of observed and standardised 30-day mortality by STEMI and NSTEMI
There was a significant correlation between the observed 30-day mortality for STEMI and NSTEMI, R2=0.72. This relationship was also significant for STEMI and NSTEMI SMR, R2=0.49 (figure 3).
Plots of ST-elevation myocardial infarction (STEMI) or non ST-elevation myocardial infarction (NSTEMI) 30-day mortality (A) and 30-day standardised mortality ratios (SMR) (B) for PCTs in England.
Funnel plots of performance
For STEMI, there were 17 (11.2%) regions above the 99.8% confidence limit and 33 (21.7%) regions below it (figure 4). For NSTEMI, there were 27 (17.8%) regions above the 99.8% confidence limit and 28 (18.4%) below it. There was no difference in the numbers of PCTs outwith the 99.8% confidence limits using SMR derived from expected deaths modelled without missing data.
Funnel plots of standardised mortality ratios (SMR) by number of cases submitted by each English hospital for ST-elevation myocardial infarction (STEMI) (A) and non ST-elevation myocardial infarction (NSTEMI) (B), 2004–7, 99.8% credible limits.
Missing data, mortality and the effects of confounding
Table 2 shows the mortality status at 30 days by STEMI and NSTEMI for missing age, sex and IMD score. For STEMI, the most important effects on 30-day mortality on a regional basis were the mean age of the patients, the proportion of females and the proportion of missing ages (table 3); a 0.1% increase in the proportion of females at the PCT level increased the risk of 30-day mortality by 63.0%. Each 10-year increase in age offered a 147% increase in the risk of 30-day mortality, and a 0.01% increase in missing age at the PCT level was associated with an increased risk of 30-day death of 8.9% Missing sex and IMD score were not statistically significant predictors of outcome for STEMI on a regional level.
Proportion of missing data by 30-day mortality status
Modelling of 30-day mortality for STEMI and NSTEMI to show the impact of confounding
For NSTEMI, the most important effects on 30-day mortality on a regional basis were the mean age of the patients, the proportion of females and the proportion of missing ages (table 3). The proportion of missing sex data also significantly predicted 30-day mortality. That is, a 0.1% increase in the proportion of females at the PCT level increased the risk of 30-day mortality by 16.0%, each 10-year increase in age offered a 151.4% increase in the risk of 30-day mortality, a 0.01% increase in missing age at the PCT level was associated with an increase risk of 30-day death of 23.3%, and a 1% increase in missing age at the PCT level was associated with an increased risk of 30-day death of up to 482.4% for NSTEMI.
Discussion
This analysis has identified several important points. First, the proportion of MINAP data missingness for age, sex and IMD score between 2004 and 2007 is low and improved with time. Second, higher proportions of missing age and sex data at the level of the PCT were associated with significantly higher 30-day mortality rates. Third, missing data significantly influenced the calculation of SMR for STEMI and NSTEMI. Fourth, geographical variation in SMR for STEMI and NSTEMI was evident by PCT. Fifth, there was a strong correlation between both observed and SMR for STEMI and NSTEMI performance by PCT, and finally, the consideration of missing data made no difference to the funnel plot representation of PCT performance for STEMI and NSTEMI.
The relationship between missing data and outcome for ACS is not new—our findings are in keeping with those from international studies.26 27 However, they do not explain why the association is apparent. Data from the CRUSADE National Quality Improvement Initiative and PREMIER study demonstrated that better medical record keeping was associated with a greater use of evidence-based medicine and lower mortality.26 27 An alternative reason (for MINAP data) may be that for patients who die, their medical records may be harder to locate and consequently clinical data are less likely to be submitted on time to CCAD.
The effects of missing MINAP data on 30-day mortality were statistically significant. However, when represented using funnel plots the impact of missing data was no longer evident. This is likely to be because the quantity of MINAP data available for analyses permits the identification of statistically significant but small effects. Of note is that our findings only relate to missing age, sex and IMD score and that the impact of missing data for other variables may be different.
Nonetheless, complete data for AMI patients is important and especially so for those who die. This is because the 30-day mortality rates (95% CI) for AMI are low (STEMI 4.3% to 4.6%; NSTEMI 5.0% to 5.2%), and therefore the ratio of signal to statistical noise will be low.4 Our strategy of modelling data missingness is one method by which concerns over data quality may be overcome. Multiple imputation of data is another,28 29 although encouraging good data collection at source is preferential.7 The aggregation of data at a PCT level may also help partly to alleviate patient-level data quality concerns. This approach facilitated mortality modelling through the resolution of patient-level data variation and missingness and hospital-specific recording biases.
After adjustment for regional-level patient factors age, sex, IMD score and data missingness, we found that variation in early mortality was evident by PCT. There was, however, no clear pattern such as a North/South divide as is typically reported for coronary heart disease in England.9 Furthermore, we found no statistically significant association at a PCT level with IMD score when others have recommended the use of IMD for case-mix adjustment.22 This may be because regional IMD scores smooth out the effects of patient-level deprivation.
Adjustment still left approximately 40% of the variance of the data unexplained. Moreover, the funnel plots indicated that even after adjusting for key covariates22 there was still considerable variability by region because many points lay outside the 99.8% credible limits. We hypothesise that much of the remaining variation in early mortality is due to hospital-specific (treatment) effects. That is, there is variation in the use of evidence-based therapies between hospitals in England. This notion is in keeping with international data in which regional variation in outcomes from STEMI and NSTEMI have been attributed to evidence-based hospital care.30–32
In light of the recalibration of the NHS by the Department of Health from ‘process targets’ to ‘outcomes of care’,8 valid representation of ACS care at a regional level is crucial and timely. This work emphasises the impact of missing MINAP data (age, sex and IMD score) to demonstrate, for the first time, regional variation in STEMI and NSTEMI 30-day mortality in England—there is important variation in ACS performance at the level of the PCT and therefore at the level of the hospital. We assume PCT SMR to be a proxy for hospital-level care because: (1) most ACS patients will attend the hospital within their PCT and nearest to their residence; (2) 30-day mortality is more representative of hospital treatment than longer-term survival; and (3) most PCTs commission secondary care at their local hospital. In addition, we identified a strong correlation between the NSTEMI and STEMI mortality; it is possible that comparable processes of care in the treatment of AMI occurred at a local level—supporting the notion of ‘strategic networks of care’ for AMI.33–37 Identifying and investigating areas of best practice may serve to help deliver optimal ACS care to all patients.
Limitations
Our analyses assume that patient-level (age, sex and IMD score) adjustment allows regional-level variation to be quantified—there are many other possible patient-level factors that could be considered. Lower-level adjustment was limited to six factors, and regional aggregation resulted in data from two to four hospitals being merged—therefore inferences of (individual) hospital-specific care may have been diluted. Our analyses only considered the impact of missing data for regional-level patient factors age, sex and IMD score—other variables (which may have greater influence) were not considered. Indeed, the addition of more variables to the model may reduce the variance seen in the regional performance.
Conclusion
Missing MINAP data are significantly associated with and therefore predict 30-day mortality after admission to hospital with STEMI and NSTEMI. The consideration of case mix, data missingess and temporal variation in mortality rates, and the regional aggregation of data clarifies the modelling of SMR. When these factors are accounted for, PCT SMR from MINAP data reveal substantial variation in early mortality that is not attributable to key patient-specific factors—therefore supporting the notion of hospital-dependent variation in the care of AMI in England.
Acknowledgments
The extract from the MINAP database was provided through the MINAP Academic Group. The author acknowledge all the hospitals in England and Wales for their contribution of data to MINAP.
References
Footnotes
Funding MINAP is funded by the Health Quality Improvement Partnership (HQIP) This study was funded by the British Heart Foundation, CPG is funded by the National Institute for Health Research as a Clinician Scientist Award Senior Lecturer in Cardiovascular Health Sciences and Honorary Consultant Cardiologist.
Competing interests None.
Ethical approval Ethical approval was not required; MINAP has PIAG approval.
Provenance and peer review Not commissioned; externally peer reviewed.