Article Text
PDF
Abstract
Objective Existing clinical prediction models (CPM) for short-term mortality after transcatheter aortic valve implantation (TAVI) have limited applicability in the UK due to moderate predictive performance and inconsistent recording practices across registries. The aim of this study was to derive a UK-TAVI CPM to predict 30-day mortality risk for benchmarking purposes.
Methods A two-step modelling strategy was undertaken: first, data from the UK-TAVI Registry between 2009 and 2014 were used to develop a multivariable logistic regression CPM using backwards stepwise regression. Second, model-updating techniques were applied using the 2013–2014 data, thereby leveraging new approaches to include frailty and to ensure the model was reflective of contemporary practice. Internal validation was performed by bootstrapping to estimate in-sample optimism-corrected performance.
Results Between 2009 and 2014, up to 6339 patients were included across 34 centres in the UK-TAVI Registry (mean age, 81.3; 2927 female (46.2%)). The observed 30-day mortality rate was 5.14%. The final UK-TAVI CPM included 15 risk factors, which included two variables associated with frailty. After correction for in-sample optimism, the model was well calibrated, with a calibration intercept of 0.02 (95% CI −0.17 to 0.20) and calibration slope of 0.79 (95% CI 0.55 to 1.03). The area under the receiver operating characteristic curve, after adjustment for in-sample optimism, was 0.66.
Conclusion The UK-TAVI CPM demonstrated strong calibration and moderate discrimination in UK-TAVI patients. This model shows potential for benchmarking, but even the inclusion of frailty did not overcome the need for more wide-ranging data and other outcomes might usefully be explored.
- aortic stenosis
- transcatheter valve interventions
This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/
Statistics from Altmetric.com
Introduction
Transcatheter aortic valve implantation (TAVI) has emerged as the indicated treatment option for patients with aortic stenosis who are intermediate to high-risk surgical candidates.1–3 Assessment of procedural risk is predominantly undertaken by the Heart Team, guided by multiple clinical prediction models (CPM).4 To this end, CPMs such as the EuroSCORE5 or the Society of Thoracic Surgeons (STS) mortality score6 have been used to guide estimation of procedural risk and employed in randomised controlled trials of TAVI.1–3 However, surgical-based models are known to perform poorly at predicting mortality in TAVI patients.7 8
Consequently, there have been recent attempts to derive procedure-specific CPMs using national registries; examples include the FRANCE-2 model,9 the Italian OBSERVANT model10 and the American College of Cardiology (ACC) model.11 Primarily, one is interested in using such TAVI CPMs to underpin procedure audit analyses across TAVI centres, to facilitate discussion of risk with the patient and to aid comparison of randomised trials. However, existing models have only moderate performance when applied in samples outside of their development cohorts.12 Indeed, within the UK, there is no validated CPM for predicting mortality post-TAVI,13 and heterogeneity in the variables recorded among national registries restricts the application of existing models into UK practice. As such, developing a CPM for UK-TAVI patients is vital, especially since one of the fundamental incentives for collecting national registry data is to monitor centre-level performance for audit and feedback purposes.
Therefore, the aim of this study was to develop and internally validate a multivariable TAVI CPM for predicting 30-day mortality in UK-TAVI patients (hereto called the UK-TAVI CPM), to facilitate clinical discussions around procedure risk with the patient during the consent process, and for national benchmarking/audit analyses.
Methods
The reporting of this manuscript adheres to the TRIPOD checklist for the reporting of multivariable prediction models.14 The completed checklist is provided as online supplementary material.
Supplementary file 1
UK-TAVI Registry
The UK-TAVI Registry prospectively collects data for every TAVI procedure conducted in the UK (34 centres) through a web-based interface.15 This study included data on all TAVI procedures between January 2009 and December 2014. Procedures were predominately undertaken via transfemoral access, using either the Edwards SAPIEN or the Medtronic CoreValve families of devices (with their various iterations over time), although other access routes and valve types were available. The registry records information on patient baseline demographics, risk factors for intervention and procedural details. Frailty information was only recorded within the registry from January 2013 through the KATZ Index of activities of daily living,16 the Canadian Study of Health and Aging (CSHA) frailty scale17 and an indicator of poor mobility (as defined in the EuroSCORE II model5).
Outcome and cohort definition
All-cause mortality tracking was independently obtained from the Office for National Statistics, with administrative censoring occurring on 31 May 2015; this information was used to define the binary endpoint of 30-day mortality after TAVI. Mortality information was unavailable for patients in Northern Ireland and Scotland; consequently, these patients were excluded, along with any patient in England and Wales with missing mortality information. Thus, the main development sample included all TAVI patients with 30-day mortality indication across England and Wales, between January 2009 and December 2014.
Predictors
For development of the UK-TAVI CPM, we considered all variables available prior to the TAVI procedure (see online supplementary table 1 for the full list of considered covariates, with translation to the UK-TAVI Registry). Our decision to include access route as a candidate covariate was made in line with previous TAVI CPM development data sets,9 11 and represents the potential risk/complications/comorbidities associated with a non-transfemoral approach. Age and sex were always included in the model, with age being mean centred. Quadratic and cubic transformations of continuous variables were considered.
Missing data
Multiple imputation by chain equations was used to handle missing data, with this analysis generating 10 imputed data sets.18 The imputation models for each baseline covariate included the majority of other variables in the UK-TAVI Registry and the survival data (both the survival event indicator and the cumulative baseline hazard19), the latter of which was included because this was used to derive 30-day mortality indication. Prior to analysis, we checked all the imputations to ensure convergence and checked that the distributions of observed and imputed values were similar. Since multiple imputation assumes a missing at random mechanism, we undertook a sensitivity analysis of this assumption as described below.
Statistical analysis
A two-stage modelling strategy was implemented to derive the CPM (figure 1). We first derived a logistic regression model within the 2009–2014 data, considering only those variables that were observed throughout this period (ie, none of the frailty measures), using the ‘majority method’ of developing CPMs within multiple imputed data.20 Here, the following steps were undertaken: (1) perform backwards selection using Akaike information criterion in each of the 10 imputed data sets, resulting in 10 (potentially unique) sets of predictors; (2) extract variables that were selected in >50% of the imputed data sets; (3) given the extracted variables, fit a model in each imputed data set, then pool the estimated coefficients and SEs across imputations according to Rubin’s rules.18 The Akaike information criterion for backwards selection corresponds to a P value of 0.157 for a variable with 1 df.
Flow chart illustrating the steps undertaken within the two-stage modelling strategy to derive and internally validate the UK-TAVI CPM. CPM, clinical prediction model; TAVI, transcatheter aortic valve implantation.
In the second stage, we updated this derived model using model-updating techniques within the last 2 years of data21 22 (figure 1). Here, a logistic regression model was fit to 30-day mortality within the 2013–2014 data, with the calculated linear predictor (LP) from the 2009–2014 derived model, KATZ, CSHA and poor mobility as covariates (see online supplementary methods for mathematical details). The likelihood ratio test was used to determine if any of the three frailty measures significantly improved the fit of the model, with only those meeting this criterion included in the final model. The reasoning for this second modelling step was twofold. First, it uses all available data as the foundations of the UK-TAVI CPM, while ensuring that the model is reflective of most contemporary practice/technology. Second, it provides a way of considering frailty variables within the model, which we hypothesised a priori would be important predictors of mortality,23 but were only recorded within the registry from January 2013.
Predictive performance was assessed through calibration and discrimination. Calibration is the agreement between the observed and expected risk, which was quantified by the calibration intercept and calibration slope, with reference values of 0 and 1, respectively.24 Discrimination of a model is its ability to distinguish those who experienced the outcome from those who did not, and was estimated with the area under the receiver operating characteristic (ROC) curve (AUC). Throughout, we use the term ‘apparent performance’ to denote the predictive performance of a model within the data set in which it was developed. As such, the apparent performance was obtained by applying the final model coefficients to each patient within the 2013–2014 data across the 10 imputed data sets and pooling performance results across imputations (using Rubin’s rules18). Internal validation was undertaken using bootstrap resampling of the 2013–2014 data to estimate optimism-corrected performance (figure 1)22 25; details of this process are given in the online supplementary material. In short, the bootstrap resampling represents sampling from the underlying population to estimate in-sample optimism, which can be subtracted from the apparent model performance. These internal validation results should be treated as the primary measures of predictive performance.
R V.3.4.026 was used for all statistical analyses. Multiple imputation of the data set was completed using the mice package,27 and the package pROC was used for constructing ROC curves.28 All other codes were written by the authors and are available upon request.
Sensitivity analyses
Instead of using multiple imputation, the first sensitivity analysis (singularly) imputed all missing continuous variables using the sex-matched median of the observed values, and missing categorical variables using the mode. For the majority of categorical variables this will result in a ‘risk factor absent’ assumption and corresponds to a viable missing not-at random assumption. Similar modelling steps to the main analysis were then performed using this single imputation strategy. The second sensitivity analysis performed the ‘majority method’ of variable selection directly within the 2013–2014 data set to examine the robustness of the two-stage modelling strategy.
Results
Between January 2009 and December 2014, up to 7070 patients were recorded in the registry. After excluding patients in Northern Ireland (n=379) and Scotland (n=193), a further 159 patients were removed due to missing life status. Hence, the development sample for the main analysis included 6339 patients (figure 2). Baseline characteristics and proportions of missing data for the main development cohort are given in table 1. The mean age of patients was 81.3 years, with 46.2% female. Most procedures were performed electively (12.5% non-elective), via transfemoral access (25.4% non-transfemoral access), and used a SAPIEN valve (56%). The proportion of missing data was low for most variables. The high percentages of missing data for the three frailty measures are due to these variables only being recorded from January 2013, after which the proportion of missing data for poor mobility, CSHA and KATZ was 1.79%, 1.65% and 11.0%, respectively.
Patient flow chart through the exclusion criteria for both the main development sample and the sensitivity analysis that modelled using 2013/2014 data only. TAVI, transcatheter aortic valve implantation.
Baseline and procedural characteristics for patients in the main development cohort
Model development
Between January 2009 and December 2014, up to 326 patients died within 30 days of the procedure (5.14%), with this decreasing to 4.14% between January 2013 and December 2014. In univariable analysis, patients with higher (mean-centred) body mass index (BMI), higher glomerular filtration rate and indication of sinus rhythm had decreased odds of 30-day mortality (table 2). Renal failure, extracardiac arteriopathy, calcification of ascending aorta, prior balloon aortic valvuloplasty, critical preoperative status, New York Heart Association class IV, poor mobility, CSHA frailty, KATZ (per point drop from 6 points), left ventricular ejection fraction <50%, non-elective procedure and non-transfemoral access each had increased odds of 30-day mortality.
Univariable odds ratios (ORs) of each baseline variable on 30-day mortality in the main analysis development cohort
The variables selected in the final multivariable model (denoted UK-TAVI CPM) are given in table 3. The model intercept was −3.6119, with 15 variables included in the model, 2 of which were frailty measures (ie, KATZ and poor mobility). One can calculate the log odds of 30-day mortality for a new patient by multiplying each covariate by the corresponding coefficient in table 3 and summing across all variables—the so-called LP; the probability of 30-day mortality can then be obtained by transforming the LP as exp(LP)/{1+exp(LP)}. Figure 3 depicts a graphical representation of this process. For example, an 85-year-old woman with a BMI of 30 kg/m2, a glomerular filtration rate of 51 mL/min/1.73 m2 and KATZ score of 4 points would have log odds=−3.6119+((85−81.25)×0.0115)+(0.1393)+((30−27.3)×−0.0257)+((30−27.3)2×0.0011)+([51/5] ×−0.0342)+((6−4)×0.2362)=−3.3604, where [51/5]=10 (ie, floor rounding). This can be converted to a probability of 30-day mortality by exp(−3.3604)/{1+exp(−3.3604)}=0.0336×100=3.36% (dotted arrows in figure 3).
Variables and coefficients included in the final multivariable UK-TAVI CPM
Graphical representation of the UK-TAVI (transcatheter aortic valve implantation) clinical prediction model (CPM). First, multiply each variable (either yes/no for categorical variables or enter the observed continuous variable) by the corresponding coefficient and then sum across all variables to obtain the linear predictor. The linear predictor can then be converted to a predicted risk using the graph or through the equation: exp(Linear Predictor)/{1+exp(Linear Predictor)}. The dotted arrows show the example described in the text. BAV, balloon aortic valvuloplasty; BMI, body mass index; eGFR, estimated glomerular filtration rate; N/A, not applicable; PA, pulmonary artery; TF, transfemoral access.
Model validation
The UK-TAVI CPM was well calibrated before and after bootstrap correction, with a calibration intercept and slope significantly close to 0 and 1, respectively (table 4); the apparent performance calibration plot is given in online supplementary figure 1. The apparent AUC of the model was 0.70 (95% CI 0.65 to 0.75), which reduced to 0.66 (95% CI 0.61 to 0.71) after bootstrap internal validation. The UK-TAVI CPM was well calibrated and had moderate discrimination across quantiles of observed predicted risk, although performance was marginally worse in the second and third quantiles (online supplementary table 2). Bootstrap-corrected performance across patient subgroups is given in online supplementary table 3.
Performance measures before (apparent) and after bootstrap-corrected optimism within the 2013–2014 data (n=2969)
We examined the predictive performance across each centre, which demonstrated that the majority of centres had an observed event rate similar to that expected from the model (figure 4). However, there was heterogeneity in the calibration-in-the-large, with some centres performing ‘better’ than expected (calibration intercept <0), and others having an observed event rate higher than that expected by the model (calibration intercept >0).
Forest plot of the calibration intercept for the UK-TAVI (transcatheter aortic valve implantation) clinical prediction model (CPM) across all centres. Centres have been sorted based on the calibration intercept. Three centres with no deaths by 30 days have been removed.
Sensitivity analyses
The first sensitivity analysis examined an alternative way of handling missing data, through a single imputation strategy. Most variables within the model derived under this approach were similar to those within the UK-TAVI CPM, but additionally included an indicator of calcified aorta, and aortic valve peak gradient (online supplementary table 4). The performance results were quantitatively similar to the main analysis model.
The second sensitivity analysis restricted the development cohort to those patients between January 2013 and December 2014 to contrast with the aforementioned two-stage modelling strategy (figure 1). The model developed under this approach included similar but fewer variables to the main analysis model (online supplementary table 5). After bootstrap internal validation, the calibration slope for 2013–2014 derived model was significantly below 1 (0.69, 95% CI 0.45 to 0.92), indicating overfitting within the smaller sample size (demonstrating an advantage of the two-stage modelling strategy). The internal validation AUC of the 2013–2014 derived model was 0.64 (95% CI 0.59 to 0.69).
Discussion
This study has derived a TAVI risk prediction model for benchmarking and audit analyses in UK patients. After covariate selection, the UK-TAVI CPM included 15 variables, including 2 frailty-related measures. The model demonstrated strong calibration after correction for in-sample optimism, but only moderate discrimination. It is, therefore, of greater value in allowing comparison of risk-adjusted outcomes between centres than in aiding clinicians to provide accurate risk assessment to patients being offered TAVI.
Risk prediction in TAVI patients is an ongoing research area; the FRANCE-2,9 OBSERVANT,10 ACC11 and the herein UK-TAVI CPM each represent models derived exclusively on patients treated by TAVI and each share similar (clinically meaningful) risk factors. However, differences in the variables that are recorded across the national registries limit the ability to obtain a single generalisable model for practical use across countries. Thus, we have demonstrated the UK-TAVI CPM to be well calibrated both before and after correction for in-sample optimism, with an AUC of 0.70 and 0.66 within development and internal validation, respectively. Similar performance has been reported from the existing TAVI models within data sets similar to that in which they were developed.9–11 For example, the FRANCE-2 model had an AUC of 0.59 in a ‘test’ subset of the French registry,9 and the AUC within an internal validation of the ACC TAVI model was 0.66.11 However, a previous analysis of the UK registry found that the AUC of the existing TAVI CPMs was below 0.64 within the UK cohort.13
The limited discrimination of the UK-TAVI CPM means that this model is not yet ready for deployment when attempting to assess risk in individual cases. While one could question the need to use TAVI CPMs in such a capacity given the emerging evidence supporting expansion into low/intermediate-risk patients,3 CPMs are fundamental to appropriately adjust for case mix when investigating postprocedural outcomes across centres. In particular, the assessment of TAVI futility should consider multiple outcomes, including mortality, quality of life and hospital readmission, but the variability in the calibration intercept per TAVI centre (figure 4) indicates the importance of case-mix adjustment when comparing mortality rates. Consequently, the herein derived model presents an important step in facilitating appropriate adjustment for case mix when making centre-level outcome comparisons in UK patients, with similar applications undertaken within the STS/ACC Transcatheter Valve Therapy Registry using the ACC model.11
However, it is inevitable that predictive performance will decrease when the UK-TAVI CPM is applied in populations distinct to that in which it was developed (external validation),22 as observed when our group applied the France, Italian and American TAVI CPMs to the UK population.13 Since the primary aim of the UK-TAVI CPM is for national benchmarking, the main implementation of this model will be within UK patients. To this end, the split sample method of internal validation (randomly splitting the data into training and test sets) is common, but this is an inefficient use of data.25 Consequently, we applied bootstrapping methods to estimate the likely performance of the model in samples drawn from the registry. While lack of new and independent data from the UK registry meant that external validation was not possible, such an assessment of model performance will need to be undertaken by independent investigators.29 As such, we encourage the validation of our model in future extracts of the UK registry to assess temporal validation. Unquestionably, the UK-TAVI CPM should not be regarded as a static tool, and the discovery of novel predictors will be fundamental for deriving future TAVI models. The rapid development of TAVI techniques, knowledge and technology, combined with the possible expansion into lower risk patients, means the model should be continuously updated. The model needs to be maintained using contemporary extracts of the registry to avoid the calibration drift commonly observed in CPMs, such as that found in the logistic EuroSCORE.30 This would be equally applicable to other contemporary risk scores used elsewhere.
Limitations
The main strength of this study is that TAVI data collection is mandatory in the UK, meaning the UK-TAVI CPM has been developed using comprehensive national data. However, several limitations need to be considered. First, since we excluded all patients from Northern Ireland and Scotland due to lack of independent mortality information, the model might not be representative of such patients. Second, this model only predicts 30-day mortality, but demonstrates the need for registries to routinely collect more wide-ranging outcomes (eg, hospital readmission, quality of life). Third, to maintain sufficient sample size, we included all data from 2009 to 2014, involving several iterations of valve types/devices. Although the modelling strategy aimed to revise the model using the 2013–2014 data, the UK-TAVI CPM will need to be updated in newer extracts to include patients exposed to only the most contemporary technology/practice. Finally, the generalisability of the model is unknown since we were unable to test the performance in independent data. Thus, we recommend independent researchers conduct future validation studies on the later extracts of the UK-TAVI Registry.
Conclusions
This analysis of the UK-TAVI Registry has developed a contemporary risk model on over 6000 TAVI patients. The validation procedure demonstrated that the model was well calibrated but achieved limited discrimination. Thus, the derived model has potential to be used for benchmarking analyses in UK-TAVI patients, but is not yet ready for deployment when attempting to assess risk in individual cases. Future external validation studies of the model within the UK population are required, and outcomes other than 30-day mortality might usefully be explored.
Key messages
What is already known on this subject?
Existing clinical prediction models (CPM) for mortality following transcatheter aortic valve implantation (TAVI) have only moderate predictive performance when applied to populations distinct from those in which they were developed. Additionally, frailty has been shown to be an important predictor of mortality following TAVI, with the recent recording of frailty in national registries presenting opportunities to include such measures within the risk prediction.
What might this study add?
This study presents a contemporary multivariable CPM for predicting 30-day mortality after TAVI that included 15 risk factors, 2 of which were measures of frailty. This is the first model to be derived from the UK-TAVI Registry and presents an important step in facilitating appropriate adjustment for case mix when making centre-level outcome comparisons. The validation procedure demonstrated that the model was well calibrated and moderately discriminating.
How might this impact on clinical practice?
The model has potential to be used for benchmarking analyses in UK-TAVI patients by adjusting for case mix, and could facilitate clinical discussions around risk during the consent process. However, further development is needed as more data become available, and other outcomes should be explored.
Acknowledgments
We acknowledge the National Institute for Cardiovascular Outcomes Research (NICOR) for providing the UK-TAVI Registry extract for this study.
References
Footnotes
Contributors GPM, MS and MAM made substantial contributions to the concept of the work in addition to designing and performing the analysis. GPM drafted the initial version of the manuscript. All of the authors interpreted the results, revised the paper critically for important intellectual content and approved the final version of the paper.
Funding This work was funded by the Medical Research Council through the Health e-Research Centre, University of Manchester (MR/K006665/1) and a grant through the North Staffordshire Heart Committee.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.