Original ArticleUpdating methods improved the performance of a clinical prediction model in new patients
Introduction
Setting a diagnosis or a prognosis in clinical practice is a multivariable process. Physicians combine patient characteristics and test results to estimate the probability that a disease or outcome is present (diagnosis) or will occur (prognosis). To guide physicians in their estimation of diagnostic and prognostic probabilities, many clinical prediction models have been developed. These prediction models are algorithms that combine patient characteristics to set a diagnosis or to predict a prognosis. Such models can improve the identification of high-risk patients, and guide medical decision making. Besides that, they can be used as a quality of care instrument, because the predicted risks can be compared with the observed actual outcomes.
The performance of prediction models needs to be tested in new patients before the models can be applied in clinical practice with confidence [1], [2]. Unfortunately, the predictive performance is often decreased when a model is tested in new patients, compared to the performance estimated in the patients who were used to derive the model. As a consequence, the original prediction model is frequently rejected, and a new prediction model is developed. While the original prediction model usually has been developed on a large data set, redevelopment then often occurs on the much smaller data set of the new patients only. When every new patient sample leads to a new prediction model, the prior information that is captured in previous studies and prediction models is neglected. This is counterintuitive to the notion that research should be based on data of as many data patients as possible. The principle of using knowledge of previous studies has been recognized in etiologic and intervention research, in which cumulative meta-analyses are more common. The alternative to redeveloping prediction models in every new patient sample is updating the existing prediction models. The updated models combine the information that is captured in the original model with the information of the new patients [3], [4], [5], [6], [7]. As a result, the updated models are adjusted to the new patients.
Recently, several updating methods have been proposed in the statistical literature [3]. The methods vary in extensiveness, which is reflected by the number of parameters that is adjusted or re-estimated. A relatively simple recalibration method, for instance, results in an updated model that has a new intercept and adjusted regression coefficients that are based on multiplication of the original coefficients with a single recalibration factor. With this method, only two parameters are estimated. More extensive updating methods estimate more parameters, for instance, when all individual regression coefficients are re-estimated.
In this study, we present an empirical example of updating a model to predict the risk of severe postoperative pain. We show the results of simple and more extensive updating methods.
Section snippets
Patients and methods
Severe postoperative pain occurs frequently after surgery. Patients at high risk of severe postoperative pain may benefit from preventative strategies. Therefore, a prediction model has been developed to predict the risk of acute severe postoperative pain.
Results
Approximately one-third of the patients in the updating (274, 36%) and test sets (100, 35%) reported severe postoperative pain in the first hour after surgery, compared to 62% (1,205) of the patients in the derivation set (Table 3). Patients in the updating and test sets were slightly older, they underwent more often ambulatory surgery, and had less often a large expected incision size than patients in the derivation set. The distribution of patient characteristics in the updating and test sets
Discussion
We tested a prediction model for the risk of acute severe postoperative pain in surgical patients. The patients were recruited in a later period and in another hospital than the patients on which the model was developed. As the predictive performance of the model was poor, updating was necessary to adjust the model to the new situation. We compared five updating methods that differed in extensiveness of the updating. In this example study, simple recalibration methods improved the calibration
Acknowledgment
We gratefully acknowledge the support by The Netherlands Organization for Scientific Research (ZON-MW 917.46.360).
References (23)
- et al.
Substantial effective sample sizes were required for external validation studies of predictive logistic regression models
J Clin Epidemiol
(2005) - et al.
Internal validation of predictive models: efficiency of some procedures for logistic regression analysis
J Clin Epidemiol
(2001) - et al.
Preoperative prediction of severe postoperative pain
Pain
(2003) - et al.
Assessing the generalizability of prognostic information
Ann Intern Med
(1999) - et al.
What do we mean by validating a prognostic model?
Stat Med
(2000) - et al.
Validation and updating of predictive logistic regression models: a study on sample size and shrinkage
Stat Med
(2004) - et al.
The importance of disease prevalence in transporting clinical prediction rules. The case of streptococcal pharyngitis
Ann Intern Med
(1986) - et al.
Transportability of a decision rule for the diagnosis of streptococcal pharyngitis
Arch Intern Med
(1986) - et al.
Applied logistic regression
(1989) - et al.
Ready-made, recalibrated, or remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery
Circulation
(1999)
Cited by (285)
Temporal validation and updating of a prediction model for the diagnosis of gestational diabetes mellitus
2023, Journal of Clinical EpidemiologyValidation of a Bayesian learning model to predict the risk for cannabis use disorder
2023, Addictive BehaviorsValidity of prognostic models of critical COVID-19 is variable. A systematic review with external validation
2023, Journal of Clinical EpidemiologyQuality of clinical prediction models in in vitro fertilisation: Which covariates are really important to predict cumulative live birth and which models are best?
2023, Best Practice and Research: Clinical Obstetrics and Gynaecology