Original Article
Updating methods improved the performance of a clinical prediction model in new patients

https://doi.org/10.1016/j.jclinepi.2007.04.018Get rights and content

Abstract

Objective

Ideally, clinical prediction models are generalizable to other patient groups. Unfortunately, they perform regularly worse when validated in new patients and are then often redeveloped. While the original prediction model usually has been developed on a large data set, redevelopment then often occurs on the smaller validation set.

Recently, methods to update existing prediction models with the data of new patients have been proposed. We used an existing model that preoperatively predicts the risk of severe postoperative pain (SPP) to compare five updating methods.

Study Design and Setting

The model was tested and updated with a set of 752 new patients (274 [36] with SPP). We studied the discrimination (ability to distinguish between patients with and without SPP) and calibration (agreement between the predicted risks and observed frequencies of SPP) of the five updated models in 283 other patients (100 [35%] with SPP).

Results

Simple recalibration methods improved the calibration to a similar extent as revision methods that made more extensive adjustments to the original model. Discrimination could not be improved by any of the methods.

Conclusion

When the performance is poor in new patients, updating methods can be applied to adjust the model, rather than to develop a new model.

Introduction

Setting a diagnosis or a prognosis in clinical practice is a multivariable process. Physicians combine patient characteristics and test results to estimate the probability that a disease or outcome is present (diagnosis) or will occur (prognosis). To guide physicians in their estimation of diagnostic and prognostic probabilities, many clinical prediction models have been developed. These prediction models are algorithms that combine patient characteristics to set a diagnosis or to predict a prognosis. Such models can improve the identification of high-risk patients, and guide medical decision making. Besides that, they can be used as a quality of care instrument, because the predicted risks can be compared with the observed actual outcomes.

The performance of prediction models needs to be tested in new patients before the models can be applied in clinical practice with confidence [1], [2]. Unfortunately, the predictive performance is often decreased when a model is tested in new patients, compared to the performance estimated in the patients who were used to derive the model. As a consequence, the original prediction model is frequently rejected, and a new prediction model is developed. While the original prediction model usually has been developed on a large data set, redevelopment then often occurs on the much smaller data set of the new patients only. When every new patient sample leads to a new prediction model, the prior information that is captured in previous studies and prediction models is neglected. This is counterintuitive to the notion that research should be based on data of as many data patients as possible. The principle of using knowledge of previous studies has been recognized in etiologic and intervention research, in which cumulative meta-analyses are more common. The alternative to redeveloping prediction models in every new patient sample is updating the existing prediction models. The updated models combine the information that is captured in the original model with the information of the new patients [3], [4], [5], [6], [7]. As a result, the updated models are adjusted to the new patients.

Recently, several updating methods have been proposed in the statistical literature [3]. The methods vary in extensiveness, which is reflected by the number of parameters that is adjusted or re-estimated. A relatively simple recalibration method, for instance, results in an updated model that has a new intercept and adjusted regression coefficients that are based on multiplication of the original coefficients with a single recalibration factor. With this method, only two parameters are estimated. More extensive updating methods estimate more parameters, for instance, when all individual regression coefficients are re-estimated.

In this study, we present an empirical example of updating a model to predict the risk of severe postoperative pain. We show the results of simple and more extensive updating methods.

Section snippets

Patients and methods

Severe postoperative pain occurs frequently after surgery. Patients at high risk of severe postoperative pain may benefit from preventative strategies. Therefore, a prediction model has been developed to predict the risk of acute severe postoperative pain.

Results

Approximately one-third of the patients in the updating (274, 36%) and test sets (100, 35%) reported severe postoperative pain in the first hour after surgery, compared to 62% (1,205) of the patients in the derivation set (Table 3). Patients in the updating and test sets were slightly older, they underwent more often ambulatory surgery, and had less often a large expected incision size than patients in the derivation set. The distribution of patient characteristics in the updating and test sets

Discussion

We tested a prediction model for the risk of acute severe postoperative pain in surgical patients. The patients were recruited in a later period and in another hospital than the patients on which the model was developed. As the predictive performance of the model was poor, updating was necessary to adjust the model to the new situation. We compared five updating methods that differed in extensiveness of the updating. In this example study, simple recalibration methods improved the calibration

Acknowledgment

We gratefully acknowledge the support by The Netherlands Organization for Scientific Research (ZON-MW 917.46.360).

References (23)

  • Janssen KJM, Kalkman CJ, Grobbee DE, Bonsel GJ, Moons KGM, Vergouwe Y. Validation of a prediction rule for severe...
  • Cited by (285)

    View all citing articles on Scopus
    View full text