Article Text
Abstract
Context A recent overview of all CVD models applicable to diabetes patients is not available.
Objective To review the primary prevention studies that focused on the development, validation and impact assessment of a cardiovascular risk model, scores or rules that can be applied to patients with type 2 diabetes.
Design Systematic review.
Data sources Medline was searched from 1966 to 1 April 2011.
Study selection A study was eligible when it described the development, validation or impact assessment of a model that was constructed to predict the occurrence of cardiovascular disease in people with type 2 diabetes, or when the model was designed for use in the general population but included diabetes as a predictor.
Data extraction A standardized form was sued to extract all data of the CVD models.
Results 45 prediction models were identified, of which 12 were specifically developed for patients with type 2 diabetes. Only 31% of the risk scores has been externally validated in a diabetes population, with an area under the curve ranging from 0.61 to 0.86 and 0.59 to 0.80 for models developed in a diabetes population and in the general population, respectively. Only one risk score has been studied for its effect on patient management and outcomes. 10% of the risk scores are advocated in national diabetes guidelines.
Conclusion Many cardiovascular risk scores are available that can be applied to patients with type 2 diabetes. A minority of these risk scores has been validated and tested for its predictive accuracy, with only a few showing a discriminative value of ≥0.80. The impact of applying these risk scores in clinical practice is almost completely unknown, but their use is recommended in various national guidelines.
- Cardiovascular disease
- diabetes mellitus
- prediction
- validation
- impact assessment
- diabetic heart disease
- macrovascular disease
- microvascular disease
- epidemiology
- coronary artery disease
- obesity
- clinical hypertension
- prevention
- diabetes
- general practice
Statistics from Altmetric.com
- Cardiovascular disease
- diabetes mellitus
- prediction
- validation
- impact assessment
- diabetic heart disease
- macrovascular disease
- microvascular disease
- epidemiology
- coronary artery disease
- obesity
- clinical hypertension
- prevention
- diabetes
- general practice
Introduction
People with type 2 diabetes have a twofold increased risk of cardiovascular disease (CVD).1 2 Guidelines for the management of type 2 diabetes advocate calculating CVD risk to guide the initiation of appropriate treatment.3–5 Over the past decades many prediction models (or risk scores) have been developed to predict CVD, of which only a small number have been specifically developed for people with type 2 diabetes.6 However, most prediction models developed for the general population take diabetes into account and can therefore be applied to the diabetes population. The performance of several prediction models has been examined in different populations for their discrimination (ability to discriminate between patients who will get the disease and those who will not) and calibration (ability to correctly quantify the absolute risk), but the outcomes have varied widely.7–9
A systematic review by Chamnan et al6 provides an overview of CVD prediction models that have been developed in diabetes populations, and prediction models for the general population that have been validated in a diabetes population. However, new prediction models for the diabetes population have been developed since this review, and many more prediction models exist that can be applied to people with diabetes. Moreover, it is unknown whether applying a certain prediction model in clinical practice affects the treatment of patients with diabetes and subsequently improves cardiovascular outcome.
Clinical prediction models need to provide accurate and validated estimates of the risk of the targeted outcome, in order to be useful for clinical decision-making. Three stages of prediction modelling can be identified: (1) model development, which includes among other aspects determining the clinically relevant predictors, assigning the relative weights to these predictors and estimating the model's predictive performance, ideally after adjustment for overfitting or optimism with internal validation techniques; (2) assessment of the model's predictive performance in new patients (external validation studies); (3) quantifying whether the use of a prediction model in daily practice indeed improves decision-making and subsequently patient outcome as compared with not using the model (model impact studies).10–15
The aim of this study was to identify all CVD prediction models (or scores or rules) that can be applied to patients with type 2 diabetes, and subsequently to assess their status and compliance in relation to the three stages of prediction modelling.
Methods
We started by systematically searching the literature for all CVD prediction models that can be applied to, or have been developed in, people with type 2 diabetes. Subsequently, we searched specifically for all studies that validated these models in patients with type 2 diabetes, followed by a search for all impact studies of these models and guidelines that incorporated these prediction models.
Search strategy
Existing CVD prediction models for patients with diabetes
At 1 April 2011, Medline was searched using a mixture of MeSH terms and truncated words to identify all papers presenting a CVD prediction model developed in patients with diabetes or that can be applied to individuals with type 2 diabetes. The precise search query is available in the appendix 1 (search term 1).
To identify additional studies we searched our own literature files, previous reviews of cardiovascular prediction models and checked citations. A study was eligible when (1) the prediction model was either developed in people with diabetes or included diabetes as a predictor, (2) the outcome of the prediction model was CVD or a cardiovascular component (ie, coronary heart disease (CHD), heart failure or stroke) and (3) it presented a specific prediction rule/model with sufficient information on all variables to calculate the CVD risk in a different population (β coefficients of the model or otherwise a scoring system/graph/score card/nomogram was provided). Exclusion criteria were non-human studies, articles in languages other than English or Dutch, studies presenting a prediction model developed in patients with previous CVD or other vascular condition (eg, hypertensive patients) or studies focusing on the added predictive value of new risk factors to an existing prediction model. Furthermore, we checked the reference list of all retrieved development studies, to determine whether any studies presenting a risk score had been missed.
External validation studies
After retrieval of all developed prediction models, we performed a second search to identify all studies that evaluated these models in a new, independent patient sample (search term 2, appendix 1). The resulting studies were added to the validation studies obtained with search term 1. Again, to identify potentially missed studies we searched our own literature files, previous reviews of cardiovascular prediction models and checked citations.
Impact studies
To find any impact studies of the retrieved cardiovascular prediction models, we first searched among the publications that were identified through the first and second search strategy. Furthermore, we developed a specific search query for impact studies for each prediction model separately. This strategy was based on search strategy for impact studies developed by Reilly and Evans16 (search term 3, appendix 1) combined with the specific prediction model's acronym, or if not available the name of the cohort in which the score was developed or first author of the paper.
Data extraction
Two authors (SVD and JWJB) independently reviewed all titles, followed by the abstract and full text of the studies. Any disagreements were resolved by a third (APK) and fourth (KGMM) reviewer. A standardised form was used for the data extraction, including outcome of the prediction model, number of events, specifics of the development population, presentation of the prediction model, type of model, results from an internal validation, selection procedure of, and variables included in, the prediction model.
For the data extraction of the external validation studies, a different data extraction sheet was used, including specifics of the validation population, number of events, type of outcome, statistical tests and measures of discrimination and calibration of the prediction model. From the retrieved impact studies additional data were extracted—namely, number of subjects, type of intervention, prediction model used in the intervention, type of outcome, number of outcomes and statistical tests.
Implementation of prediction models in guidelines
Finally, we searched various guidelines for clinical practice written in English to examine the implementation of the prediction models in countries in which the prediction models have been developed. As there are no strategies available for these types of internet searches, we specifically searched among the guidelines developed by the following organisations that are known to deal with patients with diabetes: fourth joint task force of the European Society of Cardiology and other societies on cardiovascular disease prevention in clinical practice,17 the American Diabetes Association18 National Institute for Health and Clinical Excellence (NICE),4 Joint British Society (JBS2),5 Canadian Diabetes Association,19 the Australian National Vascular Disease Prevention Alliance20 and International Diabetes Federation (IDF).21
Results
Figure 1 describes the systematic selection process of studies presenting a CVD prediction model applicable to type 2 diabetes patients. After screening 6803 titles, 218 studies remained for abstract screening; main reasons for exclusion were ‘aetiological or therapeutic research’ and ‘outcome other than CVD’. After examining the full text, 45 studies remained, of which 12 studies presented a CVD prediction model specifically developed for the diabetes population, and 33 included diabetes as a factor in the model. The majority of these models were developed in a predominantly Caucasian population and 12 were developed in Asian countries (India, China and Japan). The development study samples ranged from 698 to 1.5 million participants.
Cardiovascular prediction models developed in diabetes patient populations
Only two prediction models were developed in patients with newly diagnosed diabetes (both from the UK Prospective Diabetes Study (UKPDS study)),22 23 the other development populations were a mixture of people with varying duration of diagnosed diabetes (table 1). Six of these prediction models estimate CHD risk, while four estimate total CVD risk. The majority of the prediction models predicted 5-year risk with an average of eight predictors. The most commonly used predictors in these prediction models were age, sex, duration of diagnosed diabetes, HbA1c concentration and smoking (online appendix table 1).
Nine out of 12 studies reported the discrimination of their prediction model, with an area under receiver operating characteristic curve (AUC) ranging from 0.68 to 0.85. Eight studies also reported the calibration: all Hosmer–Lemeshow p values were >0.05, indicating no significant lack of calibration. Only half of these models examined the internal validation (performance of the model in the development population when the prediction model is corrected for overoptimism) using split samples or bootstrapping techniques. All studies reported the original model with the β coefficents, and two studies additionally presented a risk chart.
Cardiovascular prediction models developed in general populations including diabetes as predictor
Prediction models derived from general populations which included diabetes as a predictor used a variety of cardiovascular outcomes: 14 had CHD as the endpoint, 12 had CVD, eight had stroke and three had myocardial infarction (table 2). Most of these prediction models predicted 10-year risk using an average of eight predictors. The most commonly used predictors, in addition to diabetes, were age, sex, systolic blood pressure, smoking and cholesterol measurements (online appendix table 2).
Twenty of 33 studies gave a measure of validation, generally the AUC, which ranged from 0.65 to 0.86. Fourteen studies assessed calibration, with all Hosmer–Lemeshow p-values >0.05. Twelve of these prediction models were internally validated using either split sample, cross validation or bootstrapping techniques. There was a great variety in the way in which the prediction model was presented. The majority presented the original model with the β coefficients, others presented a scoring chart, nomogram or produced a computerised software program that was available through a website.
Validation studies of the CVD prediction models
Thirty studies externally validated 14 different prediction models in patients with type 2 diabetes, nine studies validated two versions of the UKPDS risk engine,22 23 10 studies validated three versions of the Framingham Prediction model41 62 65 66 and nine other prediction models were externally validated only once (table 3). The UKPDS risk engine for stroke22 was validated in two studies, which obtained very different results. One study observed an AUC of 0.61 and a poor calibration, while the other observed an AUC of 0.86 and good calibration. The UKPDS risk engine for CHD23 was validated in eight studies. Discrimination ranged from 0.65 to 0.76, and most of these studies observed poor calibration with an overestimation of the risk.
The Framingham prediction models by D'Agostino et al41 and by Anderson et al65 66 were validated three times in diabetes populations and the prediction model developed by Wilson et al62 four times. The AUC varied widely, ranging between 0.56 and 0.80 and the calibration was poor (p<0.001). Of the prediction models that were externally validated once, the Fremantle prediction model had the best discrimination with an AUC of 0.84 and a good calibration (p=0.85).
Impact studies of the CVD prediction models and incorporation in clinical guidelines
Only three studies examined the impact of applying a prediction model in clinical practice. All three studies used a randomised controlled trial design and examined the impact of the Framingham prediction model78–80 on treatment and prevention of CVD (table 4). Two of these studies were restricted to patients with a history of hypertension and the other study consisted of patients with type 2 diabetes (n=323). Half of them were allocated to an intervention group for which the cardiovascular prediction model was noted on the patient's records, while in the control group it was not. No difference was observed in prescription of glucose control, blood pressure-lowering and lipid-modifying treatments. However, restricting analysis to the high-risk group, patients in the intervention group were more likely to receive lipid-modifying or blood pressure-lowering prescriptions.78
Five out of seven guidelines recommended calculating cardiovascular risk in patients with type 2 diabetes with a prediction models that can be applied to the diabetes population (table 3). The IDF21 and NICE guidelines4 recommended using the UKPDS risk engine,23 as did the Canadian Diabetes Association,19 although this guideline mentioned PROCAM43 and the Strong Heart prediction model47 as well.
The EASD3 recommends using Framingham41 and DECODE52 as the preferred prediction models for calculating CVD risk. The Australian National Vascular Disease Prevention Alliance20 recommends using the Framingham prediction model41 as well, but they also mention the UKPDS risk engine.23 The JBS25 has developed risk charts based on the Framingham prediction model.41
Discussion
This systematic review provides an overview of all CVD prediction models that are specifically developed for, or can be applied to, patients diagnosed with type 2 diabetes to calculate future cardiovascular risk. Twelve prediction models are specifically designed for patients with type 2 diabetes and 33 have been developed in the general population accounting for diabetes as a predictor in the model. Only 31% of these prediction models have been examined for their external validity, with varying results. Interestingly, the impact on treatment and prevention of CVD by applying a prediction model has only been examined for the most commonly known CVD prediction model—that is, the Framingham prediction model.65 Nonetheless, several prediction models are incorporated in guidelines for the management of type 2 diabetes.
There are an extensive number of CVD prediction models with great variety in their quality and the methodology used to develop them. However, some of the older prediction models were developed when the statistical analysis methods for constructing such models were not yet established. The discriminative ability of both diabetes-specific CVD prediction models and prediction models that account for diabetes in the datasets from which they were developed was mostly good, with a c statistic often >0.70. However, the discriminative ability in new patients (validation studies) varied widely, as did the methods used to assess the performance in new patients. The discrimination of the prediction models designed for the general population was moderate (0.59 to 0.80) and the calibration mostly poor. This could be explained by a difference in incidence of CVD between the general and diabetes populations or that CVD prediction models developed in the general population do not account for diabetes-specific risk factors, as was suggested by Chamnan et al.6 They argued that this problem could be overcome by using only diabetes-specific prediction models for patients with diabetes. However, when diabetes-specific CVD prediction models where tested in other diabetes patient samples, the calibration was also poor with a moderate to good discrimination.
The more contemporary models, like the DCS,26 Fremantle25 or DARTS31 seem to have the best external validity, but these were validated in other patient populations only once. Therefore, more validation studies on the performance of these prediction models in different diabetes populations are needed.
The moderate performance of most prediction models suggests that it is difficult to predict CVD in patients with type 2 diabetes. Many biomarkers for CVD are emerging and being tested for their added prognostic value by extending the prediction models with these biomarkers.14 81
Few studies have assessed the impact of applying a CVD prediction model in clinical practice. We found only one study that assessed the impact of a prediction model (ie, the Framingham risk model). Consequently, it is unknown if the use of prediction models will indeed change treatment of patients with diabetes and thus reduce the number of cardiovascular complications. Even though the effect of applying a prediction model in clinical practice has not been established, guidelines recommend using CVD prediction models to determine appropriate treatment of diabetes. Interestingly, these guidelines refer both to the Framingham prediction model, which was assessed for its impact, but also to other prediction models, many of which have not been comprehensively validated. For instance, the EASD refers to the DECODE prediction model, which has only been validated once, with moderate performance. Other guidelines refer to the UKPDS risk engine, which indeed has been validated by several studies, but in most studies it showed poor calibration and severely overestimated the risk. Considering that the validation and impact of most prediction models has not been assessed, there is a great need for such studies to support their incorporation in clinical guidelines.
Prediction models which have been developed in the general population and do not correct for diabetes as a factor were excluded from this review. Therefore we did not include, for example, the Systematic COronary Risk Evaluation (SCORE).82 This model predicts fatal CVD over 10 years based on five predictors—namely, sex, age, smoking, systolic blood pressure and either total cholesterol or ratio total/high-density lipoprotein-cholesterol. The predictive ability of SCORE in patients with type 2 diabetes, however, has been assessed by three studies and was similar to other CVD prediction models included in our review.72 74 83
In conclusion, three stages of constructing a good prediction model can be identified: (1) development and internal validation of a model; (2) external validation and (3) assessment of the impact of using the model in practice. Many studies exist that present a CVD prediction model that can be applied to the diabetes population. A minority of studies have externally validated this large number of clinical prediction models in a diabetes population. Assessment of the impact on diabetes treatment and complications has been made for only one prediction model. Nevertheless, several prediction models are incorporated in guidelines for the management of type 2 diabetes and prevention of cardiovascular complications. New studies investigating prediction of CVD among patients with type 2 diabetes should, in our view, focus on further validating the performance of existing prediction models in new patients,10 improving the predictive performance of these existing models by adjusting them to local circumstances or adding new predictors14 81 84 and assessing their impact on treatment and prevention of cardiovascular events instead of developing new prediction models.11 16
Acknowledgments
This research was performed within the framework of CTMM, the Center for Translational Molecular Medicine (http://www.ctmm.nl), project PREDICCt (grant 01C-104), and supported by the Netherlands Heart Foundation, Dutch Diabetes Research Foundation and Dutch Kidney Foundation.
Appendix
Search terms used in this study
Search term 1
The search term used for identification of prediction models that are devolved in, or can be applied to, diabetes individuals: ((Validat$ OR Predict$.ti. OR Rule$) OR (Predict$ AND (Outcome$ OR Risk$ OR Model$)) OR (Decision$ AND (Model$ OR Clinical$ OR Logistic Models/)) OR (Prognostic AND (History OR Variable$ OR Criteria OR Scor$ OR Characteristic$ OR Finding$ OR Factor$ OR Model$)) OR (“risk score”[All fields] OR “prediction model”[All fields] OR “prediction rule”[All fields] OR “risk assessment” [All fields] OR “algorithm”[All fields])) AND (cardiovascular OR coronary OR cerebrovascular OR heart OR stroke) AND (diabetes OR “diabetes mellitus” OR “type 2 diabetes”) NOT (Animals[MeSH] NOT Humans[MeSH]).
Search term 2
Additional search term used to identify all validation studies of the prediction models: ((“diabetes”[All fields] OR “diabetes mellitus”[All fields] OR “type 2 diabetes”[All fields]) AND (“risk score”[All fields] OR “prediction model”[All fields] OR “risk model”[All fields] OR “predicting”[All fields] OR “predictive model”[All fields] OR “prediction tool”[All fields] OR “prediction rule”[All fields] OR “risk assessment”[All fields] OR “algorithm”[All fields] OR “validation”[All fields] or “discrimination”[All fields] OR “calibration”[All fields]) AND (“cardiovascular”[All fields] OR “coronary heart”[All fields] OR “heart”[All fields] OR “stroke”[All fields] OR “cerebrovascular”[All fields]) AND (Humans[Mesh])).
Search term 3
Search term used to identify all impact studies, which is combined with each specific risk scores acronym, or if not applicable the name of the cohort in which the score was developed or first author:
(Effectiveness [tiab] OR Comparing [tiab] OR Compared [tiab]OR Evaluate [tiab]) AND (Algorithm [tiab] OR Strategy [tiab] OR Managed [tiab] OR Management [tiab] OR Decision [tiab]) AND (cardiovascular OR coronary OR cerebrovascular OR heart OR stroke) AND (diabetes OR “diabetes mellitus” OR “type 2 diabetes”) NOT (Animals[MeSH] NOT Humans[MeSH]).
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data Supplement - Manuscript file of format pdf
Footnotes
Funding KGMM receives funding from the Netherlands Organisation for Scientific Research (project 9120.8004 and 918.10.615). APK was in receipt of a fellowship from the International Society of Hypertension.
Competing interests None.
Provenance and peer review Commissioned; externally peer reviewed.