Re: Accuracy and impact of risk assessment in primary prevention of cardiovascular disease
We applaud the systematic review by Brindle, et al (1), which explores a common use of the Framingham risk predictors. Kaiser Permanente, a large US Health Maintenance Organization, is among the organizations that use a modified Framingham risk predictor to help guide clinicians and patients in cardiovascular risk reduction interventions. The letters by Lenz and Eichler point out several issues with Brindle’s analysis. We would add several more observations.
It is not unexpected that populations with family histories of premature CHD would have CHD risk underestimated by the Framingham risk predictors, since family history is not an included variable (2, 3). The Joint British Societies’ Guidelines on Prevention of Cardiovascular Disease (4) superimpose an additional multiplier of 1.3X, on top of the predicted Framingham risk, for persons with a family history of premature CHD. This method was also adopted by Kaiser Permanente. Thus, the inclusion of the two Johns Hopkins cohorts may unfairly bias the results of the systematic review.
It has been previously documented that the Framingham risk predictor may underestimate CHD risk among people with diabetes (5). However, the dyslipidemia recommendations of the Joint British Societies (4) and the United States NCEP ATP III (6) classify diabetes as a CHD risk equivalent, and do not include a Framingham risk calculation for people with diabetes. Kaiser Permanente advises treatment of all people with diabetes aged 40 and older with statins, regardless of baseline LDL-cholesterol, based on evidence from the Heart Protection Study (7) and CARDS (8). So, although Framingham may indeed underestimate risk among people with diabetes, this would have little impact on Joint British, US or Kaiser Permanente lipid-lowering treatment recommendations.
The Brindle meta-analysis uses treated values for lipids and blood pressure. For blood pressure values in treated populations, the Anderson version of Framingham (3), which does not include a variable for antihypertensive treatment, would be expected to result in greater underestimation of risk than the Wilson version (4), which has a variable for antihypertensive treatment. For lipid values in a treated population, the West of Scotland Coronary Prevention Study (9) published their findings that suggested that the Framingham risk predictor, using observed on-treatment lipid values, accurately predicted subsequent CHD risk among the patients on placebo, but overestimated the risk among the patients on pravastatin. The ultimate effect on risk prediction accuracy of these treatment effects of opposite direction in a partially treated population is not clear.
It is also not clear how Brindle, et al, used the Framingham risk predictor. One would assume that they did not have access to individual- level data, which would force the estimation of the risk of a person with the mean values for all risk factors. This would likely skew the risk estimates, unless the values of the continuous variables were normally distributed. Furthermore, this method would not be tenable for the dichotomous variables of gender, smoking and diabetes. This particular issue could be addressed by subgroup estimates for each permutation of the dichotomous variables. But the specific methodology of the risk calculation was not explicit.
Brindle’s statement that “evidence supporting the use of cardiovascular risk scores for primary prevention is scarce” may be factual, but it overstates the case. The statement implies that the efficacy of risk scores has been examined, while the studies only examined the effectiveness. Proof of efficacy in a highly-controlled setting does not guarantee effectiveness in a more real-world setting. But lack of evidence of effectiveness, as shown by Brindle, does not prove lack of efficacy of risk assessment. Certainly, the existence of a risk score cannot affect outcomes unless that risk score is used in clinical decision-making.
In the effectiveness review, Brindle notes the disappointing impact on outcomes demonstrated by the four randomized studies of cardiovascular risk assessment. However, there is broader evidence on clinical decision support that shows decision support can improve outcomes under the right conditions (10-12). In addition to "including clinicians in the design of decision aids," as suggested by Brindle, other crucial determinants of computerized decision support usability, acceptance and effectiveness have been articulated (13, 14). Experience as reported here should not discourage us, but rather cause us to pay greater attention to design and implementation requirements for effective clinical decision support.
In summary, Brindle, et al, have done important work in examining the accuracy of the Framingham risk predictor in clinical practice. However, some sources of error have been accounted for by various groups (family history of premature CHD and diabetes), and a number of other limitations of the systematic review make it difficult to assess the remaining implications regarding accuracy of Framingham risk prediction. While the RCTs examining the effectiveness of use of cardiovascular risk assessment are disappointing, they do not disprove the efficacy of risk assessment. They may indicate short-comings in our design and implementation of clinical decision support, rather than any fundamental flaw or lack of utility of Framingham risk prediction, in particular, or decision support for cardiovascular risk assessment, in general.
Wiley V. Chan, MD
Physician, Internal Medicine
Director, Guidelines & Evidence-Based Medicine
Michael A. Krall, MD
Physician, Family Practice
Physician Lead, Decision Support and Ambulatory Patient Safety
Kaiser Permanente, Northwest
500 NE Multnomah Street
Portland, OR 97236
1. Brindle PM, Beswick AD, Fahey TF, Ebrahim SB: The accuracy and impact of risk assessment in the primary prevention of cardiovascular disease: A sytematic review. Heart 2006.
2. Anderson KM, et al. Cardiovascular Disease Risk Profiles. Am Heart J 1991;121:293-298.
3. Wilson PWF, et al. Prediction of Coronary Heart Disease Using Risk Factor Categories. Circulation 1998;97:1837-1847.
4. JBS 2: Joint British Societies’ Guidelines on Prevention of Cardiovascular Disease in Clinical Practice. Heart 2005;91(Suppl. 5):v1- v52.
5. McEwan PC, et al. The Unreliability of Framingham Risk Equations in Predicting Coronary Heart Disease (CHD) Events in Diabetes. Diabetes 2000;49(Suppl 1):A187.
6. National Cholesterol Education Program. Implications of Recent Clinical Trials for the National Cholesterol Education Program Adult Treatment Panel III Guidelines. Circulation 2004;110:227-239.
7. MRC/BHF Heart Protection Study of Cholesterol-Lowering with Simvastatin in 5963 People with Diabetes: A Randomised Placebo-Controlled Trial. Lancet 2003;361:2005-2016.
8. Colhoun HM, et al. Primary Prevention of Cardiovascular Disease with Atorvastatin in Type 2 Diabetes in the Collaborative Atorvastatin Diabetes Study (CARDS): Multicentre Randomised Placebo-Controlled Trial. Lancet 2004;364:685-696.
9. West of Scotland Coronary Prevention Group. West of Scotland Coronary Prevention Study: Identification of High-Risk Groups and Comparison with Other Cardiovascular Intervention Trials. Lancet 1996;348:1339-1342.
10. Delpierre C. Cuzin L. Fillaux J. Alvarez M. Massip P. Lang T. A systematic review of computer-based patient record systems and quality of care: more randomized clinical trials or a broader approach?. International Journal for Quality in Health Care. 16(5):407-16, 2004 Oct.
11. Garg A X, Adhikari N K, McDonald H, Rosas-Arellano M P, Devereaux P J, Beyene J, Sam J, Haynes R B, Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293(10):1223-1238.
12. Balas EA, Weingarten S, Garb CT, Blumenthal D, Boren SA, Brown GD. Improving preventive care by prompting physicians. Arch Intern Med. 2000 Feb 14;160(3):301-8.
13. Bates D, Kuperman GJ,Wang S, Gandhi T, Kittler A,Volk L, Spurr S, Khorasani S, Tanasijevic M, Middleton B. Ten Commandments for Effective Clinical Decision Support: Making the Practice of Evidence-based Medicine a Reality. J Am Med Inform Assoc 2003;10(6):523–530.
14. Krall MA, Sittig DF. Clinician’s assessments of outpatient electronic medical record alert and reminder usability and usefulness requirements. Proc AMIA Symp 2002:400-404.