rss
Heart 85:37-43 doi:10.1136/heart.85.1.37
  • Cardiovascular medicine

Comparative accuracy of cardiovascular risk prediction methods in primary care patients

Abstract

OBJECTIVE To compare the relative accuracy of cardiovascular disease risk prediction methods based on equations derived from the Framingham heart study.

DESIGN Risk factor data were collected prospectively from subjects being evaluated by their primary care physicians for prevention of cardiovascular disease. Projected cardiovascular risks were calculated for each patient with the Framingham equations, and also estimated from the risk tables and charts based on the same equations.

SETTING 12 primary care practices (46 doctors) in Birmingham.

PATIENTS 691 subjects aged 30–70 years.

MAIN OUTCOME MEASURES Sensitivity, specificity, and positive and negative predictive values of the Framingham based risk tables and charts for treatment thresholds based on projected cardiovascular disease or coronary heart disease risk.

RESULTS 59 subjects (8.5%) had projected 10 year coronary heart disease risks ⩾ 30%, and 291 (42.1%) had risks ⩾ 15%. At equivalent projected risk levels (10 year coronary heart disease ⩾ 30% and five year cardiovascular disease ⩾ 20%), the original Sheffield tables and those from New Zealand have the same sensitivities (40.0%, 95% confidence interval (CI) 26.6% to 57.8% v 41.2%, 95% CI 28.7% to 57.3%) and specificities (98.6%, 95% CI 97.2% to 99.3%v 99.7%, 95% CI 98.8% to 100%). Modifications to the Sheffield tables improve sensitivity (91.4%, 95% CI 81.3% to 96.9%) but reduce specificity (95.8%, 95% CI 93.9% to 97.3%). The revised joint British recommendations' charts have high specificity (98.7%, 95% CI 97.5% to 99.5%) and good sensitivity (84.7%, 95% CI 71.0% to 93.0%).

CONCLUSIONS The revised joint British recommendations charts appear to have the best combination of sensitivity and specificity for use in primary care patients.

Guidelines for the primary prevention of coronary heart disease published jointly by the British Cardiac, Hypertension, and Hyperlipidaemia Societies1 recommend that the use of cholesterol lowering drugs, antihypertensive agents, and aspirin should be based on the individual's projected risk of developing coronary heart disease or cardiovascular disease, and not on measurements of serum cholesterol or blood pressure alone. For cholesterol lowering treatment, individuals should have a projected risk of sustaining a coronary heart disease event in a 10 year period of ⩾ 30%, but a risk of only ⩾ 15% for initiating antihypertensive or aspirin treatment. These guidelines require quantitative risk assessment. The most widely used method for assessment of coronary heart disease and cardiovascular disease risks is based on equations derived from the Framingham heart study, which use eight weighted risk factors: age, sex, systolic or diastolic blood pressure, serum total and high density (HDL) cholesterol, and the presence and absence of left ventricular hypertrophy, diabetes mellitus, and cigarette smoking.2 To avoid the need for formal risk calculation, several types of risk tables and charts based on the Framingham equations have been developed, which can be used to estimate an individual's coronary heart disease or cardiovascular disease risk.1 3-12 With the exception of the Sheffield tables,5 13 these methods have not been validated against the Framingham equation on which they are based, so it is not possible to judge the performance of competing risk assessment methods. We have compared the accuracy of all of the published Framingham based risk tables or charts in subjects selected by their primary care physicians for coronary heart disease/cardiovascular disease risk assessment, using projected risks calculated with the Framingham equations as the reference method.

Methods

PATIENTS

Patients were selected at the discretion of their own primary care physicians, although local guidelines for primary prevention suggest that serum lipid measurements and coronary heart disease/cardiovascular disease risk assessments are indicated for those with one or more non-lipid risk factors (hypertension, smoking, diabetes mellitus, or a family history of premature coronary heart disease (onset < 55 years)). Patient data were collected between January 1998 and January 1999 from 12 general practices (46 general practitioners) served by the department of clinical biochemistry, Birmingham Heartlands Hospital. Although the Framingham equations apply to those aged 30–74 years, only those aged 30–70 years were included in our analysis, as this age range is used by all but one of the tables or charts. Individuals with left ventricular hypertrophy were also excluded as the majority of tables and charts do not include this risk factor.

DATA COLLECTION AND RISK CALCULATIONS

The methods of data collection and risk calculation have been described previously.14 In brief, the patient's systolic blood pressure, presence or absence of diabetes mellitus, cigarette smoking, and left ventricular hypertrophy (if known) were recorded on a self adhesive label attached to the pathology request form. Age and sex were recorded as usual on the same form. This was sent to the laboratory with a non-fasting blood sample, which was used for the measurement of total and HDL cholesterol. The laboratory information system software was adapted to use these data to calculate each patient's coronary heart disease/cardiovascular disease risks using the Framingham equations. Projected risks were defined as specified by each of the risk tables or charts. Cardiovascular disease risk is used only by the New Zealand tables and is projected over a five year interval. All the other tables and charts, with the exception of the Sheffield and modified Sheffield tables, use coronary heart disease risk projected over a 10 year interval. Both the original and modified Sheffield tables use coronary heart disease risk predicted for a five year interval, which was then doubled to give a projected 10 year risk (Sheffield tables) or divided by five to give an annual risk (modified Sheffield tables).

RISK ASSESSMENT TABLES

A summary of the risk factors used by each of the published tables or charts based on the Framingham equations is shown in table 1. Two observers used the clinical and biochemical data to assess coronary heart disease/cardiovascular disease risks for each patient with each of these published tables or charts. Risk was also calculated with the full Framingham equation. Calculated and estimated risks were compared where possible at the two treatment thresholds (projected 10 year coronary heart disease risk ⩾ 30% and ⩾ 15%, or at the equivalent annual coronary heart disease and five year cardiovascular disease risks) advised by the joint British recommendations. The European guidelines advise drug treatment at a projected 10 year coronary heart disease risk of ⩾ 20%, and their tables have no ⩾ 30% band: the ⩾ 20% level was used for comparison with risks calculated from the Framingham equation. The maximum 10 year coronary heart disease risk for women in the Framingham categorical method is ⩾ 27%; this projected risk level was used in our evaluation of this method, rather than the ⩾ 30% threshold.

Table 1

Summary of risk assessment methods based on the Framingham equations

The original Sheffield tables3 have 12 vertical columns of cholesterol concentrations, each column applying to a different permutation of the four categorical risk factors: diabetes, smoking, left ventricular hypertrophy, and hypertension (defined as a systolic blood pressure of > 160 mm Hg). Each row in the table applies to a different age, with rows differing by intervals of two years. By reading along the row which relates to the patient's age to the column which corresponds to their combination of risk factors, the total serum cholesterol concentration at which they would have a projected 10 year coronary heart disease risk ⩾ 30% can be determined. No account is taken of the subject's actual HDL cholesterol concentration, but an assumption is made that they would have an average value (1.15 mmol/l for men and 1.4 mmol/l for women). There are separate tables for men and women. The modified Sheffield tables4 5 use the same design, but display total to HDL cholesterol ratios rather than total cholesterol concentrations, and show the ratios that correspond to projected annual coronary heart disease risks of both ⩾ 3.0% and ⩾ 1.5%. Hypertension is also defined differently (systolic blood pressure > 140 mm Hg).

The New Zealand tables6 consist of a matrix of coloured cells with four rows and five columns. Each row relates to a different band of systolic blood pressure, which differ by 20 mm Hg (120 to 180 mm Hg), and each column to a different total to HDL cholesterol ratio (4 to 8). The colour of the cell corresponds to a different five year cardiovascular disease risk level (< 2.5%, 2.5–5%, 5–10%, 10–15%, 15–20%, and > 20%). A projected five year cardiovascular disease risk of 20% is believed to be equivalent to a 10 year coronary heart disease risk of 30%. There is a different matrix for each permutation of other risk factors (presence/absence of diabetes and smoking, sex, and for each of four decades of age from 40–70 years), giving a total of 32 matrices. The modified New Zealand tables7 also include patients aged 30–40 years, adding a further eight matrices. The updated New Zealand tables, which have only recently been published,8 further stratify subjects with projected five year cardiovascular disease risks of ⩾ 20% by including risk bands of 20–25%, 25–30%, and > 30%. The European tables9 have a similar design to the New Zealand tables, but use coronary heart disease risks and total cholesterol concentrations (4–8 mmol/l) rather than cholesterol to HDL ratios. As with the original Sheffield tables, it is assumed that the patient has an average HDL cholesterol concentration (1.0 mmol/l for men and 1.1 mmol/l for women).

The risk prediction charts published with the joint British recommendations1 have a novel design in which projected 10 year coronary heart disease risk is represented graphically as a function of systolic blood pressure (110–210 mm Hg) on the vertical axis, and cholesterol to HDL ratio (3–12) on the horizontal axis. Coronary heart disease risk is classified into three zones (< 15%, 15–30%, and > 30%), denoted by different colours. As with the New Zealand tables, there are different risk prediction charts for each permutation of the other risk factors (presence/absence of diabetes mellitus and smoking, sex, and the four decades of age from 40–70 years), giving a total of 32 charts. These charts have also recently been modified such that patients' ages are classified as 35–44, 45–54, 55–64, or 65–74 years.10

The Canadian categorical system11 consists of a nomogram in which projected five and 10 year coronary heart disease risks are both plotted as a function of age. An individual's age related coronary heart disease risk is then altered using a score derived from their other risk factors. Total and HDL cholesterol and systolic blood pressure levels are each classified into bands with different scores, and points are also applied for the other categorical risk factors (presence/absence of cigarette smoking, diabetes mellitus, left ventricular hypertrophy, and to each sex). The points score changes the age related risk. The Framingham categorical risk assessment method12 also classifies each of the continuously distributed risk factors (cholesterol or LDL cholesterol, HDL cholesterol, systolic blood pressure, and age) into bands, with a score being applied to each band. A score is also applied for each of the other categorical risk factors (presence/absence of diabetes mellitus and cigarette smoking, sex). A table is used to convert the summation of a patient's risk factor score into a projected 10 year coronary heart disease risk.

STATISTICAL METHODS

SPSS for Windows was used. Patient data are described as mean (SD) when normally distributed, or when normalised by logarithmic transformation (cholesterol to HDL ratio), or as median and 5th/95th centiles (coronary heart disease risks) when not normally distributed. The sensitivity, specificity, and positive and negative predictive values were determined for each table or chart using risks calculated from the Framingham equations as the reference method. The 95% confidence interval (CI) for each value were calculated using the exact Poisson method.

Results

We received 806 risk requests during the study. Nine patients were excluded because they were aged < 30 years, 56 because they were aged > 70 years, five because they were known to have left ventricular hypertrophy, and 45 because the request forms were not completed in full, which precluded risk calculation. Data on the remaining 691 patients were evaluated (table 2). Of these, 229 (33.1%) had a serum cholesterol concentration of ⩾ 6.5 mmol/l, 160 (23.2%) were hypertensive (systolic blood pressure ⩾ 160 mm Hg), 138 (20.0%) had diabetes mellitus, and 145 (21.0%) were current cigarette smokers. Overall, 421 patients (60.9%) had one or more cardiovascular risk factors. The median 10 year projected coronary heart disease risk was 12.6% (range 0.1–48.3%) and median five year projected cardiovascular disease risk was 8.0% (range 0.1–46.7%). In 291 individuals (42%) the projected 10 year coronary disease risk was ⩾ 15%, and in 59 (8.5%) it was ⩾ 30%. In 280 individuals (40.5%) the projected five year cardiovascular disease risk was ⩾ 10%, and in 85 (12.3%) it was ⩾ 20%. There was a strong correlation between projected coronary disease and cardiovascular disease risks (R 2 = 0.84), as the Framingham equations for these use the same risk factors but with different weightings. The regression equation between the projected five year cardiovascular disease risk (CVD5) and 10 year coronary heart disease risk (CHD10) was CVD5 = CVD10*0.661 + 0.459, and hence 10 year coronary disease risks of 30% and 15% are equivalent to five year cardiovascular disease risks of 20.3% and 10.7%, respectively. Sensitivity, specificity, positive and negative predictive values, and their respective 95% CIs for the relevant risk thresholds are shown for all of the tables and charts grouped according to their design (tables 3, 4, 5, and6).

Table 2

Details of the subjects evaluated

Table 3

Performance of the original and modified Sheffield tables

Table 4

Performance of the original and updated New Zealand tables and the European tables

Table 5

Performance of the joint British recommendations risk charts

Table 6

Performance of the categorical methods of risk assessment

The original Sheffield tables have a low sensitivity and positive predictive value, such that only 40% of those eligible for cholesterol lowering treatment would be identified, and of those so classified, one in four actually have calculated risks of < 30% (table 3). They do, however, have a high specificity so that the false positive rate is low. The patient exclusions (serum cholesterol < 5.5 mmol/l and systolic blood pressure > 160 mm Hg) suggested by the authors of the original Sheffield tables reduce the numbers in whom risk can be assessed (370 of 691, 53.5%) but do not significantly improve sensitivity, specificity, or positive and negative predictive values. Following these patient exclusions, fewer true positives are identified (17 v 28). The changes introduced in the modified Sheffield tables significantly improve sensitivity at the projected annual risk threshold of ⩾ 3% (10 year ⩾ 30%) but reduce specificity. The positive predictive value is similar to that of the original tables, such that between one in three and one in four of those classified as having annual coronary heart disease risks of ⩾ 3% actually have lower calculated risks. The modified Sheffield tables also permit assessment at an annual risk threshold of ⩾ 1.5% (10 year ⩾ 15%). Specificity is significantly worse than at the ⩾ 3% risk level, and indeed is among the lowest of all the tabular and chart systems. Sensitivity is the same at both thresholds.

Despite including the patient's HDL cholesterol and blood pressure, the New Zealand tables have similar sensitivities and specificities (table 4) to the original Sheffield tables at the equivalent 30% 10 year coronary disease and 20% five year cardiovascular disease risk levels. At the 10% five year cardiovascular disease risk threshold, specificity is significantly lower, and although sensitivity is higher this is not statistically significant. The updated New Zealand tables have significantly better sensitivities at both risk thresholds than the original tables, although this is achieved at the expense of worse specificities. The European tables, with a similar design to those from New Zealand, have better sensitivity than the original Sheffield and New Zealand tables but their specificity is low and significantly worse than most of the other risk assessment methods (table 4). Their positive predictive value is extremely low, such that between one in two to one in three of those identified as having a coronary disease risk of ⩾ 20% actually have lower calculated risks.

The charts produced jointly by the British Cardiac, Hypertension, and Hyperlipidaemia Societies appear to have significantly better specificities at both the ⩾ 15% and ⩾ 30% 10 year coronary heart disease risk levels (table 5) than the modified Sheffield tables. The higher specificity of these charts is achieved at the expense of worse sensitivity, which is significantly lower than that of the modified Sheffield tables at the ⩾ 15% 10 year risk level, but not quite significantly so at the ⩾ 30% 10 year risk level. These charts have the same specificities as the New Zealand tables, the same sensitivity at the equivalent 30% 10 year coronary heart disease and 20% five year cardiovascular disease risk levels, but a higher sensitivity at the 15% 10 year coronary disease/10% five year cardiovascular disease risk level. Recently published modifications to the charts have no significant effects on sensitivity or specificity at either risk threshold (table 5).

Of the categorical methods of risk assessment, that published by the Framingham investigators has the best performance (table 6), with specificities equivalent to those of the modified Sheffield tables and sensitivities equivalent to the joint British charts. The Canadian method does not identify those with risks ⩾ 30% well, but performs better at the ⩾ 15% risk threshold, with sensitivities and specificities similar to those of the modified Sheffield tables (table6).

Discussion

Quantitative assessment of projected coronary heart disease/cardiovascular disease risk appears now to be the accepted basis for the use of drugs in the primary prevention of cardiovascular disease.15 This concept was initially established for cholesterol lowering drugs3 16 and has been extended to the use of antihypertensive agents and aspirin.1 17 18The Framingham equations are the most widely accepted method for projecting cardiovascular disease/coronary disease risks, and are used in the British, European, and New Zealand guidelines. The equations have been validated retrospectively in British subjects.19 20 Quantitative risks calculated from such equations are often described as “absolute”, but it has been argued that this term is a misnomer as it implies a sense of certainty that cannot be applied to prospective risk prediction.21 The Framingham equations have actual or potential defects as they do not include several well established risk factors, may not apply to ethnic groups other than north Europeans, and have not been validated in a prospective fashion. We have therefore used the term “projected” rather than “absolute” risk to indicate the uncertainty of the risk prediction methodology.

Several tabular and graphical formats of the Framingham equations have been developed to avoid the need for computation of risks in clinical practice. The authors of the risk tables or charts have generally assumed that risk estimates obtained from them will agree with risks calculated using the Framingham equations. The construction of tabular and graphical methods has required simplifications of the Framingham equations which do not necessarily ensure that risk estimates will be the same as risk calculations. As risk projection is an uncertain process it would be unfortunate were the tables or charts to introduce additional errors. It is surprising that with the exception of the original and modified Sheffield tables5 13 the performance of the risk assessment methods has not been evaluated. The predictive value of a screening test is usually dependent upon the prevalence of the condition in the population. We believe that a cohort of primary care patients being assessed for coronary heart disease prevention is the most appropriate group in which to evaluate the tables or charts, rather than in higher risk patients attending hypertension,13 diabetes,14 or lipid clinics.22 Our group of patients is likely to be typical of those who are screened in primary care for prevention of coronary heart disease and cardiovascular disease.

It is clearly desirable that risk tables and charts should have a high sensitivity (true positive rate) to identify the maximum number of those who should be considered for drug treatment. It is also important that they have as low a false positive rate (defined as 100 − specificity) as possible, so that they do not incorrectly identify subjects for treatment. The false positive rate and the specificity are of central importance when risk assessment methods are applied to a population in which the prevalence of treatable risk is relatively low, as in the general population,5 or even in those who are believed to be at high risk by their primary care physicians. UK guidelines for the use of statins have been predicated in part by a consideration of the affordability of drug treatment,23 and hence treatment thresholds have been set at a relatively high level, comparable to risks observed in those with existing coronary heart disease. Risk assessment methods that have low specificities and high false positive rates necessarily increase total prescribing costs.

The original Sheffield tables are arguably the best known to UK practitioners, having been circulated with the Drug and Therapeutics Bulletin 24 and by the chief medical officer.16 We have previously argued that these tables would be inaccurate as they ignore the individual's HDL cholesterol concentration.25 Subsequent evaluations of the tables in patients with hypertension,13hyperlipidaemia,22 diabetes mellitus,14 and now the data reported here, all confirm that the original tables identify only a minority of those whose projected coronary heart disease risks qualify them for statin treatment. Moreover, the patient exclusions recommended by the Sheffield tables appear to be unnecessary. Modifying the Sheffield tables so that they use cholesterol to HDL ratios rather than total cholesterol, and a different definition of hypertension, greatly improves their sensitivity but significantly worsens specificity. The low specificity leads to a false positive rate that is approximately equal to the true positive rate. Our data are similar, although not identical, to those from an evaluation of the modified Sheffield tables in the general population.5 That study showed sensitivities that were lower, though not significantly so, than those we have found, but specificities that were significantly higher. The reason for this is not clear as the only apparent difference between the two study cohorts was in their average projected level of risk, and this should not affect sensitivities or specificities.

The New Zealand tables include both cholesterol to HDL ratios and measurements of blood pressure, and thereby have been claimed to be inherently superior to the original Sheffield tables, although no evidence was produced to substantiate this claim.7 In contrast, we have found that the sensitivities and specificities of the New Zealand and original Sheffield tables at equivalent projected cardiovascular disease/coronary disease risk thresholds are the same. The relatively poor performance of the New Zealand tables may be related to the use of cholesterol to HDL ratios as whole numbers and blood pressures in 20 mm Hg bands, which necessarily requires an individual's values to be rounded to the nearest level in the table and may result in misclassification. Although revisions to the New Zealand tables improve sensitivities, specificities are considerably worse, such that false positives will far outnumber true positives. The revised tables cannot be recommended. The European tables with a similar format also have a very low specificity, which appears to reflect the use of assumed low HDL concentrations which will tend to overestimate risk levels systematically. The Canadian graphical risk assessment method also systematically underestimates coronary disease risk at the ⩾ 30% projected 10 year risk threshold, and although performance is better at the ⩾ 15% threshold, there is no reason to advocate the use of this system. The Framingham categorical method has sensitivities equivalent to those of the joint British recommendations charts but lower specificities; it would not therefore be the method of choice for primary care patients.

The charts produced jointly by the British Cardiac, Hypertension, and Hyperlipidaemia Societies have particularly high specificities at both the ⩾ 30% and ⩾ 15% projected 10 year risk thresholds, such that the false positive rate is only 3/1000 at the higher threshold, and 3/100 at the lower threshold. These specificities are among the highest of the risk assessment methods and their sensitivities are better than the other tables (original Sheffield and New Zealand) with equivalent specificities. Revisions to the charts appear to increase sensitivity without significant loss of specificity.

An audit of clinicians' opinions on the use of tabular or chart risk assessment methods suggested that the joint British recommendations charts and the New Zealand tables were preferred to the Sheffield tables.26 The modified joint British recommendations charts appear to represent the best combination of specificity, sensitivity, and ease of use, and therefore should be the method of choice for those who elect to use tables and charts. It has been suggested, however, that the direct calculation of coronary heart disease and cardiovascular disease risks with the Framingham equations programmed for personal27 28 or laboratory computers14 is practicable, and also desirable because the accuracy of the risk assessment with respect to the Framingham equation is assured. Such approaches would target drugs most efficiently.

Acknowledgments

We thank all of the 41 general practitioners who submitted coronary heart disease risk requests during the study.

References

Responses to this article