Article Text


Hypothetical ratings of coronary angiography appropriateness: are they associated with actual angiographic findings, mortality, and revascularisation rate? The ACRE study


OBJECTIVE To determine whether ratings of coronary angiography appropriateness derived by an expert panel on hypothetical patients are associated with actual angiographic findings, mortality, and subsequent revascularisation in the ACRE (appropriateness of coronary revascularisation) study.

DESIGN Population based, prospective study. The ACRE expert panel rated hypothetical clinical indications as inappropriate, uncertain, or appropriate before recruitment of a cohort of real patients.

SETTING Royal Hospitals Trust, London, UK.

PARTICIPANTS 3631 consecutive patients undergoing coronary angiography (no exclusion criteria).

MAIN OUTCOME MEASURES Angiographic findings, mortality (n = 226 deaths), and revascularisation (n = 1556 procedures) over 2.5 years of follow up.

RESULTS The indications for coronary angiography were rated appropriate in 2253 (62%) patients. 166 (5%) coronary angiograms were performed for indications rated inappropriate, largely for asymptomatic or atypical chest pain presentations. The remaining 1212 (33%) angiograms were rated uncertain, of which 47% were in patients with mild angina and no exercise ECG or in patients with unstable angina controlled by inpatient management. Three vessel disease was more likely among appropriate cases and normal coronaries were more likely among inappropriate cases (p < 0.001). Mortality and revascularisation rates were highest among patients with an appropriate indication, intermediate in those with an uncertain indication, and lowest in the inappropriate group (log rank p = 0.018 and p < 0.0001, respectively).

CONCLUSION The ACRE ratings of appropriateness for angiography predicted angiographic findings, mortality, and revascularisation rates. These findings support the clinical usefulness of expert panel methods in defining criteria for performing coronary angiography.

  • coronary artery disease
  • coronary angiography
  • coronary artery bypass graft
  • appropriateness of coronary revascularisation

Statistics from

The management of suspected coronary artery disease (CAD) poses a central question: in which patients should coronary angiography be performed? Angiography is the most frequently performed procedure among people older than 65 years1 and the rate of angiography is predicted to continue to increase in Europe and the USA.2 3 However, despite the growing number of randomised trials of invasive management of CAD,4 considerable uncertainty remains about the appropriate indications for coronary angiography. Many of the latest3 recommendations for the use of coronary angiography are based on non-randomised studies or on unquantified expert consensus.

Quantification of the appropriateness of angiography using expert panels allows each patient's indication to be rated as appropriate, uncertain, or inappropriate, by using evidence integrated from randomised and observational studies as well as clinical opinion.5 Angiography is defined as appropriate for a specific indication when the expected health benefit (for example, from subsequent revascularisation) exceeds the expected negative consequences by a sufficiently wide margin to make it worth doing. The appropriateness method aims to provide a tool to investigate and improve the quality of patient care by identifying over and underuse.

However, the clinical relevance of the appropriateness method has not been adequately tested: a clinician needs to know whether appropriateness ratings risk stratify patients. Thus, if ratings of appropriateness have clinical validity, they should predict the presence and severity of CAD and, in the presence of CAD, subsequent revascularisation rates. Ratings should also predict survival. Previous appropriateness studies6-15 were all retrospective and tended to lack such clinically relevant measures. Furthermore, they predated the widespread use of coronary stenting and the trials of catheter based interventions and found widely differing rates of inappropriate use of coronary angiography (2%9 to 21%7).

The aim of this report therefore was to determine whether ratings of coronary angiography appropriateness derived by expert panel on hypothetical patients are associated with actual angiographic findings and over a 2.5 year follow up with mortality and subsequent revascularisation in the ACRE (appropriateness of coronary revascularisation) study.16


The overall design of the ACRE study is shown in fig1.

Figure 1

Appropriateness of coronary revascularisation (ACRE) study design.


The ACRE appropriateness ratings for coronary angiography were determined in 1995 before recruitment of the cohort and their validity and reliability have been reported.17 Using a technique developed by the RAND corporation in the USA,7 a nine member expert panel (four cardiologists, three cardiothoracic surgeons, a general physician, and a general practitioner) rated 2178 mutually exclusive indications for coronary angiography. A specific indication is defined by logical combinations of clinical presentation, test results, and current treatment. Median scores ranged from 1–9, with 1–3 being inappropriate, 4–6 uncertain, and 7–9 appropriate. Angiography was “inappropriate” when risks exceeded benefits and “uncertain” when benefits and risks approximated equality or when the expert panel was divided in its judgement. A particular indication was defined to be homogeneous, in that performing the procedure would be equally appropriate (or inappropriate) for all of them. The panel's rating of appropriateness was assigned by computer to each specific indication.


Patients were eligible for inclusion in the ACRE study if they underwent emergency or elective coronary angiography at the Royal Hospitals Trust, London between 15 April 1996 and 14 April 1997 and they lived within an area defined by five contiguous health authorities. There were no exclusion criteria and 4121 patients were identified. The resident population of the health authorities was 2.833 million (procedure rate 1700 per million) and 89% of the angiographies performed on their residents were done at this centre. Ethical approval for the study from the five local research ethics committees and written informed patient consent were obtained.


On the day of their index coronary angiography, eligible patients were identified by examination of ward admission and catheter laboratory log books. Data were extracted from case notes by trained nurses using standardised recording forms. Details were obtained on clinical presentation (RAND definitions5), functional severity of angina (Canadian Cardiovascular Society classification18), present medications, smoking status, hypertension, plasma cholesterol, diabetes, and exercise ECG results (coded as very positive, positive, indeterminate, or negative5). These data items were then used to allocate each patient to a specific, predefined clinical indication for angiography based on the RAND lists.


Angiographic findings were obtained from the free text angiogram report held in the case notes and coded blind to the clinical details by a trained coder. The severity of disease in each of 27 coronary artery segments (as defined by CASS (coronary artery surgery study)) was coded from 1 (none) to 6 (occluded) and the number of diseased vessels was calculated.19 20 To assess the reliability of this approach, two cardiologists over-read a random subsample of 209 angiograms, blinded to clinical details. The cardiologists showed good agreement beyond chance with the trained coder, with weighted κ of 0.64 and 0.63.21


Data were available from 4020 (98%) case notes and 4076 (99%) angiogram reports. In 226 (6%) patients the indication for angiography was not found in the predefined indication lists (in the majority of cases this involved angiography as part of a clinical trial or as assessment preceding coronary artery bypass grafting (CABG) after a prolonged wait). In 264 patients the case note or angiographic data were incomplete. Thus the final study population comprised 3631 (88%) patients who satisfied a predefined indication, had complete data, and were assigned an appropriateness rating.


First revascularisation procedures performed after index coronary angiography were identified by cross checking electronic information systems nationally (the National Health Service (NHS)-wide clearing system) and at the hospital against catheter laboratory and theatre log books. There were 1693 revascularisations (640 percutaneous transluminal coronary angioplasties (PTCAs) and 1053 CABGs), 1396 in those with appropriateness ratings.


Patients were followed up for death over a median (range) 30 (0–36) months. For the analyses presented here, patients were censored at the date of revascularisation or at 14 April 1999, whichever was earlier. In total, 3606 (99%) patients were flagged (using their NHS number as a unique identifier) for mortality at the Office for National Statistics central registry and the date and cause of death notified. There were 226 all cause deaths. The underlying cause of death was defined as coronary (International classification of diseases, ninth revision (ICD-9) codes 410–414, n = 119 deaths) or cardiovascular (ICD-9 codes 390–459, n = 47).


Proportions were compared using the χ2 statistic. Cox proportional hazard models were used to estimate hazard ratios and their 95% confidence intervals (CIs) for comparing revascularisation rates. Log rank p values were obtained across the three appropriateness categories. A multivariate score (from Duke22) for determining the pretest probability of CAD was estimated from published coefficients23 for age, sex, smoking, hypercholesterolaemia, diabetes, previous myocardial infarction (Q wave or not), and pain type (typical, atypical, none). All analyses were performed in SAS v.6 (SAS Institute, Cary, North Carolina, USA).


Patients in the ACRE cohort had a median (range) age of 61 (21–89) years and 29% were women (table 1). Twenty four per cent had previously undergone coronary angiography.

Table 1

Demographic and clinical characteristics of patients undergoing coronary angiography in the ACRE (appropriateness of coronary revascularisation) study (n=3631)

Table 2 shows the clinical presentation for which patients underwent coronary angiography. Clinical presentation was defined as chronic stable angina (n = 1431, 39% of the cohort), unstable angina (n = 568, 16%), acute myocardial infarction (n = 216, 6%), postmyocardial infarction (n = 167, 5%), post-revascularisation (n = 529, 15%), asymptomatic (n = 154, 4%), atypical chest pain (n = 175, 5%) near sudden death or ventricular arrhythmia (n = 70, 2%), and miscellaneous (n = 321, 9%). Exercise ECG was performed in 47% overall (table 1), although this varied by clinical presentation: chronic stable angina class I/II (70%), chronic stable angina class III/IV (60%), unstable angina (30%), acute myocardial infarction (14%), post-myocardial infarction (39%), postrevascularisation (44%), asymptomatic (84%), atypical chest pain (75%), near sudden death (32%), ventricular arrhythmia (54%), and miscellaneous (4%).

Table 2

Frequency of appropriate, uncertain, and inappropriate coronary angiography by clinical presentation: indications are listed that describe > 75% of the patients within given presentation or appropriateness category

Figure 1 shows that of the 2178 indications that were rated by the expert panel, only 241 (11.1%) were actually represented by the patients; a subset of 43 indications described 75% of the patient sample.

Overall, 166 (5%) of coronary angiograms were performed for indications rated inappropriate, 1212 (33%) uncertain, and 2253 (62%) appropriate. The panellists rated 436 indications as inappropriate; but only 166 patients (5%, represented by 39 indications) actually underwent angiography for inappropriate indications. Two clinical presentations—asymptomatic and atypical chest pain—comprised 73% of these inappropriate angiograms. The two most common uncertain clinical indications for coronary angiography (comprising 47% of the uncertain group) were patients with mild angina and no exercise ECG or patients with unstable angina controlled by inpatient management.

Table 3 shows the graded association between appropriateness categories and the number of diseased vessels at angiography. Patients who had had a previous revascularisation, and therefore were known to have CAD, were excluded from this analysis. The proportion of angiograms showing three vessel or left main stem disease was highest in the appropriate group (n = 458, 25%), intermediate in the uncertain category (n = 187, 18%), and lowest in the inappropriate category (n = 16, 11%) (p value for heterogeneity < 0.001).

Table 3

Angiographic findings in numbers of patients by appropriateness (excluding patients with previous revascularisation

The ability of appropriateness categories to predict the presence or absence of CAD was compared with the Duke score and result of the exercise ECG among 1317 patients with complete covariates for each of the three measures. The sensitivities were 779 of 967 (81%) for appropriate compared with not appropriate (uncertain and inappropriate), 831 of 967 (86%) for a Duke score in the top four quintiles, and 755 of 967 (78%) for a positive exercise ECG. The corresponding specificities were 119 of 350 (34%), 149 of 350 (43%), and 145 of 350 (41%), respectively.

The rate of all cause mortality was strongly related to the appropriateness of the angiogram (fig 2). Mortality was highest among patients undergoing angiography for appropriate indications, intermediate in the uncertain group, and lowest in the inappropriate group (logrank p = 0.018). The proportions of patients who died were 151 of 2253 (6.7%), 67 of 1211 (5.5%), and 8 of 166 (4.8%), respectively. In patients undergoing angiography for appropriate compared with those undergoing angiography for inappropriate indications, the hazard ratio for mortality was 1.75 (95% CI 0.9 to 3.6). These findings were consistent for coronary mortality (1.53, 95% CI 0.6 to 3.8; logrank p = 0.025), and cardiovascular mortality (2.08, 95% CI 0.8 to 5.1; logrank p = 0.0128).

Figure 2

All cause mortality after angiography, by appropriateness category.

The rate of revascularisation among those with CAD was also strongly related to the appropriateness of the angiogram. The revascularisation rate was highest among patients undergoing angiography for appropriate indications, intermediate in the uncertain group, and lowest in the inappropriate group (logrank p < 0.0001). The proportions of patients undergoing revascularisation were 1035 of 1676 (62%), 451 of 822 (54%), and 39 of 95 (41%), respectively. In patients undergoing angiography for appropriate compared with those undergoing angiography for inappropriate indications, the hazard ratio for revascularisation was 1.76 (95% CI 1.3 to 2.5). Figures 3 (PTCA) and 4 (CABG) show that the association of appropriateness category with subsequent revascularisation was confined to CABG. The hazard ratio for CABG in appropriate compared with inappropriate indications was 2.21 (95% CI 1.4 to 3.4), logrank p < 0.0001.

Figure 3

Percutaneous transluminal coronary angioplasty (PTCA) after angiography, by appropriateness category.

Figure 4

Coronary artery bypass grafting (CABG) after angiography, by appropriateness category.


In a cohort without exclusion criteria of over 3000 patients undergoing coronary angiography followed up for 2.5 years, the ACRE ratings of appropriateness predicted the number of diseased vessels, survival, and subsequent revascularisation. Rates were highest in patients undergoing angiography for appropriate indications, intermediate in those with an uncertain indication, and lowest in the inappropriate group. Although many previous expert panels have rated angiography appropriateness, none has prospectively shown such clinical validity. The applicability of this method in routine practice is further enhanced as the ACRE appropriateness ratings were tested in a cohort of angiography patients without exclusion criteria. For specific subsets of clinical indications, evidence, technology, and practice evolve rapidly and in these areas the ratings require refinement. However, expert panels provide the only validated method of systematically defining criteria for patients undergoing angiography, in which potential benefits are quantified.

Coronary angiography is performed to diagnose the presence and severity of CAD, to identify high risk patients, and thereby to determine the need for subsequent revascularisation. To determine the clinical relevance and validity of the appropriateness ratings, we therefore tested each of these three outcomes. Firstly, appropriateness ratings, if valid, should predict the number of diseased vessels. The hypothesis was confirmed: the three appropriateness categories showed a graded relation with the number of diseased vessels. Furthermore, the sensitivity and specificity of the appropriateness category in predicting CAD were as good as the exercise ECG or a multivariate score (from Duke) of other non-invasive predictors of CAD. Such relations have not previously been shown, although a small study showed that the presence of CAD was more likely in appropriate patients.13

Secondly, an angiogram performed for an appropriate indication should identify patients with high risk of mortality. This was confirmed. Since CABG may offer prognostic benefit in specific patient subgroups we censored the mortality follow up before revascularisation. We have previously reported how the expert panel ratings of coronary revascularisation appropriateness identified substantial underuse of these procedures; such underuse was associated with adverse clinical outcomes.16

The third test of clinical relevance was that appropriateness ratings should be associated with subsequent revascularisation rate, independent of any associations with the presence of CAD. As hypothesised, a graded association among the three appropriateness categories and revascularisation rate was seen among patients with CAD, with the lowest rate among those with inappropriate indications. This effect was striking for CABG, reflecting the association between the appropriateness of angiography and triple vessel and left main stem disease. These results are also consistent with the importance of anginal severity on maximal medication in determining the ACRE panel's ratings of appropriateness for angiography17 and in turn the importance of symptom severity in judging the need for revascularisation.

However, if the expert panel method defines indications for angiography in three clinically meaningful groups, the ratings require ongoing refinement to keep abreast of changes in evidence, technology, clinical practice, and patient expectation. Although results are based on small numbers, 11% of the inappropriate group did have triple vessel or left main stem disease and 22% of all patients undergoing angiography for inappropriate indications who had CAD were subsequently revascularised. These findings emphasise the importance of improving the rating process. It is an inherent feature of a prospective study with up to three years' follow up that some ratings do not remain current. The approach used in the ACRE study is well suited to making such refinements since it is able to quantify the frequency of specific clinical indications. For example, the trial evidence for primary angioplasty published after the expert panel met is likely to lead to more appropriate ratings for angiography in acute myocardial infarction. However, among all patients undergoing angiography this remains a much less common indication for angiography than chronic stable angina, where the evidence has changed less dramatically.

One third of the ACRE patients underwent angiography for indications rated as uncertain—that is, benefits and harms of angiography were balanced or there was insufficient evidence for the panel to judge either way. This uncertain group has been criticised by others, partly because clinical decision making is binary—a procedure is either done or not done. However, our data suggest that the judgement of “uncertain” does contribute predictive information: angiographic findings, mortality, and revascularisation rates were all intermediate between patients undergoing angiography for inappropriate and appropriate indications. The uncertain category may provide a systematic identification of areas for future research.

Appropriateness ratings may be used to determine overuse and underuse.16 Coronary angiography performed for indications rated as inappropriate may represent overuse and was found in 5% of patients. Coronary angiography carries a small but finite risk of major complication24 and is a costly procedure. Asymptomatic or atypical chest pain presentations accounted for over 70% of the inappropriate angiograms. Within these presentations, the specific indications that were rated as inappropriate showed close agreement with the latest recommendations in the American Heart Association/American College of Cardiology angiography guidelines.3 Three questions arise. Firstly, does the indication itself adequately define the patient or do new indications need to be formulated? For example, angiography may be indicated to reveal normal coronary arteries in patients frequently attending with atypical chest pain and a low pretest probability of CAD,25 26 yet the existing indication lists do not stipulate such indications. Secondly, does the rating of appropriateness reflect current evidence? Thirdly, if the indication adequately defines the patient and the rating remains evidence based, what changes in practice are required to prevent such inappropriate use in the future? It should not, however, be assumed that eradicating inappropriate use is desirable because it may be achieved at the cost of unacceptably high thresholds for investigation.

The ACRE findings do not suggest that there is significant overuse of angiography. However, a further test of their clinical validity is warranted in which the ratings are used to investigate underuse. US studies show higher proportions of underuse than of overuse.27 Future studies should use angiography appropriateness ratings in patients recruited at the time of first presentation and test the extent to which angiography was performed in patients with appropriate indications. The ACRE findings in relation to revascularisation strongly support assessing underuse with appropriateness ratings.16

There are no European guidelines on coronary angiography. The UK'sNational service framework for coronary heart disease 28 has set a standard that “people with symptoms of angina should receive appropriate investigation”. The ACRE cohort, recruited during the stenting era, is likely to be reasonably representative of UK practice because of its large geographically defined sample selected without exclusion criteria. Furthermore, expert panels show good agreement with practising physicians.29 Thus the ACRE ratings of appropriateness form a basis for articulating such government policy in order that primary and secondary care doctors, as well as cardiologists, meet the new national standards.

The unacceptably long waiting times for coronary angiography raise the question of the clinical ordering of the angiography queue in the NHS. However, simple relations between appropriateness and urgency should not be assumed.30 Appropriateness categories were related to waiting time but it was patients undergoing angiography for appropriate indications who had the longest wait (median time to angiography 91 days), compared with 58 days for inappropriate and 50 days for uncertain indications (logrank p = 0.02). Furthermore, the relation between waiting times was much stronger with a priori urgency scores than with appropriateness categories in the ACRE cohort.31

Potential limitations of the ACRE study should be considered. Firstly, like other studies of appropriateness, the ACRE study relied on clinical record data to infer why coronary angiography was performed. However, it has been shown that the written clinical records show excellent agreement with the performing physician when interviewed.32 Secondly, since the panel members rating appropriateness were recruited from the same institution as the cohort of patients, the study itself may have influenced practice. However, such a Hawthorne effect33 is unlikely because there was no relation between the proportion of appropriate patients and time since the expert panel met; over 80% of angiography procedures were performed by cardiologists who were not members of the expert panel and excluding panellists' procedures from the analysis made no difference to the results.

Patients undergoing angiography for appropriate indications had most CAD and the highest subsequent revascularisation and mortality rates, showing the prospective validity of the expert panel ratings. As the rate of angiography continues to increase, the use of such explicit criteria to improve the quality of care should be encouraged.


Contributors: Harry Hemingway is the principal investigator of the ACRE study, responsible for inception and design; he wrote the first draft of the paper and incorporated coauthors' comments. Angela Crook carried out all the statistical analyses. Adam Timmis is the co-principal investigator, and he and Rex Dawson, Gene Feder, Patrick Magee, and Alan Wood were original members of the ACRE expert panel and active members of the ACRE steering group. Shrilla Banerjee conducted the angiographic reliability study. Sue Philpott is responsible for data collection and quality control. All coauthors commented on early and late versions of the paper. The ACRE study was established with a grant from East London and the City Health Authority, and subsequently funded by a consortium of health authorities (North Essex, Barking and Havering, Redbridge and Waltham Forest), the North Thames NHS Research and Development program (RFG 258), the British Heart Foundation (PG/97216), Guidant and Boston Scientific Corporation. We gratefully acknowledge Julie Sanders and Dr Peter Mills and the participation of the patients in this research.


View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.