Send letter to journal:
Re: When Quality Hurts

Dear Editor,

The article by Bridgewater et al “Has publication of cardiac surgery outcome data been associated with changes in practice in Northwest England? An analysis of 25,730 patients undergoing CABG surgery under 30 surgeons over 8 years.” (1) has received significant international attention with headlines around the world. However before the quality junkies lapse into a swoon of self congratulation over the great leap forward that they have accomplished with scrutiny of outcomes, its time to take a reality check. Despite the fact that the Euroscore is one of the most widely-used measures of expected outcomes in Cardiothoracic modern practice, superceding the out-dated Parsonnet score (2), it is not an accurate reflection of real operative risk in the twenty-first century. Real risk is probably about half of that (3) (4) – even the original developers of the Euroscore admit that (5). The disparity between actual and predicted outcomes (by Euroscore) probably relates to the fact that surgeons now more fastidiously look for, and have a lower threshold for assignation for, all the components of the risk score. This underscores one of the main problems with such studies of quality – all independent predictors of outcomes are adjudicated and data collected and maintained by the surgeons themselves. Public scrutiny of outcomes creates such a powerful conflict of interest in data collection for the surgeons that the data is at best questionable and at worst purely a smokescreen. Similar grave concerns for the accuracy and ascertainment bias inherent in any self-reported data were raised after comparisons of outcome data after multi-vessel stenting and CABG in New York State (6). Raw data showed a survival benefit for stenting in most subgroups but after ‘adjustment’ for patient co-morbidity data routinely collected by surgeons (but not by interventionalists) the study reported the exact opposite finding. This occurred because the surgically treated patients were assigned a far greater number of co-morbid conditions than those treated percutaneously. The counter-intuitive nature of this finding (in general patients with serious co-morbidities are treated percutaneously or medically), can be well explained by the fact that in New York State there is also mandatory reporting of outcomes, and co-morbidities are required for surgical reporting but not for interventional reporting. The study investigators were sufficiently aware of the problem to report both the raw and the adjusted data, so it was quite obvious that this was statistical error due to ascertainment bias (7). However, the magnitude of the resulting bias was astounding and a warning to any other investigators that dependence on self-reported quality data portends wildly erroneous conclusions which directly contradict randomised trial data. The conclusion in the article by Bridgewater et al that mandatory reporting and public scrutiny have not resulted in risk averse behaviour but rather have improved the surgeons performance is thus simply wishful thinking not supported by data from other quality-driven states such as New York(8). A far more likely explanation for their findings is that public scrutiny increased the pressure to assign higher Euroscores, so despite the fact that they did lower risk cases (as evidenced by the lower mortality) the predicted mortality was higher. Clearly, a better marker of whether risk averse behaviour is occurring would be to measure the operative risk of the patients that the surgeons are turning down – or as a bare minimum have Euroscores objectively assessed by an independent third party. In addition, in this age of politically driven lies damned lies and statistics, it is becoming increasingly important to document the proportion of patients who have adverse outcomes because of system failures – whether that be surgery denied because of quality-induced risk averse behaviour or because of too long waiting lists with resource scarcity. These data are unfortunately not available but would be a far more sensitive and publicly meaningful measure of whether risk averse behaviour is occurring. Quite apart from the deterioration in outcomes that may be induced by risk averse behaviour, such public airing of outcome data also inevitably has a profound effect on training of new surgeons. Consultants are far less likely to allow trainees to attempt difficult procedures when adverse outcomes will be very publicly and wholly attributed to the consultant. The consequence of this is that new consultants will have to perform procedures that they have never done before or rapid deskilling of the workforce will occur. In the current hostile ‘quality’ driven environment, most new consultants would be likely to opt out of high risk cases even more, and of course never be able to teach the necessary skills to the trainees under their tutelage. With increasing pressure from our politically-driven masters to feign public accountability with outcome smokescreens, we should take care not to believe our own publicity, lest it damage the core assets of the profession – our skills and our altruism for patients rather than administrators.

REFERENCES

1. Bridgewater B, Grayson A, Brooks N, et al.
Has the publication of cardiac surgery outcome data been associated with changes in practice in Northwest England? An analysis of 25,730 patients undergoing CABG surgery under 30 surgeons over 8 years.
Heart 2007.

2. Wynne-Jones K, Jackson M, Grotte G, Bridgewater B.
Limitations of the Parsonnet score for measuring risk stratified mortality in the north west of England. The North West Regional Cardiac Surgery Audit Steering Group.
Heart 2000;84(1):71-8.

3. Yap CH, Reid C, Yii M, et al.
Validation of the EuroSCORE model in Australia.
Eur J Cardiothorac Surg 2006;29(4):441-6.

4. Bhatti F, Grayson AD, Grotte G, et al.
The logistic EuroSCORE in cardiac surgery: how well does it predict operative risk?
Heart 2006;92(12):1817-20.

5. Nashef S.
Validation of the EuroSCORE model in Australia (editorial comment).
Eur J Cardiothorac Surg 2006;29(4):446.

6. Hannan EL, Racz MJ, Walford G, et al.
Long-term outcomes of coronary-artery bypass grafting versus stent implantation.
N Engl J Med 2005;352(21):2174-83.

7. Flaherty J, Davidson C.
Coronary Artery Bypass Grafting versus stent implantation.
N Engl J Med 2005;353:735.