You are here

  • Home
  • Archive
  • Volume 99, Issue 19
  • Real time monitoring of risk-adjusted paediatric cardiac surgery outcomes using variable life-adjusted display: implementation in three UK centres

Article Text

Article menu

Download PDFPDF

Cardiovascular surgery
Original article
Real time monitoring of risk-adjusted paediatric cardiac surgery outcomes using variable life-adjusted display: implementation in three UK centres
  1. Christina Pagel1,
  2. Martin Utley1,
  3. Sonya Crowe1,
  4. Thomas Witter2,
  5. David Anderson2,
  6. Ray Samson3,
  7. Andrew McLean3,
  8. Victoria Banks4,
  9. Victor Tsang4,
  10. Katherine Brown4
  1. 1Clinical Operational Research Unit, Department of Mathematics, University College London, London, UK
  2. 2Department of Congenital Heart Disease, Evelina Children's Hospital, St Thomas’ Hospital, London, UK
  3. 3Department of Paediatric Cardiology, The Royal Hospital for Sick Children, Glasgow, UK
  4. 4Cardio respiratory and Intensive Care Division, Great Ormond Street Hospital for Children, London, UK
  1. Correspondence to Dr Christina Pagel, Clinical Operational Research Unit, Department of Mathematics, University College London, 4 Taviton Street, London WC1H 0BT, UK; c.pagel{at}ucl.ac.uk

Abstract

Objective To implement routine in-house monitoring of risk-adjusted 30-day mortality following paediatric cardiac surgery.

Design Collaborative monitoring software development and implementation in three specialist centres.

Patients and methods Analyses incorporated 2 years of data routinely audited by the National Institute of Cardiac Outcomes Research (NICOR). Exclusion criteria were patients over 16 or undergoing non-cardiac or only catheter procedures. We applied the partial risk adjustment in surgery (PRAiS) risk model for death within 30 days following surgery and generated variable life-adjusted display (VLAD) charts for each centre. These were shared with each clinical team and feedback was sought.

Results Participating centres were Great Ormond Street Hospital, Evelina Children's Hospital and The Royal Hospital for Sick Children in Glasgow. Data captured all procedures performed between 1 January 2010 and 31 December 2011. This incorporated 2490 30-day episodes of care, 66 of which were associated with a death within 30 days.The VLAD charts generated for each centre displayed trends in outcomes benchmarked to recent national outcomes. All centres ended the 2-year period within four deaths from what would be expected. The VLAD charts were shared in multidisciplinary meetings and clinical teams reported that they were a useful addition to existing quality assurance initiatives. Each centre is continuing to use the prototype software to monitor their in-house surgical outcomes.

Conclusions Timely and routine monitoring of risk-adjusted mortality following paediatric cardiac surgery is feasible. Close liaison with hospital data managers as well as clinicians was crucial to the success of the project.

  • CARDIAC SURGERY
  • CONGENITAL HEART DISEASE

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/

https://doi.org/10.1136/heartjnl-2013-303671

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    The recent Safe and Sustainable Paediatric Surgery Review of congenital heart services1 recommended changes to the provision of paediatric cardiac surgery services in the UK.2–4 The Review, managed by NHS Specialised Services, considered the delivery of congenital heart services by the NHS in England in response to concerns that smaller centres might not be in the best position to deliver sustainable 24-h care. In the absence of an accepted way to adjust for case mix, the review did not focus on outcomes but stressed in its recommendations the need for timely and meaningful reporting of mortality following paediatric cardiac surgery.1

    Currently, UK paediatric cardiac surgery outcomes for individual procedure types are published online.5 ,6 This represents an important contribution to benchmarking of outcomes but limitations include an interval of at least a year before data are published, the small number of patients in many procedure categories and the omission of some procedures from audit. Providing surgical centres with the means to monitor regularly their recent outcomes across an entire surgical programme can aid quality improvement.7–13 For specialties with heterogeneous case-mix, such monitoring should include risk adjustment. This is especially so for paediatric cardiac surgery where there have been recent advances in the number and types of available treatment. Several reported methods for risk-stratification or adjustment14–19 exist and Jacobs et al20 recently used the STS-EACTS score16 to compare the risk-adjusted outcomes of paediatric cardiac centres across North America. However, there have been no published reports of clinical teams using risk adjustment to monitor routinely their own programme-level outcomes in-house for the purpose of quality assurance.

    While academic research does often respond to clinical need, the rapid transfer from research to clinical practice is not common, delaying potential benefit to patients and the clinical community.7–10 ,13 In this paper, we describe the rapid implementation of a new risk model for paediatric cardiac surgery19 in three UK centres to facilitate, for the first time in this specialty, regular monitoring of outcomes using routinely collected data. The risk model development finished in October 2011 and the study reported here ran from November 2011 to March 2012.

    Methods

    Data

    Three UK paediatric cardiac centres were recruited to be part of this project via the Congenital Heart Disease Steering Committee of the Central Cardiac Audit Database. These centres were Great Ormond Street Hospital and Evelina Children's Hospital in London and The Royal Hospital for Sick Children in Glasgow. The centres volunteered to take part and cover the spectrum of unit sizes in the UK, from the larger (Great Ormond Street) to the smaller (Glasgow). Each centre provided data on all paediatric cardiac surgery procedures5 conducted in the period 1 January 2010 to 31 December 2011 inclusive. In order to facilitate the process, data were supplied in the format that centres currently use for mandatory submissions for national audit. Records for patients over 16 years old, and patients who only underwent catheter procedures or non-cardiac procedures were excluded from analysis.

    Only pseudonymised, routinely collected data was used with no patient or carer interaction, so no ethical approval was required for this study.

    Analysis

    For each centre, every record of a procedure was allocated to a 30-day episode of care. Each patient's first episode started with their first surgical procedure and the patient's vital status at 30 days was assigned as a primary outcome. Any further procedures within this 30-day episode constituted a secondary outcome. A surgical procedure more than 30 days after the first procedure constituted the beginning of a new episode.

    An expected risk of death was estimated for each episode of care using the partial risk adjustment in surgery (PRAiS) model,19 which partially adjusts for risk of death within 30 days following cardiac surgery in children under 16. The PRAiS model was built using a random 70% subset of 10 years of UK national audit data comprising 26 447 surgical episodes. It was then validated in the remaining 30% of the national dataset (10 597 episodes) which had not been used for model development. The risk factors included in the model are surgical procedure, diagnosis, age, weight and co-morbidity.

    For each centre, risk-adjusted outcomes over time were displayed using the variable life-adjusted display (VLAD) method7 that shows the difference between expected and observed mortality over time. In addition, further cardiac surgeries and interventional catheterisations within each individual 30-day episode of care were displayed on the VLAD chart.

    To facilitate sustainable routine monitoring without specialist analytical support, we developed software implemented within Microsoft Excel to automate the preparation of data routinely supplied by centres for national audit for use with the risk model, and to generate and display VLAD charts. Since one aim was to provide centres with a ‘ready to use’ Excel spreadsheet for generation of VLAD charts, the software was developed in close collaboration with staff responsible for data submissions within the clinical centres, including repeated prototyping taking due account of existing data structures.

    Feedback from clinical teams

    The VLAD charts were presented to multidisciplinary clinical teams within each centre to elicit feedback on the displays, leading to subsequent iterative refinements by the research team. The first time VLAD charts were shown at each centre, the analysts from University College London gave a brief introduction to the PRAiS risk model and VLAD chart methodology. This included highlighting known features of the risk model with respect to high risk cases and caveats around over-interpretation of VLAD charts without further investigation. The presentations of the VLAD charts on subsequent occasions were made by clinicians to their normal audience of the regular mortality and morbidity meeting, since one of our aims was to ascertain whether this could be embedded into the usual routine quality assurance processes of each unit. The charts were shown at the participating sites on an approximately monthly basis. At these meetings comments were sought and received ad hoc ‘from the floor’. These meetings are attended by a range of staff types; the number varies between 10 and 50 from centre to centre, and from month to month. The clinician responsible for presenting the VLAD chart and leading the discussion was debriefed by a member of the research team (KB, MU or CP) on at least three occasions.

    Results

    VLAD charts

    Centres provided a total of 4192 records, 1543 of which were excluded because: patients were over 16 (244 records), patients had only catheter procedures (1242 records), the patients did not have a cardiac procedure (56 records) or it was a duplicate record (1 record). The remaining 2649 records corresponded to 2490 30-day episodes, 66 of which were associated with a death within 30 days.

    VLAD charts for the three participating centres covering the 2-year period of review are shown in figures 1⇓–3. The risk model essentially benchmarks each centre's current short-term outcomes against UK outcomes achieved between 2007 and 2010 (see Crowe et al19) with partial adjustment for case mix. The plotted line goes up for each survival and down for each death. Over time, if outcomes are as expected based on the risk model, the end of the plotted line will tend to be close to zero. Given the low mortality associated with paediatric cardiac surgery (between 2007 and 2010, UK mortality was 2.9%19), the plotted line will rise much less steeply for a run of survivors than it will fall for a run of deaths. The overall numbers of deaths and survivors are shown in the top left hand side of the VLAD charts. Re-interventions are displayed in the form of coloured circles within the plots, such that their frequency can be reviewed by clinical teams alongside trends in risk-adjusted mortality. The three participating centres all ended the 2-year period within four survivors/deaths from what would be expected using the risk model but, importantly, the VLAD charts highlight different time periods for each centre that might have warranted further exploration.

    Figure 1
    Figure 1

    Variable life-adjusted display (VLAD) chart for centre A. Each dot represents a surgical episode.

    Figure 2
    Figure 2

    Variable life-adjusted display (VLAD) chart for centre B. Each dot represents a surgical episode.

    Figure 3
    Figure 3

    Variable life-adjusted display (VLAD) chart for centre C. Each dot represents a surgical episode.

    To help the reader interpret the VLAD charts, we have shown in figure 4 an enlarged section from the VLAD chart for centre B. Here the difference between lower- and higher-risk cases is evident: the plotted line for higher-risk patients who survived is steeper than for lower-risk survivors and vice versa for patients who died.

    Figure 4
    Figure 4

    A close-up of a section of the variable life-adjusted display (VLAD) plot for centre B showing some features of VLADs.

    We note that since calendar time is plotted on the horizontal axis, VLAD charts for centres with larger case volumes will have steeper slopes than VLAD charts for centres with a smaller case volume over the same period of time. Since the primary aim is for units to use this software for routine in-house monitoring and not for comparison with other units, this should not represent a barrier to use.

    Clinician feedback and improvements to software

    Multidisciplinary teams in each centre reported that the VLAD charts showed useful information and that they prompted important discussion in mortality and morbidity meetings that in some instances led to further actions within the programme. Examples of this include:

    • The VLAD chart from centre A (figure 1) displays a long run of survivors during 2010 and a subsequent run with several deaths in the summer/autumn of 2011 leading to a downward trend in the VLAD chart. In respect of this cluster of deaths, certain patterns and learning points were found above and beyond those identified through the individual reviews of cases that had taken place. This specific learning in centre A included a focused multi-disciplinary audit of one particular procedure type (arterial shunt) and review of protocols for that patient group including case selection issues, technical aspects of the surgery and consideration of alternative procedures in certain patients. We note that national audit has not prompted concerns about this patient group (or any other) at centre A.

    • The section of the VLAD chart showing a downward trend in the summer of 2011 at centre B led to discussion within the surgical team at that centre about case-selection of patients for the Norwood operation.

    These examples demonstrate that the response of the multidisciplinary team to adverse trends in the VLAD chart incorporated practitioners from more than one discipline, and led to ongoing quality improvement initiatives that extended outside the meeting where the chart was shown.

    In addition to mortality outcomes, the secondary outcome of ‘reintervention within 30 days’ is also shown in the VLAD charts by use of coloured dots. This can provide useful additional information to clinical teams reviewing these charts: for instance, although centre A has a long run of survivors in the summer of 2010 (figure 1), there were a cluster of reinterventions during that period.

    A review of the run of deaths shown by the VLAD chart from centre B (figure 2) during the first 6 months of 2011, while generating useful discussion of the relevant cases, also raised the issue of inter-centre differences in case-mix and the known performance of the risk model. Clinicians considered that it would be informative to compare an individual centre's case mix to the national average and we modified the software to provide this option. An example is provided in figure 5, showing the breakdown of case mix for centre B, alongside the comparator, which represents the national audit data from 2007–2010. This centre's case mix contains higher proportions of medium and high risk cases than the national average. The comparison is particularly relevant in this case, since the PRAiS risk model is known to underestimate risk slightly for the highest risk patients (with estimated risk above 10%),19 so that the VLAD chart in figure 2 is likely to present a slightly pessimistic picture.

    Figure 5
    Figure 5

    The Excel output showing the risk profile of centre B compared to the national average. CCAD, Central Cardic Audit Database.

    Review of individual deaths highlighted the importance of co-morbid conditions and medical factors that were not fully accounted for in the predictions of expected risk. For example, prematurity was included in the risk model within a ‘yes’ or ‘no’ co-morbidity field indicating gestational age less than 37 weeks, as this is the definition used for national audit.5 This does not capture the relative risk of operating at extremes of prematurity, for instance the case of a baby of only 25 weeks and weighing 600 g. Also, certain serious co-morbidities, for example end organ damage prior to surgery, were not adequately described by the data submitted for national audit, despite appropriate fields being available. Finally, important but highly specific cardiovascular risk factors such as the aortic dimensions in candidates for the high risk Norwood operation17 are not included in the routine dataset.

    The software package and data considerations

    Through working closely with the data managers, the software includes error checking and highlights obvious data anomalies for users. All required data processing is performed by the software to minimise the burden on centre data managers, so that only a straightforward ‘copy and paste’ from existing hospital databases into the Excel spreadsheet is required.

    The prototype Excel software was given to each centre at the end of the implementation period and is now being used independently by those multidisciplinary teams to inform local governance processes and quality improvement initiatives. For instance, VLAD charts covering the previous 12 months are shown once a month in a multidisciplinary meeting at Great Ormond Street Hospital and any notable features discussed (for instance, a cluster of deaths or reoperations).

    Discussion

    Our aim was to provide a means for units to monitor their own programme level outcomes for paediatric cardiac surgeries, benchmarking to recent national outcomes using a new model for risk adjustment and feeding these data back to the multidisciplinary team. We focused on team level outcomes to recognise the contribution made to care and decision making by all staff groups within the units.

    A new software package was developed that can be used by paediatric cardiac surgery centres to monitor their programme level, short-term outcomes with partial risk adjustment using the PRAiS risk model19 and VLAD charts.7 Presentation of VLAD charts to multidisciplinary meetings prompted useful discussion of observed trends in outcomes. All three centres are using the software regularly to monitor their recent outcomes in the context of mortality and morbidity conferences.

    This routine in-house monitoring of risk-adjusted outcomes complements the annual national monitoring using funnel plots already in place for a range of individual surgical procedures on the National Institute for Cardiovascular Outcomes Research (NICOR) congenital heart disease web portal. The VLAD charts have obvious advantages in that first, the charts incorporate a unit's entire case load, thus increasing the chance of any concerning trend being detectable, and second, the VLAD charts can be delivered directly to clinical teams for very timely review of risk-adjusted outcomes with obvious advantages for quality assurance.

    The PRAiS risk adjustment model was developed in 2010–2011 using paediatric cardiac surgery audit data held by NICOR by a multidisciplinary group incorporating analysts and clinicians. The PRAiS risk adjustment model includes risk factors that have face validity and are already routinely collected for national audit: these attributes ensured that this method of risk adjustment was easy to implement in routine practice in the participating cardiac centres. Validation of PRAiS19 showed that it performed as well as other published methods of risk adjustment15 ,16 which increased its acceptability for use in monitoring among clinicians. There is the potential for future improved accuracy of risk ascertainment with better data quality, in particular regarding co-morbidity information as discussed above.

    An ideal study exploring the introduction of a new monitoring system would involve only centres new to both the method of risk adjustment and the analytical team, to better mimic prospective national use. One of the centres (Great Ormond Street Hospital) co-developed the PRAiS risk model and has a long-standing productive collaboration with the analysts: this undoubtedly facilitated the success of the implementation in that centre. Close collaboration with data managers from the centres meant that the developed software could be responsive to centres’ needs—for instance, the incorporation of reasonably comprehensive error checking and allowing for a variety of data input formats. The input of clinicians into the development of the software and pilot runs of the VLAD charts in a morbidity and mortality forum allowed the research team to add useful features to the output. We acknowledge that no formal systematic evaluation of feedback was performed, rather that the clinicians responsible for presenting the plots at their respective centres were informally debriefed by the research team.

    VLAD charts evaluating trends in the results of adult cardiac surgery have used data pertaining to individual operators21–24 and alternatively, programme level data.7 The choice of programme based charts in this study reflects the fact that paediatric cardiac surgery demands high standards from cardiologists, nurses and intensive care specialists as well as surgeons as has been highlighted in recent service reviews.1 ,25 ,26 Statistical methods based on CUSUM analysis have been used previously within VLAD charts to incorporate a ‘signal’ of poor outcomes.27 Rocket tails have been applied to VLAD charts to indicate graduations of likelihood that differences between expected and observed outcomes are due to chance.28 Given the intended use of the paediatric cardiac surgery VLAD charts in a regular continuous programme of review, we chose not to include statistical signalling in order to limit scope for complacency. We additionally discussed the potential for positive or negative trends to arise by chance with clinical teams and emphasised that VLAD charts represent a starting point for investigation, not an end.

    In addition to the points above, this study highlighted other factors that have implications for future implementation of routine monitoring in clinical practice.

    When initiating risk-adjusted monitoring of outcomes within centres, the discussions with clinical teams around context and intended use should consider the sensitivities involved. Previous barriers to clinician participation in reviews of patient safety and quality concerns included fear of censure and reprisals, lack of trust and concern regarding confidentiality.29 Using the VLAD method for evaluation of recent outcomes means that on average, with a well calibrated risk model, a centre's outcomes will probably fluctuate around the zero line. We learned that clinicians’ first impressions of such plots can be that being close to the zero line is undesirable. This might be exacerbated by the fact that risk models currently used in adult cardiac surgery tend to overestimate risk of death so that VLAD charts of risk-adjusted outcomes in adult cardiac surgery normally show better performance than expected from the risk model.8 ,9 The VLAD charts do not evaluate the programme itself—quality of service is only one of many possible explanations for observed trends in a VLAD chart. Rather, the aim is to stimulate reflection on practice and assist a broader programme of continuous quality improvement.

    An unanswered question is whether such routine monitoring will lead to service improvement in paediatric cardiac surgery. There is evidence to support this in adult heart surgery,13 but with the management of congenital heart disease being far from a ‘one-stop-shop’, it seems likely that audit including review of short and longer-term outcomes will best serve the needs of this population.

    One final concern about the role of routine monitoring in surgical services arose during this study, which has relevance across many clinical areas. As hospital outcomes are increasingly published online and discussed in mainstream media, it is possible that hospital centres would see a programme of routine monitoring of outcomes as a risk rather than an opportunity. On the other hand, publishing clinical outcomes can increase trust among service users and drive improvement across an organisation.30 Perhaps further debate between clinicians and the public around the role of routine monitoring within service improvement is required.

    Acknowledgments

    We would like to acknowledge all the surgeons at each centre for supporting this project.

    References

    1. ↵
      NHS. Safe and Sustainable: Childrens Congenital Cardiac Services. NHS Specialist Services. 2011. http://www.specialisedservices.nhs.uk/safe_sustainable/childrens-congenital-cardiac-services (accessed 3 Jan 2013).
    2. ↵
      1. McGauran A
      . Radcliffe should stop doing children's heart surgery, review recommends. BMJ 2010;341:c5863–3.
      OpenUrlFREE Full Text
    3. ↵
      1. McGauran A
      . Fewer hospitals in England should perform children's heart surgery. BMJ 2011;342:d1129–9.
      OpenUrlFREE Full Text
    4. ↵
      1. Dyer C
      . Consultation process on closing children's cardiac surgery services at Brompton Hospital was fair, judges rule. BMJ 2012;344:e2896–6.
      OpenUrlFREE Full Text
    5. ↵
      CCAD. Central Cardic Audit Database: Paediatric Analysis Home Page. Congenital Heart Disease Website. 2011. http://www.ucl.ac.uk/nicor/ (accessed 29 Jan 2011).
    6. ↵
      1. Spiegelhalter DJ
      . Mortality and volume of cases in paediatric cardiac surgery: retrospective study based on routinely collected data. BMJ 2002;324:261–1.
      OpenUrlAbstract/FREE Full Text
    7. ↵
      1. Lovegrove J,
      2. Valencia O,
      3. Treasure T,
      4. et al
      . Monitoring the results of cardiac surgery by variable life-adjusted display. Lancet 1997;350:1128–30.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Nashef SA,
      2. Roques F,
      3. Michel P,
      4. et al
      . European System for Cardiac Operative Risk Evaluation (EuroSCORE). Eur J Cardiothorac Surg 1999;16:9–13.
      OpenUrlAbstract/FREE Full Text
    9. ↵
      1. Parsonnet V,
      2. Dean D,
      3. Bernstein AD
      . A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease. Circulation 1989;79:I3–12.
      OpenUrlPubMed
    10. ↵
      1. Sherlaw-Johnson C,
      2. Lovegrove J,
      3. Treasure T,
      4. et al
      . Likely variations in perioperative mortality associated with cardiac surgery: when does high mortality reflect bad practice? Heart 2000;84:79–82.
      OpenUrlAbstract/FREE Full Text
    11. ↵
      1. Nugent WC
      . Building and supporting sustainable improvement in cardiac surgery: the Northern New England experience. Semin Cardiothorac Vasc Anesth 2005;9:115–18.
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. Sibanda T,
      2. Sibanda N
      . The CUSUM chart method as a tool for continuous monitoring of clinical outcomes using routinely collected data. BMC Med Res Methodol 2007;7:46.
      OpenUrlCrossRefPubMed
    13. ↵
      1. Bridgewater B,
      2. Grayson AD,
      3. Brooks N,
      4. et al
      . Has the publication of cardiac surgery outcome data been associated with changes in practice in northwest England: an analysis of 25,730 patients undergoing CABG surgery under 30 surgeons over eight years. Heart 2007;93:744–8.
      OpenUrlAbstract/FREE Full Text
    14. ↵
      1. Lacour-Gayet F,
      2. Clarke D,
      3. Jacobs J,
      4. et al
      . The Aristotle score for congenital heart surgery. Semin Thorac Cardiovasc Surg 2004;7:185–91.
      OpenUrlCrossRef
    15. ↵
      1. Jenkins KJ,
      2. Gauvreau K,
      3. Newburger JW,
      4. et al
      . Consensus-based method for risk adjustment for surgery for congenital heart disease. J Thorac Cardiovasc Surg 2002;123:110–18.
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      1. O'Brien SM,
      2. Clarke DR,
      3. Jacobs JP,
      4. et al
      . An empirically based tool for analyzing mortality associated with congenital heart surgery. J Thorac Cardiovasc Surg 2009;138:1139–53.
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      1. Tweddell JS,
      2. Sleeper LA,
      3. Ohye RG,
      4. et al
      . Intermediate-term mortality and cardiac transplantation in infants with single-ventricle lesions: risk factors and their interaction with shunt type. J Thorac Cardiovasc Surg 2012;144:152–9.
      OpenUrlCrossRefPubMedWeb of Science
    18. ↵
      1. Kang N,
      2. Cole T,
      3. Tsang V,
      4. et al
      . Risk stratification in paediatric open-heart surgery. Eur J Cardiothorac Surg 2004;26:3–11.
      OpenUrlAbstract/FREE Full Text
    19. ↵
      1. Crowe S,
      2. Brown KL,
      3. Pagel C,
      4. et al
      . Development of a diagnosis- and procedure-based risk model for 30-day outcome after pediatric cardiac surgery. J Thorac Cardiovasc Surg. Published Online First: 17 July 2012. doi:10.1016/j.jtcvs.2012.06.023
    20. ↵
      1. Jacobs JP,
      2. O'Brien SM,
      3. Pasquali SK,
      4. et al
      . Variation in outcomes for risk-stratified pediatric cardiac surgical operations: an analysis of the STS Congenital Heart Surgery Database. Ann Thorac Surg 2012;94:564–71; discussion 571–572.
      OpenUrlCrossRefPubMedWeb of Science
    21. ↵
      1. Treasure T,
      2. Valencia O,
      3. Sherlaw-Johnson C,
      4. et al
      . Surgical performance measurement. Health Care Manag Sci 2002;5:243–8.
      OpenUrlCrossRefPubMed
    22. ↵
      1. Bridgewater B,
      2. Grayson AD,
      3. Jackson M,
      4. et al
      . Surgeon specific mortality in adult cardiac surgery: comparison between crude and risk stratified data. BMJ 2003;327:13–17.
      OpenUrlAbstract/FREE Full Text
    23. ↵
      1. Bridgewater B
      . Mortality data in adult cardiac surgery for named surgeons: retrospective examination of prospectively collected data on coronary artery surgery and aortic valve replacement. BMJ 2005;330:506–10.
      OpenUrlAbstract/FREE Full Text
    24. ↵
      1. Grant SW,
      2. Grayson AD,
      3. Jackson M,
      4. et al
      . Does the choice of risk-adjustment model influence the outcome of surgeon-specific mortality analysis? A retrospective analysis of 14,637 patients under 31 surgeons. Heart 2008;94:1044–9.
      OpenUrlAbstract/FREE Full Text
    25. ↵
      1. Murphy JF
      . The Bristol Royal Infirmary Inquiry 18th July 2001. Ir Med J 2001;94:228.
      OpenUrlPubMed
    26. ↵
      1. Moore A,
      2. Agnew T,
      3. Cole A
      . Bristol Royal Infirmary—the aftermath: six pages of analysis. Health Serv J 2001;111:11–17.
      OpenUrlPubMed
    27. ↵
      1. Sherlaw-Johnson C
      . A method for detecting runs of good and bad clinical outcomes on Variable Life-Adjusted Display (VLAD) charts. Health Care Manag Sci 2005;8:61–5.
      OpenUrlCrossRefPubMed
    28. ↵
      1. Sherlaw-Johnson C,
      2. Morton A,
      3. Robinson MB,
      4. et al
      . Real-time monitoring of coronary care mortality: a comparison and combination of two monitoring tools. Int J Cardiol 2005;100:301–7.
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      NPSA. Patient safety resources. http://www.nrls.npsa.nhs.uk/resources/?EntryId45=59883 (accessed 2 Sep 2012).
    30. ↵
      1. Abbasi K
      . Improvement in practice: Beth Israel Deaconess case study. 2010. http://www.health.org.uk/public/cms/75/76/4167/2686/BID_update_FINAL.pdf?realName=MeIeJQ.pdf (accessed 29 Oct 2012).

    Footnotes

    • Contributors CP, MU, SC, VT and KB had the original idea for the project. CP, SC analysed the data and developed the Excel software. CP, MU, SC and KB discussed the output with each unit. CP, MU, SC, TW, VB, VT and KB all contributed to the design of the Excel software and the format of the output. TW, RS and VB provided the data and feedback on the Excel software. DA, AM and VT represent the surgeons at each unit who allowed the data be used and championed the project in each unit. CP wrote the first draft of the paper and all authors read and commented on the manuscript. CP is guarantor for the paper.

    • Funding This study was funded by the National Institute of Cardiovascular Outcomes Research at UCL. The Health Foundation funded the contribution of CP, MU and SC in preparing the manuscript (grant number 6940). The funders played no role in the study design, analysis and interpretation of the data nor in the preparation of the article.

    • Competing interests The UCL Clinical Operational Research Unit received funding from the National Institute of Cardiovascular Outcomes Research to carry out the prototype software development and implementation and from the Health Foundation to disseminate the work (including manuscript preparation); CP, MU, SC may receive small amounts of income in future relating to the roll-out of an updated version of the developed VLAD software; TW, AM and KB serve alongside colleagues from various UK cardiac centres on the Congenital Heart Diseases Steering Committee at NICOR.

    • Provenance and peer review Not commissioned; externally peer reviewed.