Objective To assess the effect of various evaluation and reporting strategies in determining outlier surgeons, defined by having worse-than-expected mortality after cardiac surgery.
Methods Our study included 33 394 isolated coronary artery bypass graft (CABG) procedures performed by 136 surgeons and 12 172 surgical aortic valve replacement (SAVR) procedures performed by 113 surgeons between 2010 and 2014. Three current methodologies based on the framework of comparing observed and expected (O/E ratio) mortality, with different distributional assumptions, were examined. We further assessed the consistency of outliers detected by these three methods and the impact of using different time windows and aggregating data of CABG and SAVR procedures.
Results The three methods were consistent and detected same outliers, with the least conservative method detecting additional outliers (outliers detected for methods 1, 2 and 3: CABG 3 (2.2%), 2 (1.5%) and 8 (5.9%); SAVR 1 (0.9%), 0 (0.0%) and 11 (9.7%)). When numbers of cases recorded were low and events were rare, the two more conservative methods were unlikely to detect outliers unless the O/E ratios were extremely high. However, these two methods were more consistent in detecting the same surgeons as outliers across different time windows for assessment. Of the surgeons who performed both CABG and SAVR, none was an outlier for both procedures when assessed separately. Aggregating data from CABG and SAVR may lead to results to be dominated by the procedure that had a higher caseload.
Conclusions The choices of outlier assessment method, time window for assessment and data aggregation have an intertwined impact on detecting outlier surgeons, often representing different value assumptions toward patient protection and provider penalty. It is desirable to use different methods as sensitivity analyses, avoid aggregating procedures and avoid rare-event endpoints if possible.
- quality and outcomes of care
- cardiac surgery
- health services
Statistics from Altmetric.com
Contributors AS: responsible for the overall content as a guarantor. JM, FSR and AS: conception and design of the work. JM and AS: performed the statistical analysis. All authors contributed to the analysis, interpretation of the data, drafting and critical revision of the manuscript.
Funding This study was funded in part by the US Food and Drug Administration through grant U01FD005478.
Disclaimer No funder/supporter out of the institution was involved in any of the following aspects: design and conduct of the study; collection, management, analysis and interpretation of the data; preparation, review or approval of the manuscript; and decision to submit the manuscript for publication.
Competing interests None declared.
Patient consent Not required.
Ethics approval The study was determined exempt by the Weill Cornell Medical College institutional review board (protocol no 1209013064). Patients were not involved in developing the research question, study outcome measures, study design or conduct of the study.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.