Objective To assess the effect of various evaluation and reporting strategies in determining outlier surgeons, defined by having worse-than-expected mortality after cardiac surgery.

Methods Our study included 33 394 isolated coronary artery bypass graft (CABG) procedures performed by 136 surgeons and 12 172 surgical aortic valve replacement (SAVR) procedures performed by 113 surgeons between 2010 and 2014. Three current methodologies based on the framework of comparing observed and expected (O/E ratio) mortality, with different distributional assumptions, were examined. We further assessed the consistency of outliers detected by these three methods and the impact of using different time windows and aggregating data of CABG and SAVR procedures.

Results The three methods were consistent and detected same outliers, with the least conservative method detecting additional outliers (outliers detected for methods 1, 2 and 3: CABG 3 (2.2%), 2 (1.5%) and 8 (5.9%); SAVR 1 (0.9%), 0 (0.0%) and 11 (9.7%)). When numbers of cases recorded were low and events were rare, the two more conservative methods were unlikely to detect outliers unless the O/E ratios were extremely high. However, these two methods were more consistent in detecting the same surgeons as outliers across different time windows for assessment. Of the surgeons who performed both CABG and SAVR, none was an outlier for both procedures when assessed separately. Aggregating data from CABG and SAVR may lead to results to be dominated by the procedure that had a higher caseload.

Conclusions The choices of outlier assessment method, time window for assessment and data aggregation have an intertwined impact on detecting outlier surgeons, often representing different value assumptions toward patient protection and provider penalty. It is desirable to use different methods as sensitivity analyses, avoid aggregating procedures and avoid rare-event endpoints if possible.