Selective reporting in trials of high risk cardiovascular devices: cross sectional comparison between premarket approval summaries and published reports

Principal findings

About half (51%) of clinical studies of novel high risk cardiovascular devices remain unpublished over two years after FDA approval. Even when these studies are published, the results can differ substantially from FDA summaries in their presentation of primary endpoints. This finding remained true even when we restricted our analysis to pivotal studies. Reassuringly, a comparison of clinical design features including randomization, blinding, and number of centers as reported in summaries and publications showed that they were nearly identical. A quarter of trials, however, had discrepancies between summaries and publications in the numbers of enrolled patients, suggesting that the composition of the patient population had changed substantially and introducing a possible source of bias in reporting. This is further supported by the finding that sex and age demographics also differed between summary and publication in over 10% of the studies.

Clinicians might not be aware of the FDA device summaries and so might not critically examine these data. Thus for many high risk devices, clinical trial evidence might never be made readily available to the medical community or might be made available only after a long delay. While nearly all studies with disclosed funding were funded by industry, lengthy delays of publication have also been noted in NHLBI funded cardiovascular trials and in other NIH supported trials.5 9 10 11

Summaries and published studies differed in how their primary endpoints were presented: some endpoints labeled as primary in summaries were labeled as secondary in published studies and nearly 10% were entirely missing. These alterations mean that primary endpoints were rebranded to modify the emphasis of their findings in the literature. For instance, the summary for the “Xience V Rapid Exchange Everolimus-Eluting Stent System” (Abbott Vascular, IL, US) designates “co-primary endpoints” of “in-segment late loss at 240 days and target vessel failure at 270 days.”12 In the corresponding publication, however, the primary endpoint is “in-segment late loss at 240 days,” while the “secondary end point was ischemia-driven target vessel failure at 270 days.”13 In this case, the primary endpoint in the publication showed superiority of the tested device, whereas the rebranded secondary endpoint showed only non-inferiority. Others have reported similar findings.14 In a Cochrane review, 4-50% of randomized controlled trials exhibited discrepancies between primary outcomes registered and those published.15

Comparisons of results for primary endpoints between summaries and published studies showed that less than half of such results were identical, and 11% were substantially different. In addition, while many primary endpoints (31%) in the summaries were measured against objective performance criteria, far fewer endpoints (12%) were held to these criteria in the published literature. The significantly decreased number of objective performance criteria in the corresponding publications was the primary reason why devices seemed more favorable in the summaries than in the literature. Nonetheless, in several of these cases, the disappearance of the objective performance criteria in the literature was associated with failure of the primary endpoint result to meet target criteria in the summary. For instance, two of the three endpoints for the “7F Freezor Cardiac Cryoablation Catheter” (Medtronic Cryocath, QC, Canada) failed to meet the prespecified objective performance criteria.16 The publication for the same clinical trial, however, includes no mention of any objective performance criteria for any of its endpoints but simply reports a procedural success rate of 83% and major complication rate of 4.2%, and concludes that catheter cryoablation is a safe and clinically effective alternative to the standard of care.17

In rare cases, the endpoint reporting in the summary seemed more favorable than in the publication. For example, the summary for the “Talent Thoracic Stent Graft System” (Medtronic Vascular, MN, US) states a primary safety endpoint of all cause mortality at 12 months, to be compared with both an objective performance criterion of <29.8% and a retrospective control group. The summary reported 12 month all cause mortality of 16.1%, which both meets the objective performance criteria and is significantly lower than the retrospective control group rate of 20.6%.7 In contrast, the publication fails to mention any objective performance criteria, but reports a non-significant P value of 0.29 for the comparison with the retrospective control.8

Study limitations

A potential limitation of this study is that there is no systematic listing in the summaries of the trial principal investigators and clinical trial registries. In addition, we performed the Medline search on 15 January 2013, for publications up to 1 January 2013, and delays in indexing publications on Medline can exceed two weeks. To ensure that we did not inadvertently overlook a publication, we directly contacted the research divisions of device manufacturers and asked about any publications not found through our search algorithm. All of the manufacturers who responded confirmed that there was no publication for the respective study of interest. Another limitation is that our study focuses on selective outcome reporting and reporting of selected analyses such as follow-up intervals, and individual outcomes of composite endpoints were not explored.

Comparison with other studies

This study is the first to document selective reporting of device trials at both the study level and the outcome level, but it underscores a broader problem in the way clinical trial evidence is presented. Similar discrepancies have been identified for drug trials by comparing trial data reported in FDA New Drug Applications with the data in corresponding publications. A comparison of 128 drug trials as reported in FDA documents compared with the corresponding journal publications found differences in primary outcomes reported, types of data, results, P values, confidence intervals, and adverse event tables.5 Several types of selective outcome reporting for trials of off-label use of gabapentin have also been described, including discrepancies in numbers of patients randomized, definitions of primary outcomes, significance of reported outcomes, additions and deletions of outcomes, and differences in analyses.18 19 These discrepancies in the reporting led to more favorable efficacy data being presented in publications. These and our findings point to the importance of mandatory registration on a public clinical trials platform. Clinicaltrials.gov is an important step in this direction, but recent data show that published trials often have discrepant findings between clinicaltrials.gov and publications.20

Conclusions and policy implications

Our findings have potential international implications. In the European Union, independent organizations called notified bodies authorize device marketing. The evidence reviewed by notified bodies is not mandated to be publicly available, which makes it challenging to directly compare published data with data reviewed by notified bodies.21 As devices generally receive CE mark before FDA approval, however, it is less likely that clinical studies have been published in the medical literature at the time of CE mark.22

To protect the integrity of clinical trial research and, ultimately, safety of patients, the data from FDA reviews of devices should be readily accessible through peer reviewed publications. Although systematic reviews increasingly serve as a basis for evidence based clinical practice guidelines, documents from regulatory agencies are seldom included. Systematic reviewers might be unaware of the data that can be obtained from regulatory agencies.23 24 Thus, clinical practice guidelines might not be based on complete and accurate information about drugs or devices. Recent efforts, such as the intent of the European Medicines Agency (EMA) to make all clinical trial data for drug applications publicly available, would help enormously to fill these gaps by increasing the transparency and availability of important clinical trial data.25 26 In response to pushback, including from 33 drug companies or pharmaceutical industry associations, however, the EMA has imposed severe restrictions on the use of these data along with confidentiality agreements while making the data available only to registered users in a “view on screen only” mode.27 A recent study shows that FDA documents are more challenging to navigate than EMA reports, and the accessibility and user friendliness of FDA reports needs to be considerably improved.28 As clinicians can use devices immediately after FDA approval, it is in the public interest that all of the data be available to clinicians at that time.