Strengths and limitations of RWE studies

RWE studies include a wide range of research methodologies and data sources, although they can be broadly categorised into non-interventional studies, patient registries, claims database studies, patient surveys and electronic health record studies (table 1).2 5

They can also be categorised into prospective studies, which generally require primary data collection, or retrospective studies, which use secondary data to look back in time (ie, data initially collected for other purposes; figure 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Illustration to highlight the design and analysis time frame (relative to the study start or index date) of different types of real-world evidence studies. Arrows depict prospective or retrospective studies of various durations.

Non-interventional studies

Non-interventional studies are defined as those where treatment is prescribed in accordance with the terms of the marketing authorisation, and where the assignment of the patient to a particular therapy is entirely at the physician’s discretion, rather than decided in advance by the study protocol.22 23 As such, treatment selection and management is separate from the decision to include the patient in the study, with no diagnostic or monitoring procedures performed above those deemed necessary by the attending physician for the conduct of the study.22 23 In some instances, prospective, multicentre, non-interventional studies conducted in routine clinical practice may be referred to as pragmatic clinical trials.24 25

With the exception of registries (detailed in the next section), non-interventional studies generally fall into three categories: cohort, cross-sectional and case–control studies.11

Patient registries

Patient registries are non-interventional, observational cohort studies.2 They are typically prospective studies that involve standardised, ongoing data collection in a real-world setting to fulfil a specific predefined purpose, where management of treatment and care is determined by the patient and caregiver rather than the registry protocol.10 Ideally, the collection of registry data is uniform for all patients, including the type, method and frequency of collection. Information is then collated in a central registry database for analysis.

Advantages of registries over RCTs include the capacity to enrol a much larger and more diverse patient population with the potential for a longer follow-up period. This provides data that are more reflective of a real-world population and enables the study of longer term outcomes, including the identification of more infrequent safety outcomes.2 Registry studies also involve few or no required visits, evaluations or procedures at specialist centres because the data are collected by the attending physician as part of daily practice. Not only does this make registries potentially less expensive than RCTs, but it also means that treatment patterns reflect the daily clinical decision-making that is most relevant to healthcare providers and payers, and can help identify the most cost-effective treatment approaches.2 In addition, registries are sometimes linked with other databases, which can enable the assessment of separate outcomes such as healthcare utilisation and mortality.10

These factors make registries particularly useful for assessing (1) the natural history of a disease; (2) real-world safety and effectiveness; (3) prognosis and quality of life; (4) quality of care; and (5) cost-effectiveness of treatment strategies.10

As with all observational studies, a disadvantage inherent to patient registries is that the lack of randomisation means there is no guarantee that the patient groups are similar (because of factors such as dose adjustments), which can reduce the internal validity of the findings. Importantly, retrospective registries are also subject to survivor bias because they exclude deaths prior to the registry start date, and time from diagnosis to enrolment is longer for the patients included compared with prospective studies, meaning these registries will not encompass high-risk periods after disease onset.2 Because the data in registries are collected from a wide variety of different centres, sometimes across different countries, there can also be considerable variation in the auditing and control measures employed and the quality of the data collected.2 In addition, variations between countries and centres regarding the requirement and processes for obtaining informed patient consent can also lead to authorisation or selection bias.37 38 Furthermore, there are limits to the amount of data that can be obtained in a routine clinical practice visit, and the visits may not be scheduled at regularly timed intervals, hindering comparisons between patient groups. Despite this, the advancement of modern technologies such as mobile phone apps and cloud computing may make it far easier to collect data in a more uniform fashion and in real time, thus enabling more rigorous monitoring of data collection.2

Data on NOAC use in patients with AF or VTE are provided by several large registries, including the regional Dresden NOAC Registry, the national ORBIT AF I/II (Outcomes Registry for Better Informed Treatment of Atrial Fibrillation) registries and the multinational GARFIELD-AF (Global Anticoagulant Registry in the FIELD-Atrial Fibrillation), GARFIELD-VTE (Global Anticoagulant Registry in the FIELD-Venous Thromboembolism) and GLORIA-AF (Global Registry on Long-Term Oral Antithrombotic Treatment in Patients with Atrial Fibrillation) registries (table 2).

Results from these registries have provided insights into the management of NOAC treatment in clinical practice, including switching from warfarin to NOACs, the identification of high-risk patient populations and the high levels of inappropriate NOAC dosing that occur in clinical practice.39–42 However, reported limitations to the interpretation of these findings include missing or incomplete data, selection bias and residual/unmeasured bias due to a lack of randomisation.39–43

Long-term clinical outcomes in the majority of the above registries are currently being acquired because the studies are still ongoing, but initial results support the use of NOACs in patients with AF and patients treated for VTE. For example, in SWIVTER (SWIss Venous ThromboEmbolism Registry), a study of patients treated for VTE, the risk of recurrent VTE was similar in the rivaroxaban treatment arm compared with the conventional anticoagulation arm (1.2% vs 2.1%, P=0.29) for the propensity score-adjusted population; the risk of major bleeding was also similar (0.5% vs 0.5%, respectively, P=1.00).44 Patients with AF receiving rivaroxaban for stroke prevention in the Dresden NOAC Registry were also analysed (n=1204). The combined endpoint including stroke, transient ischaemic attack and systemic embolism occurred at a rate of 2.03 per 100 patient-years in the intention-to-treat analysis (95% CI 1.5 to 2.7). Major bleeding in the on-treatment group occurred at a rate of 3.0 per 100 patient-years. Event outcomes in the on-treatment group were higher in patients receiving rivaroxaban 15 mg once daily compared with rivaroxaban 20 mg once daily (stroke/transient ischaemic attack/systemic embolism 2.7 vs 1.25 per 100 patient-years and major bleeding 4.5 vs 2.4 per 100 patient-years, respectively).45 Similar results were observed in analyses performed in patients with AF receiving apixaban and dabigatran for stroke prevention in the Dresden NOAC Registry.46 47

Administrative and healthcare claims database studies

These studies involve retrospective (or sometimes real-time) analysis of data from administrative and healthcare claims databases containing treatment information and clinical information, such as diagnosis codes and hospital admissions/discharge dates.2 48 As such, these databases are particularly suited to longitudinal and cross-sectional analyses of healthcare utilisation and costs at the patient, group or population level,2 although insights can also be gained into associations between interventions and outcomes using the (although sometimes limited) clinical information.

The key advantages of administrative and healthcare claims database studies are that they can be performed relatively quickly and inexpensively compared with RCTs, involve a very large established patient cohort and can have a long follow-up period. This enables the identification of rare events, the determination of longer term outcomes and insight into the economic impact of interventions.2 In some instances, database information can also be linked with clinical data, such as patient-reported outcomes, laboratory assessments, medical records and physician surveys. This practice is most common in countries such as Denmark and Sweden, where every member of the populace has a unique personal identification number, enabling lifelong follow-up and linkage between databases at the individual level. The information contained within these Scandinavian administrative databases has been used to assess the comparative effectiveness and safety of the NOACs for stroke prevention in patients with AF; regression or propensity-based adjustment was used to try to reduce bias from confounding variables inherent in a non-interventional observational analysis.49–51 Similar approaches have been used in numerous studies using data from healthcare claims databases, particularly to assess the risk of bleeding events in NOAC-treated patients and adherence/persistence to treatment (table 3).52–55

Administrative and claims databases, however, lack key information about the choice of therapy. This choice can be influenced by clinical risk factors and physician and patient preferences which are not recorded in such databases. In addition, databases lack clarity in terms of insights into dosing. All these factors make inferences from administrative and claims databases, particularly direct drug comparisons, unreliable in relation to the clinical effectiveness of specific treatments. Data quality is also one of the major disadvantages of these database studies. Clinical data may not only be limited or missing (eg, health outcomes, health status, symptoms and patient characteristics, such as creatinine clearance levels, international normalised ratio values or time in therapeutic range),2 52 55 56 but the accuracy of reporting is variable (both between different countries and between centres within the same country) and coding errors are common.49 52 55 With respect to NOAC studies, there are many different codes and categories for bleeding events, further increasing the chances of coding errors (eg, gastrointestinal bleeding is often listed as ‘bleeding of unknown source’), and diagnosis codes (eg, International Classification of Diseases, Ninth or Tenth Revision) may not always differentiate between indications.52 Therefore, caution should always be exercised when interpreting treatment effectiveness and safety based on claims database analyses.

Even if the integrity of the data can be confirmed, there are additional issues inherent to claims database analyses, including limited validation, lack of a population denominator and lack of distinction between costs and charges2; follow-up duration may be limited by patients switching between insurance plans. However, selection bias is perhaps the most common and challenging methodological issue affecting both administrative and claims database analyses because treatment selection, clinical outcomes and economic outcomes could be influenced by factors that are not recorded in the database (eg, baseline health status, symptomatology and comorbidities).2 49 54 55 57

Electronic health record studies

Electronic health record studies are retrospective (or sometimes real-time), observational analyses of data sourced directly from medical records or charts.2 58 They are typically used to assess clinical treatments, procedures and outcomes.

Like claims database studies, electronic health record studies can be performed relatively quickly and inexpensively compared with RCTs, can involve a relatively large patient cohort and can have a longer follow-up. The study of rare conditions or those with a long latency between exposure and disease is also possible,58 as is the study of the real-world use of specific techniques, treatments and procedures.2 Although health record studies are usually limited to a small number of study centres, central databases such as the UK Clinical Practice Research Datalink are becoming more widely available with the proliferation of mobile technologies.2 59 These databases enable longitudinal, patient-level data collection from multiple sources and may contribute to more consistent recording and coding of information.

Although real-time clinical treatment and outcomes data are captured in electronic health record studies, there are similar issues to those of claims database studies regarding data integrity. Findings are heavily reliant on the accuracy of recorded information or the recall of individuals. Therefore, there is the potential for recording/coding and interpretive errors or bias.2 58 60 Transforming the data for research purposes requires sophisticated statistical tools, and making robust inferences on clinical effectiveness and safety based on chart review data remains a challenge.2

Nevertheless, recent electronic health record studies of NOACs have supported the findings of the pivotal RCTs and provided insights into the safety of NOAC use and treatment persistence.60–63

Patient surveys

Patient and health surveys are designed to collect health status, well-being, healthcare resource utilisation, treatment patterns, education and treatment preference information from patients, healthcare providers or the general population.2 32 They can be conducted in a variety of ways, but typically involve either online, interview or paper-based questionnaires.32 64–66 Patient surveys usually include a rigorous methodology for the collection of data and can provide unique information on the generalisability of treatments, their impacts at a patient level and adherence in the real world.2 With regard to NOACs, patient surveys have proved particularly useful in identifying factors that influence patient–physician decision-making, the need for improved patient education and adherence (including physician adherence to appropriate monitoring), and both patient and physician treatment preferences.32 64–66

Patient surveys, however, lack relevant data on specific treatments or products.2 Therefore, they cannot be used to evaluate the effectiveness or safety of treatments, other than by inference from patient or physician perceptions, and are susceptible to various forms of bias. These include recall bias, selection bias and subjectivity bias (eg, positive response bias, where a patient may respond in a way they believe to be appropriate rather than reporting their actual behaviour).2 32 Although efforts can be made to minimise these issues, such as validation of the questionnaire prior to the survey, question/response randomisation or validation of responses within the questionnaire itself,32 64 these biases should always be considered when interpreting the results of any survey.

Study overview and comparison with RCTs

The completed RWE studies listed in this review show consistent results in terms of effectiveness and safety with other studies investigating NOACs, including the findings from phase III RCTs (table 4). However, it is important to note that no comparisons between RWE studies and RCTs or other types of studies can be made.26 27

Practical and procedural considerations

Overall, there are several key aspects to consider when evaluating and comparing data from RWE studies. Perhaps most important is whether the study design is appropriate for the study outcomes. As detailed in previous sections, different study designs lend themselves to different outcomes: RCTs are the gold standard for the evaluation of efficacy and for comparing treatments; claims database studies may be the most appropriate for healthcare utilisation; cross-sectional studies can be used to assess prevalence; case–control and electronic health record studies are useful for rare conditions; and patient registries can investigate the natural history of a condition. Real-world clinical safety and effectiveness can be assessed using various RWE study types (and there is an argument that these should be obtained to provide the most balanced overview for an intervention), but it appears that prospective, non-interventional studies and patient registries provide the most robust data on these outcomes.

It is also essential that the study population reflects the target population. For example, the interpretation of health claims database findings should consider the specificity of diagnosis codes and the breadth of insurance coverage.

It is vital to consider the inherent sources of bias in the study designs when drawing the appropriate conclusions from RWE studies and comparing the findings with previous studies. These are far more numerous and varied than those of RCTs, from the positive response bias in patient surveys to the selection bias and potential for data-mining in administrative and claims database studies. There are sophisticated statistical approaches that can adjust for bias, including logistic regression adjustment, inverse probability weighting, propensity (or other) matching, instrumental variable methods and panel data models.67 However, despite these approaches, RWE study data still do not meet the reliability and accuracy afforded by the methodological rigour of RCTs and do not provide a reliable way of comparing treatment strategies.2

Lastly, one of the most common issues with RWE studies is the quality and consistency of data collection. When comparing data between studies, the type of data (eg, prospective, retrospective, adjudicated, non-interventional, database, survey), outcome definitions, choice of database codes, auditing and control measures, and methodologies to adjust for missing data, under-reporting, coding errors and others should be aligned to enable robust comparisons. Overall, inferences drawn from observational and retrospective RWE studies can be made, but this should be done with caution and interpretations need to take account of the design, robustness and quality assurance in each study.