Article Text

Original research
Novel bleeding prediction model in atrial fibrillation patients on new oral anticoagulants
  1. Ofra Barnett-Griness1,
  2. Nili Stein1,
  3. Antonio Kotler2,
  4. Walid Saliba1,3,
  5. Naomi Gronich1,3
  1. 1 Community Medicine and Epidemiology, Lady Davis Carmel Medical Center, Haifa, Israel
  2. 2 Hematology, Lady Davis Carmel Medical Center, Haifa, Israel
  3. 3 The Ruth and Bruce Rappaport Faculty of Medicine, Technion Israel Institute of Technology, Haifa, Israel
  1. Correspondence to Dr Naomi Gronich, Community Medicine and Epidemiology, Lady Davis Carmel Medical Center, Haifa, Israel; gronichn{at}


Objective Clinical models such as the HAS-BLED (standing for Hypertension, Abnormal liver/renal function, Stroke history, Bleeding history or predisposition, Labile INR, Elderly, Drug/alcohol usage) were developed to predict risk of major bleeding on vitamin K antagonists/antiplatelet therapy. We aimed to develop a model that will improve the ability to predict major bleeding events in patients with non-valvular atrial fibrillation (AF) treated with new oral anticoagulants (NOACs).

Methods Clalit Health Services is the largest of four integrated healthcare organisations in Israel, which insures 4.7 million patients (53% of the population). We identified in Clalit Health Services all patients with AF, new users of an NOAC (2013–2017), and followed them until first occurrence of a major bleeding event, death, switch to another oral anticoagulant, 30 days after discontinuation of NOAC or end of follow-up (31 December 2019). Importance of the candidate model variables was estimated by inclusion frequencies across forward selection algorithm applied to 50 bootstrap samples. Then, backward selection algorithm using the modified Bayesian Information Criterion for competing risks was applied to select predictors for the final model.

Results 47 623 patients with AF prescribed NOAC were studied. 28 055 patients with AF, initiators of apixaban (mean age 78.7, SD 9.0), were included in the first phase and had 662 major bleeding events. Nine variables were selected for inclusion in a final points-based risk-scoring system: male sex, anaemia, thrombocytopaenia (<99×103/µL), concurrent antiplatelet therapy, hypertension, prior major bleeding, risk factors for a fall, low cholesterol level and low estimated glomerular filtration rate, with apparent area-under-curve (AUC) of 0.6546. Applicability of the model was then shown for 14 118 and 5450 patients with AF, initiators of dabigatran and rivaroxaban, where the score achieved c indices of 0.62 and 0.61, respectively.

Conclusions We present a novel and simple risk score for prediction of major bleeding in patients with non-valvular AF treated with NOACs. Validation in additional cohorts is warranted.

  • atrial fibrillation

Data availability statement

Data are available upon reasonable request.

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


The factor Xa inhibitors rivaroxaban and apixaban, and the thrombin inhibitor dabigatran etexilate, named new oral anticoagulants (NOACs) largely replaced warfarin as anticoagulants for the prevention of thromboembolic complications in non-valvular atrial fibrillation (AF).1 A better safety profile was demonstrated with NOACs compared with vitamin K antagonists.2–4 Yet, 3%–6% of NOAC users experience major bleeding, including gastrointestinal (GI), intracranial and bleeding in other locations.2–5 Major bleeding events in anticoagulated patients are associated with mortality, morbidity and prolonged hospitalisations. Moreover, fear of haemorrhage causes physicians to avoid anticoagulants in patients with AF who are likely to benefit from them.6

A few clinical models,7–9 such as the HAS-BLED7 (Hypertension, Abnormal liver/renal function, Stroke history, Bleeding history or predisposition, Labile INR, Elderly, Drug/alcohol usage) and ATRIA (AnTicoagulation and Risk Factors In Atrial Fibrillation9), were developed to predict risk of major bleeding during vitamin K antagonists or antiplatelet therapy, with a modest predictive ability.1 10–13 Clinical models for prediction of major bleeding that have been developed distinctively for NOAC patients are lacking.

We assumed that differences in pharmacokinetic characteristics between vitamin K antagonists and the NOACs, and probable differences in patients’ characteristics that are treated with NOACs and had not been recommended an anticoagulant on days when only vitamin K antagonists were available might bring up new prediction variables and improve prediction. A biomarker-based ABC (age, biomarkers, clinical history)-bleeding score had been developed from clinical trials data of NOACs/warfarin.14 This score incorporated growth differentiation factor-15 and high-sensitivity cardiac troponin T, not widely available in the clinical setting.1

We developed major haemorrhage risk-scoring tool for patients with AF using demographic and clinical data of a real-life cohort of apixaban initiators. We, then, applied the model to two separate cohorts of dabigatran etexilate and rivaroxaban initiators.


Source of data

This study is based on data from the computerised database of Clalit Health Services, the largest of four integrated healthcare organisations in Israel, which insures 4.7 million patients (53% of the population). All members of the different organisations have a similar health insurance plan and similar access to health services, including low medication copayment. The electronic medical records of Clalit include data from multiple sources: records of primary care physicians, community specialty clinics, hospitalisations, laboratories and pharmacies. Diagnoses are captured in the registry by diagnosis-specific algorithms, employing International Classification of Diseases Ninth revision (ICD-9) code reading, text reading, laboratory test results and disease-specific drug usage. Reliability of electronic medical records and ICD-9 intracerebral haemorrhage coding in Clalit database has proven to be high.15 High-quality, population-based studies have been conducted based on Clalit database.16 17

Selection of study population

The first phase consisted of all Clalit adult members aged ≥40 years, with preexisting diagnoses of non-valvular AF that filled for the first time apixaban prescription between 1 November 2013 and 31 December 2017, regardless of whether they had been previously treated with other anticoagulants. We excluded 941 patients for the following reasons: patients without available serum creatinine blood test in the year prior to study entry; patients without recorded body weight in the 2 years prior to study entry and patients with estimated glomerular filtration rate <15 mL/min.

Exposure data

Use of NOACs was determined based on Clalit pharmacy records. Detailed prescription information enabled us to evaluate dosage, number of treatment days supplied and dates of prescriptions filled during the study period.

Outcome and follow-up

Outcomes were hospital discharge diagnoses representing major bleeding events (online supplemental table S1). Inpatient ICD-9 codes for major bleeding were found previously reliable.15 18 Patients were followed from the date of first filled NOAC prescription until first occurrence of any of the following: study outcome date (major bleeding), death date, switch date to another oral anticoagulant, 30 days after the discontinuation of the NOAC (defined as no evidence of the NOAC filled prescription within 30 days after the end date according to NOAC treatment days supplied in the last filled prescription) or the end of follow-up at 31 December 2019.

Supplemental material

Candidate variables

We ascertained clinical and laboratory characteristics previously associated with anticoagulation bleeding, gastrointestinal bleeding, intracranial bleeding and other major bleeding. All demographic, anthropometric, health-related habits and clinical measures were ascertained at cohort entry date and included: age, sex, body mass index, ever smoking, hypertension, diabetes mellitus, congestive heart failure, chronic obstructive pulmonary disease, active cancer (new diagnosis of cancer up to 1 year before cohort entry or anticancer drug treatment up to 6 months before cohort entry), prior major bleeding, gastrointestinal diseases (including peptic ulcer, ulcerative colitis, Crohn’s disease, diverticulosis, prior duodenal/gastric/bowel surgery), known risk factors for falls (including dementia, Parkinson’s disease, gait problem, prior fall, hip fracture), serum creatinine level, liver disease (cirrhosis, abnormal liver function tests: serum alanine aminotransferase/aspartate aminotransferase/alkaline phosphatase >3 × upper limit of normal or total bilirubin >2.24 mg/dL), serum thrombocytes count, total cholesterol and low-density lipoprotein cholesterol (LDL-C) levels: the latest measurement up to 3 years before cohort entry, concurrent prescription of an antithrombotic drug (aspirin, clopidogrel, prasugrel, ticagrelor), prescription of a drug known to inhibit P-gp/CYP3A4 (thus might increase NOAC exposure) (strong: dronedarone, clarithromycin, ketoconazole, itraconazole, indinavir, nefazodone, nelfinavir, ritonavir, saquinavir, idelalisib, ribociclib, telithromycin; moderate: amiodarone, diltiazem, erythromycin, fluconazole, verapamil, voriconazole) and prescription of a proton pump inhibitor dispensed up to 4 months before cohort entry.

Statistical methods

The population

The amount of missing data was negligible in our cohort (for any variable from the full list of candidate variables in the development cohort <0.7%; 99.2% (n=27 829) of the patients had data on all candidate variables), thus no imputation was performed.

Prediction of a major bleeding event

Time-to-event analysis methodology in the presence of competing risk was employed for deriving prediction model for a bleeding event while accounting for death before any bleeding event as a competing risk. Specifically, we used the cumulative incidence function (CIF) to describe the incidence of occurrence of an event while taking competing risks into account and the Fine-Gray subdistribution hazard model19 (CIF regression model).20 21 For descriptive purpose, the crude effect of each of the potential risk variables on major bleeding CIF was estimated using the subdistribution hazard ratio (sHR) and 95% CI.

A two-stage bootstrap procedure has been employed.22 In the first step, the variable importance of the candidate variables was estimated (figure 1). In the second step, backward selection algorithm employing the modified Bayesian Information Criterion for competing risks (BICcr)23 was used in order to select predictors for the final model, using the candidates list from step 1. The ‘crrstep’ package was used to perform this step.

Figure 1

Estimated variable importance. Variable importance of the candidate variables was determined by the frequency each variable was included across results of forward selection algorithm applied to 50 bootstrap samples (identical in size and drawn with replacement). The decreasing frequency plot was inspected for points of sharp decrease, thus creating an initial list of promising predictors. Only variables appearing in at least 50% of the forward algorithm selected multivariable models were used to create this list. We used the step 1 screening stage also to select the most suitable form for handling a continuous variable. The figure shows sharp decrease in importance after the first five predictors and a second decrease after 10 predictors, corresponding to the variables selected at least 75% and at least 50% of the bootstrapped samples, respectively. BMI, body mass index; COPD, chronic obstructive pulmonary disease; eGFR, estimated glomerular filtration rate; LDL, low-density lipoprotein.

The final list of variables identified by the selection algorithm above was entered into a subdistribution hazard regression model and the estimated parameters coefficients were used to construct the risk score. The robustness of estimates was established by referencing to the mean regression parameters achieved from fitting the model to 500 bootstrapped samples.

Model performance assessment

Model discrimination was assessed by three methods: First, the study population was stratified into quartiles, based on the model’s linear predictor (xbeta), and CIF plot was estimated and plotted by risk score quartile. Second, a concordance estimator, which is the adaptation of Harrell’s C-index to the competing risks setting,21 was obtained. Third, we plotted the receiver operating characteristic (ROC) at 1 year and estimated the AUC-ROC.21 The discrimination performance of the derived model was compared with that of five other models: a model containing all candidate variables (no selection at all) and four established clinical scores: HAS-BLED, ATRIA, ORBIT (standing for Older age, Reduced haemoglobin/haematocrit/anaemia, Bleeding history, Insufficient kidney function and Treatment with antiplatelets) and CHA2DS2-VASc (Congestive heart failure, Hypertension, Age ≥75 years, Diabetes mellitus, prior Stroke/TIA/Thromboembolism, Vascular disease, Age 65–74 years, Sex category).

Internal validation was accomplished using bootstrap cross-validation with 500 bootstrap samples, aiming to estimate the optimism (overfitting) of the apparent AUC-ROC measure, in predicting major bleeding by 1 year.

We graphically assessed internal calibration of the model by comparing and plotting the average 1-year prediction probability and the observed probability across the 10 deciles of predicted risk.20

Major bleeding risk score tool

Based on the nine-variables prediction model, we developed a points-based risk-scoring tool to estimate the 1-year major bleeding risk for each individual. The tool was developed following the methods of Austin et al,24 using the baseline CIF and the regression coefficients of the subdistribution hazards model. More specifically, a framework allowing for the use of both continuous and categorical variables in the regression model was applied. In this framework (based on that used for developing the Framingham Risk scores), as depicted in steps 1 to 7 in their section 2.2.1, the points associated with levels of each risk factor are defined relative to the points associated with an increase in a specified continuous variable. The apparent AUC was calculated for the points-based risk score tool to confirm no loss of performance due to simplification.

Generalisability of the developed risk score tool for new users of dabigatran etexilate and rivaroxaban, prescribed following an AF diagnosis, was assessed by applying the tool to cohorts of these drugs’ users, followed for first major bleeding event using the same criteria as applied for the apixaban patients. We report the Harrell’s C-index (adapted for competing risk) for each drug cohort.

Statistical software

The SAS PHREG procedure was used to estimate Fine-Gray regression parameters and for assessing model discrimination.21 The R statistical programming language was used as specified above.

Patient and public were not involved in planning or conduct of the study.


Cohort baseline characteristics, follow-up and outcome

Apixaban cohort included 28 055 adult patients with non-valvular AF who filled apixaban prescription, for the first time, between November 2013 and December 2017. The mean age was 78.8±9.0 years, and 13 405 (48.3%) were males. Baseline characteristics are presented in table 1. During follow-up, a total of 662 patients received a diagnosis of a major bleeding event (table 1). There were 3813 deaths among subjects who have not experienced major bleeding during follow-up. CIFs of major bleeding and of all-cause mortality in the entire derivation cohort are described in online supplemental figure S1.

Table 1

Baseline demographics and co-morbid conditions of the entire derivation cohort, and separately for bleeding cases, and for patients with no major bleeding, at apixaban initiation

Major bleeding risk prediction model

It is noted that all available data were used for building the prediction model in order to maximise amount of information and the generalisability of results. Before starting the building process, we have assessed that availability of 662 bleeding events will allow considering about 25–45 candidate risk factors, according to the common rule of 15–25 events per variable.

Univariate analysis of each candidate predictor is summarised in online supplemental table S2, presenting the estimated sHR of major bleeding. The first step of the selection algorithm identified an initial list of promising predictors, according to their estimated variable importance, as presented in figure 1. The second step, aiming at achieving a parsimonious multivariable model using backward selection with BICcr criteria based on the initial 10 candidates identified in the first step, retained a final list of 9 variables. The regression coefficients and sHR from a model that includes these nine variables are presented in table 2. The average regression coefficients and average sHR of these variables using 500 bootstrap samples are also provided and show comparable results.

Table 2

Nine-predictor Fine-Gray subdistribution hazard model estimates depicting adjusted effects on cumulative incidence function of major bleeding

Model performance assessment

The CIF plot by quartiles of the 9-predictors model’s linear predictor (xbeta) demonstrates that while the incidence of major bleeding in the cohort is low, there is separation between risk quartiles, where the fourth (upper) quartile is markedly separated (online supplemental figure S2).

Figure 2

ROC of prediction models at 1 year: The 9-variable model, along with the full (all variables) model, HAS-BLED, ATRIA, ORBIT and CHA2DS2-VASc. ATRIA, AnTicoagulation and Risk Factors In Atrial Fibrillation; CHA2DS2-VASc, Congestive heart failure, Hypertension, Age ≥75 years, Diabetes mellitus, prior Stroke/TIA/Thromboembolism, Vascular disease, Age 65–74 years, Sex category; HAS-BLED, Hypertension, Abnormal liver/renal function, Stroke history, Bleeding history or predisposition, Labile INR, Elderly, Drug/alcohol usage; ORBIT, Older age, Reduced /haematocrit/anaemia, Bleeding history, Insufficient kidney function and Treatment with antiplatelets.

The c-indices of the 9-variables model, along with that of a model that includes all our candidate variables, and four established clinical models are presented in online supplemental table S3. The 9-predictor model provides satisfactory discrimination (c=0.642, 95% CI=(0.62 to 0.66)) relative to the larger model (c=0.647, 95% CI=(0.62 to 0.67)) and to HAS-BLED, ATRIA, ORBIT and CHA2DS2-VASc scores (c=0.570, 0.598, 0.605, 0.563, respectively). This was confirmed in the 1-year ROC curve (figure 2). The AUCs at 1 year were: 0.658 and 0.660, for the 9-predictor model and the larger model, respectively, and 0.576, 0.616, 0.626 and 0.572 for HAS-BLED, ATRIA, ORBIT and CHA2DS2-VASc, respectively (online supplemental table S4).

The optimism-bootstrap adjusted AUC at 1 year for the 9-variables model was 0.652, informing that the optimism bias conferred by the apparent AUC measure reported above is negligible.

Calibration plot of the 9-variables prediction model is provided in online supplemental figure S3. Inspection of the predicted probability of major bleeding within 1 year to the observed probability of major bleeding by 1 year within the 10 deciles of predicted risk shows that overall, calibration appears reasonable in our very low-risk population.

Figure 3

Points-based tool for predicting major bleeding event by 1 year. The histogram refers to the risk score distribution in the cohort. Each bar represents the proportion of subjects in the cohort that was assigned the relevant score range. The histogram was divided into quartiles; each quartile was coloured differently. For example, the middle 50% of the cohort had points score between 20 and 30. The 1-year major bleeding probability can be read by the predicted risk blue curve, using the right y-axis. For example, a points score of 50 is associated with ~6% risk. Table 3 provides tabular presentation of the risk associated with each total score.

Major bleeding risk score tool

Based on the 9-variables prediction model, scores were assigned for all predictors levels to produce points-based scoring tool. The possible total score ranges from 0 to 59 (table 3). For each possible total points score, the probability of event by 1 year based on estimated CIF is summarised in table 4 and is graphically presented in figure 3, along with the points score distribution in our cohort. (Observed events in the cohort for each possible total point score are presented in online supplemental table S5.)

Table 3

NOACs Bleeding Prediction (NBP): points-based risk-scoring system* for major bleeding

Table 4

Points-based risk-scoring system: predicted 1-year major bleeding cumulative incidence

An online NBP score calculator is provided ( (for Google Chrome). The apparent AUC of the points tool for predicting 1-year major bleeding was 0.6546, only negligibly smaller than that of the original prediction model. Stratified CIF plot for quartiles of points score is presented in figure 4, confirming the discrimination of the proposed tool. Major bleeding cumulative incidence rates within 1 year were 0.75%, 1.2% and 1.8% in the first, second and third risk quartiles of the points-based score, respectively, and 2.9% in the highest-risk points-scores quartile.

Figure 4

Cumulative incidence plot of major bleeding in four risk strata, based on quartiles of points-based score. Major bleeding cumulative incidence rates within 1 year were 0.75%, 1.2%, and 1.8% in the first, second and third risk quartiles, respectively, and 2.9% in the highest risk quartile. Numbers of patients at risk are presented.

Application of the major bleeding risk score tool to other NOACs

We identified 14 118 and 5450 AF patients, new users of dabigatran etexilate and rivaroxaban, respectively, for which the tool was applied to compute risk points for major bleeding (using table 3 scoring). Baseline demographic and clinical characteristics of dabigatran and rivaroxaban cohorts are presented in online supplemental table S6. When used as the sole explanatory variable in a competing risk model, the score achieved c indices of 0.62 and 0.61, in the two cohorts, respectively, only slightly lower than the result achieved in the derivation cohort.


We present major bleeding risk-scoring tool developed for AF patients initiating NOACs. Existing clinical models had been generated for the prediction of bleeding risk associated with the older vitamin K antagonists or antiplatelet therapy.7–9 Predictive performance of the score we have developed distinctively for NOAC initiators was 0.65, as evaluated by the apparent AUC of the points-based tool, and it showed modest improvement relative to HAS-BLED, ATRIA, ORBIT and CHA2DS2-VASc scores. We suggest managing the modifiable risk factors we have identified and dictating the frequency of follow-up by the level of risk, as has been pointed out in the European Society of Cardiology (ESC) guidelines.1 It has been shown that regular bleeding risk assessment using a bleeding risk score tool is associated with reduction in bleeding outcomes.25

Most of the major bleeding events that occurred in our population were GI and intracranial haemorrhages, similar to former reports.26 Because we anticipated this, we ascertained as potential predictors GI diseases and factors that might specifically increase risk for intracranial bleeding. Indeed, the model parameters identified, following a rigorous objective process, included, in addition to the more traditional markers, a novel marker we have constructed and named ‘risk factors for a fall’ composed of history of a fall or of hip fracture, or diagnoses of gait problem, dementia or Parkinson’s disease. Our identification of low cholesterol level as a predictor of major haemorrhage goes in line with former reports by us and others of an association between reduced serum cholesterol levels and higher risk of intracranial27 28 and GI haemorrhages.29

Age was not included in our final model (but appeared as the next predictor when these were listed by their estimated variable importance as detailed above). Of note, because NOACs were covered by the Israeli national insurance plan in the beginning only for high CHADS2 score patients, the studied population was relatively old (mean age 78.7 years). Only 7.6% of patients in our derivation cohort were <65 years of age, which had been the cut-off for age that was associated with higher risk in HAS-BLED.

There were a few notable differences in model development between the score presented here, and other scores, for example, HAS-BLED. In HAS-BLED, bleeding risk was assessed in 1 year, in 3978 patients from the Euro Heart Survey on AF; 2242 of them were receiving (the older) oral anticoagulation. The rest received antiplatelet therapy alone or no antithrombotic therapy. There were 53 major bleeding cases. In development phase of the current model, we included unselected population of 28 055 apixaban patients, and 662 bleeding events were identified. In addition, we rigorously used objective statistical principles to determine which variables to include; some of them might represent markers that provide prediction. On the contrary, historical bleeding risk factors identified in the literature were included in HAS-BLED, not necessarily with statistical significance. In HAS-BLED, also, potential selection bias might have occurred because of 25% missing data regarding the occurrence of major bleeding during follow-up. Of note, our annual bleeding rate was 1.38 per 100 person-years, comparable to the reported rate in ATRIA, where first major haemorrhage occurred in 1.4% annually.9

There are a few limitations in our study. First, all three cohorts we have used were of Clalit patients. Replication of our results should be done in other cohorts. Second, similar to bleeding scores previously developed (such as the ATRIA),9 we have built the model on data from patients in whom anticoagulants have been administered. Patients who had not been prescribed NOACs at all (probably due to very high risk of bleeding) were not included in the study cohort. For example, in our cohort, there were only few patients with severe thrombocytopaenia or with cirrhosis. Underestimation of the risk associated with recurrent falls, for example, can occur, if patients with the most severe presentation would not be prescribed NOAC in the first place. In addition, this studied cohort was relatively old, and the model, thus, might be more appropriately used in older patients. Future studies should ascertain risk factors in younger patients. Third, we could not ascertain probable risk associated with use of non-steroidal anti-inflammatory drugs (NSAIDs) as these were over-the-counter medications in Israel. Finally, we included in the study patients who had been prescribed full NOAC dose, as well as patients prescribed reduced doses (eg, apixaban 2.5 mg twice daily). We have previously shown30 that 36% of patients prescribed reduced apixaban dose were not fulfilling the absolute requirement for reducing the dose. The model, thus, might underestimate their risk provided they would have received full dose.

In conclusion, we present a 9-predictor NOACs Bleeding Prediction (NBP) score we have developed to predict major bleeding. Use of this model might help clinicians in estimating the risk of bleeding on NOAC treatment of AF patients. We recommend validating the score in other cohorts.

Key messages

What is already known on this subject?

  • A few clinical models, such as the HAS-BLED (Hypertension, Abnormal liver/renal function, Stroke history, Bleeding history or predisposition, Labile INR, Elderly, Drug/alcohol usage), ATRIA (AnTicoagulation and Risk Factors In Atrial Fibrillation) and ORBIT (Older age, Reduced /haematocrit/anaemia, Bleeding history, Insufficient kidney function and Treatment with antiplatelets), were developed to predict risk of major bleeding during vitamin K antagonists or antiplatelet therapy. Pharmacokinetics and patient characteristics are not similar between new oral anticoagulant (NOAC) and the vitamin K antagonists. The ABC (age, biomarkers, clinical history)-bleeding score for NOAC patients that was constructed using clinical trials data incorporated growth differentiation factor-15 and cardiac troponin T, not widely available in the clinical setting.

What might this study add?

  • Nine-variables simple risk stratification score, with improved prediction, that includes: male sex, anaemia, thrombocytopaenia (<99×103/µL), concurrent antiplatelet therapy, hypertension, prior major bleeding, risk factors for a fall, low cholesterol level and low estimated glomerular filtration rate.

  • An online score calculator is provided ( (for Google Chrome).

How might this impact on clinical practice?

  • Use of this novel risk score, if validated, might help clinicians in estimating risk of bleeding when initiating NOAC to atrial fibrillation patients.

Data availability statement

Data are available upon reasonable request.

Ethics statements

Patient consent for publication


Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.


  • WS and NG contributed equally.

  • Contributors OB-G and NS performed the analysis; OB-G, AK, WS and NG designed the research study; OB-G and NG wrote the paper.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Linked Articles