An observational study comparing quality of care in walk-in centres with general practice and NHS Direct using standardised patients

Recruitment of clinical sites

For logistical reasons we approached sites in three geographical areas, in and around Bristol, Birmingham, and London. We invited general practices involved in research or teaching to participate, and we approached walk-in centres and NHS Direct sites on the basis of their geography. Twenty out of 25 (80%) walk-in centres, 11 out of 12 (92%) NHS Direct sites, and 24 out of 62 (39%) practices agreed to participate: we included the first 20 practices to respond. Twelve practices (19%) declined to take part, and the remainder did not respond.

Ethical approval for our study was obtained from a multicentre research ethics committee. Participating sites all gave informed consent for contact with standardised patients but were unaware when they would consult. Participating sites were ensured anonymity: outcome data were presented such that individuals could not be identified.

Selection of clinical scenarios

We chose five clinical scenarios, largely to represent problems likely to be presented by patients to walk-in centres (box 1). The scenario on postcoital contraception was intended to assess management of a common, straightforward problem, the scenario on chest pain to assess ability to exclude a potentially serious diagnosis and reassure accordingly, and the scenario on sinusitis to assess issues around antibiotic prescribing. The scenario on headache was devised to assess ability to explore psychosocial issues, and the scenario on asthma to assess the history taking on drugs and awareness of the side effects of the drugs.

Derivation of assessment criteria

We assessed clinical care against prospectively determined standards. We constructed lists of essential criteria for the adequate management of each scenario by a stepwise procedure, based on the Delphi process.13 The research team identified potential criteria for essential actions relevant to the five scenarios, by review of relevant literature and consensus based guidelines. We posted lists of potential criteria to a consensus panel of nine general practitioners and five nurse practitioners. Criteria were included as items in the final checklists if, after two rounds, 93% or more members of the panel agreed they were essential. Checklists consisted of eight to 17 items, depending on scenario and setting, with items grouped into “history taking,” “examination,” and “diagnosis, advice, and treatment.” For each scenario, checklists for the walk-in centres and general practices were identical, with the NHS Direct checklist containing all but the examination items.

Standardised patient consultations

Fifteen professional role players involved in medical education played standardised patients. They were chosen to match the roles assigned to them in terms of age and sex. Five role players, each portraying one scenario, worked in each locality. Each role player visited a particular walk-in centre or general practice once, but owing to the smaller number of sites for NHS Direct, contacted one NHS Direct site up to three times. Overall, 305 contacts were planned, 100 in walk-in centres, 100 in practices, and 105 with NHS Direct. Consultations took place over a 13 week period from July to September 2001. Standardised patients visited the general practices as temporary residents to avoid the logistical problems of registration as a new patient. They completed checklists immediately after consultations.

Training of role players

The role players were provided with detailed information about the sites and personal data relating to their roles. They received a day's training in role portrayal and clinician assessment, during which data on accuracy of portrayal and reliability of assessment were collected.

Accuracy of portrayal of standardised patients

Accuracy of portrayal, defined as the proportion of predefined clinical features presented correctly, was measured by two observers independently completing a performance item checklist during the final round of role playing.14 Across all scenarios, 1038 of 1164 (89%) clinical features were portrayed correctly. Consistency of portrayal was maintained by role players watching a videotape of their performance before each consultation during the study.

Reliability of assessment by standardised patients

Reliability of assessment was measured by comparing role players' responses to each checklist item with gold standard responses. These were the responses of two general practitioners who independently watched videotaped consultations from the training day and completed the checklists blind to role players' responses. Across all scenarios and role players κ=0.8. The reliability between role players was measured by the role players independently assessing each other's videotaped consultations. The responses of two role players observing the same videotaped scenario were compared. Across all scenarios and role players κ=0.9. The reliability of the role player over time was measured by role players assessing their training day videotapes half way through the study and comparing responses to those on the training day: κ=0.7.

Validity of standardised patients

Participating clinicians were encouraged to contact the study team, anonymously via an answer machine, if they suspected a consultation with a standardised patient. Twenty three calls were made—five about true standardised patients and 18 about true patients. Thus the detection rate was 1.7% (5 of 297), and the positive predictive value was 21.7% (5 of 23).

Analysis

For each consultation we calculated a score representing the proportion of essential criteria fulfilled for all items and separately for the three subgroups of items (history taking, examination, and diagnosis, advice, and treatment). We calculated the means of the all item and subgroup scores for each of the three primary care settings, with 95% confidence intervals calculated with design weighted survey estimators. We then undertook estimation of differences between mean scores for the three settings by using multivariable regression models, with setting and scenario included as fixed effects. We accounted for non-independence of repeat observations within individual centres in two alternative ways: models were estimated with design weighted survey estimators and with individual centre included as a random effect, as implemented in Stata.15 For each model we then added the interaction between primary care setting type and scenario and tested for significance. Where this interaction was significant we estimated separate models excluding first NHS Direct consultations,(allowing comparison of walk-in centres and practices) and second general practice consultations (allowing comparison of walk-in centres and NHS Direct). Within each scenario we estimated mean scores, mean score differences between settings, and 95% confidence intervals for these statistics by using design weighted survey estimators. Scores for the three subgroups of items were not normally distributed, so we repeated all analyses with the Wilcoxon rank sum test. In no case was the result of the non-parametric test significantly different to the analyses presented in tables 1 and 2. Finally, we calculated the variation in scores (standard deviation) within each setting.

The results of the design weighted models and random effects models were similar in all cases, and thus only the design weighted analyses are presented as these tended to be slightly more conservative than the random effects models. For the models based on all items and those restricted to history taking and diagnosis, advice, and treatment, the interactions between primary care setting and scenario were significant. Thus, we present the results of these analyses and analysis of examination items separately for comparisons between walk-in centres and general practice and then between walk-in centres and NHS Direct.

Quality of care

Walk-in centres versus general practice

Considering all scenarios together, walk-in centres achieved a significantly greater mean score for all essential items than did general practices, but there were between scenario differences. Quality of care for postcoital contraception and asthma was significantly better in walk-in centres than in general practices, that for sinusitis and headache was similar in the two settings, and that for chest pain was better in general practice, although not significantly so (table 1).

Overall, and for each scenario, walk-in centres scored better on history taking, with significant differences overall and for two individual scenarios (table 2). Overall, and for the two scenarios involving examination (chest pain and asthma), general practices scored better on examination, although for asthma the difference was not significant. Overall, there was no significant difference between quality of diagnosis, advice, and treatment provided by walk-in centres and general practices. However, for postcoital contraception and asthma, the quality of diagnosis, advice, and treatment was significantly better in walk-in centres, and for sinusitis it was significantly better in general practices. In summary, walk-in centres provided equivalent if not better quality of care than general practice, with the exception of advice and treatment of sinusitis and examination of chest pain.

Walk-in centres versus NHS Direct

Considering all scenarios together, walk-in centres achieved a significantly greater mean score for all items than did NHS Direct, again with between scenario differences. Quality of care for postcoital contraception and asthma was significantly better in walk-in centres, with no significant difference for other scenarios (table 1). Overall, and for postcoital contraception and asthma, walk-in centres scored significantly better on history taking, with the same pattern for diagnosis, advice, and treatment (table 2).

Referral

About a quarter (26%) of consultations in walk-in centres and four fifths (82%) with NHS Direct resulted in referral (table 3). Patients were advised to go to an emergency department in 5% of consultations in walk-in centres and 13% with NHS Direct, but not from consultations in general practice.

Between site variability

Considering all scenarios, the variability of quality scores, expressed as standard deviation, was highest for NHS Direct and lowest for general practice (table 4). However, variability of history taking for all scenarios was higher in general practice. Variability of quality in the three settings varied across scenarios. NHS Direct's high variability was largely the result of the postcoital contraception scenario, where referral may have substituted for consistent performance on essential items.

Access

Contacting NHS Direct was sometimes time consuming and unsatisfactory. About one quarter (25 of 99) of completed calls involved “call back” (mean wait 33 minutes). Three consultations with NHS Direct were not completed owing to length of wait for “call back.” Fewer problems were reported accessing general practice and walk-in centres.

Comparison with other studies

The accuracy of portrayal (90%) by standardised patients compared well with previous studies, as did their reliability (κ=0.7 to 0.9). 14 18 Detection (1.7%) and positive predictive value (21.7%) also compared favourably.18^–21

Although checklists were meant to consist of essential items, the proportions of items achieved in all three settings were low, a finding consistent with another study that assessed general practitioners against peer determined standards.22 Performance may not reflect competence, as clinicians exhibit efficiency by only carrying out what is necessary at a particular moment.22

Methodological issues

The study's main limitations are non-random sampling of participating sites, use of a limited number of scenarios—some more discriminating than others—and use of novel assessment checklists. Participating sites, particularly general practices, were likely to be more interested in the research question and may have provided a higher quality of care, possibly attenuating the study findings. Scenarios were chosen as typical of those seen in walk-in centres and because they were appropriate for portrayal by standardised patients. Scenarios necessitating the presence of abnormal findings or potentially involving certain types of physical examination or referral to third parties could not be included. The finding that walk-in centres offer safe care cannot necessarily be extrapolated to all clinical conditions, although we aimed to assess important, generalisable aspects of clinical care.

The ability of the scenarios to discriminate between settings was variable. Scores on the scenario for postcoital contraception were highly variable both overall and for NHS Direct in particular, perhaps because NHS Direct's management of this scenario always involved referral. Consequently, performance on history taking and advice and treatment may have been less thorough.

Scenarios did not involve children and elderly patients, who attend general practice most often. Nor did they assess some of the supposed strengths of general practice—for example, continuity and availability of past medical records. Also, the methodology, requiring all standardised patients to be temporary residents, did not lend itself to the assessment of some of these strengths. Thus, the findings do not suggest care in general practice is inferior to walk-in centres, rather that walk-in centres perform adequately and safely compared with controls.

The development of assessment checklists used by standardised patients in previous studies has received little attention, although their construction is crucial to the reliability and validity of assessment.23 Checklists used in this study seem to have high face validity and content validity, although few data were collected on other properties, such as reproducibility.

Conclusion

Standardised patients have a useful role in the assessment of quality in the UK primary care setting. Also, walk-in centres provide adequate, safe clinical care to a range of patients, compared with general practice and NHS Direct. However, the cost effectiveness of walk-in centres and their impact on workload of other healthcare providers requires further assessment.