OBJECTIVE To evaluate the costs and benefits of alternative systems of coronary heart disease monitoring in Scotland.
DESIGN An option appraisal was conducted to evaluate the costs and benefits of implementing a coronary heart disease monitoring system. This involved a review of existing Scottish datasets and relevant reports, specification of options, definition and weighting of benefit criteria by key stakeholders, assessment of options by experts, and costing of options. The options were assessed by 33 stakeholders (grouped as cardiologists, patient representatives, general practitioners, public health physicians, and policy makers), plus 13 topic experts.
SETTING Scotland (population 5.1 million).
RESULTS Between group mean benefit weights were: mortality rates and case fatality (10.6), quality of life (9.8), patient function (8.8), hospital activity (7.8), primary care activity (9.25), prescribing (5.72), socioeconomic impact (4.0), risk factors (7.4), prevalence (5.0), incidence (6.0), case registration (6.82), international comparability (4.2), breadth of coverage (8.8), and frequency (5.8). Differences between group weights were significant for prevalence (p = 0.048) and international comparability (p = 0.032). Four monitoring options were identified: acommunity epidemiology model, based on MONICA (monitoring trends and determinants in cardiovascular disease) study methodology applied to a series of eight representative communities, had the highest benefits, at an average annual discounted cost of approximately £360 000; models based on theAustralian cardiovascular disease monitoring scheme and on enhanced routine dataoffered fewer benefits at discounted average annual costs ranging from £165 000 to £195 000; finally, a coronary heart disease registry modelled on the Scottish Cancer Registry scheme would have had fewer benefits and substantially higher costs than the other options.
CONCLUSIONS The most beneficial coronary heart disease monitoring system is the community epidemiology model, based on MONICA methodology. Option appraisal potentially offers an explicit and transparent methodology for evidence based policy development.
- heart disease
- option appraisal
Statistics from Altmetric.com
Coronary heart disease remains a top national policy priority. Regular, valid monitoring (surveillance) is a key component of any such policy. The ideal coronary heart disease monitoring system should be able to assess the burden of coronary heart disease in the community; provide proper epidemiological information on the incidence and prevalence of acute myocardial infarction, angina, and heart failure; describe age and sex specific trends in different geographical areas; measure the need for services in both primary and secondary care; measure cardiovascular risk factor levels and trends; and assess the impact of medical, surgical, and preventive interventions.1-4
Current coronary heart disease monitoring in the National Health Service (NHS) has many deficiencies. Data on community based activity are limited, despite more than 80% of general practices having computerised patient records.5 Existing monitoring systems concentrate on acute episodes and deaths rather than on the disease burden in the population. Community management of angina and heart failure remains largely undocumented, and risk factor trends are similarly neglected.
Tunstall-Pedoe has outlined many of the problems which might be encountered in a coronary heart disease monitoring system.1 These include lack of pathological validation for cause of death, failure to record mild or “silent” infarctions, inadequate recording of acute and chronic medical care outside hospital, and lack of standardisation in cardiovascular risk factor measurement.
Finland,6 the USA,7-11 and Australia4 have already implemented comprehensive monitoring schemes. These offer coverage of risk factor prevalence, much enhanced data collection, and detailed analysis of acute coronary episodes.
Schemes for cancer registration and HIV surveillance have already proved the value of good quality monitoring information for decision makers.12 Better information will, however, be expensive. The additional costs and benefits of improved information therefore need to be critically examined. One possible technique for doing so is option appraisal, which is routinely used for capital projects and has shown its value in other non-capital areas such as priority setting.13 Option appraisal may also be suitable for the evaluation of new policies and new information systems, especially as it allows the benefits of the proposed service to be explicitly defined.14 The aim of the project described here was to use option appraisal to assess the costs and benefits of alternative options for monitoring coronary heart disease in Scotland.
The option appraisal followed the stages recommended by Henderson15 and HM Treasury.16 The stages include setting objectives, identification of options (including the existing system of monitoring coronary heart disease in Scotland), specification of benefit criteria, weighting the criteria to reflect their importance, scoring options against criteria, and then comparison of weighted scores with costs.
Options were identified from a 1996 policy review and from published reports.1-6 The project steering group then generated a long list of options, plus suboptions which could be added to any major option (such as population surveys, self contained modules, or research projects). The status quo was also included in order to facilitate comparisons. Reduced levels of monitoring were not considered.
A short list of options was assembled from the most promising major option and suboptions (table 1). In the sensitivity analysis, each option was compared with the next best option, with and without different suboptions.
IDENTIFYING AND MEASURING BENEFITS
Benefit criteria were defined in terms of the information that a monitoring system should contain according to experts and published reports. Fourteen benefit criteria emerged, selected on the basis of key characteristics of coronary heart disease monitoring systems (all fell within four categories: epidemiology, service utilisation, measuring effectiveness and outcomes, and system quality (table2)).
WEIGHTING AND VALUING THE BENEFITS
Coronary heart disease monitoring information is of interest to many different stakeholders, each conceivably having quite different preferences regarding the content and quality of the system's output. The project working group identified five separate groups that would be involved in collecting or using this information:
Planners and policy makers, represented by five members of the Scottish Office coronary heart disease priority team.
Ten directors of public health in Scotland.
Four cardiologists from the council of the Scottish Cardiac Society.
Ten general practitioners from the Scottish General Medical Services Committee.
Four patient representatives from the Scottish Association of Health Councils.
Each group was approached and given a brief presentation on the project and the methods of option appraisal. To elicit weights for the benefit criteria, the five groups were asked to distribute 100 points among the 14 criteria, based on the importance they attached to the various characteristics of a monitoring system. The distribution of these points thus reflected the relative importance of each criterion. Four groups chose to submit individual sets of benefit weights for each individual respondent, so a mean weight was then generated. The remaining group preferred to generate a single set of weights.
Respondents were asked to weight criteria ignoring the level of information currently provided under the status quo. This approach allowed groups such as patient representatives to express their preferences without requiring detailed knowledge of existing NHS information systems.
SCORING THE OPTIONS
Each option was then scored in terms of how well it would achieve the 14 benefits. Fourteen United Kingdom experts in different coronary heart disease related areas were invited to take part in the scoring, and 13 agreed to.
Scorers were given detailed option specifications and asked individually to assess each option by scoring each benefit criterion (from 0, the poorest possible level of monitoring, up to 100, representing perfection). Each was then interviewed to allow them to explain their judgements. Scorers were explicitly asked not to consider cost.
IDENTIFYING, MEASURING, AND VALUING COSTS
The perspective taken was the cost of each option to the NHS. Costs (including a plausible range for best and worse case scenarios) were identified by approaching organisations or individuals with appropriate expertise. Costs were examined both undiscounted, and discounted at 6% per annum (the HM Treasury suggested discount rate).
To account for periodic surveys, average annual costs were calculated over a time horizon of 10 years. A longer span was considered unrealistic, given the expected innovations in NHS information technology. All costs were held at 1998 levels.
OPTION SELECTION USING WEIGHTED BENEFIT SCORES
For each option, the option score was multiplied by the benefit weight to produce a total weighted benefit score. Those options which cost more than an alternative yet yielded fewer benefits were then discarded. Options were evaluated in terms of total cost and incremental costs over the next best option. Unquantified benefits and cost variability were also considered.
STATISTICAL ANALYSES AND SENSITIVITY ANALYSES
Group scores were summarised using means and medians as appropriate. The sensitivity analysis revolved around four key elements: benefit weights, option scores, costs, and option features. Several options were remodelled in the sensitivity analysis to include additional features. By noting the judgements behind scorers' decisions for options including similar features, it was possible to estimate the effect that their addition might have on the option score.17 Differences between those groups who submitted mean weights were tested for each benefit criterion using the Kruskal-Wallis test.
Five groups produced benefit weights (table 3). Between group mean benefit weights were: hospital activity (7.8), primary care activity (9.25), prescribing (5.72), socioeconomic impact (4.0), risk factors (7.4), prevalence (5.0), incidence (6.0), case registration (6.82), mortality rates and case fatality (10.6), patient function (8.8), quality of life (9.8), international comparability (4.2), breadth of coverage (8.8), frequency (5.8). The highest weights were generally awarded to criteria capturing outcome measures. The Scottish Cardiac Society and the directors of public health awarded around one third of their weights to these criteria alone. Weighters valued good primary care data as highly as hospital activity data. Weights for prevalence and incidence showed quite large variations both within and between groups. Socioeconomic impact and international compatibility received low weights from all groups.
The highest weights for primary care activity came from the general practitioners and the patient representatives, at 10.5 and 11.3, respectively. The Scottish Office policy makers reserved their highest weight (20) for breadth of coverage. Differences between group mean weights were significant between directors of public health and general practitioners for prevalence (p = 0.048), and between directors of public health, general practitioners, and patient representatives for international comparability (p = 0.032).
The ranges of mean and individual benefit values were tested in the sensitivity analysis.
Ten options and suboptions were shortlisted, based on four models: an enhanced routine data model which built on the existing arrangement; a community epidemiology model, based on World Health Organisation MONICA (monitoring trends and determinants in cardiovascular disease) methodology and involving eight representative communities; a model based on theAustralian cardiovascular monitoring scheme; and a coronary heart disease registration model based on the Scottish Cancer Registry scheme (table 1).
Thirteen topic experts scored the five options against the 14 benefit criteria. The highest median scores went to the community epidemiology model for 13 of the 14 criteria (table 4).
Discounted average annual option costs ranged from £165 315 for the enhanced routine data model to £827 728 for a coronary heart disease registry model (fig 1).
Staff accounted for the large majority of costs in every option. Over and above the costs quantified in the analysis, the enhanced routine data and coronary heart disease registry options would make considerable use of the centralised information resources already available at the Scottish Information and Statistics Division. All options were likely to increase the amount of doctor and nurse time spent cataloguing details. The community epidemiology and Australian cardiovascular disease monitoring scheme models were likely to be the most costly in terms of patient time for filling in survey forms. The community epidemiology model would also require patients to attend physical examinations and would therefore incur costs for travel and time off work.
WEIGHTED SCORES AND CREATION OF HYBRID OPTIONS
All 646 combinations of weights and scores possible for the four basic and enhanced options were analysed (34 sets of weights and 19 sets of scores, including means and medians). The community epidemiology option had the highest total benefit score in 603 of the 646 combinations.
Further analysis was based on mean benefit weights and median scores from scorers. The coronary heart disease registry and the original Australian cardiovascular disease option were then discarded, both having higher costs than other options yet yielding fewer benefits for all scenarios (table 5).
Ranking of the remaining options according to total benefits remained the same in each scenario: community epidemiology model, followed by an Australian model upgraded with a quality of life patient survey, then enhanced routine data upgraded with risk factor surveys, and finally the original enhanced routine data model. In each case, more benefits led to higher costs (tables 5 and 6).
IMPLICATIONS FOR DECISION MAKERS
This project has shown first, that superior strategies for monitoring coronary heart disease are available, feasible, and not particularly expensive, particularly when set in the context of the total cost of cardiac disease.1-4 It thus has considerable value for the NHS. Second, it has shown that improved primary care and outcomes monitoring are of paramount concern. Third, key stakeholders have a range of different needs from a monitoring scheme.5
Our project identified, developed, and examined four major options for monitoring coronary heart disease in Scotland, in addition to the status quo. Once the Scottish Cancer Registry model had been discarded, the three remaining options each contributed progressively higher benefits as their costs increased. Highest benefit scores were seen for the community epidemiology model, at an estimated average annual cost of around £360 000 after discounting. The higher scores and costs principally reflected the benefits from rigorous community monitoring but also the quality assured methodology and standardisation, based on MONICA methodology.2
All option appraisals have limitations. Benefit criteria have to be selected early on. In this project, some criteria turned out to be less than ideal when the scoring stage was reached. Although this presented an issue when interpreting scores, additional criteria might have overly complicated the benefit weighting process.15
Ideally, scoring would have taken place in one session attended by all scorers or through a consensus generation process. Unfortunately this proved impracticable. However, such a process might have disguised genuine differences in opinion between experts.15 ,16 Had resources permitted, we would have consulted larger numbers of stakeholders and topic experts. However, we suspect that the results would not have been very different.
True costs were difficult to gather as they are often subsumed within the overall costs of the agencies or systems in which the coronary heart disease monitoring data are gathered. Also, patient costs were not included. However, as elsewhere, staff salaries dominated.4 ,6 ,10 ,14
The results of benefit weighting and scoring need to be interpreted with care. Benefit weights are assumed to be cardinal because one point given to “incidence” represented one less point available for other criteria. However, option scores may only reflect an ordinal scale with an approximate indication of the strength of preference. Although total weighted benefit scores are listed in table 5, they are therefore best interpreted as indicating an overall ranking of options.15
The project identified several further issues for research. These include a programme budget for resources currently spent on coronary heart disease monitoring as part of routine data collection, and an assessment of the benefits of high quality monitoring for related disease groups, such as peripheral vascular disease and stroke, which could be added for little extra cost.3 ,4
Option appraisal was found to offer a transparent methodology for assessing the costs and benefits of a range of competing options for monitoring coronary heart disease.15 ,16 The results of this appraisal should be of relevance to the many groups with an interest in NHS disease monitoring—policy makers, planners, health care professionals, patients, and the public. Option appraisal should be considered more often when such policy challenges arise.
We thank the groups who provided weights: the Scottish Office coronary heart disease priority team, the directors of public health in Scotland, the council of the Scottish Cardiac Society, the Scottish General Medical Services Committee, and the Scottish Association of Health Councils. We are very grateful to the individual coronary heart disease experts who scored the detailed options: Dr J Clarke (Information and Statistics Division, Edinburgh), Dr P Doyle (Department of Health, London), Dr J Dunbar (general practitioner, Dundee), Dr P Hanlon (Department of Public Health, University of Glasgow), Dr N Hicks (Anglia and Oxfordshire Health Authority), Dr M McWhirter (Forth Valley Health Board), Dr J Pell (Greater Glasgow Health Board), and Professor D Wood (Imperial College, London). We thank the Chief Scientist's Office of the Scottish Office for funding this project.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.