Statistics from Altmetric.com
Until the turn of the century physicians were trained as apprentices, basing patient management on experience, supported by a limited understanding of the disease process and the effects of treatment. There was no “optimal management” of disease, except that defined by the teachers in the profession, who in turn based teaching largely on anecdotes.
The development of the mathematics of probability as applied to medicine,1 and the consequent introduction of the randomised clinical trial,2 has changed the basis of medical practice in many disciplines. For many common conditions, it is now possible to define contemporary optimal management of disease in terms of the probability of a defined outcome. The probabilities are usually based on evidence from clinical trials or observational studies. Clinical trials have the advantage of minimising bias and negating confounding and unanticipated variables, while observational studies have the advantage of studying usually larger and more heterogeneous populations of patients more typical of clinical practice.
Cardiovascular disease causes more premature mortality and morbidity in developed countries than any other organ disease.3 4 It is appropriate that a wealth of information now exists about the common conditions to allow definition of optimal management. When practising medicine by applying the results of clinical trials and observational studies to an individual patient, there is an inherent assumption by both doctor and patient that the outcome defined by such evidence will be achieved. What is usually missing is the measurement of that outcome.5 6
On a probability basis it is likely that different medical teams will achieve significantly different results in similar groups of patients using the same treatments. It must be a prime objective of clinical audit to apply the methods of quality assurance both to improve practice in general and to identify and eliminate the lower end of the distribution of clinical effectiveness, thus enhancing the average delivery of care.
Clinical audit may be defined in a number of ways. At its simplest, it is application of the discipline of clinical trials to all medical practice with the requirement to define the patient, disease, treatment, and outcome in a way that can be used to compare results against contemporary medical evidence. Clinical audit may also be seen in the context of quality assurance, where it is but one element of a process which results in the identification of both good and bad practice, resulting in discussion, debate, and action whereby all practice is improved and the bad is minimised.
One of Paul Wood’s7 great attributes was his meticulous collection of information on the patients he treated. None of us can now be in any doubt that we have individual and institutional responsibilities to follow his example. The recent failure of the audit process in the Bristol cardiothoracic unit,8 the subsequent pronouncements of the secretary of state for health,9 and some of the statements in the new National Health Service (NHS) information strategy10 make these responsibilities abundantly clear.
Effective clinical audit is difficult to implement; not all of the issues surrounding collection of data, analysis, interpretation of results and subsequent action have been solved although there are currently some spectacular examples of data presentation.11 This review discusses the requirements of an audit system and describes the UK national audit project in cardiology.
Background of national data collection projects
The UK has a good record in cardiology of setting up prospective databases for measurement of activity and outcome. In 1969 Graham Miller at the Brompton Hospital in London established a computer based cardiac catheter laboratory reporting system that developed an extensive diagnostic library and was subsequently widely adopted internationally.12 Catheter laboratory activity, especially in complex paediatric conditions, could be accurately described and provided the basis for many observational studies. In 1971 Raphael Balcon at the London Chest Hospital established an institutional based registry of angiographically defined coronary artery disease severity, which is still being used to generate long term outcome information.
The first national project to collect and collate audit data was started by the British pacing group in 1978. This project arose from a system developed at the National Heart Hospital in 197413to collect information on pacemaker implants. Key to the success of the project was the serendipitous definition of the main elements of quality assurance; information regarding the reasons why pacemakers were implanted, the process of implantation, and defined outcomes was collected. The simple data definitions developed have been widely adopted internationally and remain almost unchanged after 30 years.
One of the main objectives of the national pacing database (as it has since become) was the analysis of overall UK activity, in particular the analysis of the appropriateness of pacing treatment. In conjunction with best practice guidelines,14 these data have been successfully used to modify and improve the practice of pacing in the UK, both in quantity and quality terms, thus achieving the objective of a quality assurance process. In retrospect the success of the national pacing database can be attributed to its concentration in a small specialty where outcomes are based on hard and easily measured end points such as patient death or device failure, both of which can be independently checked using national registries and device manufacturer records. The lessons learnt from the national pacing database are useful in planning audit efforts in other medical disciplines.15
During the last two decades individual specialist groups in cardiovascular disease have implemented national data collection exercises. These range from simple activity counting with limited patient based information (British Cardiovascular Intervention Society) to comprehensive databases defining the process of care, such as that established by Society of Cardiothoracic Surgeons. While these efforts have provided valuable activity information there is little evidence that they have contributed to the quality of care. In the activity counting exercises, no useful patient based information is available; in the more comprehensive databases where patient information is collected, the patient population is heterogeneous and follow up is limited, so that inter-institutional comparisons, or comparisons of performance against published evidence, can be easily challenged and invalidated.
In 1996 the Department of Health was approached by six specialist groups affiliated to their respective professional bodies (the British Cardiac Society, the Society of Cardiothoracic Surgeons, and the Association of Cardiothoracic Anaesthetists). These groups were each responsible for collecting information in the cardiovascular domain. The project proposed to harmonise the data collection efforts of the groups and to design a medical audit system, based on the principles of quality assurance, to be used nationally. The project is known as the central cardiac audit database (CCAD). A three year pilot phase to assess feasibility began in May 1996.
Medical quality assurance
The process of quality assurance can be seen as a series of steps which start with defining a medical domain of interest for audit.
During the definition of the clinical audit domain, a decision has to be made as to where to “start” the process. When auditing the overall management of coronary disease, the starting point ideally should be the first clinical presentation of the patient with symptoms or signs of the disease. In practice this would require the creation of a medical record which spans primary, secondary, and tertiary care. Such a standardised record does not currently exist and, while not technically complex, the organisational and training issues involved would probably swamp the audit objective. From a tertiary care perspective it might be considered that the coronary disease audit could start with definition of the coronary anatomy by angiography. Audit per se of cardiac catheterisation is not a prime objective of CCAD although collection of overall cardiac catheter activity is a useful exercise.
The quality assurance process does not require comprehensive data collection of all activity within an institution. However, institutions generally have a requirement to collect overall activity, and it is therefore sensible to design a data collection system for quality assurance that also includes overall activity reports. In such circumstances definitions of desired outcomes are clearly possible and certainly desirable.
There is a known trend towards individuals and institutions displaying the same performance characteristics in different clinical domains. What evidence exists suggests that institution dominates the individual in determining performance,16 but the hypothesis that performance is dependent on the institution and independent of the clinical domain and the medical team needs to be tested.
In selecting the disease and treatment to be studied we suggest prioritising candidate domains in the context of categories defined in table 1.
CCAD has defined five clinical domains for audit. These domains represent the expertise and interests of the specialist subgroups comprising the project.
- Management of coronary disease by surgery or intervention. This is a type A audit where the disease is common, associated with high levels of resource utilisation and rich in evidence from randomised trials, observational studies, and guidelines.
- Use of pacemakers. This is a type B1 audit where the disease is less common but associated with high levels of resource utilisation. Although limited evidence in the form of randomised trials exists to judge appropriateness, the indications for pacing in the majority of patients are not disputed.
- Surgical management of adult valvar heart disease. This a type B2 audit, where the disease is less common but associated with high levels of resource utilisation. Little evidence in the form of randomised trials exists to judge appropriateness but outcomes are well defined.
- Use of interventional electrophysiology to manage arrhythmias. This again is a type B2 audit where the disease is less common but associated with high levels of resource utilisation. Little evidence in the form of randomised trials exists to judge appropriateness but outcomes are well defined.
- Paediatric cardiac surgery and intervention. This is a type C audit where the diseases are rare but consume high levels of resource. Evidence to judge appropriateness is very limited but outcomes are well defined and occur with enough frequency to be analysable. The rarity of the diseases makes individual performance analysis and inter-institutional comparisons more difficult.
In terms of individual and institutional priority, is it clearly important to implement type A audits; they are likely to be easier to interpret and to generate useful knowledge quickly to improve performance. Type C audits may require national or even international cooperation over long periods of time to generate useful knowledge.
The definition of the data to be collected for audit in a particular clinical domain should be considered under the following categories.
This set of data is concerned with defining the environment in which treatment is being delivered. It should provide adequate data to answer the question as to whether a particular form of treatment should be delivered in a specific environment. Table 2 indicates some of the candidate variables relevant to specialist cardiovascular care. The CCAD project does not collect these data on an individual patient basis except for reference to the institution and operators where interventions are performed. It is felt that these data are more appropriately collected at intervals by the specialist societies.
This set of data asks the question whether it was appropriate that a particular form of treatment was delivered to a particular patient (table 3). Appropriateness can be considered both as indications (for example, severity of symptoms or coronary disease) and as contraindications (for example, known risk factors for intervention). Definition of the appropriateness variables is a key step in clinical audit as it should allow comparison of the results of treatment in different institutions in similar groups of patients.
This set of data defines the process by which treatment was delivered and may include clinical variables related to a particular procedure as well as more resource orientated variables such as waiting times, length of hospital stay, and costs (table4).
Any intervention should be associated with well defined desired and undesired outcomes. These will include key clinical and quality of life variables but could also include resource orientated information such as time off work (table 5).
Once the definition of the atomic data items has been completed these data need to be turned into information (generation of information occurs when data are placed in context). The CCAD project deliberately encompasses multiple clinical interests within cardiology and an important objective is to harmonise the data definitions across these interests so that the same data item conveys the same information to the different groups in the context of that data items use. This means the development by consensus of common definitions and libraries within the cardiovascular domain.
There are basically two approaches to designing a data collection system for medical use, whether for clinical audit or general medical records.
The first is the data orientated approach commonly used for designing relational databases. In this approach a model of the medical process from which to acquire data is created and from this model entities, representing real world structures such as “patients”, “hospitals”, etc, are created and defined in terms of their attributes (such as a date of birth or sex of a patient). Data are rigorously “normalised” so that a real world data item is an attribute of a single entity and does not depend on any other item (thus “age” can never be an item in a relational database as it is derived from date of birth which in turn can only be stored as an attribute of a patient). The advantages of relational databases are that all logical questions of the process being modelled can be answered and inconsistency by duplication of data items is not possible. Data models are difficult to construct when information is highly temporal (as it usually is in medicine), and are difficult to maintain and document when representing complex processes. Data models do not adapt well to events not considered in the initial design and tend to be slow in both acquiring and presenting data when the dataset becomes very large.
The second approach is document based. Here the starting structure is a document, which should represent a real world part of a medical record (such as a patient registration document or an exercise test report). Data items become information in the context of the document in which they are contained and the same data item may be repeated in multiple documents. Documents are linked together by pointers within the document (hypertext) which identify parent, children or sibling documents. Documents can inherit information from parents and give information to children documents. Document based databases tend to be much easier to construct, maintain, and adapt in the medical environment because they represent the computer equivalent of contemporary medical records. Document based systems tend to be multimedia orientated and are easy to transmit and distribute across different computer platforms. Hypertext documents are the basis of the world wide web.
For the CCAD project we have decided to use a hypertext document based system for data collection because:
- such systems are relatively easy to design and maintain (because they relate to real world medical records)
- document based systems are highly developed to store and present temporal information
- document based systems are easy to adapt to multimedia information and easy to link to “foreign” remote information sources such as medical guidelines, data libraries, and image storage systems
- information (both patient based and system based, such as code and data libraries) can be easily transmitted from site to site using internet protocols
- document based systems have highly developed security attributes both at document and data item level
- document based systems conforming to internet protocols are at least moderately “future proofed” and are non-proprietary.
In designing a data collection system the CCAD project is very aware of its responsibility to protect both patients and health care professionals without compromising the objective of clinical audit. Anderson, in his review commissioned by the British Medical Association, has very elegantly laid down the principles of data security and confidentiality,17 and the CCAD project has adopted these principles in its system design as far as technically possible. In brief the anonymity and security features implemented are:
- all documents in the database contain an “access control list” which determines who can add to, read or edit the information in that document; documents cannot be deleted
- all document access for addition or modification of data is both time stamped and electronically signed by the user
- all information within the database whether stored or transmitted is encrypted
- there is a central “trusted authority” (at present the CCAD project itself) that issues and revokes encryption/decryption keys (based on Rivest Shamir Adleman asymmetric encryption) to individual users
- institutions taking part in CCAD have an additional encryption key with which access to the patient’s name, address, NHS number, hospital number, date of birth, and the operator’s name (for surgical or interventional procedures) can be limited to users defined by that institution. Age, sex and the first three letters of the postcode are not encrypted
- access to anonymised aggregated information is limited and determined as a matter of policy by the CCAD project.
Relation between CCAD and medical record development
The CCAD project is aware that its role in generating an audit driven data collection system could conflict with developments in electronic medical record development, in particular the development of information systems driven by computer based protocols.18Such systems are still in the early phases of development and present both organisational and technical challenges far greater than the collection of audit data in well defined medical domains. However, by adopting some of the techniques and by adhering to the technical standards likely to be adopted for medical record development, the CCAD project sees itself as the beginning of a process which will eventually lead to comprehensive electronic patient records supported by guidelines and protocols.
Future of the CCAD project
CCAD will have achieved its goals when: a dataset relevant to the audit domains of interest has been defined; the data can be shown to be collectable by a pilot group of institutions; and knowledge can be generated from the data to implement the quality assurance process both within and between participating institutions.
Following the three year pilot project there is still going to be a need for an organisation like CCAD to exist. Its future roles will include:
- maintaining the existing dataset and supporting data definitions
- at the request of the cardiology community, through the special interest groups, expanding the quality assurance process to include other clinical domains
- implementing an accreditation process whereby individual institutions are audited by CCAD from an organisational and technical standpoint as to their ability to generate quality information for medical audit
- acting as the central “trusted authority” for maintenance of data security and anonymity
- acting as the central data collection authority where data are collected, validated, and then passed to the special interest groups for analysis and distribution
- if requested, supporting and advising special interest groups on the most suitable methods for data analysis in different clinical domains
- provide advice to the special interest groups on how to interpret the data and translate the results into clinical action.
The CCAD project was set up to harmonise a variety of contemporary data collection exercises in cardiovascular medicine. It has applied the techniques of medical quality assurance in defining a dataset common to the medical domains of its client special interest groups, and categorised this dataset in terms of structure, appropriateness, process, and outcome (the “where, when, why, who and what happened” of audit) relevant to the individual medical domains. It has developed a hypertext document based data collection system with inherently strong security attributes suitable for data storage, transmission, and aggregation.
Its future role should be to maintain and expand the quality assurance process in cardiovascular medicine, act as a trusted authority to allow secure data storage and transmission, provide expertise to and accredit institutions for quality data collection, and act as a central collection authority to aggregate information for its client special interest groups. These special interest groups in turn should then have a far greater capacity than currently to complete the quality assurance cycle by analysis and distribution of knowledge to the participating institutions.