Background: Echocardiography is one of the most important diagnostic imaging modalities in paediatric cardiology. Owing to the large number of lesions, achieving expertise often requires years of training. Echocardiography is still taught using the apprenticeship model, which is time- and personnel consuming.
Objectives: To extend the echocardiography simulator EchoCom to enable simulation of congenital heart lesions and validate it for training in paediatric echocardiography.
Methods: The simulator consists of a life-size manikin, a dummy transducer with attached three-dimensional (3D) tracking system and a computer application. Transthoracic real-time (RT) 3D echocardiographic datasets were collected and embedded into the simulator. Two-dimensional images were calculated and resliced from these datasets according to the position of the tracking sensor. Ten RT 3D datasets of congenital heart lesions were selected for validation. Datasets were blinded and without additional information presented to 43 participants who were stratified according to their expertise (12 experts, 16 intermediates, 15 beginners). Participants were asked to list the relevant findings and make a diagnosis. Construct validation was tested comparing diagnostic performance for each group. Face and content validation were tested using a standardised questionnaire.
Results: Participants judged the simulator as realistic and useful. The main drawback was the adult size of the manikin. The diagnostic performance of each group differed significantly proving construct validity.
Conclusions: According to this validation the prototype simulator could make a significant contribution to training in the use of echocardiography in congenital heart disease.
Statistics from Altmetric.com
Echocardiography is one of the most important diagnostic imaging modalities in paediatric cardiology, often the only imaging necessary before surgery.1–3 Therefore, it is important for every paediatric cardiologist to develop advanced skills in performing and interpreting echocardiograms.3 Owing to the large number of congenital heart lesions and their variability, one often needs years to achieve the necessary degree of proficiency. Some of the lesions are rare and therefore, occur only once in a while even in large centres. In addition, a substantial number of the neonates and infants present with critical lesions, needing urgent diagnostic and treatment and thus are not suitable for training purposes.4 5 On the other hand, hands-on training is extremely important to ensure correct identification of such lesions when they occur, especially in remote settings.
Echocardiography is a highly operator-dependent technique. In congenital heart disease, modification of standard views is almost always necessary, making scanning even more complex. For the beginner, it is often difficult to understand the relationship between transducer position, dynamic imaging with sweeps of the scanning plane and resulting images. Presentation of images or movies during courses therefore is insufficient. Expertise can only be achieved with hands-on training that traditionally uses the apprenticeship model of teaching.3 This is often hampered by the burden placed on experienced personnel and the trainees.
We have developed a training simulator for echocardiography based on a database of real-time (RT) three-dimensional (3D) echocardiographic datasets depicting a wide range of congenital cardiac lesions and integrated them into EchoCom, an application developed by us and previously described in detail.6 Previous studies have shown the benefit of our simulator in spatial understanding and hand–eye coordination during echocardiography.7 The development of RT 3D echocardiography resulted in a dramatic improvement of image quality.8 The purpose of this study was to examine, if expert sonographers could identify congenital heart lesions correctly using the simulator and perform better than less experienced sonographers, proving construct validity. Second, we wanted to investigate the realism and usefulness of our simulator judged by users of different expertise.
The simulator is based on our application EchoCom that has been previously described in detail.6 7 The simulator consists of a life-size manikin (fig 1), a dummy transducer with attached 3D tracking system (Flock of Birds, Ascension Technology Corp, Burlington, Vermont, USA) and a computer application.9 The computer application consists of a split screen (fig 2). The right-hand screen indicates the position of the dummy transducer and scan plane with respect to the heart. The left-hand screen shows the resulting echocardiographic image. This image is calculated and resliced from stored 3D echocardiographic datasets according to the position of the sensor of the tracking system attached to the dummy transducer. Since the 2D images are derived from 3D echocardiographic datasets in real time they are not restricted by prerecorded views. Scanning is possible from all transducer positions and is not restricted to standard planes. Thus the user can interactively explore the data as if scanning a real patient.
3D Echocardiographic datasets have been recorded using a Vivid 7 scanner with 3D RT capability (3V probe, GE Healthcare, Munich, Germany). Pyramidal-shaped datasets were acquired using a matrix array RT probe. The transducer had a frequency range of 2–4 MHz. A full volume dataset that covers the whole heart or at least the major region of interest is constructed from four ECG triggered subvolumes.8 Most often, the subcostal position was used for newborns and infants, the apical position for older patients. No sedation was used in any patient. No study was recorded for the purpose of integration into the simulator but was part of our routine examination. 3D Datasets were anonymised, stored and subsequently converted into an EchoCom-specific format by a custom-made tool (Localite, St Augustin, Germany). To achieve the correct position within the manikin we registered the echocardiographic data with the virtual heart model based on anatomical landmarks.10 Since these anatomical landmarks are defined for a normal heart, individual alterations had to be made for each dataset to match the lesion-specific conditions. Currently, only grey-scale data are used, since the 3D colour Doppler technique is too immature to be used for reconstruction.
Face validity describes the realism of a simulator, content validity describes the relevance of the simulator content for the real task. Face validity and content validity were tested based on a standardised questionnaire with statements based on a rating scale of 1 (not at all) to 5 (very much). Construct validity “characterises the extent to which groups with more experience … perform better on the simulator than groups with less real-life experience”.11
We tested construct validity using datasets from our database with congenital heart disease. Criteria for the datasets chosen were (a) image quality (only good quality datasets were chosen); (b) disease spectrum (from simple to complex lesions); (c) disease incidence (importance for clinical routine); (d) predominantly intracardiac pathology and pathologies of the initial part of the great vessels (since pathologies of the descending aorta or pulmonary arteries after branching were identifiable only rarely owing to the scanning mode and resolution). Before starting the evaluation each dataset was scanned by an experienced echocardiographer using the simulator who defined which findings could be identified and were relevant for a correct diagnosis. Owing to the lower resolution of the 3D probe and the need to obtain the datasets from one transducer position very small structures or structures far away from the transducer sometimes were invisible in the datasets. Therefore not all findings had to be noted, for the validation exercise to be graded as correct diagnosis. For each dataset a list of relevant findings was made that could and had to be identified to make a correct diagnosis. If probands identified all findings correctly, a performance grade 1 was given. If probands identified some, but not all relevant findings a performance grade 0.5 was given.
Datasets were presented in random order to probands after written informed consent, without any further information except the child’s age. No clinical information was given. After an instructional period of about 5 min, probands had 1 h to scan the datasets. Participants were divided into three groups according to their expertise (experts, intermediates and beginners). Following the recommendations of the American College of Cardiology/American Heart Association12 classification of expertise was based on the number of echocardiographic studies that had been performed in congenital heart disease, the number of studies done each week and the years of training/professional experience. Experts were defined as having done more than 1000 examinations, doing more than 30 a week and having >5 years of professional experience in paediatric cardiology. Intermediates were defined as having done between 100 and 1000 studies regardless of other parameters or a total number between 10 and 100, if more than 10 were currently performed a week or if training had lasted for at least 2 years. Beginners had either performed fewer than 10 examinations or between 10 and 100 if doing <10 a week or having <2 years of training as paediatric cardiologist or paediatric echocardiographer. During the evaluation no help by the supervisor was given except for support for simulator-related issues.
If the design of the simulator is correct, experts should be able to correctly identify the lesions and should perform better than beginners and intermediates. After completion of the study a debriefing sessions at each testing site was held to demonstrate the findings of each dataset, give correct diagnosis and discuss issues related to face and content validity. No individual results were made public to ensure that the results were not used for proband assessment. Statistical methods included analysis of variance on ranks (Kruskall-Wallis test) with a p value of <0.05 considered as significant.
A total of 12 experts, 16 intermediates and 15 beginners participated in the study—37 were doctors, six were technicians. Participants were from two heart centres in Germany and the UK respectively and one expert participant from the Netherlands. None had prior experience with our ultrasound simulator.
Based on the requirements, 10 datasets were chosen for the validation. The lesions and the relevant findings for each dataset are listed in table 1, an example is given in figs 2A–C for a complex case and in fig 2D for a simple case.
Face and content validity
Figure 3 depicts the results for face and content validity. The majority of participants found the simulator and the echocardiographic image realistic. Experts graded the simulator slightly more realistic than beginners and intermediates, but graded the echocardiographic image worse. All groups found the simulator useful for training and quality assurance, with the experts grading the simulator slightly better than beginners and intermediates. Differences were not statistically significant for any parameter.
Figure 4 depicts the results of the validation. Experts had a mean (SD) performance grade of 0.98 (0.1). That means experts diagnosed almost all datasets correctly. Intermediates and beginners had a mean value of 0.69 (0.38) and 0.44 (0.43), respectively. All groups differed significantly in their performance (p<0.05).
Expanding the scope of our echocardiography simulator EchoCom to congenital heart diseases is a logical progress emerging from the importance of echocardiography in paediatric cardiology and the progress of ultrasound technology, which now enables the recording of high-quality RT 3D data. Validation of a specific simulator is a crucial step before its implementation in existing training curricula or before assessment structures can be recommended.13–15 There are several forms of validation from the simple to the very complex.
The main purpose of this study was to test the construct validity of our simulator. Construct validity, often regarded as “the central theme in validation studies”16 is defined as “the degree to which the results of the ‘training session’ as performed by the trainee on the simulator reflect the actual skill of the trainee who is being assessed”.17 It is tested by comparing groups of different levels of expertise, with experts performing better on the specific task than beginners.11 18 In simulators for endoscopy, construct validity most often is tested using certain tasks like cutting or clipping of structures.16 These tasks are substitutes for the ultimate training goal of a surgeon competent in endoscopy. Although these tasks are relevant, it is clear that they are only components of an endoscopic procedure.
To test construct validity of our simulator, instead of using subtasks like transducer handling, orientation skills or achieved image quality, we chose accuracy of diagnosis as the major outcome measure. However, we accept that diagnostic accuracy is a spectrum. While doctors with basic skills in echocardiography should at least be able to determine major findings that can for example, distinguish between simple, complex or critical lesions, the expert should be able to define the exact diagnosis. Therefore we subdivided diagnostic performance into grade 0, 0.5 and 1. In contrast to reality, in the simulator a “case” consists of one dataset, recorded from one position. Given the technical limitations in 3D echocardiography it is currently impossible to have 3D datasets that include the same information as real-life echocardiography. We have therefore, defined in the beginning, what information could be gathered from the presented data. For construct validation, 10 datasets were presented to three groups of different expertise. Experts could identify the lesions correctly in almost all cases. There was a statistically significant difference in diagnostic performance between each group proving construct validity of our simulator.
Face validity can be defined as the extent to which the simulator “looks like what it is supposed to represent”.14 It is tested by asking users “to judge the degree of resemblance between the system under study and the real activity”.13 Although this validation is simple, it is important, since it is a major factor for the acceptance and motivation of trainees to use the simulator.16 Content validity answers the question, “Does the system measure the extent of knowledge that it is intended to measure—that is, does it contain the material that should be present for the training it intends to impart?”.14 It is tested “through subjective evaluation by experts”.11
Face and content validity of our simulator have been tested previously, although not specifically for paediatric cardiology.7 19 In these previous evaluations specific interest was focused on the value of the virtual scene to support beginners understanding of image orientation and the spatial heart–scan-plane–image relationship. In this study we added a condensed questionnaire to test face and content validity in regard to congenital heart diseases. On a score of 1–5 face and content validity of the simulator was judged very positively with an overall score between 3.9 and 4.5. Experts judged the realism of the simulator slightly better than intermediates and beginners. One reason for grading low by participants, often stated in the debriefing sessions, was that the manikin is of adult size. The datasets used, however, were predominantly of neonates and infants. Thus the range of transducer movements that in reality would be only minimal had to be adapted to the adult size manikin and were larger as would be expected. A second concern of participants was the surface properties of the simulator, which was harder and more slippery than human skin. During debriefing, it became clear that these problems were seen by all groups. The reason that experts gave a better grading for the realism of the simulator might be that experts were not distracted by these drawbacks. Experts scan more instinctively based primarily on the echocardiographic images and thus “neglected” the inappropriate size of the manikin or the surface properties. This mental flexibility seems to be as yet underdeveloped in beginners.
On the other hand, experts were more critical of the image quality. Since experts (as opposed to beginners) usually acquire high-quality images this may reflect different expectations. A drawback for participants in the UK was the inability of the simulator to visualise the echocardiographic image in an anatomical fashion instead of upside-down views as in most parts of continental Europe. Although this did not worry most experts, it was a problem for intermediates and beginners. Also lack of modification of the echocardiographic image like changing depth or brightness was noted.
To improve the simulator for congenital heart disease we have now developed manikins of infant and newborn size, redesigned the surface properties and implemented an option for displaying the image in an anatomical view (see online supplementary figures). The simulator will also benefit from progress in scanner technology. In this study we used datasets recorded by a 2–4 MHz probe. This transducer is not optimal for small children and neonates. However, at the time of the study, these were the only RT 3D data available to us that could be converted into the EchoCom-specific format. Availability of datasets recorded with higher frequencies and availability of high-quality 3D colour Doppler will be just a matter of time. Despite this room for improvement most probands think that the simulator is an effective tool for training and could accelerate learning.
The simulator was also regarded as useful for quality assurance. The participants in this study were working in paediatric cardiology. However, the results of this study also have an impact on adult echocardiography. Owing to advances in cardiology and cardiac surgery, patients with congenital heart defects now reach adulthood, even with complex malformations.20 The approach to these patients is interdisciplinary. The adult size of the manikin together with datasets of adult and adolescent patients (like cases 3, 6 and 10; table 1) may make the simulator a valuable tool for adult cardiologists with an interest in congenital heart disease. The simulator is already used for training anaesthetists in transoesophageal echocardiography.19 Although the datasets used in these courses are of normal structure, advances in scanner and simulator technology have made the simulator also more attractive for this group.
We carried out a PubMed search in February 2008 and found that our simulator currently is the only simulator for echocardiography that has been scientifically evaluated world wide. Thus direct comparison of different simulators for echocardiography is not possible. There are, however, simulators for abdominal and gynaecological ultrasound.21 22 These simulators differ from ours in several aspects. First, they are commercially available products, while ours is a prototype, not commercially available. Nevertheless, EchoCom has been used successfully for training courses in many European countries. Second, the aforementioned ultrasound simulators do not have a virtual scene for orientation purposes. Beside training, simulators could be used as an objective tool for assessment as suggested by Baier et al.23 Candidates might have to show their competence on simulators before accreditation. Currently, there are no instruments for formal testing of competence in paediatric echocardiography.3 It would be too early to recommend our simulator for assessment tests, but we strongly believe that it will be a valuable tool for this in the future.
We thank the technicians of the echocardiography laboratory at Great Ormond Street Hospital, residents and consultants at Great Ormond Street Hospital and Leipzig Heart Centre who participated in this study.
See Editorial p 613
▸ Additional figures are published online only at http://heart.bmj.com/content/vol95/issue8
Competing interests: None.
Ethics approval: Ethics Committee, University of Leipzig approval obtained.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.