Article Text
PDF
Abstract
Objective Left ventricular ejection fraction (LVEF) is measured to assess haemodynamic status and cardiac function. It may be difficult to accurately measure in patients with heart failure (HF) as they are often poorly echogenic. The augmented reality (AR) technology is expected to provide real-time guidance that will enable more accurate measurements.
Methods A prospective, randomised, case-crossover simulation study was conducted to confirm the effect of AR glasses on echocardiographic interpretation in patients with HF. 22 emergency physicians participated. The participants were randomly assigned to two groups. Group A estimated the visual ejection fraction of echocardiographic video clips without the AR glasses, while group B estimated them with glasses. After a washout period, the two groups crossed over. The estimates were then compared with the ejection fraction measurements obtained by echocardiologists; intraclass correlation coefficient (ICC) was calculated.
Results The ICC with glasses (0.969, 95% CI 0.966 to 0.971) was higher than without glasses (0.705, 95% CI 0.681 to 0.727) among all participants. In the subgroup analysis, the first-year and second-year residents showed the most significant difference, with an ICC of 0.568 (95% CI 0.508 to 0.621) without glasses compared with 0.963 (95% CI 0.958 to 0.968) with glasses. For the third-year and fourth-year residents group, the ICC was 0.754 (95% CI 0.720 to 0.784) without glasses and 0.972 (95% CI 0.958 to 0.968) with glasses. Among the group of attending physicians, the ICC was 0.807 (95% CI 0.775 to 0.834) without glasses and 0.973 (95% CI 0.969 to 0.977) with glasses.
Conclusions AR glasses could be helpful in measuring LVEF and could be more helpful to those with little visual estimation experience.
- Echocardiography
- Heart failure
Data availability statement
Data are available upon reasonable request. The data and the materials of this study are available from the corresponding author (SH; brayden0819@daum.net) or the Institutional Review Board of Soonchunhyang University Bucheon Hospital (20200817@schmc.ac.kr) upon reasonable request.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
It can be challenging to accurately measure ejection fraction (EF) in patients with heart failure (HF) due to their echocardiographic windows often being poorly echogenic.
WHAT THIS STUDY ADDS
We conducted a comparison between measured EF and visually estimated EF values in 22 emergency physicians, both with and without augmented reality (AR) glasses.
The intraclass correlation coefficient (ICC) for the group using AR glasses was 0.969, whereas it was 0.705 for the group without AR glasses.
The difference in ICC values was more noticeable in the group with less medical experience (first-year and second-year residents vs third-year and fourth-year residents vs attending physicians, with values of 0.395, 0.218 and 0.166, respectively).
Real-time guidance from the AR glasses resulted in more accurate interpretations when visually estimating EF compared with not using AR glasses.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
AR glasses may potentially assist individuals with limited expertise in visually estimating EF in patients with HF.
Introduction
Left ventricular ejection fraction (LVEF) is measured via transthoracic echocardiography (TTE) to assess haemodynamic status and cardiac function during the initial treatment of critically ill patients.1 LVEF is also used during or at the end of treatment to assess the response to therapy.2 An accurate measurement is important because the LVEF value is also used as an index for patient resuscitation, and inaccurate results could be harmful to patients.1 3 In particular, patients with heart failure (HF) require a more accurate evaluation because the mechanism, prognosis and treatment plan can vary depending on the LVEF value.4 However, it may be difficult to accurately measure the LVEF in patients with HF as their echocardiographic windows are often poorly echogenic.5
Simpson’s method, Teichholz method, fractional shortening of the left ventricular wall, mitral E-point septal separation method and three-dimensional echocardiography have been used to measure LVEF.6–9 However, these methods are time-consuming and tedious and require an expert operator because the methods are complex.7 10 Recently, a machine learning algorithm method has been reported to be accurate, reproducible and effective even in critically ill patients.11 However, more studies are needed on its application in emergency department settings. Therefore, the visual estimation (VE) method, which can be used to determine the LVEF relatively quickly, easily and accurately, is widely used in the emergency department.1 7 12 By visually estimating heart function, decisions can be made without referral to a trained cardiologist for standard echocardiography, which is time-saving and cost-effective.13 However, because VE reflects the subjective interpretation of the examiner and depends on experience and skill level, the results can be inaccurate.1 14 An effective training programme using video clips was suggested in a recent study. Still, in most cases, education and examination are conducted at the bedside without sufficient training or guidance.10 15 Thus, an effective education programme and objective guidance are needed.
The development of wearable devices and augmented reality (AR) technology is rapidly progressing and this been applied to medicine. These technologies are being used in clinical settings, such as for diagnosis, treatment and surgery, and AR technology is very effective and useful in medical care.16–19 It is expected that the use of AR technology, such as for real-time guidance when diagnosing patients, will enable more accurate and faster diagnosis and could be used as a bridge for junior doctors with a short training period. Therefore, in this study, we compared and analysed how accurately the ejection fraction (EF) value can be measured using real-time AR guidance with a wearable device.
Methods
Study design
We designed a prospective, randomised, case-crossover simulation study to confirm the effects of AR glasses on the interpretation of echocardiography of patients with HF.
Study participants
Residents and attending physicians who majored in emergency medicine were included in this study. Only participants who provided written informed consent were included and those who refused to participate were excluded. Recruitment notices were sent to three emergency medicine training hospitals. Six people refused to participate and finally 22 people voluntarily participated in the experiment. They consisted of 16 residents and 6 attending physicians who participated in this experiment from January to February 2023. The participants were classified into three subgroups according to their training level, namely first-year and second-year residents, third-year and fourth-year residents, and attending physicians (with more than 10 years of clinical experience). The participants were explained the purpose and process of the experiment, and written consent was obtained. The participants’ age, sex and previous echocardiography training were collected.
Study protocol
This study was performed using a case-crossover design. After watching an educational video, the participants were randomly assigned to two groups (group A and group B). Group A visually estimated the EF of patients with HF without AR glasses, and group B estimated EF with the help of AR glasses. After a 1-hour washout period, group A visually estimated the EF with AR glasses, and group B estimated the EF without AR glasses. Participants were able to know the correct answers of the test after the study was finished. The study process is summarised in figure 1.
Study flow chart. AR, augmented reality; HR, heart failure.
Preparing the test
The video clips used in the study were made by referencing the echocardiographic procedures of 60 patients with HF. In total, 180 video clips were extracted with parasternal short axis (papillary muscle level), parasternal long axis and apical four-chamber views. These video clips were composed of continuously moving images of two or more beats for continuous playback. Among them, 11 clips were excluded due to poor video quality. Two cardiologists with more than 10 years of clinical experience and were certified as trained echocardiologists by the Korean Society of Echocardiography evaluated the clips. The HF category (HF with preserved EF, HFpEF; HF with midrange EF, HFmrEF; HF with reduced EF, HFrEF) and the measured EF (mEF) were determined using the biplane Simpson’s method. The Teichholz method was used when Simpson’s method was too difficult to use due to poor video quality. The average mEF determined by the two cardiologists was considered as the actual value. If the difference between the mEF values of the two cardiologists exceeded ±5% or if the video clips had a borderline EF value for the category (eg, 40%±5% or 50%±5%), we considered that they could be evaluated in different categories depending on the evaluator, so that 25 video clips were excluded. Ultimately, 144 video clips were used. Among them, 30 were used for educational video production and 114 (38 parasternal short axis, 38 parasternal long axis and 38 apical four-chamber view) were included in the test and were inserted into Microsoft PowerPoint.
Educational video
Participants watched a 20 min educational video consisting of about 30 video clips of echocardiography from patients with HF before completing the test. The videos were prepared on PowerPoint slides with accompanying audio recordings, covering the concept of HF, the three categories of HF and the motion of the heart of patients with HF according to EF (parasternal short axis, parasternal long axis and apical four-chamber view). The educational video was created by cardiologists.
AR glasses
We used Nreal Air (Hangzhou Tairuo Technology, Beijing, China) AR glasses. These glasses have two spatial computing cameras with the red, green, and blue (RGB) and proximity sensors and connect to a smartphone to display data, such as photos or videos, on the lens using AR technology. In addition, the menu can be manipulated using the optical touchpad. After loading the reference file (Microsoft PowerPoint) containing the video clips into the smartphone, we connected it to the AR glasses. Only the Microsoft Office mobile app was used, and no additional software was required to use AR glasses. The reference file to be linked to the AR glasses contained EF 40% and 50% video clips from the educational video and consisted of three slides showing the parasternal short axis, parasternal long axis and apical four-chamber views. One slide included two different heart motions (EF 40% and 50%). The linked reference file was located on the right side of the user’s field of view in the AR glasses, and the slide was moved using the touchpad, so the participants referred to the motion of the heart according to the EF value while solving the test (online supplemental figure 1).
Supplemental material
Outcomes
Participants solved the test with and without the AR glasses and provided visually estimated EF (vEF) and HF categories. The scores were assessed for correct and incorrect answers. A correct answer received 1 point and an incorrect answer received 0 points. First, if the EF value determined by the participant was within ±5% of the actual value, it was considered correct; here, ±5% is not a ratio but the actual value of the EF. Second, if the three HF categories which were determined by the participants matched the categories determined by the cardiologists, they were considered correct. The EF criteria for HF were HFpEF (≥50%), HFmrEF (40%–49%) and HFrEF (<40%).20
Statistical analysis
Data were analysed using IBM SPSS Statistics (V.26.0) and R (V.3.5.3; The R Foundation for Statistical Computing, Vienna, Austria) software. Categorical variables are expressed as absolute numbers with percentages, and continuous variables are expressed as mean±SD. The performance of each group was evaluated based on mean percentage error (MPE), mean absolute percentage error (MAPE) and root mean square percentage error (RMSPE). MPE evaluates predictive bias, and MAPE and RMSPE evaluate predictive accuracy. In addition, the percentage of vEF within 10% and 20% of the mEF (PEF10 and PEF20) was calculated. The intraclass correlation coefficients (ICCs) of the mEF and the vEF values were calculated and classified as inadequate (<0.7), good (0.7–0.89) or excellent (≥0.9).21 Bland-Altman plots were prepared to evaluate the extent of agreement between the mEF and the vEF. The 95% limits of agreement (LOA) were defined as mean difference ±1.96 SD of mEF and vEF. Pearson’s correlation coefficient was used to evaluate the correlations between the mEF and the vEF. Additional analyses were performed according to the training level. Bangdiwala’s agreement chart was generated to compare the categories measured by the cardiologists (mCategory) and classified as visually estimated by the emergency physicians (vCategory). A p value <0.05 was considered significant.
Results
General characteristics of the participants
The general characteristics of the participants are summarised in table 1. The average age of the participants was 31.3 years old, and 17 (77.3%) men participated. Among them, 10 (45.5%) had received previous echocardiography training.
General characteristics of the study participants
Scores
The rates of correct answers according to the ±5% criterion and the three categorical criteria are shown in table 2 and figure 2. Also, the rates of correct answers according to training level are shown in online supplemental table 1. When evaluating the ±5% criterion, the rate of correct answers was 87.0% in the glasses group, higher than in those who did not use the glasses (44.5%; p<0.001). In addition, when evaluating the three HF categorical criteria, the rate was 89.4% in the glasses group, again higher than in the other group (50.6%; p<0.001). The scores and the performance before and after the washout time based on the flow chart in figure 1 can be found in online supplemental table 2. There was no evidence of a carry-over effect for the scores of ±5% and the three categories (p=0.353 and p=0.420, respectively). The tests on using AR glasses were significant for the scores of ±5% and the three categories (p<0.05, respectively), indicating better scores in the glasses group.
Scores and performance of the non-AR glasses and AR glasses groups
Scores without and with AR glasses: (A) the difference in the scores of each group when the correct answer was defined as within 5% and (B) the changes in the score if the quantitatively measured HF category was equal to the participant’s answer. The scores were assessed for correct and incorrect answers for the test (114 video clips). A correct answer received 1 point and an incorrect answer received 0 points. AR, augmented reality; EF, ejection fraction; HF, heart failure.
Performance
The performance and the correlations between the two groups are shown in table 2 and figure 3. Overall, the MAPE and RMSPE values were lower in the glasses group. The same was true for each training level, with the difference being larger in the first-year and second-year residents group. The LOA tended to be narrower in all glass-wearing subgroups and the ICC values were higher in the glasses group overall and in each subgroup.
Bland-Altman plots of vEF and mEF according to the training level and 95% limits of agreement. AR, augmented reality; mEF, measured ejection fraction; vEF, visually estimated ejection fraction.
The ICC values of mEF and vEF according to the HF category are shown in online supplemental figure 2 and online supplemental table 3. The ICC values of all categories increased in the glasses group, being largest in HFmrEF (ICC of HFmrEF: without AR glasses 0.143 (95% CI 0.023 to 0.249, p<0.001) vs with AR glasses 0.710 (95% CI 0.670 to 0.746, p<0.001)).
Correlation between the mEF of cardiologists and the vEF of emergency physicians
A correlation analysis was conducted to determine the correlations between the vEF of emergency physicians and the mEF of cardiologists (figure 4). The correlation between the vEF and mEF values was better in the glasses group (R2=0.55 vs R2=0.94). Similar trends were found at each training level.
Scatterplot demonstrating the correlation between the mEF (x-axis) and the vEF (y-axis). Each linear regression equation is located at the top left of the scatterplot. AR, augmented reality; mEF, measured ejection fraction; R2, coefficient of determination; vEF, visually estimated ejection fraction.
Accuracy of visual categorisation of HF
Tables 3 and 4 and online supplemental figure 3 show the distribution of HF categories determined by cardiologists and emergency physicians. The accuracies of the HF category were 93.7%, 82% and 92.8% for HFpEF, HFmrEF and HFrEF in the glasses group and 54%, 45.8% and 52.1% in the group without AR, respectively.
Percentage accuracy of visual categorisation of HF into HFpEF, HFmrEF and HFrEF compared with measuring the HF category without AR glasses
Percentage accuracy of visual categorisation of HF into HFpEF, HFmrEF and HFrEF compared with measuring the HF category with AR glasses
Discussion
In this study, we confirmed that EF can be measured more accurately using an AR wearable device. The efficacy was higher for first-year and second-year residents who were unskilled in VE, and more accurate EF values were measured in all EF categories. Various attempts have been made to apply AR technology to medicine; however, to the best of our knowledge, this is the first to analyse EF measurements and their efficacy using AR technology.
Haemodynamic status and cardiac function must be immediately evaluated in critically ill patients. Therefore, emergency physicians prefer the VE method, which can be used with TTE to check cardiac function quickly; the efficacy of VE has also been proven.7 12 Accurate measurements of LVEF are important because an improper evaluation of EF can have a fatal effect on the patient’s diagnosis, treatment and prognosis.1 3 4 In this study, the group that used AR glasses showed better results in both predictive bias and predictive accuracy for the LVEF measurements than the group that did not (table 2). In previous studies, the VE method was trained by hands-on teaching, feedback or pre-education using a video clip sample.10 22 23 However, in this study, AR glasses and pre-education were used as real-time guidance. This way, guidance information was provided to the participants through a wearable device and was directly compared and reviewed during the examination. The participants measured LVEF more consistently and accurately using the glasses, which compensated for the disadvantages of VE and were highly efficient for EF measurements. This means that providing information in real time using AR technology is effective in the diagnosis and treatment of patients, as reported previously.17
Third-year and fourth-year residents and attending physicians estimated the LVEF values more accurately than the first-year and second-year residents (table 2). Experience and proficiency in the VE method are important when interpreting results.14 In addition, error values, LOA, PEF10 and PEF20 improved when the AR glasses were used in all groups compared with when the glasses were not used. In particular, the ICC value was ≥0.9 in all groups when the AR glasses were used. The improvement was significant in the first-year and second-year residents group, indicating that real-time guidance using AR glasses is more effective for those with less clinical experience. The lack of experience can be effectively overcome using AR glasses, and AR technology can be used as a bridge.
Previous studies have reported that the VE method is highly accurate.7 24 Particularly, LVEF can be accurately measured when it is normal or markedly lower.7 25 On the other hand, accurately measuring EF in the case of mildly reduced LVEF can be challenging.26 In this study, HFpEF and HFrEF had accuracies of >50% when AR glasses were not used, whereas HFmrEF did not. We assumed that accuracy was lower in this study than in previous studies due to the poor echogenicity of patients with HF.26 On the other hand, when AR glasses were used, HFpEF and HFrEF had more than 90% accuracy and HFmrEF had more than 80% (Tables 3 and 4). In addition, the predictive accuracy of the group that used AR glasses was ‘good’, and the difference was remarkable in the HFmrEF group that did not use the glasses (online supplemental table 3), suggesting that the use of glasses helped measure LVEF more accurately in those with a mildly reduced EF. HFmrEF lies in the interval between HFrEF and HFpEF and has intermediate features, but according to recent studies it is similar to HFrEF in terms of treatment and prognosis and so its importance should not be overlooked.20 Therefore, real-time guidance using AR technology is expected to be useful in the diagnosis and treatment of patients by more sensitively classifying and measuring changes in EF.
AR technology has been applied and used effectively in various fields, such as in medicine, education, science and industry.17 27 In particular, it has been applied for examinations, treatments, surgeries and procedures and has been reported to be a very useful tool.19 28 29 Also, the application of AR technology to echocardiography helps to understand anatomical structure or physiology through the interaction between actual data and virtual data.30 Similarly, in this study, sophisticated interpretation of heart function and EF has become possible because participants can examine the patient’s echocardiogram in real time by overlying the reference data using the AR glass. We confirmed that AR technology could be effective in measuring EF using the VE method in patients with HF.
However, some limitations of this study should be discussed. First, we targeted emergency medicine physicians. Therefore, applying the results to other specialists is limited. Second, the participants did not use an ultrasound probe. Handling the probe is crucial to ultrasound inspection, so additional comparative research on probe handling while wearing AR glasses is needed. Third, even though the participants did not know the correct answer for the test and had sufficient washout time, seeing the same video clips twice might cause potential bias in study results. Fourth, only ultrasound images of patients with HF were measured and compared. Further study is needed comparing the EF measurements in healthy people and patients with other underlying diseases. Fifth, it could be cost-consuming to purchase AR glasses. Simply comparing reference examples on a monitor screen could improve interpretation. AR glasses can overlie the reference examples over the patient’s echocardiogram view in real time and this could be helpful for more precise interpretation. This can be said to be the strength of AR glasses. Further study is needed to compare the efficacy between AR glasses and reference examples on a screen. Therefore, a multicentre, large-sample study is needed to generalise our results to experts in other fields and various patient groups.
Conclusion
AR glasses could be helpful in measuring LVEF using VE in patients with HF. In particular, in this study, the efficacy was better for inexperienced physicians who were unskilled in VE. Moreover, it could be effective in measuring accurate EF values in the HFpEF, HFrEF and HFmrEF categories. AR glasses could be effectively used in patients’ diagnosis, treatment and prognosis.
Supplemental material
Data availability statement
Data are available upon reasonable request. The data and the materials of this study are available from the corresponding author (SH; brayden0819@daum.net) or the Institutional Review Board of Soonchunhyang University Bucheon Hospital (20200817@schmc.ac.kr) upon reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and the study protocol was approved by the Institutional Review Board of Soonchunhyang University Bucheon Hospital (IRB number: 2022-09-007). Participants gave informed consent to participate in the study before taking part.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @rokstone
SC, SN and SH contributed equally.
Contributors SH had full access to all the data and takes responsibility for the integrity of the data and the accuracy of the data analysis. Concept and design: SC, SN, SH. Acquisition and analysis: SC, YSC, IM, JWL, CAL. Interpretation of data: SN, JEM, SH. Drafting of the manuscript: SC, SN, SH. Critical revision of the manuscript for important intellectual content: SC, IM, SH. Statistical analysis: SN, JEM. Obtaining study funding: SH. Supervision: SH. Guarantor for the manuscript: SH.
Funding This work was supported by the Soonchunhyang University Research Fund (grant number 2023-0057). This research was funded by the Society of Emergency and Critical Care Imaging, Republic of Korea (grant number 202201). The funders had no role in the design and conduct of the study.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.