Inter- and intra-observer variability of radial-endobronchial ultrasound image interpretation for peripheral pulmonary lesions

Introduction

Pulmonologists have an increasing role in the diagnosis and treatment of lung cancer, with bronchoscopy serving as the most essential tool for the proceduralist in this multi-disciplinary field (1). Historically, the diagnostic yield with conventional bronchoscopy for peripheral pulmonary lesions was as low as 20–30%. For this reason, advanced bronchoscopic techniques, such as radial probe endobronchial ultrasound (R-EBUS), have been developed in an attempt to improve diagnostic yield and are now recommended as an adjunct for the diagnosis of peripheral pulmonary lesions (2,3).

R-EBUS uses a flexible catheter with a rotating ultrasound transducer to produce a 360-degree ultrasound image. This provides real-time confirmation of lesion localization with bronchoscopy for peripheral pulmonary lesions. Following identification of the nodule, the radial probe is removed, and biopsy instruments are passed through the bronchoscope into the same segmental bronchus (4). Lesions are described as concentric when obtained images show the radial probe within and surrounded by the lesion. A lesion is described as eccentric when the images obtained show the probe towards one side of the lesion, suggesting the lesion is adjacent to the airway (Figure 1A,1B).

Figure 1 R-EBUS image and corresponding fluoroscopic view demonstrating (A) eccentric lesion suggesting lesion is adjacent to the airway, (B) concentric image suggesting lesions is within the airway. R-EBUS, radial probe endobronchial ultrasound.

R-EBUS probe positioning, and consequently image interpretation, is essential as the position of the probe relative to the lesion may predict the likelihood of obtaining a diagnosis. In a prior study, the diagnostic yield of a concentric lesion was 84%, while the yield of an eccentric lesion was 48% (5). Additional meta-analyses have shown significant differences in diagnostic yield between concentric and eccentric lesions (6,7).

While significant differences in diagnostic yield have been reported for concentric and eccentric R-EBUS images, ultrasound interpretation is subjective and may be prone to interobserver variability.Currently, there is no standardized process for classifying R-EBUS images as concentric or eccentric, and therefore, interpretation of the literature and generalizability of published findings may be challenging. This study was performed to assess inter- and intra-observer variability in the interpretation of R-EBUS images for use in peripheral pulmonary lesions. We present this article in accordance with the SURGE reporting checklist (available at https://jtd.amegroups.com/article/view/10.21037/jtd-23-998/rc).

Methods

This was a survey-based study using R-EBUS images from 100 peripheral bronchoscopy cases at Barnes-Jewish Hospital in St. Louis. For this study, 100 consecutive de-identified cases with available computed tomography (CT) scan imaging, radial EBUS images, a finalized cytopathology report, and at least 1 year of follow-up data were identified and included. The digital images from the procedure report of the included cases were collected into a survey. Figure 1A,1B are representative images included in the survey. The survey was sent to 10 fellowship trained interventional pulmonologists who had each performed over 100 cases utilizing peripheral bronchoscopy with R-EBUS. All subjects previously received training at the same institution for fellowship. Participants were contacted via institutional email and were provided with R-EBUS images on a Microsoft Excel document. They were asked to grade the images as concentric, eccentric, or no Image based on their expert interpretation and prior training. In order to accurately assess subjectivity of radial EBUS interpretation and due to the advanced training each participant received, participants were provided with no suggestions for definitions of any category of R-EBUS image. After 3 months, the same survey was repeated by the participants. Participants were instructed not to review their prior survey for the integrity of the study. Responses from the 10 participants to the first survey were evaluated for inter-observer variability between participants and a participant’s response to the first and second survey were evaluated for intra-observer variability amongst each participant. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). Ethical approval and informed consent were not required for this study in accordance with local/national guidelines of the ethics committee of Washington University in St. Louis School of Medicine.

Statistical analysis

Inter- and intra-observer variability were calculated utilizing SAS Version 9.4 for Windows. Cohen’s Kappa-coefficient was calculated for variability for both Survey 1 and Survey 2. Traditionally, a kappa result of 0 is the agreement that would occur by random chance. A kappa result between 0.01–0.2 has none to slight agreement, 0.21–0.4 fair agreement, 0.41–0.60 as moderate agreement, 0.61–0.80 as substantial agreement, and 0.81–1.0 as almost perfect agreement. In clinical practice, the minimal acceptable agreement has often been set at 0.8 (8).

Results

Survey participation

There was 100% participation among the physicians surveyed. All participants completed both surveys.

Inter-observer variability

In Survey 1, 363/1,000 (36.3%) of all images were rated as concentric, 506/1,000 (50.6%) were rated as eccentric, and 131/1,000 (13.1%) as no image. The kappa-statistic for inter-observer variability was 0.512 for concentric lesions, 0.527 for eccentric lesions, and 0.378 for no image with an overall kappa-statistic of 0.496 (Figure 2A).

Figure 2 Survey responses and corresponding k-statistic. (A) Inter-observer variability for Survey 1; (B) inter-observer variability for Survey 2; (C) intra-observer variability between Surveys 1 and 2.

In Survey 2, 386/1,000 (38.6%) of images were rated as concentric, 482/1,000 (48.2%) as eccentric, and 132/1,000 (13.2%) as no image. For Survey 2, the kappa-statistic was 0.495 for concentric lesions, 0.496 for eccentric lesions, and 0.395 for images rates as having no view. The overall kappa-statistic was 0.477 (Figure 2B).

Intra-observer variability

Of the 363 images rated as concentric in Survey 1, 326 (90%) were rated the same in Survey 2, with 32 (9%) subsequently rated as eccentric, and 5 (1%) changed to no image.

Out of 506 images initially rated as eccentric, 443 (88%) were again rated as eccentric with 49 (10%) switched to concentric, and 14 (2%) rated as no image.

Of 131 images, 113 (86%) were consistently rated as no image on both surveys, while 11 (9%) were subsequently rated as concentric, and 7 (5%) rated as eccentric on Survey 2. The overall kappa-statistic for intra-observer variability was 0.803 (Figure 2C).

Discussion

R-EBUS is a society endorsed tool to assist with the diagnosis of peripheral lung lesions (3). In addition to confirming lesion localization prior to performing biopsy, investigations have demonstrated that this tool can provide real-time, intraprocedural prognostic information regarding the likelihood of obtaining diagnostic biopsies depending on whether the ultrasound image is concentric or eccentric. If an eccentric image is obtained, the bronchoscopist has the opportunity to attempt repositioning of the probe to see if a concentric image may be obtained, and if successful, the yield is expected to be significantly improved. Unfortunately, variability in R-EBUS image interpretation may adversely affect the bronchoscopist’s ability to most accurately utilize the feedback that is being provided, thereby leading to suboptimal results. As our field gains experience with newer technologies such as advanced navigational platforms and robotic procedures, it becomes increasingly important to utilize these approaches in the most informed and standardized way.

To our knowledge, this is the first study evaluating variability amongst pulmonologists in interpreting R-EBUS images for peripheral pulmonary lesions. In this study, experienced bronchoscopists were presented with multiple radial EBUS images and were asked to classify them as either concentric, eccentric, or no image based on their prior training and without suggestion or formal definition for interpretation. There was significant variability in responses, and the kappa coefficient suggested poor agreement between the participants. However, when each participant was presented with the same set of images at a later date, there was strong agreement between responses from each participant between Surveys 1 and 2. The results of the survey suggest that each pulmonologist utilizes an intrinsic framework for evaluating R-EBUS images that allows for internal consistency, however there is a variability between physicians that is not desirable in clinical practice (8). Applying this clinically, because probe positioning within the lesion has been shown to improve diagnostic yield, some pulmonologists may be missing an opportunity to improve yield with repositioning if lesions are judged to be concentric, but are actually off-center.

R-EBUS has been investigated independently and in combination with other navigational bronchoscopic techniques such as electromagnetic navigation (EMN) and robotics for peripheral lesion localization and diagnosis (9,10). While these methods have been developed to improve diagnostic yield, there have been inconsistent results in the literature. When comparing standard bronchoscopy with thin bronchoscopy and R-EBUS, Tanner and colleagues found no significant difference in the techniques with a diagnostic yield of 49% when utilizing R-EBUS (11). Similarly, in the Acquire registry, diagnostic yield of R-EBUS alone was 57% and 47.1% with EMN and R-EBUS combined (12). This contrasts with a meta-analysis of guided bronchoscopy including over 3,000 patients which showed a pooled diagnostic yield with R-EBUS of 70.1% (13).

Although there are methodologic differences between these studies, differences in R-EBUS image interpretation may explain some discrepancy in diagnostic yield, as the overall yield is expected to be largely influenced by the relative proportion of concentric and eccentric lesions in the sampled population. Some lesions may be rated as concentric by pulmonologists if the probe is completely surrounded by the lesion, but rated as eccentric by others if the probe is only slightly off center. Studies with a higher percentage of lesions found to be concentric would be expected to have a higher diagnostic yield based on previously published literature (5-7). While some of these publications include a breakdown of concentric and eccentric images, interobserver variability in classifying concentric and eccentric images may diminish the ability to meaningfully compare one study to another. This speaks to the necessity of standardization for R-EBUS images, as there is little consensus on what imaging characteristics are consistent with each pattern.

Concentricity and accurate assessment of radial probe positioning related to peripheral lesions is one of the first and foundational steps in peripheral bronchoscopy as new studies show that radial probe ultrasound may also be helpful as a clinical prediction tool. Studies have shown significant differences in ultrasound characteristics between benign and malignant lesions such as echogenicity, size, shape, margin, and greyscale features (14,15). Another model utilizing radiographic, laboratory, and R-EBUS image findings was able to predict malignancy with accuracy up to 87.7% (16). As the field of radiomics improves, bronchoscopists should be optimizing their image to not just improve diagnostic yield, but also gain clinically predictive characteristics in the event of non-diagnostic biopsies to inform decisions about next steps in care such as referral to radiation oncology, surgery, or continued observation. However, as the data from this survey shows, image interpretation is highly variable even in the most basic of radial-EBUS image characteristics, which may affect downstream and more advanced ultrasound qualities. Proficiency and standardization is needed as we seek to advance this field.

It is unclear as to the factors leading to such variability in R-EBUS interpretation. Data regarding lesion size or location were not included in the analysis, however this may theoretically play a role in the quality and concentricity of ultrasound imaging, for example, lower lobe lesions affected by atelectasis. Further, as no formal definition exists, there may be some association of concentricity and eccentricity with symmetric and asymmetric images. As an example, an asymmetric image may still show the probe entirely within the lesion, however be interpreted as concentric or eccentric by two different pulmonologists. Limitations of the study include the use of still images for interpretation and possibly image quality. Notably, this may have had an effect in the interpretation of those images graded as “no image”. While the proceduralist would have known if a lesion was found, the survey subject did not. Further, a very large lesion with a wide concentric view with no identifiable borders could have been misinterpreted as “no image”. In general, the images that were captured during the procedure and included in the survey represent the most optimal R-EBUS image obtained. Additionally, respondents in this study were all fellowship trained Interventional Pulmonologists with considerable experience with peripheral bronchoscopy using R-EBUS. Therefore, the generalizability of findings may be questionable. Despite the inter-observer variability shown here, each participant was internally consistent with their image interpretation across the two surveys. This internal consistency may be explained by experience and the repetitive nature of cases in clinical practice. While the sample size may be somewhat limited, the study was pragmatic by including consecutive R-EBUS cases and the sample size of 100 cases and average lesion size were consistent with prior studies evaluating characteristics of R-EBUS. Additionally, all physicians surveyed were IP-fellowship trained and all trained from the same program, thereby limiting differences in training bias that may affect image interpretation.

Conclusions

As new technology in the pulmonary space evolves, it is increasingly important to understand how tools are being utilized across the space. This study demonstrates that while physicians are internally consistent with classifying images as either concentric or eccentric, there is suboptimal agreement between physicians regarding concentric or eccentric patterns, which may impact the way that this tool is utilized in clinical practice. A standardized approach and grading system for radial EBUS patterns may improve inter-observer variability and therefore the ability to standardize our review of clinical and research efforts to effectively use this technology.

Acknowledgments

Special acknowledgments go to Jennifer Ammons, RRT and Jim Sioumcas, RRT who provided respiratory and patient care during all procedures. Additional acknowledgment goes to Jack Baty who provided assistance with the statistical analysis for the manuscript.

Funding: None.

Footnote

Reporting Checklist: The authors have completed the SURGE reporting checklist. Available at https://jtd.amegroups.com/article/view/10.21037/jtd-23-998/rc

Data Sharing Statement: Available at https://jtd.amegroups.com/article/view/10.21037/jtd-23-998/dss

Peer Review File: Available at https://jtd.amegroups.com/article/view/10.21037/jtd-23-998/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://jtd.amegroups.com/article/view/10.21037/jtd-23-998/coif). A.C.C. has received research and consulting funding from Olympus, Boston Scientific, Johnson and Johnson and Intuitive Surgical. A.S. is a consultant for Olympus, which manufactures the radial EBUS device. S.P. is a consultant for Ambu Inc. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013).Ethical approval and informed consent were not required for this study in accordance with local/national guidelines of the ethics committee of Washington University in St. Louis School of Medicine.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Gaga M, Powell CA, Schraufnagel DE, et al. An official American Thoracic Society/European Respiratory Society statement: the role of the pulmonologist in the diagnosis and management of lung cancer. Am J Respir Crit Care Med 2013;188:503-7. [Crossref] [PubMed]
2. Paone G, Nicastri E, Lucantoni G, et al. Endobronchial ultrasound-driven biopsy in the diagnosis of peripheral lung lesions. Chest 2005;128:3551-7. [Crossref] [PubMed]
3. Rivera MP, Mehta AC, Wahidi MM. Establishing the diagnosis of lung cancer: Diagnosis and management of lung cancer, 3rd ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest 2013;143:e142S-e165S.
4. Chenna P, Chen AC. Radial probe endobronchial ultrasound and novel navigation biopsy techniques. Semin Respir Crit Care Med 2014;35:645-54. [Crossref] [PubMed]
5. Chen A, Chenna P, Loiselle A, et al. Radial probe endobronchial ultrasound for peripheral pulmonary lesions. A 5-year institutional experience. Ann Am Thorac Soc 2014;11:578-82. [Crossref] [PubMed]
6. Ali MS, Trick W, Mba BI, et al. Radial endobronchial ultrasound for the diagnosis of peripheral pulmonary lesions: A systematic review and meta-analysis. Respirology 2017;22:443-53. [Crossref] [PubMed]
7. Steinfort DP, Khor YH, Manser RL, et al. Radial probe endobronchial ultrasound for the diagnosis of peripheral lung cancer: systematic review and meta-analysis. Eur Respir J 2011;37:902-10. [Crossref] [PubMed]
8. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012;22:276-82.
9. Chen AC, Pastis NJ Jr, Mahajan AK, et al. Robotic Bronchoscopy for Peripheral Pulmonary Lesions: A Multicenter Pilot and Feasibility Study (BENEFIT). Chest 2021;159:845-52. [Crossref] [PubMed]
10. Chen CH, Cheng WC, Wu BR, et al. Improved diagnostic yield of bronchoscopy in peripheral pulmonary lesions: combination of radial probe endobronchial ultrasound and rapid on-site evaluation. J Thorac Dis 2015;7:S418-25. [Crossref] [PubMed]
11. Tanner NT, Yarmus L, Chen A, et al. Standard Bronchoscopy With Fluoroscopy vs Thin Bronchoscopy and Radial Endobronchial Ultrasound for Biopsy of Pulmonary Lesions: A Multicenter, Prospective, Randomized Trial. Chest 2018;154:1035-43. [Crossref] [PubMed]
12. Ost DE, Ernst A, Lei X, et al. Diagnostic Yield and Complications of Bronchoscopy for Peripheral Lung Lesions. Results of the AQuIRE Registry. Am J Respir Crit Care Med 2016;193:68-77. [Crossref] [PubMed]
13. Wang Memoli JS, Nietert PJ, Silvestri GA. Meta-analysis of guided bronchoscopy for the evaluation of the pulmonary nodule. Chest 2012;142:385-93. [Crossref] [PubMed]
14. Zheng X, Wang L, Chen J, et al. Diagnostic value of radial endobronchial ultrasonographic features in predominant solid peripheral pulmonary lesions. J Thorac Dis 2020;12:7656-65. [Crossref] [PubMed]
15. Badiei A, Nguyen P, Jersmann H, et al. Radial Endobronchial Ultrasound Greyscale Texture Analysis Using Whole-Lesion Analysis Can Characterise Benign and Malignant Lesions without Region-of-Interest Selection Bias. Respiration 2019;97:78-83. [Crossref] [PubMed]
16. Choi J, Zo S, Kim JH, et al. Nondiagnostic, radial-probe endobronchial ultrasound-guided biopsy for peripheral lung lesions: The added value of radiomics from ultrasound imaging for predicting malignancy. Thorac Cancer 2023;14:177-85. [Crossref] [PubMed]