Pleural invasion of peripheral cT1 lung cancer by deep learning analysis of thoracoscopic images: a retrospective pilot study

Introduction

Pathological pleural invasion (pPL) in non-small cell lung cancer (NSCLC) has a prognostic impact, with pT2 classifier even when the size is ≤3 cm (1,2). This is determined by postoperative pathology evaluation including elastic stain (3), and the accuracy of intraoperative visual estimation of pPL is reported to be 62.8% (4). Several studies examining preoperative and intraoperative methods to predict visceral pPL provided limited results (4,5). It is noted that pPL may be present even when the tumor does not touch the pleural surface on preoperative computed tomography (CT) (6). However, this difficulty did not affect the clinical practice as lobectomy was a standard surgical treatment for fit patients with NSCLC regardless of the tumor size or the presence of pPL.

Recently, two large prospective randomized controlled trials of surgical treatment of small (cT1 ≤2 cm) peripheral NSCLC, JCOG0802/WJOG4607L (lobectomy vs. segmentectomy) (7) and CALGB/Alliance140503 (lobectomy vs. segmentectomy or wedge resection) (8), showed that sublobar resection yields long-term oncological outcomes non-inferior to that of lobar resection with pulmonary function benefit. Thus, sublobar resection is becoming a standard of care in this selected population. Also, retrospective studies have reported similar results between segmental resection and lobectomy for T1c (2–3 cm) (9,10). Criteria for sublobar resection may be extended to the size of 3 cm or smaller.

Although it was not investigated in subgroup analysis in the two randomized trials, retrospective study demonstrated that pPL is a risk factor for locoregional recurrence during segmentectomy (11). Theoretically, if pPL can be properly assessed intraoperatively (ideally even without frozen section diagnosis), converting to lobectomy may reduce the risk of local recurrence associated with sublobar resection. Existing reports include attempts of real time evaluation of pPL using autofluorescence combined with internal thoracoscopic 5-ALA (12), and pleural touch cytology (13), but a simpler and more accurate system is desired. Recently, improved image recognition of artificial intelligence (AI) has been demonstrated in the diagnosis of gastric cancer by endoscopy (14), prediction of gastric cancer depth (15), and differentiation of skin diseases (16). Thus, we hypothesized that AI could predict pPL by learning from high definition thoracoscopic images of direct observation of pleural surfaces.

Our ultimate goal is to develop a real-time diagnostic support system for pathological visceral pPL of lung cancer using AI that has been trained on images of video-assisted thoracoscopic surgery (VATS). As an initial step, we conducted a retrospective pilot study to develop a deep learning algorithm for the prediction of pPL using images already collected with VATS, and to evaluate the feasibility of the concept. We present this article in accordance with the STROBE reporting checklist (available at https://jtd.amegroups.com/article/view/10.21037/jtd-24-1510/rc).

Methods

Patient selection

The institutional review board at the Cancer Institute Hospital approved the study on Jan 25, 2022 (No. 2021-GB-107) and informed consent was waived due to its retrospective nature of research. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments.

Among consecutive patients who underwent curative-intent VATS pulmonary resection for cT1 (≤3 cm) N0M0 NSCLC (TNM version 8th) from 5/2020 to 3/2022, patients with available video that showed visceral pleural surface changes due to tumor on thoracoscopic images were included. Cases with cTis/T1mi and peritumoral adhesions during surgery were excluded because these were already shown to be cured with sublobar resection with enough margin in a prospective study (JCOG0804) (17).

Preoperative examination included physical examinations, pulmonary function test, cardiac assessment, (head, chest and abdominal) CT scan, and 2-deoxy-2-[18F] fluoro-D-glucose positron emission tomography (FDG-PET). Sublobar resection was selected when a patient is not fit for lobectomy in terms of cardiopulmonary reserve or other comorbidities. The degree of pPL (pPL0, no pPL; pPL 1, invasion beyond the elastic layer of the visceral pleura; pPL2, invasion to the visceral pleural surface) was assessed based on permanent pathology by a board-certified pathologist specializing in lung cancer (H.N.).

All surgery was performed by board certified thoracic surgeons in our center which is a major tertiary cancer center in Japan. The surgical pPL score was prospectively assessed intraoperatively at the beginning of pulmonary resection as determined by the Japan Lung Cancer Society. Recorded videos were retrospectively collected and used for the deep learning analysis.

Model building

VATS pulmonary resection was performed through 3 or 4 ports using a 5 or 10 mm thoracoscope (Olympus Corporation, Tokyo, Japan or Karl Stolz SE & Co. KG, Tuttlingen, Germany) or the thoracoscope of a surgical robot (Intuitive Surgical, Sunnyvale, CA, USA). Video files were pooled in our database as MP4 files. A coordinating researcher (K.H.) annotated the video with timestamps, corresponding to the time when the tumor was recognizable in the video.

Video files were broken down into still images, with 1 image per second when a tumor wasn’t visible and 2 images per second when it was visible, due to the large discrepancy in tumor-visible versus non-visible times. The resulting .jpg files were then used to train and evaluate two deep learning models with a ResNet50 architecture.

The study design and algorithm construction are shown in Figure 1. The first model, a tumor recognition model, was trained using all images in which the tumor was visible and a randomly selected subset of images in which the tumor was not visible. This subset of images in which the tumor is visible, contained twice as many images as the tumor-visible images. The second model, a pPL prediction model, was trained using all pPL− (pPL0) and pPL+ (pPL1 or pPL2) tumor-visible images.

Figure 1 Study design and construction of algorithms. AI, artificial intelligence; NSCLC, non-small cell lung cancer.

All images were resized and normalized before being passed onto the models. All training images were also augmented by randomly flipping images horizontally, vertically and/or rotating them by a maximum of 45 degrees and black borders were removed from images that were used for the pPL prediction model to improve accuracy. Both models were trained using 5-fold cross-validation on the training dataset (64 patients) and evaluated on the test dataset (16 patients) using PyTorch v1.11.0, with optimization of hyperparameters such as batch size, learning rate and weight decay, using Optuna v2.10.1.

Assessment of the models and statistical analyses

Image-level predictions made by the pPL prediction model were converted to patient-level predictions by taking the majority vote of all image-level predictions in each patient. Comparison of the clinical characteristics was performed for continuous and categorical variables using t-test or Mann-Whitney test, and Fisher’s exact test, respectively.

Accuracy, recall, precision, specificity, and F1 scores were calculated to assess model performances. Gradient-weighted class activation mapping (Grad-CAM), implemented using pytorch-grad-cam v.1.4.0, was used to visualize the regions in the images most important for the classification models.

The predictive ability of the pPL prediction model was compared to the surgeon’s intraoperative evaluation of pPL using McNemar’s test. These statistical analyses were performed in RStudio 2022.07.1 build 554 (Posit, Boston, MA, USA) using R v4.1.1 and the R package rstatix v0.7.0, and GraphPad Prism version 8 (GraphPad Software Inc., La Jolla, CA, USA). All P values were two-sided, and P<0.05 was considered statistically significant.

Results

Clinical characteristics of patients

Characteristics of study participants are shown in Table 1. Sublobar resection (36.3%) was chosen due to poor cardiopulmonary reserve or other comorbidities. Among the 80 patients (age 69±10 years, 42.5% female, tumor diameter 20±7 mm), pPL1 and pPL2 were found in 18 and 4 patients, respectively. Compared to the pPL− group (N=58), the pPL+ group (N=22) was significantly older, with tumors having a larger solid diameter and more pure solid nodules, and had higher maximum standardized uptake values (SUVmax) on PET-CT. In the resected specimen, pPL+ group was associated with nodal metastasis.

Tumor recognition model

Among 422,873 still images extracted from all 80 patients’ video files, 2,074 still images were noted to show tumors, as assessed by surgeons. Because of the large class imbalance, a total of 4,148 out of 420,799 images (twice the number of images that show tumors) that didn’t show tumors were randomly selected, creating the tumor recognition dataset with a 1:2 ratio of tumors to no-tumors (Table 2).

These images were used to fine-tune a pretrained ResNet50 model using 5-fold cross-validation. Each of the models of the 5-fold cross-validation was evaluated on the same test dataset, with the final score being the median score across all 5 folds. The total numbers of predictions made by all 5 models are shown in Table 3. The final accuracy, precision, recall, specificity and F-value of this tumor recognition model were 77.7%, 78.7%, 48.8%, 93.6% and 59.8%, respectively.

pPL prediction model

Among 2,074 still images in which a tumor was visible, 1,466 were pPL− and 608 were pPL+. These images were used to fine-tune a pretrained ResNet50 model using 5-fold cross-validation. Each of the models of the 5-fold cross-validation was evaluated on the same test data, with the final reported score being the median score across all 5 folds. The total number of predictions made by all 5 models are shown in Table 4. The accuracy, precision, recall, specificity and F-value of this pPL prediction model were 49.6%, 33.5%, 13.2%, 79.5% and 19.0%, respectively. A representative Grad-CAM image is shown in Figure 2, where the deep learning model’s focus is indicated at the center of the tumor.

Figure 2 An original thoracoscopic image of a tumor with pathological pleural invasion (A) and an integrated image with a Grad-CAM heat map representing the deep learning model’s focus (B). Grad-CAM, gradient-weighted class activation mapping.

Comparison of diagnostic performance between surgeons’ evaluation and deep learning model performance

To compare the evaluations made by the surgeons and the model, image-level predictions were summarized per patient by taking the majority evaluation for each image of that patient. Summarizing the predictions at the patient-level, the accuracy of surgeons and deep learning predictions was found to be 0.75 and 0.69 respectively. Here, no statistical significance was found between the surgeons’ evaluation and the deep learning model predictions (P=0.32, McNemar’s test).

Discussion

In this study, we conducted a pilot study to investigate the feasibility of predicting pathologic pPL of peripherally located NSCLC from thoracoscopic images using a deep learning algorithm. The overall prediction performance was comparable to that of board-certified thoracic surgeons specializing in lung cancer surgery.

Our ultimate goal is to develop software as a medical device that can be integrated into thoracoscopes or surgical robots (we have filed a patent for this). With this software, surgeons would be able to switch from sublobar resection to lobectomy when there is a high probability of pPL as a precision surgery strategy (Figure 3).

Figure 3 Concept of precision surgery strategy using this deep learning algorithm. If the algorithm detects a high probability of pleural invasion, the planned segmentectomy (or sublobar resection) can be changed to a lobectomy to reduce the chance of local recurrence. NSCLC, non-small cell lung cancer.

The frequency of pPL in peripheral cT1 tumors was reported to be approximately 4–13% in the previous reports (7,18). The reason why the proportion of pPL in our study (22/88, 27.5%) was higher compared to the previous reports was partly because we only included peripheral cT1 tumors with an apparent pleural surface change on thoracoscopic observation. As surgeons would not be inclined to suspect pPL from the tumors without any surface changes, so would not need software support. Therefore, such tumors without surface changes were excluded from our study population. Nevertheless, it is noted that upstaging (pT2) due to pPL of cT1 tumor with pleural surface change on VATS is a frequently encountered scenario (approximately one-fourth in our study population).

Although it is well documented that patients with tumors with pPL (pT2) have a worse survival rate compared to patients with tumors without pPL (pT1) after complete resection (19), it is still uncertain whether peripheral cT1 tumors with pPL are oncologically better treated with lobectomy as compared to sublobar resection (segmentectomy or wedge resection). Although it was not available when we initiated this study, secondary analysis from CALGB 14503 showed patients with lung cancer with pPL had similar recurrence-free survival or disease-free survival between sublobar resection and lobectomy (20). If these findings are validated in the real-world settings, “conversion to lobectomy when pPL is detected” would not be necessary.

There were a few studies that investigated the predictive ability of pPL with preoperative imaging modalities. The predictive accuracy of pPLs of cT1 tumors that are in contact with pleural surfaces from preoperative 3D-CT was 77% (21). The presence of pleural tags (one or more linear pleural tags with soft tissue component at the pleural end) on preoperative CT in NSCLC was predictive of visceral pPL with an accuracy of 71% (6). SUVmax on ¹⁸F-FDG PET in adenocarcinoma with pleural contact on preoperative CT was predictive of pPL with an area under the curve (AUC) of 0.84 (22). In subsolid cT1 peripheral adenocarcinoma, pleural contact [odds ratio (OR): 8.30], pleural thickening (OR: 3.97), solid proportion ≥50% (OR: 4.64) and nodule size ≥2 cm (OR: 2.99) were predictive of pPL (23), which were consistent with our results. We speculated that direct observation of the pleural surface with a high-resolution video would provide more precise information of visceral pleura compared to preoperative imaging modalities, which therefore could lead to higher predictive ability of visceral pPL.

In our study, a total of 422,873 still images were extracted from the 80 patient surgery video files. A subset of these images were used for training a deep learning model. While a lot of still images were extracted, the variation between images is quite limited. Complex deep learning models typically need a large amount of varied input data. In our study, we effectively trained a model on 64 videos and validated it on 16 videos. This likely does not capture enough variation to train a versatile model, which is reflected in the results of the model (Tables 3,4). Both the tumor recognition model’s and pPL model’s results show that the deep learning model predicts many positive images as negative; only 1,113 out of 2,410 tumor images (Table 3) and 221 out of 1,630 pPL+ images (Table 4) were correctly predicted by the deep learning model, indicating that there’s a lot of room for improvement. As the purpose of this study was to investigate the feasibility of constructing a deep learning model to predict pPL from operation videos, further analysis with more patients, ideally at a multi-institutional level, is required as a next step. We also recognize that an AI model with 69% accuracy, which cannot outperform expert thoracic surgeons, is still not powerful enough for our purpose (i.e., converting the operative mode based on the prediction). An accuracy of at least 90% or more is probably needed to be accepted in clinical settings. For this, the optimal observation mode (distance, angle, light intensity, thoracoscopy) needs to be further tuned with the prospect of future data collection. Integrating other clinical factors might be another option.

The results of this preliminary study showed that the model has a moderate predictive ability that is comparable to surgeons, despite having a limited training dataset. It is therefore possible that an even higher predictive ability can be achieved with further adjustments to the deep learning model and/or by using more data. This pilot study is limited by its inherent nature of being a small retrospective pilot study. Therefore, external validation with a larger population is warranted.

Conclusions

In conclusion, deep learning analysis of thoracoscopic images of lung cancer surgery showed the possibility of recognizing tumors with pleural changes and prediction of pPL to a comparable degree to surgeons’ evaluation. We will examine whether further learning can provide predictions outperforming those of surgeons.

Acknowledgments

This paper was presented in the 31st annual meeting of the European Association for Thoracic Surgery, Milano, Italy, June 4–6, 2023.

Footnote

Reporting Checklist: The authors have completed the STROBE reporting checklist. Available at https://jtd.amegroups.com/article/view/10.21037/jtd-24-1510/rc

Data Sharing Statement: Available at https://jtd.amegroups.com/article/view/10.21037/jtd-24-1510/dss

Peer Review File: Available at https://jtd.amegroups.com/article/view/10.21037/jtd-24-1510/prf

Funding: This study was funded by the Takeda Science Foundation (2021 Medical Research Fund) and the divisional funding at the Cancer Institute Hospital.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://jtd.amegroups.com/article/view/10.21037/jtd-24-1510/coif). C.D. and J.S. are from Humanome Lab. Inc. (a for-profit company). K.H., C.D., J.S. and M.M. report a patent application has been submitted for the pleural invasion prediction on thoracoscopic lung cancer surgery data algorithm and dataset (application number: 2023-053340). The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The institutional review board at the Cancer Institute Hospital approved this study on Jan 25, 2022 (No. 2021-GB-107) and informed consent was waived due to its retrospective nature of research. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Neri S, Yoshida J, Ishii G, et al. Prognostic impact of microscopic vessel invasion and visceral pleural invasion in non-small cell lung cancer: a retrospective analysis of 2657 patients. Ann Surg 2014;260:383-8. [Crossref] [PubMed]
2. Yoshida J, Nagai K, Asamura H, et al. Visceral pleura invasion impact on non-small cell lung cancer patient survival: its implications for the forthcoming TNM staging based on a large-scale nation-wide database. J Thorac Oncol 2009;4:959-63. [Crossref] [PubMed]
3. Travis WD, Brambilla E, Rami-Porta R, et al. Visceral pleural invasion: pathologic criteria and use of elastic stains: proposal for the 7th edition of the TNM classification for lung cancer. J Thorac Oncol 2008;3:1384-90.
4. Takizawa H, Tsuboi M, Kajiura K, et al. Intraoperative diagnosis of pleural invasion of lung cancer patients: Evaluation of accuracy. The Japanese Association for Chest Surgery 2015;29:576-81.
5. Kim H, Goo JM, Kim YT, et al. CT-defined Visceral Pleural Invasion in T1 Lung Adenocarcinoma: Lack of Relationship to Disease-Free Survival. Radiology 2019;292:741-9. [Crossref] [PubMed]
6. Hsu JS, Han IT, Tsai TH, et al. Pleural Tags on CT Scans to Predict Visceral Pleural Invasion of Non-Small Cell Lung Cancer That Does Not Abut the Pleura. Radiology 2016;279:590-6. [Crossref] [PubMed]
7. Saji H, Okada M, Tsuboi M, et al. Segmentectomy versus lobectomy in small-sized peripheral non-small-cell lung cancer (JCOG0802/WJOG4607L): a multicentre, open-label, phase 3, randomised, controlled, non-inferiority trial. Lancet 2022;399:1607-17. [Crossref] [PubMed]
8. Altorki N, Wang X, Kozono D, et al. Lobar or Sublobar Resection for Peripheral Stage IA Non-Small-Cell Lung Cancer. N Engl J Med 2023;388:489-98. [Crossref] [PubMed]
9. Landreneau RJ, Normolle DP, Christie NA, et al. Recurrence and survival outcomes after anatomic segmentectomy versus lobectomy for clinical stage I non-small-cell lung cancer: a propensity-matched analysis. J Clin Oncol 2014;32:2449-55. [Crossref] [PubMed]
10. Chan EG, Chan PG, Mazur SN, et al. Outcomes with segmentectomy versus lobectomy in patients with clinical T1cN0M0 non-small cell lung cancer. J Thorac Cardiovasc Surg 2021;161:1639-48.e2. [Crossref] [PubMed]
11. Koike T, Koike T, Yoshiya K, et al. Risk factor analysis of locoregional recurrence after sublobar resection in patients with clinical stage IA non-small cell lung cancer. J Thorac Cardiovasc Surg 2013;146:372-8. [Crossref] [PubMed]
12. Kitada M, Ohsaki Y, Matsuda Y, et al. Photodynamic diagnosis of pleural malignant lesions with a combination of 5-aminolevulinic acid and intrinsic fluorescence observation systems. BMC Cancer 2015;15:174. [Crossref] [PubMed]
13. Saito Y, Yamakawa Y, Kiriyama M, et al. Diagnosis of visceral pleural invasion by lung cancer using intraoperative touch cytology. Ann Thorac Surg 2002;73:1552-6; discussion 1556-7. [Crossref] [PubMed]
14. Hirasawa T, Aoyama K, Tanimoto T, et al. Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images. Gastric Cancer 2018;21:653-60. [Crossref] [PubMed]
15. Zhu Y, Wang QC, Xu MD, et al. Application of convolutional neural network in the diagnosis of the invasion depth of gastric cancer based on conventional endoscopy. Gastrointest Endosc 2019;89:806-815.e1. [Crossref] [PubMed]
16. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017;542:115-8. [Crossref] [PubMed]
17. Yoshino I, Moriya Y, Suzuki K, et al. Long-term outcome of patients with peripheral ground-glass opacity-dominant lung cancer after sublobar resections. J Thorac Cardiovasc Surg 2023;166:1222-1231.e1. [Crossref] [PubMed]
18. Tsutani Y, Miyata Y, Nakayama H, et al. Oncologic outcomes of segmentectomy compared with lobectomy for clinical stage IA lung adenocarcinoma: propensity score-matched analysis in a multicenter study. J Thorac Cardiovasc Surg 2013;146:358-64. [Crossref] [PubMed]
19. Rami-Porta R, Ball D, Crowley J, et al. The IASLC Lung Cancer Staging Project: proposals for the revision of the T descriptors in the forthcoming (seventh) edition of the TNM classification for lung cancer. J Thorac Oncol 2007;2:593-602. [Crossref] [PubMed]
20. Altorki N, Wang X, Damman B, et al. Recurrence of Non-Small Cell Lung Cancer With Visceral Pleural Invasion: A Secondary Analysis of a Randomized Clinical Trial. JAMA Oncol 2024;10:1179-86. [Crossref] [PubMed]
21. Ebara K, Takashima S, Jiang B, et al. Pleural invasion by peripheral lung cancer: prediction with three-dimensional CT. Acad Radiol 2015;22:310-9. [Crossref] [PubMed]
22. Tanaka T, Shinya T, Sato S, et al. Predicting pleural invasion using HRCT and 18F-FDG PET/CT in lung adenocarcinoma with pleural contact. Ann Nucl Med 2015;29:757-65. [Crossref] [PubMed]
23. Ahn SY, Park CM, Jeon YK, et al. Predictive CT Features of Visceral Pleural Invasion by T1-Sized Peripheral Pulmonary Adenocarcinomas Manifesting as Subsolid Nodules. AJR Am J Roentgenol 2017;209:561-6. [Crossref] [PubMed]