Development of a machine learning-based model to predict prognosis of resected invasive pulmonary adenocarcinoma

Introduction

Invasive pulmonary adenocarcinoma (IPA) is the most prevalent subtype of lung cancer and poses a significant threat to global health. Although neoadjuvant and adjuvant immunotherapies or targeted therapies have remarkably improved patient health, some patients still experience tumor recurrence and metastasis (1). Therefore, it is imperative to create disease-free survival (DFS) prediction models for patients with IPA to accurately address their concerns about recurrence and guide individualized management.

The International Association for the Study of Lung Cancer (IASLC) has proposed an updated histological grading system to clearly discriminate the prognosis of heterogeneous IPA (2), and recent studies have validated its robustness among various populations (3-7), including patients in Asia, Europe, and America. Several prognostic models have been proposed for lung adenocarcinoma patients using histopathomics (8), genomics (9,10), transcriptomics (11,12), and radiomics (13,14). However, the clinical application of these models is not as convenient as that of the novel grading system. Additionally, several other clinicopathological factors may also affect the prognosis. Previous studies have shown that visceral pleural invasion (VPI) and lymphovascular invasion (LVI) are independent risk factors for early lung cancer after surgery, which is consistent with the results of our study (15-17). Carcinoembryonic antigen (CEA), as one of the most common tumor markers playing an important role in the screening and diagnosis of lung cancer, has also been shown to be an independent prognostic factor for resected non-small cell lung cancer (18,19). Hence, whether an integrative prognosis model with clinicopathological features, as well as the novel grading system, could enhance the classification ability warrants further exploration.

Machine learning, a cutting-edge branch of artificial intelligence (AI), has emerged as a crucial field of interest, offering sophisticated methods, techniques, and tools for analyzing data from the biological sciences. This technology can learn from data samples, enabling it to forecast individual patient survival and develop clinical AI models that boast significantly enhanced predictive accuracy (20). In light of these considerations, the aim of the present study was to construct a prognostic model of IPA patients to predict DFS through machine learning based on the novel grading system. We present this article in accordance with the TRIPOD+AI reporting checklist (available at https://jtd.amegroups.com/article/view/10.21037/jtd-2025-1669/rc).

Methods

Patient enrollment and screen

This retrospective study was approved by the Institutional Ethics Committee of Hangzhou First People’s Hospital (No. ZN-20230928-0224-01). Informed consent was waived due to the retrospective design of the study, in accordance with national legislation and institutional requirements. This study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. A total of 1,150 lung cancer patients who underwent surgical resection between January 2015 and October 2020 at the Department of Thoracic Surgery, Hangzhou First People’s Hospital were screened and 670 IPA patients were finally included in this retrospective study. The exclusion criteria were as follows: (I) squamous carcinoma, large cell carcinoma, carcinoid, and other types; (II) adenocarcinoma in situ (AIS), invasive mucinous adenocarcinoma (IMA), minimally invasive adenocarcinoma (MIA), and other variants of adenocarcinoma; (III) received neoadjuvant anti-tumor treatments; and (4) lost to follow-up.

Clinicopathological characteristics

Clinicopathological characteristics were obtained by reviewing patients’ electronic medical records, including age, gender, smoking history, package years (1 package-year = smoking 20 cigarettes/day for 1 year), tumor location, surgical procedure, p-TNM staging (according to the 8th edition of the American Joint Committee on Cancer staging manual), tumor size, tumor stage, nodal status, VPI, lymphovascular invasion (LVI), CEA levels, body mass index (BMI), and treatment history. DFS was recorded by clinic or telephone follow-up until September 2022. DFS was defined as the time from surgical resection to the first documented recurrence or death from any cause. Patients who died without documented recurrence were counted as events.

Histological evaluation and grading criteria

As proposed by the novel IASLC grading system (2), all pathological sections were reevaluated by professional pathologists, and the grades were defined as follows: grade 1, lepidic predominant tumor with no or less than 20% high-grade patterns (solid, micropapillary, or complex gland); grade 2, acinar or papillary predominant tumor, both with no or less than 20% high-grade patterns; and grade 3, any tumor with 20% or more high-grade patterns.

Model building

The Python package ‘scikit-survival’ (21) was employed in the model building and evaluation section. Survival models based on (I) the conventional Cox proportional hazard model; (II) penalized Cox model with least absolute shrinkage and selection operator (Lasso); (III) gradient boosted (GB) model, and (IV) random survival forest (RSF) model, were compared. All clinicopathological features including the new grade were input for model building except for the gene information since only 201/670 cases had complete gene mutation details, which may limit the integration of molecular predictors in the current study. A 5-fold cross-validation analysis was performed to maximize the Harrell Concordance index as the performance metric by repeating the cross-validation procedure 100 times and calculating the mean value of the classification accuracy. In our study, 105 recurrence events were observed among 670 patients. The final Lasso model included 16 predictors and the events per predictor (EPP) ratio was approximately 6.6, which is within the commonly recommended threshold of 5 to 10 for stable model development. All candidate predictor variables were fully observed in our study.

Model evaluation

A time-dependent receiver operating characteristic (ROC) curve was drawn, and the area under the ROC curve (AUC) as well as C-index were calculated to assess the prediction ability of the classifier. Calibration curve analysis was applied to estimate the calibration effect, and decision curve analysis (DCA) was utilized to assess the clinical value. SHapley Additive exPlanations (SHAP) is a method to explain the importance of the features in the classifications and thus a Python package ‘sHAP’ (22) was utilized to calculate the SHAP value of each variable in the model and observe the impact of the variable on the classification. Finally, the Python package ‘Pynomo’ (23) was applied to delineate nomograms for predicting survival. Patients in the test set were stratified into low risk (<10%), intermediate risk (10–20%), and high risk (>20%). The number of patients in each group, as well as sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), were reported. Kaplan-Meier curves and calibration plots were generated for each risk stratum.

Code availability

The source code used in the manuscript are deposited at GitHub and freely accessible (https://github.com/Huang111999/LC_ML).

Statistical analysis

Categorical variables were expressed as percentages and evaluated using the Pearson Chi-squared test or Fisher’s exact test. DFS was estimated using Kaplan-Meier curves and survival differences were assessed using the log-rank test. All tests were two-sided, and a P value <0.05 was considered statistically significant.

Results

Patient characteristics

A total of 670 IPA patients who underwent surgical resection between January 2015 and October 2020 at our site were finally included in this retrospective study (Figure 1). The baseline clinicopathological characteristics including the IASLC grading system of the training and test sets are shown in Table 1 and summarized below. There was no difference in baseline data (except for smoking history and lateral side) between the training and test sets.

Figure 1 Flowchart of study design and patient selection. AIS, adenocarcinoma in situ; AUC, area under the curve; IMA, invasive mucinous adenocarcinoma; MIA, minimally invasive adenocarcinoma; SHAP, SHapley Additive exPlanations.

Of these patients, 62.7% were female, 57.3% were older than 60 years old, and 74.9% were non-smokers. For the ever-smokers, the median package years were 35 (21.875, 45). Tumors originating in the right lung constituted 60.7% of the cases, and 66.7% tumors were situated in either the upper or middle lobes. In terms of the surgical procedures, about 75.8% of patients underwent lobectomy. Regarding tumor pathology, most tumors were described as grade 2 (53.3%), without VPI (73.3%), without LVI (93.9%), <2 cm (69.3%), pT1 (69.3%), pN0 (86.6%), and stage I (84.1%). Most patients had normal CEA levels (81.2%) while normal BMI values (61.2%). Median follow-up time was 29 months, and 105 (15.7%) recurrence events occurred by the last follow-up timepoint.

Construction of machine learning based model for predicting the DFS of IPA patients

In the training set, four models available in the scikit-survival package, including Cox proportional hazards (Cox), Lasso regression (Lasso), RSF, and gradient boosting (GB), were compared, and 5-fold cross-validation was performed for iterative testing and tuning. The model with the highest C-index was chosen for further validation. As shown in Figure 2A, the 2-year AUC values of Lasso, Cox, RSF, and GB models, as well as the grading system alone were 0.906, 0.895, 0.902, and 0.878, respectively. The 3-year AUC values of Lasso, Cox, RSF, and GB models were 0.894, 0.886, 0.884, and 0.870, respectively. The 4-year AUC values of Lasso, Cox, RSF, and GB models were 0.901, 0.893, 0.889, and 0.877, respectively. The 5-year AUC values of Lasso, Cox, RSF, and GB models were 0.927, 0.923, 0.908, and 0.894, respectively. All models had superior 2-, 3-, and 4-year AUC values than the grading system alone. The C-index values of Lasso, Cox, RSF, and GB models were 0.880, 0.870, 0.874, and 0.855, respectively. Hence, the Lasso model showed superior performance across almost all time points in predicting DFS when compared to the other models. During the Lasso regression model construction, the alpha value was set with the best C-index score, and the factors were entered into the model and nomogram development (Figure S1). The Lasso model performed well in predicting DFS of IPA patients in both the training and test set at 2-year (training set: AUC =0.906, test set: AUC =0.862), at 3-year (training set: AUC =0.894, test set: AUC =0.879), at 4-year (training set: AUC =0.901, test set: AUC =0.902), at 5-year (training set: AUC =0.927, test set: AUC =0.887) (Figure 2).

Figure 2 Performance matrix of machine-learning models for predicting survival outcomes in resected IPA patients. (A) Time-dependent AUC value of the Lasso, Cox, GB, RSF models, and the grading system for predicting DFS in the training set. (B) Time-dependent AUC value of the Lasso model for predicting DFS in the test set. AUC, area under the receiver operating characteristic curve; DFS, disease-free survival; GB, gradient boosting; IPA, invasive pulmonary adenocarcinoma; Lasso, least absolute shrinkage and selection operator regression; RSF, random survival forest.

Evaluation of the constructed model

The Lasso model-related calibration curves displayed good consistency in the probability of 2-, 3-, 4-, and 5-year survival between the actual DFS and the predicted DFS in the training and test sets (Figure 3). Meanwhile, the DCA curves of 2-, 3-, 4-, and 5-year survival in the training and test sets also demonstrated good clinical utility, showing a preferable positive net benefit (Figure 4).

Figure 3 Calibration curves of the Lasso model in the training and test sets. Calibration curves for (A) 2-year, (B) 3-year, (C) 4-year, and (D) 5-year prognostic models in the training set and the test set. DFS, disease-free survival; Lasso, least absolute shrinkage and selection operator.

Figure 4 Decision curve analysis of the Lasso model in the training and test sets. Decision curve analysis curves for (A) 2-year, (B) 3-year, (C) 4-year, and (D) 5-year prognostic models in the training set and the test set. DCA, decision curve analysis; DFS, disease-free survival; Lasso, least absolute shrinkage and selection operator.

The contribution of predictors to the model was further evaluated using the SHAP value and the novel pathological grade showed the highest value, indicating its essential role in the model construction to predict DFS (Figure 5A). The nomogram of 2-, 3-, and 5-year DFS was then established and is shown in Figure 5B, and patients in the test set were then stratified into low risk (<10%), intermediate risk (10–20%), and high risk (>20%) groups based on the risk score. Notably, the DFS of the low-risk group was remarkably higher than that of the intermediate-risk group (P=0.003) and high-risk group (P<0.001) (Figure 5C). The model achieved a sensitivity of 0.75 and a specificity of 0.913 for predicting 3-year DFS with a good calibration (Figure S2), which indicated a strong potential for risk stratification.

Figure 5 Construction of nomogram for predicting DFS in resected invasive lung adenocarcinoma patients. (A) The predictors’ contribution to the model evaluated by SHAP; (B) nomogram of the established model for predicting 2-, 3-, and 5-year DFS; (C) survival cures of low-risk (<10%), intermediate-risk (10–20%), high-risk (>20%) groups in the test set stratified by the risk score of the model. BMI, body mass index; CEA, carcinoembryonic antigen; DFS, disease-free survival; LVI, lymphovascular invasion; SHAP, SHapley Additive exPlanations; VPI, visceral pleural invasion.

Discussion

In current clinical practice, genomic or transcriptomic profiling may greatly add to the economic burden and inconvenience. Therefore, reliable, convenient, and applicable models are clearly needed. In this study, we aimed to develop a prognostic model for patients with IPA based on the updated IASLC grading system and other clinically available clinicopathological factors.

To construct an accurate prognosis model, a machine learning algorithm was employed and different methods were compared. Lasso regression was found to be more suitable for model construction. Notably, 16 basic clinicopathological features of patients with IPA were considered in the final model, which could be helpful in providing a comprehensive and accurate prediction. Our findings revealed that the grading system, nodal status, tumor size, and CEA levels, were relatively vital variables affecting DFS. Among these, the grading system was the most important, further validating the importance of the updated IASLC grading system.

Nodal status and tumor size are well-established prognostic factors, reflecting tumor burden and dissemination potential. Elevated CEA levels likely indicate biological aggressiveness and subclinical metastasis. The integration of these factors enhances DFS prediction by capturing both histopathological severity and clinical indicators of tumor progression.

Recently, machine learning-based prognosis models have attracted increasing attention in clinical and translational cancer research. For instance, artificial neural networks or the ComplEx-N3 model have been employed to predict recurrence in patients with early-stage non-small cell lung cancer (24,25). However, recurrence was mistaken as an outcome without consideration of time support. Kinoshita et al. showed that the AI prognostic model of resected early stage non-small cell lung cancer using XGBoost exhibited good performance, with an AUC of DFS at 5 years of 0.890 (26), whereas our study focused on patients with early-stage IPA.

To the best of our knowledge, this is the first study to create AI prognostic models for patients with IPA by the updated IASLC grading system. Four machine learning algorithms were implemented, five-fold cross-validation for iterative testing and tuning was utilized, and the established model achieved robust performance in both the training and test sets. Of note, our model showed superior AUC values than the grading system alone, which implies that integration of other clinical parameters is also important to predict recurrence precisely. Instead of other reported models that integrate multi-omics data, which makes the prediction unaffordable, our model is convenient and could be freely and easily used by clinical oncologists or patients. Nonetheless, our study had some limitations. Only 15.7% of patients experienced recurrence events by the last follow-up timepoint, which may restrict model development and limit the model’s applicability. Notably, the impact of gene status on prognosis was not included because above half of patients did not perform the next-generation sequencing profiling. The sample size was limited, especially for Grade 3 cases and stages II and III samples. Additionally, all cases were from a single center, only internal validation was performed, and external data validation in the future may support the model in the clinical setting. Future studies should focus on: Multicenter external validation to confirm generalizability; Incorporation of gene mutation or multi-omics data to refine biological insight; Prospective studies to evaluate the model’s impact on clinical decision-making and patient outcomes.

Conclusions

Our study successfully constructed a prognostic model for risk stratification based on the grading system using a machine learning technique, which may provide oncologists and surgeons with an effective tool for recurrence prediction and early medical intervention to improve the survival of patients with IPA. Further external validation and integration of molecular data are warranted to enhance clinical applicability and generalizability.

Acknowledgments

We thank all patients, pathologists, surgeons and oncologists involved in this study.

Footnote

Reporting Checklist: The authors have completed the TRIPOD+AI reporting checklist. Available at https://jtd.amegroups.com/article/view/10.21037/jtd-2025-1669/rc

Data Sharing Statement: Available at https://jtd.amegroups.com/article/view/10.21037/jtd-2025-1669/dss

Peer Review File: Available at https://jtd.amegroups.com/article/view/10.21037/jtd-2025-1669/prf

Funding: This study was supported in part by grants from the Medical and Health Technology Project of Hangzhou (No. A20220840) and The Construction Fund of Key Medical Disciplines of Hangzhou (No. 2025HZGF06).

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://jtd.amegroups.com/article/view/10.21037/jtd-2025-1669/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. This retrospective study was approved by the Institutional Ethics Committee of Hangzhou First People’s Hospital (No. ZN-20230928-0224-01). Informed consent was waived due to the retrospective design of the study, in accordance with national legislation and institutional requirements. This study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Rajaram R, Huang Q, Li RZ, et al. Recurrence-Free Survival in Patients With Surgically Resected Non-Small Cell Lung Cancer: A Systematic Literature Review and Meta-Analysis. Chest 2024;165:1260-70. [Crossref] [PubMed]
2. Moreira AL, Ocampo PSS, Xia Y, et al. A Grading System for Invasive Pulmonary Adenocarcinoma: A Proposal From the International Association for the Study of Lung Cancer Pathology Committee. J Thorac Oncol 2020;15:1599-610. [Crossref] [PubMed]
3. Deng C, Zheng Q, Zhang Y, et al. Validation of the Novel International Association for the Study of Lung Cancer Grading System for Invasive Pulmonary Adenocarcinoma and Association With Common Driver Mutations. J Thorac Oncol 2021;16:1684-93. [Crossref] [PubMed]
4. Rokutan-Kurata M, Yoshizawa A, Ueno K, et al. Validation Study of the International Association for the Study of Lung Cancer Histologic Grading System of Invasive Lung Adenocarcinoma. J Thorac Oncol 2021;16:1753-8. [Crossref] [PubMed]
5. Fujikawa R, Muraoka Y, Kashima J, et al. Clinicopathologic and Genotypic Features of Lung Adenocarcinoma Characterized by the International Association for the Study of Lung Cancer Grading System. J Thorac Oncol 2022;17:700-7. [Crossref] [PubMed]
6. Park BJ, Woo W, Cha YJ, et al. Proposal of a revised International Association for the Study of Lung Cancer grading system in pulmonary non-mucinous adenocarcinoma: The importance of the lepidic proportion. Lung Cancer 2023;175:1-8. [Crossref] [PubMed]
7. Hou L, Wang T, Chen D, et al. Prognostic and predictive value of the newly proposed grading system of invasive pulmonary adenocarcinoma in Chinese patients: a retrospective multicohort study. Mod Pathol 2022;35:749-56. [Crossref] [PubMed]
8. Diao S, Chen P, Showkatian E, et al. Automated Cellular-Level Dual Global Fusion of Whole-Slide Imaging for Lung Adenocarcinoma Prognosis. Cancers (Basel) 2023;15:4824. [Crossref] [PubMed]
9. Jones GD, Brandt WS, Shen R, et al. A Genomic-Pathologic Annotated Risk Model to Predict Recurrence in Early-Stage Lung Adenocarcinoma. JAMA Surg 2021;156:e205601. [Crossref] [PubMed]
10. Yu J, Hu Y, Xu Y, et al. LUADpp: an effective prediction model on prognosis of lung adenocarcinomas based on somatic mutational features. BMC Cancer 2019;19:263. [Crossref] [PubMed]
11. Li Y, Ge D, Gu J, et al. A large cohort study identifying a novel prognosis prediction model for lung adenocarcinoma through machine learning strategies. BMC Cancer 2019;19:886. [Crossref] [PubMed]
12. Shi R, Bao X, Unger K, et al. Identification and validation of hypoxia-derived gene signatures to predict clinical outcomes and therapeutic responses in stage I lung adenocarcinoma patients. Theranostics 2021;11:5061-76. [Crossref] [PubMed]
13. Kim H, Goo JM, Lee KH, et al. Preoperative CT-based Deep Learning Model for Predicting Disease-Free Survival in Patients with Lung Adenocarcinomas. Radiology 2020;296:216-24. [Crossref] [PubMed]
14. Huang Y, Liu Z, He L, et al. Radiomics Signature: A Potential Biomarker for the Prediction of Disease-Free Survival in Early-Stage (I or II) Non-Small Cell Lung Cancer. Radiology 2016;281:947-57. [Crossref] [PubMed]
15. Jiwangga D, Cho S, Kim K, et al. Recurrence Pattern of Pathologic Stage I Lung Adenocarcinoma With Visceral Pleural Invasion. Ann Thorac Surg 2017;103:1126-31. [Crossref] [PubMed]
16. Shimizu K, Yoshida J, Nagai K, et al. Visceral pleural invasion is an invasive and aggressive indicator of non-small cell lung cancer. J Thorac Cardiovasc Surg 2005;130:160-5. [Crossref] [PubMed]
17. Neri S, Yoshida J, Ishii G, et al. Prognostic impact of microscopic vessel invasion and visceral pleural invasion in non-small cell lung cancer: a retrospective analysis of 2657 patients. Ann Surg 2014;260:383-8. [Crossref] [PubMed]
18. Tomita M, Shimizu T, Ayabe T, et al. Prognostic significance of tumour marker index based on preoperative CEA and CYFRA 21-1 in non-small cell lung cancer. Anticancer Res 2010;30:3099-102.
19. Muley T, Dienemann H, Ebert W. CYFRA 21-1 and CEA are independent prognostic factors in 153 operated stage I NSCLC patients. Anticancer Res 2004;24:1953-6.
20. Haug CJ, Drazen JM. Artificial Intelligence and Machine Learning in Clinical Medicine, 2023. N Engl J Med 2023;388:1201-8. [Crossref] [PubMed]
21. Pölsterl S. scikit-survival: A Library for Time-to-Event Analysis Built on Top of scikit-learn. J Mach Learn Res 2020;21:1-6.
22. Lundberg SM, Nair B, Vavilala MS, et al. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nat Biomed Eng 2018;2:749-60. [Crossref] [PubMed]
23. Martínez-Pagán P, Roschier L. PyNomo software dataset for sciences and engineering nomogram construction. Data Brief 2022;45:108661. [Crossref] [PubMed]
24. Lorenc A, Romaszko-Wojtowicz A, Jaśkiewicz Ł, et al. Exploring the efficacy of artificial neural networks in predicting lung cancer recurrence: a retrospective study based on patient records. Transl Lung Cancer Res 2023;12:2083-97. [Crossref] [PubMed]
25. Janik A, Torrente M, Costabello L, et al. Machine Learning-Assisted Recurrence Prediction for Patients With Early-Stage Non-Small-Cell Lung Cancer. JCO Clin Cancer Inform 2023;7:e2200062. [Crossref] [PubMed]
26. Kinoshita F, Takenaka T, Yamashita T, et al. Development of artificial intelligence prognostic model for surgically resected non-small cell lung cancer. Sci Rep 2023;13:15683. [Crossref] [PubMed]