Validation of the American College of Surgeons surgical risk calculator for thoracic surgery

Introduction

Background

Due to advances in medical and surgical treatments, high-risk patients are now being considered for surgery, which may result in higher rates of postoperative complications and death. Therefore, it is crucial to assess the risk of morbidity and mortality after medical or surgical procedures, which can be affected by patient age, body mass index (BMI), lung function, cancer stage, concomitant diseases, and other risk factors (1-3).

Rationale and knowledge gap

The reported overall complication rate after thoracic surgery under general anesthesia ranges from 15% to 37.5% (4). Pulmonary and cardiac morbidity are the main causes of mortality, with reported mortality rates of 1.2–2.9% after a lobectomy and 3.2–6.8% after pneumonectomy (2,5,6). Traditionally, surgeons rely on their own clinical experience to identify patients at “high risk” for postoperative complications, but this judgment is not always based on precise risk assessments (7). Today, indices can be used to identify “high-risk” patients and to assess surgical risks. Two of the most common predictive tools for surgical risk assessment are the Revised Cardiac Risk Index (RCRI) (8), which estimates the risk of cardiac complications after non-cardiac surgery, and the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) surgical risk calculator (SRC) (7). The ACS NSQIP database was developed to predict risk-adjusted surgical outcomes and improve the quality of surgical care (9). In 2013, the ACS NSQIP presented a web-based SRC that is currently based on clinical data (patient demographics, comorbidities, and complications observed up to 30-days after surgery) continuously collected from more than 500 hospitals (7). It estimates the 30-day postoperative mortality and morbidity risk for a variety of surgical procedures, including thoracic surgery.

To date, the suitability of the universal ACS NSQIP SRC in thoracic surgery has only been evaluated for patients undergoing elective lung resection, with inconsistent results (10,11). There is no data how the SRC will perform in the general thoracic surgery population (emergency chest trauma, surgery for empyema, pleural and mediastinal diseases, etc.). Annually, approximately 700 thoracic surgeries of various types are performed in our center. These patients are also at risk for unfavorable postoperative events.

Objective

The aim of this study was to validate the ACS NSQIP SRC and explore its suitability for general thoracic surgery. We compared the predicted ACS NSQIP risk outcomes with the observed incidence of 30-day morbidity and mortality after any thoracic procedure conducted in our study cohort. We present this article in accordance with the STARD reporting checklist (available at https://jtd.amegroups.com/article/view/10.21037/jtd-24-611/rc).

Methods

This single-center, blinded (regardless of precalculated ACS NSQIP risk score), retrospective study included a consecutive series of patients, who underwent a thoracic surgery at the University Hospital Basel from May 1, 2021 to March 1, 2022 and who had signed the general research consent form of our institution. Patients who clearly stated that they would not agree in providing their clinical data for scientific purposes were excluded. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the local ethics committee Ethikkommission Nordwest-und Zentralschweiz (EKNZ) (No. BASEC 2022-02163, granted on December 15, 2022). All included patients signed the general research consent form of our institution.

Study design

For all patients included, we collected data on 20 preoperative risk factors [including surgical procedure, age group, sex, functional status, emergency of the surgery, American Society of Anesthesiologists (ASA) class, use of steroids, ascites, systemic sepsis, ventilator dependent, disseminated cancer, diabetes, hypertension, congestive heart failure, dyspnoea, current smoker, history of chronic obstructive pulmonary disease (COPD), dialysis, renal failure, and BMI] according to the ACS NSQIP SRC from the hospital’s internal electronic medical records system. Based on these risk factors and prior to the surgical procedure, we generated a perioperative risk profile including “average risk” for the outcomes “serious complication”, “any complication”, “pneumonia”, “cardiac complication”, “urinary tract infection”, “surgical site infection”, “venous thromboembolism”, “renal failure”, “readmission”, “return to OR”, “discharge to nursing or rehabilitation facility”, “sepsis”, and “death” on the SRC website (Figure S1). The predicted risks for all outcomes generated by the SRC were then copied into the data file for analysis. The resulting postoperative 30-day morbidity and mortality was graded based on the Clavien-Dindo classification (12) for surgical complications, which is used to record and classify postoperative complications, then blindly documented the complications (regardless of the precalculated ACS NSQIP risk score). The complications were subsequently categorized based on the SRC outcomes, where any complications of Grade IIIa and above was considered as a “serious complication”. The outcome “serious complications” is defined according to the ACS NSQIP SRC as combination of various outcomes.

Statistical methods and analysis

Considering a “serious complication” as primary study outcome, we calculated the sample size needed to achieve a 95% confidence interval (CI) width of ≤20% of the area under the receiver operating characteristic curve (AUROC) as a measure of discrimination. For an incidence of 15% serious complications (e.g., primary outcome and AUROC 60%), 262 patients are sufficient to achieve this precision. This sample size would yield a narrower CI if the AUROC was 80% or higher. We would not consider the ACS NSQIP SRC to be a useful risk estimation tool if discrimination was markedly below an AUROC of 80%.

We calculated the AUROC with CI for each outcome to assess discriminative ability. An AUROC between 50% and 70% suggested no discrimination (i.e., ability to distinguish patients with and without disease or condition based on the test), 70% to 80% was considered acceptable, 80% to 90% was considered excellent, and >90% was considered outstanding (13). In addition, we calculated the Brier score to check the calibration and used nonparametric AUROCs and scatterplots of the ACS NSQIP scores for visualization.

We conducted three sensitivity analyses for all SRC outcomes, except for “discharge to a nursing or rehabilitation facility”, as patients in Swiss health care system can choose to either be discharged at home or to a rehabilitation center. First, we repeated the analyses described above, including only lung resections. Second, we investigated whether single risk factors of the SRC model provided a better prediction of the two outcomes: “serious complication” and “any complication” compared to the entire score (see Appendix 1 for details). Third, we assessed the risk for the occurrence of a specific event associated with an exceedance of the “average risk” provided by the SRC for the respective outcome by calculating odds ratios and 95% CI.

We performed descriptive statistics of the study cohort; patient characteristics of the SRC are provided in numbers and percents. This included a comparison of patients with and without events using Fisher’s exact test. Surgical procedures were divided into four main groups with their respective Current Procedural Terminology (CPT) codes—lung resection, pleural empyema, chest wall interventions, and various (Table S1). Unspecific results were interpreted as false-positive or false-negative and reported in the analysis.

All analyses were conducted using Stata 16.0 (StataCorp LLC, College Station, TX, USA).

Results

Our study cohort included 283 patients (Figure S2 Flow diagram) who underwent lung resections (n=174), pleural empyema surgery (n=16), chest-wall interventions (n=39), and other thoracic surgeries (n=54) (CPT codes see Table S1). Nearly half of the patients (48%) were younger than 65 years, 75% were classified as ≥ ASA III, 48% as overweight and obese, 39% required hypertensive medication, 27% were current smokers, 25% had dyspnea, and 13% had severe COPD. “Serious complications” occurred in 41 patients. Of these, 56% were younger than 65 years, 75% were classified as ≥ ASA III, 32% were overweight and obese, 34% had dyspnea, and 29% had severe COPD (Table 1).

Association of predicted and observed “serious complications”

The observed frequency of “serious complications” within 30 days was 14.5%, whereas the predicted mean was 10.9%. The analysis of the discriminative ability of ACS NSQIP SRC in our study cohort yielded an AUROC of 59% (95% CI: 49–68%) for “serious complication”, indicating no useful discrimination (Figure 1). The Brier score was moderate at 0.13 (Table 2). Restricting the analysis to lung resections increased the AUROC to 63% (95% CI: 53–73%), suggesting low discrimination (Table 3). “Serious complications” were equally frequent among patients above and below the predicted ACS NSQIP SRC average risk (odds ratio, 1.02; P=0.96) (Table 4).

Figure 1 Left hand side: AUROC for predicted risk of “serious complication”. Right hand side: scatterplot of predicted risks of patients with [1] and without [0] “serious complication”. AUROC, area under the receiver operating characteristic.

Prediction of other outcomes

The observed postoperative frequencies of “any complication” and “return to OR” within 30 days were 26.1% and 11.7% compared to the mean predicted values of 12% and 3.6%, respectively. The AUROC was 67% for “any complication” (Figure 2) and 58% for “return to OR”, expressing no useful prediction. Excellent discrimination was only observed for “renal failure” (Table 2), with an AUROC of 99%. However, this was based on only two events. Discrimination with respect to “pneumonia” (AUROC, 76%) was fair (Figure 3), with a 95% CI ranging from poor to excellent discrimination (61–92%). Limiting the analysis exclusively to lung resection showed comparable results. The score’s predictive ability for “any complication” showed similar results when compared to the entire cohort (AUROC, 67% vs. 68%), yet the predictive value remained unsatisfactory (Tables 2,3). For all outcomes, the Brier score was low particularly for outcomes with very few events.

Figure 2 Left hand side: AUROC for predicted risk of “any complication”. Right hand side: scatterplot of predicted risks of patients with [1] and without [0] “any complication”. AUROC, area under the receiver operating characteristic.

Figure 3 Left hand side: AUROC for predicted risk of “pneumonia”. Right hand side: scatterplot of predicted risks of patients with [1] and without [0] “pneumonia”. AUROC, area under the receiver operating characteristic.

The likelihood of experiencing “any complication” was greater for patients with above average risk scores (odds ratio, 1.8; P=0.03). This was also observed with “return to OR”, although it did not reach statistical significance (Table 4).

Including selected, statistically significant risk factors (dyspnea, ASA, severe COPD, and BMI) as single postoperative risk predictors for “serious complication” and “any complication” showed that each of the selected variables significantly improved the basic model, consisting of the logit of the ACS NSQIP SRC alone. Therefore, the SRC alone did not perform as well as these single variables did (Tables 1,5).

Discussion

Key findings

The results of our study show that the ACS NSQIP SRC has a low discriminative ability for our primary and secondary outcomes. These findings suggest an overall low predictive ability of the SRC for 30-day mortality and morbidity risk after general thoracic surgery in our cohort.

Strengths and limitations

In terms of limitations, we must acknowledge that our study was conducted retrospectively. In addition our cohort was relatively small (n=283), although a meta-analysis of six studies evaluating the accuracy of the ACS NSQIP score in emergency abdominal surgery showed a median cohort size of 300 patients (ranging from 69 to 758 patients) (14). Nevertheless, our pre-calculated patient count was sufficient to validate the score as predictor of “serious complication”, “any complication”, and “return to OR”, but this was not the case for outcomes with frequencies lower than 10%.

Comparison with similar research

The ACS NSQIP SRC has been validated in various surgical specialities. Some studies have reported excellent prognostic ability of the ACS NSQIP SRC, while others have found it to be an inadequate predictive tool (10,11,14-16).

A 2015 study from Washington University, analysing the calculated ACS NSQIP risk for patients undergoing surgery (n=277) or stereotactic body radiation therapy (SBRT) (n=195) for stage I lung cancer published results similar to ours (10). The observed rate for “serious complications” in patients undergoing lung resection was twice as high as predicted by the score (16.6% vs. 8.8%), showing its tendency to underestimate the occurrence of serious complications. Their reported AUROC for wedge resections was 65% (95% CI: 56–74%). This is consistent with our results (63%) and suggests a poor discriminative ability of the score (Table 3). In the lobectomy group, they observed an acceptable AUROC of 78% (95% CI: 73–82%). Patients in this group had a lower ASA, a median forced expiratory volume in 1 second (FEV1) of 82.7%, a diffusing capacity of the lung for carbon monoxide (DLCO) of 73.6% (which speaks against COPD), and were less likely to have history of congestive heart failure. Our study cohort included 13% patients with severe COPD, more than twice as many with congestive heart failure, and 17% with ASA II.

On the other hand, Chudgar et al. reported results comparable to the risk score estimations in their analysis in a cohort of patients undergoing lung resection (11). Observed complication rates were generally lower than estimated (“serious complication” 7.4% vs. 9.2%) with almost identical results for “any complication”, pneumonia, and cardiac complications. Despite examining more than 2,000 recruited cases and a fair number of events, only the discriminative ability for cardiac complications was excellent (AUROC, 82%; 95% CI: 70–95%). For most of the outcomes, the AUROC was merely acceptable (95% CI: 70–79%). The Department of Thoracic Surgery, who conducted the present study, participates in the Society of Thoracic Surgeon’s General Thoracic Surgery Database (STS GTSD), which provides preoperative prediction models comparable to the ACS NSQIP reports (17). Thus, the score may perform better for participating hospitals in the USA.

None of the above-mentioned studies compared observed morbidity and mortality in patients with a risk score above and below average risk, which we think is a main function of the score, allowing practitioners to consider a predicted risk as acceptable. A 2018 Filipino study validated the score’s ability to predict below and above average risk, with major adverse cardiac events (MACE) as a primary outcome for non-cardiac surgeries and compared it to RCRI risk estimations (15). According to their results, the SRC provided a low sensitivity and an excellent discriminative ability for MACE (AUROC, 93%) for above and below average risk, where we observed acceptable results (AUROC, 73%). However, they do not report on “serious complication” and “any complication”. In these outcomes, we observed no significant differences (a small event prevalence for above average risk), suggesting poor discriminative ability. The patient cohort in this study was 84% ASA I, predominantly female (63%), and had a mean age of 54.58 years, all of which are associated with low morbidity and mortality (13).

Studies we found validating the ACS NSQIP SRC in the field of thoracic surgery considered cohorts limited to patients undergoing lung resection (10,11). Although, pulmonary resection may be the most common procedure performed in thoracic surgery, operations on the chest-wall, mediastinum, diaphragm, esophagus, pleura and pleura space, trachea, and traumatic injuries of chest-wall or organs within the chest are part of the daily work of any thoracic surgeon (6,18). We think that the ACS NSQIP SRC cannot be validated as a universal risk stratification tool in thoracic surgery based only on a specific range of surgical procedures.

A 2016 analysis of the European Society of Thoracic Surgeons database by Brunelli et al., suggested two models (i.e., EuroLung1 for morbidity and EuroLung2 for mortality) to predict outcomes following anatomic lung resections. Variables such as male sex, age, decreased predicted postoperative FEV1 (reduced in patients with COPD), BMI, and others were associated with 30-day morbidity and mortality risk (6). This corresponds to our analysis of the ACS NSQIP SRC risk factors. We observed that male sex, ASA, COPD, and BMI as individual risk factors predicted complications better than the ACS NSQIP SRC alone (Tables 1,4).

Explanations of findings

Our study evaluated the suitability of the ACS NSQIP SRC in predicting postoperative complications after general thoracic surgery. The AUROC for “serious complications”, “any complications”, and “return to OR” yielded values below the threshold for useful prediction of postoperative 30-day morbidity and mortality in our study cohort. As observed in the results, discrimination for “pneumonia” was fair, with a 95% CI ranging from poor to excellent. This might indicate that the SRC score was generally low in our study cohort, rather than a good calibration (Table 2). Moreover, restricting the analysis to lung resections, as suggested by other colleagues, did not improve the discriminative ability of the SRC (10,11).

Implications and actions needed

Although, the ACS NSQIP SRC is considered comparable to procedure-specific calculators (7), we think that the lack of specific risk factors for thoracic surgery led to a low sensitivity and specificity in our study cohort, and therefore, cannot consider it as a valid tool for routine clinical practice. As reported in the results, individual risk factors such as dyspnea, ASA, severe COPD, and BMI significantly improved the predictive accuracy of the model over the ACS NSQIP SRC alone. Our findings highlight the obvious need to further develop a risk prediction tool specifically for thoracic surgery.

Conclusions

The ACS NSQIP SRC exhibited low sensitivity, specificity, and low AUROC for postoperative 30-day morbidity and mortality in our study cohort. Such outcomes can mislead patients and surgeons in shared decision-making. Therefore, we think it cannot be considered as valid risk estimation tool for general thoracic surgery.

Acknowledgments

The authors would like to thank Cecile Buenter for editorial and language assistance and Dr. Christian Schindler, Swiss Tropical and Public Health Institute, University of Basel, for statistical assistance and advice.

Funding: This work was supported by the Department of Thoracic Surgery, University Hospital Basel (No. FO 126 102).

Footnote

Reporting Checklist: The authors have completed the STARD reporting checklist. Available at https://jtd.amegroups.com/article/view/10.21037/jtd-24-611/rc

Data Sharing Statement: Available at https://jtd.amegroups.com/article/view/10.21037/jtd-24-611/dss

Peer Review File: Available at https://jtd.amegroups.com/article/view/10.21037/jtd-24-611/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://jtd.amegroups.com/article/view/10.21037/jtd-24-611/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the local ethics committee Ethikkommission Nordwest-und Zentralschweiz (EKNZ) (No. BASEC 2022-02163, granted on December 15, 2022). All included patients signed the general research consent form of our institution.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. García-Miguel FJ, Serrano-Aguilar PG, López-Bastida J. Preoperative assessment. Lancet 2003;362:1749-57. [Crossref] [PubMed]
2. Agostini P, Cieslik H, Rathinam S, et al. Postoperative pulmonary complications following thoracic surgery: are there any modifiable risk factors? Thorax 2010;65:815-8. [Crossref] [PubMed]
3. Cohen MM, Duncan PG, Tate RB. Does anesthesia contribute to operative mortality? JAMA 1988;260:2859-63.
4. Iyer AP, Yadav S. Postoperative Care and Complications After Thoracic Surgery. In: Firstenberg MS (ed). Principles and Practice of Cardiothoracic Surgery. InTech; 2013. Available online: http://dx.doi.org/10.5772/55351
5. Ginsberg RJ. Lung cancer surgery: acceptable morbidity and mortality, expected results and quality control. Surg Oncol 2002;11:263-6. [Crossref] [PubMed]
6. Brunelli A, Salati M, Rocco G, et al. European risk models for morbidity (EuroLung1) and mortality (EuroLung2) to predict outcome following anatomic lung resections: an analysis from the European Society of Thoracic Surgeons database. Eur J Cardiothorac Surg 2017;51:490-7. [Crossref] [PubMed]
7. Bilimoria KY, Liu Y, Paruch JL, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg 2013;217:833-42.e1-3.
8. Fleisher LA, Fleischmann KE, Auerbach AD, et al. 2014 ACC/AHA guideline on perioperative cardiovascular evaluation and management of patients undergoing noncardiac surgery: executive summary: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 2014;130:2215-45. [Crossref] [PubMed]
9. Ingraham AM, Richards KE, Hall BL, et al. Quality improvement in surgery: the American College of Surgeons National Surgical Quality Improvement Program approach. Adv Surg 2010;44:251-67. [Crossref] [PubMed]
10. Samson P, Robinson CG, Bradley J, et al. The National Surgical Quality Improvement Program risk calculator does not adequately stratify risk for patients with clinical stage I non-small cell lung cancer. J Thorac Cardiovasc Surg 2016;151:697-705.e1. [Crossref] [PubMed]
11. Chudgar N, Yan S, Hsu M, et al. The American College of Surgeons Surgical Risk Calculator performs well for pulmonary resection: A validation study. J Thorac Cardiovasc Surg 2022;163:1509-1516.e1. [Crossref] [PubMed]
12. Dindo D, Demartines N, Clavien PA. Classification of surgical complications: a new proposal with evaluation in a cohort of 6336 patients and results of a survey. Ann Surg 2004;240:205-13. [Crossref] [PubMed]
13. Hosmer DW, Lemeshow S (2000) Applied Logistic Regression. Ed Ke-2. John Wiley and Sons, New York. Available online: https://doi.org/10.1002/0471722146
14. Parkin CJ, Moritz P, Kirkland O, et al. What is the Accuracy of the ACS-NSQIP Surgical Risk Calculator in Emergency Abdominal Surgery? A Meta-Analysis. J Surg Res 2021;268:300-7. [Crossref] [PubMed]
15. Yap MKC, Ang KF, Gonzales-Porciuncula LA, et al. Validation of the American College of Surgeons Risk Calculator for preoperative risk stratification. Heart Asia 2018;10:e010993. [Crossref] [PubMed]
16. Hers TM, Van Schaik J, Keekstra N, et al. Inaccurate Risk Assessment by the ACS NSQIP Risk Calculator in Aortic Surgery. J Clin Med 2021;10:5426. [Crossref] [PubMed]
17. Dyas AR, Bronsert MR, Henderson WG, et al. A comparison of the National Surgical Quality Improvement Program and the Society of Thoracic Surgery Cardiac Surgery preoperative risk models: a cohort study. Int J Surg 2023;109:2334-43. [Crossref] [PubMed]
18. Klepetko W, Aberg TH, Lerut AE, et al. Structure of general thoracic surgery in Europe. Eur J Cardiothorac Surg 2001;20:663-8. [Crossref] [PubMed]