The quality of systematic reviews on the treatment of stage I non-small cell lung cancer for individualized decision-making: improved but further refinements are needed
Editorial

The quality of systematic reviews on the treatment of stage I non-small cell lung cancer for individualized decision-making: improved but further refinements are needed

Yajia Sun1, Qianling Shi2, Anya Shi3, Yaolong Chen1,4,5,6

1School of Public Health, Lanzhou University, Lanzhou, China; 2The First School of Clinical Medicine, Lanzhou University, Lanzhou, China; 3The Second Clinical Medical Hospital, Lanzhou University, Lanzhou, China; 4Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences, Lanzhou University, Lanzhou, China; 5Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China; 6Lanzhou University Institute of Health Data Science, Lanzhou, China

Correspondence to: Yaolong Chen. Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, No. 199 Donggang West Road, Lanzhou 730000, China. Email: chevidence@lzu.edu.cn.

Submitted Jul 17, 2022. Accepted for publication Sep 09, 2022.

doi: 10.21037/jtd-22-1128

Lung cancer has a high incidence and mortality rate worldwide, with the vast majority of patients having non-small cell lung cancer (NSCLC), of which nearly one-third of patients are in an early stage (1,2). The American College of Chest Physicians (ACCP), the National Comprehensive Cancer Network (NCCN) and other institutions and organizations have published several clinical practice guidelines for stage I and II NSCLC (3-5) addressing diagnosis, treatment and management. In light of the trend towards diagnosis of early-stage NSCLC at a younger age, as well as evolving treatment modalities, Professor Detterbeck and colleagues recently published a management guide on treatment options for stage I NSCLC that provides protocols and frameworks for clinicians and patients seeking to individualize clinical decisions, (6) based on thorough research evidence including the three systematic reviews archived in the same series (7-9). To assess the methodological and reporting quality of the systematic reviews underpinning the decision-making framework (6), we used A MeaSurement Tool to Assess Systematic Reviews (AMSTAR) 2 (10) and The Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) 2020 (11). Overall, the results of both AMSTAR 2 and PRISMA 2020 were improved compared to previous systematic reviews of lung cancer treatments (12).

In terms of methodological quality, this series of systematic reviews met the majority of AMSTAR 2 criteria. Among the 13 relevant domains of AMSTAR 2 (three additional domains relate to meta-analysis which was not performed), seven domains were assessed with “Yes” or “Partial yes” (Table 1). These systematic reviews present clear research questions, study inclusion and exclusion criteria, as well as the process of literature retrieval, screening and evaluation. The review authors appropriately present a qualitative synthesis of the results, quality appraisal of included studies, and reach well-founded summaries. Actually, no matter whether performed the meta-analysis, systematic review should consider the possible impact of the risk of bias of the included studies when interpreting and discussing the research results. Compared with randomized controlled trials (RCTs), non-randomized comparisons (NRCs) may lead to inaccurate effect estimates due to confounders (both known and unknown) that may differ among treatment groups. Therefore, this series of systematic reviews (includes both RCTs and NRCs) deserves recognition for the improvement in the key area of completing the consideration of the methodological quality of included studies when forming conclusions.

Table 1

AMSTAR 2 assessment results of the series systematic reviews regarding the guide for managing patients with stage I NSCLC

AMSTAR 2 Part 2 of the guide Part 3 of the guide Part 4 of the guide
1. Did the research questions and inclusion criteria for the review include the components of PICO? Yes Yes Yes
2. Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol? No No No
3. Did the review authors explain their selection of the study designs for inclusion in the review? No No No
4. Did the review authors use a comprehensive literature search strategy? Partial yes Partial yes Partial yes
5. Did the review authors perform study selection in duplicate? Yes Yes Yes
6. Did the review authors perform data extraction in duplicate? No No No
7. Did the review authors provide a list of excluded studies and justify the exclusions? No No No
8. Did the review authors describe the included studies in adequate detail? Yes Yes Yes
9. Did the review authors use a satisfactory technique for assessing the RoB in individual studies that were included in the review? Yes Yes Yes
10. Did the review authors report on the sources of funding for the studies included in the review? No No No
11. If meta-analysis was performed did the review authors use appropriate methods for statistical combination of results? No meta-analysis conducted No meta-analysis conducted No meta-analysis conducted
12. If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis? No meta-analysis conducted No meta-analysis conducted No meta-analysis conducted
13. Did the review authors account for RoB in individual studies when interpreting/discussing the results of the review? Yes Yes Yes
14. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review? No No No
15. If they performed quantitative synthesis did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review? No meta-analysis conducted No meta-analysis conducted No meta-analysis conducted
16. Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review? Yes Yes Yes

NSCLC, non-small cell lung cancer; AMSTAR 2, A MeaSurement Tool to Assess Systematic Reviews 2; PICO, population, intervention, control group, outcome; RoB, risk of bias.

For reporting quality, among the 27 items in PRISMA 2020, a total of 22 items (81.5%) were evaluated as “reported” (Table 2). The researchers clearly presented the review purpose, the study inclusion and exclusion criteria, sources of literature, search strategies, processes for literature screening and data extraction, methods used for variable adjustment, and they specified the risk of bias assessment tool used in these systematic reviews. This detailed description of the methods used promotes transparency and replicability.

Table 2

PRISMA 2020 assessment results of the series systematic reviews regarding the guide for managing patients with stage I NSCLC

PRISMA 2020 Part 2 of the guide Part 3 of the guide Part 4 of the guide
1. Identify the report as a systematic review Yes Yes Yes
2. See the PRISMA 2020 for Abstracts checklist Yes Yes Yes
3. Describe the rationale for the review in the context of existing knowledge Yes Yes Yes
4. Provide an explicit statement of the objective(s) or question(s) the review addresses Yes Yes Yes
5. Specify the inclusion and exclusion criteria for the review and how studies were grouped for the syntheses Yes Yes Yes
6. Specify all databases, registers, websites, organisations, reference lists and other sources searched or consulted to identify studies. Specify the date when each source was last searched or consulted Yes Yes Yes
7. Present the full search strategies for all databases, registers and websites, including any filters and limits used Yes Yes Yes
8. Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record and each report retrieved, whether they worked independently, and if applicable, details of automation tools used in the process Yes Yes Yes
9. Specify the methods used to collect data from reports, including how many reviewers collected data from each report, whether they worked independently, any processes for obtaining or confirming data from study investigators, and if applicable, details of automation tools used in the process Yes Yes Yes
10a. List and define all outcomes for which data were sought. Specify whether all results that were compatible with each outcome domain in each study were sought (e.g., for all measures, time points, analyses), and if not, the methods used to decide which results to collect Yes Yes Yes
10b. List and define all other variables for which data were sought (e.g., participant and intervention characteristics, funding sources). Describe any assumptions made about any missing or unclear information Partial Yes Partial Yes Partial Yes
11. Specify the methods used to assess risk of bias in the included studies, including details of the tool(s) used, how many reviewers assessed each study and whether they worked independently, and if applicable, details of automation tools used in the process Yes Yes Yes
12. Specify for each outcome the effect measure(s) (e.g., risk ratio, mean difference) used in the synthesis or presentation of results No No No
13a. Describe the processes used to decide which studies were eligible for each synthesis [e.g., tabulating the study intervention characteristics and comparing against the planned groups for each synthesis (item #5)] No No No
13b. Describe any methods required to prepare the data for presentation or synthesis, such as handling of missing summary statistics, or data conversions Yes Yes Yes
13c. Describe any methods used to tabulate or visually display results of individual studies and syntheses Yes Yes Yes
13d. Describe any methods used to synthesize results and provide a rationale for the choice(s). If meta-analysis was performed, describe the model(s), method(s) to identify the presence and extent of statistical heterogeneity, and software package(s) used Yes Yes Yes
13e. Describe any methods used to explore possible causes of heterogeneity among study results (e.g., subgroup analysis, meta-regression) No No No
13f. Describe any sensitivity analyses conducted to assess robustness of the synthesized results No No No
14. Describe any methods used to assess risk of bias due to missing results in a synthesis (arising from reporting biases) No No No
15. Describe any methods used to assess certainty (or confidence) in the body of evidence for an outcome Partial yes Partial yes Partial yes
16a. Describe the results of the search and selection process, from the number of records identified in the search to the number of studies included in the review, ideally using a flow diagram Yes Yes Yes
16b. Cite studies that might appear to meet the inclusion criteria, but which were excluded, and explain why they were excluded No No No
17. Cite each included study and present its characteristics Yes Yes Yes
18. Present assessments of risk of bias for each included study Yes Yes Yes
19. For all outcomes, present, for each study: (I) summary statistics for each group (where appropriate) and (II) an effect estimate and its precision (e.g., confidence/credible interval), ideally using structured tables or plots Yes Yes Yes
20a. For each synthesis, briefly summarise the characteristics and risk of bias among contributing studies Yes Yes Yes
20b. Present results of all statistical syntheses conducted. If meta-analysis was done, present for each the summary estimate and its precision (e.g., confidence/credible interval) and measures of statistical heterogeneity. If comparing groups, describe the direction of the effect Yes Yes Yes
20c. Present results of all investigations of possible causes of heterogeneity among study results No No No
20d. Present results of all sensitivity analyses conducted to assess the robustness of the synthesized results No No No
21. Present assessments of risk of bias due to missing results (arising from reporting biases) for each synthesis assessed No No No
22. Present assessments of certainty (or confidence) in the body of evidence for each outcome assessed Partial yes Partial yes Partial yes
23a. Provide a general interpretation of the results in the context of other evidence Yes Yes Yes
23b. Discuss any limitations of the evidence included in the review Yes No Yes
23c. Discuss any limitations of the review processes used Yes Yes Yes
23d. Discuss implications of the results for practice, policy, and future research Yes Yes Yes
24a. Provide registration information for the review, including register name and registration number, or state that the review was not registered No No No
24b. Indicate where the review protocol can be accessed, or state that a protocol was not prepared No No No
24c. Describe and explain any amendments to information provided at registration or in the protocol No No No
25. Describe sources of financial or non-financial support for the review, and the role of the funders or sponsors in the review Yes Yes Yes
26. Declare any competing interests of review authors Yes Yes Yes
27. Report which of the following are publicly available and where they can be found: template data collection forms; data extracted from included studies; data used for all analyses; analytic code; any other materials used in the review No No No

Yes = reported; Partial yes = partial reported; No = not reported. NSCLC, non-small cell lung cancer; PRISMA 2020, The Preferred Reporting Items for Systematic Reviews and Meta-analyses 2020.

Nonetheless, there are some shortcomings in this series of systematic reviews. Firstly, these reviews did not describe a research protocol or proposal. As a critical domain of AMSTAR 2 and an important item in PRISMA, the first step in formulating a systematic review, a research protocol or proposal can help reduce bias in study conduct and reporting, improve research quality, diminish publication bias, and improve the credibility of conclusions (13). In addition, the prospective registration of research protocols can facilitate cooperation and collaboration across systematic review teams. In 2010, based on the major changes between research protocols and published content of Cochrane systematic reviews as well as problems such as excessive duplication between systematic reviews, some scholars proposed that systematic reviews should be registered to reduce publication bias and avoid duplication of work. Studies have shown that prospectively registered systematic reviews have a longer production cycle and higher methodological quality than unregistered systematic reviews (14,15). And therefore, AMSTAR 2 and PRISMA 2020 have also emphasized in their respective domains or items, involving research protocols and its explanations of the details of any amendments to the information provided in the registration or protocol.

Secondly, although this series of systematic reviews reported the funding sources for the reviews themselves, they did not present the funding sources for the included studies. Funders can potentially have an important impact on the technical quality and credibility of systematic reviews and their conclusions. PRISMA 2020 and AMSTAR 2 both require authors to describe all sources of financial and non-financial support for the review, and the role of the funders or sponsors when drafting the systematic review. AMSTAR 2 also requires researchers to report funding information on the studies included in the review. While funding for systematic reviews and other research is essential and engenders important collaborations, funders have been demonstrated to bias results and conclusions in favor of funders’ interests (16). A 2003 exploratory study of the impact of financial conflicts of interest on biomedical research reported that industry-funded research tends to have outcomes favorable to the funding company (17). In the same year, a systematic review of pharmaceutical industry sponsorship of research showed that studies that did not report industry funding had poor methodological quality (18). In response, CONSORT 2010 added a new item addressing the need for researchers to report all funding sources for clinical trials, and the role of funders. Although there are reported inaccuracies in reporting of funders for RCTs (19) and the association between funders’ interests and study conclusions cannot be proven as causal, clear and explicit reporting could help readers critically judge research findings. Similarly, as a synthesis of evidence from original research, systematic reviews should include this information also to help the reader assess the validity and credibility of the review’s conclusions.

Lastly, this series of systematic reviews lacks several other important elements of high-quality systematic reviews. The rationale for the selection of specific study designs for inclusion is unclear; data extraction was not reported as verified; and the methods used to explore possible causes of heterogeneity among studies are not described. In addition, sensitivity analyses to evaluate the stability of pooled results are not reported and adequately explained. The selection between RCTs and NRCs when they addressed the same question is necessary for the author to consider the completeness of conclusions. The duplication of data extraction is necessary to ensure the accuracy of results as well, especially for NRCs (due to the complicated data and potential confounding) (10). Besides, the description of the heterogeneity method is needed to show apparently and let readers assess the appropriateness of the explanation of results even though the systematic review did not conduct the meta-analysis (11).

In conclusion, the management guide by Detterbeck and colleagues provides an important clinical decision framework for the choices of treatment modality for patients with stage I NSCLC. This decision framework is based on systematic reviews, whose methodological and reporting quality have greatly improved compared with previous reviews. However, several important aspects of high-quality systematic reviews are lacking and these missing elements may potentially impact the reviews’ conclusion. In order to present clear, transparent and valid conclusions in systematic reviews and related clinical guidance, we sincerely recommend that future researchers in this field strictly adhere to the highest standards for development methods and reporting. The AMSTAR instruments and the PRISMA reporting checklist are extremely useful for assessing the quality and ensuring complete reporting of systematic reviews. Authors of systematic reviews should pay attention to the important steps (including selection rationale, duplicate data extraction, heterogeneity and sensitivity approaches, etc.) which are often missed when conducting research, and strive to make high-quality reviews to enrich and inspire clinical practice. Besides, systematic reviewers and developers of clinical guidance need to pay particular attention to: (I) a detailed research protocol, with prospective registration on relevant registration platforms or publications; and (II) funding and other support for primary research included in the review, and for the review itself. Using such rigorous and transparent approaches for systematic reviews helps to ensure that resulting clinical guidance is of the highest quality for patients and healthcare providers.

Acknowledgments

Susan L. Norris provided editorial assistance.

Funding: None.

Footnote

Provenance and Peer Review: This article was commissioned by the editorial office, Journal of Thoracic Disease. The article did not undergo external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://jtd.amegroups.com/article/view/10.21037/jtd-22-1128/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

  1. World Health Organization. WHO Cancer Regional Profile 2020 [DB/OL]. [2022-08-13]. Available online: https://www.who.int/health-topics/cancer#tab=tab_1
  2. Duan J, Tan F, Bi N, et al. Expert consensus on perioperative treatment for non-small cell lung cancer. Transl Lung Cancer Res 2022;11:1247-67. [Crossref] [PubMed]
  3. Scott WJ, Howington J, Feigenberg S, et al. Treatment of non-small cell lung cancer stage I and stage II: ACCP evidence-based clinical practice guidelines (2nd edition). Chest 2007;132:234S-42S.
  4. Howington JA, Blum MG, Chang AC, et al. Treatment of stage I and II non-small cell lung cancer: Diagnosis and management of lung cancer, 3rd ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest 2013;143:e278S-313S.
  5. Schneider BJ, Daly ME, Kennedy EB, et al. Stereotactic Body Radiotherapy for Early-Stage Non-Small-Cell Lung Cancer: American Society of Clinical Oncology Endorsement of the American Society for Radiation Oncology Evidence-Based Guideline. J Clin Oncol 2018;36:710-9. [Crossref] [PubMed]
  6. Detterbeck FC, Blasberg JD, Woodard GA, et al. A guide for managing patients with stage I NSCLC: deciding between lobectomy, segmentectomy, wedge, SBRT and ablation-part 1: a guide to decision-making. J Thorac Dis 2022;14:2340-56. [Crossref] [PubMed]
  7. Detterbeck FC, Mase VJ Jr, Li AX, et al. A guide for managing patients with stage I NSCLC: deciding between lobectomy, segmentectomy, wedge, SBRT and ablation-part 2: systematic review of evidence regarding resection extent in generally healthy patients. J Thorac Dis 2022;14:2357-86. [Crossref] [PubMed]
  8. Bade BC, Blasberg JD, Mase VJ Jr, et al. A guide for managing patients with stage I NSCLC: deciding between lobectomy, segmentectomy, wedge, SBRT and ablation-part 3: systematic review of evidence regarding surgery in compromised patients or specific tumors. J Thorac Dis 2022;14:2387-411. [Crossref] [PubMed]
  9. Park HS, Detterbeck FC, Madoff DC, et al. A guide for managing patients with stage I NSCLC: deciding between lobectomy, segmentectomy, wedge, SBRT and ablation-part 4: systematic review of evidence involving SBRT and ablation. J Thorac Dis 2022;14:2412-36. [Crossref] [PubMed]
  10. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017;358:j4008. [Crossref] [PubMed]
  11. Page MJ, Moher D, Bossuyt PM, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ 2021;372: [Crossref] [PubMed]
  12. Xiang Y, Zhang M, Gao J, et al. Quality evaluation of Chinese literatures on systematic review and Meta-analysis of lung cancer treatment literature. China Journal of Traditional Chinese Medicine and Pharmacy 2015;30:1216-8.
  13. Ge L, Tian JH, Li YN, et al. Association between prospective registration and overall reporting and methodological quality of systematic reviews: a meta-epidemiological study. J Clin Epidemiol 2018;93:45-55. [Crossref] [PubMed]
  14. Allers K, Hoffmann F, Mathes T, et al. Systematic reviews with published protocols compared to those without: more effort, older search. J Clin Epidemiol 2018;95:102-10. [Crossref] [PubMed]
  15. Sideri S, Papageorgiou SN, Eliades T. Registration in the international prospective register of systematic reviews (PROSPERO) of systematic review protocols was associated with increased review quality. J Clin Epidemiol 2018;100:103-10. [Crossref] [PubMed]
  16. Lundh A, Lexchin J, Mintzes B, et al. Industry sponsorship and research outcome. Cochrane Database Syst Rev 2017;2:MR000033. [PubMed]
  17. Bekelman JE, Li Y, Gross CP. Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA 2003;289:454-65. [Crossref] [PubMed]
  18. Lexchin J, Bero LA, Djulbegovic B, et al. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 2003;326:1167-70. [Crossref] [PubMed]
  19. Hakoum MB, Jouni N, Abou-Jaoude EA, et al. Characteristics of funding of clinical trials: cross-sectional survey and proposed guidance. BMJ Open 2017;7:e015997. [Crossref] [PubMed]
Cite this article as: Sun Y, Shi Q, Shi A, Chen Y. The quality of systematic reviews on the treatment of stage I non-small cell lung cancer for individualized decision-making: improved but further refinements are needed. J Thorac Dis 2022;14(10):3660-3666. doi: 10.21037/jtd-22-1128

Article Options

Download Citation

Share