A brief appraisal on the GRADE methodology and the development of recommendations on the management of prolonged air leaks by the PALAS Study Group
Editorial on Prolonged Air Leak after Lung Surgery: Prediction, Prevention and Management

A brief appraisal on the GRADE methodology and the development of recommendations on the management of prolonged air leaks by the PALAS Study Group

Francesco Zaraca1, Luca Bertolaccini2^; PALAS Study Group

1Department of Vascular and Thoracic Surgery, Regional Hospital, Bozen-Bolzano, Italy; 2Department of Thoracic Surgery, IEO, European Institute of Oncology IRCCS, Milan, Italy

^ORCID: 0000-0002-1153-3334.

*Collaborators from PALAS Study Group are listed in the Footnote.

Investigators of PALAS Study Group: Alessandro Brunelli, Department of Thoracic Surgery, St. James’s University Hospital, Leeds, UK; Amr Abdellateef, Department of Cardiothoracic Surgery, Mansoura University Hospital, Mansoura, Egypt; Firas Abu Akar, Department of Cardiothoracic Surgery, Al-Makassed Charitable Society Hospital, East Jerusalem, Palestine; Florian Augustin, Department of Visceral, Transplant, and Thoracic Surgery, Medical University of Innsbruck, Innsbruck, Austria; Tim Batchelor, Department of Thoracic Surgery, Bristol Royal Infirmary, Bristol, UK; Alessandro Bertani, Division of Thoracic Surgery and Lung Transplantation, Department for the Treatment and Study of Cardiothoracic Diseases and Cardiothoracic Transplantation, IRCCS ISMETT-UPMC, Palermo, Italy; Oriana Ciani, Centre for Research on Health and Social Care Management (CERGAS), SDA Bocconi School of Management, Milan, Italy; Roberto Crisci, Division of Thoracic Surgery, Thoracic Surgery Unit, University of L’Aquila, G. Mazzini Hospital, Teramo, Italy; Thomas D’Amico, Department of Surgery, Duke Cancer Institute, Durham, NC, USA; Xavier Benoit D’Journo, Department of Thoracic Surgery, Aix-Marseille Université, APHM, Hôpital Nord, Marseille, France; Wentao Fang, Department of Thoracic Surgery, Shanghai Chest Hospital, Shanghai Jiao Tong University, Shanghai, China; Miroslav Janík, Department of Thoracic Surgery, Slovak Medical University, University Hospital Bratislava, Bratislava, Slovakia; Marcelo Jiménez, Department of Thoracic Surgery, Salamanca University Hospital, Salamanca, Spain; Andreas Kirschbaum, Department of Visceral, Thoracic and Vascular Surgery, University of Marburg, Marburg, Germany; Marko Kostic, Clinic for Thoracic Surgery, University Clinical Center of Serbia, University of Belgrade, Belgrade, Serbia; Richard Lazzaro, Department of Cardiothoracic Surgery, Lenox Hill Hospital, New York, NY, USA; Marco Lucchi, Division of Thoracic Surgery, Department of Surgical, Medical and Molecular Pathology and Critical Care Medicine, University of Pisa, Pisa, Italy; Alessandro Marra, Department of Thoracic Surgery, Rems-Murr-Klinikum Winnenden, Winnenden, Germany; Sudish Murthy, Department of Thoracic and Cardiovascular Surgery, Heart, Vascular and Thoracic Institute, Cleveland Clinic, Cleveland, OH, USA; Calvin S. H. Ng, The Chinese University of Hong Kong, Prince of Wales Hospital, Hong Kong, China; Dania Nachira, Department of General Thoracic Surgery, Fondazione Policlinico Universitario A. Gemelli IRCCS, Università Cattolica del Sacro Cuore, Rome, Italy; René Horsleben Petersen, Department of Cardiothoracic Surgery, Copenhagen University Hospital, Rigshospitalet, Copenhagen, Denmark; Vadim Pischik, Department of Thoracic Surgery, St. Petersburg City Oncology Center, St. Petersburg State University, Saint Petersburg, Russia; Isabelle Opitz, Department of Thoracic Surgery, University Hospital Zurich, Zurich, Switzerland; Lorenzo Spaggiari, Department of Thoracic Surgery, IEO, European Institute of Oncology IRCCS, Milan, Italy; Department of Oncology and Hemato-Oncology, University of Milan, Milan, Italy; Paula A. Ugalde, Department of Thoracic Surgery, Institut Universitaire de Pneumologie et Cardiologie de Quebec, Quebec, QC, Canada; Fernando Vannucci, Thoracic Surgery Unit, Hospital Pró-Cardíaco-Botafogo, Rio de Janeiro, Brazil; Thoracic Surgery Unit, Hospital Central da Polícia Militar, Rio de Janeiro, Brazil.

Correspondence to: Luca Bertolaccini, MD, PhD, FCCP. Department of Thoracic Surgery, IEO, European Institute of Oncology IRCCS, Via Ripamonti 435, 20141 Milan, Italy. Email: luca.bertolaccini@gmail.com.

Submitted Apr 13, 2022. Accepted for publication Jul 18, 2022. Published online Aug 04, 2022.

doi: 10.21037/jtd-22-498


Introduction

Reduction of practice variation in prevention and management of prolonged air leak (PAL) after pulmonary resection remains one of the most critical issues to be solved, especially in the era of Enhanced Recovery After Surgery (ERAS). The lack of evidence-based guidelines on PAL burdens the routine activity of the thoracic surgeon, especially in the younger generation. Recently, the PALAS Study Group, through an international 3-round Delphi survey, reached an agreement on some relevant issues concerning the detection and management of intraoperative alveolar air leak and PAL but failed on others (1).

Therefore, it is a top priority for the PALAS Study Group to provide new generations of thoracic surgeons with clear and easily applicable recommendations on preventing, detecting, and managing PAL after pulmonary resection. Here, we describe the methodology that we will use.


What is GRADE methodology?

Grading of Recommendations, Assessment, Development and Evaluation (GRADE) is a tool that systematic reviewers and guideline developers use to evaluate the quality of evidence and determine whether to propose an intervention (1). GRADE is distinct from previous appraisal tools. It separates evidence quality and recommendation strength and assesses the quality of evidence for each outcome. Lastly, observational studies can be upgraded if they meet specified criteria. The GRADE technique consists of five distinct steps.

  • Step 1: assign randomised controlled trials a high ranking and observational studies a low ranking a priori. Randomised controlled trials receive a better beginning grade than observational research because they are typically less prone to bias.
  • Step 2: diminish or increase the initial rating. It is not unusual for randomised controlled trials and observational research to be downgraded due to detectable bias. Additionally, observational studies can be enhanced when high-quality research demonstrates consistent findings.
  • Step 3: for each crucially significant outcome, provide a final rating of the evidence’s quality as high, moderate, poor, or very low (Table 1). Upgrades and downgrades necessitate sound judgment and consideration. As a general guideline, one category per issue should be moved up or down. For example, the quality of evidence for an outcome studied in randomised controlled trials may begin as high and decrease to moderate due to a high risk of bias in the included studies, then to low if there was significant unexplained heterogeneity between the trials, and finally to very low if there were few events leading to confidence intervals that included both significant benefits and harms (Table 2).
  • Step 4: consider other variables that affect the intensity of a suggestion for a course of action. A strong recommendation does not always follow high-quality evidence. Apart from the quality of evidence, recommendations must consider other variables. The first consideration is the balance of desired and unintended consequences. Specific therapies, such as antibiotics prophylaxis to prevent infections associated with certain thoracic operations, have benefits and few adverse effects, making a solid recommendation uncontroversial. When the benefit-harm ratio is unclear, it is critical to examine patient values and preferences and expenses carefully.
  • Step 5: provide a recommendation that is either strong or weak. When the treatment’s net benefit is evident, patient values and circumstances are unlikely to influence the decision to pursue treatment, and a strong recommendation is justified (3).

Table 1

Grade the overall certainty of evidence (2)

Grade Quality level Definition
A High We are confident that the actual impacts closely match the estimated effects
B Moderate The true impacts are likely to be similar to the estimates, but there is a chance that they are significantly different
C Low The actual impacts may differ significantly from the estimated effects
D Very low The estimates are quite uncertain and frequently will be wildly inaccurate

Table 2

Method of rating the certainty of the evidence for an outcome (2)

Step 1: starting grade according to study design
   Randomised trials = high
   Observational studies = low
Step 2: lower if
   Risk of bias
    Serious
    Very serious
   Inconsistency
    Serious
    Very serious
   Indirectness
    Serious
    Very serious
   Imprecision
    Serious
    Very serious
   Publication bias
    Serious
    Very serious
Step 3: higher if
   Large effect
    Large
    Very large
   Dose response
    Evidence of a gradient
   All plausible confounding
    Would reduce a demonstrated effect
    Would suggest a spurious effect when results show no effect
Step 4: determine final grade for quality of evidence
   High
   Moderate
   Low
   Very low

The GRADE summary of findings tables provides quick access to the most current evidence. To serve patients, as mentioned previously, a thoroughly informed dialogue about the benefits, risks, and practical implications of choosing one alternative over another is necessary. Utilising the medical literature to resolve patient concerns necessitates the transformation of these interactions into structured clinical inquiries. To provide satisfactory answers to clinical concerns, strong systematic reviews and appropriate meta-analyses are required that convey the best available data in easily digestible formats, rather than relying on selective interpretation of the literature. GRADE’s solution is to summarise findings tables in systematic reviews and guidelines. The tables of summary findings include vital outcomes, the number of studies and patients, the relative and absolute impacts, the degree of certainty (quality) of the findings, and straightforward language (4). Frequently, surrogate results do not transfer into an actual benefit (5,6). Instead, GRADE users place a premium on patient-centred outcomes. As opposed to important but not critical outcomes, defining crucial outcomes for patients can be accomplished through formal processes such as systematic reviews of pertinent literature and patient focus groups. Patient-important outcomes are classified into two categories: binary and continuous. Dichotomous outcomes refer to events that can occur or do not occur. The occurrence of such events in treated and untreated individuals could be assessed in relative terms using risk ratios or odds ratios or in absolute terms using hazard ratios for time-to-event outcomes (survival analysis) (4).


Other techniques

Other techniques for developing evidence-based clinical guidelines have been created in the past. The Scottish Intercollegiate Guidelines Network (SIGN) was established primarily to address effectiveness issues. The SIGN method evaluates the quality of evidence-based on the type of study in which it was published and places a premium on trial design rather than trial quality. SIGN assigns quality to studies based on their design, with little regard for their outcomes or publishing bias. For instance, using the SIGN classification approach, even the most poorly conducted randomised controlled trials can be classed as valuable, and any recommendations based on this evidence are scored as strong. While SIGN is subjective, the categorisation system requires that all recommendations from randomised controlled trials or meta-analyses of randomised controlled trials obtain a high-quality grade (even if there is a substantial probability of bias), making the method reasonably reproducible. On the contrary, the GRADE system considers additional factors when making recommendations, but they are not included in assessing the quality of evidence.


Why GRADE methodology?

Medicine is a challenging profession to practice. All clinicians battle every day and throughout their careers with fiduciary responsibilities. They can, however, refer to recognised principles of optimal clinical decision-making, several of which are related to evidence-based practice. GRADE is a critical component of evidence-based medicine (EBM). It is widely recognised as the most acceptable approach for summarising and interpreting data and guiding decision-making (4).

In 2004, a panel of international specialists in methodology and practice guidelines produced the initial version of the GRADE technique to assess the quality of evidence supporting medical interventions and develop recommendations. Since then, the group has released a six-part series aimed at clinicians who use GRADE guidelines and a series of papers aimed at assisting writers of systematic reviews and guideline groups that use GRADE in their work. Since its inception, over 100 organisations worldwide have endorsed or implemented GRADE, including the World Health Organization, the Cochrane Collaboration, the Joanna Briggs Institute, the American College of Physicians, DynaMed Plus, and UpToDate. Any physician employing formal guidelines or recommendations in online literature such as UpToDate is now unlikely to escape encountering GRADE suggestions. The GRADE approach’s popularity stems from its numerous advantages: establishing rigorous and comprehensive criteria for assessing the quality of evidence (also known as certainty or confidence in evidence); assessing the quality of evidence using systematic summaries of the entire body of relevant studies rather than selected individual studies; assessing the quality of evidence for each relevant outcome; progressing from evidence to recommendations, evaluating the importance of outcomes from a patient perspective, including both benefit and harm (clinicians, patients and policymakers) (7).


The limitations of GRADE methodology

An often-noted shortcoming of GRADE standards is that they are narrowly focused on a subset of the broader area. GRADE always prioritises the most rigorous evidence available for each outcome. This may be different for beneficial outcomes (randomised controlled trials are more likely to explore) than for uncommon unfavourable effects (requiring observational studies). GRADE does not rely on unsupported opinions. Third, many suggestions in many guidelines are conditional, which might be a source of frustration. The issue is not with GRADE but with the low quality of evidence, which makes decisions (extra) susceptible to patient values and preferences and other contextual factors. Conditional recommendations are subject to change considering new evidence. Thus, guidelines are crucial in identifying research gaps and driving the generation of new practice-changing data that would justify a guideline update (4). In some situations, the grade for the internal validity of a study could properly lower the overall grade by three grades (e.g., from high to very low) as opposed to the two-grade decline advised by the GRADE system (8).

On the other hand, GRADE is less concerned with the type of study and evaluates numerous aspects of the available data when determining the quality. Nonetheless, the GRADE system is time-consuming since it requires users to examine various factors when assessing quality, subject to some degree of subjectivity. Thus, one legitimate critique of the GRADE system is that it looks to have been established by academics for academics. If the greater complexity of GRADE in comparison to other approaches prevents clinical guideline developers from comprehending the process, GRADE will be unable to achieve its goal of standardisation. One possible option is to develop alternate techniques for displaying data to make the recommendations more user-friendly for surgeons, who are the ultimate users. Surgeons may find the guidelines less complicated and easier to comprehend if merely the quality assessment outcome was published. This suggestion was created following a rigorous evaluation utilising the GRADE approach (9).


Conclusions

Clinicians make several daily decisions with their patients on the use of tests or therapies. The typical approach to these decisions is based on the clinician’s training and expertise. However, applying EBM concepts to comprehend and use evidence about management options and knowledge of the critical nature of patients’ values and preferences are required for effective patient care. Clinicians employing GRADE-compliant guidelines will examine the suggestions’ applicability to the patient in front of them. Clinicians working with developing evidence and/or limited solid advice will assist patients best by utilising the GRADE domains (4). GRADE is expected to play a variety of functions in the broader ecology of evidence. Innovative approaches may improve the efficiency of systematic reviews and GRADE applications. The rapid recommendations collaboration between the British Medical Journal and the Making GRADE the Irresistible Choice (MAGIC) group is an example of this initiative, which uses GRADE methodology to rapidly generate a small number of recommendations addressing a specific clinical context in response to potentially practice-changing randomised controlled trials. Additional GRADE-related activities involve patient engagement in evaluating the evidence and providing recommendations (7). For instance, the Italian Association of Medical Oncologist (AIOM) Clinical Practice Guidelines were developed in accordance to the GRADE methodology (10-12). Nonetheless, obstacles persist. To fully exploit the benefits of GRADE, it will be necessary to improve GRADE’s understanding and application in guideline or recommendations creation and at the bedside. Thoracic surgeons’ associations should endeavour to strengthen their members’ understanding and application of GRADE concepts, and those responsible for developing clinical practice guidelines should frequently incorporate the GRADE approach. Clinicians committed to providing the best possible care for their patients will incorporate GRADE principles into their daily practice.


Acknowledgments

Funding: This work was partially supported by the Italian Ministry of Health with Ricerca Corrente and 5x1000 funds.


Footnote

Provenance and Peer Review: This article was commissioned by the editorial office, Journal of Thoracic Disease, for the series “Prolonged Air Leak after Lung Surgery: Prediction, Prevention and Management”. The article has undergone external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://jtd.amegroups.com/article/view/10.21037/jtd-22-498/coif). The special series “Prolonged Air Leak after Lung Surgery: Prediction, Prevention and Management” was sponsored by Bard Limited. Bard Limited has no interference on the contents of the special series. FZ served as the unpaid Guest Editor of the series. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.


References

  1. Zaraca F, Brunelli A, Pipitone MD, et al. A Delphi Consensus report from the "Prolonged Air Leak: A Survey" study group on prevention and management of postoperative air leaks after minimally invasive anatomical resections. Eur J Cardiothorac Surg 2022;62:ezac211. [Crossref] [PubMed]
  2. Balshem H, Helfand M, Schünemann HJ, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011;64:401-6. [Crossref] [PubMed]
  3. Goldet G, Howick J. Understanding GRADE: an introduction. J Evid Based Med 2013;6:50-4. [Crossref] [PubMed]
  4. Chu DK, Golden DBK, Guyatt GH. Translating Evidence to Optimize Patient Care Using GRADE. J Allergy Clin Immunol Pract 2021;9:4221-30. [Crossref] [PubMed]
  5. Ciani O, Buyse M, Garside R, et al. Comparison of treatment effect sizes associated with surrogate and final patient relevant outcomes in randomised controlled trials: meta-epidemiological study. BMJ 2013;346:f457. [Crossref] [PubMed]
  6. Ciani O, Federici CB. Value Lies in the Eye of the Patients: The Why, What, and How of Patient-reported Outcomes Measures. Clin Ther 2020;42:25-33. [Crossref] [PubMed]
  7. Alhazzani W, Guyatt G. An overview of the GRADE approach and a peek at the future. Med J Aust 2018;209:291-2. [Crossref] [PubMed]
  8. Malmivaara A. Methodological considerations of the GRADE method. Ann Med 2015;47:1-5. [Crossref] [PubMed]
  9. Bertolaccini L, Rocco G. The forest of methodology and the writing of evidence-based medicine papers. Eur J Cardiothorac Surg 2018;54:615-21. [Crossref] [PubMed]
  10. Facchinetti F, Pilotto S, Metro G, et al. Treatment of metastatic non-small cell lung cancer: 2018 guidelines of the Italian Association of Medical Oncology (AIOM). Tumori 2019;105:3-14. [Crossref] [PubMed]
  11. Passiglia F, Bertolaccini L, Del Re M, et al. Diagnosis and treatment of early and locally advanced non-small-cell lung cancer: The 2019 AIOM (Italian Association of Medical Oncology) clinical practice guidelines. Crit Rev Oncol Hematol 2020;148:102862. [Crossref] [PubMed]
  12. Passiglia F, Pilotto S, Facchinetti F, et al. Treatment of advanced non-small-cell lung cancer: The 2019 AIOM (Italian Association of Medical Oncology) clinical practice guidelines. Crit Rev Oncol Hematol 2020;146:102858. [Crossref] [PubMed]
Cite this article as:Zaraca F, Bertolaccini L; on behalf of PALAS Study Group. A brief appraisal on the GRADE methodology and the development of recommendations on the management of prolonged air leaks by the PALAS Study Group. J Thorac Dis 2023;15(2):839-844. doi: 10.21037/jtd-22-498

Download Citation