Time to get real: critical and imperative change required in evidence evaluation

The central theme of this special issue of the Journal of Thoracic Disease (JTD) is real-world effectiveness research (observational studies and pragmatic trials) in the context of respiratory disease.

There has been growing recognition in recent years of the need for real-world methodologies and the value of real-life evidence to supplement and complement the existing safety and efficacy evidence reported in registration randomized controlled trials (RCTs). The European Respiratory Society/American Thoracic Society Taskforce paper on asthma control and exacerbations was one of the first formal papers in this space (1). The Taskforce proposed the use of composite measures when evaluating asthma control and called for measurement properties to be validated in clinical trials and in “large, prospective studies in ‘real-world’ settings (e.g., trials designed pragmatically to reflect everyday clinical practice) to ensure they provide content validity as well as reflect clinically meaningful outcomes”. The Cochrane Collaboration also called for this broader approach to evidence evaluation when reviewing the effects of ICS use on linear growth in children, recognising the need for longer outcome periods than offered by traditional RCTs, specifically advising: “research efforts should concentrate on evaluating the long-term effects of inhaled steroids” (2). Around the same time, the Brussels Declaration on Asthma echoed the sentiment by stating in 1 of its 10 key points that there is a need to: “include evidence from real world studies in treatment guidelines,” (3) and Sir Michael Rawlins, the Chairman of the United Kingdom’s National Institute for Health and Care Excellence (NICE) (4), added his voice to the debate by suggesting that RCTs should be complemented by a “diversity of approaches that involve analyzing the totality of the evidence base”. Sir Rawlins further argued that RCTs “often miss the value of a therapeutic intervention and tend to be carried out in specific types of patients for relatively short periods of time”. More recently, the Global Initiative for Asthma (GINA) highlighted the need to evaluate patients’ risk as a core part of disease management and listed a number of real-world factors among key potential risks, e.g., comorbidities, inhaler technique and medication adherence (5).

These high-level calls for a more inclusive and integrated approach to evidence synthesis grew out of the clinic and a very real clinical need. Practicing clinicians (particularly primary care clinicians) were struggling to apply a one-size-fits-all guideline approach. In the clinic, they were not faced with one standard patient, but a plethora of different patient types arising from complex interactions of real-life factors: demographic (age, weight), lifestyle (e.g., socioeconomic status, smoking status), behavioural (medication implementation; inhaler technique) and clinical (e.g., comorbid conditions).

The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) system has been the most widely accepted methodology for developing guidelines (6). GRADE classifies the quality of evidence relating to a pre-specified clinical question as very low, low, moderate, or high based on the methodological quality of the evidence and likelihood of outcome bias. GRADE’s quality assessment starts with RCTs as high-quality evidence and observational studies as low-quality evidence. Such categorizations largely reflect the degree of internal validity of RCTs compared with observational studies (more pragmatically designed trials sitting somewhere between). RCTs are designed to evaluate direct cause-and-effect between a trial intervention (versus comparators) and an outcome. They are typically conducted in centres of excellence and designed to minimize extraneous variables that might ‘mask’ the effect of the intervention(s) under trial. This results in RCTs having high internal validity, i.e., confidence that measured treatment effects are ‘true’ within the population and context considered in that trial. To achieve this, however, they have to exclude any patients that may introduce uncertainty or, ‘confound’ the results. Patients’ lifestyle characteristics (e.g., current smoking status), socioeconomic or educational characteristics, behavioural characteristics (e.g., religious beliefs, medication attitudes) and/or clinical characteristics (e.g., comorbidities, reversibility of airflow obstruction) are, therefore, systematically controlled for in the trial design. Similarly, variability in quality of clinical care, monitoring and exposure to the trial intervention(s) has to be standardized. This tends to result in a highly-interventional ecology of care involving close monitoring and assurance that the intervention is being implemented as prescribed.

As such, RCTs appraise whether a clinical intervention is efficacious and safe for the duration of the trial period (typically relatively short-term) in ideal patients managed ideally. This translates to exclusion of approximately 90% of the patients with asthma and COPD managed in routine care (7). The high internal validity of RCT design therefore means they are limited in their external validity, i.e., the ability to which their results can be assumed to hold true for the general patient population (≥90% of whom fail to meet at least one standard RCT inclusion criterion).

Contrastingly, observational studies do not interfere with the usual ecology of care (routine care precedes as normal and outcomes are simply observed). In terms of study population, they can take a very inclusive approach to population selection (e.g., any patients with a diagnostic label for COPD) or a more focused approach by pre-defining a more specific subpopulation to increase confidence in the results (e.g., patients with spirometry-confirmed COPD, a positive smoking history, aged 60 years or above and in receipt of at least one prescription for a licensed COPD therapy in the last year). Their non-interventional ecology of care and their ability to observe clinical outcomes at a population level result (i.e., across age ranges, socioeconomic categories, clinical and demographic subgroups) means observational studies having better external validity than RCTs. Yet the ability to extrapolate their results to the general patient population comes at the expense of robust internal validity. The potential signal measured between an intervention and an outcome in an observational study may be the result of (or influenced by) variations in the clinical approaches (prescribing decisions and management) and patient characteristics encompassed by their more inclusive design.

In the absence of one study design that offers both strong internal and external validity, there is a need to integrate different sources of evidence to get the most complete clinical picture possible. Yet it is important to interpret the respective findings with care. Critical to accurate interpretation of available evidence is to understand the specific research question being addressed. For instance, an RCT may address the following question: ‘is intervention A non-inferior to intervention B in terms of exacerbation management when used over 6 months in adult patients with (highly-characterized) asthma treated at a severe asthma service?’ In contrast, an observational study may aim to address an apparently similar, yet distinctly different, question: ‘is intervention A better than intervention B in terms of exacerbation management when used over 24 months in adult patients treated for asthma in routine primary care?’ The observational study will consider a broader patient population, over a longer period of exposure and consider the benefit of one intervention over the other while the RCT will seek to demonstrate that the new intervention (A) is at least as good as existing option (B) when used in a specialist setting of highly selective patients over a relatively short period of time. Accuracy and skill in evidence reporting, appraisal and interpretation are key to integrating these different findings.

Simple, practical tools will help to support the community in achieving that end. In terms of appraising the quality of published observational studies, the literature assessment tool developed by a joint Respiratory Effectiveness Group–European Academy of Asthma Allergy and Clinical Immunology Taskforce (REG-EAACI; see REG 2016 summit report later in this issue) will be a valuable addition to our armamentarium. Also useful in terms of conceptualizing the relative contributions of different studies is a framework for integrating different evidence sources, proposed by Roche et al. on behalf of the REG (8). The framework classifies all studies by the degree to which their study population (Y-axis) and ecology of care (X-axis) reflect real life (Figure 1A-D). These two key study characteristics are graphically represented in terms of their relative positioning along the two axes—for ecology of care, their position along a continuous scale from “highly-controlled efficacy RCT management and follow-up” at one end to “usual care (i.e., real-life practice)” at the other (Figure 1C). In turn, the population axis categorizes a study’s population along a continuum from a “highly-selected” population through to a “clinically diagnosed” population and on to “managed as condition X” (i.e., managed as the condition under evaluation with or without a properly confirmed physician diagnosis) (Figure 1D). This framework provides a way of visualizing the relative position and contribution of a specific study/research question compared with that of existing studies, i.e., the gap in the evidence base that the study is seeking to address.

Figure 1 Framework to support conceptual integration of different study designs in terms of the extent to which their design reflects real-life (8). (A) Framework; (B) position of evidence classified as high-quality by GRADE; (C) examples of means by which studies can be designed to include a more real-life ecology of care, and associated position along the X-axis of the framework; (D) examples of means by which studies can be designed to include a more real-life patient population, and associated position along the Y-axis of the framework. Adapted from Roche et al. (8) with permission from the publisher (9).

As an early and committed advocate for greater recognition of real-life research, we believe the community is now approaching a point of critical change. Real-life research is no longer the ‘poor relation’ or ‘critical opponent’ to RCTs; it is increasingly seen as a ‘supportive and helpful friend’. With this change in perception of real-life evidence the field is poised to come of age. Clinicians and clinical academics have a critical role in helping to nurture and guide this evolution and to set and share best practice standards so that rigor continues to be applied. Only in this way can real-life studies stand shoulder-to-shoulder with RCTs and warrant full integration into clinical decision making: something that I believe is a clinical imperative.

The Respiratory Effectiveness Group (REG; www.effectivenessevaluation.org) was set up 3 years ago to respond to the growing recognition of real-life research in respiratory medicine and in response to the need to standardize methodologies and approaches in this evolving field. The group provides a platform for experts (and interested parties) to come together to defragment and share best practice, set quality standards, identify unmet research needs that real-world methodologies are best placed to address and to shape the field so that it is responsibly nurtured, managed and adopted at the points of both research/evidence generation and clinical management/guideline development. The REG collaborators met in April this year for their third annual summit, themed: ‘Impact and Influence of Real-World Respiratory Medicine’. It was a chance for delegates both to reflect on the evolution of the field so far and to consider how it can and should move forward.

As the REG 2016 summit report in this issue outlines, the event debated and discussed important areas of current and potential influence of real-life research, in clinical practice, market access, guideline development and regulatory issues as well as showcased the results from some of the on-going real-world studies being delivered by REG’s specialty working groups. The summit also featured six thematic abstract sessions across the 2 days: Safety and Risk; Diagnostic Practice; Pragmatic Approaches; Validation and Tool Development; Healthcare Resource Utilisation; Late Breaking Research. The high-quality research presented in these sessions is also included in this issue. The scope of the work presented is testimony to the breadth and depth of interest and expertise in real-world research within the respiratory community. The reason for this, to me, is obvious—there is widespread recognition that (contrary to what many of the guidelines purport) patients refuse to fit into neat boxes or to behave in the logical and rational manner expected by their clinicians. Similarly, healthcare professionals sometimes stray from the letter of the guidelines and, as a result, may offer patients less than optimal therapy. Until one of these realities changes, daily clinical practice requires a complex interplay between experience and training, research and judgment and must draw not only on data from RCTs, but also on real-life evidence—pragmatically designed studies that better reflect real people and clinical practice.

References

1. Reddel HK, Taylor DR, Bateman ED, et al. An official American Thoracic Society/European Respiratory Society statement: asthma control and exacerbations: standardizing endpoints for clinical asthma trials and clinical practice. Am J Respir Crit Care Med 2009;180:59-99. [Crossref] [PubMed]
2. Sharek PJ, Bergman DA. Beclomethasone for asthma in children: effects on linear growth. Cochrane Database Syst Rev 2000.CD001282. [PubMed]
3. Holgate S, Bisgaard H, Bjermer L, et al. The Brussels Declaration: the need for change in asthma management. Eur Respir J 2008;32:1433-42. [Crossref] [PubMed]
4. Rawlins M. De Testimonio: on the evidence for decisions about the use of therapeutic interventions. Clin Med (Lond) 2008;8:579-88. [Crossref] [PubMed]
5. Global Initiative for Asthma. GINA-GOLD diagnosis of disease of chronic airflow limitation: asthma, COPD and asthma-COPD overlap syndrome (ACOS), 2015. Accessed Feb 23, 2016. Available online: http://ginasthma.org
6. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924-6. [Crossref] [PubMed]
7. Herland K, Akselsen JP, Skjønsberg OH, et al. How representative are clinical study patients with asthma or COPD for a larger "real life" population of patients with obstructive lung disease? Respir Med 2005;99:11-9. c. [Crossref] [PubMed]
8. Roche N, Reddel HK, Agusti A, et al. Integrating real-life studies in the global therapeutic research framework. Lancet Respir Med 2013;1:e29-30. [Crossref] [PubMed]
9. Price D, Brusselle G, Roche N, et al. Real-world research and its importance in respiratory medicine. Breathe (Sheff) 2015;11:26-38. [Crossref] [PubMed]

David B. Price