The role of small or single-institution prospective studies in thoracic research

Interventional studies are designed specifically to test the impact of a treatment or management strategy on an outcome of interest. These are frequently prospective studies that compare a treated cohort with control patients, and can take the form of randomized controlled trials, pre- and post-implementation comparisons, or non-randomized trials (1). These study designs are better suited than retrospective observational studies to assess the causative effect of an intervention, but still have limitations and can be susceptible to different types of bias. Often, considerable effort and resources are required to plan and conduct an interventional study, and many factors can affect the eventual impact of the results.

In this article, we will highlight two viewpoints regarding the role of small or single-institution prospective interventional studies within the field of thoracic surgery: first, that these studies are beneficial and can positively impact the field, and second, that these studies can have such significant limitations that they may have no meaningful impact. A summary of these viewpoints is included in Table 1. We will also review elements that can make for successful design and execution of small prospective interventional studies.

Table 1 A summary of highlighted benefits and limitations of small prospective studies
Full table

Pro: these studies are beneficial and can positively impact the field

Small or single-institutional prospective studies can be employed for a variety of goals. These studies can be useful for analyzing elements of daily clinical care, adding data to dogmatic practice. They can also assess preliminary safety and efficacy of new or innovative clinical practices, establish feasibility of an intervention, or demonstrate proof of concept that provides the basis for a larger study. They may also be the ideal medium for collecting detailed subject patient data or communicating relevant clinical experiential data to the field. Several examples of such studies exist within thoracic surgery, and demonstrate the utility of this type of research.

Thoracic surgeons regularly use historical dogma to make daily clinical decisions, such as in the post-operative management of chest tubes, but there is often a lack of data that support these decisions. Prospective studies can help fill the gap. Cerfolio and Bryant (2) produced a single-surgeon, prospective study to challenge the often-cited “rule” that it is unsafe to discontinue a chest tube with an output of 250 mL/day or more. By applying a prospective algorithm of removing all chest tubes with less than 450 mL/day, without an air leak and without chylothorax, they found that over a 10-year period of over 8,600 pulmonary resections, only 0.55% of patients were readmitted for symptomatic pleural effusion. When compared to the overall group, the cohort of patients that had chest tubes discontinued with 250–450 mL/day output had no significant difference in readmission rate. Additionally, patients that were discharged early on postoperative day 2 or 3 carried the same readmission rate as the rest of the population. The significance of small, perspective studies in the management of chest tubes is not confined to this study. Cerfolio et al. (3) also utilized a prospective randomized trial to shed light on whether water seal or suction is superior for resolving air leaks after elective pulmonary resection. After developing a system to quantify severity of air leaks, they followed postoperative leaks over time in preoperatively-randomized water seal and suction groups matched to control for risk factors of leak. The randomization aspect of the trial was halted after 67% of air leaks under water seal had resolved by postoperative day 3 and only 7% of leaks under suction had resolved by this time (P=0.001). They concluded that water seal is superior to suction for resolving mild to moderate air leaks according to their described severity ranking, however, patients with large air leaks are at risk for pneumothorax on water seal and should be placed to suction to avoid this complication. These types of findings would be difficult or impractical to study in a large randomized trial and allow surgeons to utilize data-driven evidence when making clinical patient care decisions. While the decision of when to transition to water seal and when to pull a chest tube may seem mundane, these types of daily considerations have real implications on length of stay, treatment costs, and the patient’s experience. Therefore, it is imperative that these decisions are evidence-based. Small, single-institution prospective studies can serve as a relatively quick, affordable, and generalizable source of evidence to better inform surgeons on daily clinical decisions that are often dictated by dogma and anecdotal evidence.

Small-scale prospective studies also have the potential to provide a relatively simple and timely means to evaluate the safety and efficacy of new interventions that are either not developed enough or not appropriate for more rigorous, large-scale, randomized studies. One example is the use of office spirometry for preoperative risk stratification before lung resection instead of the traditionally used formal laboratory-based spirometry. In a small-scale, prospective, single-institution clinical trial, Hudson et al. (4) created 52 clinical propensity matches between patients assessed with office spirometry or laboratory spirometry. Outcome analysis showed no statistical difference between the matched office spirometry and laboratory spirometry patients in terms of discharge from the hospital, 30-day mortality, postoperative respiratory failure, major morbidity rate, and length of stay. The study also estimated that appropriately using office spirometry rather than laboratory spirometry would result in approximately $38,000 in annual institutional cost savings. In summary, by appropriately implementing a single-institution, small-scale, prospective clinical trial, Hudson et al. were able to effectively demonstrate the sufficiency, safety, and potential cost savings of using only office spirometry for preoperative pulmonary function screening before lung resection in low-risk patients. Findings like this can have practice-changing implications in the field of thoracic surgery. At a minimum, prospective trials can serve as the first form of safety validation for newer diagnostic and treatment patterns that are not yet widely accepted or are still being scaled. The prospective study design is often able to show the potential efficacy, safety, and cost-savings for new clinical interventions by taking full advantage of single-institution data that may otherwise be underutilized.

A limiting factor in executing large-multicenter trials is often cost. When a novel idea or intervention is presented in the surgical arena, demonstrating proof of concept can be valuable prior to organizing expensive multicenter trials. Smaller-scale studies can provide this preliminary assurance. Postoperative pneumonia after elective pulmonary resection is a common and life-threatening complication. In addition to affecting morbidity and mortality, postoperative pneumonia can greatly affect cost and length of stay. After recognizing that oral hygiene regimens lower ventilator-associated pneumonias, Washington University in St. Louis (5) designed a single-arm interventional study to assess the effect and feasibility of oral hygiene on pneumonia rates after pulmonary resection. This study enrolled 62 patients into a pre- and postoperative chlorhexidine toothbrushing intervention cohort and compared their postoperative pneumonia rate (1.6%) to that of a time-matched contemporaneous cohort (4.9%). While not statistically significant (P=0.35), this small-scale study showed an encouraging trend towards lower rates of pneumonia that supports further larger-scale studies, and provides a point estimate for appropriate statistical power calculations. As important as effectiveness, this study tested feasibility of the proposed oral hygiene regimen. Preoperative adherence to the oral hygiene regimen was high (87–100%). Postoperatively, 80% of patients opted to continue participation, with median adherence of 86% in this group. This revealed that patients were able to comply with the inexpensive oral hygiene regimen proposed. In summary, this study revealed that patients could adhere to a perioperative oral hygiene regimen that may hold promise for decreasing postoperative pneumonia rates. Small-scale studies like this are an affordable and realistic way of assessing feasibility of a new intervention while also gathering preliminary data before planning larger appropriately-powered studies to better test effectiveness.

Prospective studies can also provide key insight into a particular disease process that is otherwise difficult to study through feasible means of subjective follow-up. An example of this is Giant Paraesophageal Hernias (GPEH) that are relatively rare and often misunderstood. Stringham et al. (6) studied 106 GPEH patients at a single institution and found that surgical repair is safe and carries a low postoperative complication rate. Additionally, the study compared a new, radiographic recurrence definition (any amount of stomach above the diaphragm on barium swallow) to the conventional recurrence definition most often used in GPEH literature (>2 cm or >10% of the stomach above the diaphragm) and found a 1-year radiographic recurrence rate of 32.7% compared to a conventional recurrence rate of 18.8%. Interestingly the patient satisfaction scores were similar between the smaller and larger GPEH recurrences (57.1% vs. 52.6%, respectively) but both were significantly higher than the overall preoperative satisfaction score average of 2.9%. Finally, this study used the validated quality of life metric (GERD-HRQL), where high scores suggest poor quality of life, to show that patients without recurrence had significantly improved scores from 22.5 to 3.5 at 1 year from surgery, but that a recurrence of any kind was associated with an increase in score to 13.0. Prospective studies like this can help give thoracic surgeons a better understanding of a disease and specifically the subjective ramifications that it has on patient experience. This GPEH study was able to show convincingly the significant impact that a recurrence has on a patient postoperatively. It also revealed a potential new paradigm in the diagnosis of GPEH recurrence. This type of a prospective study was able to take advantage of a relatively small sample size, with a high follow-up rate (96.2% at 1 year) to gain outcome data about a surgical disease. With quality of life and other subjective indictors becoming more important in surgical patient care and potentially even in medical billing, small-scale, feasible, prospective trials with high follow-up rates could provide thoracic surgeons much needed insight into the patient experience.

When a new treatment or intervention is introduced, multiple centers often discover similar shortcomings before independently adjusting and adapting. Small prospective studies on new practices can provide helpful insight into recognized trends, problems encountered, and knowledge gained that may be useful to many institutions. One example is enhanced recovery after surgery (ERAS) pathways. There is a growing trend toward standardized ERAS protocols as data shows improved outcome measures with standardized and intentional postoperative care. These pathways can vary greatly between institutions. Martin et al. (7) at the University of Virginia introduced two thoracic ERAS protocols, one for VATS (n=81) and one for thoracotomy (n=58), and compared outcomes data to historic controls after 1 year. Both protocols showed significant decreases in postoperative opioids, total fluid balance, and hospital cost. ERAS after thoracotomy also showed a significant decrease in length of stay. While their preliminary data on the improvement of outcome measures is valuable, their detailed narrative of interventions made, problems encountered, and corresponding adjustments made to the pathway over the course of a year is widely useful to institutions developing similar protocols. By sharing this insight into the development of new interventions such as ERAS protocols, small prospective studies can encourage an efficient and collaborative approach across many institutions with the shared goal of improving patient experience.

Stepwise improvements in patient care can result from studies like these. Appropriate design, analysis, and presentation of small-scale studies are critical for maximizing the impact of the findings, but the attributes highlighted here demonstrate the utility that a small or single-institution study can have within the field.

Con: these studies can have such significant limitations that they may have no meaningful impact

Small studies can be subject to inadequate statistical study design, methodological pitfalls, or other shortcomings that can severely limit their impact, or may preclude publication altogether. Potential limitations in study planning, conduct, analysis, and publication merit consideration.

One major risk with small or single-institution studies is that they may be underpowered and subject to type II error, meaning that even if there is a real effect of the intervention, the analysis may not show a statistical difference. One example of this is demonstrated by a small preliminary prospective randomized trial by Geddes et al. (8) that examined the effect of lung volume reduction surgery (LVRS) versus medical management for patients with severe emphysema. They randomized a total of 48 patients and found that, while there were statistically significant improvements in pulmonary function studies, walking distance, and quality of life, there was not an observed benefit in mortality with one strategy versus the other. In the subsequent National Emphysema Treatment Trial (9), which randomized a total of 1,218 patients to LVRS or medical treatment, there was sufficient statistical power to perform subgroup analyses based on the distribution of emphysema and baseline exercise capacity. This trial showed that for certain subgroups of patients with high baseline exercise capacity, there actually were significant differences in mortality: in predominantly upper-lobe disease, LVRS actually was associated with a lower risk of mortality, whereas for patients with non-upper lobe emphysema, LVRS carried a higher risk of mortality. These disparate findings are very important to patient selection, but were not apparent in smaller studies. The statistical power, or the ability to detect a difference in outcomes between groups if it exists, is predominantly dependent on sample size and the effect size of the intervention. Within a small-scale study, interventions that have a large effect may yield statistically significant results, while those that have a smaller, yet real, effect may not. While data on estimated effect size can be used to determine the necessary sample size to avoid a futile underpowered study, this is not always done for a variety of reasons, ranging from lack of availability of the necessary data in the context of preliminary studies to simply poor trial planning. Even if a goal sample size is estimated properly, it may not prove feasible to enroll the target number of patients in a single-institution study. It is therefore necessary to interpret negative results of small studies with trial design and statistical power in mind.

Another concern is the risk of false positive results, or a type I error. Rare events may have an unwarranted effect on a study’s conclusions if they occur in a small population, and this can pose a significant threat to the validity of a study. A notable example of such questionable findings is an analysis of stereotactic body radiation therapy (SBRT) versus lobectomy for stage I lung cancer (10), which pooled a total of 58 patients from two randomized trials (STARS and ROSEL) that were closed early due to poor enrollment. Of the 27 patients in the surgery group, 6 died, reflecting a much higher than expected mortality rate from lobectomy. Only one of the 31 patients receiving SBRT died, which resulted in their analysis showing a statistically significant difference in survival. A statistical study by Samson et al. (11) examined the stability of the survival estimates from small versus large cohorts of patients by performing a bootstrapping analysis of an institutional sample of stage I lung cancer patients who would have met criteria for inclusion in one of the randomized trials. They found that with a cohort of 27 patients (the actual number included in the pooled analysis), expected survival estimates at 3-year varied widely from 46–100%, whereas with a cohort of 515 (the planned surgical enrollment of one of the randomized trials), the estimates ranged only from 70–85%. This is the reason that many clinical trials have planned preliminary analyses that take a very conservative early approach to avoid prematurely declaring superiority of a treatment based on chance events in a small sample size (12). Small studies that demonstrate much higher than expected event rates in one or both treatment arms should be interpreted with caution.

Additionally, confounding can be an issue in these studies. Randomization should theoretically distribute risk factors between intervention and control groups, but for practical reasons, many of these prospective studies are not randomized. As a result, an apparent effect of the intervention may in fact be due to one or more unequally distributed risk factors for the outcome of interest between the groups: confounding. A common approach to adjusting for potential confounding variables is performance of a multivariable analysis. A valid and reliable multivariable analysis including all relevant covariates may not be able to be performed with too few outcomes. A general rule is that ten outcomes are required for each variable included in the model (13), meaning that a small sample size in a study with few outcomes can preclude performance of a statistically sound multivariable analysis. Even when performed properly with sufficient outcomes, multivariable analysis only removes confounding from variables that were measured and included in the model, again emphasizing the importance of a properly designed study. Furthermore, some prospective studies are performed with a single interventional arm only, and initially encouraging results may not be confirmed when an intervention is tested in a trial, where additional important prognostic variables are taken into account. For example, Voynov et al. (14) reported outcomes from a series of 110 patients with stage I lung cancer who, because of poor lung function, underwent sublobar resection with placement of a brachytherapy mesh along the surgical margin and surrounding visceral pleura. They found an encouraging local control rate of 90% at 5 years, prompting enthusiasm for use of brachytherapy. The subsequent ACOSOG Z4032 trial was performed (15) and 224 high-risk patients undergoing sublobar resection were randomized to receipt of brachytherapy versus surgery alone. This trial demonstrated that there was no difference in rates of local control (HR 1.01, P=0.98) or 3-year survival (71% for both groups, P=0.97). This demonstrates the importance of an appropriate comparison group and consideration of relevant covariates.

Another limitation is that results from small studies conducted in a single institution may not be reproducible or generalizable elsewhere. Idiosyncratic circumstances may exist at small centers that contribute to study outcomes. Additionally, if rigorous inclusion and exclusion criteria are implemented that significantly limit the population, a study may be assessing the efficacy of the intervention—that is, how well it works in an ideal setting—rather than the effectiveness of the intervention—or how well it might be expected to work under more commonplace real-world circumstances (16). Often in studies, the patients most likely to benefit from an intervention are selected for participation through strict enrollment criteria. This frequently contributes to the ‘success’ of a study in demonstrating a statistically significant benefit of an intervention, but leaves providers questioning the expected benefit to their general patient population. The results may not be able to be extrapolated, or, while the intervention may still have an effect, it may be much smaller in magnitude than that demonstrated in the study. Attention to the patient population and study design can help a provider interpret how pragmatic the findings of a study are, and consideration of appropriate patient selection in trial planning can help maximize the applicability of the results.

Finally, significant publication bias affects this type of study, even for properly executed trials: many of the studies that do not show a statistically significant difference may never be published. Many reasons for the lower rate of publication of negative studies are often cited, though the most common explanation is that they are never submitted for consideration. Often this is due to perceived lack of interest in the results, an assumption that a journal will not be willing to publish the study, or that there is no personal or patient benefit to publication of negative results (17). This bias in publication not only limits the available primary literature, but also can inappropriately impact the findings of systematic reviews and meta-analyses, leading essentially to inaccuracies in what is frequently considered the highest level of evidence (17).

Because of these potential shortcomings, small studies can sometimes be considered futile efforts. Alone, they may not provide sufficient credible evidence to modify practice within the field, or even for individual practitioners. The limitations reviewed here demonstrate why clinical practice may not change in response to small or single-institution study findings.

Summary and recommendations

Consideration of certain elements can make for successful design and execution of small prospective studies. First, one should be mindful of several qualities in the conception and development of the project proposal. An appropriately focused clinical question must be selected. The effect of a single intervention should be investigated in a clearly defined patient population with specified inclusion and exclusion criteria. The trial itself needs to be properly designed, with an appropriate comparison or control group selected that is subject to the same predefined criteria. Randomization of patients to intervention or control groups is the gold-standard approach to minimize confounding, but time-matched, propensity-matched, or historical controls may be reasonable in certain circumstances. Thought should also be given to sample size. If preliminary data is available, power calculations should be performed to dictate target enrollment. For novel interventions, small pilot studies may be useful to assess feasibility of the intervention and to obtain a point estimate of effect size to perform power calculations for a planned larger trial. Furthermore, small-scale studies are appropriate when testing an intervention that deviates from existing standard of care and may carry safety concerns. Interval analysis and monitoring should be planned. Careful design is critical to the success of a small trial.

Second, adequate resources are needed to recruit and enroll patients. Cost-savings approaches to study design can help somewhat, but this is an especially relevant concern when significant costs may be associated with a single intervention or additional personnel are needed for detailed follow-up that is more in-depth than routine standard of care. Frequently, traditional sources of funding are granted on the basis of existing preliminary data. Sponsorship for the small trials that generate that initial data can be harder to come by. Industry, departmental, or societal support may be options in some cases. These avenues should be pursued, as having sufficient staff and support can be critical for meeting target enrollment and successfully conducting the intervention. Individual centers may be volunteering efforts to conduct these studies, which may not be sustainable or may substantially limit what can be done. There certainly is a need for more readily available funding to support promising efforts to conduct small-scale studies that may provide intriguing results or provide a basis for larger trials.

Finally, it is essential to have an appropriate plan for analysis of the study data. Often, due to power and sample size concerns, there can be significant challenges getting these studies published. Clearly describing the purpose of the study and the design can be helpful when presenting pilot data. An analysis plan should be decided on a priori, and again it is critical to have a proper comparison group. Appropriate statistics should be used and the results should be reported with a discussion of the limitations. When conducted, analyzed, and presented appropriately, small studies can have an impact on the field.

Acknowledgements

Funding: TR Semenkovich was supported by the Barnes Jewish Hospital Foundation and the Division of Cardiothoracic Surgery at Washington University in St. Louis. TR Semenkovich and M Subramanian were supported by National Institutes of Health (NIH) Grant Number 2T32HL7776-21.

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

References

1. Thiese MS. Observational and interventional study design types; an overview. Biochem Med (Zagreb) 2014;24:199-210. [Crossref] [PubMed]
2. Cerfolio RJ, Bryant AS. Results of a prospective algorithm to remove chest tubes after pulmonary resection with high output. J Thorac Cardiovasc Surg 2008;135:269-73. [Crossref] [PubMed]
3. Cerfolio RJ, Bass C, Katholi CR. Prospective randomized trial compares suction versus water seal for air leaks. Ann Thorac Surg 2001;71:1613-7. [Crossref] [PubMed]
4. Hudson JL, Bell JM, Crabtree TD, et al. Office-Based Spirometry: A New Model of Care in Preoperative Assessment for Low-Risk Lung Resections. Ann Thorac Surg 2018;105:279-86. [Crossref] [PubMed]
5. Semenkovich TR, Frederiksen C, Hudson JL, et al. Postoperative Pneumonia Prevention in Pulmonary Resections: A Feasibility Pilot Study. Ann Thorac Surg 2019;107:262-70. [Crossref] [PubMed]
6. Stringham JR, Phillips JV, McMurry TL, et al. Prospective study of giant paraesophageal hernia repair with 1-year follow-up. J Thorac Cardiovasc Surg 2017;154:743-51. [Crossref] [PubMed]
7. Martin LW, Sarosiek BM, Harrison MA, et al. Implementing a Thoracic Enhanced Recovery Program: Lessons Learned in the First Year. Ann Thorac Surg 2018;105:1597-604. [Crossref] [PubMed]
8. Geddes D, Davies M, Koyama H, et al. Effect of lung-volume-reduction surgery in patients with severe emphysema. N Engl J Med 2000;343:239-45. [Crossref] [PubMed]
9. Fishman A, Martinez F, Naunheim K, et al. A randomized trial comparing lung-volume-reduction surgery with medical therapy for severe emphysema. N Engl J Med 2003;348:2059-73. [Crossref] [PubMed]
10. Chang JY, Senan S, Paul MA, et al. Stereotactic ablative radiotherapy versus lobectomy for operable stage I non-small-cell lung cancer: a pooled analysis of two randomised trials. Lancet Oncol 2015;16:630-7. [Crossref] [PubMed]
11. Samson P, Keogan K, Crabtree T, et al. Interpreting survival data from clinical trials of surgery versus stereotactic body radiation therapy in operable Stage I non-small cell lung cancer patients. Lung Cancer 2017;103:6-10. [Crossref] [PubMed]
12. Meyers BF, Puri V, Broderick SR, et al. Lobectomy versus stereotactic body radiotherapy for stage I non-small cell lung cancer: Post hoc analysis dressed up as level-1 evidence? J Thorac Cardiovasc Surg 2015;150:468-71. [Crossref] [PubMed]
13. Katz MH. Multivariable analysis: a primer for readers of medical research. Ann Intern Med 2003;138:644-50. [Crossref] [PubMed]
14. Voynov G, Heron DE, Lin CJ, et al. Intraoperative (125)I Vicryl mesh brachytherapy after sublobar resection for high-risk stage I non-small cell lung cancer. Brachytherapy 2005;4:278-85. [Crossref] [PubMed]
15. Fernando HC, Landreneau RJ, Mandrekar SJ, et al. Impact of brachytherapy on local recurrence rates after sublobar resection: results from ACOSOG Z4032 (Alliance), a phase III randomized trial for high-risk operable non-small-cell lung cancer. J Clin Oncol 2014;32:2456-62. [Crossref] [PubMed]
16. Singal AG, Higgins PD, Waljee AK. A primer on effectiveness and efficacy trials. Clin Transl Gastroenterol 2014;5:e45. [Crossref] [PubMed]
17. Johnson RT, Dickersin K. Publication bias against negative results from clinical trials: three of the seven deadly sins. Nat Clin Pract Neurol 2007;3:590-1. [Crossref] [PubMed]