Structured and evidence-based training of technical skills in respiratory medicine and thoracic surgery

Introduction

Competent performance of technical procedures is essential for both thoracic surgeons and respiratory physicians (1-3). The list of technical thoracic procedures has expanded widely within the last decades due to an evolution of new technologies. This creates massive demands for obtaining new knowledge and skills of the physicians in respiratory medicine and thoracic surgery.

Previously, education and training in technical procedures have often taken place bed-side fully integrated as part of daily clinical work. The extensive growth in number and complexity of procedures combined with increased demand of efficacy, number of patients, and focus on patient safety have made training in new approaches necessary in order to benefit both patients and trainees (4). The traditional apprenticeship model, the “see one, do one, teach one”-strategy, where trainees watch the procedure being performed by experienced colleagues, and then are expected to have gained adequate competences and skills, is no longer considered adequate (5,6). Current guidelines from the international societies recommend volume thresholds when certifying trainees in technical thoracic procedures (2,7). Fixed numbers of supervised procedures prior to independent performance do however not ensure adequate competencies, because trainees differ in learning pace, motivation, and pre-existing skills. Larger syllabuses are often taught in courses, which extend from a couple of hours to days, and often as classroom-based didactic lectures, even though research proves interactive teaching models are far more effective (8). Theoretical knowledge of a given procedure is possible to gain from a lecture, but the necessary essential motor skills are not possible learn in a theoretic session.

Looking at the growing number of published literature within medical education, and education in technical procedures, the previous educational strategy is today not considered as best evidence medical education. Researchers have searched for structured and evidence-based ways of not only developing methods for education, but also ways of ensuring sufficient competence following education.

Kern has presented a six-step, systematic approach for curriculum development (9). The approach is today widely used, because it, in a practical, evidence-based way, emphasizes considerations on how an educational program should be developed, implemented and re-evaluated. It adheres to the requirements of a modern healthcare system, where education is time, and time is money. Kern’s six-step approach complies:

• Problem identification and general needs assessment;
• Targeted needs assessment;
• Goals and objectives;
• Educational strategies;
• Implementation;
• Evaluation and feedback.

To improve the effectiveness, educators need to evaluate and elaborate each step of the approach, see Figure 1. The aim of this article is (I) to summarize current state of education in technical procedures in respiratory medicine and thoracic surgery, and (II) based on Kern’s framework, discuss and elaborate on future recommendations.

Figure 1 Kern’s six-step approach for continuing curriculum development. The steps are not consecutive, but should be evaluated fluently as curriculum development is a dynamic process.

Problem identification and general needs assessment

First step is the identification and analysis of the problem, issue or need, that is to be addressed by the curriculum (10). The identification should be based on a review of existing literature and available information on e.g., current postgraduate medical education, and the educators must elaborate on how the problem is currently being addressed. The difference between how the educational issue is addressed and how it should be addressed is the general needs assessment. The complete problem identification and needs assessment must be done at a national or international level in order to ensure generalizability.

Generally, national learning objectives, both technical and non-technical in respiratory medicine and thoracic surgery, are provided by the National Health Services in collaboration with the societies for the specific speciality, and statement of aims rely on the syllabus established by the international societies e.g., European Respiratory Society (ERS) or American Board of Thoracic Surgery (ABTS) (11-13). In most cases, the presented statements of aims, are lists of knowledge and skills needed in order to handle, diagnose or treat patients with pulmonary or thoracic diseases, but it is unclear what the evidence behind these lists are. A structured approach to identify, which technical procedures should be learned, is desired, rather than one expert’s opinion, financial interests of a company, or availability of simulators or phantoms for a specific technical procedure. Nayahangan et al. present a general needs assessment of procedures suitable for simulation-based training (SBT) (14). The stringent and structured study design is judiciously and deliberated, and future general needs assessment can advantageously be carried out as this study, which additionally encourages cross boarder collaboration in order to generalize curriculum development, and provide comparable results.

In March 2018 ERS published the “Update of the ERS international Adult Respiratory Medicine syllabus for postgraduate training” (11), and following in May 2018, the report of general needs assessment in thoracic surgery was published by the European Society of Thoracic Surgeons (ESTS) (15). The aim of the work and publications was to define minimum standards to address the heterogeneity of training across Europe, and enable a generalizable education in respiratory medicine and thoracic surgery. A Delphi-like process, as described by Nayahangan et al. (14), was in both cases performed.

No specific or definitive number of respondents is needed when performing a Delphi-process, it depends on the complexity of the topic evaluated, but response time and deadlines can influence the choice and the number of participants (16). It is recommended that the number of experts should be large enough to provide a representative gathering of the opinions on the content (16). It is unclear how many respondents were used in the ERS publication, but the process of the preparation of syllabus in thoracic surgery included 334 respondents (representing 36 countries) in first round. Trainee members of ESTS were invited, because the task force had experienced dissatisfaction among the younger surgeons due to missing education and training in specific procedures they found relevant. Therefore the trainees could contribute to the overall assessment of general needs relevant in a daily clinical work, even though a Delphi process is said to include experts only (16).

Hence, this way of identifying and prioritizing the educational modules and syllabus, meets the recommendations by Kern, and is a commonly used and favoured technique for problem identification and needs assessment, but requires adaption into the setting. The structured, international general needs assessments performed in respiratory medicine and thoracic surgery are available, and can form the foundation of targeted needs assessments on a national or institutional level, when having the geographical differences in disease prevalence, and resources available in mind.

Targeted needs assessment

The definition of the targeted needs assessment, is “the process by which curriculum developers apply the knowledge learned from the general needs assessment to their particular learners and learning environment” (9). Main focus on this step is to identify the target trainees and targeted learning environment, e.g., are the target trainees of the curriculum in the specific technical procedure junior surgeon residents, or experienced surgeon with competences in similar procedures. A well-done targeted needs assessment prevents inappropriate decisions that can cause additional work, bad feedback, and continuing restructuring with massive work load. Examples of possible avoidable issues/problems that could have been solved with a sufficient targeted needs assessment are; duplicated objectives and tasks (e.g., which is taught in another course), or wrong aligned level of knowledge and skills (e.g., the tasks are either too easy or too difficult to the targeted trainees). If such problems arise, it will decrease trainees’ motivation, the outcome benefit will be lower than expected, and the financial expense will be wasted (10). Therefore, a curriculum developer should identify resources to align the strategy and assess the targeted trainees’ experiences, expectations, existing proficiencies (including cognitive, affective and psychomotor), and the environmental opportunities (already existing curriculum, needs of stakeholders, financial barriers, and resources). Understanding the targeted trainees’ environment also contributes to understanding trainees’ behaviour and attitude towards a specific procedure or skill. Different methods are being used for targeted needs assessment, and can be done by questionnaires, tests, focus groups interviews or observations (9). These methods can be time consuming but sufficient considerations and reflections can benefit the next parts of the curriculum development, and at the end be timesaving and cost-effective.

Goals and objectives

When relevant procedures are clarified for the targeted trainees and targeted environment, characterization of the structured goals have to be made. The goals are often broad educational objectives of the curriculum, and describe an end toward which, an effort is directed, and are usually not measureable (10). Contrary, in order to prove the effect of an educational intervention, clear measureable objectives (and outcome measures) are needed, and may include cognitive, affective and psychomotor skill objectives to the trainee, especially if the curriculum covers technical procedures.

Presented in Figure 2 is a modified version of the framework for assessing clinical competences by George Miller, Millers’ pyramid (17). The lower level represents the basic facts (knows), followed by the applied knowledge (knows how), the hands-on (shows how), and performance (does). The framework acknowledges the consecutive development when learning a new skill with increased complexity when moving towards the top, and invites to the use of competency-based learning and assessment.

Figure 2 A modified version of Miller’s pyramid for assessing clinical competences.

Trainees have different and individual learning paces, as well as different prerequisites when training a new technical procedure. Competency-based training allows these differences so that all trainees reach same adequate, and pre-defined level of proficiency for a given task; some trainees just use more time or more guidance in order to reach the level. Thus, when writing objectives, considerations must be done on, which step of the Miller’s pyramid the procedure is being taught. It is essential prior to implementation of measurable assessment tools, and additionally if the assessment tool should be implemented as formative or summative test, during the training or at the end. A theoretical test as a measure of a practical competence is not proper, and a practical test in a simulation-based environment does not equal perspective to other findings in a clinical setting.

Already published literature has contributed with assessment tools in technical procedures with solid validity evidence in respiratory medicine and thoracic surgery (18-22). Savran et al. have developed theoretical tests in endosonography and video-assisted thoracoscopic surgery (VATS) lobectomy, because acquisition of sufficient theoretical knowledge provides trainees with the necessary foundation to develop technical skills (19,23). SBT may be a flexible and effective approach for training technical skills. Konge et al. present a study using simulation as a tool for assessment of competences in endobronchial ultrasound (EBUS) (18). The trainees in the study had to identify six anatomical landmarks followed by two transbronchial fine-needle aspirations of lymph node stations, and an EBUS-expert supervised the trainees and assessed competences by a structured score sheet. Jensen et al. developed and gathered validity evidence for the first virtual reality simulator for VATS lobectomy (24). The model is a right upper lobe lobectomy based on the principles of a standardized anterior approach including dissection, division of vessels, fissures and bronchus with staplers (25). Today, new software in the simulators makes it possible to assess technical proficiency, progression, and provide reliable and structured feedback. Because of this, it is possible for the trainee to restrict the training effort on the pitfalls identified until an adequate level is obtained and then subsequently proceed to a clinical setting. SBT are therefore able to replace the apprenticeship training in the beginning of the learning curve, but does however not provide experts in a clinical setting. A structured assessment tool with proven solid validity evidence has been developed for assessment of competences in bronchoscopy in a clinical setting (20). The assessment tool includes several aspects of a technical procedure; anatomical knowledge, technical skills, and the ability to describe the procedure. Thus, the complexity has further increased when compared to the simulation-based assessment. As an example the tool for assessment of competences in thoracic ultrasound, also evaluates on the ability to reflect on indications for the procedure (e.g., tentative diagnoses, information to the patient), and conclusions based on the procure (e.g., integrate the examination findings with the patient history) (26). The VATSAT (VATS assessment tool) was developed by Jensen et al. using the Delphi method and a group of international VATS experts. It consists of eight items with a score between one and five (27). Petersen et al. provided validity evidence for VATSAT in a real clinical situation with VATS lobectomies performed in a Danish nationwide study. Video recordings of 60 VATS lobectomies performed at all four centers of thoracic surgery in Denmark by 18 thoracic surgeons were rated using the VATSAT. The two raters were blinded to surgeon and center (28). Observational assessment in a clinical setting is still needed, and can be done by direct observation using a structured tool filled out by an experienced supervisor.

Gathering validity evidence for assessment tools is vital to their use (29). A significant issue with the increasing number of publications in medical education is the lack of proven validity evidence, wrong or outdated methods (30). Downing and Yudkowsky describe Validity as the single most important characteristic of assessment data, and that “validity refers to the evidence presented to support or refute the meaning or interpretation assigned to assessment data (test)” (29). So, gathering validity evidence for a test is done to ensure that the assessment tool or test measures what it is meant to measure, and in a meaningful way. It ensures that no trainee with adequate level fails, and then spend more time than needed practising, costing the institution time and money unnecessarily. It also ensures that no trainee who does not possess adequate level of competence pass, and potentially put patients at risk by mistakes or wrong interpretations (29,31,32). Pass/fail standards are needed for summative tests, and is an essential part of exploring validity evidence (33).

So, in conclusion, when determining the educational objectives for a curriculum of a technical procedure, considerations must encompass to which level the procedure should be learned, and how should the development and skills be measured.

Educational strategies

Once the goals and objectives have been determined, the next step is to consider which educational strategy is more favourable to accomplish goals and objectives. It is important to consider both the content of the curriculum, and the educational methods. Maintenance of congruence between the objectives and educational methods are crucial, but so are feasible methods in terms of resources and administrative possible solutions. Involved educators must decide how much time and effort they will use on the educational program. Studies have proved that online or web-based sessions like podcasts, are non-inferior to traditional classroom-based education in terms of obtainment of theoretical knowledge, and today, trainees are used to navigate online and respond to web-based learning by increasing satisfaction (34-36). Workload of both educators and trainees is more flexible when using web-based learning, the educators avoid long confrontation hours, which reduce already discounted hours away from the clinical work, but still, this method requires continuously evaluation in order to keep material up to date (36).

Traditionally, courses are being held as classroom-based courses, introducing new technical procedures to the trainees. The trainees then repeat and train the procedure for the day, and are then ready to implement the procedure in daily practice. First of all, a fixed timeframe or a fixed number of procedures do not ensure adequate competences. Secondly, the effect of whole-day courses is inferior to shorter task training sessions (37), and thirdly, if the procedure trained does not correspond to the daily work done by the trainee, the effect could potentially be minimized. One potential way to address these issues is by using SBT. Several studies within respiratory medicine and thoracic surgery have proven that trainees can benefit from SBT and that is possible to use SBT for assessing competences in a solid and valid manner (18,21,38). SBT addresses the challenges of patient safety, making it possible for novices to train motor skills at the stage where the movements are slow and inconsistent, and where large parts of the movements are controlled consciously without using patients as hands-on training facilities (Figure 3) (39). Additionally, when the basic skills have evolved, and the movements are done automatically, it is possible to repeat and train more complex high-stake cases in a calm environment, which increases trainee motivation and confidence (40). The cost of acquiring a simulator can be significant, and SBT is not the answer to all issues in the teaching of technical procedures. The implementation and use of SBT must be deliberate and embedded into a full curriculum, because SBT is not suitable for knowledge acquisition (know) or for implementing the procedure into context in a clinical setting (Does from Miller’s pyramid). Konge et al. recommend a four-step approach for the development of competency in video-assisted thoracic surgery (VATS) lobectomy including SBT. The approach complies the basic principles of the mastery learning concept, and as seen in Figure 4, covers the aspects of Miller’s pyramid (41). The trainees will train to mastery, and the training is individualized, thus, the depended variable is not the test score but numbers of examinations performed during the training period.

Figure 3 A trainee practices bronchoscopy using a simulator, supervised by an instructor in the Regional Center for Technical Simulation at Odense University Hospital, Denmark.

Figure 4 The four-step mastery learning approach for medical simulation-based training of technical procedures, recommended by Konge et al. (24). The approach is compared to the levels from Millers pyramid.

Implementation

Support and flexibility from the institutions are prerequisites in order to implement, and thus benefit from SBT as an educational strategy in a curriculum of technical procedures. The institutions must accept the time away from the clinical ward as an investment, which will pay off in effective time (working hours), decreased number of complications or calls for assistance, but the win is not directly and immediately measureable. The “Danish model” comprises four regional Simulation Centres, where physicians from hospitals and departments within the same region share the same models, simulators, instructors and material. Each centre evaluates the use and need of courses, and can decide to move a specific course to a smaller hospital for a period of time. The centres are more cost effective, than if each hospital should offer courses within the same procedures, and the model also contributes to more uniform education and the possibility for educational research.

It is a necessary process to go through all the steps in Kerns’ model, because curriculum development is time-consuming, and therefore the result must be validated, systematic and as effective as possible. The infrastructure and technology e.g. simulators, are not enough to comply the demands of teaching technical procedures. A challenge is to effectively utilize the tools whether it is simulators, phantoms, or other devices used for education. A major component of the challenge lies in attracting institutional members to apply this strategy of education. Several technical procedures within respiratory medicine and thoracic surgery are also done by physicians from other specialities, e.g., pleural drainage performed by anaesthesiologists, and thoracic ultrasound performed by emergency physicians (42,43). One method to streamline, and make a curriculum more generalizable, and thus cost beneficial, is to explore the multi-speciality procedures. The utilization of resources could be increased if these procedures are identified and the needs assessments and subsequently steps are done with a broader variety of specialities.

A benefit of SBT when considering the implementation strategy, is the possibility of self-directed training. Self-directed learning is trainee-centred, individualized, and flexible to working hours. The trainee can be introduced to the procedure and simulator by the expert (educator), and is afterwards able to train on their own or supervised by an institute member or medical student. Many simulators are able to assess the trainee automatically which allows identification of pitfalls or incomplete whole or part tasks, and the expert supervision can be saved for more complex tasks, or for assessment after implementation of the procedure in a clinical setting. A well-defined simulation-based curriculum in endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA) has been implemented using medical students as supervisors during self-directed SBT (22,44). Studies are needed to sustain the fact that experts are not necessary during SBT to acquire the greatest possible benefit from SBT—actually, self-directed learning might increase retention (45).

A recurrent issue in published literature within medical education is the patient-related outcome or benefit of the educational intervention (46). Such educational interventions are hard to measure because of the irregular links that potentially exist between the intervention and patient outcome. Research warns against relying on surrogate measures, but no persuasive and convincing methods have been presented in order to address this issue.

Evaluation and feedback

The sixth step “Evaluation and feedback” closes the loop in the curriculum development cycle, and provides information to guide the curriculum or individual into improvement so that (I) the individual meets the objectives defined in step III, and (II) the curriculum evolves and remain up-to-date (10). Kirkpatrick developed a model for evaluating and analyzing educational programs, which is widely used in the assessment of educational interventions (47). The model describes four levels (Figure 5). Level 1 is the trainees’ reaction, and trainees’ satisfaction with the educational intervention. The objective of this level is a straightforward quantitative analysis, but it is important to remember that an optimistic reaction to a curriculum does not equal a positive learning outcome, but an unfavourable reaction definitely makes it less likely that the trainee paid attention to the curriculum. Level two of the learning outcome criteria, is the evaluation of obtained knowledge, skills and behaviour (individuals meet the objectives). The one-group pre-test and post-test design is often used (48-50), but is today considered obsolete due to minimal internal and low external validity (51). When using assessment tools, like described under Goals and objectives, including minimum pass/fails standards, the evaluation of learning can be assessed effectively and evidence-based. The evaluation of behaviour (Level 3) assess whether the skill or competence is transferred to a clinical setting. Trainees must apply the learning into context, and change behaviour either immediately or as a continuous remodelling if the teaching of a technical procedure should be effective. Clear design on how this outcome should be measured, is highly relevant in order to decrease the subjective judgement by educators related to the educational programme, which must be expected as a factor that can affect the reliability and consistency of the evaluation. The use of e.g., identical SBT with uniform assessment tools across respiratory medicine and thoracic surgery, can lead to a large amount of data for future research in the teaching of technical thoracic procedures. Hopefully, the gathering of comparable data can also make the patient-related outcome more feasible.

Figure 5 The four levels of educational evaluation (Kirkpatrick’s model).

Conclusions

In the future, structured curricula have to be developed when introducing and educating trainees to perform new thoracic technical procedures. ERS and ESTS have made a substantial contribution by performing the needs assessments, and this work can form the foundation to forthcoming programs, but educators have to elaborate on more than just the topic in order to establish a structured, evidence-based educational program. Clear goals and objectives must be established, assessment tools with solid validity evidence are crucial, and educational strategies must be decided with available recourses in mind.

The authors have presented a practical guide, using a recognized framework, on how the development can be manufactured, and the future can potentially be as told by Vozenilek “see one, simulate many, do one competently and teach many” (4).

Acknowledgments

Funding: None.

Footnote

Provenance and Peer Review: This article was commissioned by the Guest Editor (Gilbert Massard) for the series “Training in Pulmonary Medicine and Surgery” published in Journal of Thoracic Disease. The article has undergone external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/jtd.2019.02.39). The series “Training in Pulmonary Medicine and Surgery” was commissioned by the editorial office without any funding or sponsorship. RH Petersen serves as an unpaid editorial board member of Journal of Thoracic Disease from Sep 2018 - Aug 2022. CB Laursen reports personal fees from GE, outside the submitted work. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Accreditatiomn Counsil for Graduate Medical Education. ACGME Program Requirements for Graduate Medical Education in Pulmonary Diseases and Critical Care (Internal Medicine) Chicago: ACGME, 2017.
2. Bolliger CT, Mathur PN, Beamis JF, et al. ERS/ATS statement on interventional pulmonology. European Respiratory Society/American Thoracic Society. Eur Respir J 2002;19:356-73. [PubMed]
3. Ernst A, Silvestri GA, Johnstone D, et al. Interventional pulmonary procedures: Guidelines from the American College of Chest Physicians. Chest 2003;123:1693-717. [Crossref] [PubMed]
4. Vozenilek J, Huff JS, Reznek M, et al. See one, do one, teach one: advanced technology in medical education. Acad Emerg Med 2004;11:1149-54. [Crossref] [PubMed]
5. Iobst WF, Sherbino J, Cate OT, et al. Competency-based medical education in postgraduate medical education. Med Teach 2010;32:651-6. [Crossref] [PubMed]
6. Carraccio C, Wolfsthal SD, Englander R, et al. Shifting paradigms: from Flexner to competencies. Acad Med 2002;77:361-7. [Crossref] [PubMed]
7. Prakash U. Guidelines for Training and Practice of Interventional Pulmonology: By the Numbers? Journal of Bronchology and Interventional Pulmonoly 2003;10.
8. Hew KF, Lo CK. Flipped classroom improves student learning in health professions education: a meta-analysis. BMC Med Educ 2018;18:38. [Crossref] [PubMed]
9. Patricia A. Thomas DEK, Mark T. Hughes, Belinda Y. Chen. Curriculum Development for Medical Education - A Six-Step Approach. Third ed. Baltimore, Maryland Johns Hopkins University Press; 2016.
10. Khamis NN, Satava RM, Alnassar SA, et al. A stepwise model for simulation-based curriculum development for clinical skills, a modification of the six-step approach. Surg Endosc 2016;30:279-87. [Crossref] [PubMed]
11. Tabin N, Mitchell S, O'Connell E, et al. Update of the ERS international Adult Respiratory Medicine syllabus for postgraduate training. Breathe (Sheff) 2018;14:19-28. [Crossref] [PubMed]
12. Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Graduate Medical Education in Thoracic Surgery. 2018.
13. American Board of Thoracic Surgery. ABTS Cardiothoracic Surgery Curriculum Outline. 2011.
14. Nayahangan LJ, Stefanidis D, Kern DE, et al. How to identify and prioritize procedures suitable for simulation-based training: Experiences from general needs assessments using a modified Delphi method and a needs assessment formula. Med Teach 2018;40:676-83. [Crossref] [PubMed]
15. Massard G, Tabin N, Mitchell S, et al. A harmonized European training syllabus for thoracic surgery: report from the ESTS-ERS task force. Eur J Cardiothorac Surg 2018;54:214-20. [Crossref] [PubMed]
16. de Villiers MR, de Villiers PJ, Kent AP. The Delphi technique in health sciences education research. Med Teach 2005;27:639-43. [Crossref] [PubMed]
17. Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65:S63-7. [Crossref] [PubMed]
18. Konge L, Clementsen PF, Ringsted C, et al. Simulator training for endobronchial ultrasound: a randomised controlled trial. Eur Respir J 2015;46:1140-9. [Crossref] [PubMed]
19. Savran MM, Clementsen PF, Annema JT, et al. Development and validation of a theoretical test in endosonography for pulmonary diseases. Respiration 2014;88:67-73. [Crossref] [PubMed]
20. Konge L, Larsen KR, Clementsen P, et al. Reliable and valid assessment of clinical bronchoscopy performance. Respiration 2012;83:53-60. [Crossref] [PubMed]
21. Jensen K, Bjerrum F, Hansen HJ, et al. Using virtual reality simulation to assess competence in video-assisted thoracoscopic surgery (VATS) lobectomy. Surg Endosc 2017;31:2520-8. [Crossref] [PubMed]
22. Naur TMH, Konge L, Nayahangan LJ, et al. Training and certification in endobronchial ultrasound-guided transbronchial needle aspiration. J Thorac Dis 2017;9:2118-23. [Crossref] [PubMed]
23. Savran MM, Hansen HJ, Petersen RH, et al. Development and validation of a theoretical test of proficiency for video-assisted thoracoscopic surgery (VATS) lobectomy. Surg Endosc 2015;29:2598-604. [Crossref] [PubMed]
24. Jensen K, Hansen HJ, Petersen RH, et al. Evaluating competency in video-assisted thoracoscopic surgery (VATS) lobectomy performance using a novel assessment tool and virtual reality simulation. Surg Endosc 2019;33:1465-73. [Crossref] [PubMed]
25. Hansen HJ, Petersen RH. Video-assisted thoracoscopic lobectomy using a standardized three-port anterior approach - The Copenhagen experience. Ann Cardiothorac Surg 2012;1:70-6. [PubMed]
26. Skaarup SH, Laursen CB, Bjerrum AS, et al. Objective and Structured Assessment of Lung Ultrasound Competence. A Multispecialty Delphi Consensus and Construct Validity Study. Ann Am Thorac Soc 2017;14:555-60. [Crossref] [PubMed]
27. Jensen K, Petersen RH, Hansen HJ, et al. A novel assessment tool for evaluating competence in video-assisted thoracoscopic surgery lobectomy. Surg Endosc 2018;32:4173-82. [Crossref] [PubMed]
28. Petersen RH, Gjeraa K, Jensen K, et al. Assessment of competence in video-assisted thoracoscopic surgery lobectomy: A Danish nationwide study. J Thorac Cardiovasc Surg 2018;156:1717-22. [Crossref] [PubMed]
29. Downing SM, Yudkowsky R. Assessment in Health Proffessions Education. New York & Londin: Routledge, 2009.
30. Borgersen NJ, Naur TMH, Sorensen SMD, et al. Gathering Validity Evidence for Surgical Simulation: A Systematic Review. Ann Surg 2018;267:1063-8. [Crossref] [PubMed]
31. S M. Validity Educational Testing Service1987;1987.
32. Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ 2003;37:830-7. [Crossref] [PubMed]
33. Jorgensen M, Konge L, Subhi Y. Contrasting groups' standard setting for consequences analysis in validity studies: reporting considerations. Adv Simul (Lond) 2018;3:5. [Crossref] [PubMed]
34. Edrich T, Stopfkuchen-Evans M, Scheiermann P, et al. A Comparison of Web-Based with Traditional Classroom-Based Training of Lung Ultrasound for the Exclusion of Pneumothorax. Anesth Analg 2016;123:123-8. [Crossref] [PubMed]
35. Gold JP, Begg WB, Fullerton D, et al. Successful implementation of a novel internet hybrid surgery curriculum: the early phase outcome of thoracic surgery prerequisite curriculum e-learning project. Ann Surg 2004;240:499-507; discussion -9.
36. Cook DA, Levinson AJ, Garside S. Time and learning efficiency in Internet-based learning: a systematic review and meta-analysis. Adv Health Sci Educ Theory Pract 2010;15:755-70. [Crossref] [PubMed]
37. Savion-Lemieux T, Penhune VB. The effects of practice and delay on motor skill learning and retention. Exp Brain Res 2005;161:423-31. [Crossref] [PubMed]
38. Konge L, Colella S, Vilmann P, et al. How to learn and to perform endoscopic ultrasound and endobronchial ultrasound for lung cancer staging: A structured guide and review. Endosc Ultrasound 2015;4:4-9. [Crossref] [PubMed]
39. Taylor JA, Ivry RB. The role of strategies in motor learning. Ann N Y Acad Sci 2012;1251:1-12. [Crossref] [PubMed]
40. Sun Y, Pan C, Li T, et al. Airway management education: simulation based training versus non-simulation based training-A systematic review and meta-analyses. BMC Anesthesiol 2017;17:17. [Crossref] [PubMed]
41. Konge L, Petersen RH, Ringsted C. Developing competency in video-assisted thoracic surgery (VATS) lobectomy. J Thorac Dis 2018;10:S2025-8. [Crossref] [PubMed]
42. Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Graduate Medical Education in Anesthesiology. Chicago ACGME, 2018.
43. Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Graduate Medical Education in Emergency Medicine. Chicago: ACGME, 2017.
44. Farr A, Clementsen P, Herth F, et al. Endobronchial ultrasound: launch of an ERS structured training programme. Breathe (Sheff) 2016;12:217-20. [Crossref] [PubMed]
45. Brydges R, Nair P, Ma I, et al. Directed self-regulated learning versus instructor-regulated learning in simulation training. Med Educ 2012;46:648-56. [Crossref] [PubMed]
46. Brydges R, Hatala R, Zendejas B, et al. Linking simulation-based educational assessments and patient-related outcomes: a systematic review and meta-analysis. Acad Med 2015;90:246-56. [Crossref] [PubMed]
47. McGaghie WC. Mastery learning: it is time for medical education to join the 21st century. Acad Med 2015;90:1438-41. [Crossref] [PubMed]
48. Cuca C, Scheiermann P, Hempel D, et al. Assessment of a new e-learning system on thorax, trachea, and lung ultrasound. Emerg Med Int 2013;2013:145361 [Crossref] [PubMed]
49. Smolle J, Prause G, Smolle-Juttner FM. Emergency treatment of chest trauma--an e-learning simulation model for undergraduate medical students. Eur J Cardiothorac Surg 2007;32:644-7. [Crossref] [PubMed]
50. Colt HG, Davoudi M, Murgu S, et al. Measuring learning gain during a one-day introductory bronchoscopy course. Surg Endosc 2011;25:207-16. [Crossref] [PubMed]
51. Marsden E, Torgerson CJSingle group. pre- and post-test research designs: Some methodological concerns. Oxford Review of Education 2012;38:583-616. [Crossref]