Lessons learned from the Dutch Institute for Clinical Auditing: the Dutch model for quality assurance in lung cancer treatment

Introduction

As in the rest of the world, in The Netherlands lung cancer is the leading cause of cancer related mortality (1,2). On a population of almost 17 million inhabitants, in 2016 over 12,000 persons were diagnosed with primary lung cancer (2,3). Of these, the vast majority is non-small cell lung cancer (NSCLC).

In The Netherlands, there is a national multidisciplinary evidence-based guideline on the diagnoses and treatment of NSCLC, which is revised about every 5 years. Despite this, in 2010, a study using population-based data from the Dutch Cancer Registry showed significant regional differences and between-hospital variation in treatment patterns and outcomes for patients with NSCLC and other malignancies, though reasons for these differences could hardly be identified (4). In order to improve the quality and equality of cancer care in The Netherlands, multidisciplinary quality standards were developed, by The Dutch Federation of Oncologic Societies (5). In these standards general and cancer specific requirements for optimal cancer care are described, including organization of care, the presence of certain facilities and minimum volume standards.

At the same time, nationwide clinical audits facilitated by the Dutch Institute for Clinical Auditing (DICA) were introduced in The Netherlands. Clinical auditing is a process of systematic analysis of quality of healthcare, with the aim to improve patient outcomes. With a clinical audit system, guideline adherence, patient outcomes and other quality indicator results can be accurately studied and compared.

The first Dutch lung cancer-specific audit started in 2012 and was focussed on surgical treatment. Since then more specialties involved in lung cancer care have joined and nowadays quality of lung cancer care is evaluated multidisciplinary in the nation-wide Dutch Lung Cancer Audit (DLCA).

This article provides insight into the DLCA; its core principles, initiation and development, first results and what lessons can be learned from the successful Dutch experience.

Methods

Aim

The care for patients with lung cancer ideally takes place in a multidisciplinary setting, for both the diagnostic and treatment process. The DLCA therefore is a collaboration of multiple disciplines involved in the treatment of lung cancer. The aim of the DLCA is to evaluate the multidisciplinary care for lung cancer patients, with the potential to improve care processes and outcomes on a national level.

Development

The development of the DLCA was facilitated by DICA and design was in accordance with the DICA blueprint (6). In 2012 the first national quality registry on lung cancer was initiated, focussing on surgical treatment: the Dutch Lung Surgery Audit (7). In The Netherlands, lung surgery is performed by cardiothoracic surgeons and by general surgeons with a specialization in lung surgery. Initially, mainly the hospitals with general surgeons participated in the audit, but from 2015 on all cardiothoracic centres joined as well. In 2014, a quality registry was launched focussing on the radiotherapeutic treatment of lung cancer: the Dutch Lung Radiotherapy Audit. As of 2016, in addition to these two registries, pulmonologists joined the audit. Therefore, from 2016 on the audit was renamed as the DLCA, with sub-registries for lung oncologists (DLCA-L), surgeons (DLCA-S) and radiation-oncologists (DLCA-R), together encompassing the whole care path of lung cancer patients in Dutch hospitals.

The DLCA was developed in close collaboration with all relevant professional associations (the Dutch Society of Physicians for Lung Diseases and Tuberculosis—NVALT, The Netherlands Association for Cardio-Thoracic Surgery—NvT, the Dutch Society for Lung Surgery—NVvL-NVvH and the Dutch Society for Radiotherapy and Oncology—NVRO).

Organisation

The organizational structure of the DLCA is visualized in Figure 1. Clinicians mandated by their professional association and a patient representative form a joint clinical audit board (CAB). The CAB is responsible for the development and progress of the complete audit. Overarching quality issues, interdisciplinary quality indicators and joined meetings are the responsibility of the board. In addition, the three sub-registries have their own scientific committee (SC), responsible for the content of the audit and participation of their colleagues in the institutions providing lung cancer care. In the CAB each SC is represented by its chairman. The audit is supported by the DICA scientific bureau, which in turn is backed by a methodological council and a privacy committee. The SC’s have approximately three separate meetings a year in which the datasets, results and future goals are discussed. The joint results and objectives are discussed in the CAB approximately twice a year.

Figure 1 Organisational structure of DLCA. L, lung-oncologists; R, radiotherapeutic-oncologists; S, surgical-oncologists; MRDM, Medical Research Data Management. DICA, Dutch Institute for Clinical Auditing; DLCA, Dutch Lung Cancer Audit; FMS, Federatie Medisch Specialisten/Federation Medical Specialists; ZiNL, Zorginstituut Nederland/National Health Care Institute; ZN, Zorgverzekeraars Nederland; NFU, Nederlandse Federatie van Universitair medische centra/Netherlands Federation of University Medical Centres; NVZ, Nederlandse Vereniging van Ziekenhuizen/Dutch Hospital Association; NPCF, Patiëntenfederatie Nederland/Dutch Patient Federation.

Funding

The development and implementation of all DLCA sub-registries were project based. These projects were funded and executed via quality improvement grants from the federation of medical specialists [Federatie Medisch Specialisten (FMS), Stichting Kwaliteitsgelden Medisch Specialisten (SKMS)]. Since 2017, the DLCA is completely financed by an umbrella organization of ten healthcare insurance companies in The Netherlands [Zorgverzekeraars Nederland (ZN)]. Apart from funding, these companies do not influence the DLCA organisation. Costs of data registration for participating hospitals are not centrally compensated.

Inclusion

The DLCA includes all patients with primary lung cancer of any stage. In addition, in the DLCA-S there are audit possibilities for patients undergoing surgery for other mediastinal diseases, lung metastasis or benign lung diseases. In the DLCA-L, besides primary lung cancer, there is also a minimal registration of patients with malignant mesothelioma and thymomas or thymic carcinoma. In the DLCA-R only patients with stage I–III disease, treated with curative intent, are included. In the DLCA-L and -S this selection does not apply.

Dataset

The collected data is primarily based on established or future quality indicators reflecting quality of care on a hospital level and potential casemix factors one should account for in between-hospital comparisons. The International Consortium of Health Outcome Measurement (ICHOM) standard dataset was adopted as much as possible (8).

Registered information for casemix adjustment includes baseline patient (e.g., age, gender, performance score) and tumour characteristics (e.g., disease stage and histology). The development of a suitable casemix model is subject of a separate methodology that is described elsewhere (9). Furthermore, the registry includes items regarding processes of care [e.g., modalities used in the diagnostic process, time to treatment and evaluation of the patient in a multidisciplinary team (MDT) meeting] and outcomes (e.g., short-term mortality, complications or toxicity, reinterventions and length-of-hospital-stay).

The content of the dataset is evaluated by the SC and can be adjusted on a yearly base.

Data collection and security

Data collection is preferably prospective and takes place through a secured web-based survey system or via batches of data uploaded by the hospital. Data can be supplemented or modified online when needed, for instance when follow-up information is available. An example of the web-interface of the DLCA-S data collection and feedback report is shown in Figure 2. Hospitals can decide themselves which method they prefer and who carries out data collection (for example: clinicians themselves or trained data-managers). For every hospital, the final responsibility for the completeness and correctness of collected data rests with a clinician. The ownership of the own data remains with the hospital. Current data dictionaries are freely available online (10).

Figure 2 Illustration: “What does the doctor see?”. Example of the web-interface of the DLCA data collection (A), feedback reports (B) and authorization portal for external transparency (C).

There is a close cooperation with a data processor: Medical Research Data Management (MRDM) for data collection, encryption and safe storage. MRDM has a user agreement with all participating hospitals. A service-desk is available by telephone or e-mail during working hours for all questions.

Data quality

Assurance of data quality takes place in multiple ways. One of these is the on-site verification of registered data in the (electronic) patient records of the hospital, by an independent third party. During the data verification, the completeness of patient inclusion by hospitals is checked, as well as the accuracy of the most important data on patient level. Verification takes place for the first time approximately 3 years after the start of the registry. Thus, a registry has to be more “mature” for this data validity check. The data verification process and results of the DLCA-S verification in 2016, as well as other methods to assure data quality are described as a separate topic by Hoeijmakers et al. (11).

Auditing process

Feedback information is provided through weekly updated online reports, of which an example of the DLCA-S is shown in Figure 2. Participating hospitals can use these reports to continuously monitor their results compared to a national benchmark. DICA provides two types of online reports: “the basic report” and “the indicator report”. The more unprocessed information on the treated population is displayed in the basic report and divided into different sections. The indicator scores are displayed in the indicator report, typically in funnel plots with 95% confidence intervals around the national average or a defined norm and adjusted for casemix factors when relevant.

Quality indicators and transparency

To reflect quality of care on a hospital level, quality indicators were developed. Quality indicators primarily serve as information for healthcare providers (internal use). Clinicians thus play a leading role in the development and determination of the DLCA indicator sets.

Since quality information is also of interest for other parties in the Dutch healthcare system (e.g., insurance companies, patient federations, government), a part of the set is agreed to be of use as transparent information (external use or “transparency”). Indicators are tested on relevance, validity, reliability and feasibility (12). The decision whether an indicator is suitable for external use is made tripartite with mandated representatives from the SC and the external parties in Figure 1. In accordance with the Donabedian concept, indicator sets consist of structure, process and outcome indicators (13). Twice a year the content of indicator sets for public transparency is discussed between all relevant parties in a meeting facilitated by DICA’s scientific bureau.

Public transparency of hospital specific indicator scores follows a stepwise model: participation and structure indicators are released in the first year of the audit, process indicators in the second and outcome indicators in the third. External indicator scores are calculated after a “database-lock” three months after expiry of the registration year. Hospitals are obliged to provide external parties with their external indicator scores. Hospital boards are facilitated to share this information with different stakeholders after the annual database lock through a web-based authorization portal facilitated by DICA, shown in Figure 2.

Statistical analysis

Information of all patients registered in the DLCA for primary lung cancer between 1 January 2012 and 31 December 2016 (DLCA-R: 2014–2016, DLCA-L: 2016) was used for analysis. A minimum number of items per patient was required in order to consider a patient eligible for analysis. Descriptive statistics were used to assess patient, tumour and treatment characteristics for all analysable patients with a NSCLC registered in the DLCA-L, -R or -S in 2016.

Statistical analyses were performed using SPSS (IBM SPSS Statistics for Macintosh, Version 23.0).

Results

Growing participation in the DLCA

Figure 3 displays the development of the DLCA from 2012 on. The number of participating hospitals and included patients with a NSCLC increased over time. The participation of all cardiothoracic centres in the DLCA-S from 2015 on is also clearly visible in this figure.

Figure 3 Evolution of the DLCA. Number of participating hospitals and number of registered patients with NSCLC per sub-registry. L, lung-oncologists; R, radiotherapeutic-oncologists; S, surgical-oncologists; DLCA, Dutch Lung Cancer Audit; NSCLC, non-small cell lung cancer.

With the more multidisciplinary character of the audit, the number of cooperating specialists in the DLCA CAB and SCs rose to 55 (representing five medical professional associations) and one patient representative.

Patient characteristics

In 2016, the total numbers of registered patients in DLCA-L, -R and -S were respectively 4,544, 2,883 and 2,391. For the DLCA-L 4,192 patients (92.3%) were considered eligible for analysis. For the DLCA-R and -S the number of analysable patients were 2,767 (96.0%) and 2,349 (98.2%) respectively.

Of these analysable patients, in DLCA-L 3,502 (83.5%) were diagnosed with NSCLC. In the DLCA-R 2,427 (87.7%) and in the DLCA-S 1,979 (84.2%) NSCLC patients were included.

Patient and tumour characteristics of all patients with a NSCLC included in the DLCA in 2016 are shown in Table 1, stratified per DLCA sub-registry. As expected, there are differences in these characteristics between the sub-registries, with the surgically treated patients being younger, with better performance score and more frequently having a clinically stage I–II NSCLC.

Table 1 Patient and tumour characteristics of patients a NSCLC in the DLCA-L, -R and -S in 2016
Full table

Diagnostic characteristics

Of all analysable patients with NSCLC registered in the DLCA-L in 2016 (n=3,502), 2,867 (81.9%) had pathologically proven disease. Of these the majority was proven histologically (1,566, 54.6%) (Table 2). Of all analysable patients with a NSCLC registered in the DLCA-R (n=2,427), 1,230 (50.7%) had pathologically proven disease, 703 (29.0%) did not and in 494 (20.4%) it was not recorded in the database (data not shown).

Table 2 Diagnostic and treatment characteristics of patients with a NSCLC in the DLCA-L in 2016 (N=3,502)
Full table

The most used invasive diagnostic, according to the DLCA-L, was endoscopic ultrasound, with 985 of 3,502 (28.1%) undergoing an endoesophageal ultrasound (EUS) and/or endobronchial ultrasound (EBUS) in the diagnostic work-up of a NSCLC (Table 2).

Treatment characteristics

Treatment plan (DLCA-L)

Of all analysable patients with NSCLC registered in the DLCA-L in 2016 (n=3,502), the primary treatment goal was curative in 1,449 patients (41.4%) and palliative in 1,787 patients (51.0%). In 266 patients (7.6%) information on the treatment plan is missing.

An active anti-tumour treatment (n=2,350) comprised of combinations of surgery, radiotherapy, chemotherapy, immunotherapy and targeted therapy. Surgery, whether or not combined with another treatment, was planned for 615 patients (26.2%). Radiotherapy, whether or not combined with another non-surgical treatment, was applied in 933 patients (39.7%). And another 618 patients (26.3%) were planned for systemic therapy only (Table 2).

Radiotherapy (DLCA-R)

Of all analysable patients with a (stage I–III) NSCLC undergoing radiotherapeutic treatment and registered in the DLCA-R (n=2,427), most were treated with Stereotactic Body Radiation Therapy (SBRT, n=1,294, 53.3%). Other patients were treated with conventional radiotherapy (296, 12.2%) or chemoradiotherapy (837, 34.5%), of which 64.2% was delivered concurrently (Table 3).

Table 3 Treatment characteristics of patients with a NSCLC in the DLCA-R in 2016 (N=2,427)
Full table

Surgery (DLCA-S)

Of all analysable patients with NSCLC undergoing surgery and registered in the DLCA-S (n=1,979), 166 (8.4%) underwent a pneumonectomy, 1,618 (81.8%) a (bi)lobectomy, 55 (2.8%) an anatomic segment resection, 127 (6.4%) a subparenchymal resection and 13 (0.7%) did not undergo a resection. Most operations were started with a minimal invasive technique, video or robotic assisted resection (VATS or RATS): 1,409 (71.2%), with a conversion rate of 18.7% (n=263) (Table 4).

Table 4 Treatment characteristics of patients with a NSCLC in the DLCA-S in 2016 (N=1,979)
Full table

Quality indicators

In Table 5 all quality indicators that are part of the externally transparent indicator sets in 2016, 2017 and 2018 are demonstrated. Also shown in Table 5 are the results of national quality indicators of 2016, with for all indicators the number of patients included in the indicator calculation (“denominator”) and the percentage meeting the indicator definition (numerator divided by denominator). For the two volume indicators only the numerators are displayed.

Table 5 External indicator sets for lung carcinoma from 2016–2018 and national indicator scores in 2016
Full table

The structure indicator “completeness of data entry…” is scored “yes” on patient level when all items required for calculating external indicators are registered and thereby gives an indication on the validity of other indicator results. In 2016, the average data completeness on patient level was 90.7% in the DLCA-S and 84.0% in the DLCA-R.

The percentage of patients with NSCLC discussed in a meeting prior to radiotherapeutic treatment was 95.1%. Of the surgically treated patients with a NSCLC, 97.1% was discussed in a postoperative MDT meeting, an increase of 15% compared to the 82.2% in 2012.

Outcome indicator results are calculated over a period of two consecutive years. The national average 30-day/in-hospital mortality of patients after a resection for primary lung cancer was 2.3% in 2015–2016. The national average 90-day mortality of all patients treated with combined chemoradiotherapy for a primary NSCLC was 6.4% in 2015–2016.

Discussion

In The Netherlands, clinical audits are integrated as a part of the professional quality system. This report provides insight in the DLCA, one of the first nationwide-implemented quality registries to evaluate the multidisciplinary care for patients with lung cancer worldwide. The DLCA was developed according to the blueprint of the DICA, one of the leading organizations facilitating clinical auditing in the Netherlands. Although the audit in its current format is still “immature” (with 2016 as the first registration year for lung oncologists), the core principles are clear. In this paper the first results were presented.

Several initiatives to monitor quality of (surgical) lung cancer care have been developed worldwide (14-19). The design and intents of these initiatives differ in various ways.

The DLCA distinguishes itself from other initiatives through; the central role of clinicians, weekly updated feedback information with national benchmark information, participation of all major treating specialisms, a centrally financed system and close collaboration with other parties in healthcare with tripartite agreements on data transparency. Furthermore, participation in the DLCA has been incorporated in the professional quality system, thereby stimulating nationwide implementation and unbiased information, in contrast with registries with a more voluntary nature. Implementation of evidence-based guidelines and quality standards is evaluated with the audit, on a local as well as a national level.

Design and implementation of the DLCA sub-registries has been a phased process. After independent data-verification of the surgical part of the audit, the data of the DLCA-S are considered mature and the data of the DLCA-R will follow soon. The pulmonologists joined the DLCA-L only recently and the number of analysable patients with NSCLC included in this sub-registry in 2016 is limited to 35–40 percent of the national incidence (2). It is expected that case ascertainment will rapidly increase over time, especially from the moment hospitals are provided with benchmarked feedback (6,7,20,21). The great incentive for clinicians to participate in the audit is the information they receive on the quality of their performance in clinical practice with indicator results benchmarked to the national average (intrinsic motivation). In addition, The Netherlands Healthcare Inspectorate demands participation in the audit, insurance companies use the audit information for reimbursement and the National Healthcare Institute demands indicator scores from the audit for public transparency, which makes participation more or less mandatory for hospitals (external stimulus).

In this first year of multidisciplinary collaboration in the DLCA, one of the biggest advantages experienced by all specialism was the opportunity to address overlapping issues in the combined CAB and SC meetings and the quick implementation of new knowledge into national clinical practice. This has contributed to the implementation of TNM8 in the DLCA in 2017, only a few months after publication (22), leading to a nationwide in-hospital adoption of TNM8.

Sub-registries in themselves also provided important information already. DLCA-S data showed that national use of minimally invasive techniques (VATS/RATS) is high, around 70%. Internationally this percentage varies between 22–63% (23,24). Postoperative mortality after primary lung cancer resection in the Netherlands has been as low as between 2.0% and 2.5% from the start of the DLCA-S. This is comparable to international data (23-26).

An important issue that arose from the DLCA-S is the unfavourable quality of staging compared to for example Denmark (27,28). In the DLCA-S staging accuracy was assessed comparing clinical with pathological TNM stage, regardless of whether discrepancies influenced treatment strategy. This definition differs from Danish studies, which reported inaccuracy only if this had clinical consequences. Nevertheless, the DLCA studies demonstrated there is room for improvement in preoperative staging in the Netherlands. In the diagnostic path leading to a clinical stage, pulmonologists play a major role. Hence, to improve pre-treatment staging, a multidisciplinary approach is essential.

The primary aim of clinical auditing is to improve outcomes for patients by providing meaningful, actable, benchmarked, short-cycled feedback information on daily clinical practice to the MDT in hospitals. Thereby stimulating improvement initiatives on both local and national level. Ultimately, quality assurance for the whole clinical care path of every lung cancer patient is intended. Therefore, the DLCA evolves from a procedure-based mono-disciplinary audit to a condition based multidisciplinary audit, tracking patients from diagnosis until death.

Simultaneously, a shift of focus on structural and process indicators towards outcome indicators, clinical as well as patient reported, is intended. The standard set for lung cancer of the International Consortium of Health Outcome Measurement (ICHOM) was adopted, to be able to participate in international comparisons in the near future. On the other hand, the DLCA is also a platform for clinicians to develop new meaningful quality indicators. DICA’s well-respected agreement with external stakeholders on “stepwise” transparency is imperative in this context, because it gives clinicians the opportunity to evaluate the validity of an indicator and its results, before hospital specific information is made public.

Initially, the idea was to set up the DLCA as a multidisciplinary audit in which multiple disciplines distributed over various hospitals could contribute to registration of one patient. Unfortunately, the construction of such a “chain registry” has not been achieved yet. Largely, this is due to privacy legislation, causing difficulties in sharing patient data across different hospitals (29). Additionally, such a “chain registry” needs clear agreements on who registers what and how, since part of the feedback information (including externally transparent indicators) will be based on information provided by a clinician one might not know. Taking into account the aforementioned issue of low case ascertainment in the novice DLCA-L, completeness of data that should be relied on is not guaranteed. Therefore, the current design was chosen: three DLCA sub-registries (DLCA-L, -R and -S). This has the disadvantage that some information is repeatedly registered unnecessarily.

In extension to this, a general limitation of quality registries is the administrative burden associated with data collection, which frequently rests on the shoulders of clinicians themselves. Still, to evaluate the quality of all essential points in the patients’ care path a substantial amount of data is needed. In addition, proper casemix adjustments are imperative in between-hospital comparisons, for which a set of patient and disease characteristics have to be registered for each case. A meaningful registration may be an administrative burden, though on the other hand reduces the obligations to provide less-meaningful -but externally imposed- indicators to other partners in healthcare (e.g., insurance companies).

One of the solutions to reduce administrative burden is (partly) automated data extraction from existing data sources [e.g., electronic patient records (EPDs), structured reports of diagnostics, treatment or pathology]. Being part of a larger platform, like DICA, can be an advantage in this, when close cooperation is sought between the registry platform, the data processor and hospital-IT-providers.

The main challenge of the DLCA in the (near) future is the integration of three separate sub-registries, the DLCA-L, -R and -S into one “chain registry” system as described above. This integrated system facilitates registration of data by different disciplines and institutions in one patient record, thereby maximizing multidisciplinary quality evaluation possibilities and minimizing administrative burden of the registration.

In addition, there should be more focus on outcome indicators in the audit. Next to clinical outcomes, functional outcomes and quality of life are of great value for patients. Measuring such patient reported outcomes (PROMs) or patient reported experiences (PREMs) in daily practice can be challenging, especially low response rates can hamper valid comparisons between hospitals. Though, linkage of patient reported data to clinical data could provide clinicians with “new” valuable information and can facilitate (shared) personalized treatment decisions.

In conclusion, with the start of the DLCA in 2016, there is a unique nationwide audit system to evaluate the quality of multidisciplinary lung cancer care. The DLCA is accepted and implemented on a nationwide level, enabling healthcare providers insight in their performance together with a national benchmark, and providing other stakeholders with a transparent evaluation of this performance. When challenges of shared data input and access—mostly concerning privacy legalisation—are solved, the accomplishment of a completely integrated audit remains a main aspiration. The possibilities of multidisciplinary quality evaluation will be maximized further, with the highest aim of continuous healthcare improvement.

Acknowledgements

The authors thank all clinicians, physician assistants, nurses and data-managers that register patients in the DLCA.

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

References

1. World Health Orginization. Cancer fact sheet [Internet]. Updated February 2017. Available online: http://www.who.int/mediacentre/factsheets/fs297/en/
2. Integraal Kankercentrum Nederland (IKNL). Cijfers over kanker [Internet]. Available online: https://www.cijfersoverkanker.nl
3. Statistics Netherlands. Population key figures [Internet]. Available online: http://statline.cbs.nl/Statweb/publication/?DM=SLEN&PA=37296ENG&D1=0-2,19-21,23-27,52-55,57-68&D2=60-66&LA=EN&VW=T
4. Wouters MW, Siesling S, Jansen-Landheer ML, et al. Variation in treatment and outcome in patients with non-small cell lung cancer by region, hospital type and volume in the Netherlands. Eur J Surg Oncol 2010;36 Suppl 1:S83-92. [Crossref] [PubMed]
5. SONCOS (Stichting Oncologische Samenwerking). Standardization report 5. 2017. Available online: https://www.soncos.org
6. Van Leersum NJ, Snijders HS, Henneman D, et al. The dutch surgical colorectal audit. Eur J Surg Oncol 2013;39:1063-70. [Crossref] [PubMed]
7. ten Berge M, Beck N, Heineman DJ, et al. Dutch Lung Surgery Audit: A National Audit Comprising Lung and Thoracic Surgery Patients. Ann Thorac Surg 2018;106:390-7. [Crossref] [PubMed]
8. Mak KS, Van Bommel ACM, Stowell C, et al. Defining a standard set of patient-centred outcomes for lung cancer. Eur Respir J 2016;48:852-60. [Crossref] [PubMed]
9. Beck N, Hoeijmakers F, van der Willik EM, et al. National comparison of hospital performances in lung cancer surgery: the role of casemix adjustment. Ann Thorac Surg 2018;106:412-20. [Crossref] [PubMed]
10. DICA/MRDM. Data dictionaries of DICA facilitated audits [Internet]. Available online: https://www.mrdm.nl/showcase/downloaden
11. Hoeijmakers F, Beck N, Prins HA, et al. How to improve the reliability/quality of data? J Thorac Dis 2018. In press.
12. Mainz J. Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care 2003;15:523-30. [Crossref] [PubMed]
13. Donabedian A. The quality of care. How can it be assessed? JAMA 1988;260:1743-8. [Crossref] [PubMed]
14. Society of Thoracic Surgeons. STS national database [Internet]. Available online: http://www.sts.org/national-database
15. European Society of Thoracic Surgeons. ESTS international database [Internet]. Available online: http://www.ests.org
16. Nationella Kvalitetsregister. Swedish National Quality Registry for Lung Cancer [Internet]. Available online: http://kvalitetsregister.se/englishpages/findaregistry/registerarkivenglish/nationalqualityregistryforlungcancer.2280.html
17. Falcoz PE, Conti M, Brouchet L, et al. The Thoracic Surgery Scoring System (Thoracoscore): Risk model for in-hospital death in 15,183 patients requiring thoracic surgery. J Thorac Cardiovasc Surg 2007;133:325-32. [Crossref] [PubMed]
18. Jakobsen E, Palshof T, Østerlind K, et al. Data from a national lung cancer registry contributes to improve outcome and quality of surgery: Danish results. Eur J Cardiothorac Surg 2009;35:348-52; discussion 352. [Crossref] [PubMed]
19. Rich AL, Tata LJ, Stanley RA, et al. Lung cancer in England: Information from the National Lung Cancer Audit (LUCADA). Lung Cancer 2011;72:16-22. [Crossref] [PubMed]
20. Busweiler LA, Wijnhoven BP, van Berge Henegouwen MI, et al. Early outcomes from the Dutch Upper Gastrointestinal Cancer Audit. Br J Surg 2016;103:1855-63. [Crossref] [PubMed]
21. van Bommel AC, Spronk PE, Vrancken Peeters MT, et al. Clinical auditing as an instrument for quality improvement in breast cancer care in the Netherlands: The national NABON Breast Cancer Audit. J Surg Oncol 2017;115:243-9. [Crossref] [PubMed]
22. Goldstraw P, Chansky K, Crowley J, et al. The IASLC lung cancer staging project: Proposals for revision of the TNM stage groupings in the forthcoming (eighth) edition of the TNM Classification for lung cancer. J Thorac Oncol 2016;11:39-51. [Crossref] [PubMed]
23. Seder CW, Salati M, Kozower BD, et al. Variation in Pulmonary Resection Practices between the Society of Thoracic Surgeons and the European Society of Thoracic Surgeons General Thoracic Surgery Databases. Ann Thorac Surg 2016;101:2077-84. [Crossref] [PubMed]
24. Green A, Hauge J, Iachina M, et al. The mortality after surgery in primary lung cancer: Results from the Danish Lung Cancer Registry. Eur J Cardiothorac Surg 2016;49:589-94. [Crossref] [PubMed]
25. Morgant MC, Pagès PB, Orsini B, et al. Time trends in surgery for lung cancer in France from 2005 to 2012: A nationwide study. Eur Respir J 2015;46:1131-9. [Crossref] [PubMed]
26. Powell HA, Tata LJ, Baldwin DR, et al. Early mortality after surgical resection for lung cancer: an analysis of the English National Lung cancer audit. Thorax 2013;68:826-34. [Crossref] [PubMed]
27. Heineman DJ, ten Berge MG, Daniels JM, et al. The Quality of Staging Non-Small Cell Lung Cancer in the Netherlands: Data From the Dutch Lung Surgery Audit. Ann Thorac Surg 2016;102:1622-9. [Crossref] [PubMed]
28. Jakobsen E, Green A, Oesterlind K, et al. Nationwide quality improvement in lung cancer care: The role of the Danish Lung Cancer Group and Registry. J Thorac Oncol 2013;8:1238-47. [Crossref] [PubMed]
29. Dutch Senate. Voting - Change of the Law on use of civil service number in healthcare, in relation to electronic information exchange in healthcare (31466). 2011:4-6. Available online: http://www.eerstekamer.nl/behandeling/20110405/stemming_wetsvoorstel_verworpen_en/f=/viozdip1gx6j.pdf