Predictive tools in VVECMO patients: handicap or benefit for clinical practice?

Venovenous extracorporeal membrane oxygenation (VVECMO) is being increasingly employed as a rescue intervention and a temporary treatment for patients with severe acute respiratory failure who are refractory to conventional treatment. There has not been established a universally accepted consensus on the indication for VVECMO. Many clinicians will follow the suggestions for use of the international Extracorporeal Life Support Organization (1), which also lists a number of relative contra-indications known to increase mortality. However, the final decision is made by the discretion of experienced clinicians and may vary between institutions as well as therapists. Due to the lack of rigid guidelines, several risk prediction models have been developed aiming to aid clinicians in their decision for VVECMO.

Hilder and colleagues recently introduced us to the PRESET score, a new prediction score for hospital mortality in VVECMO patients (2). Besides externally validating four pre-existing risk scores [ECMOnet score (3), RESP score (4), PRESERVE score (5) and Roch score (6)], they constructed a new model incorporating five extrapulmonary variables. They validated the PRESET score in an independent, external cohort. In their local cohort, their novel model predicted mortality more accurate than previous scores and was therefore claimed to be a more precise choice for decision support in patients with acute respiratory distress syndrome to be placed on VVECMO.

Rozencwajg et al. have made a systematic overview of pre-existing risk models up to 2016 and compared them (7). The PRESET risk model consisted of arterial pH at admission, mean arterial pressure, lactate, platelet concentrations and pre-ECMO length of hospital stay. They have categorized each variable, yielding an end total score between 0–15. Referring to the table by Rozencwajg and colleagues (reproduced with modifications in Table 1), we can see that some of the variables are overlapping with known factors, whereas others are new.

Table 1 Survival predictive models for patients on VVECMO for ARDS
Full table

The differences in the models and their performance in new patient cohorts lead to questions regarding the feasibility of using mortality risk prediction models in VVECMO patients. Every prediction rule will only be as good as the collection of underlying data. Differences in the model composition may be a result of the heterogeneity of the VVECMO databases, in terms of size, population and the data variables collected. A priori pre-selection of patients by institutional guidelines may irrevocably alter the final results for a prediction model. For example, age will not be a relevant factor if older patients are denied ECMO on principle. Further hazards may be related to small numbers of patients included or correlation between variables, like arterial pH and lactate, or mean arterial pressure and lactate. The varying in-hospital mortality rates might reflect the differences in the decision for ECMO, but may also be related to local treatment variations.

We observe an increased attention for prediction models in the literature and in clinical practice. Unfortunately, a prediction model often works well for the local population of patients, but shows significantly poorer predictive abilities when applied to external cohorts. When acceptance or denial of a possible treatment may be potentially crucial for survival, an area under the receiver-operating-curve (AUC) of 0.70 must not be considered to be adequate itself. Five out of seven prediction models for VVECMO patients have been externally validated (Table 1). As expected, the models generally performed best in the patient cohort from which they were developed. Poorer performance in new patient cohorts led to the construction of new models. The new model by Hilder et al. adds to the total list, but it is difficult to excerpt what they did differently. The challenges and limitations brought up by Rozencwajg et al. remain unsolved: Hilder et al. used the same statistical methods for development (logistic regression analyses) and validation (AUC). They did not perform bootstrapping nor mixed or random effects models as asked for by Rozencwajg et al. We are at a standstill.

The outlined main purpose of the risk models has been to aid individual case management. It is indisputable that the prediction rules do not replace clinical evaluation of the patient. They represent a supplementary tool for clinicians in their decision-making process. Other purposes mentioned include use in research and for quality improvement. However, has anyone employed any risk models to their patients? Did they experience any usefulness? And not at least, did they evaluate their usefulness?

It is important to improve the scientific approaches for evaluating prediction models. We should adapt to general guidelines for prediction modelling, following the steps of development, validation and updating, impact and implementation, as outlined by Toll et al. (13). Moreover, rather than starting from scratch for each new patient cohort, we should try to build on previous findings and see if we can adjust or update it, rather than replace it. Janssen et al. have described methods of updating prediction rules, from adjustment of the intercept only, to adjustment of regression coefficients of predictors with or without inclusion of additional predictors (14). The updated model should be based on additional patient data, thereby expanding the dataset, yielding better risk estimates and improving its calibration and/or discrimination.

Whereas the evaluation of calibration and discrimination often are useful first steps in evaluating a model or in comparing two prediction models against each other, the AUC value is insufficient to demonstrate that a model would improve decision-making (15). Novel measures related to clinical usefulness, including calculation of net reclassification and decision curve analyses have been well-established (16). Closer attention to these guidelines and keeping updated with statistical methods and tools may help us lift our research to the next level.

We want to encourage a shift in study focus, from continuously developing new models, to elaborating the ones we have, continue to improve them and work on integration into clinical practice. It has been discussed whether large patient heterogeneity amongst those supported with VVECMO may limit the usefulness. Evaluating the usefulness of current models will help us further in the discussion.

Acknowledgements

None.

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

References

1. Extracorporeal Life Support Organization: ELSO Guidelines: Patient Care Practice Guidelines. Available online: https://www.elso.org/resources/guidelines.aspx
2. Hilder M, Herbstreit F, Adamzik M, et al. Comparison of mortality prediction models in acute respiratory distress syndrome undergoing extracorporeal membrane oxygenation and development of a novel prediction score: the PREdiction of Survival on ECMO Therapy-Score (PRESET-Score). Crit Care 2017;21:301. [Crossref] [PubMed]
3. Pappalardo F, Pieri M, Greco T, et al. Predicting mortality risk in patients undergoing venovenous ECMO for ARDS due to influenza A (H1N1) pneumonia: the ECMOnet score. Intensive Care Med 2013;39:275-81. [Crossref] [PubMed]
4. Schmidt M, Bailey M, Sheldrake J, et al. Predicting survival after extracorporeal membrane oxygenation for severe acute respiratory failure. The Respiratory Extracorporeal Membrane Oxygenation Survival Prediction (RESP) Score. Am J Respir Crit Care Med 2014;189:1374-82. [Crossref] [PubMed]
5. Schmidt M, Zogheib E, Rozé H, et al. The PRESERVE mortality risk score and analysis of long-term outcomes after extracorporeal membrane oxygenation for severe acute respiratory distress syndrome. Intensive Care Med 2013;39:1704-13. [Crossref] [PubMed]
6. Roch A, Hraiech S, Masson E, et al. Outcome of acute respiratory distress syndrome patients treated with extracorporeal membrane oxygenation and brought to a referral center. Intensive Care Med 2014;40:74-83. [Crossref] [PubMed]
7. Rozencwajg S, Pilcher D, Combes A, et al. Outcomes and survival prediction models for severe adult acute respiratory distress syndrome treated with extracorporeal membrane oxygenation. Crit Care 2016;20:392. [Crossref] [PubMed]
8. Enger T, Philipp A, Videm V, et al. Prediction of mortality in adult patients with severe acute lung failure receiving veno-venous extracorporeal membrane oxygenation: a prospective observational study. Crit Care 2014;18:R67. [Crossref] [PubMed]
9. Liu X, Xu Y, Zhang R, et al. Survival predictors for severe ARDS patients treated with extracorporeal membrane oxygenation: A retrospective study in China. PLoS One 2016;11:e0158061. [Crossref] [PubMed]
10. Cheng YT, Wu MY, Chang YS, et al. Developing a simple preinterventional score to predict hospital mortality in adult venovenous extracorporeal membrane oxygenation: A pilot study. Medicine (Baltimore) 2016;95:e4380. [Crossref] [PubMed]
11. Klinzing S, Wenger U, Steiger P, et al. External validation of scores proposed for estimation of survival probability of patients with severe adult respiratory distress syndrome undergoing extracorporeal membrane oxygenation therapy: a retrospective study. Crit Care 2015;19:142. [Crossref] [PubMed]
12. Brunet J, Valette X, Buklas D, et al. Predicting Survival after extracorporeal membrane oxygenation for ARDS: An External Validation of RESP and PRESERVE Scores. Respir Care 2017;62:912-9. [Crossref] [PubMed]
13. Toll DB, Janssen KJM, Vergouwe Y, et al. Validation, updating and impact of clinical prediction rules: A review. J Clin Epidemiol 2008;61:1085-94. [Crossref] [PubMed]
14. Janssen KJ, Moons KG, Kalkman CJ, et al. Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 2008;61:76-86. [Crossref] [PubMed]
15. Mallett S, Halligan S, Thompson M, et al. Interpreting diagnostic accuracy studies for patient care. BMJ 2012;345:e3999. [Crossref] [PubMed]
16. Steyerberg EW, Yvonne V. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J 2014;35:1925-31. [Crossref] [PubMed]