Mortality prediction algorithms for patients undergoing primary percutaneous coronary intervention

Introduction

Mortality risk of ST-segment elevation myocardial infarction (STEMI) patients shows high variability. In order to assess individual risk, a number of mathematical models (scoring algorithms) have been developed. Yet, as treatment approaches evolve over time with improving outcomes and as ever older patients with complex disease patterns are treated invasively, there is a need to build new risk prediction algorithms to maintain/increase prognostic accuracy. One of the most relevant improvements of therapy is primary percutaneous coronary intervention (PPCI), since, compared with fibrinolysis, it further reduces mortality. Therefore, it is the treatment of choice according to both American and European guidelines (1-3). Prediction algorithms may provide useful information for patients/relatives and help physicians to allocate hospital resources. Moreover, they may contribute to an improved quality of care as they can be used for risk adjustment in inter-organizational comparisons of health care providers with different case mixes. They also enable intra organizational quality monitoring. Furthermore, risk models may be helpful in clinical trial design identifying patients with the needed risk profile thereby increasing statistical power/reducing sample size and costs.

Methods

PubMed (https://pubmed.gov) was searched for English language mortality risk models that were developed using (at least in part) data of STEMI patients (Table 1). Other risk prediction algorithms (e.g., different derivations of the SYNTAX score, which was developed excluding cases with myocardial infarction) were not considered (4,5). After identifying the models, we sought for their external validation studies. Only reports with populations involving STEMI and primary PCI as a treatment modality (at least partly) were analyzed.

Table 1 Characteristics of the derivation data set of mortality risk models
Full table

General characteristics of risk predicting algorithms

The general characteristics of the derivation and validation studies of the analyzed mortality risk models are summarized in Tables 1 and 2.

Table 2 Validation studies of mortality risk models in patients with ST-segment elevation myocardial infarction
Full table

Clinical setting

Most of the studied risk predicting algorithms were constructed using data of patients with STEMI exclusively (6-13) (Table 1). Yet, the “Acute Coronary Treatment and Intervention Outcomes Network Registry-Get With the Guidelines” (AR-G) model was derived from mixed data of STEMI and non-STEMI cases (14), whereas the “Global Registry of Acute Coronary Events” GRACE 2.0 score was developed using data of all three types (i.e., STEMI, non-STEMI, and unstable angina pectoris) of acute coronary syndrome (ACS) (15). In contrast, the developers of the “National Cardiovascular Data Registry for Catheterization Percutaneous Coronary Intervention” NCDR CathPCI model used data of all coronary artery disease (CAD) patients who underwent PCI, regardless of disease acuity (16). Similarly to the training data sets, clinical settings of the validation studies also vary (Table 3).

Table 3 Characteristics and composition of mortality risk models
Full table

Treatment modality

In the development set of the “Thrombolysis in Myocardial Infarction” (TIMI) risk score, the oldest of the analyzed algorithms (6), and the derivative “Dynamic TIMI” model (12) patients were treated with fibrinolysis. In contrast, newer scores used primary PCI (7-11,13) or PCI (16), depending on the clinical setting, as a treatment modality. Nevertheless, in the AR-G algorithm for STEMI and non STEMI (14) and in the GRACE 2.0 model developed for all types of the ACS (15), the therapeutic modality was not specified. Likewise, treatment was not restricted to primary PCI/PCI in many of the validation studies (Table 2).

Source of data

Older models often used randomized controlled trials (RCTs) as a data source for derivation (6,7,9,10,12) (Table 1). While RCTs generally have excellent data quality, due to the strict inclusion and exclusion criteria, their participants may not be fully representative of the whole population. Moreover, in the case of some models, patients with cardiogenic shock were completely excluded from the derivation data set (6,7,9,12). Missing or under-representation of prognostically important factors in the derivation data set may result in a systematic misestimation of the regression coefficients and biased prediction, limiting the generalizability of the algorithm. Newer models, however, usually employed derivation data from single- or multi-center registries representing “real-world” patients (8,11,13-16). Likewise, the risk prediction algorithms were mostly validated using registries (Table 2).

Prediction end point

Three of the studied scores were constructed to predict the risk of in-hospital deaths (11,14,16), whereas three used 30-day mortality risk as an outcome measure (6,8,13). The “Assessment of Pexelizumab in Acute Myocardial Infarction” (APEX-AMI) (10) and the “Primary Angioplasty in Myocardial Infarction” (PAMI) (7) models were developed for forecasting 90-day and 6-month mortality risks, respectively. For long-term (1 year) prognosis, the” Controlled Abciximab and Device Investigation to Lower Late Angioplasty Complications” (CADILLAC) (9), the dynamic TIMI (12), and the GRACE 2.0 (15) scores were developed. Latter was also designed to predict in-hospital, 6-month, and 3-year mortality risks. Irrespective of the outcome measure used for derivation, most validation studies used in hospital, 30-day, and/or 1-year mortality risks as prediction end points (Table 2).

Predictors/time of assessment

Some of the models use exclusively predictors that are available at presentation like demographic and historical data, presentation and ECG characteristics (6,7) (“admission model”), while others also make use of findings/results of the coronary intervention and/or more time consuming imaging/laboratory studies/in-hospital events assessing risk later during the hospital stay (8-11,13-16) or only at the time of discharge (12) (“discharge model”) (Table 3, Figure 1). The most common variables used in the models are age, which is a predictor in each of the studied models (6-16), Killip class/presence of cardiogenic shock/hemodynamic instability (6-12,14-16), heart rate (6,7,10,12-15) and systolic blood pressure at admission (6,10,12-15), ECG localization of the infarction (6-8,10,12), renal function (9,10,14-16), ischemia time (6,8,11,12), and history of diabetes mellitus (6,7,11,12). Each of the variables presented in Table 3 was independently associated with mortality being parts of one or more models. Yet, researchers have to maintain a balance between including too many predictors and model parsimony. Omitting better treatment options, such as primary PCI (6,12) and/or under-representation of other important prognostic factors (e.g., cardiogenic shock) (6,7,9,12) may cause biased prediction. On the other hand, using too many variables may result in loss of precision in the estimation of the coefficients and the predictions of new responses.

Figure 1 Composition of mortality risk scores. Height of the bars shows the number of predictors needed for calculation of the score. Color of the predictor groups corresponds with the time needed for the availability of predictors: blue: variables that are available at or soon after admission (presentation characteristics and procedural data); orange: laboratory and imaging studies requiring some more time; green: in hospital events that can only be assessed at the time of discharge. True admission models are the TIMI and PAMI scores, whereas dynamic TIMI can only be calculated at the time of discharge. With the exception of the GRACE 2.0 and ALPHA models, there is a trend that newer algorithms became more complex with more predictors. TIMI, thrombolysis in myocardial infarction; PAMI, primary angioplasty in myocardial infarction.

Usage

Classic risk scores used points usually derived from the odds/hazard ratios of the predictors and provided mainly relative risk classes. Modern models, however, make use of more complex statistical methods allowing non-linear associations between continuous predictors and the outcome (e.g., APEX-AMI, AR-G, GRACE 2.0, ALPHA), which makes “manual” calculations somewhat difficult. Hence, these algorithms sometimes come with an online calculator/mobile app providing both relative and absolute risks (e.g., GRACE 2.0, ALPHA).

Characteristics of individual risk models

The characteristics of the derivation and validation studies of the analyzed mortality risk models are summarized in Tables 1-3 and Figure 1. Here we give a short description of each of the analyzed algorithms sorted by the treatment modality used in the derivation data set and publication date.

Scores without treatment specification

AR-G

The “Acute Coronary Treatment and Intervention Outcomes Network (ACTION) Registry-Get With the Guidelines (GWTG)” (AR-G) score was constructed using data of both STEMI and non STEMI patients to forecast in hospital deaths (14). Similarly to the GRACE models for all three types of ACS, treatment modality was not specified. Yet, most patients (~80%) were treated with PCI, some 14% with fibrinolysis, whereas around 6% did not get reperfusion therapy. The score consists of 7 patient related and 2 laboratory parameters (Tables 1-3, Figure 1). Though the predictive ability of the model was similarly good in both the training and validation data sets (14), the score has been poorly validated externally (17).

GRACE 2.0

The GRACE 2.0 models (two distinct algorithms for 1-year and 3-year mortality risks) were derived from an international multi-center registry of 32,037 and 1,274 ACS patients, respectively (15). Together with the GRACE 1.0 model, they are capable of predicting in-hospital, 6-month, 1-year, and 3-year mortality risks. Besides 6 parameters that are available at the presentation it evaluates serum creatinine and cardiac necrosis biomarker levels (Tables 1-3, Figure 1; in the “mini” version of the model creatinine level may be substituted by the history of renal failure). The GRACE 1.0 and 2.0 scores have been extensively validated in the STEMI setting (13,15,17-31). An online calculator is available at https://www.outcomes-umassmed.org/grace/acs_risk2/index.html.

Fibrinolysis scores

TIMI

The thrombolysis in myocardial infarction score was developed from data of a multi-center RCT on STEMI patients who were treated with fibrinolysis (6). The model employs 10 variables that are all readily available at admission (Tables 1-3, Figure 1). After the GRACE score, TIMI is the second most validated model (9,13,18,19,24,25,29,30,32,33).

Dynamic TIMI

Similarly to the TIMI model, the dynamic TIMI score has also been constructed using data of a multi-center RCT: the 10 predictors of the TIMI risk score have been amended by 6 major clinical in-hospital events such as myocardial infarction, arrhythmia, major bleeding, stroke, congestive heart failure/shock, and renal failure (Tables 1-3, Figure 1) (12). Therefore, the dynamic TIMI risk score can only be calculated at the time of discharge. This RCT-derived score has been relatively poorly validated (12,19).

Primary PCI/PCI scores

PAMI

The “Primary Angioplasty in Myocardial Infarction” risk score was developed using a mixed population of 3,252 patients from the multi-center PAMI RCTs and the registry arm of the PAMI-2 study to forecast 6-month mortality risk (7). Despite the registry arm, cases with cardiogenic shock were excluded. Likewise to the TIMI model, it solely employs 5 simple parameters that are all accessible at presentation (Tables 1-3, Figure 1). Being the oldest primary PCI model, the PAMI score is well evaluated (9,13,18,19,25,29).

Zwolle

The Zwolle risk score was developed using a single-center registry of 1,791 STEMI patients treated with primary PCI (8). The model consists of 6 variables: besides basic demographics, presentation and ECG characteristics it also makes use of 2 procedural parameters: the presence of triple vessel disease and final TIMI flow (Tables 1-3, Figure 1). The Zwolle risk score was validated in several studies (8,9,13,18,19).

CADILLAC

The risk model was derived to predict 1-year mortality risk from data of the “Controlled Abciximab and Device Investigation to Lower Late Angioplasty Complications” multi-center trial, excluding high-risk cases (9). The algorithm employs 7 variables including procedural and laboratory parameters: age, Killip class, baseline left ventricular ejection fraction (LVEF) as assessed by ventriculography before PCI, presence of triple vessel disease, final TIMI flow grade, anemia, and renal failure (Tables 1-3, Figure 1). Thus, this is the only model that makes use of three predictor categories i.e., presentation characteristics, procedural data, and imaging/laboratory studies. Though LVEF less than 40% was the most important predictor in the model [odds ratio =3.50, 95% confidence interval (CI): 2.07–5.75], ventriculography is rarely performed before/during primary PCI limiting the practical value of the original model. Nevertheless, LVEF can also be estimated using echocardiography as it was done in some of the validation studies (13,19,23). Despite the exclusion of high-risk patients from the derivation data set, the model performs well in registry-based validation analyses as well (9,13,19,23,25,29).

APEX-AMI

Characteristics of the “Assessment of Pexelizumab in Acute Myocardial Infarction” (APEX-AMI) score are shown in Table 1 (10). The model has 6 variables that are available at admission and renal function as a laboratory parameter (Tables 1-3, Figure 1). The c-statistic for 90-day mortality in the derivation set was 0.82. Though the APEX-AMI model underwent internal validation, to our knowledge, only one external validation study was performed showing good predictive ability (13).

NCDR CathPCI

The NCDR CathPCI is a simplified, user-friendly model with 7 patient-related (pre-procedural) parameters and 1 laboratory variable (glomerular filtration rate) to predict in-hospital mortality (16). It was derived from the National Cardiovascular Data Registry using data of both elective and acute PCI procedures (Tables 1-3, Figure 1). Besides external temporal validation using the same registry, it also underwent a fully external evaluation showing good predictive ability (16,22).

EuroHeart STEMI PCI

The EuroHeart ST-segment elevation myocardial infarction PCI model (EH STEMI PCI) was constructed using a multi-center registry to predict in-hospital mortality of patients treated with primary PCI (11). Besides 9 patient-related parameters (this is the only algorithm that contains gender as a predictor), it includes 5 procedural variables (Tables 1-3, Figure 1). Though the model seems to exhibit excellent discriminatory power, to our knowledge it did not undergo fully external validation (it was only validated using the very same registry randomly divided into derivation and validation data sets) (11).

ALPHA

Transradial primary PCI has been shown to reduce mortality risk in several randomized trials and is gaining popularity worldwide (34-37). In light of that, current European guidelines on STEMI and myocardial revascularization give a class IA indication for routine radial access (2,3). To reflect this change in practice, a new risk model including vascular access site has been constructed for predicting 30-day mortality in patients treated with primary PCI (13). The ALPHA model consists of 5 simple predictors such as Age, need for Life support (i.e., cardiac arrest on or prior to admission), systolic blood Pressure and Heart rate at admission, and vascular Access site (Tables 1-3, Figure 1). Besides internal validation, the model underwent temporal and fully external evaluations both suggesting high discriminatory power: c-statistic =0.87 (95% CI: 0.81–0.93) and 0.86 (95% CI: 0.84–0.88), respectively. Moreover, the predictive power of the score was stable for up to one year (13,24). With only 5 parameters that are available at or soon after the presentation the score can be calculated early using the online calculator (https://alphascore.org) providing both relative risk class and absolute 30-day mortality risk. ALPHA is the only mortality risk model that includes the access site as a variable representing contemporary PCI practice. Using the model, patients who may benefit most from transradial access may be identified at presentation. Then, even before coronary angiography, when the actual vascular access site is still not known, estimating the absolute risks for the two approaches and subtracting the radial from the femoral one, the resulting difference equals the absolute risk reduction that is attributable to transradial access. Despite these advantages, the model still awaits further (international) validation.

Comparative validation

Halkin et al. compared the performance of the CADILLAC score (9) with that of the TIMI (6), PAMI (7), and Zwolle (8) models in 900 patients of the Stent-Primary Angioplasty in Myocardial Infarction (Stent-PAMI) trial (Table 2) (38). Though the authors state that the CADILLAC score compared favorably with these previous models in prognostic performance, data of pairwise comparisons has not been reported (9).

Lev et al. analyzed the TIMI (6), PAMI (7), CADILLAC (9), and GRACE (15) models in 855 hemodynamically stable patients from a single-center registry (25). According to the authors, the CADILLAC, TIMI, and PAMI risk scores all had relatively high predictive accuracy for 30-day and 1-year mortality (Table 2), with slight superiority of the CADILLAC score. Surprisingly, the discriminative ability of the GRACE model was not found to be statistically significant. Nevertheless, the results of pairwise comparisons have not been published.

Using a single-center registry, Raposeiras-Roubín et al. studied the AR-G (14) and GRACE (15) models in STEMI patients, who were only partly treated with primary PCI, with in-hospital mortality risk as an outcome measure (Table 2). They found no statistical difference in the predictive performance of the two scores (17).

Méndez-Eirín et al. studied the TIMI (6), PAMI (7), CADILLAC (9), and GRACE (15) models in a single-center cohort of STEMI patients who were treated with primary or rescue PCI, using mortality risks at 30 days and 1 year as end point (Table 2) (29). They found that the TIMI, CADILLAC, and GRACE scores had greater discriminatory power for both 30-day and 1-year mortality than the PAMI model. Also, at 30 days, the GRACE model predicted statistically better than the TIMI model.

The TIMI (6) and GRACE (15) scores were also compared by Timóteo et al. using 607 patients of a single-center registry (Table 2). With that sample size the GRACE score was found to have a better predictive performance for in-hospital but not for 30-day mortality, despite numerical difference (30).

Abelin et al. analyzed the discriminative abilities of the TIMI (6), PAMI (7), Zwolle (8), and GRACE (15) models in a single-center cohort of 501 STEMI patients treated with primary PCI (Table 2) (18). With that sample size, there was no statistically significant difference regarding the predictive accuracy of the TIMI, GRACE, and Zwolle scores for 30-day mortality risk, but the GRACE model was superior to the PAMI algorithm (P<0.01).

Littnerova et al. analyzed the capability of the TIMI (6), dynamic TIMI (12), PAMI (7) Zwolle (8), CADILLAC (9), and GRACE (15) scores to predict mortality from 6 months up to 3 years in 593 STEMI patients of a single-center registry who underwent primary PCI (19). The best predictive values for long-term mortality risk were obtained by the GRACE algorithm, followed by the CADILLAC, Zwolle, and dynamic TIMI models. In contrast, the TIMI and PAMI risk scores were less good at long-term predictions (Table 2). Nevertheless, not all pairwise comparisons were made, the predictive value of the models was compared with that of the GRACE score as a reference.

The GRACE (15) and the NCDR CathPCI (16) algorithms were compared by Timóteo et al. using a single-center cohort of 2,148 ACS patients (of whom 70.9% had STEMI) treated with PCI (22). The authors found that the predictive power of the GRACE model for in hospital mortality risk was statistically greater than that of the NCDR CathPCI score (Table 2).

More recently, Hizoh et al. comparatively determined the c-statistic of the TIMI (9), PAMI (7), Zwolle (8), CADILLAC (9), APEX-AMI (10), GRACE 2.0 (15), and ALPHA (13) models for 30-day mortality risk using a single-center registry cohort of 505 patients (Table 2). The ALPHA, GRACE 2.0, APEX-AMI, and CADILLAC models predicted 30-day mortality risk better than the PAMI score [ALPHA vs. PAMI: difference =0.10 (95% CI: 0.03–0.16), P=0.005; GRACE 2.0 vs. PAMI: difference =0.09 (95% CI: 0.03–0.15), P=0.004; APEX-AMI vs. PAMI: difference =0.08 (95% CI: 0.02–0.15), P=0.01; CADILLAC vs. PAMI: difference =0.08 (95% CI: 0.01–0.14), P=0.02], the remaining comparisons revealed no statistically significant differences. The same group also compared the predictive performance of the ALPHA, GRACE 2.0, and TIMI models for 30 day risk of death in 5,203 patients using a national multi-center registry (24). The analysis showed a high discriminatory power of the GRACE 2.0 model: c-statistic =0.87 (95% CI: 0.85–0.89). Similarly, the ALPHA score performed well with a c-statistic of 0.86 (95% CI: 0.84–0.88). The difference between the two algorithms was not statistically significant (P=0.19). In contrast, the predictive ability of the TIMI score was somewhat weaker with a c-statistic of 0.81 (95% CI: 0.79–0.83). Compared with the GRACE 2.0 and ALPHA models, the difference was statistically significant (P<0.0001, in both comparisons). Thus, the predictive ability of the ALPHA score was similar to that of the more complex GRACE 2.0 model whereas both models performed statistically better than the TIMI score from the fibrinolysis era.

Limitations

In the present review we studied the discriminatory power of the models using receiver operating characteristic (ROC) curve analysis and the c-statistic (39,40). Though this approach is popular and widely accepted, it has some drawbacks. ROC analysis may present an overly optimistic picture of the model on data sets with a class imbalance (i.e., numbers of controls and cases differ substantially), like in the present validation studies. In such situations, presentation of the so called “precision recall curve” would be more appropriate, since these calculations do not make use of the true negatives, they are only concerned with the correct prediction of the less frequent positive events (41). Unfortunately, neither of the studies give such information. Moreover, we did not analyze calibration of the models, because in some works no information is available on that or authors report the result of the Hosmer-Lemeshow test, which is not considered to be an appropriate measure of model fit because of limited power and poor interpretability (39,40).

Also, we did not perform a (network) meta analysis for several reasons. The studied populations were substantially heterogeneous: different clinical settings (STEMI vs. non-STEMI and STEMI vs. ACS), various treatment modalities (primary PCI vs. fibrinolysis vs. no reperfusion therapy), different baseline risks (populations of randomized clinical trials vs. registry data), diverse prediction end points (the inherently heterogeneous in-hospital vs. 30-day vs. 6-month vs. 1-year mortality risks) that might have introduced substantial bias into the results of a network meta-analysis. Moreover, some authors did not provide a 95% confidence interval for the point estimate of the c-statistic which would be necessary for meta-analyses.

Conclusions

Mortality prediction algorithms are useful tools for patients, physicians, and clinical researchers that are also essential for quality control. Though the extensively validated GRACE model was not particularly derived from data of invasively treated STEMI patients, it also performs well in the era of transradial primary PCI. Similarly, the Zwolle, CADILLAC, APEX-AMI, and ALPHA models, that were constructed using primary PCI data, all seem to have comparable discriminative abilities. In contrast, the admission model TIMI, which was developed in the fibrinolysis era, might have less predictive power. Finally, the primary PCI admission model PAMI is likely the weakest among the comparatively studied risk models concerning discriminatory ability. Despite a large number of models, as treatment approaches evolve over time with improving outcomes and as ever older patients with complex disease patterns are treated invasively, new or updated risk prediction algorithms are needed to maintain/increase prognostic accuracy.

Acknowledgments

Funding: None.

Footnote

Provenance and Peer Review: This article was commissioned by the Guest Editor (Ion S. Jovin) for the series “Interventional Cardiology” published in Journal of Thoracic Disease. The article was sent for external peer review organized by the Guest Editor and the editorial office.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/jtd.2019.12.83). The series “Interventional Cardiology” was commissioned by the editorial office without any funding or sponsorship. BM reports personal fees from Biotronik, Medtronic, Abbott, Astra Zeneca, Sanofi Aventis, Servier as well as grants from Boston Scientific, Medtronic, and Abbott, outside the submitted work. The other authors have no other of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. O’Gara PT, Kushner FG, Ascheim DD, et al. 2013 ACCF/AHA Guideline for the Management of ST-Elevation Myocardial Infarction. J Am Coll Cardiol 2013;61:e78-140. [Crossref] [PubMed]
2. Ibanez B, James S, Agewall S, et al. 2017 ESC Guidelines for the management of acute myocardial infarction in patients presenting with ST-segment elevation: The Task Force for the management of acute myocardial infarction in patients presenting with ST-segment elevation of the European Society of Cardiology (ESC). Eur Heart J 2018;39:119-77. [Crossref] [PubMed]
3. Neumann F-J, Sousa-Uva M, Ahlsson A, et al. 2018 ESC/EACTS Guidelines on myocardial revascularization. Eur Heart J 2019;40:87-165. [Crossref] [PubMed]
4. Sianos G, Morel M-A, Kappetein AP, et al. The SYNTAX Score: an angiographic tool grading the complexity of coronary artery disease. EuroIntervention 2005;1:219-27. [PubMed]
5. Ong ATL, Serruys PW, Mohr FW, et al. The SYNergy between percutaneous coronary intervention with TAXus and cardiac surgery (SYNTAX) study: Design, rationale, and run-in phase. Am Heart J 2006;151:1194-204. [Crossref] [PubMed]
6. Morrow DA, Antman EM, Charlesworth A, et al. TIMI risk score for ST-elevation myocardial infarction: A convenient, bedside, clinical score for risk assessment at presentation: An intravenous nPA for treatment of infarcting myocardium early II trial substudy. Circulation 2000;102:2031-7. [Crossref] [PubMed]
7. Addala S, Grines CL, Dixon SR, et al. Predicting mortality in patients with ST-elevation myocardial infarction treated with primary percutaneous coronary intervention (PAMI risk score). Am J Cardiol 2004;93:629-32. [Crossref] [PubMed]
8. De Luca G, Suryapranata H, van ’t Hof AW, et al. Prognostic assessment of patients with acute myocardial infarction treated with primary angioplasty: implications for early discharge. Circulation 2004;109:2737-43. [Crossref] [PubMed]
9. Halkin A, Singh M, Nikolsky E, et al. Prediction of mortality after primary percutaneous coronary intervention for acute myocardial infarction: The CADILLAC risk score. J Am Coll Cardiol 2005;45:1397-405. [Crossref] [PubMed]
10. Stebbins A, Mehta RH, Armstrong PW, et al. A Model for Predicting Mortality in Acute ST-Segment Elevation Myocardial Infarction Treated With Primary Percutaneous Coronary Intervention. Circ Cardiovasc Interv 2010;3:414-22. [Crossref] [PubMed]
11. de Mulder M, Gitt A, van Domburg R, et al. EuroHeart score for the evaluation of in-hospital mortality in patients undergoing percutaneous coronary intervention. Eur Heart J 2011;32:1398-408. [Crossref] [PubMed]
12. Amin ST, Morrow DA, Braunwald E, et al. Dynamic TIMI risk score for STEMI. J Am Heart Assoc 2013;2:e003269. [Crossref] [PubMed]
13. Hizoh I, Gulyas Z, Domokos D, et al. A novel risk model including vascular access site for predicting 30-day mortality after primary PCI: The ALPHA score. Cardiovasc Revasc Med 2017;18:33-9. [Crossref] [PubMed]
14. Chin CT, Chen AY, Wang TY, et al. Risk adjustment for in-hospital mortality of contemporary patients with acute myocardial infarction: the acute coronary treatment and intervention outcomes network (ACTION) registry-get with the guidelines (GWTG) acute myocardial infarction mortality mode. Am Heart J 2011;161:113-22.e2. [Crossref] [PubMed]
15. Fox KAA, Fitzgerald G, Puymirat E, et al. Should patients with acute coronary disease be stratified for management according to their risk? Derivation, external validation and outcomes using the updated GRACE risk score. BMJ Open 2014;4:e004425. [Crossref] [PubMed]
16. Peterson ED, Dai D, DeLong ER, et al. Contemporary mortality risk prediction for percutaneous coronary intervention: results from 588,398 procedures in the National Cardiovascular Data Registry. J Am Coll Cardiol 2010;55:1923-32. [Crossref] [PubMed]
17. Raposeiras-Roubín S, Abu-Assi E, Cabanas-Grandío P, et al. Walking Beyond the GRACE (Global Registry of Acute Coronary Events) Model in the Death Risk Stratification During Hospitalization in Patients With Acute Coronary Syndrome. JACC Cardiovasc Interv 2012;5:1117-25. [Crossref] [PubMed]
18. Abelin AP, David RB, Gottschall CA, et al. Accuracy of Dedicated Risk Scores in Patients Undergoing Primary Percutaneous Coronary Intervention in Daily Clinical Practice. Can J Cardiol 2014;30:125-31. [Crossref] [PubMed]
19. Littnerova S, Kala P, Jarkovsky J, et al. GRACE Score among Six Risk Scoring Systems (CADILLAC, PAMI, TIMI, Dynamic TIMI, Zwolle) Demonstrated the Best Predictive Value for Prediction of Long-Term Mortality in Patients with ST-Elevation Myocardial Infarction. PLoS One 2015;10:e0123215. [Crossref] [PubMed]
20. Parenica J, Kala P, Pavkova MG, et al. Natriuretic peptides, nitrite/nitrate and superoxide dismutase have additional value on top of the GRACE score in prediction of one-year mortality and rehospitalisation for heart failure in STEMI patients - Multiple biomarkers prospective cohort study. Int J Cardiol 2016;211:96-104. [Crossref] [PubMed]
21. Huang W, FitzGerald G, Goldberg RJ, et al. Performance of the GRACE Risk Score 2.0 Simplified Algorithm for Predicting 1-Year Death After Hospitalization for an Acute Coronary Syndrome in a Contemporary Multiracial Cohort. Am J Cardiol 2016;118:1105-10. [Crossref] [PubMed]
22. Timóteo AT, Monteiro AV, Portugal G, et al. Validation of two US risk scores for percutaneous coronary intervention in a single-center Portuguese population of patients with acute coronary syndrome. Rev Port Cardiol 2016;35:73-8. [Crossref] [PubMed]
23. Yu T, Tian C, Song J, et al. ACTION (acute coronary treatment and intervention outcomes network) registry-GWTG (get with the guidelines) risk score predicts long-term mortality in acute myocardial infarction. Oncotarget 2017;8:102559-72. [Crossref] [PubMed]
24. Hizoh I, Banhegyi G, Domokos D, et al. Comparative Validation of the ALPHA Score, a Novel Risk Model Including Vascular Access Site for Predicting 30-Day Mortality in Patients Treated With Primary PCI. J Am Coll Cardiol 2018;72:B320-1. [Crossref]
25. Lev EI, Kornowski R, Vaknin-Assa H, et al. Comparison of the Predictive Value of Four Different Risk Scores for Outcomes of Patients With ST-Elevation Acute Myocardial Infarction Undergoing Primary Percutaneous Coronary Intervention. Am J Cardiol 2008;102:6-11. [Crossref] [PubMed]
26. Elbarouni B, Goodman SG, Yan RT, et al. Validation of the Global Registry of Acute Coronary Event (GRACE) risk score for in-hospital mortality in patients with acute coronary syndrome in Canada. Am Heart J 2009;158:392-9. [Crossref] [PubMed]
27. Abu-Assi E, Ferreira-González I, Ribera A, et al. Do GRACE (Global Registry of Acute Coronary events) risk scores still maintain their performance for predicting mortality in the era of contemporary management of acute coronary syndromes? Am Heart J 2010;160:826-834.e1-3.
28. Yusufali A, Zubaid M, Al-Zakwani I, et al. Validation of the GRACE risk score for hospital mortality in patients with acute coronary syndrome in the Arab Middle East. Angiology 2011;62:390-6. [Crossref] [PubMed]
29. Méndez-Eirín E, Flores-Ríos X, García-López F, et al. Comparison of the prognostic predictive value of the TIMI, PAMI, CADILLAC, and GRACE risk scores in STEACS undergoing primary or rescue PCI. Rev Esp Cardiol 2012;65:227-33. [Crossref] [PubMed]
30. Timóteo AT, Papoila AL, Lopes JP, et al. Is it possible to simplify risk stratification scores for patients with ST-segment elevation myocardial infarction undergoing primary angioplasty? Rev Port Cardiol 2013;32:967-73. [Crossref] [PubMed]
31. Fujii T, Suzuki T, Torii S, et al. Diagnostic Accuracy of Global Registry of Acute Coronary Events (GRACE) Risk Score in ST-Elevation Myocardial Infarction for In-Hospital and 360-Day Mortality in Japanese Patients. Circ J 2014;78:2950-4. [Crossref] [PubMed]
32. Morrow DA, Antman EM, Parsons L, et al. Application of the TIMI risk score for ST-elevation MI in the National Registry of Myocardial Infarction 3. JAMA 2001;286:1356-9. [Crossref] [PubMed]
33. Selvarajah S, Fong AYY, Selvaraj G, et al. An Asian validation of the TIMI risk score for ST-segment elevation myocardial infarction. PLoS One 2012;7:e40249. [Crossref] [PubMed]
34. Mehta SR, Jolly SS, Cairns J, et al. Effects of radial versus femoral artery access in patients with acute coronary syndromes with or without ST-segment elevation. J Am Coll Cardiol 2012;60:2490-9. [Crossref] [PubMed]
35. Romagnoli E, Biondi-Zoccai G, Sciahbasi A, et al. Radial versus femoral randomized investigation in ST-segment elevation acute coronary syndrome. J Am Coll Cardiol 2012;60:2481-9. [Crossref] [PubMed]
36. Karrowni W, Vyas A, Giacomino B, et al. Radial Versus Femoral Access for Primary Percutaneous Interventions in ST-Segment Elevation Myocardial Infarction Patients. JACC Cardiovasc Interv 2013;6:814-23. [Crossref] [PubMed]
37. Valgimigli M, Gagnor A, Calabró P, et al. Radial versus femoral access in patients with acute coronary syndromes undergoing invasive management: a randomised multicentre trial. Lancet 2015;385:2465-76. [Crossref] [PubMed]
38. Grines CL, Cox DA, Stone GW, et al. Coronary angioplasty with or without stent implantation for acute myocardial infarction. Stent Primary Angioplasty in Myocardial Infarction Study Group. N Engl J Med 1999;341:1949-56. [Crossref] [PubMed]
39. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010;21:128-38. [Crossref] [PubMed]
40. Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: Seven steps for development and an ABCD for validation. Eur Heart J 2014;35:1925-31. [Crossref] [PubMed]
41. Davis J, Goadrich M. The relationship between Precision-Recall and ROC curves. In: Proceedings of the 23rd international conference on Machine learning - ICML ’06.Vol 73. New York, New York, USA: ACM Press, 2006:233-40.