Machine learning-based model development for predicting risk factors of prolonged intra-aortic balloon pump therapy in patients with coronary artery bypass grafting

Abstract

Machine learning algorithms are frequently used to clinical risk prediction. Our study was designed to predict risk factors of prolonged intra-aortic balloon pump (IABP) use in patients with coronary artery bypass grafting (CABG) through developing machine learning-based models. Patients who received perioperative IABP therapy were divided into two groups based on their length of IABP implantation longer than the 75th percentile for the whole cohort: normal (≤ 10 days) and prolonged (> 10 days) groups. Seven machine learning-based models were created and evaluated, and then the Shapley Additive exPlanations (SHAP) method was employed to further illustrate the influence of the features on model. In our study, a total of 143 patients were included, comprising 56 cases (38.16%) in the prolonged group. The logistic regression model was considered the final prediction model according to its most excellent performance. Furthermore, feature important analysis identified left ventricular end-systolic or diastolic diameter, preoperative IABP use, diabetes, and cardiac troponin T as the top five risk variables for prolonged IABP implantation in patients. The SHAP analysis further explained the features attributed to the model. Machine learning models were successfully developed and used to predict risk variables of prolonged IABP implantation in patients with CABG. This may help early identification for prolonged IABP use and initiate clinical interventions.

Introduction

Short-to-midterm mortality in acute coronary syndrome (ACS) patients complicating cardiogenic shock remains high at rates between 40% and 60% [1,2,3,4]. The intra-aortic balloon pumping (IABP), as one of hemodynamic support devices, is implanted into the aorta to temporarily support cardiac output in cardiogenic shock patients [5]. The registry and experimental trials have suggested that it can elevate diastolic blood pressure through promoting forward flow from a high-capacitance reservoir to a low-capacitance vessels, thereby improving coronary and peripheral perfusion and preserving the cardiac function [4, 5]. Despite the supporting evidence for the benefits of IABP, recent IABP-SHOCK II trial (IABP in Cardiogenic Shock II) do not exhibit a beneficial effect of IABP use on 30-day and one-year mortality, which may be associated with IABP-caused complications [6, 7]. Indeed, numerous previous studies have extensively explored the complications related to IABP implantation, such as hemorrhage, limb ischemia, embolization, and thrombocytopenia [8,9,10]. Additionally, renal function damage is a frequent complication observed in patients undergoing IABP treatment [11]. These complications have been reported to exhibit a positive correlation with the duration of IABP use [10, 12]. However, increased IABP use may impact in a certain degree the length of hospital stay (LOS), the duration of intensive care unit (ICU) stay, hospital costs, and in-hospital death. Accordingly, exploring risk factors for prolonged use of IABP may be of great significance for patients.

Coronary artery bypass grafting (CABG) is the most common heart operation which is performed for treating ACS patients in cardiac surgery centers. During the perioperative period of CABG, these patients complicating cardiogenic shock are frequently required for using IABP support therapy. Some previous studies primarily focus on investigating the effects of IABP implantation timing on clinical outcomes [13, 14]. However, there still are few studies for exploring risk factors of IABP itself use in patients, specifically in CABG individuals. As a result, a highly effective prediction tool for prolonged IABP use in patients is expected to be developed. Machine learning is a novel artificial intelligence-based modeling tool and has been recognized as excellent tool for biomedical research, customized treatment, and computer-aided diagnosis [15]. It is gradually being applied in clinical research and practice to achieve various tasks, such as risk stratification, diagnostic classification, and survival prediction [16]. The aim of this study was to create and evaluate supervised machine learning models to perform risk prediction of prolonged IABP implantation in patients with CABG.

Methods

Patient population

This retrospective study was approved by the Ethics Review Committee of The First Affiliated Hospital of Nanjing Medical University (ethics number: 2019-SR-313.A1). Considering the nature of the retrospective study, patients’ informed consent was waived by the Ethics Review Committee of the hospital. Between January 2015 to December 2019, all adult patients who underwent an isolated CABG surgery and received perioperative IABP support therapy were enrolled into this study in a way of the chronological mode. Standard median sternotomy, cardiopulmonary bypass (CPB), and aortic cross-clamp were applied in all patients. The timing of IABP implantation in patients was evaluated by the entire medical team. The indications for IABP insertion in all patients were: (1) blood pressure decreasing progressively under the therapy of two vasoactive drugs, (2) mean arterial pressure < 50 mmHg, (3) cardiac index (CI) < 2.2 L/(min·m²), (4) mean arterial pressure < 50 mmHg, and (5) urine volume < 0.5 mL/(kg·h). In this study, all patients who met the inclusion criteria were enrolled during the study period, and a total of 143 patients were collected.

Data collection

Patients were characterized by 46 rapid available preoperative variables (including demographics, comorbidities, coronary artery lesion and angiography, echocardiography, electrocardiogram (ECG), and laboratory indicators). Moreover, information on continuous renal replacement therapy (CRRT) and tracheotomy in patients during the IABP insertion was collected. LOS, length of ICU, in-hospital mortality, and hospital costs were recorded. All data were input and audited by experienced physicians using the electronic medical record (EMR) system of the hospital. Some previous publications defined prolonged duration of IABP to be between 2 and 14 days; however, this definition has not been adopted by all investigators. In this study, we stratified the patients into the following three groups based on the length of IABP therapy (LOIT): the lowest quartile (25th quartile), the median (25th–75th quartiles), and the highest quartile (75th quartile). In this study, we defined the 75th quartile of LOIT (10 days) as the demarcation line between normal and prolonged periods: normal LOIT (Nor-LOIT, ≤ 10 days) and prolonged LOIT (Pro-LOIT, > 10 days) groups.

Feature selection for modeling

The Boruta algorithm is used to evaluate the importance of each variable in a circular manner, comparing the importance of original variables and its shadow variables in each iteration round. If the importance of original variable is significantly higher than that of the shadow variable, it is considered important. Conversely, if original variable is considerably less important than its shadow counterpart, it is deemed unimportant. In this study, the Boruta algorithm was performed to select the most crucial features associated with clinical outcome from the collected variables. Subsequently, these variables were employed to the construction and development of the models, which could effectively avoid overfitting and optimize hyperparameters.

Machine learning models development and validation

The original dataset collected in this study was randomly separated into a training (90%) dataset and an internal validation (10%) dataset. The training dataset was employed to train the models, while the validation dataset was used for the evaluation and selection of the models. During this process, 10-fold cross-validation was performed. In this study, seven machine learning models were developed to predict the risk factors of prolonged LOIT in patients with CABG surgery, including logistic regression, LightGBM, Gaussian Naive Bayes (GNB), multi-layer perceptron neural network (MLP), k-nearest neighbors (KNN), support vector machine (SVM), and Complement Naive Bayes (CNB).

Then, performance of machine learning models was measured using the area under the receiver operating characteristic curve (AUC) with associated 95% confidence interval (CI). The accuracy (ACC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1-score were calculated for further evaluation of the model performance. The calibration plot was visualized to assess the model’s calibration by calculating the Brier score, where a smaller Brier value indicates higher accuracy of the model. This implies that the discrepancy between predicted outcomes and actual clinical practice outcomes is minimized. The final predictive model in this study was chosen based on its superior performance. Figure 1shows the flowchart of this study.

Fig. 1

Overall flowchart of the study. IABP, intra-aortic balloon pump; CABG, coronary artery bypass grafting; GNB, Gaussian Naive Bayes; MLP, multi-layer perceptron neural network; KNN, k-nearest neighbors; SVM, support vector machine; CNB, Complement Naive Bayes; AUC, area under the receiver operating characteristic curve; SHAP, Shapley Additive exPlanations

Shapley Additive exPlanations (SHAP) application

The SHAP method was applied to enhance the interpretability of the final predictive model, and the SHAP summary plot was used to illustrate the influence of model features. The SHAP dependence plot was used to analyze the importance of individual features affecting model output. The SHAP force plot was utilized to visually represent the impact of the features on the final model in individual patients.

Statistical analysis

Continuous variables were presented as mean ± standard deviations (SD) or median (interquartile spacing). Continuous variables with a normalized distribution among two groups were analyzed using Students’ t-test. Continuous variables with non-normalized distribution were analyzed using the Mann-Whitney U test. Categorical variables were presented as numerical values (proportions) and analyzed using the chi-square test. All statistical analysis in this study was conducted using IBM SPSS statistics software (version 25.0, IBM Corp., Armonk, New York, USA). The values of p < 0.05 represented a significant statistical difference.

Results

Baseline characteristics of patients

A total of 143 patients who underwent CABG and received perioperative IABP support therapy were enrolled in this study, comprising 116 males (81.12%), with a median age of 66 years and 19 mortalities (13.29%). Among these patients, 56 cases (39.16%) were included into the Pro-LOIT group, with 46 males (82.14%) and eight mortalities (14.29%). The significant differences were observed between the two groups regarding diabetes, total cholesterol (TC), high-density lipoprotein cholesterol (HDL-C), fast blood glucose, cardiac troponin T (cTnT), New York Heart Association (NYHA) classification, left ventricular end-systolic dimension (LVDs), left ventricular ejection fraction (LVEF), left ventricular fraction shortening (LVFS), tracheotomy, preoperative IABP insertion, LOS, length of ICU, and hospital costs. The detail content is shown in Table 1.

Table 1 Baseline characteristics of patients in the two groups

Feature selection based on the Boruta algorithm

The Boruta algorithm was employed to identify the most crucial variables associated with Pro-LOIT in patients. Ultimately, 7 variables were identified and used to develop machine learning models. This selection process was effective in avoiding overfitting and optimizing hyperparameters. However, the selection based on the Boruta algorithm did not imply that importance of the variables was analyzed in this study. These selected variables included tracheotomy, preoperative IABP, LVEF, cTnT, NYHA classification, LVFS, and diabetes. The corresponding results are shown in Fig. 2.

Fig. 2

The Boruta algorithm for feature selection. When importance of feature exceeds the value of ShadowMax, these features are selected for developing models

Models’ performance in predicting prolonged LOIT risk

The predictive performance of machine learning models is illustrated in Figs. 3–4; Tables 2 and 3. Our findings showed that the models had various abilities to predict the risk factors associated with prolonged LOIT in patients. Compared to other models, the logistic regression model exhibited an excellent predictive performance due to its highest AUC (0.799, 95%CI: 0.711–0.887, Fig. 3) in the training set and (0.774, 95%CI: 0.630–0.919, Fig. 4) in the validation set. Furthermore, the highest ACC, sensitivity, and F1-score in the two datasets of the logistic regression model were found (Tables 2 and 3). The calibration curve plotting was created and is shown in Fig. 5. The predictive probability of the logistic regression model was the closest to clinical practice outcome. Based on these findings, we eventually considered the logistic regression model as the predictive model for prolonged LOIT.

Fig. 3

The receiver operating characteristic curves (ROCs) of machine learning models in the training set. GNB, Gaussian Naive Bayes; MLP, multi-layer perceptron neural network; KNN, k-nearest neighbors; SVM, support vector machine; CNB, Complement Naive Bayes; AUC, area under the receiver operating characteristic curve

Fig. 4

The receiver operating characteristic curves (ROCs) of machine learning models in the validation set. GNB, Gaussian Naive Bayes; MLP, multi-layer perceptron neural network; KNN, k-nearest neighbors; SVM, support vector machine; CNB, Complement Naive Bayes; AUC, area under the receiver operating characteristic curve

Fig. 5

The calibration plots (reliability curves) of machine learning models. GNB, Gaussian Naive Bayes; MLP, multi-layer perceptron neural network; KNN, k-nearest neighbors; SVM, support vector machine; CNB, Complement Naive Bayes

Table 2 The parameters of models in the training set

Table 3 The parameters of models in the validation set

The logistic regression model explanation and application

The feature importance for the logistic regression model was analyzed using the SHAP value. This showed the greatest discriminatory capacity in the validation cohort. According to the obtained SHAP value, Fig. 6A exhibits the weight of 7 clinical variables and Fig. 6B provides an overview of the impact (positive or negative aspects) of factors on the logistic regression model. Subsequently, the correlation between variables and the risk factors associated with prolonged LOIT is displayed in Fig. 7A-F, with the positive and negative association. Subsequently, we randomly selected one patient from the validation cohort to exhibit a visual interpretation for an individual patient (Fig. 7G). The logistic regression model predicted the probability of prolonged LOIT in patient to be 55.30%. The result indicated that serum cTnT of 7579.0 pg/L, preoperative IABP use, and LVFS of 29.6% were the top three contributors to this prediction.

Fig. 6

The logistic regression model’s interpretation based on the SHAP. (A) The importance ranking of the top 7 variables according to the mean (|SHAP value|). (B) The importance ranking of the top 7 risk factors. The higher SHAP value of a feature is shown, the higher risk of prolonged LOIT the patient would have. The red part in feature value represents higher value

Fig. 7

The SHAP dependency plot for the top 6 clinical features contributing to the logistic regression model and interpretation of model prediction results with the one sample. (A-F) LVEF, LVFS, preoperative IABP, diabetes, cTnT, and NYHA. (G) Model predictions by randomly drawing a single sample from the validation cohort

Discussion

In this study, we developed seven machine learning models using selected clinical crucial features to identify the risk factors associated with prolonged LOIT in patients with CABG surgery. Compared to other models, the logistic regression model had the most well predictive performance, which could be confirmed by its highest AUC, ACC, sensitivity, and F1-score, as well as an excellent calibration. Additionally, the SHAP analysis was constructed to exhibit the importance of the variables and how particular compound substructures influence prolonged LOIT in CABG patients in the created logistic regression model. Our result revealed that LVEF, LVFS, preoperative IABP, diabetes, and cTnT were the top five most important variables contributing to the logistic regression model. Finally, the SHAP personal analysis was used to facilitate the individualized predictions.

With the rise of artificial intelligence, machine learning algorithms are expected to become a crucial tool to optimize risk prediction and clinical assessment system [16]. Machine learning models based on artificial intelligence have been successfully developed in the field of perioperative medicine for risk stratification, prediction of intraoperative events, and intensive care medicine [17]. The models could help clinicians improve clinical outcomes by accurately predicting complications and suggesting optimal treatment strategies in real-time [17]. The development of several machine learning-based models has enabled the prediction of perioperative outcomes in patients undergoing cardiac surgeries [18,19,20,21]. However, there is a lack of studies on the prediction of risk factors associated with prolonged LOIT in patients undergoing CABG surgery. Furthermore, the development of personalized systems is imperative for accurately predicting outcomes among specific operator groups, which highlights the importance of machine learning models [22, 23]. The aim of personalized medicine has been to make models match the individual across multiple scales to solve clinical issues. During the development of models, it is imperative to emphasize the importance of conducting selection of characteristic variables prior to model development, which is beneficial for identifying optimal parameters and avoiding the model overfitting. Then, the SHAP analysis was used to further demonstrate the weights assigned by the model to relevant factors. The individual explanations generated by SHAP analysis help doctors’ comprehension of why the model provided specific recommendations for high-risk decisions.

To further confirm how factors contribute to the model, we calculated SHAP feature importance and feature effects. The LVEF and LVFS of patient are crucial echocardiographic indicators that reflect the systolic function of left ventricle. A decrease in the parameters indicates impaired left ventricular systolic function. A study revealed a close relationship between the impact of IABP on mortality and the severity of cardiogenic shock, suggesting that cardiac function may affect clinical consequences in patients [24]. Another clinical study investigated the association between preoperative cardiac function, including left systolic function, and perioperative IABP use in patients undergoing elective off-pump coronary artery bypass surgery [25]. Compared to those with normal cardiac function, patients with reduced left ventricular systolic function received IABP support therapy more frequently during the perioperative period of cardiac surgery [25]. Patients with left ventricular dysfunction are more prone to hemodynamic disturbances and related complications during the perioperative period, and therefore may require ventricular assist support or extended support duration. In our study, the LVEF and LVFS of patients in the Pro-LOIT group were lower than those in the Nor-LOIT group, indicating a reduced cardiac function in the prolonged IABP group. The SHAP analysis further identified LVEF and LVFS as the most crucial variables affecting the outcome of prolonged IABP utilization within the logistic regression model in our study. These findings underscore the importance of monitoring left ventricular systolic parameters in patients. Due to the superior sensitivity and specificity of serum cTnT levels as a diagnostic marker, it has been become the current standard laboratory determinant for myocardial injury [26]. Previous studies have shown serum cTnT as a prognostic variable in cardiac or noncardiac surgery, with an elevated serum cTnT levels being related to postoperative complications and mortality following surgeries [27,28,29]. Our model identified serum cTnT levels as crucial variable of prolonged IABP implantation in patients. This suggests that there is a positive relationship between cardiac injury and IABP support therapy.

Previous studies have extensively explored the influence of the timing of IABP implantation on postoperative clinical outcomes, including mortality, LOS, ICU, and complications [13, 30,31,32]. Our study revealed that preoperative IABP implantation was an important risk factor affecting the LOIT of patients. Preoperative IABP implantation frequently implies that when ACS patients with cardiogenic shock are unable to immediately undergo CABG surgery, and IABP is only considered a temporary support device to improve clinical symptoms of patient. Previous study indicates that patients receiving preoperative IABP use have a higher risk of cardiac dysfunction, intraoperative complications, and postoperative ICU stay than those without preoperative IABP use [30]. On this basis, we recognize that preoperative IABP implantation partly reflects a patient’s worse functional status and resistance to surgical and clinical support measures. However, there is study on report that preoperative IABP use reduces postoperative mortality in high-risk populations of patients undergoing CABG surgery [33], which in part underscoring the clinical benefit of preoperative IABP implantation. Our study found no significant difference in in-hospital mortality between patients in the Pro-LOIT and Nor-LOIT group, which may be explained by the lack of population stratification based on the risk.

Diabetes mellitus (DM) is a systemic disorder of glucose metabolism, characterized by insulin resistance, hyperglycemia, and hyperinsulinism, along with dyslipidemia. DM has been regarded as a risk factor for a variety of cardiovascular and cerebrovascular diseases, such as coronary heart disease, myocardial infarction, heart failure, ischemic and hemorrhagic cerebral infarction [34, 35]. Previous studies have investigated the associated risk factors of IABP insertion during CABG surgery and found that DM is an associated factor for intraoperative IABP implantation [36, 37]. Clinical studies have found that ACS patients with DM have higher risks of heart failure and short- and long-term mortality than ACS patients without DM [38]. We identified a promoting effect of diabetes on prolonged IABP implantation, which may be attributed to global cardiac function and the patient’s own resistance to disease. Additionally, despite the logistic regression model not considering blood glucose levels as a risk factor, patients in the Pro-LOIT group had higher fasting blood glucose levels than those in the Nor-LOIT group, consistent with the finding that diabetes was more common in the Pro-LOIT group. However, it should be noted that fasting plasma glucose levels were within the normal range in both groups, which may have been due to the use of glucose-lowering medications or insulin.

Created machine learning models in this study overcome complex relationship between various variables and display good performance in predicting the risk factors associated with prolonged LOIT. Importantly, the logistic regression model exhibited most excellent predictive ability. Furthermore, our findings showed that LOS, length of ICU, and hospitalization costs were significantly higher in the Pro-LOIT group than in the Nor-LOIT group. To extent degree, this suggests that early intervention based on machine learning model-identified risk factors for prolonged LOIT may help to improve these clinical outcomes in patients. However, it must be mentioned that more samples and data are required for supporting whether this clinical application is effective. Likewise, we ought to consider the existed limitations in this study. Firstly, the nature of our study was retrospective, which may have biased the results to some extent. Secondly, there is a lack of the external validation, which might affect the generalizability of our findings and models. Therefore, in future, the external data need to be used. Thirdly, it is unclear whether the constructed risk prediction model can be translated into actual clinical benefits for patients, so prospective, multicenter studies are needed to evaluate.

Conclusion

Created machine learning models in this study were used for personalized prediction of prolonged LOIT in patients with CABG. Our results have revealed that the logistic regression model exhibits a good predictive performance and identifies the risk factors associated with prolonged LOIT. This may contribute to improving perioperative.