Development of a predictive model for major adverse cardiac events in a coronary artery bypass and valve population

Abstract

Background

Quality improvement initiatives in cardiac surgery largely rely on risk prediction models. Most often, these models include isolated populations and describe isolated end-points. However, with the changing clinical profile of the cardiac surgical patients, mixed populations models are required to accurately represent the majority of the surgical population. Also, composite model end-points of morbidity and mortality, better reflect outcomes experienced by patients.

Methods

The model development cohort included 4,270 patients who underwent aortic or mitral valve replacement, or mitral valve repair with/without coronary artery bypass grafting, or isolated coronary artery bypass grafting. A composite end-point of infection, stroke, acute renal failure, or death was evaluated. Age, sex, surgical priority, and procedure were forced, a priori, into the model and then stepwise selection of candidate variables was utilized. Model performance was evaluated by concordance statistic, Hosmer-Lemeshow Goodness of Fit, and calibration plots. Bootstrap technique was employed to validate the model.

Results

The model included 16 variables. Several variables were significant such as, emergent surgical priority (OR 4.3; 95% CI 2.9-7.4), CABG + Valve procedure (OR 2.3; 95% CI 1.8-3.0), and frailty (OR 1.7; 95% CI 1.2-2.5), among others. The concordance statistic for the major adverse cardiac events model in a mixed population was 0.764 (95% CL; 0.75-0.79) and had excellent calibration.

Conclusions

Development of predictive models with composite end-points and mixed procedure population can yield robust statistical and clinical validity. As they more accurately reflect current cardiac surgical profile, models such as this, are an essential tool in quality improvement efforts.

Background

Quality improvement initiatives (QI), a cornerstone of cardiac surgery, have largely relied on predictive models to advance the quality of care cardiac surgical patients receive. For the last two decades, coronary artery bypass grafting (CABG) has dominated clinical practice in cardiac surgery, and therefore the majority of quality improvement initiatives have focused on surgical outcomes following isolated CABG surgery [1–4]. More recently, however, there has been an increase in valve and valve-CABG cases in cardiac surgery [5] and existing predictive models for isolated CABG may not accurately reflect current practice profiles. These models may achieve great statistical validity but lack clinical validity as the models are applicable to only a decreasing percentage of the clinical practice. In order to maintain continued success in quality improvement, it is important to delineate risk profiles for a group of mixed procedures, including CABG, valve, and valve + CABG, that characterize current clinical practice.

To accommodate the shift in the profile of cardiac surgical practice, isolated valve and valve plus CABG models have been developed [5–10]. Some debate exists regarding the validity of developing models with hetero-geneous procedures with some advocating for single procedure models [7, 9, 10]. Homogeneity within the population allows for simplicity of model development with improved reliability but limits sample size available for development and validation [5]. Although mixed procedural models may be confounded or biased by different pathophysiological and risk profiles, they improve sample size [5] and increase their relevance to current surgical practice.

Largely, mixed procedural models have focused on aortic/mitral or valve +/− CABG, with few CABG, valve and CABG +valve models present in the literature. The EuroSCORE model, one of the mostly widely used cardiac surgical prognostic models, has successfully achieved both statistical and clinical relevance as it applies to a mixed surgical practice including CABG, valve, and CABG + valve. However its development and validation cohort were derived from a population sample that is over fifteen years old [11] limiting its applicability to a contemporary surgical practice. Recently, the EuroSCORE II updated its risk model obviating some of the limitations of the original model [12]. However, the performance of this heterogeneous procedure model performed poorly in risk prediction for several of its component procedures types [13, 14]. In addition, both the EuroSCORE and the EuroSCORE II are only a mortality model.

Many models describe death as a lone end point, despite that fact that cardiac patients experience a variety of relevant morbidity. These alternate surgical outcomes are also important quality indicators for cardiac surgical care [15] and targets for QI initiatives. Subsequently, modeling composite end points such as, major adverse cardiac events (MACE), that included both important morbidity and mortality may provide more insight to the surgical outcomes experienced by patients as well as increase statistical power in low frequency end points.

Comprehensive models that include most major cardiac surgical procedures may allow for better understanding of patient risk profiles and facilitate quality improvement initiatives directed towards the majority of patients seen in current practice. The objective of this paper is to develop a CABG, valve, CABG + valve morbidity and mortality (MACE) model that can be applied to the majority of patients undergoing cardiac surgery.

Methods

Data source and study population

This study is a retrospective cohort design. The Maritime Heart Center Cardiac Surgery Registry is a detailed clinical database housed at the Queen Elizabeth II Health Science Center (QEII HSC) Halifax, Nova Scotia, an academic tertiary care centre performing 1200 cardiac cases per year. It includes pre-, intra-, and post-operative data prospectively collected on all cardiac surgical cases performed at the QEII HSC from 1995 to present. Trained abstractors collect data, and a database administrator maintains the registry. The database is audited annually.

The model development cohort included all patients undergoing isolated coronary artery bypass grafting (CABG), isolated aortic valve replacement, isolated mitral valve repair or replacement with or without concomitant CABG performed at the QEII HSC since 2004 to 2009. The development cohort was restricted to these years in order to maintain a modern population relevant to current clinical practice.

The primary outcome was a composite end point defined as Major Adverse Cardiac Events (MACE) that included in-hospital death, stroke (persisting at discharge or transient), acute renal failure (new post-operative renal failure or acute on chronic (>50% increase from baseline creatinine)), or infection (sepsis, pneumonia, or deep sternal wound infection).

Statistical methods

Variable selection

Candidate variables for model development included the following preoperative characteristics: age, sex, diabetes, frailty, chronic obstructive pulmonary disease, redo sternotomy, atrial fibrillation, hemoglobin, peripheral vascular disease, cerebro-vascular disease, creatinine, ejection fraction < 40%, New York Heart Association classification (NYHA I-IV) and body mass index (BMI). Surgery-related data such as urgency of surgery, and procedure type were also included. These candidate variables were chosen a priori and selected through rigorous review of the literature [1–9, 16–18] (Table 1).

Table 1 Model variable definitions

Multi-colinearity of candidate variables was assessed with variance inflation factor (VIF) (Additional file 1: see Statistical Methods). The linear relationships of the natural variables and their transformations were assessed through locally weighted scatterplot smoothing (LOESS) regression [19] (see Additional file 1: Linearity and Transformations and Figure 2-3). The WHO classification of BMI was used [20].

Model evaluation

A multivariate logistic regression analysis was used to describe MACE.

Age, sex, procedure type and surgical priority were chosen a priori and forced into the model. Stepwise selection was implemented for the remainder of the candidate variables. The concordance statistic and −2 log likelihood were evaluated to assess the contribution of each variable to the model. If a variable did not contribute to an increase in the C or −2 log likelihood statistic it was not retained in the model.

Model discrimination was determined using the concordance statistic [21]. Model calibration was assessed by the Hosmer-Lemeshow goodness-of-fit statistic [22, 23] as well as calibration plots [24]. Deciles of observed and predicted probabilities of MACE were plotted for the calibration plots [5]. Bootstrap procedure was used to internally validate the model.

All statistical analysis was performed using SAS software version 9.2 (SAS, Cary, NC).

Approval for conducting this study was obtained from the Institutional Review Board of the Capital District Health Authority. The requirement to obtain informed consent was waived under Section 2.1c of the Tri-Council Policy Statement. All personal identifiers were stripped prior to data analysis to ensure patient anonymity and confidentiality.

The authors had full access to the data and take full responsibility for its integrity. All authors have read and agree to the manuscript as written.

Results

Population

A total of 4,270 patients underwent CABG, valve (aortic valve replacement, mitral valve replacement, or mitral valve repair) or CABG + valve at the QEII HSC from Jan 2004 to Dec 2009. The model cohort was 65% of the total case volume (6,525) performed during the study period. The distributions of risk factors in the development cohort are displayed in Table 2. The prevalence of MACE in this cohort was 15.7% (n=669). The prevalence of MACE was higher in the CABG plus valve group (32%, n=155) than in the isolated procedures (CABG 13%, n=416; Valve 14%, n=98). The frequencies of MACE for each procedure as well as the components of MACE are summarized in Table 3.

Table 2 Distribution of risk factors in the model development cohort

Table 3 Frequency of MACE and MACE components in the model development cohort

Model development

Assessment of the variance inflation for each variable revealed that no variable exceeded 4.0 allowing all variables to remain in the final model (see Additional file 1: Table 5).

By LOESS regression, the squared transformation of the continuous variable age had the most linear relationship with the logodds of the outcome (see Additonal file 1: Figure 2-3). Hemoglobin and creatinine had non-linear relationships with the outcome despite transformations. The inflection points of the natural variable were taken to create categorical variables. Hemoglobin inflection points were 115 and 135 and creatinine was 115, 140, and 160.

A total of 16 variables remain in the logistic regression model (Table 4). Significant predictors of MACE include variables such as Frailty, BMI >35, all levels of Creatinine, DM, Emergent and Urgent status and CABG + valve procedure type. Model beta coefficients are presented in the Additional file 1: Table 5.

Table 4 Logistic regression for MACE in a CABG/Valve population

Model performance

The concordance statistic for the logistic regression was 0.764, which is equivalent to an ROC of 76.4% (95% CI; 75–79). The Hosmer-Lemeshow goodness of fit statistic was not significant (p=0.3133).

The deciles of observed over predicted probabilities of MACE are plotted (Figure 1). Each data points falls on or very near the ideal line indicating excellent calibration.

Figure 1

Calibration plot of observed risk versus predicted risk.

Discussion

This paper outlines the development of a logistic regression model. Our model is unique for three reasons: 1) it performs well in a heterogeneous population including CABG, valve, and CABG + valve patients, 2) it predicts a composite outcome of quality indicators including death and major morbidities and, 3) it was developed for a contemporary cohort that represents a contemporary cardiac surgery practice. Also, we could not identify in the literature any models that include a CABG, valve and valve + CABG population with a composite end point.

Although many models exist that describe isolated CABG or isolated valve [1–9, 16–18], their use is limited to only a percentage of the cardiac surgical population. EuroSCORE, perhaps the most widely recognized cardiac surgical predictive model, is a mixed population model that has had great success in research and as a quality improvement tool [11, 18]. However, its derivation cohort is over 15 years old restricting its applicability to a contemporary practice. Our model is derived from a 2004–2009 cohort allowing for more current application. The EuroSCORE II has failed to perform better then the original model in certain procedure type subsets [13, 14]. Furthermore both EuroSCORE models only predicts mortality, and not other important quality indicators.

Provided the model performs well, the advantage of a heterogeneous population model exceeds that of single-procedure models in its ability to describe to majority of surgical patients and can help facilitate quality improvement efforts. However, certain predictive variables, such as descriptions of coronary artery disease or valve disease severity, cannot be included in the model, as they do not apply to the entire model derivation cohort. This might be problematic, as some variables like left main disease have previously been identified in the literature as highly important variables to include in CABG mortality models [25]. Although our model cannot contain this variable (as it would be entirely co-linear with CABG patients), it does contain other clinically relevant variables previously used in other published models [1–9, 16–18].

Modeling composite outcomes allows for a broader prediction of important post-operative events rather than being limited to a single outcome. Also, the components of our composite are each identified as quality indicators in cardiac surgery [15] allowing for improved clinical validity. The benefit of a correctly chosen composite outcome allows for more detailed description and prediction of the clinical population increasing the clinical relevance of the model.

The model discrimination is high with a ROC 76.4% (75–79, 95% CI) that exceeds published recommendations [26] and is similar to other published cardiac surgical models [1–9, 16–18]. This indicates that the model has good predictive ability. The bootstrap procedure, a form of internal validation [27, 28], allows for estimation of the 95% confidence interval. The tight 95% confidence interval provides a further estimation of reliability of the model. The calibration plot allows for a visual representation of the model’s performance (Figure 1). The observed to expected (O/E) data points fall on or very near the ideal line indicating excellent calibration of the model. The model performed well for each individual procedure subset as well (isolated CABG, isolated valve, CABG/valve).

Conclusions

We provide a statistically and clinically relevant model that is an essential tool in the era of quality improvement. The model includes heterogeneous procedures and a composite outcomes increasing its utility.

As the profile of the cardiac surgery patients changes, so must the predictive models used to describe this group. This is of utmost importance in the field of quality assessment and improvement. Models such as the one reported in this manuscript, assists many QI techniques. They can be used to perform pre-operative predictive risk matching to allow for comparison of matched groups and can risk adjust surgeon specific surgical outcomes for report carding [24, 26, 29]. The benefit of the mixed population and composite end points facilitates describing a contemporary clinical practice so that QI efforts are more productive. Developers of such models must be dedicated to upholding high statistical standards, so that the QI efforts actually benefit the patients. As QI efforts become a staple in cardiac surgery practice, models such as these are essential in propelling advancement in this field and improving outcomes for our patients.