- Research article
- Open Access
- Open Peer Review
Development of a predictive model for major adverse cardiac events in a coronary artery bypass and valve population
Journal of Cardiothoracic Surgery volume 8, Article number: 177 (2013)
Quality improvement initiatives in cardiac surgery largely rely on risk prediction models. Most often, these models include isolated populations and describe isolated end-points. However, with the changing clinical profile of the cardiac surgical patients, mixed populations models are required to accurately represent the majority of the surgical population. Also, composite model end-points of morbidity and mortality, better reflect outcomes experienced by patients.
The model development cohort included 4,270 patients who underwent aortic or mitral valve replacement, or mitral valve repair with/without coronary artery bypass grafting, or isolated coronary artery bypass grafting. A composite end-point of infection, stroke, acute renal failure, or death was evaluated. Age, sex, surgical priority, and procedure were forced, a priori, into the model and then stepwise selection of candidate variables was utilized. Model performance was evaluated by concordance statistic, Hosmer-Lemeshow Goodness of Fit, and calibration plots. Bootstrap technique was employed to validate the model.
The model included 16 variables. Several variables were significant such as, emergent surgical priority (OR 4.3; 95% CI 2.9-7.4), CABG + Valve procedure (OR 2.3; 95% CI 1.8-3.0), and frailty (OR 1.7; 95% CI 1.2-2.5), among others. The concordance statistic for the major adverse cardiac events model in a mixed population was 0.764 (95% CL; 0.75-0.79) and had excellent calibration.
Development of predictive models with composite end-points and mixed procedure population can yield robust statistical and clinical validity. As they more accurately reflect current cardiac surgical profile, models such as this, are an essential tool in quality improvement efforts.
Quality improvement initiatives (QI), a cornerstone of cardiac surgery, have largely relied on predictive models to advance the quality of care cardiac surgical patients receive. For the last two decades, coronary artery bypass grafting (CABG) has dominated clinical practice in cardiac surgery, and therefore the majority of quality improvement initiatives have focused on surgical outcomes following isolated CABG surgery [1–4]. More recently, however, there has been an increase in valve and valve-CABG cases in cardiac surgery  and existing predictive models for isolated CABG may not accurately reflect current practice profiles. These models may achieve great statistical validity but lack clinical validity as the models are applicable to only a decreasing percentage of the clinical practice. In order to maintain continued success in quality improvement, it is important to delineate risk profiles for a group of mixed procedures, including CABG, valve, and valve + CABG, that characterize current clinical practice.
To accommodate the shift in the profile of cardiac surgical practice, isolated valve and valve plus CABG models have been developed [5–10]. Some debate exists regarding the validity of developing models with hetero-geneous procedures with some advocating for single procedure models [7, 9, 10]. Homogeneity within the population allows for simplicity of model development with improved reliability but limits sample size available for development and validation . Although mixed procedural models may be confounded or biased by different pathophysiological and risk profiles, they improve sample size  and increase their relevance to current surgical practice.
Largely, mixed procedural models have focused on aortic/mitral or valve +/− CABG, with few CABG, valve and CABG +valve models present in the literature. The EuroSCORE model, one of the mostly widely used cardiac surgical prognostic models, has successfully achieved both statistical and clinical relevance as it applies to a mixed surgical practice including CABG, valve, and CABG + valve. However its development and validation cohort were derived from a population sample that is over fifteen years old  limiting its applicability to a contemporary surgical practice. Recently, the EuroSCORE II updated its risk model obviating some of the limitations of the original model . However, the performance of this heterogeneous procedure model performed poorly in risk prediction for several of its component procedures types [13, 14]. In addition, both the EuroSCORE and the EuroSCORE II are only a mortality model.
Many models describe death as a lone end point, despite that fact that cardiac patients experience a variety of relevant morbidity. These alternate surgical outcomes are also important quality indicators for cardiac surgical care  and targets for QI initiatives. Subsequently, modeling composite end points such as, major adverse cardiac events (MACE), that included both important morbidity and mortality may provide more insight to the surgical outcomes experienced by patients as well as increase statistical power in low frequency end points.
Comprehensive models that include most major cardiac surgical procedures may allow for better understanding of patient risk profiles and facilitate quality improvement initiatives directed towards the majority of patients seen in current practice. The objective of this paper is to develop a CABG, valve, CABG + valve morbidity and mortality (MACE) model that can be applied to the majority of patients undergoing cardiac surgery.
Data source and study population
This study is a retrospective cohort design. The Maritime Heart Center Cardiac Surgery Registry is a detailed clinical database housed at the Queen Elizabeth II Health Science Center (QEII HSC) Halifax, Nova Scotia, an academic tertiary care centre performing 1200 cardiac cases per year. It includes pre-, intra-, and post-operative data prospectively collected on all cardiac surgical cases performed at the QEII HSC from 1995 to present. Trained abstractors collect data, and a database administrator maintains the registry. The database is audited annually.
The model development cohort included all patients undergoing isolated coronary artery bypass grafting (CABG), isolated aortic valve replacement, isolated mitral valve repair or replacement with or without concomitant CABG performed at the QEII HSC since 2004 to 2009. The development cohort was restricted to these years in order to maintain a modern population relevant to current clinical practice.
The primary outcome was a composite end point defined as Major Adverse Cardiac Events (MACE) that included in-hospital death, stroke (persisting at discharge or transient), acute renal failure (new post-operative renal failure or acute on chronic (>50% increase from baseline creatinine)), or infection (sepsis, pneumonia, or deep sternal wound infection).
Candidate variables for model development included the following preoperative characteristics: age, sex, diabetes, frailty, chronic obstructive pulmonary disease, redo sternotomy, atrial fibrillation, hemoglobin, peripheral vascular disease, cerebro-vascular disease, creatinine, ejection fraction < 40%, New York Heart Association classification (NYHA I-IV) and body mass index (BMI). Surgery-related data such as urgency of surgery, and procedure type were also included. These candidate variables were chosen a priori and selected through rigorous review of the literature [1–9, 16–18] (Table 1).
Multi-colinearity of candidate variables was assessed with variance inflation factor (VIF) (Additional file 1: see Statistical Methods). The linear relationships of the natural variables and their transformations were assessed through locally weighted scatterplot smoothing (LOESS) regression  (see Additional file 1: Linearity and Transformations and Figure 2-3). The WHO classification of BMI was used .
A multivariate logistic regression analysis was used to describe MACE.
Age, sex, procedure type and surgical priority were chosen a priori and forced into the model. Stepwise selection was implemented for the remainder of the candidate variables. The concordance statistic and −2 log likelihood were evaluated to assess the contribution of each variable to the model. If a variable did not contribute to an increase in the C or −2 log likelihood statistic it was not retained in the model.
Model discrimination was determined using the concordance statistic . Model calibration was assessed by the Hosmer-Lemeshow goodness-of-fit statistic [22, 23] as well as calibration plots . Deciles of observed and predicted probabilities of MACE were plotted for the calibration plots . Bootstrap procedure was used to internally validate the model.
All statistical analysis was performed using SAS software version 9.2 (SAS, Cary, NC).
Approval for conducting this study was obtained from the Institutional Review Board of the Capital District Health Authority. The requirement to obtain informed consent was waived under Section 2.1c of the Tri-Council Policy Statement. All personal identifiers were stripped prior to data analysis to ensure patient anonymity and confidentiality.
The authors had full access to the data and take full responsibility for its integrity. All authors have read and agree to the manuscript as written.
A total of 4,270 patients underwent CABG, valve (aortic valve replacement, mitral valve replacement, or mitral valve repair) or CABG + valve at the QEII HSC from Jan 2004 to Dec 2009. The model cohort was 65% of the total case volume (6,525) performed during the study period. The distributions of risk factors in the development cohort are displayed in Table 2. The prevalence of MACE in this cohort was 15.7% (n=669). The prevalence of MACE was higher in the CABG plus valve group (32%, n=155) than in the isolated procedures (CABG 13%, n=416; Valve 14%, n=98). The frequencies of MACE for each procedure as well as the components of MACE are summarized in Table 3.
Assessment of the variance inflation for each variable revealed that no variable exceeded 4.0 allowing all variables to remain in the final model (see Additional file 1: Table 5).
By LOESS regression, the squared transformation of the continuous variable age had the most linear relationship with the logodds of the outcome (see Additonal file 1: Figure 2-3). Hemoglobin and creatinine had non-linear relationships with the outcome despite transformations. The inflection points of the natural variable were taken to create categorical variables. Hemoglobin inflection points were 115 and 135 and creatinine was 115, 140, and 160.
A total of 16 variables remain in the logistic regression model (Table 4). Significant predictors of MACE include variables such as Frailty, BMI >35, all levels of Creatinine, DM, Emergent and Urgent status and CABG + valve procedure type. Model beta coefficients are presented in the Additional file 1: Table 5.
The concordance statistic for the logistic regression was 0.764, which is equivalent to an ROC of 76.4% (95% CI; 75–79). The Hosmer-Lemeshow goodness of fit statistic was not significant (p=0.3133).
The deciles of observed over predicted probabilities of MACE are plotted (Figure 1). Each data points falls on or very near the ideal line indicating excellent calibration.
This paper outlines the development of a logistic regression model. Our model is unique for three reasons: 1) it performs well in a heterogeneous population including CABG, valve, and CABG + valve patients, 2) it predicts a composite outcome of quality indicators including death and major morbidities and, 3) it was developed for a contemporary cohort that represents a contemporary cardiac surgery practice. Also, we could not identify in the literature any models that include a CABG, valve and valve + CABG population with a composite end point.
Although many models exist that describe isolated CABG or isolated valve [1–9, 16–18], their use is limited to only a percentage of the cardiac surgical population. EuroSCORE, perhaps the most widely recognized cardiac surgical predictive model, is a mixed population model that has had great success in research and as a quality improvement tool [11, 18]. However, its derivation cohort is over 15 years old restricting its applicability to a contemporary practice. Our model is derived from a 2004–2009 cohort allowing for more current application. The EuroSCORE II has failed to perform better then the original model in certain procedure type subsets [13, 14]. Furthermore both EuroSCORE models only predicts mortality, and not other important quality indicators.
Provided the model performs well, the advantage of a heterogeneous population model exceeds that of single-procedure models in its ability to describe to majority of surgical patients and can help facilitate quality improvement efforts. However, certain predictive variables, such as descriptions of coronary artery disease or valve disease severity, cannot be included in the model, as they do not apply to the entire model derivation cohort. This might be problematic, as some variables like left main disease have previously been identified in the literature as highly important variables to include in CABG mortality models . Although our model cannot contain this variable (as it would be entirely co-linear with CABG patients), it does contain other clinically relevant variables previously used in other published models [1–9, 16–18].
Modeling composite outcomes allows for a broader prediction of important post-operative events rather than being limited to a single outcome. Also, the components of our composite are each identified as quality indicators in cardiac surgery  allowing for improved clinical validity. The benefit of a correctly chosen composite outcome allows for more detailed description and prediction of the clinical population increasing the clinical relevance of the model.
The model discrimination is high with a ROC 76.4% (75–79, 95% CI) that exceeds published recommendations  and is similar to other published cardiac surgical models [1–9, 16–18]. This indicates that the model has good predictive ability. The bootstrap procedure, a form of internal validation [27, 28], allows for estimation of the 95% confidence interval. The tight 95% confidence interval provides a further estimation of reliability of the model. The calibration plot allows for a visual representation of the model’s performance (Figure 1). The observed to expected (O/E) data points fall on or very near the ideal line indicating excellent calibration of the model. The model performed well for each individual procedure subset as well (isolated CABG, isolated valve, CABG/valve).
We provide a statistically and clinically relevant model that is an essential tool in the era of quality improvement. The model includes heterogeneous procedures and a composite outcomes increasing its utility.
As the profile of the cardiac surgery patients changes, so must the predictive models used to describe this group. This is of utmost importance in the field of quality assessment and improvement. Models such as the one reported in this manuscript, assists many QI techniques. They can be used to perform pre-operative predictive risk matching to allow for comparison of matched groups and can risk adjust surgeon specific surgical outcomes for report carding [24, 26, 29]. The benefit of the mixed population and composite end points facilitates describing a contemporary clinical practice so that QI efforts are more productive. Developers of such models must be dedicated to upholding high statistical standards, so that the QI efforts actually benefit the patients. As QI efforts become a staple in cardiac surgery practice, models such as these are essential in propelling advancement in this field and improving outcomes for our patients.
Body mass index
Coronary artery bypass grafting
- QEII HSC:
Queen Elizabeth II health science center
- QI Quality:
Locally weighted scatter plot smoothing
Major adverse cardiac events
New York heart association
Variance inflation factor
World Health Organization.
Shahian DM, O'Brien SM, Filardo G, Ferraris VA, Haan CK, Rich JB: The society of thoracic surgeons 2008 cardiac surgery risk models: part 1–coronary artery bypass grafting surgery. Ann Thorac Surg. 2009, 88: S2-S22. 10.1016/j.athoracsur.2009.05.053.
Jones RH, Hannan EL, Hammermeister KE, Delong ER, O'Connor GT, Luepker RV: Identification of preoperative variables needed for risk adjustment of short-term mortality after coronary artery bypass graft surgery. The working group panel on the cooperative CABG database project. J Am Coll Cardiol. 1996, 28: 1478-1487. 10.1016/S0735-1097(96)00359-2.
Shroyer AL, Coombs LP, Peterson ED, Eiken MC, DeLong ER, Chen A: The society of thoracic surgeons: 30-day operative mortality and morbidity risk models. Ann Thorac Surg. 2003, 75: 1856-1864. 10.1016/S0003-4975(03)00179-6. discussion 1864–5
O’Connor G, Plume S, Olmstead E, Coffin LH, Morton JR, Maloney CT: Multivariate prediction of in-hospital mortality associated with coronary artery bypass surgery. Northern new england cardiovascular disease study group. Circ. 1992, 85: 2110-2118. 10.1161/01.CIR.85.6.2110.
Shahian DM, O'Brien SM, Filardo G, Ferraris VA, Haan CK, Rich JB: The society of thoracic surgeons 2008 cardiac surgery risk models: part 3–valve plus coronary artery bypass grafting surgery. Ann Thorac Surg. 2009, 88: S43-S62. 10.1016/j.athoracsur.2009.05.055.
OBrien S, Shahian D, Filardo G, Ferraris VA, Haan CK, Rich JB: The society of thoracic surgeons 2008 cardiac surgery risk models: part 2--isolated valve surgery. Ann Thorac Surg. 2009, 88: S23-S42. 10.1016/j.athoracsur.2009.05.056.
Hannan EL, Racz MJ, Jones RH, Gold JP, Ryan TJ, Hafner JP: Predictors of mortality for patients undergoing cardiac valve replacements in new york state. Ann Thorac Surg. 2000, 70: 1212-1218. 10.1016/S0003-4975(00)01968-8.
Hannan EL, Wu C, Bennett EV, Carlson RE, Culliford AT, Gold JP: Risk index for predicting in-hospital mortality for cardiac valve surgery. Ann Thorac Surg. 2007, 83: 921-929. 10.1016/j.athoracsur.2006.09.051.
Jamieson WR, Edwards FH, Schwartz M, Bero JW, Clark RE, Grover FL: Risk stratification for cardiac valve replacement. national cardiac surgery database. database committee of the society of thoracic surgeons. Ann Thorac Surg. 1999, 67: 943-951. 10.1016/S0003-4975(99)00175-7.
Nowicki ER, Birkmeyer NJ, Weintraub RW, Leavitt BJ, Sanders JH, Dacey LJ: Multivariable prediction of in-hospital mortality associated with aortic and mitral valve surgery in northern new england. Ann Thorac Surg. 2004, 77: 1966-1977. 10.1016/j.athoracsur.2003.12.035.
Roques F, Nashef SA, Michel P, Gauducheau E, de Vincentiis C, Baudet E: Risk factors and outcome in european cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg. 1999, 15: 816-822. 10.1016/S1010-7940(99)00106-2. discussion 822–3
Nashef SA, Roques F, Sharples LD, Nilsson J, Smith C, Goldstone AR, Lockowandt U: EuroSCORE II. Eur J Cardiothorac Surg. 2012, 41 (4): 734-744. 10.1093/ejcts/ezs043. Epub 2012 Feb 29
Howell NJ, Head SJ, Freemantle N, van der Meulen TA, Senanayake E, Menon A, Kappetein AP, Pagano D: The new EuroSCORE II does not improve prediction of mortality in high-risk patients undergoing cardiac surgery: a collaborative analysis of two European centres. Eur J Cardiothorac Surg. 2013, Epub ahead of print
Di Dedda U, Pelissero G, Agnelli B, De Vincentiis C, Castelvecchio S, Ranucci M: Accuracy, calibration and clinical performance of the new EuroSCORE II risk stratification system. Eur J Cardiothorac Surg. 2013, 43 (1): 27-32. 10.1093/ejcts/ezs196. Epub 2012 Jul 20
Guru V, Anderson GM, Fremes SE, O'Connor GT, Grover FL, Tu JV: The identification and development of canadian coronary artery bypass graft surgery quality indicators. J Thorac Cardiovasc Surg. 2005, 130: 1257-
Rankin JS, Hammill BG, Ferguson TB, Glower DD, O'Brien SM, DeLong ER: Determinants of operative mortality in valvular heart surgery. J Thorac Cardiovasc Surg. 2006, 131: 547-57. 10.1016/j.jtcvs.2005.10.041.
Ambler G, Omar RZ, Royston P, Kinsman R, Keogh BE, Taylor KM: Generic, simple risk stratification model for heart valve surgery. Circulation. 2005, 112: 224-31. 10.1161/CIRCULATIONAHA.104.515049.
Nashef SA, Roques F, Hammill BG, Peterson ED, Michel P, Grover FL: Validation of european system for cardiac operative risk evaluation (EuroSCORE) in north american cardiac surgery. Eur J Cardiothorac Surg. 2002, 22: 101-5. 10.1016/S1010-7940(02)00208-7.
Cleveland WS: Robust locally weighted regression and smoothing scatter- plots. J Amer Stats Ass. 1979, 74: 829-36. 10.1080/01621459.1979.10481038.
WHO: WHO Technical Report Series 854. Physical status: the use and interpretation of anthropometry. Report of a WHO Expert Committee. 1995, Geneva: World Health Organization
Hanley JA, McNeil BJ: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982, 143: 29-36.
Hosmer DW, Lemeshow S: A goodness-of-fit test for the multiple regression mdel. Commun Statistics. 1980, A10: 1043-1069.
Marcin JP, Romano PS: Size matters to a model's fit. Crit Care Med. 2007, 35: 2212-3. 10.1097/01.CCM.0000281522.70992.EF.
Harrell FE, Lee KL, Mark DB: Tutorial in biostatistics multivariable prognostic models: issues in developing mdels, evaluating assumptions and adequacy, and measuring and reducing errors. Stats Med. 1996, 15: 361-87. 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4.
Tu JV, Sykora K, Naylor CD: Assessing the outcomes of coronary artery bypass graft surgery: how many risk factors are enough? steering committee of the cardiac care network of ontario. J Am Coll Cardiol. 1997, 30: 1317-23. 10.1016/S0735-1097(97)00295-7.
Spiegelhalter DJ: Probabilistic prediction in patient management and clinical trials. Stat Med. 1986, 5: 421-33. 10.1002/sim.4780050506.
Concato J, Feinstein AR, Holford TR: The risk of determining risk with multivariable models. Ann Intern Med. 1993, 118: 201-10. 10.7326/0003-4819-118-3-199302010-00009.
Altman DG, Royston P: What do we mean by validating a prognostic model?. Stat Med. 2000, 19: 453-73. 10.1002/(SICI)1097-0258(20000229)19:4<453::AID-SIM350>3.0.CO;2-5.
Harrell F, Califf R, Pryor D, Lee K, Rosati R: Evaluating the yield of medical tests. JAMA. 1982, 247: 2543-6. 10.1001/jama.1982.03320430047030.
This work was supported by the Heart and Stroke Foundation of Canada grant-in-aid and a Nova Scotia Health Research Foundation salary grant (PSO-SRA-2010–6546).
The authors declare that they have no competing interests.
CRH carried out the data analysis and authored the manuscript. KJB provided statistical support. CRH, KJB and RB participated in the design of the study. All authors contributed to the editing of the manuscript and have approved the final manuscript.
Electronic supplementary material
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
Herman, C.R., Buth, K.J., Légaré, J. et al. Development of a predictive model for major adverse cardiac events in a coronary artery bypass and valve population. J Cardiothorac Surg 8, 177 (2013). https://doi.org/10.1186/1749-8090-8-177
- Cardiac surgery
- Predictive model