The Pre-brief
Decision-making in the Intensive Care Unit (ICU) is difficult. As patients and therapy become more complex, there is a risk of cognitive overload, affecting decision-making. Humans are prone to making decisions laden with cognitive biases, and (1,2). Furthermore, fatigue and cognitive stressors can increase cognitive and implicit bias (3). Amongst the difficult mental tasks, arguably the most difficult for an intensivist is determining a patient’s trajectory. Specifically, identifying patients for a proper de-escalation of aggressive care is important but often inconsistent (4). How frequently have you been asked the question, “Doctor, what are my chances?” To address some of the difficulties mentioned above, I joined a group of data science researchers to develop a mortality prediction tool (5,6). Machine learning can provide insight to help improve decision tools; this in turn may help alleviate the cognitive burden for clinical decision-making. Discerning patient trajectory and the likelihood of survival during an ICU stay can help families make appropriate patient-centered care decisions. With these problems in mind, the data science research group from Cooper/Rowan University Medical School is developing a machine learning app to estimate survival probability accurately and with transparency.
Background
The Mortality Prediction Tool is based upon a previous work presented at two engineering conferences and has been improved over time (5,6). The Mortality Prediction Tool was compared to prior severity score models, including APACHE IV and SAPS, as benchmark comparisons and was found to have better predictive metrics (5). The Mortality Prediction tool was retrospectively validated on the eICU Collaborative Research (eICU) and Medical Information Mart for Intensive Care (MIMIC) data sets. Our group used clinical features measured in the first 24 hours of ICU admission. MIMIC-III contains de-identified data for over 60,000 ICU admissions from the Harvard medical system. The eICU database includes data from 208 intensive care units (ICUs) across the United States for 200,859 patient encounters. The age cutoff for inclusion into our model was 18 to 99 years of age. The training datasets’ patient population is diverse with patients from Medical, Surgical, Neurologic and Cardiac ICUs. Two neural network models were introduced: Deterministic and Stochastic, but our focus will be on the latter.
In the Stochastic or Bayesian setting, the unknown parameters are considered to have probability distributions. The primary outputs for the Stochastic model are a mortality probability and a confidence score. The mortality probability output is the mean of the probability distribution. The confidence score is dependent on the variance of the probability distribution. A high variance for the probability distribution would have a low confidence score, and a low variance would have a high confidence score. The confidence score can be better understood by analyzing the probability distribution curve in figure 1. X is the distribution of probabilities from 0 to 1, and the Y-axis is the frequency of any given probability output. The green curve would have the highest confidence score because the variance of the probability outputs is low. The orange curve would have the lowest confidence score due to the wide variance. The confidence score is a feature that makes this model distinct, which to the best of our knowledge, is the first mortality prediction tool to do so. A Bayesian (Stochastic) model is preferable in this scenario because the output variance can quantify Uncertainty in the prediction. Finally, a major criticism of complex neural networks is that they are black boxes. To provide more transparency of the model process, there is an explanation plot that reflects the relative importance of the feature inputs. Performance metrics for the model shown in figure 2.
How to Use This Tool
ICU admission lab and exam findings should be used for information (see figure 3). GCS score should be based upon the exam performed before intubation and sedation if available. If the GCS scores are not available, they can be left blank, and the model will impute potential values (this will reduce the strength of the prediction). Mean Arterial Pressure (MAP) should reflect the value before the addition of pressors.
Hypothetical Clinical Example
71 y/o female with a history of CKD IV, COPD, HFpEF, and failure to thrive presents to the ICU with DOE and AMS. She is hypotensive and hypoxic with an inability to protect her airway, therefore promptly intubated. Prior to intubation she is on NIV with an fi02 of 100% with a Pao2 of 110. She is started on levophed at 10 mcg to maintain a MAP > 65. It is 3 am, and it has been a long and grueling shift. You are about to sit down with the family to discuss the likely trajectory for their loved one. Relevant exam and laboratory finding inputs are noted in figure 4. The family states that she does not have a living will, but they do not believe she would have wanted aggressive care, but they are adamant about maintaining full code status. They go on to state that the most important thing for their loved ones was independence but they are confident that she will be able to come back home. They ask, “What are her chances?” How can you use this tool in this clinical scenario? What is the interpretation of the output?
Interpretation
The output of the Stochastic model (model A) is shown below (figure 5). The model output indicates a high probability for death during the patient’s ICU stay, however the low confidence score indicates that the predicted probability has a wide variance. The explanation plot demonstrates the clinical features that pushed the prediction towards mortality (red) versus survival (green) (figure 6). Based upon the model output, the patient’s lactate and creatinine were the strongest features that pushed the prediction towards the high mortality probability. The probability output for this patient can be taken with a greater degree of caution than for an output with higher confidence, but it does not mean the output is useless. If this model were run 1000 times, the output would be more often closer to 90 than not. A safe interpretation would be that this patient does have a high probability of death (let us assume > 50%) during this hospitalization.
How would I use the output in this case?
The decision to continue aggressive care is complicated and multi-faceted. No model is 100% accurate, but it can help clinicians, families, and patients make more informed decisions when used conservatively and only as one data point amongst the other important aspects. As with other decision aids, this tool’s value lies in its ability to reinforce and/or question a provider’s initial assessment. The patient’s values should be at the center of the decision-making. In this hypothetical case, the family stated that they do not believe that their loved one would want this level of care, but they are not yet ready to deescalate the aggressive care. The tool can be used to provide a gauge of the likelihoods of survival. So how would I reply to the question, “What are her chances?” I would say that there is a moderate to high likelihood that she will not will not survive this ICU stay. It is at this point that I would ask the family to reflect on tracheostomy, long term acute care hospitalization, and all the complications that come with advanced age, comorbid conditions, and acute ICU interventions.
Limitations
- Our work has yet to be peer reviewed.
- Our model has inputs for only 12 features (we intended to keep the inputs as minimal as possible while maintaining performance). We may gain incremental improvements in the metrics with additional features, but this comes at the expense of ease of use.
- The model does not account for diagnosis. Performance may vary on a patient presenting with a stroke versus congestive heart failure or surgical disease with clear surgical solution.
- GCS remains one of the strongest and consistent predictors for our model despite not controlling for pre-hospital polysubstance or medication use.
- This model has yet to be validated prospectively in an external fashion.
- This model has not been validated against physician gestalt
The Debrief
Our Mortality Predictor Tool shows promise as a way to gauge initial trajectories of ICU patients to help families and providers make difficult decisions in a complex landscape. The future direction is to validate our app prospectively. The app should not be used as the primary means for clinical decision-making as patient care is multi-faceted and should not be driven by one result. It is available for use with the link below.
- ICU patient trajectory can be difficult to ascertain.
- Fatigue and cognitive stressors can increase the degree of bias in clinical decision-making.
- Machine learning can be helpful as clinical support tools.
- Complex machine learning models have been seen as black boxes, but our models provide a degree of transparency via an explanation plot.
- Providing uncertainty is a strength of the model, as with a low confidence score, you should be more cautious with the output.
Mortality Prediction Tool link: https://mortalitytool.com/
This work was supported by the U.S. Department of Education Graduate Assistance in Areas of National Need (GAANN) Grant Number P200A180055 and the National Science Foundation Award No. OAC-2008690.
Github link for more information about the App: https://github.com/jrepifano/mortality-tool
References
- DeCato TW, Engelberg RA, Downey L, Nielsen EL, Treece PD, Back AL, Shannon SE, Kross EK, Curtis JR. Hospital variation and temporal trends in palliative and end-of-life care in the ICU. Crit Care Med. 2013 Jun;41(6):1405-11. doi: 10.1097/CCM.0b013e318287f289. PMID: 23518869; PMCID: PMC4009379.
- J. R. Epifano, R. P. Ramachandran, S. Patel, and G. Rasool, “Towards an explainable mortality prediction model,”in2020 IEEE 30th International Workshop on MachineLearning for Signal Processing (MLSP), pp. 1–6, IEEE,2020.
- Dimah Dera, Ghulam Rasool, and Nidhal Bouaynaya, “Extended Variational Inference for Propagating Uncertainty in Convolutional Neural Networks,” in 2019 IEEE 29th Interna- tional Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2019, pp. 1–6.
- Wachter RM, Luce JM, Hearst N, Lo B. Decisions about resuscitation: inequities among patients with different diseases but similar prognoses. Ann Intern Med. 1989 Sep 15;111(6):525-32. doi: 10.7326/0003-4819-111-6-525. PMID: 2549825.
- Christakis NA, Asch DA. Biases in how physicians choose to withdraw life support. Lancet. 1993 Sep 11;342(8872):642-6. doi: 10.1016/0140-6736(93)91759-f. PMID: 8103146.
- Pollard, T., Johnson, A., Raffa, J. et al. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Sci Data 5, 180178 (2018).
- Saposnik G, Redelmeier D, Ruff CC, Tobler PN. Cognitive biases associated with medical decisions: a systematic review. BMC Med Inform Decis Mak. 2016;16(1):138. Published 2016 Nov 3. doi:10.1186/s12911-016-0377-1
- Johnson TJ, Hickey RW, Switzer GE, et al. The Impact of Cognitive Stressors in the Emergency Department on Physician Implicit Racial Bias. Acad Emerg Med. 2016;23(3):297-305. doi:10.1111/acem.12901
