Oakland score to identify low-risk patients with lower gastrointestinal bleeding performs well among emergency department patients

Abstract

Background

The Oakland Score predicts risk of 30-day adverse events among hospitalized patients with lower gastrointestinal bleeding (LGIB) possibly identifying patients who may be safe for discharge. The Oakland Score has not been studied among emergency department (ED) patients with LGIB. The Oakland Score composite outcome includes re-bleeding, defined as additional blood transfusion requirements and/or a further decrease in hematocrit (Hct) >/= 20% after 24 h in clinical stability; red blood cell transfusion; therapeutic intervention to control bleeding, including surgery, mesenteric embolization, or endoscopic hemostasis; in-hospital death, all cause; and re-admission with further LGIB within 28 days. Prediction variables include age, sex, previous LGIB admission, systolic blood pressure, heart rate, and hemoglobin concentration, and scores range from 0 to 35 points, with higher scores indicating greater risk.

Methods

Retrospective cohort study of adult (≥ 18 years old) patients with a primary ED diagnosis of LGIB across 21 EDs from March 1st, 2018, through March 1st, 2020. We excluded patients who were more likely to have upper gastrointestinal bleeding (esophago-gastroduodenoscopy without LGIB evaluation), patients who left against medical advice or prior to ED provider evaluation, ED patients without active health plan membership, and patients with incomplete Oakland Score variables. We assessed predictive accuracy by reporting the area under the receiver operator curve (AUROC) and sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios at multiple clinically relevant thresholds.

Results

We identified 8,283 patients with LGIB, 52% were female, mean age was 68, 49% were non-White, and 27% had an adverse event. The AUROC for predicting an adverse event was 0.85 (95% CI 0.84–0.86). There were 1,358 patients with an Oakland Score of </=8; 4.9% had an adverse event, and sensitivity of the Oakland Score at this threshold was 97% (95% CI 96%−98%).

Conclusion

The Oakland Score had high predictive accuracy among ED patients with LGIB. Prospective evaluation is needed to understand if the risk score could augment ED decision-making and improve outcomes and resource utilization.

Introduction

Background

Lower gastrointestinal bleeding (LGIB), or bleeding distal to the ligament of Treitz [1], contributes to approximately 100,000 hospital admissions in the U.S. annually [2,3,4]. Compared to upper gastrointestinal bleeding (UGIB), LGIB is less likely to present with hemorrhagic shock or require red blood cell (RBC) transfusions, and the in-hospital mortality rate for LGIB is lower, approximately 2–4% [5,6,7]. It is estimated that in the U.S. alone, healthcare for gastrointestinal (GI) bleeding incurred direct costs of $5 billion in 2014 [8], with individual costs for LGIB between $22,142 and $28,749 per hospitalization, or $4,492 per bed-day [9].

Although bleeding often resolves spontaneously, elderly patients and those with co-morbid conditions face higher risks of adverse outcomes [5, 7, 10]. Some data suggest that rates of hospitalization, resource utilization, and length of hospital stay for LGIB have surpassed UGIB [11,12,13]. Because most cases are self-limited, there is an increased interest in developing prediction models to identify patients that may be safely managed outside the hospital.

Importance

While validated risk scores exist for UGIB, decision support for LGIB is needed. The 2023 American College of Gastroenterology (ACG) guideline recommended using risk stratification tools such as the Oakland Score to triage hospitalized patients [2]. The Oakland Score predicts likelihood of safe hospital discharge, defined as the absence of all of the following after presentation: re-bleeding, defined as additional blood transfusion requirements and/or a further decrease in hematocrit (Hct) >/= 20% after 24 h in clinical stability; RBC transfusion; therapeutic intervention to control bleeding, including surgery, mesenteric embolization, or endoscopic hemostasis; in-hospital death, all cause; and re-admission with further LGIB within 28 days [14]. The original Oakland Score, derived and validated by Oakland et al. in 2017 among a nationally representative sample of hospitalized patients with LGIB in the U.K., used seven variables, including age, sex, history of LGIB, rectal examination findings, heart rate, systolic blood pressure, and hemoglobin level, to provide clinicians with a risk score between 0 and 35. Lower scores suggest patients at lower risk for adverse outcomes. The subsequent external validation study, which dropped the digital rectal examination (DRE) finding variable from the score due to limited electronic availability and included over 46,000 patients from 14 hospitals across the U.S., found that a threshold of ≤ 8 or ≤ 10 identified patients at very low risk for an adverse event, with sensitivity of 98.4% and 96.0%, respectively [15].

Goals

Prior validation studies of the Oakland Score have been limited to hospitalized patients [11, 15]. A recent single center Korean study of 376 patients found the Oakland Score was highly sensitive among low-risk ED patients [16]. Given the majority of unscheduled hospital admissions come through the ED [17], our goal was to assess how the Oakland Score performs in an ED population prior to the decision to admit to the hospital. We describe patient characteristics and outcomes among a large cohort of ED patients with LGIB and report performance of the Oakland Score to identify low-risk patients who may be safe for outpatient management.

Methods

Study design and setting

This retrospective cohort study was conducted among health plan members of Kaiser Permanente Northern California (KPNC) across 21 medical centers with associated EDs. KPNC is a large, integrated healthcare delivery system providing care for over 4.5 million members, including over 1.5 million annual ED visits. Of the 21 EDs, 13 had an observation unit during the study period, although protocols, number of beds, and primary treatment team (ED versus hospitalist) varied between sites. KPNC demographics reflect the ethnic and socioeconomic diversity of the surrounding areas [18]. KPNC utilizes an integrated and comprehensive electronic health record (EHR), including all ED, inpatient, and outpatient records with associated laboratory, pharmacy, and imaging data. The KPNC Institutional Review Board approved the study protocol and waived informed consent.

Selection of participants

We identified adult (>/= 18 years) health plan members with an ED encounter for LGIB between March 1st, 2018, and March 1st, 2020, using International Classification of Disease 10 (ICD-10 codes) to identify encounters with a primary diagnosis of LGIB or undifferentiated GI hemorrhage (Appendix, Table 1) [19]. To limit misclassification of patients with UGIB, we excluded patient encounters with a completed esophagogastroduodenoscopy (EGD) study without a concomitant study to evaluate for LGIB (specifically, we searched for evidence of colonoscopy, separating therapeutic from diagnostic, sigmoidoscopy, CT angiography, CT colonography, or tagged RBC scan) during hospitalization (Appendix, Table 2). We excluded patients who left against medical advice, eloped prior to ED provider evaluation, or who had incomplete or missing Oakland Score components [14]. Lastly, we excluded patients who had a non-KPNC ED encounter, for which full encounter data was not available, within seven days of the index ED visit, to ensure accurate capture of LGIB evaluation and outcomes.

Table 1 Oakland Score components with assigned points

Table 2 Patient characteristics, overall study cohort and among those with and without an adverse event

Variables

We extracted the six Oakland Score variables (dropping DRE findings because of inconsistent electronic capture) from the EHR, shown in Table 1 with their assigned points. Each eligible patient encounter was assigned a score between 0 and 35, with hemoglobin carrying the most weight (up to + 22 points). We used the first ED heart rate, systolic blood pressure, and hemoglobin concentration to calculate the score.

We collected patient demographics, healthcare utilization, and medications (anti-platelets and anti-coagulants) from the EHR and vital signs and laboratory values from the ED visit. We report patient co-morbidity burden using the Elixhauser Co-morbidity Index, both total Elixhauser score and key diagnostic categories included within the index [20]. We assessed neighborhood socioeconomic status at the census block track level, using the most recent American Community Service values that are built on 2010 Census data [21]. We assessed race and ethnicity data (Asian, Black, Hispanic, non-Hispanic White, other [including American Indian or Alaska Native, Native Hawaiian or other Pacific Islander, and multiple races or ethnicities], unknown, or missing).

Outcome

The primary outcome was the composite outcome used in the derivation and validation studies [14, 15], ‘safe for hospital discharge,’ as previously defined (Appendix, Table 2). Re-bleeding among hospitalized patients was defined as 20% or greater decrease in hematocrit after 24 h in which the hematocrit was stable (< 10% variation). We defined re-bleeding among discharged patients as a repeat ED visit for LGIB within 72 h or a repeat outpatient hematocrit within seven days of discharge that was 20% less than the last ED hematocrit.

Statistical analysis

We describe patient characteristics among the full sample and among those with and without an adverse event, shown in Table 2. We report rates of the composite outcome score for each adverse event at multiple Oakland Score thresholds. We assessed predictive accuracy using area under the receiver operator curve (AUROC) and precision recall curve from a logistic regression model. We examined sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), likelihood ratios, and F1 scores for several cutoffs of the Oakland Score (score </=7, </=8, </=9, </=10, </=15). We performed a variable importance list using a logistic regression model with all Oakland Score variables. Finally, as a sensitivity analysis, we examined the utility of additional variables (anti-platelet or anticoagulant medications and colonoscopy in the past two years) in the performance of a logistic regression with the validated Oakland Score predictors.

There was no missingness in the components of the Oakland Score (cases with missing variables were excluded as with the original studies) or among the outcome measures.

Data management and cleaning was done in SAS version 9.4 (SAS Institute Inc), and statistical analysis was performed using R version 4.3.1. Two-sided α < 0.05 indicated statistical significance.

Results

There were 8,283 patients with an ED visit for LGIB during the study period who met study inclusion criteria (Appendix, Fig. 1). The median age was 67.7 years, 48.6% were male, and 48.9% were non-White (Table 2). We found 3,627 (43.8%) were directly discharged from the ED or transferred, and an additional 2,367 (28.6%) were discharged after a brief observation stay (generally < 24 h); the remaining 2,289 (27.6%) were admitted to the hospital. There were high rates of co-morbid illness, including liver, kidney, and heart disease, and 43.8% and 29.1% of patients were taking anti-platelet medications and anti-coagulant medications, respectively.

Fig. 1

Oakland Score distribution by adverse event status

Overall, 2,243 patients (27.1%) experienced an adverse event. Patients without an adverse event were younger (mean age 65.6 versus 73.4) with lower rates of prior admissions for LGIB (5.8% versus 12.5%), lower rates of kidney, liver, heart disease, or coagulopathy, and they were less likely to be taking oral anticoagulant or antiplatelet medications and had higher presenting hemoglobin values (12.6 g/dL versus 9.0 g/dL).

The median Oakland Score was 13 with an interquartile range of 9 to 20. Figure 1 shows the overlap of Oakland Scores with rates of adverse events, demonstrating that increasing scores were associated with higher probability of adverse events. We found 1,358 (16.4%) had an Oakland Score of </=8 and 2,743 (33.1%) had a score of </=10. Rates of the composite Oakland Score outcome and rates of individual adverse events at different low-risk thresholds are seen in Table 3. We found 4.9% and 5.9% of patients had an adverse event using a threshold of </=8 or </=10, respectively, with red blood cell transfusion being the most common adverse event. Transfusion also contributed to nearly all of the increase in adverse events moving from a score of </=8 to </=10. There was no increase in the proportion of patients who died or required surgery, a 0.1% increase in the proportion that required mesenteric embolization, and a 0.1% decrease in the proportion that required a therapeutic colonoscopy.

Table 3 Rates of the Oakland Score composite outcome as well as individual adverse events at various lower risk thresholds

The Oakland Score maintained a high sensitivity (97.0%) for safe discharge among patients scoring 8, with a specificity of 21.4%. Increasing the threshold to 10 points lowered the sensitivity to 92.8% and increased specificity to 42.8%. Additional Oakland Score performance characteristics at various thresholds can be seen in Table 4.

Table 4 Performance characteristics of the Oakland Score at various thresholds with 95% confidence intervals

Figures 2 and 3 show the AUROC and area under the precision recall curve (AUPRC), respectively. The AUROC for predicting a patient safe for discharge was 0.85 (95% CI, 0.84–0.86), while the AUPRC was 0.85. The most important variables in predicting an adverse event, in order of influence on the outcome, were hemoglobin concentration, systolic blood pressure, age >/=70, and HR >/=110 (Appendix, Fig. 2). The variables with limited influence on the outcome included use of anti-coagulant medications, heart rate of 70–89, history of LGIB in the prior two years, male gender, and history of colonoscopy in the prior two years.

Fig. 2

Area under the receiver operator curve for the Oakland Score

Fig. 3

Area under the precision recall curve for the Oakland Score

We found that overall, ED admission decision-making correlated with predicted risk of an adverse event, with ED discharge more common among lower-risk patients and admission of higher-risk patients. Table 5 shows that 8.4% of very low-risk patients (score </=7) and 8.0% of low-risk patients (score 8–9) were admitted to the hospital, while 19.0% of moderate-risk (score 11–14) and 18.8% of high-risk patients (score >/=15) were discharged, and adverse event rates were 12.0% and 51.7%, respectively, in these risk strata. Among discharged patients who were moderate risk, 22 (2.3%) had a re-bleed, and five (0.5%) had LGIB readmission. Among discharged patients who were high risk, 33 (5.9%) had a re-bleed, and seven (1.3%) had LGIB readmission. The ED discharge rate of high-risk patients varied nearly 3-fold across the 21 EDs in our health system.

Table 5 ED disposition (direct discharge, treatment in an observation area followed by discharge, or hospital admission), adverse event rates, and ED mortality by predicted risk (Oakland Score threshold)

Discussion

To our knowledge, this is the first large, community-based, multi-center external validation of the Oakland Score to assess whether the prediction tool can accurately identify ED patients with LGIB who are at low risk for an adverse event and therefore may be appropriate for safe ED outpatient care. Among a diverse cohort of 8,283 patients across 21 EDs, we found the Oakland Score was highly sensitive to identifying low-risk patients. Generally, ED provider admission decisions correlated with predicted risk, although we did find wide variation between EDs regarding discharge of moderate and high-risk patients, and some discharged patients experienced adverse events, suggesting clinical decision support might help aid risk stratification.

Because we shifted the study population further upstream (from hospitalized patients to ED patients), it included a larger proportion of low and low/moderate risk patients compared to the Oakland Score derivation and validation studies, which included only hospitalized patients. ED patients include those across the risk spectrum, including younger, healthier patients who are less likely to have a severe cause of LGIB or experience an adverse event. Only 27.1% of patients in our cohort experienced an adverse event, compared to 52.1% among the hospitalized cohort in the external validation study [15]. We found 16.4% of patients were predicted to be low risk (Oakland Score of </=8) in our population, while only 8.7% were predicted to be low risk among hospitalized patients in the external Oakland Score validation study [15].

Our findings of risk model performance suggest the Oakland Score can be used to safely identify low-risk patients who may be safe for discharge. Using a </=8 threshold, the Oakland Score was 97% sensitive to identify patients who had an adverse event, comparable to the sensitivity found in the external validation study among hospitalized patients (98.4%) [15]. The recent Korean validation study of the Oakland Score among ED patients found a sensitivity of 100% using a </=8 threshold.

Several risk scores have been proposed to help risk stratify patients with LGIB, including the Strate, NOBLADS, BLEED, SHA₂PE, and Oakland scores, each with slightly different goals and predictive accuracy [2, 22, 23]. The Strate score uses seven clinical variables, does not require bloodwork, and is designed to predict severe LGIB. The NOBLADS score consists of eight variables and is designed to predict severe bleeding. The BLEED score consists of five variables and is designed to predict an in-hospital complication. The SHA₂PE score consists of seven variables and was developed among ED patients with LGIB. In an external validation study of the SHA₂PE score and comparison to the Oakland Score among a cohort of 595 hospitalized patients with LGIB, both scores performed well; the AUCs for the Oakland Score and SHA₂PE score were 0.85 (95% CI 0.82–0.89) and 0.797 (95% CI 0.75–0.84), respectively [23, 24].

We found the Oakland Score had an AUROC of 0.85 (95% CI 0.84–0.86) suggesting good discriminative performance among ED patients with LGIB. Similar to findings in the validation study of hospitalized patients [15], specificity and positive predictive values were low among patients predicted to be low risk, suggesting the risk estimates are safe and conservative. This matches providers’ desire for a decision support tool that limits misclassifying high-risk patients as low risk.

We found 42.4% of patients were directly discharged from the ED, and an additional 28.6% were discharged after a brief observation stay. In general, ED providers discharged patients predicted to be lower risk, and adverse events among discharged patients were infrequent. We found that 8.1% of low-risk patients were hospitalized, and 18.8% of high-risk patients were discharged, suggesting an opportunity to better match predicted risk with intensity of care (hospitalization) with a risk stratification tool. However, it is important to recognize that a variety of factors, many of which are not captured by the Oakland Score, may influence a provider’s decision to admit a patient, including medical comorbidities, social determinants of health, and other clinical considerations.

Hospital admissions for LGIB are costly [9], and prior studies show that bleeding is self-limited in most patients [3, 5]. Utilizing risk prediction tools like the Oakland Score during admission decision-making could help optimize resource allocation while also mitigating potential negative effects of prolonged or unnecessary hospital stays [2]. Furthermore, the Oakland Score may assist providers in identifying true high-risk patients who may benefit from close observation and treatment in the hospital. ACG guidelines on the management of LGIB support use of risk stratification tools including the Oakland Score and threshold of </=8, and our findings among ED patients suggest this score may be useful to stratify patients prior to the decision to admit or discharge a patient. Other scores described in the ACG guidelines include the NOBLADS score and the SHA₂PE score for identifying low-risk patients [2]. Risk prediction scores are designed to support clinical decision making and should not be used in lieu of a clinician’s judgment.

Oakland et al. suggest that increasing the threshold from 8 to 10 points identifies a larger portion of low-risk patients while maintaining good discriminative performance. Identifying the ideal threshold score for safe discharge requires balancing the risk of higher rates of adverse events among patients identified as low risk and safe for discharge with the goal of optimizing resource utilization decisions. Increasing the “low-risk” threshold from </=8 to </=10 doubled the number of patients in this category (from 16.4 to 33.1%), while sensitivity decreased from 97 to 93%, and rates of adverse events increased from 4.9 to 5.9%. This 1% difference in adverse events was predominantly driven by need for transfusion and not by death or invasive procedures.

More than two-thirds of patients in our cohort experienced no adverse outcomes as previously defined. The most common adverse event was RBC transfusion, consistent with other large observational studies of patients with LGIB [4, 15]. The Oakland Score composite outcome includes a broad spectrum of invasive interventions and serious outcomes. Among the 2,743 patients identified as low risk using the </=10 threshold, rates of serious invasive procedures or in-hospital mortality were low: less than 10 patients required mesenteric embolization or surgery and three patients died. There were 38 patients who had evidence of a re-bleed after ED discharge in this group, although only 10 were admitted to the hospital for LGIB within 28 days.

As with the prior external validation study [15], we found the score maintained high predictive accuracy even without DRE findings. The remaining six variables were sufficient to maintain high sensitivity to identify low-risk patients. Among these, hemoglobin concentration and systolic blood pressure had the highest clinical significance in predicting an adverse event, aligning with their respective weightings in the score calculation. Although we hypothesized that incorporating additional clinical variables (such as use of anti-platelets or anti-coagulants) might further improve model discrimination, these variables were found to be non-predictive in the Oakland Score derivation study [14], and had little influence on the outcome in our dataset (Appendix, Fig. 2). A simple model, including only demographic factors, vital signs, and one blood test, allows for ease of calculation and may increase interpretability and provider trust, compared to more complex models.

Further study is needed to prospectively validate the Oakland Score at the point of care and compare ED provider gestalt with risk predictions. Additional investigations should also include model performance among different populations to determine potential for bias.

Limitations

The main limitation of this study is its retrospective design. Origin of bleeding is difficult to confirm retrospectively among patients with an ED diagnosis code of undifferentiated GI hemorrhage. Although we tried to limit inclusion of patients with UGIB, there may have been some mis-categorization. There may be unmeasured severity of illness or patient preferences that may have driven hospital admission decision-making.

Additionally, other systems may have less access to follow up care after ED discharge, and this may impact risk thresholds and admission decision-making, potentially limiting the generalizability of our findings.

Conclusion

We present findings from a large, diverse, multi-center external validation study of the Oakland Risk Score among ED patients diagnosed with LGIB. We found that the Oakland Score maintained high predictive accuracy in identifying low-risk patients who may be amenable to outpatient care in this novel application of the score to patients prior to the hospital admission decision. Next steps could include a prospective implementation study to ensure the safety of risk estimates.