External Validation of a Model for Prediction of Patients at Low Risk of Deterioration in Pneumonia

Introduction: Community-acquired pneumonia (CAP) complicated by acute respiratory failure is common and increases the risk of advanced respiratory support or death. Excluding low-risk patients can improve prognostic enrichment in future trials. We hypothesized that external validation of a previously published machine learning model for early prediction of hospitalized patients with CAP at low risk of respiratory deterioration or death would demonstrate adequate generalizability.


Methods: A gradient boosting machine (GBM) model predicting need for advanced respiratory support or death was previously developed and internally validated using routine clinical data, including demographics, comorbidities, vitals, and labs from the first 6 hours of admission in 4,379 CAP patients with acute hypoxic respiratory failure at Mayo Clinic, Rochester, Minnesota. Risk thresholds were set to maximize sensitivity while ensuring an NPV of 0.85 or greater, to screen out low-risk patients and enrich higher-risk cases for a prospective trial. The model demonstrated superior discrimination during internal testing (area under the curve (AUC) = 0.71; specificity = 0.57; sensitivity = 0.72) compared to the pneumonia severity index (PSI) (AUC = 0.65; specificity = 0.91; sensitivity = 0.24; P value < 0.001). External validation was performed using the MIMIC-IV database, a separate cohort from Beth Israel Deaconess Medical Center in Boston.


Results: The GBM model demonstrated good predictive performance, with an AUC of 0.69, sensitivity of 0.75, specificity of 0.52, NPV of 0.84, and PPV of 0.40, consistent with the prior internal validation results. For secondary outcomes, the external validation of the GBM model demonstrated consistent performance for in-hospital mortality (AUC 0.73 vs. 0.73), the need for invasive mechanical ventilation (AUC 0.72 vs. 0.74), and non-invasive mechanical ventilation (AUC 0.71 vs. 0.73).


Conclusions: This external validation of our GBM model using the MIMIC-IV data supports its generalizability. Further multicenter prospective validation is needed to assess clinical utility.