External validation of predictive models for antibiotic susceptibility of urine culture

Abstract

Objective

To develop, externally validate, and test a series of computer algorithms to accurately predict antibiotic susceptibility test (AST) results at the time of clinical diagnosis, up to 3 days before standard urine culture results become available, with the goal of improving antibiotic stewardship and patient outcomes.

Patients and Methods

Machine learning algorithms were developed and trained to predict susceptibility or resistance using over 4.7 million discrete AST classifications from urine cultures in a cohort of adult patients from outpatient and inpatient settings from 2012 to 2022. The algorithms were validated on a cohort from a geographically-distant hospital system, ~1931 km (~1200 miles) from the training cohort facilities, from the same time period. Finally, algorithms were clinically validated in a contemporary cohort and compared to the empiric therapy prescribed by clinicians. Appropriateness of the antibiotics selected by clinicians and the algorithm during the clinical validation was compared.

Results

Algorithms were accurate during clinical validation (area under the receiver operating characteristic curve [AUC] 0.71–0.94) for all 11 tested antibiotics. The algorithms’ accuracy improved as the organism was identified (AUC 0.79–0.97). In external validation in a geographically-distant cohort, the algorithms remained accurate even without additional training on this group (AUC 0.69–0.87). When the algorithms were trained on the antibiogram from the geographically-distant hospital, the accuracy improved (AUC 0.70–0.93). When algorithms’ performances were tested against clinicians in a contemporary cohort for the empiric prescription of oral antibiotics, the drug agent suggested by the algorithms more frequently resulted in adequate empiric coverage.

Conclusions

Machine learning algorithms trained on a large dataset are accurate in prediction of urine culture susceptibility vs resistance up to 3 days prior to urine AST availability. Clinical implementation of such an algorithm could improve both clinical care and antimicrobial stewardship.