Preparing for the next COVID: Deep Reinforcement Learning trained Artificial Intelligence discovery of multi-modal immunomodulatory control of systemic inflammation in the absence of effective anti-microbials

Abstract

BackgroundDespite a great deal of interest in the application of artificial intelligence (AI) to sepsis/critical illness, most current approaches are limited in their potential impact: prediction models do not (and cannot) address the lack of effective therapeutics and current approaches to enhancing the treatment of sepsis focus on optimizing the application of existing interventions, and thus cannot address the development of new treatment options/modalities. The inability to test new therapeutic applications was highlighted by the generally unsatisfactory results from drug repurposing efforts in COVID-19. HypothesisAddressing this challenge requires the application of simulation-based, model-free deep reinforcement learning (DRL) in a fashion akin to training the game-playing AIs. We have previously demonstrated the potential of this method in the context of bacterial sepsis in which the microbial infection is responsive to antibiotic therapy. The current work addresses the control problem of multi-modal, adaptive immunomodulation in the circumstance where there is no effective anti-pathogen therapy (e.g., in a novel viral pandemic or in the face of resistant microbes). MethodsThis is a proof-of-concept study that determines the controllability of sepsis without the ability to pharmacologically suppress the pathogen. We use as a surrogate system a previously validated agent-based model, the Innate Immune Response Agent-based Model (IIRABM), for control discovery using DRL. The DRL algorithm trains an AI on simulations of infection where both the control and observation spaces are limited to operating upon the defined immune mediators included in the IIRABM (a total of 11). Policies were learned using the Deep Deterministic Policy Gradient approach, with the objective function being a return to baseline system health. ResultsDRL trained an AI policy that improved system mortality from 85% to 10.4%. Control actions affected every one of the 11 targetable cytokines and could be divided into those with static/unchanging controls and those with variable/adaptive controls. Adaptive controls primarily targeted 3 different aspects of the immune response: 2nd order pro-inflammation governing TH1/TH2 balance, primary anti-inflammation, and inflammatory cell proliferation. DiscussionThe current treatment of sepsis is hampered by limitations in therapeutic options able to affect the biology of sepsis. This is heightened in circumstances where no effective antimicrobials exist, as was the case for COVID-19. Current AI methods are intrinsically unable to address this problem; doing so requires training AIs in contexts that fully represent the counterfactual space of potential treatments. The synthetic data needed for this task is only possible through the use of high-resolution, mechanism-based simulations. Finally, being able to treat sepsis will require a reorientation as to the sensing and actuating requirements needed to develop these simulations and bring them to the bedside.