Dulberg, Z. | Search bioRxiv

bioRxiv2025

On the duality of pain and pleasure processing: Why two dimensions of valence may be better than one

Reinforcement learning treats reward maximization as a single objective, such that pain avoidance is implicit in pleasure seeking. However, humans appear to have distinct neural systems for processing pain and pleasure. This paper investigates the computational advantages of this separation through grid-world experiments. We demonstrate that modular architectures employing distinct max and min operators for value propagation outperform monolithic models in non-stationary environments. This separation allows agents to simultaneously grow and shrink learned values without interference, enabling both efficient reward collection and punishment avoidance. Additionally, these separate systems can be dynamically arbitrated using a mood-like mechanism for rapid adaptation. Our results suggest that separate pain and pleasure systems may have evolved to enable safe and efficient learning in changing environments. Nature has placed mankind under the governance of two sovereign masters, pain and pleasure Jeremy Bentham

neuroscience↗

bioRxiv2024

Adapting to loss: A normative account of grief

Grief is a reaction to loss that is observed across human cultures and even in other species. While the particular expressions of grief vary significantly, universal aspects include experiences of emotional pain and frequent remembering of what was lost. Despite its prevalence, and its obvious nature, considering grief from a normative perspective is puzzling: Why do we grieve? Why is it painful? And why is it sometimes prolonged enough to be clinically impairing? Using the framework of reinforcement learning with memory replay, we offer answers to these questions and suggest, counter-intuitively, that grief may have normative value with respect to reward maximization.

neuroscience↗

bioRxiv2022

Having "multiple selves" helps learning agents explore and adapt in complex changing worlds

Satisfying a variety of conflicting needs in a changing environment is a fundamental challenge for any adaptive agent. Here, we show that designing an agent in a modular fashion as a collection of subagents, each dedicated to a separate need, powerfully enhanced the agents capacity to satisfy its overall needs. We used the formalism of deep reinforcement learning to investigate a biologically relevant multi-objective task: continually maintaining homeostasis of a set of physiologic variables. We then conducted simulations in a variety of environments and compared how modular agents performed relative to standard monolithic agents (i.e., agents that aimed to satisfy all needs in an integrated manner using a single aggregate measure of success). Simulations revealed that modular agents: a) exhibited a form of exploration that was intrinsic and emergent rather than extrinsically imposed; b) were robust to changes in non-stationary environments, and c) scaled gracefully in their ability to maintain home-ostasis as the number of conflicting objectives increased. Supporting analysis suggested that the robustness to changing environments and increasing numbers of needs were due to intrinsic exploration and efficiency of representation afforded by the modular architecture. These results suggest that the normative principles by which agents have adapted to complex changing environments may also explain why humans have long been described as consisting of multiple selves. Significance StatementAdaptive agents must continually satisfy a range of distinct and possibly conflicting needs. In most models of learning, a monolithic agent tries to maximize one value that measures how well it balances its needs. However, this task is difficult when the world is changing and needs are many. Here, we considered an agent as a collection of modules each dedicated to a particular need and competing for control of action. Compared to the standard monolithic approach, modular agents were much better at maintaining homeostasis of a set of internal variables in simulated environments, both static and changing. These results suggest that having multiple selves may represent an evolved solution to the universal problem of balancing multiple needs in changing environments.

neuroscience↗