Neural mechanisms for reward-modulated vector learning and navigation: from social insects to embodied agents

Abstract

Despite their small size, insect brains are able to produce robust and efficient navigation in complex environments. Specifically in social insects, such as ants and bees, these navigational capabilities are guided by orientation directing vectors generated by a process called path integration. During this process, they integrate compass and odometric cues to estimate their current location as a vector, called home vector for guiding them back home on a straight path. They further acquire and retrieve path integration-based vector memories anchored globally to the nest or visual landmarks. Although existing computational models reproduced similar behaviors, they largely neglected evidence for possible neural substrates underlying the generated behavior. Therefore, we present here a model of neural mechanisms in a modular closed-loop control - enabling vector navigation in embodied agents. The model consists of a path integration mechanism, reward-modulated global and local vector learning, random search, and action selection. The path integration mechanism integrates compass and odometric cues to compute a vectorial representation of the agents current location as neural activity patterns in circular arrays. A reward-modulated learning rule enables the acquisition of vector memories by associating the local food reward with the path integration state. A motor output is computed based on the combination of vector memories and random exploration. In sim-ulation, we show that the neural mechanisms enable robust homing and localization, even in the presence of external sensory noise. The proposed learning rules lead to goal-directed navigation and route formation performed under realistic conditions. This provides an explanation for, how view-based navigational strategies are guided by path integration. Consequently, we provide a novel approach for vector learning and navigation in a simulated embodied agent linking behavioral observations to their possible underlying neural substrates.\n\nAuthor SummaryDesert ants survive under harsh conditions by foraging for food in temperatures over 60{degrees} C. In this extreme environment, they cannot, like other ants, use pheromones to track their long-distance journeys back to their nests. Instead they apply a computation called path integration, which involves integrating skylight compass and odometric stimuli to estimate its current position. Path integration is not only used to return safely to their nests, but also helps in learning so-called vector memories. Such memories are sufficient to produce goal-directed and landmark-guided navigation in social insects. How can small insect brains generate such complex behaviors? Computational models are often useful for studying behavior and their underlying control mechanisms. Here we present a novel computational framework for the acquisition and expression of vector memories based on path integration. It consists of multiple neural networks and a reward-based learning rule, where vectors are represented by the activity patterns of circular arrays. Our model not only reproduces goal-directed navigation and route formation in a simulated agent, but also offers predictions about neural implementations. Taken together, we believe that it demonstrates the first complete model of vector-guided navigation linking observed behaviors of navigating social insects to their possible underlying neural mechanisms.