Learning the structure of the world: The adaptive nature of state-space and action representations in multi-stage decision-making

Abstract

State-space and action representations form the building blocks of decision-making processes in the brain; states map external cues to the current situation of the agent whereas actions provide the set of motor commands from which the agent can choose to achieve specific goals. Although these factors differ across environments, it is not currently known whether or how accurately state and action representations are acquired by the agent because previous experiments have typically provided this information a priori through instruction or pre-training. Here we show that, in the absence of such a priori knowledge, state and action representations adapt to reflect the structure of the world. We used a sequential decision-making task in rats in which they were required to pass through multiple states before reaching the goal, and for which the number of states and how they map onto external cues were not known a priori. We found that, early in training, animals selected actions as if the task was not sequential and outcomes were the immediate consequence of the most proximal action. During the course of training, however, rats recovered the true structure of the environment and made decisions based on the expanded state-space, reflecting the multiple stages of the task. We found a similar pattern with actions; early in training animals only considered the execution of single actions whereas, after training, they created useful action sequences that expanded the set of available actions. We conclude that the profile of choices shows a gradual shift from simple representations of actions and states to more complex structures compatible with the structure of the world.