Habituation and Goal-directed arbitration mechanisms and failures under partial observability

Abstract

AO_SCPLOWBSTRACTC_SCPLOWWe often need to make decisions under incomplete information (partial observability) and the brain manages to add the right minimal context to the decision-making. Partial observability may also be handled by other mechanisms than adding contextual experience / memory. We propose that parallel and sequential arbitration of Habituation (Model-Free, MF) and Goal-Directed (Model-Based, MB) behavior may be at play to deal with partial observability "on-the-fly", and that MB may be of different types (going beyond the MF/MB dichotomy [4]). To illustrate this, we identify, describe and model with Reinforcement Learning (RL) a behavioral anomaly (an habituation failure) occurring during the so-called Hotel Elevators Rows (HER, for short) task: a prototypical partial observation situation that can be reduced to the well studied Two and One Sequence Choice Tasks. The following hypothesis are supported by RL simulation results: (1) a parallel (semi)model-based successor representation mechanism is operative while learning to habituate which detects model-based mismatches and serves as an habituation surveillance, (2) a retrospective inference is triggered to identify the source of the habituation failure (3) a model-free mechanism can trigger model-based mechanisms in states in which habituation failed. The "failures" in the title refer to: the habituation failures that need to be monitored and surveilled (1) and to the failures that we identified in prototypical state of the art Model-Based algorithms (like DynaQ) when facing partial observability. As other research on MF/MB arbitration shows, the identification of these new mechanisms could shine light into new treatments for addiction, compulsive behavior (like compulsive checking) and understand better accidents caused by habituation behaviors.