Recurrent Neural Network Exploration Strategies During Reinforcement Learning Depend on Network Capacity

Abstract

Artificial neural networks constitute simplified computational models of neural circuits that might help understand how the biological brain solves and represents complex tasks. Previous research revealed that recurrent neural networks (RNNs) with 48 hidden units show human-level performance in restless four-armed bandit tasks but differ from humans with respect to the task strategy employed. Here we systematically examined the impact of network capacity (no. of hidden units) on computational mechanisms and performance. Computational modeling was applied to investigate and compare network behavior between capacity levels as well as between RNNs and human learners. Using a task frequently employed in human cognitive neuroscience work as well as in animal systems neuroscience work, we show that high-capacity networks displayed increased directed exploration and attenuated random exploration relative to low-capacity networks. RNNs with 576 hidden units approached "human-like" exploration strategies, but the overall switch rate and the level of perseveration still deviated from human learners. In the context of the resource-rational framework, which posits a trade-off between reward and policy complexity, human learners may devote more resources to solving the task, albeit without performance benefits over RNNs. Taken together, this work reveals the importance of network capacity on exploration strategies during reinforcement learning and therefore contributes to the goal of building neural networks that behave "human-like" to possibly gain insights into computational mechanisms in human brains.