Multisensory integration across reference frames with additive feed-forward networks

Abstract

The integration of multiple sensory inputs is essential for human perception and action in uncertain environments. This process includes reference frame transformations as different sensory signals are encoded in different coordinate systems. Studies have shown multisensory integration in humans is consistent with Bayesian optimal inference. However, neural mechanisms underlying this process are still debated. Different population coding models have been proposed to implement probabilistic inference. This includes a recent suggestion that explicit divisive normalization accounts for empirical principles of multisensory integration. However, whether and how divisive operations are implemented in the brain is not well understood. Indeed, all existing models suffer from the curse of dimensionality and thus fail to scale to real-world problems. Here, we propose an alternative model for multisensory integration that approximates Bayesian inference: a multilayer-feedforward neural network of multisensory integration (MSI) across different reference frames trained on the analytical Bayesian solution. This model displays all empirical principles of multisensory integration and produces similar behavior to that reported in Ventral intraparietal (VIP) neurons in the brain. The model achieved this without a neatly organized and regular connectivity structure between contributing neurons, such as required by explicit divisive normalization. Overall, we show that simple feedforward networks of purely additive units can approximate optimal inference across different reference frames through parallel computing principles. This suggests that it is not necessary for the brain to use explicit divisive normalization to achieve multisensory integration. Significance StatementThis research presents an alternative model to divisive normalization models of multisensory integration in the brain. Our study demonstrates that a feed-forward neural network can achieve optimal multisensory integration across different reference frames without explicitly implementing divisive operations, challenging the long-held assumption that such operations are necessary for multisensory integration. The model displays all the empirical principles of multisensory integration, producing similar behavior to that reported in Ventral intraparietal (VIP) neurons in the brain. This work offers profound insights into the putative neural computations underlying multisensory processing.