Unifying sparse coding, predictive coding, and divisive normalization

Abstract

Sparse coding, predictive coding and divisive normalization have each been found to be principles that underlie the function of neural circuits in many parts of the brain, supported by substantial experimental evidence. However, the connections between these related principles are still poorly understood. Sparse coding and predictive coding can be reconciled into a learning framework with predictive structure and sparse responses, termed as sparse/predictive coding. However, how sparse/predictive coding (a learning model) is connected with divisive normalization (not a learning model) is still not well investigated. In this paper, we show how sparse coding, predictive coding, and divisive normalization can be described within a unified framework, and illustrate this explicitly within the context of a two-layer neural learning model of sparse/predictive coding. This two-layer model is constructed in a way that implements sparse coding with a network structure that is constructed by implementing predictive coding. We demonstrate how a homeostatic function that regulates neural responses in the model can shape the nonlinearity of neural responses in a way that replicates different forms of divisive normalization. Simulations show that the model can learn simple cells in the primary visual cortex with the property of contrast saturation, which has previously been explained by divisive normalization. In summary, the study demonstrates that the three principles of sparse coding, predictive coding, and divisive normalization can be connected to provide a learning framework based on biophysical properties, such as Hebbian learning and homeostasis, and this framework incorporates both learning and more diverse response nonlinearities observed experimentally. This framework has the potential to also be used to explain how the brain learns to integrate input from different sensory modalities. Author SummaryComputational principles are often proposed to reveal the neural computations underlying brain functions. In the past three decades, sparse coding, predictive coding and divisive normalization have been three influential computational principles that have much success in different areas of neuroscience. Sparse coding offers insights into how the brain learns meaningful associations based on the hypothesis of brain being very efficient. With an emphasis on prediction, predictive coding provides an appealing hierarchical framework of only sending prediction errors to higher layers. Divisive normalization is a mathematical equation designed to account for the extensive nonlinearities in the brain. All these three computational principles along their variants have greatly improved our understanding of the underlying mechanism of the brain. Though connection between sparse and predictive coding has been studied previously, how sparse/predictive coding is connected to a seemingly different principle, divisive normalization, to provide a unified understanding of the brain is still unclear. In this paper, we show that sparse coding, predictive coding and divisive normalization can be connected from first principles. We propose a learning framework that is based on the hypothesis of efficiency, implemented with a predictive structure and displays response nonlinearities of divisive normalization. This framework can be potentially examined and used in a broader context such as multi-sensory integration.