Visual Image Reconstruction from Human Brain Activity using Linear Image Decoders plus Nonlinear Noise Suppression

Abstract

In recent years, substantial strides have been made in the field of visual image reconstruction, particularly in its capacity to generate high-quality visual representations from human brain activity while considering semantic information. This advancement not only enables the recreation of visual content but also provides valuable insights into the intricate processes occurring within high-order functional brain regions, contributing to a deeper understanding of brain function. However, considering fusion semantics in reconstructing visual images from brain activity involves semantic-to-image guide reconstruction and may ignore underlying neural computational mechanisms, which does not represent true reconstruction from brain activity. In response to this limitation, our study introduces a novel approach that combines linear mapping with nonlinear noise suppression to reconstruct visual images perceived by subjects based on their brain activity patterns. The primary challenge associated with linear mapping lies in its susceptibility to noise interference. To address this issue, we leverage a flexible denoised deep convolutional neural network, which can suppress noise from linear mapping. Our investigation encompasses linear mapping as well as the training of shallow and deep autoencoder denoised neural networks, including a pre-trained, state-of-the-art denoised neural network. The outcome of our study reveals that combining linear image decoding with nonlinear noise reduction significantly enhances the quality of reconstructed images from human brain activity. This suggests that our methodology holds promise for decoding intricate perceptual experiences directly from brain activity patterns without semantic information. Moreover, the model has strong neural explanatory power because it shares structural and functional similarities with the visual brain.