Decoding and reconstruction of surface materials from EEG

Abstract

The human visual system can easily recognize object material categories and estimate surface properties such as glossiness and smoothness. A number of psychophysical and computational studies suggest that the material perception depends on global feature statistics of the entire image at multiple processing levels. Neural representations of such global features, which is independent of precise retinotopy, may be captured even by EEG that have low spatial resolution. To test this possibility, here we measured visual evoked potentials (VEPs) for 191 natural images consisting of 20 categories of materials. We then sought to classify material categories and surface properties from the VEPs, and to reconstruct the rich phenomenological appearance of materials themselves via neural representations of global features as estimated from the VEPs. As a result, we found that material categories were correctly classified by the VEPs even at latencies of 150 ms or less. The apparent surface properties were also significantly classified within 175 ms (lightness, colorfulness, and smoothness) and after 200 ms (glossiness, hardness, and heaviness). In a subsequent reverse-correlation analysis, we further found that the VEPs at these latencies are highly correlated with low- and high-level global feature statistics of the surface images; Portilla-Simoncelli texture statistics and style information in deep convolutional neural network (dCNN), indicating that neural activities about such global features are reflected in the VEPs that enabled successful classification of materials. To demonstrate this idea more directly, we trained deep generative models (MVAE models) that reconstruct the surface image itself from the VEPs via style information (gram matrix of the dCNN output). The model successfully reconstructed realistic surface images, a part of which were nearly indistinguishable from the original images. These findings suggest that the neural representation of statistical image features, which were formed at short latencies in the visual cortex and reflected even in EEG signals, not simply enable human visual system to recognize material categories and evaluate surface properties but provides the essential basis for rich and complex phenomenological qualities of natural surfaces.