Feedforward computational models of vision do not explain expert neural processing of visual Braille in the human visual system

Abstract

The human visual system, and in particular the Visual Word Form Area (VWFA), adapts to process letters and words, even when the stimuli do not share canonical script features, like Braille. Here we set-up to compare the organization of typical orthographic and peculiar visual scripts such as Braille in computational models. In a first experiment, we looked at how Braille letters are represented in an illiterate Convolutional Neural Network (AlexNet) and compared them to Latin alphabet and to Line Braille, a custom line-based script. We observed a predisposition of the network, pre-trained to perform object recognition, for line-based scripts. This finding suggests an initial advantage of line junctions over Braille in processing scripts likely based on typical visual computations applied to the visual world. In a second experiment, we trained two benchmark neural network architectures (AlexNet, CORnet Z) to classify words in the Latin script (literacy acquisition) and then in the Braille script (expertise acquisition). We modelled the processing of reading visual Braille and explored the networks representations at different layers. We observed clustering of features based on the visual properties of the scripts and not by the networks expertise. Unlike human participants, the representations of linguistic categories do not converge to a model of the linguistic (orthographic, phonological, semantic) properties. Overall, the lack of alignment between the visual processing of the trained computational models and neural data recorded in expert humans suggests that the fundamental processing of reading cannot be fully explained by simple feed-forward visual processing of the script, but likely relies on additional mechanisms including interactive relations between the visual and linguistic systems.