Katti, H. | Search bioRxiv

bioRxiv2020

A separable neural code in monkey IT enables perfect CAPTCHA decoding

Reading distorted letters is easy for us but so challenging for machine vision that it is used on websites as CAPTCHAs (Completely Automated Public Turing test to tell Computers and Humans Apart). How does our brain solve this problem? One solution is to have neurons invariant to letter distortions but selective for letter combinations. Another is for neurons to separately encode letter distortions and combinations. Here, we provide evidence for the latter using neural recordings in the monkey inferior temporal (IT) cortex. Neurons encoded letter distortions as a product of letter and distortion tuning, and letter combinations as a sum of letters. These rules were sufficient for perfect CAPTCHA decoding and were also present in neural networks trained for word recognition. Taken together, our findings suggest that a separable neural code enables efficient letter recognition.

neuroscience↗

bioRxiv2019

Do deep neural networks see the way we do?

Deep neural networks have revolutionized computer vision, and their object representations match coarsely with the brain. As a result, it is widely believed that any fine scale differences between deep networks and brains can be fixed with increased training data or minor changes in architecture. But what if there are qualitative differences between brains and deep networks? Do deep networks even see the way we do? To answer this question, we chose a deep neural network optimized for object recognition and asked whether it exhibits well-known perceptual and neural phenomena despite not being explicitly trained to do so. To our surprise, many phenomena were present in the network, including the Thatcher effect, mirror confusion, Webers law, relative size, multiple object normalization and sparse coding along multiple dimensions. However, some perceptual phenomena were notably absent, including processing of 3D shape, patterns on surfaces, occlusion, natural parts and a global advantage. Our results elucidate the computational challenges of vision by showing that learning to recognize objects suffices to produce some perceptual phenomena but not others and reveal the perceptual properties that could be incorporated into deep networks to improve their performance.

neuroscience↗