Pollick, F. E. | Search bioRxiv

bioRxiv2022

Patterns of Saliency and Semantic Features Distinguish Gaze of Expert and Novice Viewers of Surveillance Footage

When viewing the actions of others, we not only see patterns of body movements, but we also "see" the intentions and social relations of people, enabling us to understand the surrounding social environment. Previous research has shown that experienced forensic examiners--Closed Circuit Television (CCTV) operators--convey superior performance in identifying and predicting hostile intentions from surveillance footages than novices. However, it remains largely unknown what visual content CCTV operators actively attend to when viewing surveillance footage, and whether CCTV operators develop different strategies for active information seeking from what novices do. In this study, we conducted computational analysis for the gaze-centered stimuli captured by experienced CCTV operators and novices eye movements when they viewed the same surveillance footage. These analyses examined how low-level visual features and object-level semantic features contribute to attentive gaze patterns associated with the two groups of participants. Low-level image features were extracted by a visual saliency model, whereas object-level semantic features were extracted by a deep convolutional neural network (DCNN), AlexNet, from gaze-centered regions. We found that visual regions attended by CCTV operators versus by novices can be reliably classified by patterns of saliency features and DCNN features. Additionally, CCTV operators showed greater inter-subject correlation in attending to saliency features and DCNN features than did novices. These results suggest that the looking behavior of CCTV operators differs from novices by actively attending to different patterns of saliency and semantic features in both low-level and high-level visual processing. Expertise in selectively attending to informative features at different levels of visual hierarchy may play an important role in facilitating the efficient detection of social relationships between agents and the prediction of harmful intentions. Author SummaryImagine seeing a person walking toward another person menacingly on the street, we may instantly feel that some physical confrontation will happen in the next second. However, it remains unclear how we efficiently infer social intentions and outcomes from the observed dynamic visual input. To answer this question, CCTV experts, who have years of experience on observing social scenes and making online predictions of the action outcomes, provide a unique perspective. Here, we collected experts and novices eye movements when observing different action sequences and compared the attended visual information between groups. A saliency model was used to compare low-level visual features such as luminance and color, and a deep convolutional neural network was used to extract object-level semantic visual features. Our findings showed that experts obtained different patterns of low-level and semantic-level features in visual processing compared to novices. Thus, the expertise in selectively attending to informative features at different levels of visual hierarchy may play an important role in facilitating the efficient detection of social relationships between agents and the prediction of harmful intentions.

neuroscience↗

bioRxiv2018

Comparing fMRI inter-subject correlations between groups

Inter-subject correlation (ISC) based analysis is a conceptually simple approach to analyze functional magnetic resonance imaging (fMRI) data acquired under naturalistic stimuli such as a movie. We describe and validate the statistical approaches for comparing ISCs between two groups of subjects implemented in the ISC toolbox, which is an open source software package for ISC-based analysis of fMRI data. The approaches are based on permutation tests. We validated the approaches using five different data sets from the ICBM functional reference battery tasks. First, we created five null datasets (one for each task) by dividing the subjects into two matched groups and assumed that no group difference exists. Second, based on one null dataset, we created datasets with simulated ISC differences of varying size between the two groups. Based on the experiments with these two types of data, we recommend the use of subject-wise permutations, instead of element-wise permutations. The tests based on subject-wise permutations led to correct false positive rates. We observed that the null-distributions should be voxel-specific and not based on pooling all voxels across the brain as is typical in fMRI. This was the case even if studentized permutation tests were used. Additionally, we experimented with an fMRI dataset acquired using a dance movie stimulus for comparison of a group of adult males on the autism spectrum to a matched typically developed group. The experiment confirmed the differences between voxel-based permutation tests and global model based permutation tests.

neuroscience↗