Search bioRxiv⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.1101/2024.01.24.576956

The effect of semantic content on the perception of audiovisual movieclips

Kesoglou, A. M.·Mikellidou, K.
View original source ↗Export metadata ↓

Abstract

Our brain is skilled with the ability to perceive and process multimodal stimuli. This process known as crossmodal perceptual integration, has been in the research spotlight for a long time, providing evidence for the integration of information coming from different modalities. Prior research mostly utilized pictures and focused on the semantic content of a single sound or word. The present study aims to investigate crossmodal perceptual integration in realistic conditions using short movieclips (1500ms) and auditory meaningful three-word sentences to evaluate target detection in terms of accuracy and response times. Two experimental tasks were developed using PsychoPy, where participants had to indicate whether a target (noun for Experiment 1, verb for Experiment 2) was present or absent. In trials without a target, target-related information was always present, either through one of the two senses (vision or audition; incongruent condition) or through both senses (congruent condition). We observed superior performance when the target was absent generally (Mexp1 =93.9%, SDexp1 =0.04, Mexp2 =84.2%, SDexp2 =0.182) compared to when it was present (Mexp1 =82.3%, SDexp1 =0.143, Mexp2 =73.7%, SDexp2 =0.184). Moreover, superior performance was noted in incongruent target-related movieclips, which significantly decreased during congruent target-related movieclips. In Experiment 1, we observed that in the audio condition when the target-related word was a noun, participant performance was superior compared to when it was a verb (M=99.4% vs. M=86.7%; tincVerb vs. incNoun =-8.428, p=.001). In Experiment 2, the judgment scores were similarly high in incongruent movieclips and significantly lower in congruent ones regardless of whether the target-related information presented was a verb or noun. The present results provide evidence regarding the role of complexity of semantics, and especially the diverse role verbs and nouns could play in crossmodal perceptual integration in more realistic situations. Our findings can enrich the content of learning techniques, as well as the design of AI models, by taking advantage of the supporting role of semantic audiovisual information, while taking into consideration the potential confusion that the complexity of semantic information can induce to perception experience.

Explore related subjects

neurosciencePreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Kesoglou, A. M., Mikellidou, K.. 2024-01-25. The effect of semantic content on the perception of audiovisual movieclips. https://doi.org/10.1101/2024.01.24.576956

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Attention Across Scales: From Individual Variation to Social Hierarchies and Brain Networks in Semi-Free-Ranging Macaques

Attention is a fundamental brain function supporting perception, decision-making, and social behavior, and its dysfunction profoundly impairs daily life. It is both dynamic and stable, varying across observations and individuals, changing across the lifespan, and being shaped by social and environmental experience. Yet capturing this complexity remains a central challenge in neuroscience. Here, we integrated longitudinal behavioral assessments of semi-free-ranging macaques living in naturalistic social groups with resting-state fMRI. We quantified performance across days, ages, and social hierarchies and related it to intrinsic brain organization. Distinct attentional phenotypes emerged, including individuals with reduced attentional control. Performance followed an inverted-U lifespan trajectory, improving from childhood to adulthood before declining. Social status modulated attentional performance. Critically, nonlinear lifespan trajectories and associations with individual attentional differences were most clearly expressed in frontoparietal connectivity. Together, these findings reveal how sustained attention is organized across scales, providing a biological framework for its individual diversity, social modulation, and neural basis.

neuroscience↗
bioRxiv2026

Decoding natural scenes from patterned optogenetic responses in mouse visual cortex

A central challenge in developing visual cortical prostheses is to determine how visual stimuli should be transformed into effective patterns of cortical stimulation. Although advances in stimulation technologies, including optogenetics, provide increasingly precise control over cortical activity, it remains unclear whether artificially evoked activity can reproduce the information content of naturally evoked visual representations. Here we establish a quantitative framework for evaluating visual encoding strategies by decoding cortical responses evoked by natural vision and patterned optogenetic stimulation. We developed a novel dual-modal paradigm in awake mice to bridge the gap between endogenous photostimulation and artificial network driving. By co-expressing the high-performance calcium indicator GCaMP6s and the red-shifted, ultra-sensitive opsin rsChRmine-oScarlet in the primary visual cortex (V1), we successfully translated dynamic natural movie frames into patterned, spatiotemporal optogenetic stimulation. Quantitative comparisons of macro-scale dynamics demonstrated that this patterned optogenetic injection evokes cortical states highly comparable and representationally aligned with those driven by actual visual photostimulation. To systematically evaluate the fidelity of these responses, we developed STAR, a deep learning model featuring spatial and temporal attention mechanisms, and successfully reconstructed the frames of natural movies from V1 signals under both experimental modalities. Collectively, our results demonstrate that complex sensory information can be both naturally encoded and synthetically injected into V1 circuits with high decoding fidelity. This work provides an empirical and computational proof-of-concept for intelligent, closed-loop biomimetic encoders, establishing a robust framework for next-generation cortical visual neuroprostheses and bidirectional brain-machine interfaces.

neuroscience↗
bioRxiv2026

Why Is Spontaneous Blink Timing Informative? An Adaptive Scheduling Perspective

Spontaneous eye blinks have long been linked to cognitive processing, yet how task demands shape blink timing and its relationship to behavioral performance remains unclear. We examined spontaneous blink behavior in 576 adults performing two variants of the Continuous Performance Task (CPT). Blink occurrence and timing were most strongly modulated by the experimental condition in the more demanding CPT-AX task, whereas their association with response time was stronger in the CPT-X task, where more consistent blink timing predicted faster responses. This dissociation suggests that task structure changes not only blink behavior but also the behavioral relevance of blink timing. These findings are consistent with an adaptive scheduling account of spontaneous blinking and provide a conceptual framework for understanding when and why blink timing contains chronometric information about ongoing cognition.

neuroscience↗