Biology subjects

Sawayama, M.

Publications and source records attributed to Sawayama, M..

12 recordsLinked to original sources

bioRxiv2026

Subspace Image Synthesis for Style-Invariant Object Recognition

Human-like behavior in visual models does not necessarily imply human-like internal computation. When a model recognizes objects robustly across changes in visual style, what internal computations support this behavior, and which are relevant to human perception? We addressed this question by pruning style-robust neural networks into candidate submodels corresponding to two computational strategies. The first retained channels whose responses were stable across style changes for fixed content, yielding a Content Stability Score (CSS)-based submodel. The second retained channels whose responses were stable across content changes for fixed style, yielding a Style Stability Score (SSS)-based submodel. Across ImageNet, Stylized ImageNet, and model-versus-human benchmarks, the two submodel families showed comparable object-recognition performance. We then used their exclusive feature subspaces to generate matched images with Gram-matrix-based neural style transfer and asked human observers which image was more similar in visual style to a reference image. Images generated from the CSS-based submodel were selected significantly more often than chance, even though CSS preferentially retained channels whose individual activation patterns were relatively insensitive to style. These results suggest that representations preserving content across style changes are more consistent with human judgments of visual style. More broadly, they demonstrate a framework for isolating candidate computational subspaces from large artificial models and testing their perceptual relevance through image synthesis.

neuroscience↗

bioRxiv2026

Task-general brain state dynamics relate to trial-to-trial performance and individual response strategy

Behavior fluctuates from moment to moment, yet individuals also show characteristic response strategies across tasks. Whether these two levels of variation relate to common brain dynamics is unknown. Using an inside-out approach across 10 cognitive tasks during fMRI, we identified recurring brain states and related their dynamics to behavior within and between individuals. Default mode network (DMN) dominant and action mode network (AMN) dominant states occurred in all tasks. Within individuals, momentary state identity (the state occupied at a given moment) tracked trial-to-trial performance. The AMN-dominant state was associated with slower, more variable responses across tasks, whereas its relationship to accuracy depended on the task. Across individuals, switching between the two states, but not time spent in either, showed cross-task consistency, and frequent switchers showed faster, less accurate responding. Thus, distinct aspects of the same task-general brain state dynamics, momentary state identity and state switching, relate to behavior at both levels.

neuroscience↗

bioRxiv2025

Probing the content of semantic representations in body-selective regions

The extrastriate body area (EBA) has been known to selectively respond to human body parts, but recent work suggests it may encode richer scene semantics. However, it remains unclear what aspects of natural scenes constitute the semantic representation in the EBA. Here we address this question by analyzing the relationship between object co-occurrence in natural scene captions and EBA responses predicted by caption-based encoding models. This revealed object category pairs associated with EBA activity, leading to the hypothesis that EBA encodes the speed of human body motion implied in scenes. A subsequent behavioral experiment and correlation analyses confirmed this, showing strong correlations between EBA responses and implied motion ratings. In addition, the representation of implied body motion extends into the fusiform body area (FBA). Variance partitioning further showed that while body-related features such as the number of people and body size also contributed, implied motion uniquely explained the largest portion of EBA and FBA responses. Overall, semantic representations in body-selective regions are jointly shaped by multiple body-related features, with implied motion as the primary contributor. Our co-occurrence-based approach, combined with brain-to-model mapping methods, offers a novel, interpretable framework for understanding high-level visual representations in natural scene perception.

neuroscience↗

bioRxiv2025

Revisiting Color Efficient Coding through Material Perception

An essential objective of early visual processing is to handle the redundant inputs from natural environments efficiently. For instance, cone signals from natural environments are highly correlated across cone types, indicating channel redundancy. Early visual processing transforms the signals into color and luminance information, such as principle component analysis, and thus achieves efficient and decorrelated representations of natural scenes [1-3]. Building on these findings, previous research has investigated the effect of color on visual tasks such as object recognition using grayscale conversion, which separates luminance from color [4, 5]. However, recent work suggests that when focusing on object materials, color and luminance remain highly redundant due to complex optical properties [6, 7]. Although this finding indicates that there may be a more efficient decomposition of signals, the specific algorithms remain unknown. This study derives that a classic computer graphics algorithm, the median cut [8], offers a novel approach to enhance visual processing efficiency while capturing diagnostic features to separate material from object geometry information (Fig. 1a and 1b). Human behavioral experiments show that color reduction based on the algorithm disturbs material classification while preserving object recognition (Fig. 1c). These findings suggest that object geometric structures are available only from low-bit information. Finally, considering material information can be estimated from summary statistics of image sub-spaces, this study suggests an efficient decomposition of input color images (Fig. 1d). O_FIG O_LINKSMALLFIG WIDTH=200 HEIGHT=146 SRC="FIGDIR/small/644715v1_fig1.gif" ALT="Figure 1"> View larger version (46K): org.highwire.dtl.DTLVardef@1ae46baorg.highwire.dtl.DTLVardef@a56667org.highwire.dtl.DTLVardef@e61973org.highwire.dtl.DTLVardef@90ffbe_HPS_FORMAT_FIGEXP M_FIG O_FLOATNOFig. 1.C_FLOATNO (a) Demonstration of canceling the material appearance difference (wet and dry) of the same stone image while maintaining its geometric structure using a color reduction algorithm called the median cut. The reduced color image consists of four color points in a RGB space, corresponding to 2-bits index color. (b) Schematic visualization of the median cut. The input color space is divided with a plane according to the median of the cumulative histogram of each axis. Each point in the left figure represents a pixel of the RGB color image. The divisions by the plane define the subspaces, and all points within each subspace are replaced with a single mean value. The right figure shows an example with 2-bits. According to the histogram of each subspace, the intensity order of the mean values is maintained consistent with the original distribution (Supplementary Fig. S1). (c) Results of human behavioral experiments. One hundred participants engaged in ensemble 2AFC tasks involving three material classifications and one object classification: Glossy vs. Matte, Wet vs. Dry, Translucent vs. Opaque for material classification, and Animal vs. House for object classification. Participants were divided into four groups based on the color condition to ensure that no participant saw the same original image. Violin plots in each panel illustrate the accuracy distribution for these image conditions, with the mean accuracy indicated by red dots. The horizontal dashed line marks the chance level accuracy. The results demonstrate that color reduction decreases material classification accuracy but does not affect object classification accuracy. (d) Color image decomposition by the median cut with Gaussian ellipsoid projection. While the median cut preserves the geometric structure of an image, the residual material information can be efficiently represented by the summary statistics of the Gaussian ellipsoid within each subspace, i.e., the mean and standard deviation. C_FIG

neuroscience↗

bioRxiv2024

Probing the Link Between Vision and Language in Material Perception

We can visually discriminate and recognize a wide range of materials. Meanwhile, we use language to express our subjective understanding of visual input and communicate relevant information about the materials. Here, we investigate the relationship between visual judgment and language expression in material perception to understand how visual features relate to semantic representations. We use deep generative networks to construct an expandable image space to systematically create materials of well-defined and ambiguous categories. From such a space, we sampled diverse stimuli and compared the representations of materials from two behavioral tasks: visual material similarity judgments and free-form verbal descriptions. Our findings reveal a moderate but significant correlation between vision and language on a categorical level. However, analyzing the representations with an unsupervised alignment method, we discover structural differences that arise at the image-to-image level, especially among materials morphed between known categories. Moreover, visual judgments exhibit more individual differences compared to verbal descriptions. Our results show that while verbal descriptions capture material qualities on the coarse level, they may not fully convey the visual features that characterize the materials optical properties. Analyzing the image representation of materials obtained from various pre-trained data-rich deep neural networks, we find that human visual judgments similarity structures align more closely with those of the text-guided visual-semantic model than purely vision-based models. Our findings suggest that while semantic representations facilitate material categorization, non-semantic visual features also play a significant role in discriminating materials at a finer level. This work illustrates the need to consider the vision-language relationship in building a comprehensive model for material perception. Moreover, we propose a novel framework for quantitatively evaluating the alignment and misalignment between representations from different modalities, leveraging information from human behaviors and computational models.

neuroscience↗

bioRxiv2023

Decoding time-resolved neural representations of orientation ensemble perception

The visual system is capable of computing summary statistics of multiple visual elements at a glance. While numerous studies have demonstrated ensemble perception across different visual features, the timing at which the visual system forms an ensemble representation remains unclear. This is mainly because most previous studies did not uncover time-resolved neural representations during ensemble perception. Here we used orientation ensemble discrimination tasks along with EEG recordings to decode orientation representations over time while human observers discriminated an average of multiple orientations. We observed alternation in orientation representations over time, with stronger neural representations of the individual elements in a set of orientations, but we did not observe significantly strong representations of the average orientation at any time points. We also found that a cumulative average of the orientation representations over approximately 500 ms converged toward the average orientation. More importantly, this cumulative orientation representation significantly correlated with the individual difference in the perceived average orientation. These findings suggest that the visual system gradually extracts an orientation ensemble, which may be represented as a cumulative average of transient orientation signals, through selective processing of a subset of multiple orientations that occurs over several hundred milliseconds.

neuroscience↗

bioRxiv2023

Early-stage neural processing reveals distinctive responses to saliency-differentiated colors in various types of color vision

Individuals with minority color vision phenotypes have been reported to exhibit enhanced color discrimination and color recognition, which deviate from predictions based on their receptoral sensitivities. However, the specific mechanisms underlying this enhanced sensitivity remain unclear. In this study, we examined the commonality and diversity of neural activity between typical and anomalous trichromats in response to differences in color saliency. Electroencephalography was recorded during an oddball task, in which participants discriminated each of two target stimuli, blue-green and red, from a green standard stimulus. The chromaticity of the stimulus was identical across participants, whereas the relationship of saliency between the target stimuli was expected to be reversed between color vision types. The spatiotemporal dynamics of event-related potentials (ERPs) were analyzed using cluster-based permutation tests. Typical trichromats demonstrated faster behavioral and neural responses to the more salient red target stimulus, with pronounced neural activity spreading from the occipital to the parietal regions in the comparison between the target stimuli. Anomalous trichromats also exhibited similar temporal patterns toward the more salient target stimulus when each target stimulus was compared with the green standard stimulus, indicating comparable processing toward saliency across color vision types. Although a similarity was observed, neither behavioral nor neural responses in anomalous trichromats reflected saliency contrast differences. In addition, a comparative analysis of ERPs between color vision types did not reveal any distinct differences in either target stimulus. Given the large variations in color sensitivity in individuals with anomalous trichromacy, further investigation is required to understand the detailed neural processing in individuals with various color vision types.

neuroscience↗

bioRxiv2022

Translucency perception emerges in deep generativerepresentations for natural image synthesis

Humans constantly assess the appearance of materials to plan actions, such as stepping on icy roads without slipping. Visual inference of materials is challenging because a given material can appear dramatically different in various scenes. This problem especially stands out for translucent materials (e.g., skin, soap), whose appearance strongly depends on lighting, geometry, and viewpoint. Despite this, humans can still distinguish between different materials, and it remains unsolved how to systematically discover visual features pertinent to material inference. Here, we developed an unsupervised style-based image generation model to identify perceptually relevant dimensions for material appearances from natural photographs. We find the model can synthesize images of convincing material appearances. Importantly, without supervision, human-understandable scene attributes, including objects shape, material, and body color, spontaneously emerge in the models layer-wise latent space in a scale-specific manner. Crucially, the middle-layers of the latent space selectively encode translucency features correlated with perception, suggesting that translucent impressions are established in mid-to-low spatial scale features. Our findings indicate that humans may use the scale-specific statistical structure of natural images to efficiently represent material properties across contexts, and our approach is widely applicable in discovering perceptually relevant features from complex stimuli for many visual inference tasks.

neuroscience↗

bioRxiv2021

Crystal or Jelly? Effect of Color on the Perception of Translucent Materials with Photographs of Real-world Objects

Translucent materials are ubiquitous in nature (e.g. teeth, food, wax), but our understanding of translucency perception is limited. Previous work in translucency perception has mainly used monochromatic rendered images as stimuli, which are restricted by their diversity and realism. Here, we measure translucency perception with photographs of real-world objects. Specifically, we use three behavior tasks: binary classification of "translucent" versus "opaque", semantic attribute rating of perceptual qualities (see-throughness, glossiness, softness, glow and density), and material categorization. Two different groups of observers finish the three tasks with color or grayscale images. We find that observers agreements depend on the physical material properties of the objects such that translucent materials generate more inter-observer disagreements. Further, there are more disagreements among observers in the grayscale condition in comparison to that in color condition. We also discover that converting images to grayscale substantially affects the distributions of attribute ratings for some images. Furthermore, ratings of see-throughness, glossiness, and glow could predict individual observers binary classification of images in both grayscale and color conditions. Lastly, converting images to grayscale alters the perceived material categories for some images such that observers tend to misjudge images of food as non-food and vice versa. Our result demonstrates color is informative about material property estimation and recognition. Meanwhile, our analysis shows mid-level semantic estimation of material attributes might be closely related to high-level material recognition. We also discuss individual differences in our results and highlight the importance of such consideration in material perception.

neuroscience↗

bioRxiv2019

Visual discrimination of optical material properties: a large-scale study

Complex visual processing involved in perceiving the object materials can be better elucidated by taking a variety of research approaches. Sharing stimulus and response data is an effective strategy to make the results of different studies directly comparable and can assist researchers with different backgrounds to jump into the field. Here, we constructed a database containing several sets of material images annotated with visual discrimination performance. We created the material images using physically-based computer graphics techniques and conducted psychophysical experiments with them in both laboratory and crowdsourcing settings. The observers task was to discriminate materials on one of six dimensions (gloss contrast, gloss distinctness-of-image, translucent vs. opaque, metal vs. plastic, metal vs. glass, and glossy vs. painted). The illumination consistency and object geometry were also varied. We used a non-verbal procedure (an oddity task) applicable for diverse use-cases such as cross-cultural, cross-species, clinical, or developmental studies. Results showed that the material discrimination depended on the illuminations and geometries and that the ability to discriminate the spatial consistency of specular highlights in glossiness perception showed larger individual differences than in other tasks. In addition, analysis of visual features showed that the parameters of higher-order color texture statistics can partially, but not completely, explain task performance. The results obtained through crowdsourcing were highly correlated with those obtained in the laboratory, suggesting that our database can be used even when the experimental conditions are not strictly controlled in the laboratory. Several projects using our dataset are underway.

neuroscience↗

bioRxiv2019

A computational mechanism for seeing dynamic deformation

Human observers perceptually discriminate the dynamic deformation of materials in the real world. However, the psychophysical and neural mechanisms responsible for the perception of dynamic deformation have not been fully elucidated. By using a deforming bar as the stimulus, we showed that the spatial frequency of deformation was a critical determinant of deformation perception. Simulating the response of direction-selective units (i.e., MT pattern motion cells) to stimuli, we found that the perception of dynamic deformation was well explained by assuming a higher-order mechanism monitoring the spatial pattern of direction responses. Our model with the higher-order mechanism also successfully explained the appearance of a visual illusion wherein a static bar apparently deforms against a tilted drifting grating. In particular, it was the lower spatial frequencies in this pattern that strongly contributed to the deformation perception. Finally, by manipulating the luminance of the static bar, we observed that the mechanism for the illusory deformation was more sensitive to luminance than contrast cues. Significance StatementFrom the psychophysical and computational points of view, the present study tried to answer the question, "how do human observers see deformation?". In the psychophysical experiment, we used a clip wherein a bar dynamically deformed. We also tested the illusory deformation of a bar, which was caused by tilted drifting grating, because it was unclear whether the illusory deformation could be described by our model. In the computational analysis, in order to explain psychophysical data for deformation perception, it was necessary to assume an additional unit monitoring the spatial pattern of direction responses of MT cells that were sensitive to local image motion.

neuroscience↗

bioRxiv2019

Haptic metameric textures

Humans sense spatial patterns through their eyes and hands. Past studies have revealed differences (as well as similarities) between vison and touch in texture processing (e.g., eye is good at detecting texture boundaries, while hand can discriminate subtle texture differences), but the underlying computational differences remains poorly understood. Here we transcribed various textures as surface relief patterns by 3D-printing, and analyzed the tactile discrimination performance regarding the sensitivity to image statistics. The results suggest that touch is sensitive to texture differences in lower-order statistics (e.g., statistics of local amplitude spectrum), while may not to those in the higher-order statistics (e.g., joint statistics of local orientations). In agreement with this, we found that pairs of synthesized textures differing only in higher-order statistics were nearly indiscriminable (metameric) by touch, while easily discriminable by vision. Our findings show that touch and vision sense spatial information using different and complementary computational strategies.

neuroscience↗