Schueler, T. | Search bioRxiv

bioRxiv2021

Realism Matters. Eye movements in VR reveal prior tool knowledge, intended interaction, and end-state comfort planning

AO_SCPLOWBSTRACTC_SCPLOWSeminal studies on human cognitive behavior have been conducted in controlled laboratory settings, demonstrating that visual attention is mainly goal-directed and allocated based on the action performed. However, it is unclear how far these results generalize to cognition in more naturalistic settings. The present study investigates active inference processes revealed by eye movements during interaction with familiar and novel tools with two levels of realism of the action affordance. We presented participants with 3D tool models that were either familiar or unfamiliar, oriented congruent or incongruent to their handedness, and asked participants to interact with them by lifting or using. Importantly, we used the same experimental design in two setups. In the first experiment, participants interacted with a VR controller in a low realism environment; in the second, they performed the task with an interaction setup that allowed differentiated hand and finger movements in a high realism environment. We investigated the differences in odds of fixations and their eccentricity towards the tool parts before action initiation. The results show that participants fixate more on the tools effector part before action initiation for the use task for unfamiliar tools. Furthermore, with more realistic action affordances, subjects fixate more on the tools handle as a function of the tools orientation, well before the action was executed. Secondly, the spatial viewing bias on the tool reveals early fixations are influenced by the task and the familiarity of the tools. In contrast, later fixations are associated with the manual planning of the interaction. In sum, the findings from the experiments suggest that fixations are made in a task-oriented way to plan the intended action well before action initiation. Further, with more realistic action affordances, fixations are made towards the proximal goal of optimally planning the grasp even though the perceived action on the tools is identical for both experimental setups. Taken together, proximal and distal goal-oriented planning is contextualized to the realism of action/interaction afforded by an environment.

neuroscience↗

bioRxiv2021

Just-in-time: gaze guidance behavior while action planning and execution in VR

Natural eye movements have primarily been studied for over-learned activities such as tea-making, sandwich-making, and hand-washing, which have a fixed sequence of associated actions. These studies indicate a sequential activation of low-level cognitive schemas facilitating task completion. However, it is unclear if these action schemas are activated in the same pattern when a task is novel and a sequence of actions must be planned in the moment. Here, we recorded gaze and body movements in a naturalistic task to study action-oriented gaze behavior. In a virtual environment, subjects moved objects on a life-size shelf to achieve a given order. In order to compel cognitive planning, we added complexity to the sorting tasks. Fixations aligned with the action onset showed gaze as tightly coupled with the action sequence, and task complexity moderately affected the proportion of fixations on the task-relevant regions. Our analysis further revealed that the gaze was allocated to action-relevant targets just in time. Planning behavior predominantly corresponded to a greater visual search for task-relevant objects before the action onset. The results support the idea that natural behavior relies very little on working memory, and humans refrain from encoding objects in the environment to plan long-term actions. Instead, they prefer just-in-time planning by searching for action-relevant items in the moment, directing their body and hand to it, monitoring the action until it is terminated, and moving on to the next action. Author SummaryEye movements in the natural environment have primarily been studied for over-learned and habitual everyday activities (tea-making, sandwich-making, hand-washing) with a fixed sequence of associated actions. These studies show eye movements correspond to a specific order of actions learned over time. In this study, we were interested in how humans plan and execute actions for tasks that do not have an inherent action sequence. To that end, we asked subjects to sort objects based on object features on a life-size shelf in a virtual environment as we recorded their eye and body movements. We investigated the general characteristics of gaze behavior while acting under natural conditions. Our paper provides a comprehensive approach to preprocess naturalistic gaze data in virtual reality. Furthermore, we provide a data-driven method of analyzing the different action-oriented functions of gaze. The results show that bereft of a predefined action sequence, humans prefer to plan only their immediate actions where eye movements are used to search for the target object to immediately act on, then to guide the hand towards it, and to monitor the action until it is terminated. Such a simplistic approach ensures that humans choose sub-optimal behavior over planning under sustained cognitive load.

neuroscience↗