Self-guided search in immersive VR reveals active hierarchical planning

Abstract

Despite the ubiquitous nature of visual search in our daily lives, our knowledge of the underlying mechanisms is almost entirely derived from studies using simple, synthetic stimuli briefly flashed on a computer monitor. In real life, however, visual search occurs in three-dimensional environments over much longer timescales and involves the dynamic coordination of attention, memory, and action planning, generating novel predictions that cannot be tested with traditional approaches. Here, we examined how self-guided visual search unfolds under naturalistic conditions that extend over space and time. We asked participants (N=80) to find specific objects (e.g., a green lamp, an orange shelf) in an immersive, visually rich virtual reality environment using range-limited teleportation (2 m steps). Consistent with our hypotheses, we found that self-guided search engaged a hierarchical information-seeking policy characterized at each stage by recurring plan-and-execute cycles paired with subgoal-specific attentional priorities. Subgoal priorities progressed from wayfinding to identification of clusters of target-similar objects, and then to specific target features. These search priorities progressively narrowed the search space through coordinated eye, head, and body movements that recurred throughout the task. We conclude that naturalistic search is hierarchical, built from sequential subgoals with shifting attentional priorities that single-level tasks cannot capture. Moreover, immersive VR with eye-head-body tracking dissociates motor planning from execution moments and reveals how looking content shifts with each task subgoal while the person moves through space, something desktop tasks cannot capture.