Leibbrandt, R. | Search bioRxiv

bioRxiv2024

Combining Unity with machine vision to create low latency, flexible, and simple virtual realities

O_LIIn recent years, virtual reality arenas have become increasingly popular for quantifying visual behaviors. By using the actions of a constrained animal to control the visual scenery, the animal is provided the perception of moving through a simulated environment. As the animal is constrained in space, this allows detailed behavioral quantification. Additionally, as the world is generally computer-generated this allows for mechanistic quantification of visual triggers of behavior. C_LIO_LIWe created a novel virtual arena combining machine vision with the gaming engine Unity. For tethered flight, we enhanced an existing multi-modal virtual reality arena, MultiMoVR (Kaushik et al., 2020) but tracked hoverfly wing movements using DeepLabCut-live (DLC-live, Kane et al., 2020). For trackball experiments, we recorded the motion of a ball that a tethered crab was walking on using FicTrac (Moore et al., 2014). In both cases, real-time tracking was interfaced with Unity to control the movement of the tethered animals avatars in the virtual world. We developed a user-friendly Unity Editor interface, CAVE, to simplify experimental design and data storage without the need for coding. C_LIO_LIWe show that both the DLC-live-Unity and the FicTrac-Unity configurations close the feedback loop effectively with small delays, less than 50 ms. Our FicTrac-Unity integration highlighted the importance of closed-loop feedback by reducing behavioral artifacts exhibited by the crabs in open-loop scenarios. We show that Eristalis tenax hoverflies, using the DLC-live-Unity integration, navigate towards flowers. The effectiveness of our CAVE interface is shown by implementing experimental sequencing control based on avatar proximity to virtual structures. C_LIO_LIOur results show that combining Unity with machine vision tools such as DLC-live and FicTrac provides an easy and flexible virtual reality (VR) environment that can be readily adjusted to new experiments and species. This can be implemented programmatically in Unity, or by using our new tool CAVE, which allows users to design and implement new experiments without programming in code. We provide resources for replicating experiments and our interface CAVE via GitHub, together with user manuals and instruction videos, for sharing with the wider scientific community. C_LI

animal behavior and cognition↗

bioRxiv2022

Male hoverflies (Eristalis tenax) do not use heuristic rules based on retinal size and speed to initiate indoor pursuits of artificial targets

The ability to visualize small moving objects is vital for the survival of many animals, as these could represent predators or prey. For example, predatory insects, including dragonflies, robber flies and killer flies, perform elegant, high-speed pursuits of both biological and artificial targets. Many non-predatory insects, including male hoverflies and blowflies, also pursue targets during territorial or courtship interactions. To date, most hoverfly pursuits were studied outdoors. To investigate naturalistic hoverfly (Eristalis tenax) pursuits under more controlled settings, we constructed an indoor arena that was large enough to encourage naturalistic behavior. We presented artificial beads of different sizes, moving at different speeds, and filmed pursuits with two cameras, allowing subsequent 3D reconstruction of the hoverfly and bead position as a function of time. We show that male E. tenax hoverflies are unlikely to use strict heuristic rules based on angular size or speed to determine when to start pursuit, at least in our indoor setting. We found that hoverflies pursued faster beads when the trajectory involved flying downwards towards the bead. Furthermore, we show that target pursuit behavior can be broken down into two stages. In the first stage the hoverfly attempts to rapidly decreases the distance to the target by intercepting it at high speed. During the second stage the hoverflys forward speed is correlated with the speed of the bead, so that the hoverfly remains close, but without catching it. This may be similar to dragonfly shadowing behavior, previously coined motion camouflage.

animal behavior and cognition↗