HandMol: Coupling WebXR, AI and HCI technologies for Immersive, Natural, Collaborative and Inclusive Molecular Modeling

Abstract

Except for isolated developments and specific software extensions, molecular graphics and modeling have historically been stuck at flat screens for visualization, mouse operations for molecular manipulation, menus and command line interfaces for controls, and single-user interfaces that only allow collaboration by streaming video hence limited to just sharing the view of the user operating the software. We demonstrate here how various technologies are ripe enough to enable much more fluent, immersive and natural human-computer interactions that in turn facilitate collaboration between human users, using affordable hardware through the internet and without even installing any specialized programs. For this, we introduce HandMol, a web app that exploits (i) WebXR for molecular visualization and manipulation in virtual reality, (ii) speech recognition coupled to a large language model to pass commands orally, (iii) speech synthesis for auditory feedback, (iv) WebRTC to communicate multiple instances of the tool without even requiring a server, and (v) external APIs to flexibly account for molecular mechanics, exemplified here with an endpoint running an AMBER forcefield for protein and nucleic acids and another running a DFT-trained neural network, ANI-2x, to allow exploration of conformation and some simple reactivity at high speed and accuracy. We show example applications to situations from daily work and education in chemistry and structural biology where HandMol can provide an advantage over traditional software: exploring and explaining molecular conformations and reactivity, docking and undocking small molecules into/out of protein pockets, threading molecules through nanopores, preparing systems for molecular simulations and for protein design, etc. We also present a brief study showing how users, even with limited or even no experience in VR, can significantly benefit from these kinds of technologies. As a draft prototype for the moment, HandMol is made available free of charge and without registration at https://go.epfl.ch/handmol, in (optional but greatly appreciated) exchange for feedback on usability and on features expected for this kind of tools.