Search bioRxiv⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.1101/2023.10.24.563804

AI-powered smart glasses for sensing and recognition of human-robot walking environments

Rossos, D.·Mihailidis, A.·Laschowski, B.
View original source ↗Export metadata ↓

Abstract

Environment sensing and recognition can allow humans and robots to dynamically adapt to different walking terrains. However, fast and accurate visual perception is challenging, especially on embedded devices with limited computational resources. The purpose of this study was to develop a novel pair of AI-powered smart glasses for onboard sensing and recognition of human-robot walking environments with high accuracy and low latency. We used a Raspberry Pi Pico microcontroller and an ArduCam HM0360 low-power camera, both of which interface with the eyeglass frames using 3D-printed mounts that we custom-designed. We trained and optimized a lightweight and efficient convolutional neural network using a MobileNetV1 backbone to classify the walking terrain as either indoor surfaces, outdoor surfaces (grass and dirt), or outdoor surfaces (paved) using over 62,500 egocentric images that we adapted and manually labelled from the Meta Ego4D dataset. We then compiled and deployed our deep learning model using TensorFlow Lite Micro and post-training quantization to create a minimized byte array model of size 0.31MB. Our system was able to accurately predict complex walking environments with 93.6% classification accuracy and had an embedded inference speed of 1.5 seconds during online experiments using the integrated camera and microcontroller. Our AI-powered smart glasses open new opportunities for visual perception of human-robot walking environments where embedded inference and a low form factor is required. Future research will focus on improving the onboard inference speed and miniaturization of the mechatronic components.

Explore related subjects

bioengineeringPreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Rossos, D., Mihailidis, A., Laschowski, B.. 2023-10-29. AI-powered smart glasses for sensing and recognition of human-robot walking environments. https://doi.org/10.1101/2023.10.24.563804

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Single-Component Photo-Crosslinked Lipoic Acid Functionalized Mucin Hydrogels: Dynamic Mucus Mimetics for Antiviral Protection

Native mucus protects epithelial surfaces through viscoelastic networks of disulfide-linked mucins. However, commercially available mucins often fail to reconstitute comparable mechanical and barrier properties, while synthetic and hybrid mucus mimetics require polymeric crosslinkers that may alter mucus-like composition. Inspired by the native architecture, we directly functionalized porcine gastric and bovine submaxillary mucins with -lipoic acid to develop a single-component, photo-crosslinkable hydrogels. Upon UV irradiation (365 nm), the modified mucins rapidly form disulfide-crosslinked networks without added photo-initiator/ polymeric crosslinkers, two orders of magnitude faster as compared with PEG-SH approaches. The resulting mucin-lipoic acid (MUC-LA) hydrogels exhibit mucus-relevant properties, including tunable viscoelasticity, redox responsiveness, self-healing, and size-dependent restrictions on nanoparticle transport. In transwell-based infection models, MUC-LA hydrogels (2-3%) mimicking the healthy airway mucus (G' {approx} 10-20 Pa) suppress epithelial infection by SARS-CoV-2 and influenza A virus. Crosslinked MUC-LA provides greater protection against SARS-CoV-2 than the non-crosslinked, whereas both formulations suppress influenza A infection, suggesting distinct contributions from network formation and mucin-associated interactions. These findings establish a dynamic, mucus-mimetic platform integrating rapid photo-gelation, controllable mechanics, selective transport, and antiviral barrier functionality. MUC-LA hydrogels offer opportunities to investigate mucosal transport and host-pathogen interactions, with potential applications in drug delivery, and as biomaterials.

bioengineering↗
bioRxiv2026

Population-level emulation enables fast Bayesian calibration of cardiac digital twins

Purpose: Patient-specific cardiac digital twins have the potential to guide individualized therapy, but generating these in a routine clinical workflow remains challenging. This is especially true when the twins must carry quantified uncertainty: Bayesian calibration requires thousands of simulations per patient, and the emulators that reduce this cost are typically trained one patient at a time. We aimed to train a single emulator across a patient cohort, so that a new patient can be calibrated in minutes without new simulations. Methods: We trained a population-level Gaussian process emulator whose inputs included a 12-dimensional latent encoding of the clinical features of each patient, so that a single emulator represents the whole cohort. We compared spending simulation budgets on resampling the real cohort with augmenting the cohort with digital chimeras, synthetic patients interpolated between real patients. Seven parameters of a closed-loop heart and circulation model were calibrated by Markov chain Monte Carlo to ten clinical biomarkers in 26 patients with left bundle branch block, each using an emulator trained without that patient, and the calibrated posteriors were compared against the full simulator. Results: A single emulator, trained without the held-out patient, calibrated the digital twin of each patient in 59.7-63.0 s per patient, versus 15 h with the simulator. Resampling the real cohort was the more efficient training strategy, bringing every output to R2 greater than or equal to 0.9, which augmentation did not achieve for the same simulation budget. For held-out patients, both strategies performed comparably (median leave-one-out normalized root-mean-square error 11.0% vs. 10.7%; p = 0.745), and error rose with distance from the cohort centroid (Spearman rho = +0.406). The calibrated models showed no cohort-wide bias (median standardized residual -0.12 against the clinical targets), and 95% predictive intervals contained 78.5% of targets, with the largest misfit in systolic blood pressure. Conclusion: Removing the need for new simulations for every new patient is a meaningful step toward uncertainty-aware cardiac digital twins that fit within a routine clinical workflow. Because the simulator, surrogate and sampler are interchangeable, the approach used here can be applied to other (cardiac) models and cohorts.

bioengineering↗
bioRxiv2026

Multi-Structure Segmentation of Type B Aortic Dissection from CT Images with U-Net

Type B aortic dissection (TBAD) is a life-threatening condition in which a tear in the intimal layer separates the aortic wall into a true and a false lumen. Assessment of the lumen, the intimal flap, and false-lumen thrombosis from CT images is important for risk analysis and treatment planning. This work applied U-Net-based deep neural networks to the automatic segmentation of these structures in TBAD. We collected 46 CT scans from 20 patients, with voxel-wise annotations of three structures: the aortic lumen, the intimal flap, and the thrombus. We explored four models: two convolutional networks, nnU-Net 2D and 3D, and two transformer-based architectures, TransUNet 2D and 3D. We observed that (1) for both nnU-Net and TransUNet, the 3D models outperformed their 2D counterparts on all three structures. The difference was small for the aortic lumen but large for the thin flap and the thrombus. The best model, nnU-Net 3D, reached a DSC of 93.05% for the lumen, 50.22% for the flap, and 69.94% for the thrombus. (2) The intimal flap was the most difficult structure for all models. Even the best 3D models showed large thrombus boundary errors in a few scans. These results provide a baseline for developing more advanced models for accurate segmentation of these structures in TBAD.

bioengineering↗