Search bioRxiv⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.1101/2024.11.24.625099

Intraoperative Augmented Reality for Vitreoretinal Surgery using Edge Computing

Ye, R. Z.·Iezzi, R.
View original source ↗Export metadata ↓

Abstract

PurposeAugmented reality (AR) may allow vitreoretinal surgeons to leverage microscope-integrated digital imaging systems to analyze and highlight key retinal anatomic features in real-time, possibly improving safety and precision during surgery. By employing convolutional neural networks (CNNs) for retina vessel segmentation, a retinal coordinate system can be created that allows pre-operative images of capillary non-perfusion or retinal breaks to be digitally aligned and overlayed upon the surgical field in real-time. Such technology may be useful in assuring thorough laser treatment of capillary non-perfusion or in using pre-operative optical coherence tomography (OCT) to guide macular surgery when microscope-integrated OCT (MIOCT) is not available. MethodsThis study is a retrospective analysis involving the development and testing of a novel image registration algorithm for vitreoretinal surgery. Fifteen anonymized cases of pars plana vitrectomy with epiretinal membrane peeling, along with corresponding preoperative fundus photographs and optical coherence tomography (OCT) images, were retrospectively collected from the Mayo Clinic database. We developed a TPU (Tensor-Processing Unit)-accelerated CNN for semantic segmentation of retinal vessels from fundus photographs and subsequent real-time image registration in surgical video streams. An iterative patch-wise cross-correlation (IPCC) algorithm was developed for image registration, with a focus on optimizing processing speeds and maintaining high spatial accuracy. The primary outcomes measured were processing speed in frames per second (FPS) and the spatial accuracy of image registration, quantified by the Dice coefficient between registered and manually aligned images. ResultsWhen deployed on an Edge TPU, the CNN model combined with our image registration algorithm processed video streams at a rate of 14 FPS, which is superior to processing rates achieved on other standard hardware configurations. The IPCC algorithm efficiently aligned pre-operative and intraoperative images, showing high accuracy in comparison to manual registration. ConclusionThis study demonstrates the feasibility of using TPU-accelerated CNNs for enhanced AR in vitreoretinal surgery.

Explore related subjects

bioengineeringPreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Ye, R. Z., Iezzi, R.. 2024-11-26. Intraoperative Augmented Reality for Vitreoretinal Surgery using Edge Computing. https://doi.org/10.1101/2024.11.24.625099

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Single-Component Photo-Crosslinked Lipoic Acid Functionalized Mucin Hydrogels: Dynamic Mucus Mimetics for Antiviral Protection

Native mucus protects epithelial surfaces through viscoelastic networks of disulfide-linked mucins. However, commercially available mucins often fail to reconstitute comparable mechanical and barrier properties, while synthetic and hybrid mucus mimetics require polymeric crosslinkers that may alter mucus-like composition. Inspired by the native architecture, we directly functionalized porcine gastric and bovine submaxillary mucins with -lipoic acid to develop a single-component, photo-crosslinkable hydrogels. Upon UV irradiation (365 nm), the modified mucins rapidly form disulfide-crosslinked networks without added photo-initiator/ polymeric crosslinkers, two orders of magnitude faster as compared with PEG-SH approaches. The resulting mucin-lipoic acid (MUC-LA) hydrogels exhibit mucus-relevant properties, including tunable viscoelasticity, redox responsiveness, self-healing, and size-dependent restrictions on nanoparticle transport. In transwell-based infection models, MUC-LA hydrogels (2-3%) mimicking the healthy airway mucus (G' {approx} 10-20 Pa) suppress epithelial infection by SARS-CoV-2 and influenza A virus. Crosslinked MUC-LA provides greater protection against SARS-CoV-2 than the non-crosslinked, whereas both formulations suppress influenza A infection, suggesting distinct contributions from network formation and mucin-associated interactions. These findings establish a dynamic, mucus-mimetic platform integrating rapid photo-gelation, controllable mechanics, selective transport, and antiviral barrier functionality. MUC-LA hydrogels offer opportunities to investigate mucosal transport and host-pathogen interactions, with potential applications in drug delivery, and as biomaterials.

bioengineering↗
bioRxiv2026

Population-level emulation enables fast Bayesian calibration of cardiac digital twins

Purpose: Patient-specific cardiac digital twins have the potential to guide individualized therapy, but generating these in a routine clinical workflow remains challenging. This is especially true when the twins must carry quantified uncertainty: Bayesian calibration requires thousands of simulations per patient, and the emulators that reduce this cost are typically trained one patient at a time. We aimed to train a single emulator across a patient cohort, so that a new patient can be calibrated in minutes without new simulations. Methods: We trained a population-level Gaussian process emulator whose inputs included a 12-dimensional latent encoding of the clinical features of each patient, so that a single emulator represents the whole cohort. We compared spending simulation budgets on resampling the real cohort with augmenting the cohort with digital chimeras, synthetic patients interpolated between real patients. Seven parameters of a closed-loop heart and circulation model were calibrated by Markov chain Monte Carlo to ten clinical biomarkers in 26 patients with left bundle branch block, each using an emulator trained without that patient, and the calibrated posteriors were compared against the full simulator. Results: A single emulator, trained without the held-out patient, calibrated the digital twin of each patient in 59.7-63.0 s per patient, versus 15 h with the simulator. Resampling the real cohort was the more efficient training strategy, bringing every output to R2 greater than or equal to 0.9, which augmentation did not achieve for the same simulation budget. For held-out patients, both strategies performed comparably (median leave-one-out normalized root-mean-square error 11.0% vs. 10.7%; p = 0.745), and error rose with distance from the cohort centroid (Spearman rho = +0.406). The calibrated models showed no cohort-wide bias (median standardized residual -0.12 against the clinical targets), and 95% predictive intervals contained 78.5% of targets, with the largest misfit in systolic blood pressure. Conclusion: Removing the need for new simulations for every new patient is a meaningful step toward uncertainty-aware cardiac digital twins that fit within a routine clinical workflow. Because the simulator, surrogate and sampler are interchangeable, the approach used here can be applied to other (cardiac) models and cohorts.

bioengineering↗
bioRxiv2026

Multi-Structure Segmentation of Type B Aortic Dissection from CT Images with U-Net

Type B aortic dissection (TBAD) is a life-threatening condition in which a tear in the intimal layer separates the aortic wall into a true and a false lumen. Assessment of the lumen, the intimal flap, and false-lumen thrombosis from CT images is important for risk analysis and treatment planning. This work applied U-Net-based deep neural networks to the automatic segmentation of these structures in TBAD. We collected 46 CT scans from 20 patients, with voxel-wise annotations of three structures: the aortic lumen, the intimal flap, and the thrombus. We explored four models: two convolutional networks, nnU-Net 2D and 3D, and two transformer-based architectures, TransUNet 2D and 3D. We observed that (1) for both nnU-Net and TransUNet, the 3D models outperformed their 2D counterparts on all three structures. The difference was small for the aortic lumen but large for the thin flap and the thrombus. The best model, nnU-Net 3D, reached a DSC of 93.05% for the lumen, 50.22% for the flap, and 69.94% for the thrombus. (2) The intimal flap was the most difficult structure for all models. Even the best 3D models showed large thrombus boundary errors in a few scans. These results provide a baseline for developing more advanced models for accurate segmentation of these structures in TBAD.

bioengineering↗