Suvarna, A. | Search bioRxiv

bioRxiv2023

DeltaAI: Semi-Autonomous Tissue Grossing Measurements and Recommendations using Neural Radiance Fields for Rapid, Complete Intraoperative Histological Assessment of Tumor Margins

Mohs Micrographic Surgery (MMS) aims to excise cutaneous cancer with real-time margin analysis. However, manual tissue grossing and analysis can be inefficient, so we propose DeltaAI, a novel workflow that utilizes Neural Radiance Fields (NeRF) to enable rapid tissue grossing and generate a 3D model in an augmented reality (AR) environment. In our study, we captured 30-second videos of 17 MMS specimens using a photogrammetry turntable and cellphone camera. Preprocessing the tissues with segmentation models, we created a dataset of 923, 360-degree-view, images per video (17 videos). Using COLMAP, we estimated poses for sparse tissue reconstructions and trained the NeRF model for 3D volumetric tissue renderings. The results demonstrated that DeltaAI generated more accurate and complete 360-degree, 3D tissue renderings compared to previous models, while also achieving significantly faster runtimes. Our proposed semi-autonomous NeRF-based workflow has the potential to enhance the speed of MMS specimen processing, measurement, report generation, and margin assessment. It can inform real-time grossing decisions, automate the export of electronic health record data, and facilitate time-efficient and complete cancer excisions. Moreover, DeltaAI can contribute to the wider adoption of AI technology in clinical settings by improving tissue modeling for manual grossing.

pathology↗

bioRxiv2022

Assessment of Emerging Pretraining Strategies in Interpretable Multimodal Deep Learning for Cancer Prognostication

Deep learning models have demonstrated the remarkable ability to infer cancer patient prognosis from molecular and anatomic pathology information. Studies in recent years have demonstrated that leveraging information from complementary multimodal data can improve prognostication, further illustrating the potential utility of such methods. Model interpretation is crucial for facilitating the clinical adoption of deep learning methods by fostering practitioner understanding and trust in the technology. However, while prior works have presented novel multimodal neural network architectures as means to improve prognostication performance, these approaches: 1) do not comprehensively leverage biological and histomorphological relationships and 2) make use of emerging strategies to "pretrain" models (i.e., train models on a slightly orthogonal dataset/modeling objective) which may aid prognostication by reducing the amount of information required for achieving optimal performance. Here, we develop an interpretable multimodal modeling framework that combines DNA methylation, gene expression, and histopathology (i.e., tissue slides) data, and we compare the performances of crossmodal pretraining, contrastive learning, and transfer learning versus the standard procedure in this context. Our models outperform the existing state-of-the-art method (average 11.54% C-index increase), and baseline clinically driven models. Our results demonstrate that the selection of pretraining strategies is crucial for obtaining highly accurate prognostication models, even more so than devising an innovative model architecture, and further emphasize the all-important role of the tumor microenvironment on disease progression.

pathology↗