Achieving high-resolution single-cell segmentation in convoluted cancer spheroids via Bayesian optimization and deep-learning

Abstract

Accurate quantification of drug responses in 3D tumor-immune co-cultures remains challenging because complex spatial architecture and cellular heterogeneity limit the interpretability of bulk viability assays. Here, we present MorphoNavigator-3D ('Morphological Navigator in 3D';MoNa-3D), an automated framework for high-resolution, annotation-free single-cell analysis in complex 3D co-cultures. The approach integrates optimized live-cell staining, deep learning-based segmentation, and Bayesian optimization (BO) to adapt end-to-end image-analysis workflows across diverse experimental conditions. MoNa-3D was applied to clear cell renal cell carcinoma (ccRCC)-immune cell 3D-spheroid co-cultures, exposed to PI3K/mTOR pathway inhibitors and immunomodulatory compounds in a high-content imaging-based drug screen. The pipeline was used to extract multiscale phenotypic features encompassing ATP-based cell viability, morphology, nuclear remodeling, spatial dispersion, and immune infiltration. This analysis resolved distinct drug-induced phenotypes: PI3K/mTOR inhibitors promoted spheroid disintegration, nuclear enlargement, and immune exclusion, whereas immunomodulators preserved spheroid architecture and T-cell engagement. Multivariate phenotypic integration distinguished drug classes and revealed intra-class variation, including divergent spatial responses to dual PI3K/mTOR versus mTORC1 inhibition. These phenotypes were consistent with known drug mechanisms, supporting the biological interpretability of the framework. Together, these findings establish MoNa-3D as a generalizable platform for multidimensional phenotypic profiling across complex 3D multicellular systems, supporting applications in drug discovery, tumor-immune interaction studies, and precision oncology.