Explainable Multimodal Prognostic Modeling Uncovers New Clinically Relevant Pathological Features in a Multi-Institutional Pancreatic Neuroendocrine Tumor Cohort
Although many patients with a well-differentiated pancreatic neuroendocrine tumor (PanNET) are cured after surgical resection, it is hard to predict outcome in these patients. Current prognostic assessment relies largely on Ki-67 proliferation index, tumor stage, and established adverse pathologic features, such as vascular and perineural invasion. However, individually, these variables incompletely explain why patients with similar clinicopathologic profiles sometimes experience markedly different outcomes. To address this challenge, we assembled three independent cohorts with a surgically resected PanNET. In total, we included 1,438 patients with approximately 30,000 histology images, and approximately 12,000 person-years of follow-up. We separately modeled structured clinicopathologic information with Cox regression and tumor-centered pathology images with deep learning on an internal cohort treated at Johns Hopkins Hospital and performed multimodal continuous-time survival modeling to reveal which clinical and pathologic features are most important in predicting patient outcome. Across Cox models of censored time to recurrence-free survival, a text-based clinicopathological model that incorporated known prognosticators achieved a C-index of approximately 0.87, the image-based histology model approximately 0.80, and decision-level late multimodal fusion approximately 0.88 on the internal test set. The clinical model demonstrated strong prognostic discrimination using established risk factors, mainly relying on the TNM stage and Ki-67 index. Primary tumor histology retained substantial prognostic signal, with analysis restricted to tumor-centered regions to reduce reliance on unrelated tissues that might indirectly indicate disease extent. Interpretation of high-attention image tiles revealed regions of importance in the primary tumor, including recognized adverse morphologic features such as the leading edge of the tumor and foci of vascular or perineural invasion. Unexpectedly, intratumoral and peritumoral stroma accounted for nearly 50% of the model's high-attention regions, suggesting that the stroma may yield equal prognostic information to neoplastic cells in recurrence-prediction models. To test generalizability, we further evaluated the trained models in independent cohorts from two institutions with different demographics, where discrimination varied by site (mean late-fusion C-indices of 0.864 and 0.723). Together with the improved discrimination achieved through late fusion, our findings suggest that current clinicopathologic variables can be modeled to form reliable prognosticators, and that routine hematoxylin and eosin (H&E) stained slides of the primary tumor contain additional prognostically important information not fully captured by conventional clinicopathologic assessment, such that together, the two sources provide complementary information.