Multimodal histopathologic models stratify hormone receptor-positive early breast cancer
For patients with hormone receptor-positive, early breast cancer without HER2 amplification, multigene expression assays including Oncotype DX (R) recurrence score (RS) have been clinically validated to identify patients who stand to derive added benefit from adjuvant cytotoxic chemotherapy. However, cost and turnaround time have limited its global adoption despite recommendation by practice guidelines. We investigated if routinely available hematoxylin and eosin (H&E)-stained pathology slides could act as a surrogate triaging data substrate by predicting RS using machine learning methods. We trained and validated a multimodal transformer model, Orpheus, using 6,203 patients across three independent cohorts, taking both H&E images and their corresponding synoptic text reports as input. We showed accurate inference of recurrence score from whole-slide images (r = 0.63 (95% C.I. 0.58 - 0.68); n = 1,029), the raw text of their corresponding reports (r = 0.58 (95% C.I. 0.51 - 0.64); n = 972), and their combination (r = 0.68 (95% C.I. 0.64 - 0.73); n = 964) as measured by Pearsons correlation. To predict high-risk disease (RS>25), our model achieved an area under the receiver operating characteristic curve (AUROC) of 0.89 (95% C.I. 0.83 - 0.94), and area under the precision recall curve (AUPRC) of 0.64 (95% C.I. 0.60 - 0.82), compared to 0.49 (95% C.I. 0.36 - 0.64) for an existing nomogram based on clinical and pathologic features. Moreover, our model generalizes well to external international cohorts, effectively identifying recurrence risk (r = 0.61, p < 10-4, n = 452; r = 0.60, p < 10-4, n = 575) and high-risk status (AUROC = 0.80, p < 10-4, AUPRC = 0.68, p < 10-4, n = 452; AUROC = 0.83, p < 10-4, AUPRC = 0.73, p < 10-4, n = 575) from whole-slide images. Probing the biologic underpinnings of the model decisions uncovered tumor cell size heterogeneity, immune cell infiltration, a proliferative transcription program, and stromal fraction as correlates of higher-risk predictions. We conclude that at an operating point of 94.4% precision and 33.3% recall, this model could help increase global adoption and shorten lag between resection and adjuvant therapy.