Prasanna, P. | Search bioRxiv

bioRxiv2024

Performance of GPT-4 on the American College of Radiology In-Service Examination

ObjectivesNo study has evaluated the ability of ChatGPT-4 to answer image-rich diagnostic radiology board exam questions or assessed for model drift in GPT-4s image interpretation abilities. In our study we evaluate GPT-4s performance on the American College of Radiology (ACR) 2022 Diagnostic Radiology In-Training Examination (DXIT). MethodsQuestions were sequentially input into GPT-4 with a standardized prompt. Each answer was recorded and overall accuracy was calculated, as was logic-adjusted accuracy, and accuracy on image-based questions. This experiment was repeated several months later to assess for model drift. ResultsGPT-4 achieved 58.5% overall accuracy, lower than the PGY-3 average (61.9%) but higher than the PGY-2 average (52.8%). Adjusted accuracy was 52.8%. GPT-4 showed significantly higher (p = 0.012) confidence for correct answers (87.1%) compared to incorrect (84.0%). Performance on image-based questions was notably poorer (p < 0.001) at 45.4% compared to text-only questions (80.0%), with adjusted accuracy for image questions of 36.4%. When the questions were repeated, GPT-4 chose a different answer 25.5% of the time and there was a small but insignificant decrease in accuracy. DiscussionGPT-4 performed between PGY-2 and PGY-3 levels on the 2022 DXIT, but significantly poorer on image-based questions, and with large variability in answer choices across time points. This study underscores the potential and risks of using minimally-prompted general AI models in interpreting radiologic images as a diagnostic tool. Implementers of general AI radiology systems should exercise caution given the possibility of spurious yet confident responses.

bioinformatics↗

bioRxiv2022

Deep learning-based approach for the characterization and quantification of histopathology in mouse models of colitis

Inflammatory bowel disease (IBD) is a chronic immune-mediated disease of the gastrointestinal tract. While therapies exist, response can be limited within the patient population. As such, researchers have studied mouse models of colitis to further understand its pathogenesis and identify new treatment targets. Although bench methods like flow cytometry and RNA-sequencing can characterize immune responses with single-cell resolution, whole murine colon specimens are processed at once. Given the simultaneous presence of colonic regions that are involved or uninvolved with abnormal histology, processing whole colons may lead to a loss of spatial context. Detecting these regions in hematoxylin and eosin (H&E)-stained colonic tissues offers the downstream potential of quantifying immune populations in areas with and without disease involvement by immunohistochemistry on serially sectioned slides. This could provide a complementary, spatially-aware approach to further characterize populations identified by other methods. However, detection of such regions requires expert interpretation by pathologists and is a tedious process that may be difficult to perform consistently across experiments. To this end, we have trained a deep learning model to detect Involved and Uninvolved regions from H&E-stained colonic slides across controls and three mouse models of colitis - the dextran sodium sulfate (DSS) chemical induction model, the recently established intestinal epithelium-specific, inducible Klf5{Delta}IND (Villin-CreERT2;Klf5fl/fl) genetic model, and one that combines both induction methods. The trained classifier allows for extraction of Involved colonic regions across mice to cluster and identify histological classes. Here, we show that quantification of Involved and Uninvolved image patch classes in swiss rolls of colonic specimens can be utilized to train a linear determinant analysis classifier to distinguish between mouse models. Such an approach has the potential for revealing histological links and improving synergy between various colitis mouse model studies to identify new therapeutic targets and pathophysiological mechanisms.

immunology↗