Evaluating Large Language Models for Gene-to-Phenotype Mapping: The Critical Role of Full-Text Database Access

Abstract

Transformer-based large language models (LLMs) have demonstrated significant potential in the biological and medical fields due to their ability to effectively learn from large-scale, diverse datasets and perform a wide range of downstream tasks. However, LLMs are limited by issues such as information processing inaccuracies and data confabulation. These limitations hinder their utility for literature searches and other tasks requiring accurate and comprehensive extraction of information from extensive scientific literature. In this study, we evaluated the performance of various LLMs in accurately retrieving peer-reviewed literature and mapping correlations between 198 genes and six phenotypes: bone formation, cartilage formation, fibrosis, cell proliferation, tendon formation, and ligament formation. Our analysis included three types of models. First, standard transformer-based LLMs (ChatGPT-4o and Gemini 1.5 Pro). Second, specialized LLMs with dedicated custom databases containing peer-reviewed articles (SciSpace and ScholarAI). Third, specialized LLMs without dedicated databases (PubMedGPT and ScholarGPT). Using human-curated gene-to-phenotype mappings as the ground truth, we found that specialized LLMs with dedicated databases achieved the highest accuracy (>80%) in gene-to-phenotype mapping. Additionally, these models were able to provide relevant peer-reviewed publications supporting each gene-to-phenotype correlation. These findings underscore the importance of database augmentation and specialization in enhancing the reliability and utility of LLMs for biomedical research applications.