ChatGPT identifies gender disparities in scientific peer review

Verharen, J. P. H.

doi:10.1101/2023.07.18.549552

Home/Archive/bioRxiv

bioRxiv · 10.1101/2023.07.18.549552

ChatGPT identifies gender disparities in scientific peer review

Verharen, J. P. H.

View original source ↗Export metadata ↓

Abstract

The peer review process is a critical step in ensuring the quality of scientific research. However, its subjectivity has raised concerns. To investigate this issue, I examined over 500 publicly available peer review reports from 200 published neuroscience papers in 2022-2023. OpenAIs generative artificial intelligence ChatGPT was used to analyze language use in these reports. It demonstrated superior performance compared to traditional lexicon- and rule-based language models. As expected, most reviews for these published papers were seen as favorable by ChatGPT (89.8% of reviews), and language use was mostly polite (99.8% of reviews). However, this analysis also demonstrated high levels of variability in how each reviewer scored the same paper, indicating the presence of subjectivity in the peer review process. The results further revealed that female first authors received less polite reviews than their male peers, indicating a gender bias in reviewing. In addition, published papers with a female senior author received more favorable reviews than papers with a male senior author, for which I discuss potential causes. Together, this study highlights the potential of generative artificial intelligence in performing natural language processing of specialized scientific texts. As a proof of concept, I show that ChatGPT can identify areas of concern in scientific peer review, underscoring the importance of transparent peer review in studying equitability in scientific publishing.

Explore related subjects

scientific communication and education Preprint

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Verharen, J. P. H.. 2023-07-19. ChatGPT identifies gender disparities in scientific peer review. https://doi.org/10.1101/2023.07.18.549552

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Evaluating Large Language Models as Tools to Navigate Researchers in Rapidly Evolving Research Landscapes: A Case Study in Cancer Drug Response Prediction

Large Language Models (LLMs) have emerged as promising tools for assisting researchers in automating and accelerating the synthesis of literature reviews. However, their reliability is a significant concern due to issues like factual inaccuracies and hallucinations. The key question is whether LLMs can reliably provide comprehensive, up-to-date overviews and analyses. This study evaluates the performance of three leading LLMs (OpenAI's ChatGPT, Google's Gemini, and DeepSeek) on the complex task of generating a comprehensive survey paper on deep learning for cancer Drug Response Prediction (DRP). By testing both standard and Deep Research (DR) / Deep Think (DT) modes of LLMs with prompts of varying detail, this paper assesses key academic dimensions, including reference management, content quality, and analytical depth. Key findings reveal that while DR modes of LLMs significantly improve reliability by eliminating hallucinations, performance variations exist across models and prompts. A trade-off between reference quantity and integration quality was observed, and even the best-performing models lacked the analytical depth of human experts, often requiring extensive human supervision. The study concludes that LLMs currently serve as powerful assistive tools but still cannot replace the critical validation and synthesis provided by human researchers. Choosing the best LLM to use depends on the task in hand, while several strategies can be implemented to improve the produced output.

scientific communication and education↗

bioRxiv2024

Sci-comm "behind the scenes": Gendered narratives of scientific outreach activities in the life sciences

Science communication (sci-comm) encompasses activities that promote scientific literacy, inform policy, and inspire future scientists. Despite its value, sci-comm is often considered less pretigious than research and other internal academic activities. In Australia, scientific organisations (e.g., learned societies) rely on members to perform a range of sci-comm activities, typically unpaid. In this pilot study, we surveyed 88 Australian life sciences organisations to examine who performs sci-comm and why. Most respondents were early-career women employed in university research positions. Participants largely agreed that organisational sci-comm brought limited career benefits and was often viewed as feminised or "care work" relative to their research roles. Yet, respondents also cited personal and professional gains and most wished to continue in such roles. Our findings suggest "invisible" sci-comm performed for scientific organisations is disproportionately undertaken by women at early career stages, with implications for career progression and gender equity in STEM.

scientific communication and education↗

bioRxiv2024

Career Navigator: An online platform to streamline professional development and career education for graduate bioscientists

Graduate professional development is a highly dynamic enterprise that prepares graduate students for personal and career success in a variety of fields, including the biosciences. National policies, funding awards, and institutional programs have generated myriad tools and services for graduate bioscience students, including new learning resources, events, connections to prospective employers, and opportunities to strengthen academic and professional portfolios. These interventions are welcome and have done much to enhance graduate bioscience training, but they may also be overwhelming for trainees. To streamline professional development and career education information for the bioscience graduate students at our institution, we tested a model where we built a centralized web portal of career development resources. Here we present our strategy and best practices for website design. We show data that students preferred a centralized online portal over other forms of resource communication; that programming, paired communication and environmental factors (e.g. remote learning and work as in the COVID-19 pandemic) combined to increase sustained engagement with the site; and that harnessing website analytics is an effective way to measure site utilization and generate insights on programming and resource development. This data, in turn, fits into broader priorities to evaluate interventions in graduate bioscience education.

scientific communication and education↗