Search bioRxiv⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.1101/2020.11.26.400028

How to best evaluate applications for junior fellowships? Remote evaluation and face-to-face panel meetings compared

Bieri, M.·Roser, K.·Heyard, R.·Egger, M.
View original source ↗Export metadata ↓

Abstract

ObjectivesTo test a simplified evaluation of fellowship proposals by analyzing the agreement of funding decisions with the official evaluation, and to examine the use of a lottery-based decision for proposals of similar quality. DesignThe study involved 134 junior fellowship proposals (Postdoc.Mobility). The official method used two panel reviewers who independently scored the application, followed by triage and discussion of selected applications in a panel. Very competitive/uncompetitive proposals were directly funded/rejected without discussion. The simplified procedure used the scores of the two panel members, with or without the score of an additional, third expert. Both methods could further use a lottery to decide on applications of similar quality close to the funding threshold. The same funding rate was applied, and the agreement between the two methods analyzed. SettingSwiss National Science Foundation (SNSF). ParticipantsPostdoc.Mobility panel reviewers and additional expert reviewers. Primary outcome measurePer cent agreement between the simplified and official evaluation method with 95% confidence intervals (95% CI). ResultsThe simplified procedure based on three reviews agreed in 80.6% (95% CI 73.9-87.3) with the official funding outcome. The agreement was 86.6% (95% CI 80.8-92.4) when using the two reviews of the panel members. The agreement between the two methods was lower for the group of applications discussed in the panel (64.2% and 73.1%, respectively), and higher for directly funded/rejected applications (range 96.7% to 100%). The lottery was used in eight (6.0%) of 134 applications (official method), 19 (14.2%) applications (simplified, three reviewers) and 23 (17.2%) applications (simplified, two reviewers). With the simplified procedure, evaluation costs could have been halved and 31 hours of meeting time saved for the two 2019 calls. ConclusionAgreement between the two methods was high. The simplified procedure could represent a viable evaluation method for the Postdoc.Mobility early career instrument at the SNSF.

Explore related subjects

scientific communication and educationPreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Bieri, M., Roser, K., Heyard, R., Egger, M.. 2020-11-26. How to best evaluate applications for junior fellowships? Remote evaluation and face-to-face panel meetings compared. https://doi.org/10.1101/2020.11.26.400028

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Evaluating Large Language Models as Tools to Navigate Researchers in Rapidly Evolving Research Landscapes: A Case Study in Cancer Drug Response Prediction

Large Language Models (LLMs) have emerged as promising tools for assisting researchers in automating and accelerating the synthesis of literature reviews. However, their reliability is a significant concern due to issues like factual inaccuracies and hallucinations. The key question is whether LLMs can reliably provide comprehensive, up-to-date overviews and analyses. This study evaluates the performance of three leading LLMs (OpenAI's ChatGPT, Google's Gemini, and DeepSeek) on the complex task of generating a comprehensive survey paper on deep learning for cancer Drug Response Prediction (DRP). By testing both standard and Deep Research (DR) / Deep Think (DT) modes of LLMs with prompts of varying detail, this paper assesses key academic dimensions, including reference management, content quality, and analytical depth. Key findings reveal that while DR modes of LLMs significantly improve reliability by eliminating hallucinations, performance variations exist across models and prompts. A trade-off between reference quantity and integration quality was observed, and even the best-performing models lacked the analytical depth of human experts, often requiring extensive human supervision. The study concludes that LLMs currently serve as powerful assistive tools but still cannot replace the critical validation and synthesis provided by human researchers. Choosing the best LLM to use depends on the task in hand, while several strategies can be implemented to improve the produced output.

scientific communication and education↗
bioRxiv2021

Inferring livestock movement networks from archived data to support infectious disease control in developing countries

The use of network analysis to support livestock disease control in low middle-income countries (LMICs) has historically been hampered by the cost of generating empirical data in the absence of animal movement recording schemes. To fill this gap, methods which exploit freely available demographic and archived molecular data can be used to generate livestock networks based on gravity and phylogeographic modelling techniques, respectively. However, questions remain on the performance of these methods in capturing the topology of empirical networks. Here, we compare output from these network methodologies to a network constructed from either empirical data or randomly generated data. To facilitate this comparison, the spread of infectious diseases was simulated, it is this evaluation that demonstrates their potential utility to inform robust livestock disease control strategies. The molecular network was the closest approximation to the empirical network, both in relation to topological and epidemic characteristics, whereas size of epidemics in the gravity network tended to be larger, better agreement across all three networks was observed when; a) total nodes infected, b) percentage infection take off were compared. These methods consistently identified the same important animal movement and trade hotspots as the empirical networks. We therefore consider this proof-of-concept that demographic data such as censuses and archived molecular data could be repurposed to inform livestock disease management in LMICs. Author summaryLive animal movements in Africa represent a significant risk of transmission and spread of infectious diseases in livestock populations, and therefore, have direct implications on the food security of the continent. Here we explore the potential utility of available data to support control strategies, by comparing movement networks inferred from such data i.e. census and pathogen molecular data using gravity modelling and phylogeography respectively. Their utility is evaluated by comparing their topology and disease spread characteristics to empirical live animal movement. Based on our results, we posit that archived data can be repurposed to support infectious disease control on the African continent.

scientific communication and education↗
bioRxiv2021

scite: a smart citation index that displays the context of citations and classifies their intent using deep learning

Citation indices are tools used by the academic community for research and research evaluation which aggregate scientific literature output and measure scientific impact by collating citation counts. Citation indices help measure the interconnections between scientific papers but fall short because they only display paper titles, authors, and the date of publications, and fail to communicate contextual information about why a citation was made. The usage of citations in research evaluation without due consideration to context can be problematic, if only because a citation that disputes a paper is treated the same as a citation that supports it. To solve this problem, we have used machine learning and other techniques to develop a "smart citation index" called scite, which categorizes citations based on context. Scite shows how a citation was used by displaying the surrounding textual context from the citing paper, and a classification from our deep learning model that indicates whether the statement provides supporting or disputing evidence for a referenced work, or simply mentions it. Scite has been developed by analyzing over 23 million full-text scientific articles and currently has a database of more than 800 million classified citation statements. Here we describe how scite works and how it can be used to further research and research evaluation.

scientific communication and education↗