Search bioRxiv⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.1101/2023.06.12.544611

Methodological considerations for behavioral studies relying on response time outcomes through online crowdsourcing platforms

McConnell, P. A.·Finetto, C.·Heise, K.-F.
View original source ↗Export metadata ↓

Abstract

This study explored challenges associated with online crowdsourced data collection, particularly focusing on longitudinal tasks with time-sensitive outcomes like response latencies. The research identified two significant sources of bias: technical shortcomings such as low, variable frame rates, and human factors, contributing to high attrition rates. The study also explored potential solutions to these problems, such as enforcing hardware acceleration and defining study-specific frame rate thresholds, as well as pre-screening participants and monitoring hardware performance and task engagement over each experimental session. This study provides valuable insights into improving the quality and reliability of data collected via online crowdsourced platforms and emphasizes the need for researchers to be cognizant of potential pitfalls in online research.

Source connections

Explore related subjects

scientific communication and educationPreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

McConnell, P. A., Finetto, C., Heise, K.-F.. 2023-06-13. Methodological considerations for behavioral studies relying on response time outcomes through online crowdsourcing platforms. https://doi.org/10.1101/2023.06.12.544611

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Evaluating Large Language Models as Tools to Navigate Researchers in Rapidly Evolving Research Landscapes: A Case Study in Cancer Drug Response Prediction

Large Language Models (LLMs) have emerged as promising tools for assisting researchers in automating and accelerating the synthesis of literature reviews. However, their reliability is a significant concern due to issues like factual inaccuracies and hallucinations. The key question is whether LLMs can reliably provide comprehensive, up-to-date overviews and analyses. This study evaluates the performance of three leading LLMs (OpenAI's ChatGPT, Google's Gemini, and DeepSeek) on the complex task of generating a comprehensive survey paper on deep learning for cancer Drug Response Prediction (DRP). By testing both standard and Deep Research (DR) / Deep Think (DT) modes of LLMs with prompts of varying detail, this paper assesses key academic dimensions, including reference management, content quality, and analytical depth. Key findings reveal that while DR modes of LLMs significantly improve reliability by eliminating hallucinations, performance variations exist across models and prompts. A trade-off between reference quantity and integration quality was observed, and even the best-performing models lacked the analytical depth of human experts, often requiring extensive human supervision. The study concludes that LLMs currently serve as powerful assistive tools but still cannot replace the critical validation and synthesis provided by human researchers. Choosing the best LLM to use depends on the task in hand, while several strategies can be implemented to improve the produced output.

scientific communication and education↗
bioRxiv2023

Analysis of the comparative strengths of intramural and extramural grant funding mechanisms

Funding agencies use a variety of mechanisms to fund research. The National Institutes of Health in the United States, for example, employs scientists to perform research at its own laboratories (intramural research), and it also awards grants to pay for research at external institutions such as universities (extramural research). Here, using data from 1594 intramural grants and 97054 extramural grants funded between 2009 and 2019, we compare the scholarly outputs from these two funding mechanisms in terms of number of publications, relative citation ratio and clinical metrics. We find that extramural awards are more cost-effective for producing outputs commonly used for academic evaluation, such as publications and citations (per dollar), while intramural awards are more cost-effective for generating research that influences future clinical work, more closely in line with the agencys health goals. These findings provide evidence that institutional incentives associated with different funding mechanisms drive their comparative strengths.

scientific communication and education↗
bioRxiv2023

ABCal: a Python package for Author Bias Computation and Scientometric Plotting for Reviews and Meta-Analyses

Systematic reviews are critical summaries of the exiting literature on a given subject and, when combined with meta-analysis, provides a quantitative synthesis of evidence to direct and inform future research. Such reviews must, however, account for complex sources of between study heterogeneity and possible sources of bias, such as publication bias. This paper presents the methods and results of a research study using a newly developed software tool called ABCal (version 1.0.2) to compute and assess author bias in the literature, providing a quantitative measure for the possible effect of overrepresented authors introducing bias to the overall interpretation of the literature. ABCal includes a new metric referred to as author bias, which is a measure of potential biases per paper when the frequency or proportions of contributions from specific authors are considered. The metric is able to account for a significant portion of the observed heterogeneity between studies included in meta-analyses. A meta-regression between observed effect measures and author bias values revealed that higher levels of author bias were associated with higher effect measures while lower author bias was evident for studies with lower effect measures. Furthermore, the softwares capabilities to analyse authorship contributions and produce scientometric plots was able to reveal distinct patterns in both the temporal and geographic distributions of publications, which may relate to any evident publication bias. Thus, ABCal can aid researchers in gaining a deeper understanding of the research landscape and assist in identifying both key contributors and holistic research trends.

scientific communication and education↗