The unbiased estimation of the fraction of variance explained by a model

Abstract

The correlation coefficient squared, r2, is often used to validate quantitative models on neural data. Yet it is biased by trial-to-trial variability: as trial-to-trial variability increases, measured correlation to a models predictions decreases; therefore, models that perfectly explain neural tuning can appear to perform poorly. Many solutions to this problem have been proposed, but some prior methods overestimate model fit, the utility of even the best performing methods is limited by the lack of confidence intervals and asymptotic analysis, and no consensus has been reached on which is the least biased estimator. We provide a new estimator, [Formula], that outperforms all prior estimators in our testing, and we provide confidence intervals and asymptotic guarantees. We apply our estimator to a variety of neural data to validate its utility. We find that neural noise is often so great that confidence intervals of the estimator cover the entire possible range of values ([0, 1]), preventing meaningful evaluation of the quality of a models predictions. We demonstrate the use of the signal-to-noise ratio (SNR) as a quality metric for making quantitative comparisons across neural recordings. Analyzing a variety of neural data sets, we find ~ 40% or less of some neural recordings do not pass even a liberal SNR criterion. Author SummaryQuantifying the similarity between a model and noisy data is fundamental to the verification of advances in scientific understanding of biological phenomena, and it is particularly relevant to modeling neuronal responses. A ubiquitous metric of similarity is the correlation coefficient. Here we point out how the correlation coefficient depends on a variety of factors that are irrelevant to the similarity between a model and data. While neuroscientists have recognized this problem and proposed corrected methods, no consensus has been reached as to which are effective. Prior methods have wide variation in their precision, and even the most successful methods lack confidence intervals, leaving uncertainty about the reliability of any particular estimate. We address these issues by developing a new estimator along with an associated confidence interval that outperforms all prior methods.