Valid and powerful statistical test for decoding accuracy-proposal of Permutation-based Information Prevalence Inference using the i-th order statistic

Abstract

In fMRI decoding studies using pattern classification, a "second-level" group statistical test is typically performed after "first-level" decoding analyses for individual participants. Neuroscientists often test the mean decoding accuracy across participants against the chance-level accuracy (e.g., one-sample Student t-test) to verify whether brain activation includes information about the label (i.e., cognitive content). However, Allefeld et al., (2016) notified that positive results for such tests only indicate that "there are some people in the population whose fMRI data carry information about the experimental condition." Thus, such tests cannot provide inferences about the trend of the majority. They proposed an alternative method, in which prevalence inference is implemented. In this study, I extended their method and propose statistical test "information prevalence inference using the i-th order statistic (i-test)". Compared with the method proposed in the previous study, i-test has high statistical power to provide an inference regarding major population trends. In i-test, the i-th lowest sample decoding accuracy (the i-th order statistic) is compared to the null distribution to test whether the proportion of the higher-than-chance decoding accuracy in the population (information prevalence) is higher than the threshold. Thus, a significant result of the i-test can be interpreted as "the majority of the population has information about the label in the brain." Numerical calculation identified its high statistical power. Also, theoretical detail is provided, and the use of this method in an fMRI decoding study is demonstrated. Allefeld, C., Gorgen, K., Haynes, J.D., 2016. Valid population inference for information-based imaging: From the second-level t-test to prevalence inference. Neuroimage.