Classification-based tests for neuroimaging data analysis: comparison of best practices

In neuroimaging data analysis, classification algorithms are frequently used to discriminate between two populations of interest, like patients and healthy controls, or between stimuli presented to the subject, like face and house. Usually, the ability of the classifier to discriminate populations is used within a statistical test, in order to evaluate scientific hypotheses. In the literature, different procedures are adopted to carry out such tests, like using permutations, assuming the binomial model or using confidence intervals. Moreover multiple choices are made by practitioners when implementing those tests, like the actual classification algorithm or the use of a resampling scheme. In this work we analyze those procedures and some of those choices with respect to their effect on the Type I (false discovery) and Type II (sensitivity) errors. With a simulation study, we compare the different procedures and show the impact in practice. The final aim is to characterize the best practices and give more insight for their use.