An extensive analysis of software commonly used for functional MRI (fMRI) has found false-positive rates of up to 70%, which may have adversely influenced the results of thousands of neuroimaging examinations, according to a Swedish/U.K. study published online on 28 June.
The researchers retrospectively performed a multitude of random group analyses to calculate false-positive rates of statistical parametric mapping (SPM) software, offerings from the Oxford Centre for Functional MRI of the Brain's Software Library (FSL), and software designed for the analysis and visualization of functional MR neuro images (AFNI). Among the results was the discovery of false activity in brain imaging in 60% of the cases, compared with a standard expected error rate of only 5% in fMRI studies.
"These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results," noted lead author Anders Eklund, PhD, and colleagues from Linköping University in Linköping, Sweden, and the University of Warwick in Coventry, U.K. (Proceedings of the National Academy of Sciences, 28 June 2016).
Unvalidated data
The current study builds upon a 2012 study by Eklund et al of 1,484 rest datasets, which concluded that fMRI software results are not always reliable and that a "better model of the temporal correlations in fMRI rest data is needed" (NeuroImage, Vol. 61:3, July 2012, pp. 565-578).
Dr. Anders Eklund from Linköping University. Image courtesy of Linköping University."We knew that no one had extensively tested the (types of) software with real fMRI data," Eklund wrote in an email to AuntMinnieEurope.com. "Several researchers have tested the softwares with simulated data generated by a computer, but simulated data cannot mimic all properties of real fMRI data."
Additionally, the statistical methods utilized in fMRI software are built on a number of assumptions, the authors noted, and if one or more of those assumptions are incorrect, the results also will be inaccurate.
One challenge was the difficult task of simulating a human brain and an MR scanner. Eklund, who works in Linköping University's department of computer and information science and the department of biomedical engineering, said it is even harder to simulate a human brain inside an MR scanner.
There also is the expense to obtain real fMRI data, which he estimated at 5,000 Swedish kronor (527 euros) per hour for an MRI exam. Thus scanning 500 people, for example, is not often financially feasible.
"In this work, we, therefore, downloaded fMRI data from 671 healthy controls, made possible through several data-sharing initiatives like the 1,000 Functional Connectomes Project, which saved us a lot of time and a lot of money," he added.
Using this data with different experimental designs, researchers estimate the incidence of false positives should be in the range of 5%. However, they discovered the most common software packages for fMRI analysis -- SPM, FSL, AFNI -- can produce false-positive rates of up to 70%.
Software options
To help remedy the deficiencies, Eklund and colleagues offered options to the aforementioned software packages.
Instead of SPM and FSL software, researchers suggested the statistical nonparametric mapping (SnPM) toolbox, which is available through the University of Warwick. It is designed to provide a framework for nonparametric permutation and randomization tests for independent observations based on a lower number of statistical assumptions to provide results considerably closer to the expected 5% false-positive rate.
"The AFNI software has also been improved due to our discoveries," he added. "We also encourage all researchers to share the statistical maps, for example through neurovault.org, and the raw fMRI data, for example through openfmri.org," he added. "If the original fMRI data is available, other researchers can replicate the results and new analyses can be applied several years later."
