In which it is once more demonstrated that scientific evidence is VASTLY less reliable than other types of evidence, because, in most cases, no one ever bothers to actually check the results:
A whole pile of “this is how your brain looks like” MRI-based science has been invalidated because someone finally got around to checking the data.
The problem is simple: to get from a high-resolution magnetic resonance imaging scan of the brain to a scientific conclusion, the brain is divided into tiny “voxels”. Software, rather than humans, then scans the voxels looking for clusters.
When you see a claim that “scientists know when you’re about to move an arm: these images prove it”, they’re interpreting what they’re told by the statistical software.
Now, boffins from Sweden and the UK have cast doubt on the quality of the science, because of problems with the statistical software: it produces way too many false positives.
In this paper at PNAS, they write: “the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results.”
For example, a bug that’s been sitting in a package called 3dClustSim for 15 years, fixed in May 2015, produced bad results (3dClustSim is part of the AFNI suite; the others are SPM and FSL).
That’s not a gentle nudge that some results might be overstated: it’s more like making a bonfire of thousands of scientific papers.
It is not even remotely reasonable to take scientific evidence at face value anymore, much less the pseudoscience so often being substituted for the results produced by genuine, if often flawed, scientody.
It is not even remotely surprising that the flaw the scientists failed to pick up in this situation was statistical, for as I’ve previously observed, most scientists have very little training in math or statistics, and despite their habit of regularly citing statistics, most of them are more or less statistically illiterate.
Never forget that while there are certainly some brilliant scientists, most of them are literal midwits as there are relatively few credentialed scientists with IQs over 132. A study of all the U.S. PhD recipients in 1958 reported an average IQ of 123; the Flynn effect notwithstanding, it is highly unlikely that the average IQ of today’s increasingly diverse and vibrant PhD recipients has risen since then.