The reproducibility crisis in science, which is the fact that most peer-reviewed studies published in the most-reputable science journals cannot be replicated, is being significantly exacerbated by the fact that the nonreplicable studies are cited far more often than the replicable ones:
Papers in leading psychology, economic and science journals that fail to replicate and therefore are less likely to be true are often the most cited papers in academic research, according to a new study by the University of California San Diego’s Rady School of Management.
Published in Science Advances, the paper explores the ongoing “replication crisis” in which researchers have discovered that many findings in the fields of social sciences and medicine don’t hold up when other researchers try to repeat the experiments.
The paper reveals that findings from studies that cannot be verified when the experiments are repeated have a bigger influence over time. The unreliable research tends to be cited as if the results were true long after the publication failed to replicate.
“We also know that experts can predict well which papers will be replicated,” write the authors Marta Serra-Garcia, assistant professor of economics and strategy at the Rady School and Uri Gneezy, professor of behavioral economics also at the Rady School. “Given this prediction, we ask ‘why are non-replicable papers accepted for publication in the first place?’”
The link between interesting findings and nonreplicable research also can explain why it is cited at a much higher rate—the authors found that papers that successfully replicate are cited 153 times less than those that failed.
“Interesting or appealing findings are also covered more by media or shared on platforms like Twitter, generating a lot of attention, but that does not make them true,” Gneezy said.
Serra-Garcia and Gneezy analyzed data from three influential replication projects which tried to systematically replicate the findings in top psychology, economic and general science journals (Nature and Science). In psychology, only 39 percent of the 100 experiments successfully replicated. In economics, 61 percent of the 18 studies replicated as did 62 percent of the 21 studies published in Nature/Science.
With the findings from these three replication projects, the authors used Google Scholar to test whether papers that failed to replicate are cited significantly more often than those that were successfully replicated, both before and after the replication projects were published. The largest gap was in papers published in Nature/Science: non-replicable papers were cited 300 times more than replicable ones.
Apparently my prediction that science would be as reliably false as government press releases may have been an observation, not a prediction. Putting the reproducibility problem together with the non-replicable citation problem indicates that present scientage – the current scientific knowledge base – in Nature/Science is only 0.5 percent reliable.
Granted, that’s half-a-percent more reliable than government press releases, but still reliably wrong. In other words, almost everything that “studies show” has no support from an actual scientific perspective.