Severe Publication Bias Contributes to Illusory Sleep Consolidation in the Motor Sequence Learning Literature

Abstract

We explored the possibility of publication bias in the sleep and explicit motor sequence learning literature by applying precision effect test (PET) and precision effect test with standard errors (PEESE) weighted regression analyses to the 88 effect sizes from. recent comprehensive literature review (Pan & Rickard, 2015). Basic PET analysis indicated pronounced publication bias; that is, the effect sizes were strongly predicted by their standard error. When variables that have previously been shown to both moderate the sleep gain effect and substantially reduce unaccounted for effect size heterogeneity were included in that analysis, evidence for publication bias remained strong. The estimated postsleep gain was negative, suggesting forgetting rather than facilitation, and it was statistically indistinguishable from the estimated postwake gain. In. qualitative review of. smaller group of more recent studies we observed that (a) small sample sizes—a major factor behind the publication bias—are still the norm, (b) use of demonstrably flawed experimental design and analysis remains prevalent, and (c) when authors conclude in favor of sleep-dependent consolidation, they frequently do not cite the articles in which those methodological flaws have been demonstrated. We conclude that there is substantial publication bias, that there is no consolidation-based, absolute performance gain following sleep, and that strong conclusions regarding the hypothesis of less forgetting after sleep than after wakefulness should await further research. Recommendations are made for reducing publication bias in future work.