Комментарий к статье «Критика парапсихологических мета-анализов Рэдина и Нельсона о генераторах случайных чисел» Мартина Шуба

I would like to reiterate the point of my JSE paper (Scargle, 2000), namely the Rosenthal formula for assessing the potential influence of publication bias on meta-analyses is simply wrong. Its idea is to estimate the number of papers, unpublished as a result of publication bias, that would be consistent with the positive results of a meta-analysis being merely a statistical fluke. The starting equation of Rosenthal’s analysis postulates that the z-values of the papers in the filedrawer are normally distributed, with zero mean. This is in direct contradiction to the definition of the filedrawer as a biased repository of papers with smaller-than-average z-values. The mistake that many have made is to think of the filedrawer as unbiased, and that its effect is merely to statistically dilute the effects of the meta-analytic results. But publication bias is postulated to be biased.
A simple thought experiment shows that publication bias resulting in approximately equal numbers of published and unpublished papers could very easily explain the positive result of any meta-analysis. Simply construct a set of
hypothetical unpublished papers equal in number to those in the meta-analysis, with z-values equal to the negatives of those in the meta-analysis. If the metaanalysis had a positive mean z, the z-values in this set would be smaller on average than those in the meta-analysis, and their distribution would not be unreasonable to find as the result of a publication-based bias in favor of smaller z-values. Combining this hypothetical filedrawer with the studies in the metaanalysis results in a data set with zero mean z, by construction. Hence any logic that results in the claim that a huge number of papers must be languishing in the filedrawer must be wrong.
In this important paper, Martin Schub makes some comments relevant to fixing up Rosenthal-type estimates, but the above argument places a better, direct upper limit on how big the filedrawer must be. (Of course it could be, but need not be, much larger.)