对所有“不具有统计学意义”的研究进行荟萃分析能否得出“重要”结论？

荟萃分析包括大量研究，所有研究均报告P值大于0.05。整体荟萃分析报告的P值是否可能小于0.05？什么情况下

（我很确定答案是肯定的，但是我想提供参考或解释。）

statistical-significance meta-analysis combining-p-values

— 哈维·莫图尔斯基
source

我对荟萃分析了解不多，但给我的印象是它不涉及任何假设检验，仅是对总体影响的估计，在这种情况下，没有要说的重要概念。

— Kodiologist '16

好吧，在一天结束时，荟萃分析只是加权平均值。当然，您可以为该加权平均值建立假设检验。参见，例如，Borenstein，Michael等。“对荟萃分析的固定效应和随机效应模型进行了基本介绍。” 研究合成方法1.2（2010）：97-111。

— boscovich

其他答案也很好，但很简单：两个研究在p = 0.9时有意义，但在p = 0.95时不明显。两项独立研究均显示p> = 0.9的概率仅为0.01，因此您的荟萃分析可能显示出p = 0.99的显着性

— Barrycarter

极限：没有一个度量可以为（非平凡的）假设提供足够的证据，以使其具有较小的

p

$p$ 值，但是可以有足够大的度量集合。

— 埃里克·塔

p值既不表示“统计上显着”，也不表示无关紧要。从一个重要的结论中我们可以理解什么？这是一个荟萃分析的结论吗？

— Subhash C. Davar

Answers:

从理论上讲，是的...

个别研究的结果可能微不足道，但综合来看，结果可能会很重要。

从理论上讲，你可以通过处理结果进行 $y_i$ 研究 $i$ 像任何其他随机变量。

令 $y_i$ 是一些随机变量（例如，研究的估计值 $i$ ）。然后，如果 $y_i$ 是独立的且 $E[y_i]=\mu$ ，则可以通过以下方式一致地估计均值：

\hat{μ} = \frac{1}{n} \sum_{i} y_{i}

$\hat{\mu} = \frac{1}{n} \sum_i y_i$

加入更多的假设，让 $\sigma^2_i$ 是估计的方差 $y_i$ 。然后，您可以使用逆方差加权有效地估算 $\mu$ ：

\hat{μ} = \sum_{i} w_{i} y_{i} w_{i} = \frac{1 / σ_{i}^{2}}{\sum_{j} 1 / σ_{j}^{2}}

$\hat{\mu} = \sum_i w_i y_i \quad \quad w_i = \frac{1 / \sigma^2_i}{\sum_j 1 / \sigma^2_j}$

在这两种情况下 $\hat{\mu}$ 可能是在一定的信心水平上显显著即使个别估计都没有。

但是可能会有大问题，需要注意的问题...

如果 $E[y_i] \neq \mu$ 然后荟萃分析可能不收敛到 $\mu$ （即平均值的荟萃分析的是不一致的估计器）。

例如，如果对发布负面结果存有偏见，那么这种简单的荟萃分析可能会前后矛盾且带有偏见！就像仅通过观察硬币未落尾的翻转来估计硬币翻转落下的概率一样！
和可能不是独立的。例如，如果两项研究和基于相同的数据，则在荟萃分析中将和视为独立的可能会大大低估标准误和夸大统计意义。您的估计仍然是一致的，但是标准误差需要合理地考虑研究中的互相关。 $y_i$ $y_j$ $i$ $j$ $y_i$ $y_j$
组合（1）和（2）可能特别糟糕。

例如，平均民意测验的荟萃分析往往比任何单个民意测验更准确。但是将轮询平均在一起仍然很容易受到相关误差的影响。过去的选举中发生的事情是，年轻的退出民意调查工作者可能倾向于采访其他年轻人而不是老人。如果所有出口民意测验都产生相同的错误，那么您可能会认为估算值不错，您可能会认为这是一个不错的估计（出口民意测验是相关的，因为它们使用相同的方法进行出口民意测验，并且此方法会产生相同的错误）。

毫无疑问，更熟悉荟萃分析的人们可能会提出更好的例子，更细微的问题，更复杂的估算技术等...，但这涉及到一些最基本的理论和一些更大的问题。如果不同的研究产生独立的随机误差，那么荟萃分析可能会非常有用。如果跨研究的错误是系统性的（例如，每个人都低估了年长选民等），那么研究的平均值也将不正确。如果您低估了相关研究的程度或相关误差的程度，那么您实际上就高估了样本总数，而低估了标准误差。

一致性定义等还有各种实际问题...

— 马修·冈恩
source

我批评忽略了效应大小之间的依赖性的荟萃分析（即，许多效应大小是基于相同的参与者，但被视为独立的）。作者说没什么大不了的，无论如何我们只是对主持人感兴趣。我在这里指出您的意思：将它们“独立进行荟萃分析可能会大大低估标准误差和夸大统计意义”。是否有证明/模拟研究表明为什么会这样？我有很多参考资料说相关错误意味着SE被低估了...但是我不知道为什么？

— 马克·怀特

@MarkWhite基本思想并不比

。如果对所有

我们有

和

对于

然后

Var (\frac{1}{n} \sum_{i} X_{i}) = \frac{1}{n^{2}} (\sum_{i} Var (X_{i}) + \sum_{i \neq j} Cov (X_{i}, X_{j}))

$\operatorname{Var}\left( \frac{1}{n} \sum_i X_i \right) = \frac{1}{n^2} \left( \sum_{i} \operatorname{Var}(X_i) + \sum_{i \neq j} \operatorname{Cov}(X_i, X_j) \right)$

i

$i$

Var (X_{i}) = σ^{2}

$\operatorname{Var}(X_i) = \sigma^2$

Cov (X_{i}, X_{j}) = 0

$\operatorname{Cov}(X_i, X_j) = 0$

i \neq j

$i\neq j$

，您的标准误是

Var (\frac{1}{n} \sum_{i} X_{i}) = \frac{σ^{2}}{n}

$\operatorname{Var}\left( \frac{1}{n} \sum_i X_i \right) = \frac{\sigma^2}{n}$

。另一方面，如果协方差项为正且较大，则标准误差将更大。

\frac{σ}{\sqrt{n}}

$\frac{\sigma}{\sqrt{n}}$

— 马修·冈恩

@MarkWhite我不是一个荟萃分析专家，我真的不知道什么是一个如何的重要来源应该做的现代，荟萃分析。从概念上讲，对相同数据进行重复分析肯定是有用的（就像对某些主题的深入研究一样），但这与在新的独立主题上再现发现并不相同。

— 马修·冈恩

嗯，也就是说：效果大小的总方差来自（a）它的方差和（b）它与其他效果大小的协方差。如果协方差为0，则标准误差估计很好；但是，如果它与其他效应大小协变量，则需要考虑该差异，而忽略它意味着我们低估了差异。这就像方差由A和B的两个部分组成，而忽略依赖关系则假设B部分为0时不是吗？

— 马克·怀特

此外，这似乎是一个很好的来源（尤其是专栏2）：nature.com/neuro/journal/v17/n4/pdf/nn.3648.pdf

— 马克·怀特

是。假设你有从p值独立研究。 $N$ $N$

费舍尔测试

（编辑-为响应下面的@mdewey的有用评论，区分不同的元测试很重要。我在下面阐明了mdewey提到的另一个元测试的情况）

经典费希尔元测试（参见费舍尔（1932），“用于研究工作者的统计方法”）的统计具有零分布，如对于均匀RV 。

F = - 2 \sum_{i = 1}^{N} \ln (p_{i})

$F=-2\sum_{i=1}^N\ln(p_i)$

χ_{2 N}^{2}

$\chi^2_{2N}$

- 2 \ln (U) \sim χ_{2}^{2}

$-2\ln(U)\sim\chi^2_2$

U

$U$

Let $\chi^2_{2N}(1-\alpha)$ denote the $(1-\alpha)$ -quantile of the null distribution.

Suppose all p-values are equal to $c$ , where, possibly, $c>\alpha$ . Then, $F=-2N\ln(c)$ and $F>\chi^2_{2N}(1-\alpha)$ when

c < \exp (- \frac{χ_{2 N}^{2} (1 - α)}{2 N})

$c < \exp\left(-\frac{\chi^2_{2N}(1-\alpha)}{2N}\right)$ For example, for

α = 0.05

$\alpha=0.05$ and

N = 20

$N=20$ , the individual

p

$p$ -values only need to be less than

> exp(-qchisq(0.95, df = 40)/40)
[1] 0.2480904

Of course, what the meta statistic tests is "only" the "aggregate" null that all individual nulls are true, which is to be rejected as soon as only one of the $N$ nulls is false.

EDIT:

Here is a plot of the "admissible" p-values against $N$ , which confirms that $c$ grows in $N$ , although it seems to level off at $c\approx0.36$ .

I found an upper bound for the quantiles of the $\chi^2$ distribution

χ_{2 N}^{2} (1 - α) \leq 2 N + 2 \log (1 / α) + 2 \sqrt{2 N \log (1 / α)},

$\chi^2_{2N}(1-\alpha)\leq 2N+2\log(1/\alpha)+2\sqrt{2N\log(1/\alpha)},$ here, suggesting that

χ_{2 N}^{2} (1 - α) = O (N)

$\chi^2_{2N}(1-\alpha)=O(N)$ so that

\exp (- \frac{χ_{2 N}^{2} (1 - α)}{2 N})

$\exp\left(-\frac{\chi^2_{2N}(1-\alpha)}{2N}\right)$ is bounded from above by

\exp (- 1)

$\exp(-1)$ as

N \to \infty

$N\to\infty$ . As

\exp (- 1) \approx 0.3679

$\exp(-1)\approx0.3679$ , this bound seems reasonably sharp.

Inverse Normal test (Stouffer et al., 1949)

The test statistic is given by

Z = \frac{1}{\sqrt{N}} \sum_{i = 1}^{N} Φ^{- 1} (p_{i})

$Z=\frac{1}{\sqrt{N}}\sum_{i=1}^N\Phi^{-1}(p_i)$ with

Φ^{- 1}

$\Phi^{-1}$ the standard normal quantile function. The test rejects for large negative values, viz., if

Z < - 1.645

$Z < -1.645$ at

α = 0.05

$\alpha=0.05$ . Hence, for

p_{i} = c

$p_i=c$ ,

Z = \sqrt{N} Φ^{- 1} (c)

$Z=\sqrt{N}\Phi^{-1}(c)$ . When

c < 0.5

$c<0.5$ ,

Φ^{- 1} (c) < 0

$\Phi^{-1}(c)<0$ and hence

Z \to_{p} - \infty

$Z\to_p-\infty$ as

N \to \infty

$N\to\infty$ . If

c \geq 0.5

$c\geq0.5$ ,

Z

$Z$ will take values in the acceptance region for any

N

$N$ . Hence, a common p-value less than 0.5 is sufficient to produce a rejection of the meta test as

N \to \infty

$N\to\infty$ .

More specifically, $Z < -1.645$ if $c<\Phi(-1.645/\sqrt{N})$ , which tends to $\Phi(0)=0.5$ from below as $N\to\infty$ .

— Christoph Hanck
source

+1 and wow! did not expect there to be an upper bound at all, let alone

1 / e

$1/e$ .

— amoeba says Reinstate Monica

Thanks :-). I had not expected one either before I saw the plot...

— Christoph Hanck

Interestingly the method due to Fisher is the only one of the commonly used methods which has this property. For most of the others what you call F increases with N if $c>0.5) and decreases otherwise. That applies to Stouffer's method and Edgington's method as well as methods based on logits and on mean of p. The various methods which are special cases of Wilkinson's method (minimum p, maximum p, etc) have different properties again.

— mdewey

@mdewey, that is interesting indeed, I just picked Fisher's test purely because it came to my mind first. That said, by "only one", do you mean the specific bound

1 / e

$1/e$ ? Your comments, that I try to spell out in my edit, suggest to me that Stouffer's method also has an upper bound, that turns out to be 0.5?

— Christoph Hanck

I am not going to have time to go into this for another week but I think if you have ten studies with

p = 0.9

$p=0.9$ you get an overall

p

$p$ as close to unity as makes no difference. There may be a one- versus two-sided issue here. If you want to look at more material I have a draft of extra stuff to go into my R package <code>metap</code> here which you are free to use to expand your answer if you wish.

— mdewey

The answer to this depends on what method you use for combining $p$ -values. Other answers have considered some of these but here I focus on one method for which the answer to the original question is no.

The minimum $p$ method, also known as Tippett's method, is usually described in terms of a rejection at the $\alpha_*$ level of the null hypothesis. Define

p_{[1]} \leq p_{[2]} \dots p_{[k]}

$p_{[1]} \le p_{[2]} \dots p_{[k]}$ for the

k

$k$ studies. Tippett's method then evaluates whether

p_{[1]} < 1 - (1 - α_{*})^{\frac{1}{k}}

$\begin{equation} p_{[1]} < 1 - (1 - \alpha_*)^{\frac{1}{k}} \end{equation}$

It is easy to see the since the $k$ th root of a number less than unity is closer to unity the last term is greater than $\alpha_*$ and hence the overall result will be non-significant unless $p_{[1]}$ is already less than $\alpha_*$ .

It is possible to work out the critical value and for example if we have ten primary studies each with a $p$ -values of 00.05 so as close to significant as can be then the overall critical value is 0.40. The method can be seen as a special case of Wilkinson's method which uses $p_{[r]}$ for $1\le r\le k$ and in fact for the particular set of primary studies even $r=2$ is not significant ( $p=0.09$ )

L H C Tippett's method is described in a book The methods of statistics. 1931 (1st ed) and Wilkinson's method is here in an article "A statistical consideration in psychological research"

— mdewey
source

Thanks. But note that most meta-analysis methods combine effect sizes (accounting for any difference in sample size), and do not combine P values.

— Harvey Motulsky

@HarveyMotulsky agreed, combining p-values is a last resort but the OP did tag his question with the combining-p-values tag so I responded in that spirit

— mdewey

I think that your answer is correct.

— Subhash C. Davar