对于什么模型，MLE的偏差下降快于方差？

$\hat\theta$ $\theta^*$ $n$ $\lVert\hat\theta-\theta^*\rVert$ $O(1/\sqrt n)$ $\lVert \mathbb E\hat\theta - \theta^*\rVert$ $\lVert \mathbb E\hat\theta - \hat\theta\rVert$ $O(1/\sqrt{n})$

我对具有比更快地收缩的偏差的模型感兴趣，但是其中的误差不会以这种更快的速率收缩，因为偏差仍以收缩。特别是，我想知道足够的条件来使模型的偏差以的速率收缩。 $O(1/\sqrt n)$ $O(1/\sqrt n)$ $O(1/n)$

— 迈克·伊兹比基（Mike Izbicki）
source

确实？要么？

∥θ^−θ∗∥=(θ^−θ∗)2 $\lVert\hat\theta-\theta^*\rVert = (\hat\theta-\theta^*)^2$

— Alecos Papadopoulos

我是专门询问L2规范的，是的。但是如果它使问题更易于回答，我也会对其他规范感兴趣。

— Mike Izbicki

(θ^−θ∗)2 $(\hat \theta -\theta^*)^2$ 是。

Op(1/n) $O_p(1/n)$

— Alecos Papadopoulos

抱歉，我看不懂您的评论。对于维的L2范数，，因此收敛速度为。我同意，如果我们将其平方，那么它将收敛为。

d $d$

∥a−b∥=∑di=1(ai−bi)2−−−−−−−−−−−−√ $\Vert a-b\Vert = \sqrt{\sum_{i=1}^d (a_i-b_i)^2}$

O(1/n−−√) $O(1/\sqrt n)$

O(1/n) $O(1/n)$

— Mike Izbicki

您是否看到过岭回归（Hoerl＆Kennard 1970）的论文？我相信它给出了设计矩阵+罚金的条件，而这是正确的。

— dcl

Answers:

通常，您需要的模型中MLE并非渐近正态，而是会收敛到其他某种分布（并且收敛速度更快）。当被估计的参数位于参数空间的边界时，通常会发生这种情况。直观上，这意味着MLE将“仅从一侧”接近参数，因此它“提高了收敛速度”，因为它不会因在参数周围“来回”移动而“分散注意力”。

一个标准的例子，是用于MLE $\theta$ 的IID样品中 $U(0,\theta)$ 一致的RV的The MLE这里是最大阶统计量，

θ^n = u (n)

$\hat \theta_n = u_{(n)}$

其有限样本分布为

F θ^n = ( θ ^ n ) n θ n, f θ^= n ( θ ^ n ) n - 1 θ n

$F_{\hat \theta_n} = \frac {(\hat \theta_n)^n}{\theta ^n},\;\;\; f_{\hat \theta}=n\frac {(\hat \theta_n)^{n-1}}{\theta ^n}$

E (θ^n) = n n + 1 θ ⟹ B (θ^) = - 1 n + 1 θ

$\mathbb E(\hat \theta_n) = \frac {n}{n+1}\theta \implies B(\hat \theta) = -\frac {1}{n+1}\theta$

所以。但是，相同的增长率也适用于方差。 $B(\hat \theta_n) = O(1/n)$

你也可以验证，以获得极限分布，我们需要看看变量，（即我们需要通过规模），因为 $n(\theta - \hat \theta_n)$ $n$

P [n (θ - θ^n) \leq z] = 1 - P [θ^n \leq θ - (z / n)]

$P[n(\theta - \hat \theta_n)\leq z] = 1-P[\hat \theta_n\leq \theta - (z/n)]$

= 1 - 1 θ n \cdot (θ + - z n) n = 1 - θ n θ n \cdot (1 + - z / θ n) n

$=1-\frac 1 {\theta^n}\cdot \left(\theta + \frac{-z}{n}\right)^n = 1-\frac {\theta^n} {\theta^n}\cdot \left(1 + \frac{-z/\theta}{n}\right)^n$

\to 1 - e - z / θ

$\to 1- e^{-z/\theta}$

which is the CDF of the Exponential distribution.

I hope this provides some direction.

— Alecos Papadopoulos
source

This is getting close, but I'm specifically interested in situations where the bias shrinks faster than the variance.

— Mike Izbicki

@MikeIzbicki Hmm ...偏差收敛取决于分布的一阶矩，（平方根）方差也是“一阶”量值。那时我不确定这是否有可能发生，因为看来这意味着极限分布的时刻会以彼此不兼容的收敛速度“上升”……我会考虑一下。

— Alecos Papadopoulos

在我其他答案中的评论之后（并再次查看OP的标题！），这是对该问题的不太严格的理论探索。

We want to determine whether Bias $B(\hat \theta_n) = E(\hat \theta_n) - \theta$ may have different convergence rate than the square root of the Variance,

B (θ^n) = O (1 / n δ), Var (θ^n) - - - - - - - \sqrt = O (1 / n γ), γ \neq δ ? ? ?

$B(\hat \theta_n) = O(1/n^{\delta}),\;\;\; \sqrt {\text{Var}(\hat \theta_n)} = O(1/n^{\gamma}), \;\;\gamma \neq \delta \;???$

We have

B (θ^n) = O (1 / n δ) ⟹ lim n δ E (θ^n) < K ⟹ lim n 2 δ [E (θ^n)] 2 < K'

$B(\hat \theta_n) = O(1/n^{\delta}) \implies \lim n^{\delta}\mathbb E(\hat \theta_n) < K \implies \lim n^{2\delta}[\mathbb E(\hat \theta_n)]^2 < K'$

⟹ [E (θ^n)] 2 = O (1 / n 2 δ) (1)

$\implies [\mathbb E(\hat \theta_n)]^2 = O(1/n^{2\delta}) \tag{1}$

while

Var (θ^n) - - - - - - - \sqrt = O (1 / n γ) ⟹ lim n γ E (θ^2 n) - [E (θ^n)] 2 - - - - - - - - - - - - - \sqrt < M

$\sqrt {\text{Var}(\hat \theta_n)} = O(1/n^{\gamma}) \implies \lim n^{\gamma}\sqrt{\mathbb E (\hat \theta_n^2) - [\mathbb E(\hat \theta_n)]^2 }<M$

⟹ lim n 2 γ E (θ^2 n) - n 2 γ [E (θ^n)] 2 - - - - - - - - - - - - - - - - - - \sqrt < M

$\implies \lim \sqrt{n^{2\gamma}\mathbb E (\hat \theta_n^2) - n^{2\gamma}[\mathbb E(\hat \theta_n)]^2 }<M$

⟹ lim n 2 γ E (θ^2 n) - lim n 2 γ [E (θ^n)] 2 < M' (2)

$\implies \lim n^{2\gamma}\mathbb E (\hat \theta_n^2) - \lim n^{2\gamma}[\mathbb E(\hat \theta_n)]^2 < M' \tag{2}$

We see that $(2)$ may hold happen if

A) both components are $O(1/n^{2\gamma})$ , in which case we can only have $\gamma = \delta$ .

B) But it may also hold if

lim n 2 γ [E (θ^n)] 2 \to 0 ⟹ [E (θ^n)] 2 = o (1 / n 2 γ) (3)

$\lim n^{2\gamma}[\mathbb E(\hat \theta_n)]^2 \to 0 \implies [\mathbb E(\hat \theta_n)]^2 = o(1/n^{2\gamma}) \tag{3}$

For $(3)$ to be compatible with $(1)$ , we must have

$n^{2\gamma} < n^{2\delta} \implies \delta > \gamma\tag {4}$

So it appears that in principle it is possible to have the Bias converging at a faster rate than the square root of the variance. But we cannot have the square root of the variance converging at a faster rate than the Bias.

— Alecos Papadopoulos
source

How would you reconcile this with the existence of unbiased estimators like ordinary least squares? In that case,

$B(\hat\theta)=0$ , but

$\sqrt{Var(\hat\theta)} = O(1/\sqrt n)$ .

— Mike Izbicki

@MikeIzbicki Is the concept of convergence/big-O applicable in this case? Because here

$B(\hat \theta)$ is not "

$O()$ -anything" to begin with.

— Alecos Papadopoulos

In this case,

$\mathbb E\hat\theta=\theta^*$ , so

$B(\hat\theta) = \lVert \mathbb E \hat\theta - \theta^*\rVert = 0 = O(1) = O(1/n^0)$ .

— Mike Izbicki

@MikeIzbicki But also

$B(\hat \theta) = O(n)$ or

$B(\hat \theta) =O(1/\sqrt{n})$ or any other you care to write down. So which one is the rate of convergence here?

— Alecos Papadopoulos

@MikeIzbicki I have corrected my answer to show that it is possible in principle to have the Bias converging faster, although I still think the "zero-bias" example is problematic.

— Alecos Papadopoulos