最小二乘估计量方差中

如果$X$是满秩，逆$X^TX$存在并且我们得到的最小二乘估计

$$\hat\beta = (X^TX)^{-1}XY$$ 和 $$\operatorname{Var}(\hat\beta) = \sigma^2(X^TX)^{-1}$$

我们如何在方差公式中直观地解释？推导技术对我来说很清楚。$(X^TX)^{-1}$

考虑一个没有常数项的简单回归，其中单个回归值以样本均值为中心。然后 $X'X$为（$n$倍）其样本方差，和$(X'X)^{-1}$其recirpocal。因此，回归变量中的方差=变异性越高，系数估算器的方差越小：解释变量的变异性越大，我们可以更准确地估计未知系数。

为什么？因为回归变量的变化越大，它包含的信息就越多。当回归变量很多时，这将推广到其方差-协方差矩阵的逆矩阵，该矩阵还考虑了回归变量的协方差。在极端情况下，其中$X'X$是对角的，则对于每个估计的系数的精度仅取决于相关联的回归（给出的误差项的方差）的方差/变性。

观看的一个简单的方法$\sigma^2 \left(\mathbf{X}^{T} \mathbf{X} \right)^{-1}$是作为基体（多变量）的类似物$\frac{\sigma^2}{\sum_{i=1}^n \left(X_i-\bar{X}\right)^2}$，这是在简单OLS回归斜率系数的方差。人们甚至可以得到σ2对于该方差， ∑ n i = 1 X 2 i通过省略模型中的截距，即通过原点进行回归来实现。$\frac{\sigma^2}{\sum_{i=1}^n X_i^2}$

从这些公式中的任一个，可以看出，预测变量的较大可变性通常将导致对其系数的更精确估计。这是在实验设计中经常采用的想法，通过选择（非随机）预测变量的值，人们试图使（X T X ）的决定因素尽可能大，该决定因素是变化的量度。$\left(\mathbf{X}^{T} \mathbf{X} \right)$

高斯随机变量的线性变换有帮助吗？使用规则是，如果，X 〜Ñ（μ ，Σ ），然后甲X + b 〜Ñ（甲μ + b ，甲Ť Σ 甲）。$x \sim \mathcal{N}(\mu,\Sigma)$$Ax + b ~ \sim \mathcal{N}(A\mu + b,A^T\Sigma A)$

假设，即ÿ = X β + ε是基础模型和ε 〜Ñ（0 ，σ 2）。$Y = X\beta + \epsilon$$\epsilon \sim \mathcal{N}(0, \sigma^2)$

$$\therefore Y \sim \mathcal{N}(X\beta,\sigma^2)\\ X^TY \sim \mathcal{N}(X^TX\beta, X\sigma^2 X^T)\\ (X^TX)^{-1}X^TY \sim \mathcal{N}[\beta,(X^TX)^{-1} \sigma^2]$$

因此（X T X ）− 1 X T只是一个复杂的缩放矩阵，可转换Y的分布。$(X^TX)^{-1}X^T$$Y$

希望对您有所帮助。

I'll take a different approach towards developing the intuition that underlies the formula $\text{Var}\,\hat{\beta}=\sigma^2 (X'X)^{-1}$. When developing intuition for the multiple regression model, it's helpful to consider the bivariate linear regression model, viz.,

$$y_i=\alpha+\beta x_i + \varepsilon_i, \quad i=1,\ldots,n.$$ $\alpha+\beta x_i$ is frequently called the deterministic contribution to $y_i$, and $\varepsilon_i$ is called the stochastic contribution. Expressed in terms of deviations from the sample means $(\bar{x},\bar{y})$, this model may also be written as $$(y_i-\bar{y}) = \beta(x_i-\bar{x})+(\varepsilon_i-\bar{\varepsilon}), \quad i=1,\ldots,n.$$

To help develop the intuition, we will assume that the simplest Gauss-Markov assumptions are satisfied: $x_i$ nonstochastic, $\sum_{i=1}^n(x_i-\bar{x})^2>0$ for all $n$, and $\varepsilon_i \sim \text{iid}(0,\sigma^2)$ for all $i=1,\ldots,n$. As you already know very well, these conditions guarantee that

$$\text{Var}\,\hat{\beta}=\tfrac{1}{n}\sigma^2(\text{Var}\,x)^{-1}\text{,}$$ where $\text{Var}\,x$ is the sample variance of $x$. In words, this formula makes three claims: "The variance of $\hat{\beta}$ is inversely proportional to the sample size $n$, it is directly proportional to the variance of $\varepsilon$, and it is inversely proportional to the variance of $x$."

Why should doubling the sample size, ceteris paribus, cause the variance of $\hat{\beta}$ to be cut in half? This result is intimately linked to the iid assumption applied to $\varepsilon$: Since the individual errors are assumed to be iid, each observation should be treated ex ante as being equally informative. And, doubling the number of observations doubles the amount of information about the parameters that describe the (assumed linear) relationship between $x$ and $y$. Having twice as much information cuts the uncertainty about the parameters in half. Similarly, it should be straightforward to develop one's intuition as to why doubling $\sigma^2$ also doubles the variance of $\hat{\beta}$.

Let's turn, then, to your main question, which is about developing intuition for the claim that the variance of $\hat{\beta}$ is inversely proportional to the variance of $x$. To formalize notions, let us consider two separate bivariate linear regression models, called Model $(1)$ and Model $(2)$ from now on. We will assume that both models satisfy the assumptions of the simplest form of the Gauss-Markov theorem and that the models share the exact same values of $\alpha$, $\beta$, $n$, and $\sigma^2$. Under these assumptions, it is easy to show that $\text{E}\,\hat{\beta}{}^{(1)}=\text{E}\,\hat{\beta}{}^{(2)}=\beta$; in words, both estimators are unbiased. Crucially, we will also assume that whereas $\bar{x}^{(1)}=\bar{x}^{(2)}=\bar{x}$, $\text{Var}\,x^{(1)}\ne \text{Var}\,x^{(2)}$. Without loss of generality, let us assume that $\text{Var}\,x^{(1)}>\text{Var}\,x^{(2)}$. Which estimator of $\hat{\beta}$ will have the smaller variance? Put differently, will $\hat{\beta}{}^{(1)}$ or $\hat{\beta}{}^{(2)}$ be closer, on average, to $\beta$? From the earlier discussion, we have $\text{Var}\,\hat{\beta} {}^{(k)} =\tfrac{1}{n}\sigma^2/\text{Var}\,x{}^{(k)})$ for $k=1,2$. Because $\text{Var}\,x^{(1)}>\text{Var}\,x^{(2)}$ by assumption, it follows that $\text{Var}\,\hat{\beta}{}^{(1)} <\text{Var}\,\hat{\beta}{}^{(2)}$. What, then, is the intuition behind this result?

Because by assumption $\text{Var}\,x^{(1)}>\text{Var}\,x^{(2)}$, on average each $x_i^{(1)}$ will be farther away from $\bar{x}$ than is the case, on average, for $x_i^{(2)}$. Let us denote the expected average absolute difference between $x_i$ and $\bar{x}$ by $d_x$. The assumption that $\text{Var}\,x^{(1)}>\text{Var}\,x^{(2)}$ implies that $d_x^{(1)} >d_x^{(2)}$. The bivariate linear regression model, expressed in deviations from means, states that $d_y = \beta d_x^{(1)}$ for Model $(1)$ and $d_y = \beta d_x^{(2)}$ for Model $(2)$. If $\beta\ne0$, this means that the deterministic component of Model $(1)$, $\beta d_x^{(1)}$, has a greater influence on $d_y$ than does the deterministic component of Model $(2)$, $\beta d_x^{(2)}$. Recall that the both models are assumed to satisfy the Gauss-Markov assumptions, that the error variances are the same in both models, and that $\beta^{(1)}=\beta^{(2)}=\beta$. Since Model $(1)$ imparts more information about the contribution of the deterministic component of $y$ than does Model $(2)$, it follows that the precision with which the deterministic contribution can be estimated is greater for Model $(1)$ than is the case for Model $(2)$. The converse of greater precision is a lower variance of the point estimate of $\beta$.

It is reasonably straightforward to generalize the intuition obtained from studying the simple regression model to the general multiple linear regression model. The main complication is that instead of comparing scalar variances, it is necessary to compare the "size" of variance-covariance matrices. Having a good working knowledge of determinants, traces and eigenvalues of real symmetric matrices comes in very handy at this point :-)

Say we have $n$ observations (or sample size) and $p$ parameters.

The covariance matrix $\operatorname{Var}(\hat{\beta})$ of the estimated parameters $\hat{\beta}_1,\hat{\beta}_2$ etc. is a representation of the accuracy of the estimated parameters.

If in an ideal world the data could be perfectly described by the model, then the noise will be $\sigma^2= 0$. Now, the diagonal entries of $\operatorname{Var}(\hat{\beta})$ correspond to $\operatorname{Var}(\hat{\beta_1}),\operatorname{Var}(\hat{\beta_2})$ etc. The derived formula for the variance agrees with the intuition that if the noise is lower, the estimates will be more accurate.

In addition, as the number of measurements gets larger, the variance of the estimated parameters will decrease. So, overall the absolute value of the entries of $X^TX$ will be higher, as the number of columns of $X^T$ is $n$ and the number of rows of $X$ is $n$, and each entry of $X^TX$ is a sum of $n$ product pairs. The absolute value of the entries of the inverse $(X^TX)^{-1}$ will be lower.

Hence, even if there is a lot of noise, we can still reach good estimates $\hat{\beta_i}$ of the parameters if we increase the sample size $n$.

I hope this helps.

Reference: Section 7.3 on Least squares: Cosentino, Carlo, and Declan Bates. Feedback control in systems biology. Crc Press, 2011.

This builds on @Alecos Papadopuolos' answer.

Recall that the result of a least-squares regression doesn't depend on the units of measurement of your variables. Suppose your X-variable is a length measurement, given in inches. Then rescaling X, say by multiplying by 2.54 to change the unit to centimeters, doesn't materially affect things. If you refit the model, the new regression estimate will be the old estimate divided by 2.54.

The $X'X$ matrix is the variance of X, and hence reflects the scale of measurement of X. If you change the scale, you have to reflect this in your estimate of $\beta$, and this is done by multiplying by the inverse of $X'X$.