为什么将牛顿法用于逻辑回归优化称为迭代重加权最小二乘？

为什么将牛顿法用于逻辑回归优化称为迭代重加权最小二乘？

对我来说似乎不清楚，因为逻辑损失和最小二乘损失是完全不同的东西。

简介：GLM通过费舍尔评分来拟合，正如Dimitriy V. Masterov所指出的那样，牛顿-拉夫森与预期的Hessian相匹配（即，我们使用Fisher信息的估计值而不是观察到的信息）。如果我们使用规范链接函数，则发现观察到的Hessian等于预期的Hessian，因此在这种情况下NR和Fisher评分相同。无论哪种方式，我们都会看到Fisher评分实际上是在拟合加权最小二乘线性模型，并且由此估计的系数估计值在对数回归可能性最大时收敛。除了减少对已经解决的问题进行逻辑回归拟合之外，我们还获得了能够在最终WLS拟合中使用线性回归诊断程序以了解我们的逻辑回归的好处。

我将重点放在逻辑回归上，但是对于GLM中最大似然的更一般的观点，我建议本章的 15.3节介绍这一点，并在更一般的情况下得出IRLS（我认为这是从John Fox的Applied回归分析和广义线性模型（）。

$^*$见尾注

---

可能性和得分函数

我们将通过迭代形式的东西来拟合我们的GLM

$$b^{(m+1)} = b^{(m)} - J^{-1}_{(m)}\nabla \ell(b^{(m)})$$ ，其中$\ell$是对数似然和$J_{m}$将或者对数似然的观测或预期的Hessian。

我们的链接函数是一个函数$g$映射条件均值$\mu_i = E(y_i | x_i)$到我们的线性预测器，所以我们对均值模型是$g(\mu_i) = x_i^T\beta$。令$h$为将线性预测变量映射到均值的逆链接函数。

对于Logistic回归，我们具有独立观察的伯努利似然，因此

$$\ell(b; y) = \sum_{i=1}^n y_i\log h(x_i^T b) + (1 - y_i) \log(1 - h(x_i^Tb)).$$ 取导数， = n ∑ i=1xijh'（x T i b）（ y $$\frac{\partial \ell}{\partial b_j} = \sum_{i=1}^n \frac{y_i}{h(x_i^T b)} h'(x_i^T b) x_{ij} - \frac{1 - y_i}{1 - h(x_i^T b)} h'(x_i^T b) x_{ij}$$ =∑ixijh'（x T i b $$= \sum_{i=1}^n x_{ij} h'(x_i^T b) \left(\frac{y_i}{h(x_i^T b)} - \frac{1 - y_i}{1 - h(x_i^T b)} \right)$$ $$= \sum_i x_{ij} \frac{h'(x_i^T b)}{h(x_i^T b)(1 - h(x_i^T b))}(y_i - h(x_i^T b)).$$

---

使用规范链接

现在，假设我们使用规范链接函数。那么g − 1 c（x ）：= h c（x ）= $g_c = \text{logit}$所以ħ ' C ^ =ħÇ⋅（1-H ^C ^），这意味着这简化为 ＆PartialD;＆$g^{-1}_c(x) := h_c(x) = \frac{1}{1+e^{-x}}$$h_c' = h_c \cdot (1-h_c)$ ，以便 ＆dtri;ℓ（b;Ý）=XŤ（Ý - Ý）。 此外，仍然使用ħÇ， ∂2

$$\frac{\partial \ell}{\partial b_j} = \sum_i x_{ij} (y_i - h_c(x_i^T b))$$ $$\nabla \ell (b; y) = X^T (y - \hat y).$$ $h_c$ $$\frac{\partial^2 \ell}{\partial b_k \partial b_j} = - \sum_i x_{ij} \frac{\partial}{\partial b_k} h_c(x_i^T b) = - \sum_i x_{ij}x_{ik} \left[h_c(x_i^T b) (1 - h_c(x_i^T b))\right].$$

令 然后我们有 H=− X T WX， 并记下它如何不再包含 y i，所以E（H）=H（我们将其视为b的函数，所以唯一的随机数是y本身）。因此，我们证明了在逻辑回归中使用规范链接时，费舍尔评分与牛顿-拉夫森等效。也

$$W = \text{diag}\left(h_c(x_1^T b)(1 - h_c(x_1^T b)), \dots, h_c(x_n^T b)(1 - h_c(x_n^T b))\right) = \text{diag}\left(\hat y_1(1 - \hat y_1), \dots, \hat y_n (1 - \hat y_n)\right).$$ $$H = -X^TWX$$ $y_i$$E(H) = H$$b$$y$-XŤW¯¯X将总是严格负定的，虽然数值如果 ý我太接近0或1则我们可以具有权重四舍五入到0，它可以使ħ负半定，因此计算单数。$\hat y_i \in (0,1)$ $-X^TWX$$\hat y_i$$0$$1$$0$$H$

$z = W^{-1}(y - \hat y)$

$$\nabla \ell = X^T(y - \hat y) = X^T W z.$$

$$b^{(m+1)} = b^{(m)} + (X^T W_{(m)} X)^{-1}X^T W_{(m)} z_{(m)}$$ $(X^T W_{(m)} X)^{-1}X^T W_{(m)} z_{(m)}$$\hat \beta$$z_{(m)}$$X$

在检查此R：

set.seed(123)
p <- 5
n <- 500
x <- matrix(rnorm(n * p), n, p)
betas <- runif(p, -2, 2)
hc <- function(x) 1 /(1 + exp(-x)) # inverse canonical link
p.true <- hc(x %*% betas)
y <- rbinom(n, 1, p.true)

# fitting with our procedure
my_IRLS_canonical <- function(x, y, b.init, hc, tol=1e-8) {
  change <- Inf
  b.old <- b.init
  while(change > tol) {
    eta <- x %*% b.old  # linear predictor
    y.hat <- hc(eta)
    h.prime_eta <- y.hat * (1 - y.hat)
    z <- (y - y.hat) / h.prime_eta

    b.new <- b.old + lm(z ~ x - 1, weights = h.prime_eta)$coef  # WLS regression
    change <- sqrt(sum((b.new - b.old)^2))
    b.old <- b.new
  }
  b.new
}

my_IRLS_canonical(x, y, rep(1,p), hc)
# x1         x2         x3         x4         x5 
# -1.1149687  2.1897992  1.0271298  0.8702975 -1.2074851

glm(y ~ x - 1, family=binomial())$coef
# x1         x2         x3         x4         x5 
# -1.1149687  2.1897992  1.0271298  0.8702975 -1.2074851 

他们同意。

---

非规范链接功能

$\frac{h'}{h(1-h)} = 1$$\nabla \ell$$H$$E(H)$ in our Fisher scoring.

Here's how this will go: we already worked out the general $\nabla \ell$ so the Hessian will be the main difficulty. We need

$$\frac{\partial^2 \ell}{\partial b_k \partial b_j} = \sum_i x_{ij} \frac{\partial}{\partial b_k}h'(x_i^T b) \left(\frac{y_i}{h(x_i^T b)} - \frac{1 - y_i}{1 - h(x_i^T b)} \right)$$ $$= \sum_i x_{ij}x_{ik} \left[h''(x_i^T b) \left(\frac{y_i}{h(x_i^T b)} - \frac{1 - y_i}{1 - h(x_i^T b)} \right) - h'(x_i^T b)^2\left(\frac{y_i}{h(x_i^T b)^2} + \frac{1-y_i}{(1-h(x_i^T b))^2} \right)\right]$$

Via the linearity of expectation all we need to do to get $E(H)$ is replace each occurrence of $y_i$ with its mean under our model which is $\mu_i=h(x_i^T\beta)$. Each term in the summand will therefore contain a factor of the form

$$h''(x_i^T b) \left(\frac{h(x_i^T \beta)}{h(x_i^T b)} - \frac{1 - h(x_i^T \beta)}{1 - h(x_i^T b)} \right) - h'(x_i^T b)^2\left(\frac{h(x_i^T \beta)}{h(x_i^T b)^2} + \frac{1-h(x_i^T \beta)}{(1-h(x_i^T b))^2} \right).$$ But to actually do our optimization we'll need to estimate each $\beta$, and at step $m$ $b^{(m)}$ is the best guess we have. This means that this will reduce to $$h''(x_i^T b) \left(\frac{h(x_i^T b)}{h(x_i^T b)} - \frac{1 - h(x_i^T b)}{1 - h(x_i^T b)} \right) - h'(x_i^T b)^2\left(\frac{h(x_i^T b)}{h(x_i^T b)^2} + \frac{1-h(x_i^T b)}{(1-h(x_i^T b))^2} \right)$$ $$= - h'(x_i^T b)^2\left(\frac{1}{h(x_i^T b)} + \frac{1}{1-h(x_i^T b)} \right)$$ $$= -\frac{h'(x_i^T b)^2}{h(x_i^T b)(1-h(x_i^T b))}.$$ This means we will use $J$ with $$J_{jk} = -\sum_i x_{ij}x_{ik} \frac{h'(x_i^T b)^2}{h(x_i^T b)(1-h(x_i^T b))}.$$

Now let

$$W^* = \text{diag}\left(\frac{h'(x_1^T b)^2}{h(x_1^T b)(1-h(x_1^T b))} ,\dots, \frac{h'(x_n^T b)^2}{h(x_n^T b)(1-h(x_n^T b))}\right)$$ and note how under the canonical link $h_c' = h_c \cdot (1-h_c)$ reduces $W^*$ to $W$ from the previous section. This lets us write $$J = -X^TW^*X$$ except this is now $\hat E(H)$ rather than necessarily being $H$ itself, so this can differ from Newton-Raphson. For all $i$ $W_{ii}^* > 0$ so aside from numerical issues $J$ will be negative definite.

We have

$$\frac{\partial \ell}{\partial b_j} = \sum_i x_{ij} \frac{h'(x_i^T b)}{h(x_i^T b)(1 - h(x_i^T b))}(y_i - h(x_i^T b))$$ so letting our new working response be $z^* = D^{-1}(y-\hat y)$ with $D=\text{diag}\left(h'(x_1^T b), \dots, h'(x_n^T b)\right)$, we have $\nabla \ell = X^TW^*z^*$.

All together we are iterating

$$b^{(m+1)} = b^{(m)} + (X^T W_{(m)}^* X)^{-1}X^T W_{(m)}^* z_{(m)}^*$$ so this is still a sequence of WLS regressions except now it's not necessarily Newton-Raphson.

---

I've written it out this way to emphasize the connection to Newton-Raphson, but frequently people will factor the updates so that each new point $b^{(m+1)}$ is itself the WLS solution, rather than a WLS solution added to the current point $b^{(m)}$. If we wanted to do this, we can do the following:

$$b^{(m+1)} = b^{(m)} + (X^T W_{(m)}^* X)^{-1}X^T W_{(m)}^* z_{(m)}^*$$ $$= (X^T W_{(m)}^* X)^{-1}\left(X^T W_{(m)}^* Xb^{(m)}+ X^TW^*_{(m)}z_{(m)}^* \right)$$ $$= (X^T W_{(m)}^* X)^{-1}X^TW_{(m)}^*\left(Xb^{(m)}+ z_{(m)}^* \right)$$ so if we're going this way you'll see the working response take the form $\eta^{(m)} + D^{-1}_{(m)}(y - \hat y^{(m)})$, but it's the same thing.

---

Let's confirm that this works by using it to perform a probit regression on the same simulated data as before (and this is not the canonical link, so we need this more general form of IRLS).

my_IRLS_general <- function(x, y, b.init, h, h.prime, tol=1e-8) {
  change <- Inf
  b.old <- b.init
  while(change > tol) {
    eta <- x %*% b.old  # linear predictor
    y.hat <- h(eta)
    h.prime_eta <- h.prime(eta)
    w_star <- h.prime_eta^2 / (y.hat * (1 - y.hat))
    z_star <- (y - y.hat) / h.prime_eta

    b.new <- b.old + lm(z_star ~ x - 1, weights = w_star)$coef  # WLS

    change <- sqrt(sum((b.new - b.old)^2))
    b.old <- b.new
  }
  b.new
}

# probit inverse link and derivative
h_probit <- function(x) pnorm(x, 0, 1)
h.prime_probit <- function(x) dnorm(x, 0, 1)

my_IRLS_general(x, y, rep(0,p), h_probit, h.prime_probit)
# x1         x2         x3         x4         x5 
# -0.6456508  1.2520266  0.5820856  0.4982678 -0.6768585 

glm(y~x-1, family=binomial(link="probit"))$coef
# x1         x2         x3         x4         x5 
# -0.6456490  1.2520241  0.5820835  0.4982663 -0.6768581 

and again the two agree.

---

Comments on convergence

Finally, a few quick comments on convergence (I'll keep this brief as this is getting really long and I'm no expert at optimization). Even though theoretically each $J_{(m)}$ is negative definite, bad initial conditions can still prevent this algorithm from converging. In the probit example above, changing the initial conditions to b.init=rep(1,p) results in this, and that doesn't even look like a suspicious initial condition. If you step through the IRLS procedure with that initialization and these simulated data, by the second time through the loop there are some $\hat y_i$ that round to exactly $1$ and so the weights become undefined. If we're using the canonical link in the algorithm I gave we won't ever be dividing by $\hat y_i (1 - \hat y_i)$ to get undefined weights, but if we've got a situation where some $\hat y_i$ are approaching $0$ or $1$, such as in the case of perfect separation, then we'll still get non-convergence as the gradient dies without us reaching anything.