Logum中的灾难性取消

我正在尝试在相对精度较低的双精度浮点中实现以下功能：

l o g s u m (x, y) = \log (\exp (x) + \exp (y))

$\mathrm{logsum}(x,y) = \log(\exp(x) + \exp(y))$

这在统计应用程序中被广泛使用以添加在对数空间中表示的概率或概率密度。当然，或都很容易溢出或下溢，这很不好，因为日志空间最初是用来避免下溢的。这是典型的解决方案： $\exp(x)$ $\exp(y)$

l o g s u m (x, y) = x + l o g 1 p (\exp (y - x))

$\mathrm{logsum}(x,y) = x + \mathrm{log1p}(\exp(y - x))$

Cancellation from $y-x$ does happen, but is mitigated by $\exp$ . Worse by far is when $x$ and $\mathrm{log1p}(\exp(y - x))$ are close. Here's a relative error plot:

enter image description here

The plot is cut off at $10^{-14}$ to emphasize the shape of the curve $\mathrm{logsum}(x,y) = 0$ , about which the cancellation occurs. I've seen error up to $10^{-11}$ and suspect that it gets much worse. (FWIW, the "ground truth" function is implemented using MPFR's arbitrary-precision floats with 128-bit precision.)

I've tried other reformulations, all with the same result. With $\log$ as the outer expression, the same error occurs by taking a log of something near 1. With $\mathrm{log1p}$ as the outer expression, cancellation happens in the inner expression.

Now, the absolute error is very small, so $\exp(\mathrm{logsum}(x,y))$ has very small relative error (within an epsilon). One might argue that, because a user of $\mathrm{logsum}$ is really interested in probabilities (not log probabilities), this terrible relative error isn't a problem. It's likely that it usually isn't, but I'm writing a library function, and I'd like its clients to be able to count on relative error not much worse than rounding error.

It seems I need a new approach. What might it be?

floating-point stability numerics

— Neil Toronto
source

I don't understand your last paragraph. "within an epsilon" doesn't mean anything to me. Do you mean a Unit in the Last Place? As for users being interested in probabilities, a small log probability error will result in a large probability error, so this is not the case.

— Aron Ahmadia

Out of curiosity, have you tried taking the "best of" of your two methods and plotting the error of that? Then all you need is the right logic to detect which case you're in (hopefully being less costly or part of the required cost of the algorithm anyway), then switching to the appropriate method.

— Aron Ahmadia

@AronAhmadia: "Within an epsilon" means relative error less than a double-precision floating-point epsilon, which is about 2.22e-16. For normal (i.e. not subnormal) floats, it corresponds to about an ulp. Also, if

a

$a$ is the absolute error of

x

$x$ , then the relative error of

\exp (x)

$\exp(x)$ is

\exp (a) - 1

$\exp(a)-1$ , which is nearly the identity function near zero. IOW, small absolute error for

x

$x$ implies small relative error for

\exp (x)

$\exp(x)$ .

— Neil Toronto

Addendum: When absolute error

a

$a$ is near zero. When

a > 1

$a > 1$ , for example, you're right: the relative explodes.

— Neil Toronto

The formula

l o g s u m (x, y) = max (x, y) + l o g 1 p (\exp (- abs (x - y))

$\mathrm{logsum}(x,y)=\max(x,y)+\mathrm{log1p}(\exp(-\operatorname{abs}(x-y))$ should be numerically stable. It generalizes to a numerically stable computation of

\log \sum_{i} e^{x_{i}} = ξ + \log \sum_{i} e^{x_{i} - ξ}, ξ = max_{i} x_{i}

$\log \sum_i e^{x_i} = \xi+ \log\sum_i e^{x_i-\xi},~~~\xi=\max_i x_i$

In case the logsum is very close to zero and you want high relative accuracy, you can probably use

l o g s u m (x, y) = max (x, y) + l e x p (x - y)

$\mathrm{logsum}(x,y)=\max(x,y)+\mathrm{lexp}(x-y)$ using an accurate (i.e., more than double precision) implementation of

l e x p (z) := \log (1 + e^{- | z |})

$\mathrm{lexp}(z):=\log(1+e^{-|z|})$ which is nearly linear for small

z

$z$ .

— Arnold Neumaier
source

In terms of absolute error, it is. In terms of relative error, it's awful when the output is near zero.

— Neil Toronto

@NeilToronto: Please give an example with two explicit inputs

x

$x$ and

y

$y$ , so that I can play with it.

— Arnold Neumaier

For x = -0.775 and y = -0.6175, I get 62271 ulps error and 1.007e-11 relative error.

— Neil Toronto

Calculate highly accurate data points in the range of interest - at least ttwo different ranges are needed because of the asymptotic behavior. One can use the defining expression for z not close to zero. For the exceptional range fit a rational function of sufficiently high degree to it to get the desired accuracy. For numerical stability, use bernstein polynomials or Tchebychev polynomials in numerator and denominator, adapted to the interval of interest. At the end, expand into a continued fraction and find out how much one can truncate the coefficients without imapiring the accuracy.

— Arnold Neumaier

This gives

l = l (z)

$l=l(z)$ To get

m

$m$ do the same but applied to the function lexp(z)-l(z).

— Arnold Neumaier