在scikit学习LinearRegression中找到p值(重要性)


Answers:


162

这有点矫kill过正,但让我们尝试一下。首先让我们使用statsmodel找出p值应该是什么

import pandas as pd
import numpy as np
from sklearn import datasets, linear_model
from sklearn.linear_model import LinearRegression
import statsmodels.api as sm
from scipy import stats

diabetes = datasets.load_diabetes()
X = diabetes.data
y = diabetes.target

X2 = sm.add_constant(X)
est = sm.OLS(y, X2)
est2 = est.fit()
print(est2.summary())

我们得到

                         OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.518
Model:                            OLS   Adj. R-squared:                  0.507
Method:                 Least Squares   F-statistic:                     46.27
Date:                Wed, 08 Mar 2017   Prob (F-statistic):           3.83e-62
Time:                        10:08:24   Log-Likelihood:                -2386.0
No. Observations:                 442   AIC:                             4794.
Df Residuals:                     431   BIC:                             4839.
Df Model:                          10                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const        152.1335      2.576     59.061      0.000     147.071     157.196
x1           -10.0122     59.749     -0.168      0.867    -127.448     107.424
x2          -239.8191     61.222     -3.917      0.000    -360.151    -119.488
x3           519.8398     66.534      7.813      0.000     389.069     650.610
x4           324.3904     65.422      4.958      0.000     195.805     452.976
x5          -792.1842    416.684     -1.901      0.058   -1611.169      26.801
x6           476.7458    339.035      1.406      0.160    -189.621    1143.113
x7           101.0446    212.533      0.475      0.635    -316.685     518.774
x8           177.0642    161.476      1.097      0.273    -140.313     494.442
x9           751.2793    171.902      4.370      0.000     413.409    1089.150
x10           67.6254     65.984      1.025      0.306     -62.065     197.316
==============================================================================
Omnibus:                        1.506   Durbin-Watson:                   2.029
Prob(Omnibus):                  0.471   Jarque-Bera (JB):                1.404
Skew:                           0.017   Prob(JB):                        0.496
Kurtosis:                       2.726   Cond. No.                         227.
==============================================================================

好的,让我们重现一下。这有点过头了,因为我们几乎要使用矩阵代数重现线性回归分析。但是到底。

lm = LinearRegression()
lm.fit(X,y)
params = np.append(lm.intercept_,lm.coef_)
predictions = lm.predict(X)

newX = pd.DataFrame({"Constant":np.ones(len(X))}).join(pd.DataFrame(X))
MSE = (sum((y-predictions)**2))/(len(newX)-len(newX.columns))

# Note if you don't want to use a DataFrame replace the two lines above with
# newX = np.append(np.ones((len(X),1)), X, axis=1)
# MSE = (sum((y-predictions)**2))/(len(newX)-len(newX[0]))

var_b = MSE*(np.linalg.inv(np.dot(newX.T,newX)).diagonal())
sd_b = np.sqrt(var_b)
ts_b = params/ sd_b

p_values =[2*(1-stats.t.cdf(np.abs(i),(len(newX)-len(newX[0])))) for i in ts_b]

sd_b = np.round(sd_b,3)
ts_b = np.round(ts_b,3)
p_values = np.round(p_values,3)
params = np.round(params,4)

myDF3 = pd.DataFrame()
myDF3["Coefficients"],myDF3["Standard Errors"],myDF3["t values"],myDF3["Probabilities"] = [params,sd_b,ts_b,p_values]
print(myDF3)

这给了我们。

    Coefficients  Standard Errors  t values  Probabilities
0       152.1335            2.576    59.061         0.000
1       -10.0122           59.749    -0.168         0.867
2      -239.8191           61.222    -3.917         0.000
3       519.8398           66.534     7.813         0.000
4       324.3904           65.422     4.958         0.000
5      -792.1842          416.684    -1.901         0.058
6       476.7458          339.035     1.406         0.160
7       101.0446          212.533     0.475         0.635
8       177.0642          161.476     1.097         0.273
9       751.2793          171.902     4.370         0.000
10       67.6254           65.984     1.025         0.306

因此,我们可以从statsmodel复制值。


2
我的var_b都是Nans是什么意思?线性代数部分失败有任何根本原因吗?
famargar

很难猜测为什么会这样。我将查看您的数据结构,并将其与示例进行比较。这可能提供了一个线索。
JARH

1
看起来codenp.linalg.inv有时即使矩阵不可逆也可以返回结果。这可能是问题所在。
JARH

6
@famargar我也遇到了所有问题nan。对我来说,是因为我X的是我的数据样本,所以该索引已关闭。调用时会导致错误pd.DataFrame.join()。我进行了这一行更改,现在似乎可以使用:newX = pd.DataFrame({"Constant":np.ones(len(X))}).join(pd.DataFrame(X.reset_index(drop=True)))
pault

1
@ mLstudent33“概率”列。
skeller88

52

scikit-learn的LinearRegression不会计算此信息,但是您可以轻松地扩展该类来做到这一点:

from sklearn import linear_model
from scipy import stats
import numpy as np


class LinearRegression(linear_model.LinearRegression):
    """
    LinearRegression class after sklearn's, but calculate t-statistics
    and p-values for model coefficients (betas).
    Additional attributes available after .fit()
    are `t` and `p` which are of the shape (y.shape[1], X.shape[1])
    which is (n_features, n_coefs)
    This class sets the intercept to 0 by default, since usually we include it
    in X.
    """

    def __init__(self, *args, **kwargs):
        if not "fit_intercept" in kwargs:
            kwargs['fit_intercept'] = False
        super(LinearRegression, self)\
                .__init__(*args, **kwargs)

    def fit(self, X, y, n_jobs=1):
        self = super(LinearRegression, self).fit(X, y, n_jobs)

        sse = np.sum((self.predict(X) - y) ** 2, axis=0) / float(X.shape[0] - X.shape[1])
        se = np.array([
            np.sqrt(np.diagonal(sse[i] * np.linalg.inv(np.dot(X.T, X))))
                                                    for i in range(sse.shape[0])
                    ])

        self.t = self.coef_ / se
        self.p = 2 * (1 - stats.t.cdf(np.abs(self.t), y.shape[0] - X.shape[1]))
        return self

这里被盗。

您应该看一下statsmodels,以便在Python中进行这种统计分析。


好。这不会起作用,因为sse是一个标量,所以sse.shape实际上并不意味着任何东西。
阿舒(Ashu)

15

编辑:可能不是正确的方法,请参阅评论

您可以使用sklearn.feature_selection.f_regression。

单击此处获取scikit学习页面


1
那是F检验吗?我认为线性回归的p值通常是针对每个回归变量的,这是一个测试,系数的零为0?为了获得一个好的答案,有必要对功能进行更多的解释。
wordforthewise

@wordsforthewise文档页面说返回的值是p_values的数组。因此,对于每个单独的回归变量而言,这确实是一个价值。
ashu

1
不要使用此方法,因为它不正确!它执行单变量回归,但是您可能想要一个多元回归
user357269 '19

1
不,不要使用f_regression。拟合数据后,每个系数的实际p值应来自每个系数的t检验。sklearn中的f_regression来自单变量回归。它没有建立模式,只是计算每个变量的f得分。与sklearn中的chi2函数相同这是正确的:将statsmodels.api导入为sm mod = sm.OLS(Y,X)
Richard Liang

@RichardLiang,使用sm.OLS()是为任何算法计算p值(多变量)的正确方法吗?(例如决策树,svm,k均值,逻辑回归等)?我想要一种获取p值的通用方法。谢谢
Gilian

11

elyase的答案https://stackoverflow.com/a/27928411/4240413中的代码实际上无效。请注意,sse是一个标量,然后尝试对其进行迭代。以下代码是修改后的版本。并不是很干净,但是我认为它或多或少地起作用。

class LinearRegression(linear_model.LinearRegression):

    def __init__(self,*args,**kwargs):
        # *args is the list of arguments that might go into the LinearRegression object
        # that we don't know about and don't want to have to deal with. Similarly, **kwargs
        # is a dictionary of key words and values that might also need to go into the orginal
        # LinearRegression object. We put *args and **kwargs so that we don't have to look
        # these up and write them down explicitly here. Nice and easy.

        if not "fit_intercept" in kwargs:
            kwargs['fit_intercept'] = False

        super(LinearRegression,self).__init__(*args,**kwargs)

    # Adding in t-statistics for the coefficients.
    def fit(self,x,y):
        # This takes in numpy arrays (not matrices). Also assumes you are leaving out the column
        # of constants.

        # Not totally sure what 'super' does here and why you redefine self...
        self = super(LinearRegression, self).fit(x,y)
        n, k = x.shape
        yHat = np.matrix(self.predict(x)).T

        # Change X and Y into numpy matricies. x also has a column of ones added to it.
        x = np.hstack((np.ones((n,1)),np.matrix(x)))
        y = np.matrix(y).T

        # Degrees of freedom.
        df = float(n-k-1)

        # Sample variance.     
        sse = np.sum(np.square(yHat - y),axis=0)
        self.sampleVariance = sse/df

        # Sample variance for x.
        self.sampleVarianceX = x.T*x

        # Covariance Matrix = [(s^2)(X'X)^-1]^0.5. (sqrtm = matrix square root.  ugly)
        self.covarianceMatrix = sc.linalg.sqrtm(self.sampleVariance[0,0]*self.sampleVarianceX.I)

        # Standard erros for the difference coefficients: the diagonal elements of the covariance matrix.
        self.se = self.covarianceMatrix.diagonal()[1:]

        # T statistic for each beta.
        self.betasTStat = np.zeros(len(self.se))
        for i in xrange(len(self.se)):
            self.betasTStat[i] = self.coef_[0,i]/self.se[i]

        # P-value for each beta. This is a two sided t-test, since the betas can be 
        # positive or negative.
        self.betasPValue = 1 - t.cdf(abs(self.betasTStat),df)

8

拉取p值的一种简单方法是使用statsmodels回归:

import statsmodels.api as sm
mod = sm.OLS(Y,X)
fii = mod.fit()
p_values = fii.summary2().tables[1]['P>|t|']

您将获得一系列可以操纵的p值(例如,通过评估每个p值来选择要保留的顺序):

在此处输入图片说明


使用sm.OLS()是计算任何算法的p值(多变量)的正确方法吗?(例如决策树,svm,k均值,逻辑回归等)?我想要一种获取p值的通用方法。谢谢
吉莲

7

p_value在f统计信息中。如果要获取值,只需使用以下几行代码:

import statsmodels.api as sm
from scipy import stats

diabetes = datasets.load_diabetes()
X = diabetes.data
y = diabetes.target

X2 = sm.add_constant(X)
est = sm.OLS(y, X2)
print(est.fit().f_pvalue)

3
这不能回答问题,因为您使用的库与上述库不同。
绅士

@gented在哪种情况下一种计算方法会比另一种更好?
唐吉

6

在多变量回归的情况下,@ JARH的答案可能有误。(我没有足够的声誉来发表评论。)

在以下行中:

p_values =[2*(1-stats.t.cdf(np.abs(i),(len(newX)-1))) for i in ts_b]

t值遵循度的卡方分布len(newX)-1而不是度的卡方分布len(newX)-len(newX.columns)-1

所以这应该是:

p_values =[2*(1-stats.t.cdf(np.abs(i),(len(newX)-len(newX.columns)-1))) for i in ts_b]

(有关更多详细信息,请参见t值以进行OLS回归


5

您可以将scipy用作p值。此代码来自scipy文档。

>>> from scipy import stats
>>> import numpy as np
>>> x = np.random.random(10)
>>> y = np.random.random(10)
>>> slope, intercept, r_value, p_value, std_err = stats.linregress(x,y)

1
我认为这不适用于在拟合过程中使用的多个矢量
O.rka '18

1

对于单行代码,您可以使用pingouin.linear_regression函数(免责声明:我是Pingouin的创建者),该函数可使用NumPy数组或Pandas DataFrame与单变量/多元回归配合使用,例如:

import pingouin as pg
# Using a Pandas DataFrame `df`:
lm = pg.linear_regression(df[['x', 'z']], df['y'])
# Using a NumPy array:
lm = pg.linear_regression(X, y)

输出是一个数据帧,其中包含每个预测变量的beta系数,标准误差,T值,p值和置信区间,以及拟合的R ^ 2和调整后的R ^ 2。

By using our site, you acknowledge that you have read and understand our Cookie Policy and Privacy Policy.
Licensed under cc by-sa 3.0 with attribution required.