如何从pandas DataFrame中选择一个或多个null的行而不显式列出列?


232

我有一个约30万行和约40列的数据框。我想找出是否有任何行包含空值-并将这些“空”行放入单独的数据框中,以便我可以轻松地探索它们。

我可以显式创建一个遮罩:

mask = False
for col in df.columns: 
    mask = mask | df[col].isnull()
dfnulls = df[mask]

或者我可以做类似的事情:

df.ix[df.index[(df.T == np.nan).sum() > 1]]

有没有更优雅的方法(找到行中包含null的行)?

Answers:


383

[已更新以适应现代pandas,它已isnull成为一种方法DataFrame。]

您可以使用isnullany构建布尔系列,并使用它来索引您的框架:

>>> df = pd.DataFrame([range(3), [0, np.NaN, 0], [0, 0, np.NaN], range(3), range(3)])
>>> df.isnull()
       0      1      2
0  False  False  False
1  False   True  False
2  False  False   True
3  False  False  False
4  False  False  False
>>> df.isnull().any(axis=1)
0    False
1     True
2     True
3    False
4    False
dtype: bool
>>> df[df.isnull().any(axis=1)]
   0   1   2
1  0 NaN   0
2  0   0 NaN

[较老pandas:]

您可以使用函数isnull代替方法:

In [56]: df = pd.DataFrame([range(3), [0, np.NaN, 0], [0, 0, np.NaN], range(3), range(3)])

In [57]: df
Out[57]: 
   0   1   2
0  0   1   2
1  0 NaN   0
2  0   0 NaN
3  0   1   2
4  0   1   2

In [58]: pd.isnull(df)
Out[58]: 
       0      1      2
0  False  False  False
1  False   True  False
2  False  False   True
3  False  False  False
4  False  False  False

In [59]: pd.isnull(df).any(axis=1)
Out[59]: 
0    False
1     True
2     True
3    False
4    False

导致相当紧凑:

In [60]: df[pd.isnull(df).any(axis=1)]
Out[60]: 
   0   1   2
1  0 NaN   0
2  0   0 NaN

75
def nans(df): return df[df.isnull().any(axis=1)]

然后,当您需要时可以键入:

nans(your_dataframe)

1
df[df.isnull().any(axis=1)]工作但抛出UserWarning: Boolean Series key will be reindexed to match DataFrame index.。如何以一种不触发该警告消息的方式更明确地重写此内容?
维沙尔'18

3
@vishal我想您需要做的就是添加loc这样的;df.loc[df.isnull().any(axis=1)]
詹姆斯·德雷珀

2
顺便说一句-您不应该命名您的匿名(lambda)函数。始终使用def语句而不是将lambda表达式直接绑定到标识符的赋值语句。
donrondadon

0

.any()并且.all()非常适合极端情况,但不适用于要查找特定数量的空值的情况。这是完成我认为您要问的事情的一种非常简单的方法。它很冗长,但很实用。

import pandas as pd
import numpy as np

# Some test data frame
df = pd.DataFrame({'num_legs':          [2, 4,      np.nan, 0, np.nan],
                   'num_wings':         [2, 0,      np.nan, 0, 9],
                   'num_specimen_seen': [10, np.nan, 1,     8, np.nan]})

# Helper : Gets NaNs for some row
def row_nan_sums(df):
    sums = []
    for row in df.values:
        sum = 0
        for el in row:
            if el != el: # np.nan is never equal to itself. This is "hacky", but complete.
                sum+=1
        sums.append(sum)
    return sums

# Returns a list of indices for rows with k+ NaNs
def query_k_plus_sums(df, k):
    sums = row_nan_sums(df)
    indices = []
    i = 0
    for sum in sums:
        if (sum >= k):
            indices.append(i)
        i += 1
    return indices

# test
print(df)
print(query_k_plus_sums(df, 2))

输出量

   num_legs  num_wings  num_specimen_seen
0       2.0        2.0               10.0
1       4.0        0.0                NaN
2       NaN        NaN                1.0
3       0.0        0.0                8.0
4       NaN        9.0                NaN
[2, 4]

然后,如果您像我一样,并且想要清除这些行,则只需编写以下代码:

# drop the rows from the data frame
df.drop(query_k_plus_sums(df, 2),inplace=True)
# Reshuffle up data (if you don't do this, the indices won't reset)
df = df.sample(frac=1).reset_index(drop=True)
# print data frame
print(df)

输出:

   num_legs  num_wings  num_specimen_seen
0       4.0        0.0                NaN
1       0.0        0.0                8.0
2       2.0        2.0               10.0
By using our site, you acknowledge that you have read and understand our Cookie Policy and Privacy Policy.
Licensed under cc by-sa 3.0 with attribution required.