使用scikit-learn分为多个类别


80

我正在尝试使用scikit-learn的监督学习方法之一将文本片段分类为一个或多个类别。我尝试过的所有算法的预测功能仅返回一个匹配项。

例如,我有一段文字:

"Theaters in New York compared to those in London"

而且我已经训练了算法,可以为我输入的每个文本片段选择一个位置。

在上面的例子中,我希望它返回New YorkLondon,但是只返回New York

是否可以使用scikit-learn返回多个结果?甚至以下一个最高概率返回标签?

谢谢你的帮助。

-更新

我尝试使用,OneVsRestClassifier但每条文本我仍然只能得到一个选择。以下是我正在使用的示例代码

y_train = ('New York','London')


train_set = ("new york nyc big apple", "london uk great britain")
vocab = {'new york' :0,'nyc':1,'big apple':2,'london' : 3, 'uk': 4, 'great britain' : 5}
count = CountVectorizer(analyzer=WordNGramAnalyzer(min_n=1, max_n=2),vocabulary=vocab)
test_set = ('nice day in nyc','london town','hello welcome to the big apple. enjoy it here and london too')

X_vectorized = count.transform(train_set).todense()
smatrix2  = count.transform(test_set).todense()


base_clf = MultinomialNB(alpha=1)

clf = OneVsRestClassifier(base_clf).fit(X_vectorized, y_train)
Y_pred = clf.predict(smatrix2)
print Y_pred

结果:['纽约''伦敦''伦敦']

Answers:


111

您想要的就是多标签分类。Scikits-learn可以做到这一点。参见此处:http : //scikit-learn.org/dev/modules/multiclass.html

我不确定您的示例出了什么问题,我的sklearn版本显然没有WordNGramAnalyzer。也许这是使用更多训练示例或尝试使用其他分类器的问题?但是请注意,多标签分类器希望目标是元组/标签列表的列表。

以下对我有用:

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.multiclass import OneVsRestClassifier

X_train = np.array(["new york is a hell of a town",
                    "new york was originally dutch",
                    "the big apple is great",
                    "new york is also called the big apple",
                    "nyc is nice",
                    "people abbreviate new york city as nyc",
                    "the capital of great britain is london",
                    "london is in the uk",
                    "london is in england",
                    "london is in great britain",
                    "it rains a lot in london",
                    "london hosts the british museum",
                    "new york is great and so is london",
                    "i like london better than new york"])
y_train = [[0],[0],[0],[0],[0],[0],[1],[1],[1],[1],[1],[1],[0,1],[0,1]]
X_test = np.array(['nice day in nyc',
                   'welcome to london',
                   'hello welcome to new york. enjoy it here and london too'])   
target_names = ['New York', 'London']

classifier = Pipeline([
    ('vectorizer', CountVectorizer(min_n=1,max_n=2)),
    ('tfidf', TfidfTransformer()),
    ('clf', OneVsRestClassifier(LinearSVC()))])
classifier.fit(X_train, y_train)
predicted = classifier.predict(X_test)
for item, labels in zip(X_test, predicted):
    print '%s => %s' % (item, ', '.join(target_names[x] for x in labels))

对我来说,这将产生输出:

nice day in nyc => New York
welcome to london => London
hello welcome to new york. enjoy it here and london too => New York, London

希望这可以帮助。


1
我试着删除最后两个结合城市名称的培训示例,然后我得到了:欢迎来到纽约。在这里也可以在伦敦享受它=>纽约它不再返回两个标签。对我来说,如果我训练两个城市的组合,它只会返回两个标签。我想念什么吗?再次感谢您的所有帮助
-CodeMonkeyB

1
这只是一个玩具数据集,我不会从中得出太多结论。您是否对实际数据尝试过此过程?
mwv 2012年

3
@CodeMonkeyB:您应该真的接受这个答案,从编程的角度来看这是正确的。它在实践中是否有效取决于您的数据,而不是代码。
Fred Foo 2012年

2
还有其他人遇到与min_n和相关的问题吗max_n?我需要改变他们ngram_range=(1,2)工作
emmagras

1
出现此错误:ValueError:您似乎正在使用旧的多标签数据表示形式。不再支持序列序列;请改用二进制数组或稀疏矩阵。
MANU

61

编辑:建议使用MultiLabelBinarizer为Python 3,scikit-learn 0.18.1更新。

我也一直在努力,并且对mwv的出色回答(可能有用)进行了一些改进。它使用文本标签作为输入而不是二进制标签,并使用MultiLabelBinarizer对其进行编码。

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.multiclass import OneVsRestClassifier
from sklearn.preprocessing import MultiLabelBinarizer

X_train = np.array(["new york is a hell of a town",
                    "new york was originally dutch",
                    "the big apple is great",
                    "new york is also called the big apple",
                    "nyc is nice",
                    "people abbreviate new york city as nyc",
                    "the capital of great britain is london",
                    "london is in the uk",
                    "london is in england",
                    "london is in great britain",
                    "it rains a lot in london",
                    "london hosts the british museum",
                    "new york is great and so is london",
                    "i like london better than new york"])
y_train_text = [["new york"],["new york"],["new york"],["new york"],["new york"],
                ["new york"],["london"],["london"],["london"],["london"],
                ["london"],["london"],["new york","london"],["new york","london"]]

X_test = np.array(['nice day in nyc',
                   'welcome to london',
                   'london is rainy',
                   'it is raining in britian',
                   'it is raining in britian and the big apple',
                   'it is raining in britian and nyc',
                   'hello welcome to new york. enjoy it here and london too'])
target_names = ['New York', 'London']

mlb = MultiLabelBinarizer()
Y = mlb.fit_transform(y_train_text)

classifier = Pipeline([
    ('vectorizer', CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('clf', OneVsRestClassifier(LinearSVC()))])

classifier.fit(X_train, Y)
predicted = classifier.predict(X_test)
all_labels = mlb.inverse_transform(predicted)

for item, labels in zip(X_test, all_labels):
    print('{0} => {1}'.format(item, ', '.join(labels)))

这给了我以下输出:

nice day in nyc => new york
welcome to london => london
london is rainy => london
it is raining in britian => london
it is raining in britian and the big apple => new york
it is raining in britian and nyc => london, new york
hello welcome to new york. enjoy it here and london too => london, new york

13
labelBinarizer已经过时了。使用lb = preprocessing.MultiLabelBinarizer()替代
罗马

1
它没有给英国,因为唯一的输出标签是New YorkLondon
umop aplsdn

2
根据scikit-learn,除sklearn.svm.SVC之外,所有线性模型均支持One-Vs-All,并且以下各项还支持multilabel:决策树,随机森林,最近邻居,因此我不会将LinearSVC()用于此类型任务(又称多
标签

2
@mindstorm提到的Fyi One-Vs-All与scikit学习类“ OneVsRestClassifier”相对应(注意,“ Rest”而不是“ all”)。此scikit-learn帮助页面进行了说明。
lucid_dreamer

1
正如@mindstorm提到的,在此页面上,文档的确提到:“一个对所有:除sklearn.svm.SVC外的所有线性模型”。但是,来自scikit-learn文档的另一个多标签示例显示了带有此行的多标签示例classif = OneVsRestClassifier(SVC(kernel='linear'))。困惑。
lucid_dreamer

8

我也碰到了这个问题,对我来说,问题是我的y_Train是一个字符串序列,而不是一个字符串序列。显然,OneVsRestClassifier将基于输入标签格式来决定是使用多类还是多标签。所以改变:

y_train = ('New York','London')

y_train = (['New York'],['London'])

显然,这种情况将来会消失,因为所有标签的中断都相同:https : //github.com/scikit-learn/scikit-learn/pull/1987


8

更改此行以使其在新版本的python中工作

# lb = preprocessing.LabelBinarizer()
lb = preprocessing.MultiLabelBinarizer()

2

很少有多种分类,例如:

示例1:

import numpy as np
from sklearn.preprocessing import LabelBinarizer
encoder = LabelBinarizer()

arr2d = np.array([1, 2, 3,4,5,6,7,8,9,10,11,12,13,14,1])
transfomed_label = encoder.fit_transform(arr2d)
print(transfomed_label)

输出为

[[1 0 0 0 0 0 0 0 0 0 0 0 0 0]
 [0 1 0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 1 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 1 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 1 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 1 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 1 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 1 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 1 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 1 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 1 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 1 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0 1 0]
 [0 0 0 0 0 0 0 0 0 0 0 0 0 1]
 [1 0 0 0 0 0 0 0 0 0 0 0 0 0]]

示例2:

import numpy as np
from sklearn.preprocessing import LabelBinarizer
encoder = LabelBinarizer()

arr2d = np.array(['Leopard','Lion','Tiger', 'Lion'])
transfomed_label = encoder.fit_transform(arr2d)
print(transfomed_label)

输出为

[[1 0 0]
 [0 1 0]
 [0 0 1]
 [0 1 0]]
By using our site, you acknowledge that you have read and understand our Cookie Policy and Privacy Policy.
Licensed under cc by-sa 3.0 with attribution required.