<p>我们可以创建一个定制的tfidf转换器,它可以接受一个列数组,然后在应用<code>.fit()</code>或<code>.transform()</code>之前连接它们。在</p>
<p>试试这个!在</p>
<pre><code>from sklearn.base import BaseEstimator,TransformerMixin
class custom_tfidf(BaseEstimator,TransformerMixin):
def __init__(self,tfidf):
self.tfidf = tfidf
def fit(self, X, y=None):
joined_X = X.apply(lambda x: ' '.join(x), axis=1)
self.tfidf.fit(joined_X)
return self
def transform(self, X):
joined_X = X.apply(lambda x: ' '.join(x), axis=1)
return self.tfidf.transform(joined_X)
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.compose import ColumnTransformer
dataset = pd.DataFrame({"a":["word gone wild","word gone with wind"],
"b":[" gone fhgf wild","gone with wind"],
"c":[1,2]})
tfidf = TfidfVectorizer(min_df=0)
clmn = ColumnTransformer([("tfidf", custom_tfidf(tfidf), ['a','b'])],remainder="passthrough")
clmn.fit_transform(dataset)
#
array([[0.36439074, 0.51853403, 0.72878149, 0. , 0. ,
0.25926702, 1. ],
[0. , 0.438501 , 0. , 0.61629785, 0.61629785,
0.2192505 , 2. ]])
</code></pre>
<p>注:可能需要为每个列创建一个tfidf向量器,然后创建一个字典,其中key作为列名,value作为fitted vectorizer。此字典可在转换相应列时使用</p>