读入SQL文件并使用计数向量器获取单词出现次数

2024-03-29 08:13:30 发布

您现在位置:Python中文网/ 问答频道 /正文

用一个字来读一个文件。你知道吗

到目前为止,我有以下代码:

import re
import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer




df = pd.read_sql(q, dlconn)
print(df)

count_vect = CountVectorizer()
X_train_counts= count_vect.fit_transform(df)

print(X_train_counts.shape)
print(count_vect.vocabulary_)

这将给出'cat': 1, 'dog': 0的输出

它似乎只取了animal列的名称并从那里开始计数。你知道吗

如何让它访问完整的列并得到一个显示列中每个单词及其频率的图表?你知道吗


Tags: 文件代码importrenumpypandasdfas
1条回答
网友
1楼 · 发布于 2024-03-29 08:13:30

根据the ^{} docs,方法fit_transform()需要字符串的iterable。 它不能直接处理DataFrame。你知道吗

但是在数据帧上迭代会返回列的标签,而不是值。我建议你试试^{}。你知道吗

试着这样做:

value_list = [
    row[0]
    for row in df.itertuples(index=False, name=None)]
print(value_list)
print(type(value_list))
print(type(value_list[0]))

X_train_counts = count_vect.fit_transform(value_list)

value_list中的每个值都应该是str类型。 如果有帮助请告诉我们。你知道吗


下面是一个小例子:

>>> import pandas as pd
>>> df = pd.DataFrame(['my big dog', 'my lazy cat'])
>>> df
             0
0   my big dog
1  my lazy cat

>>> value_list = [row[0] for row in df.itertuples(index=False, name=None)]
>>> value_list
['my big dog', 'my lazy cat']

>>> from sklearn.feature_extraction.text import CountVectorizer
>>> cv = CountVectorizer()
>>> x_train = cv.fit_transform(value_list)
>>> x_train
<2x5 sparse matrix of type '<class 'numpy.int64'>'
    with 6 stored elements in Compressed Sparse Row format>
>>> x_train.toarray()
array([[1, 0, 1, 0, 1],
       [0, 1, 0, 1, 1]], dtype=int64)
>>> cv.vocabulary_
{'my': 4, 'big': 0, 'dog': 2, 'lazy': 3, 'cat': 1}

现在可以显示每行的字数(每个输入字符串分别显示):

>>> for word, col in cv.vocabulary_.items():
...     for row in range(x_train.shape[0]):
...         print('word:{:10s} | row:{:2d} | count:{:2d}'.format(word, row, x_train[row,col]))
word:my         | row: 0 | count: 1
word:my         | row: 1 | count: 1
word:big        | row: 0 | count: 1
word:big        | row: 1 | count: 0
word:dog        | row: 0 | count: 1
word:dog        | row: 1 | count: 0
word:lazy       | row: 0 | count: 0
word:lazy       | row: 1 | count: 1
word:cat        | row: 0 | count: 0
word:cat        | row: 1 | count: 1

您还可以显示总字数(行总数):

>>> x_train_sum = x_train.sum(axis=0)
>>> x_train_sum
    matrix([[1, 1, 1, 1, 2]], dtype=int64)
>>> for word, col in cv.vocabulary_.items():
...     print('word:{:10s} | count:{:2d}'.format(word, x_train_sum[0, col]))
word:my         | count: 2
word:big        | count: 1
word:dog        | count: 1
word:lazy       | count: 1
word:cat        | count: 1

>>> with open('my-file.csv', 'w') as f:
...     for word, col in cv.vocabulary_.items():
...         f.write('{};{}\n'.format(word, x_train_sum[0, col]))

这应该说明你如何使用你拥有的工具。你知道吗

相关问题 更多 >