Python:计算词频并将它们放入多个字典中

2024-06-16 13:56:50 发布

您现在位置:Python中文网/ 问答频道 /正文

我是python新手,现在我有一个txt文件如下:

Doc 1
aaa bbb ccc ddd ...

Doc 2
eee fff ggg hhh ...

Doc 3
aaa ggg iii kkk ...

...

Doc 11
eee ttt uuu zzz ...

基本上,我想做的是计算每个文档的术语频率,并将它们放入11个不同的字典中(比如“for Doc1,{aaa':10,'bbb':5…}),最后构建术语文档矩阵。我当前的代码如下:

# split te text file into 11 documents(paragraphs) 
f = open('filename.txt', 'r')
data = f.read()
docs = data.split("\n\n")

# creat 11 tf dictionaries
dictstr = 'tf'
dictlist = [dictstr + str(i) for i in range(10)]

for i in range(10):
    for line in docs[i]:
        tokens = line.split()
        for term in tokens:
            term = term.lower()
            term = term.replace(',', '')
            term = term.replace('"', '')
            term = term.replace('.', '')
            term = term.replace('/', '')
            term = term.replace('(', '')
            term = term.replace(')', '')

            if not term in dict['tfi']:
                dict['tfi'][term] = 1

            else:
                dict['tfi'][term] += 1

在最后的“如果-否则”步骤中有一些问题,我被困在这里了。有人能告诉我怎么处理吗?(不想使用“熊猫”之类的其他软件包)谢谢!
The txt resource's here


Tags: in文档txtfordocdictreplacesplit
1条回答
网友
1楼 · 发布于 2024-06-16 13:56:50

这段代码读入您提供的文件,一次性删除不需要的字符(与每次使用.replace创建新字符串相比),并将字数保存在名为result的dict中。键是doc nums('XXX9'->;'tf9'),值是带有单词计数的collections.Counter对象。你知道吗

>>> import re
... from collections import Counter
... 
... with open('filename.txt', 'r') as f:
...     data = f.read().lower()
... 
... clean_data = re.sub(r'[,"./()]', '', data)
... 
... result = {}
... for line in clean_data.splitlines():
...     if not line:
...         continue  # skip blank lines
...     elif line.startswith('xxx'):
...         doc_num = 'tf{}'.format(line[3:])
...     else:
...         result[doc_num] = Counter(line.split())
... 
>>> list(result.keys())
['tf7', 'tf10', 'tf5', 'tf2', 'tf9', 'tf4', 'tf11', 'tf3', 'tf6', 'tf8', 'tf1']

>>> for k, v in list(result['tf1'].items())[:15]:
...     print("'{}': {}".format(k, v))
... 
'class': 1
'then': 1
'emerge': 1
'industry': 1
'common': 1
'ourselves': 2
'models': 1
'short': 1
'mgi': 1
'it': 1
'actionable': 1
'time': 1
'why': 1
'theory': 1
'equip': 2

如果需要做任何更改来帮助回答您的问题,请告诉我!你知道吗

相关问题 更多 >