读取给定文本文件中的所有单词，并打印每个单词的计数

Question

你知道吗测试.txt包含以下句子（如果一只土拨鼠可以的话，一只土拨鼠需要多少木头 （查克·伍德）

这个程序应该读取给定文本文件中的所有单词（直到eof） 并打印出每个单词的计数。这个词应该是 不区分大小写（所有大写），标点符号应为 移除，输出应按 频率。你知道吗

不过，我遇到了一个简单的问题，那就是数台词，而不是字数，帮帮一个兄弟。你知道吗

Make a translation table for getting rid of non-word characters

dropChars = "!@#$%ˆ& ()_+-={}[]|\\:;\"’<>,.?/1234567890"
dropDict = dict([(c, '') for c in dropChars])
dropTable = str.maketrans(dropDict)

Read a file and build the table.

f = open("Test.txt")
testList=list()
lineNum = 0
table = {} # dictionary: words -> set of line numbers
for line in f:
    testList.append(line)
for line in testList :
    lineNum += 1
    words = line.upper().translate(dropTable).split()
    for word in words:
        if word in table:
            table[word].add(lineNum)
        else:
            table[word] = {lineNum}
f.close()

Print the table

for word in sorted(table.keys()):
    print(word, end = ": ")
    for lineNum in sorted(table[word]):
        print(lineNum, end = " ")
    print()

Answer 1

首先，你必须决定你对一个词的定义是什么。

定义1：单词是由空格分隔的字符序列。所以“you've”是一个单词，“o'clock”也是一个单词。你知道吗

定义2：一个词是“一个独特的有意义的元素的讲话或写作”。在这种情况下，“you've”是两个不同的单词（you+have），“o'clock”是一个单词。你知道吗

因此，如果您运行：

import string
import re
import nltk
import pandas as pd

s = "How much wood would a woodchuck chuck if a woodchuck could chuck wood. \n And also another line you've read from the file with something else. I wake up daily before eight o'clock."

def tokenize(text,semantic=True,sep=" "):
    if semantic:
        #Definition 2
        return nltk.word_tokenize(text)
    else:
        #Definition 1
        return [x for x in text.split(sep) ]

def remove_punctuation(text):
    pattern = re.compile('[{}]'.format(re.escape(string.punctuation)))
    return list(filter(None, [pattern.sub('',token) for token in text]))

def lowercase(text):
    return [token.lower() for token in text]

result = nltk.FreqDist(remove_punctuation(lowercase(tokenize(s)))).most_common()

table = pd.DataFrame(result)

table.to_csv('result.csv')

然后您将得到以下csv文件：

注意“ve”（来自“you've”）是一个独立的词。你知道吗

但是如果在标记化中将semantic=True改为semantic=False

result = nltk.FreqDist(remove_punctuation(lowercase(tokenize(s,semantic=False)))).most_common()

然后你会得到：

然而，在我们的频率表中将“have”写成“ve”并不是很人性化。我们可以利用收缩映射来解决这个问题。你知道吗

import string
import re
import nltk
import pandas as pd

s = "How much wood would a woodchuck chuck if a woodchuck could chuck wood. \n And also another line you've read from the file with something else. I wake up daily before eight o'clock."

CONTRACTION_MAP = {"ain't": "is not", "aren't": "are not","can't": "cannot", 
                   "can't've": "cannot have", "'cause": "because", "could've": "could have", 
                   "couldn't": "could not", "couldn't've": "could not have","didn't": "did not", 
                   "doesn't": "does not", "don't": "do not", "hadn't": "had not", 
                   "hadn't've": "had not have", "hasn't": "has not", "haven't": "have not", 
                   "he'd": "he would", "he'd've": "he would have", "he'll": "he will", 
                   "he'll've": "he he will have", "he's": "he is", "how'd": "how did", 
                   "how'd'y": "how do you", "how'll": "how will", "how's": "how is", 
                   "I'd": "I would", "I'd've": "I would have", "I'll": "I will", 
                   "I'll've": "I will have","I'm": "I am", "I've": "I have", 
                   "i'd": "i would", "i'd've": "i would have", "i'll": "i will", 
                   "i'll've": "i will have","i'm": "i am", "i've": "i have", 
                   "isn't": "is not", "it'd": "it would", "it'd've": "it would have", 
                   "it'll": "it will", "it'll've": "it will have","it's": "it is", 
                   "let's": "let us", "ma'am": "madam", "mayn't": "may not", 
                   "might've": "might have","mightn't": "might not","mightn't've": "might not have", 
                   "must've": "must have", "mustn't": "must not", "mustn't've": "must not have", 
                   "needn't": "need not", "needn't've": "need not have","o'clock": "of the clock", 
                   "oughtn't": "ought not", "oughtn't've": "ought not have", "shan't": "shall not",
                   "sha'n't": "shall not", "shan't've": "shall not have", "she'd": "she would", 
                   "she'd've": "she would have", "she'll": "she will", "she'll've": "she will have", 
                   "she's": "she is", "should've": "should have", "shouldn't": "should not", 
                   "shouldn't've": "should not have", "so've": "so have","so's": "so as", 
                   "this's": "this is",
                   "that'd": "that would", "that'd've": "that would have","that's": "that is", 
                   "there'd": "there would", "there'd've": "there would have","there's": "there is", 
                   "they'd": "they would", "they'd've": "they would have", "they'll": "they will", 
                   "they'll've": "they will have", "they're": "they are", "they've": "they have", 
                   "to've": "to have", "wasn't": "was not", "we'd": "we would", 
                   "we'd've": "we would have", "we'll": "we will", "we'll've": "we will have", 
                   "we're": "we are", "we've": "we have", "weren't": "were not", 
                   "what'll": "what will", "what'll've": "what will have", "what're": "what are", 
                   "what's": "what is", "what've": "what have", "when's": "when is", 
                   "when've": "when have", "where'd": "where did", "where's": "where is", 
                   "where've": "where have", "who'll": "who will", "who'll've": "who will have", 
                   "who's": "who is", "who've": "who have", "why's": "why is", 
                   "why've": "why have", "will've": "will have", "won't": "will not", 
                   "won't've": "will not have", "would've": "would have", "wouldn't": "would not", 
                   "wouldn't've": "would not have", "y'all": "you all", "y'all'd": "you all would",
                   "y'all'd've": "you all would have","y'all're": "you all are","y'all've": "you all have",
                   "you'd": "you would", "you'd've": "you would have", "you'll": "you will", 
                   "you'll've": "you will have", "you're": "you are", "you've": "you have" } 

# Credit for this function: https://www.kaggle.com/saxinou/nlp-01-preprocessing-data
def expand_contractions(sentence, contraction_mapping): 

    contractions_pattern = re.compile('({})'.format('|'.join(contraction_mapping.keys())),  
                                      flags=re.IGNORECASE|re.DOTALL) 
    def expand_match(contraction): 
        match = contraction.group(0) 
        first_char = match[0] 
        expanded_contraction = contraction_mapping.get(match) if contraction_mapping.get(match) else contraction_mapping.get(match.lower())                        
        expanded_contraction = first_char+expanded_contraction[1:] 
        return expanded_contraction 

    expanded_sentence = contractions_pattern.sub(expand_match, sentence) 
    return expanded_sentence 

def tokenize(text,semantic=True,sep=" "):
    if semantic:
        #Definition 2
        return nltk.word_tokenize(text)
    else:
        #Definition 1
        return [x for x in text.split(sep) ]

def remove_punctuation(text):
    pattern = re.compile('[{}]'.format(re.escape(string.punctuation)))
    return list(filter(None, [pattern.sub('',token) for token in text]))

def lowercase(text):
    return [token.lower() for token in text]

result = nltk.FreqDist(remove_punctuation(lowercase(tokenize(expand_contractions(s,CONTRACTION_MAP))))).most_common()

table = pd.DataFrame(result)

table.to_csv('result.csv')

那么问题就解决了。你知道吗

Answer 2

f=打开（'测试.txt'）

cnt=0 对于f.read（）.split（）中的单词： 打印（word） cnt+=1 打印cnt

这也许对你有帮助，兄弟…虽然我也是python的新手。你知道吗

Answer 3

此代码：

from collections import Counter
data = open( 'Test1.txt' ).read()  # read the file
data = ''.join( [i.upper() if i.isalpha() else ' ' for i in data] )   # remove the punctuation
c = Counter( data.split() )   # count the words
c.most_common()

印刷品：

[('A', 2), ('CHUCK', 2), ('WOODCHUCK', 2), ('WOOD', 2), ('WOULD', 1), ('COULD', 1), ('HOW', 1), ('MUCH', 1), ('IF', 1)]

我想知道代码是不是太短了？=）