<p>首先,你必须决定你对<strong>一个词的定义是什么。</strong></p>
<p>定义1:单词是由空格分隔的字符序列。所以“you've”是一个单词,“o'clock”也是一个单词。你知道吗</p>
<p>定义2:一个词是“一个独特的有意义的元素的讲话或写作”。在这种情况下,“you've”是两个不同的单词(you+have),“o'clock”是一个单词。你知道吗</p>
<p>因此,如果您运行:</p>
<pre><code>import string
import re
import nltk
import pandas as pd
s = "How much wood would a woodchuck chuck if a woodchuck could chuck wood. \n And also another line you've read from the file with something else. I wake up daily before eight o'clock."
def tokenize(text,semantic=True,sep=" "):
if semantic:
#Definition 2
return nltk.word_tokenize(text)
else:
#Definition 1
return [x for x in text.split(sep) ]
def remove_punctuation(text):
pattern = re.compile('[{}]'.format(re.escape(string.punctuation)))
return list(filter(None, [pattern.sub('',token) for token in text]))
def lowercase(text):
return [token.lower() for token in text]
result = nltk.FreqDist(remove_punctuation(lowercase(tokenize(s)))).most_common()
table = pd.DataFrame(result)
table.to_csv('result.csv')
</code></pre>
<p>然后您将得到以下csv文件:</p>
<p><a href="https://i.stack.imgur.com/58uCE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/58uCE.png" alt="enter image description here"/></a></p>
<p>注意“ve”(来自“you've”)是一个独立的词。你知道吗</p>
<p>但是如果在标记化中将semantic=True改为semantic=False</p>
<pre><code>result = nltk.FreqDist(remove_punctuation(lowercase(tokenize(s,semantic=False)))).most_common()
</code></pre>
<p>然后你会得到:</p>
<p><a href="https://i.stack.imgur.com/wfp2w.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wfp2w.png" alt="enter image description here"/></a></p>
<p>然而,在我们的频率表中将“have”写成“ve”并不是很人性化。我们可以利用收缩映射来解决这个问题。你知道吗</p>
<pre><code>import string
import re
import nltk
import pandas as pd
s = "How much wood would a woodchuck chuck if a woodchuck could chuck wood. \n And also another line you've read from the file with something else. I wake up daily before eight o'clock."
CONTRACTION_MAP = {"ain't": "is not", "aren't": "are not","can't": "cannot",
"can't've": "cannot have", "'cause": "because", "could've": "could have",
"couldn't": "could not", "couldn't've": "could not have","didn't": "did not",
"doesn't": "does not", "don't": "do not", "hadn't": "had not",
"hadn't've": "had not have", "hasn't": "has not", "haven't": "have not",
"he'd": "he would", "he'd've": "he would have", "he'll": "he will",
"he'll've": "he he will have", "he's": "he is", "how'd": "how did",
"how'd'y": "how do you", "how'll": "how will", "how's": "how is",
"I'd": "I would", "I'd've": "I would have", "I'll": "I will",
"I'll've": "I will have","I'm": "I am", "I've": "I have",
"i'd": "i would", "i'd've": "i would have", "i'll": "i will",
"i'll've": "i will have","i'm": "i am", "i've": "i have",
"isn't": "is not", "it'd": "it would", "it'd've": "it would have",
"it'll": "it will", "it'll've": "it will have","it's": "it is",
"let's": "let us", "ma'am": "madam", "mayn't": "may not",
"might've": "might have","mightn't": "might not","mightn't've": "might not have",
"must've": "must have", "mustn't": "must not", "mustn't've": "must not have",
"needn't": "need not", "needn't've": "need not have","o'clock": "of the clock",
"oughtn't": "ought not", "oughtn't've": "ought not have", "shan't": "shall not",
"sha'n't": "shall not", "shan't've": "shall not have", "she'd": "she would",
"she'd've": "she would have", "she'll": "she will", "she'll've": "she will have",
"she's": "she is", "should've": "should have", "shouldn't": "should not",
"shouldn't've": "should not have", "so've": "so have","so's": "so as",
"this's": "this is",
"that'd": "that would", "that'd've": "that would have","that's": "that is",
"there'd": "there would", "there'd've": "there would have","there's": "there is",
"they'd": "they would", "they'd've": "they would have", "they'll": "they will",
"they'll've": "they will have", "they're": "they are", "they've": "they have",
"to've": "to have", "wasn't": "was not", "we'd": "we would",
"we'd've": "we would have", "we'll": "we will", "we'll've": "we will have",
"we're": "we are", "we've": "we have", "weren't": "were not",
"what'll": "what will", "what'll've": "what will have", "what're": "what are",
"what's": "what is", "what've": "what have", "when's": "when is",
"when've": "when have", "where'd": "where did", "where's": "where is",
"where've": "where have", "who'll": "who will", "who'll've": "who will have",
"who's": "who is", "who've": "who have", "why's": "why is",
"why've": "why have", "will've": "will have", "won't": "will not",
"won't've": "will not have", "would've": "would have", "wouldn't": "would not",
"wouldn't've": "would not have", "y'all": "you all", "y'all'd": "you all would",
"y'all'd've": "you all would have","y'all're": "you all are","y'all've": "you all have",
"you'd": "you would", "you'd've": "you would have", "you'll": "you will",
"you'll've": "you will have", "you're": "you are", "you've": "you have" }
# Credit for this function: https://www.kaggle.com/saxinou/nlp-01-preprocessing-data
def expand_contractions(sentence, contraction_mapping):
contractions_pattern = re.compile('({})'.format('|'.join(contraction_mapping.keys())),
flags=re.IGNORECASE|re.DOTALL)
def expand_match(contraction):
match = contraction.group(0)
first_char = match[0]
expanded_contraction = contraction_mapping.get(match) if contraction_mapping.get(match) else contraction_mapping.get(match.lower())
expanded_contraction = first_char+expanded_contraction[1:]
return expanded_contraction
expanded_sentence = contractions_pattern.sub(expand_match, sentence)
return expanded_sentence
def tokenize(text,semantic=True,sep=" "):
if semantic:
#Definition 2
return nltk.word_tokenize(text)
else:
#Definition 1
return [x for x in text.split(sep) ]
def remove_punctuation(text):
pattern = re.compile('[{}]'.format(re.escape(string.punctuation)))
return list(filter(None, [pattern.sub('',token) for token in text]))
def lowercase(text):
return [token.lower() for token in text]
result = nltk.FreqDist(remove_punctuation(lowercase(tokenize(expand_contractions(s,CONTRACTION_MAP))))).most_common()
table = pd.DataFrame(result)
table.to_csv('result.csv')
</code></pre>
<p>那么问题就解决了。你知道吗</p>
<p><a href="https://i.stack.imgur.com/6KpGJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6KpGJ.png" alt="enter image description here"/></a></p>