如何总结对话中每个人的字数？

Question

我开始学习Python，并试图编写一个程序来导入一个文本文件，计算总字数，计算特定段落中的字数（由每个参与者说，用'P1'、'P2'等描述），从字数中排除这些字词（例如'P1'等），并分别打印段落。在

多亏了@James Hurford我得到了这个代码：

words = None
with open('data.txt') as f:
   words = f.read().split()
total_words = len(words)
print 'Total words:', total_words

in_para = False
para_type = None
paragraph = list()
for word in words:
  if ('P1' in word or
      'P2' in word or
      'P3' in word ):
      if in_para == False:
         in_para = True
         para_type = word
      else:
         print 'Words in paragraph', para_type, ':', len(paragraph)
         print ' '.join(paragraph)
         del paragraph[:]
         para_type = word
  else:
    paragraph.append(word)
else:
  if in_para == True:
    print 'Words in last paragraph', para_type, ':', len(paragraph)
    print ' '.join(paragraph)
  else:
    print 'No words'

我的文本文件如下所示：

P1: Bla bla bla.

P2: Bla bla bla bla.

P1: Bla bla.

P3: Bla.

下一步我要做的是总结每个参与者的单词。我只能打印，但我不知道如何归还/再利用它们。在

我需要一个新的变量，其中包含每个参与者的单词数，我可以在以后操作它，此外还可以总结每个参与者所说的单词，例如

^{pr2}$

有没有办法把“你”或“它”等算作两个词？在

有什么办法解决吗？在

Answer 1

I would need a new variable with word count for each participant that I could manipulate later on

不，您需要一个^{}（Python2.7+，否则使用defaultdict(int)）将人员映射到字数。在

from collections import Counter
#from collections import defaultdict

words_per_person = Counter()
#words_per_person = defaultdict(int)

for ln in inputfile:
    person, text = ln.split(':', 1)
    words_per_person[person] += len(text.split())

现在，words_per_person['P1']包含P1的字数，假设text.split()是一个足够好的标记符。（语言学家不同意单词的定义，所以你总是会得到一个近似值。）

Answer 2

你可以用两个变量。一个用来记录某人在说什么，另一个用来记录讲话人的段落。为了存储段落并关联段落所属的人，请使用dict和person作为键，并使用与该键相关联的段落列表。在

para_dict = dict()
para_type = None

for word in words:
    if ('P1' in word or
        'P2' in word or
        'P3' in word ):
        #extract the part we want leaving off the ':'
        para_type = word[:2]
        #create a dict with a list of lists 
        #to contain each paragraph the person uses
        if para_type not in para_dict:
            para_dict[para_type] = list()
        para_dict[para_type].append(list())
    else:
        #Append the word to the last list in the list of lists
        para_dict[para_type][-1].append(word)

从这里你可以总结出这样说的字数

^{pr2}$

Answer 3

恭喜你用Python开始你的冒险之旅！并不是这篇文章中的所有内容现在都有意义，但请将其添加到书签中，如果以后有帮助的话，可以回来看看。最终你应该尝试从脚本到软件工程，这里有一些想法给你！在

强大的力量带来了巨大的责任感，作为Python开发人员，您需要比其他语言更严格地约束自己，因为这些语言不支持您的工作，并实施“好”的设计。在

我发现从自上而下的设计开始是有帮助的。在

def main():
    text = get_text()
    p_text = process_text(text)
    catalogue = process_catalogue(p_text)

轰！你刚刚写了整个程序，现在你只需要回来填空！当你这样做的时候，它似乎没有那么吓人。就我个人而言，我认为自己不够聪明，不能解决很大的问题，但我擅长解决小问题。所以让我们一次解决一件事。我将从“处理文本”开始。在

^{pr2}$

我还不太清楚这些东西意味着什么，但我知道文本问题往往遵循一种叫做“map/reduce”的模式，这意味着您对某个东西执行和操作，然后将其清理并合并，所以我添加了一些占位符函数。如果有必要的话，我可以回去再加一些。在

现在让我们写下“过程目录”。我本可以写“过程”的，但这听起来很蹩脚。在

def process_catalogue(p_text): 
    speakers = make_catalogue(c_text)
    s_speakers = sum_words_per_paragraph_items(speakers)
    t_speakers = total_word_count(s_speakers)

很酷。还不错。你可能会和我不同，但我认为把项目汇总起来，每段数单词，然后数一数所有单词。在

因此，在这一点上，我可能会制作一个或两个小的“lib”（库）模块来填充剩余的函数。为了使您能够在不担心导入的情况下运行此程序，我将把它全部保存在一个.py文件中，但最终您将学会如何拆分这些文件，使其看起来更好。所以我们就这么做吧。在

#           #
# == process_text == #
#           #

def bundle_dialogue_items(lines):
    cur_speaker = None
    paragraphs = Counter()
    for line in lines:
        if re.match(p, line):
            cur_speaker, dialogue = line.split(':')
            paragraphs[cur_speaker] += 1
        else:
            dialogue = line

        res = cur_speaker, dialogue, paragraphs[cur_speaker]
        yield res


def filter_dialogue_items(lines):
    for name, dialogue, paragraph in lines:
        if dialogue:
            res = name, dialogue, paragraph
            yield res

def clean_dialogue_items(flines):
    for name, dialogue, paragraph in flines:
        s_dialogue = dialogue.strip().split()
        c_dialouge = [clean_word(w) for w in s_dialogue]
        res = name, c_dialouge, paragraph
        yield res

a和一个小助手函数

#          - #
# == aux functions == #
#          - #

to_clean = string.whitespace + string.punctuation
def clean_word(word):
    res = ''.join(c for c in word if c not in to_clean)
    return res

所以它可能不明显，但这个库是作为一个数据处理管道设计的。数据处理有几种方法，一种是流水线处理，另一种是批处理。让我们来看看批处理。在

#            - #
# == process_catalogue == #
#            - #

speaker_stats = 'stats'
def make_catalogue(names_with_dialogue):
    speakers = {}
    for name, dialogue, paragraph in names_with_dialogue:
        speaker = speakers.setdefault(name, {})
        stats = speaker.setdefault(speaker_stats, {})
        stats.setdefault(paragraph, []).extend(dialogue)
    return speakers



word_count = 'word_count'
def sum_words_per_paragraph_items(speakers):
    for speaker in speakers:
        word_stats = speakers[speaker][speaker_stats]
        speakers[speaker][word_count] = Counter()
        for paragraph in word_stats:
            speakers[speaker][word_count][paragraph] += len(word_stats[paragraph])
    return speakers


total = 'total'
def total_word_count(speakers):
    for speaker in speakers:
        wc = speakers[speaker][word_count]
        speakers[speaker][total] = 0
        for c in wc:
            speakers[speaker][total] += wc[c]
    return speakers

所有这些嵌套字典都变得有点复杂了。在实际的生产代码中，我会用一些更可读的类来替换它们（同时添加测试和docstring！！），但我不想让这件事比现在更令人困惑！好吧，为了您的方便，下面是整个事情的组合。在

import pprint
import re
import string
from collections import Counter

p = re.compile(r'(\w+?):')


def get_text_line_items(text):
    for line in text.split('\n'):
        yield line


def bundle_dialogue_items(lines):
    cur_speaker = None
    paragraphs = Counter()
    for line in lines:
        if re.match(p, line):
            cur_speaker, dialogue = line.split(':')
            paragraphs[cur_speaker] += 1
        else:
            dialogue = line

        res = cur_speaker, dialogue, paragraphs[cur_speaker]
        yield res


def filter_dialogue_items(lines):
    for name, dialogue, paragraph in lines:
        if dialogue:
            res = name, dialogue, paragraph
            yield res


to_clean = string.whitespace + string.punctuation


def clean_word(word):
    res = ''.join(c for c in word if c not in to_clean)
    return res


def clean_dialogue_items(flines):
    for name, dialogue, paragraph in flines:
        s_dialogue = dialogue.strip().split()
        c_dialouge = [clean_word(w) for w in s_dialogue]
        res = name, c_dialouge, paragraph
        yield res


speaker_stats = 'stats'


def make_catalogue(names_with_dialogue):
    speakers = {}
    for name, dialogue, paragraph in names_with_dialogue:
        speaker = speakers.setdefault(name, {})
        stats = speaker.setdefault(speaker_stats, {})
        stats.setdefault(paragraph, []).extend(dialogue)
    return speakers


def clean_dict(speakers):
    for speaker in speakers:
        stats = speakers[speaker][speaker_stats]
        for paragraph in stats:
            stats[paragraph] = [''.join(c for c in word if c not in to_clean)
                                for word in stats[paragraph]]
    return speakers


word_count = 'word_count'


def sum_words_per_paragraph_items(speakers):
    for speaker in speakers:
        word_stats = speakers[speaker][speaker_stats]
        speakers[speaker][word_count] = Counter()
        for paragraph in word_stats:
            speakers[speaker][word_count][paragraph] += len(word_stats[paragraph])
    return speakers


total = 'total'


def total_word_count(speakers):
    for speaker in speakers:
        wc = speakers[speaker][word_count]
        speakers[speaker][total] = 0
        for c in wc:
            speakers[speaker][total] += wc[c]
    return speakers


def get_text():
    text = '''BOB: blah blah blah blah
blah hello goodbye etc.

JERRY:.............................................
...............

BOB:blah blah blah
blah blah blah
blah.
BOB: boopy doopy doop
P1: Bla bla bla.
P2: Bla bla bla bla.
P1: Bla bla.
P3: Bla.'''
    text = get_text_line_items(text)
    return text


def process_catalogue(c_text):
    speakers = make_catalogue(c_text)
    s_speakers = sum_words_per_paragraph_items(speakers)
    t_speakers = total_word_count(s_speakers)
    return t_speakers


def process_text(text):
    b_text = bundle_dialogue_items(text)
    f_text = filter_dialogue_items(b_text)
    c_text = clean_dialogue_items(f_text)
    return c_text


def main():

    text = get_text()
    c_text = process_text(text)
    t_speakers = process_catalogue(c_text)

    # take a look at your hard work!
    pprint.pprint(t_speakers)


if __name__ == '__main__':
    main()

因此，对于这个应用程序来说，这个脚本几乎肯定是大材小用了，但关键是要看看可读的、可维护的、模块化的Python代码可能是什么样子。在

可以肯定的是，输出如下：

{'BOB': {'stats': {1: ['blah',
                       'blah',
                       'blah',
                       'blah',
                       'blah',
                       'hello',
                       'goodbye',
                       'etc'],
                   2: ['blah',
                       'blah',
                       'blah',
                       'blah',
                       'blah',
                       'blah',
                       'blah'],
                   3: ['boopy', 'doopy', 'doop']},
         'total': 18,
         'word_count': Counter({1: 8, 2: 7, 3: 3})},
 'JERRY': {'stats': {1: ['', '']}, 'total': 2, 'word_count': Counter({1: 2})},
 'P1': {'stats': {1: ['Bla', 'bla', 'bla'], 2: ['Bla', 'bla']},
        'total': 5,
        'word_count': Counter({1: 3, 2: 2})},
 'P2': {'stats': {1: ['Bla', 'bla', 'bla', 'bla']},
        'total': 4,
        'word_count': Counter({1: 4})},
 'P3': {'stats': {1: ['Bla']}, 'total': 1, 'word_count': Counter({1: 1})}}