从多个网页收集格式化内容

Question

我在做一个研究项目，需要一个节目记录的内容来获取数据。问题是，文本是为特定的wiki（Arrested Development wiki）格式化的，而我需要它们是机器可读的。你知道吗

下载所有这些成绩单并重新格式化的最佳方式是什么？Python's HTMLParser是我的最佳选择吗？你知道吗

Answer 1

我用python编写了一个脚本，它将wiki脚本的链接作为输入，然后将文本文件中的脚本的纯文本版本作为输出。我希望这对你的项目有帮助。你知道吗

from pycurl import *
import cStringIO
import re

link = raw_input("Link to transcript: ")
filename = link.split("/")[-1]+".txt"

buf = cStringIO.StringIO()

c = Curl()
c.setopt(c.URL, link)
c.setopt(c.WRITEFUNCTION, buf.write)
c.perform()
html = buf.getvalue()
buf.close()

Speaker = ""
SpeakerPositions = [m.start() for m in re.finditer(':</b>', html)]

file = open(filename, 'w')

for x in range(0, len(SpeakerPositions)):
    if html[SpeakerPositions[x] + 5] != "<":

        searchpos = SpeakerPositions[x] - 1
        char = ""
        while char != ">":
            char = html[searchpos]
            searchpos = searchpos - 1
            if char != ">":
                Speaker += char

        Speaker = Speaker[::-1]
        Speaker += ": "

        searchpos = SpeakerPositions[x] + 5
        char = ""
        while char != "<":
            char = html[searchpos]
            searchpos = searchpos + 1
            if char != "<":
                Speaker += char

        Speaker = Speaker.replace("&#160;", "")
        file.write(Speaker + "\n")
        Speaker = ""

file.close()