如何从网站提取危险边缘问题和答案？

Question

这是我目前写的一个脚本，虽然有点乱。你可以在这里查看：http://pastebin.com/prpdJXsq

#Jeopardy!
#Goal is to create a list of lists ie.
#[[Category 1, Question 1, Answer 1], [Category 1, Question 2, Answer 2]]
#First iteration will just be Q


import urllib.request, re

Question = []

first_game_id = 3458
last_game_id = 3713

for gameid in range(first_game_id, last_game_id):
    webpageid = "http://www.j-archive.com/showgame.php?game_id=" + str(gameid)
    temp=urllib.request.urlopen(webpageid)
    webpage=temp.read()
    temp.close()
    for line in webpage:
        if question != None:
            Question.append(question)
print(Question)

#wrong.  ??? = figure out which re to insert?

question = re.match('clue_text\"></td>')
answer= re.match'correct_response&quot;&gt;???&'


#trying to use re match and compile to match the string and output tuple?
import urllib.request, re
webpageid = "http://www.j-archive.com/showgame.php?game_id=" + str(3713)
temp=urllib.request.urlopen(webpageid)
webpage=temp.read()
temp.close()

question=re.compile(r'clue_text">*?</td>')

Question = []
##
##for line in webpage:
##    print(line)
##
##    if question.match(line) != None:
##        Question.append(question)
##
##print(Question)

我还是个新手，最多也就是个初学者，正在尝试写一个Python脚本，从这个很棒的网站提取每一个《危险边缘》的问题和答案：http://www.j-archive.com/showseason.php?season=27

我的大致思路是参考我在这里找到的伪代码，针对一个类似的问题，但我只做到这一步：Jeopardy questions in Excel or other database format?

任何建设性的批评或者直接的嘲讽我都非常欢迎。

Answer 1

我建议你使用 lxml，并利用它的 XPath 支持：

import lxml.html

doc = lxml.html.parse('http://www.j-archive.com/showgame.php?game_id=1')
# get all td's with class="clue_text", these are the clues
clues = doc.xpath('//td[@class="clue_text"]')
# create a dict of clue_id, clue_text
clues_by_id = dict((x.attrib['id'], x.text) for x in clues)