BeautifulSoup生成不一致的结果

Question

我正在使用BeautifulSoup从Reddit侧边栏中提取一些子Reddit的数据，但是每次运行脚本时，我的结果都会发生很大的变化。你知道吗

具体来说，sidebar_urls中的结果会随着迭代而变化；有时会导致[XYZ.com/abc, XYZ.com/def]，有时只返回[XYZ.com/def]，最后有时返回[]。你知道吗

你知道为什么使用下面的代码会发生这种情况吗？你知道吗

sidebar_urls = []

for i in range(0, len(reddit_urls)):
    req = urllib.request.Request(reddit_urls[i], headers=headers)
    resp = urllib.request.urlopen(req)
    soup = BeautifulSoup(resp, 'html.parser')

    links = soup.find_all(href=True)

    for link in links:
        if "XYZ.com" in str(link['href']):
            sidebar_urls.append(link['href'])

Answer 1

似乎你有时会得到一个没有侧边栏的页面。这可能是因为Reddit将您识别为机器人，并返回一个默认页面，而不是您期望的页面。请考虑在请求页面时使用User-Agent字段标识自己：

reddit_urls = [
    "https://www.reddit.com/r/leagueoflegends/",
    "https://www.reddit.com/r/pokemon/"
]

# Update this to identify yourself
user_agent = "me@example.com"

sidebar_urls = []
for reddit_url in reddit_urls:
    response = requests.get(reddit_url, headers={"User-Agent": user_agent})
    soup = BeautifulSoup(response.text, "html.parser")

    # Find the sidebar tag
    side_tag = soup.find("div", {"class": "side"})
    if side_tag is None:
        print("Could not find a sidebar in page: {}".format(reddit_url))
        continue

    # Find all links in the sidebar tag
    link_tags = side_tag.find_all("a")
    for link in link_tags:
        link_text = str(link["href"])
        sidebar_urls.append(link_text)

print(sidebar_urls)