如何使用python3.5和beauthoulsoup来刮取href

Question

我想用python3.5和BeautifulSoup从网站https://www.kickstarter.com/discover/advanced?category_id=16&woe_id=23424829&sort=magic&seed=2449064&page=1中获取每个项目的href。在

这是我的准则

#Loading Libraries import urllib import urllib.request from bs4 import BeautifulSoup #define URL for scraping theurl = "https://www.kickstarter.com/discover/advanced?category_id=16&woe_id=23424829&sort=magic&seed=2449064&page=1" thepage = urllib.request.urlopen(theurl) #Cooking the Soup soup = BeautifulSoup(thepage,"html.parser") #Scraping "Link" (href) project_ref = soup.findAll('h6', {'class': 'project-title'}) project_href = [project.findChildren('a')[0].href for project in project_ref if project.findChildren('a')] print(project_href)

Answer 1

试试这样的方法：

import urllib.request
from bs4 import BeautifulSoup

theurl = "https://www.kickstarter.com/discover/advanced?category_id=16&woe_id=23424829&sort=magic&seed=2449064&page=1"
thepage = urllib.request.urlopen(theurl)

soup = BeautifulSoup(thepage)

project_href = [i['href'] for i in soup.find_all('a', href=True)]
print(project_href)

这将返回所有href实例。正如我在您的链接中看到的，很多href标记中都有#。您可以通过一个简单的正则表达式来避免这些链接，或者忽略#符号。在

^{pr2}$

这仍然会给你一些垃圾链接，比如/discover?ref=nav，所以如果你想缩小它的范围，就为你需要的链接使用一个合适的正则表达式。在

编辑：

要解决您在评论中提到的问题：

soup = BeautifulSoup(thepage)
for i in soup.find_all('div', attrs={'class' : 'project-card-content'}):
    print(i.a['href'])