使用登录进行Python web抓取

Question

我试图通过密码保护的网站登录，以便访问受保护的页面，我有电子邮件和密码名称以及csrf-token。但当我尝试访问受保护的页面时，它不允许我，并将我重定向回登录。任何帮助都将非常棒！我试图访问的站点是

https://www.usertesting.com/users/sign_in

import requests
from lxml import html

session_requests = requests.session()

login_url = "https://www.usertesting.com/users/sign_in"
result = session_requests.get(login_url)

tree = html.fromstring(result.text)
authenticity_token = list(set(tree.xpath("//meta[@name='csrf-token']/@content")))[0]

userInfo = {
    "user[email]": "email", 
    "user[password]": "password", 
    "csrf-token": authenticity_token
}

result = session_requests.post(
    login_url, 
    data = userInfo, 
    headers = dict(referer=login_url)
)

url = 'https://www.usertesting.com/my_dashboard'

result = session_requests.get(
    url, 
    headers = dict(referer = url)
)

print result.content

Answer 1

试着看看这个https://kazuar.github.io/scraping-tutorial/来寻找你想要的答案。总之，你需要检查网页，在你开始你的完整的抓取程序之前，你应该编写另一个函数，输入用户名、密码，然后进入网站。完成后，开始完整的脚本编写