Python+scrapy+web scraping：页面未被刮取

Question

我想从这个页面上抓取电影标题：https://www.imdb.com/list/ls055386972/。我编写了以下代码：

import scrapy
from scrapy import Spider
from scrapy.http import Request
import re
import pymysql
import sys
import hashlib
from datetime import *
#import time
import csv
import os
import requests


class MoviesSpider(scrapy.Spider):

  name = 'movies' #name of the spider
  allowed_domains = ['imdb.com/list/ls055386972/']
  start_urls = ['http://imdb.com/list/ls055386972//']


  def parse(self, response):
      #events = response.xpath('//*[@property="url"]/@href').extract()
      links = response.xpath('//h3[@class]/a/@href').extract()

      final_links = []

      for link in links:
          final_link = 'http://www.imdb.com' + link 
          final_links.append(final_link)     

      for final_link in final_links:
          absolute_url = response.urljoin(final_link)
          yield Request(absolute_url, callback = self.parse_movies)

          #process next page url
          #next_page_url = response.xpath('//a[text() = "Next"]/@href').extract_first()
          #absolute_next_page_url = response.urljoin(next_page_url)
          #yield Request(absolute_next_page_url)    

  def parse_movies(self, response):

      title  = response.xpath('//div[@class = "title_wrapper"]/h1[@class]/text()').extract_first()

      yield{
                'title': title,
      } 

但它不会刮到任何东西。我收到以下错误消息：

2019-03-04 18:08:37 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://www.imdb.com/list/ls055386972//> (referer: None)
2019-03-04 18:08:37 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.imdb.com/list/ls055386972//>: HTTP status code is not handled or not allowed
2019-03-04 18:08:37 [scrapy.core.engine] INFO: Closing spider (finished)

打印（“最终链接”）生成到各个电影页的正确链接：

[u'https://www.imdb.com/title/tt0068646/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0108052/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0050083/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0118799/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0060196/?ref_=ttls_li_tt',..........]

Answer 1

不确定关于Scrapy，但如果您使用下面的代码，您将得到所需的输出。你知道吗

import requests
import lxml
from lxml import html

url = "https://www.imdb.com/list/ls055386972/"

r = requests.get(url)
tree = html.fromstring(r.content)
domain = "https://www.imdb.com/"

movies_list = tree.xpath('//div[@class="lister-item-content"]/h3/a')
for i, movie in enumerate(movies_list, 1):
  print(i, movie.text, domain+movie.get('href'))


1 The Godfather https://www.imdb.com//title/tt0068646/?ref_=ttls_li_tt
2 Schindler's List https://www.imdb.com//title/tt0108052/?ref_=ttls_li_tt
3 12 Angry Men https://www.imdb.com//title/tt0050083/?ref_=ttls_li_tt
4 Life Is Beautiful https://www.imdb.com//title/tt0118799/?ref_=ttls_li_tt
5 The Good, the Bad and the Ugly https://www.imdb.com//title/tt0060196/?ref_=ttls_li_tt
6 The Shawshank Redemption https://www.imdb.com//title/tt0111161/?ref_=ttls_li_tt
7 The Pursuit of Happyness https://www.imdb.com//title/tt0454921/?ref_=ttls_li_tt
8 Seven Samurai https://www.imdb.com//title/tt0047478/?ref_=ttls_li_tt
9 The Intouchables https://www.imdb.com//title/tt1675434/?ref_=ttls_li_tt
10 Central Station https://www.imdb.com//title/tt0140888/?ref_=ttls_li_tt
11 Requiem for a Dream https://www.imdb.com//title/tt0180093/?ref_=ttls_li_tt
12 A Beautiful Mind https://www.imdb.com//title/tt0268978/?ref_=ttls_li_tt
13 Hachi: A Dog's Tale https://www.imdb.com//title/tt1028532/?ref_=ttls_li_tt
14 Taken https://www.imdb.com//title/tt0936501/?ref_=ttls_li_tt
15 My Sassy Girl https://www.imdb.com//title/tt0293715/?ref_=ttls_li_tt
16 Amores Perros https://www.imdb.com//title/tt0245712/?ref_=ttls_li_tt
17 The Shining https://www.imdb.com//title/tt0081505/?ref_=ttls_li_tt
18 Apocalypto https://www.imdb.com//title/tt0472043/?ref_=ttls_li_tt
19 Gladiator https://www.imdb.com//title/tt0172495/?ref_=ttls_li_tt
20 Cast Away https://www.imdb.com//title/tt0162222/?ref_=ttls_li_tt
21 The Dark Knight https://www.imdb.com//title/tt0468569/?ref_=ttls_li_tt
22 The Pianist https://www.imdb.com//title/tt0253474/?ref_=ttls_li_tt
23 Titanic https://www.imdb.com//title/tt0120338/?ref_=ttls_li_tt
24 3-Iron https://www.imdb.com//title/tt0423866/?ref_=ttls_li_tt
25 Braveheart https://www.imdb.com//title/tt0112573/?ref_=ttls_li_tt
26 It's a Wonderful Life https://www.imdb.com//title/tt0038650/?ref_=ttls_li_tt
27 Spring, Summer, Fall, Winter... and Spring https://www.imdb.com//title/tt0374546/?ref_=ttls_li_tt
28 Alien https://www.imdb.com//title/tt0078748/?ref_=ttls_li_tt
29 Memories of Murder https://www.imdb.com//title/tt0353969/?ref_=ttls_li_tt
30 The Return https://www.imdb.com//title/tt0376968/?ref_=ttls_li_tt
31 I Saw the Devil https://www.imdb.com//title/tt1588170/?ref_=ttls_li_tt
32 Children of Heaven https://www.imdb.com//title/tt0118849/?ref_=ttls_li_tt
33 A Separation https://www.imdb.com//title/tt1832382/?ref_=ttls_li_tt
34 The Sixth Sense https://www.imdb.com//title/tt0167404/?ref_=ttls_li_tt
35 A Moment to Remember https://www.imdb.com//title/tt0428870/?ref_=ttls_li_tt
36 Departures https://www.imdb.com//title/tt1069238/?ref_=ttls_li_tt
37 Wo de fu qin mu qin https://www.imdb.com//title/tt0235060/?ref_=ttls_li_tt
38 Saving Private Ryan https://www.imdb.com//title/tt0120815/?ref_=ttls_li_tt
39 The Bridge on the River Kwai https://www.imdb.com//title/tt0050212/?ref_=ttls_li_tt
40 Ben-Hur https://www.imdb.com//title/tt0052618/?ref_=ttls_li_tt
41 The Exorcist https://www.imdb.com//title/tt0070047/?ref_=ttls_li_tt
42 The Secret in Their Eyes https://www.imdb.com//title/tt1305806/?ref_=ttls_li_tt
43 Léon: The Professional https://www.imdb.com//title/tt0110413/?ref_=ttls_li_tt
44 The Green Mile https://www.imdb.com//title/tt0120689/?ref_=ttls_li_tt
45 Gran Torino https://www.imdb.com//title/tt1205489/?ref_=ttls_li_tt
46 Kill Bill: Vol. 1 https://www.imdb.com//title/tt0266697/?ref_=ttls_li_tt
47 Jurassic Park https://www.imdb.com//title/tt0107290/?ref_=ttls_li_tt
48 Terminator 2: Judgment Day https://www.imdb.com//title/tt0103064/?ref_=ttls_li_tt
49 Back to the Future https://www.imdb.com//title/tt0088763/?ref_=ttls_li_tt
50 Finding Nemo https://www.imdb.com//title/tt0266543/?ref_=ttls_li_tt

Answer 2

您还没有解析解析函数的起始url，下面是工作代码。你知道吗

class MoviesSpider(BaseSpider):

  name = 'movies' #name of the spider
  allowed_domains = ['imdb.com']
  start_url = 'http://imdb.com/list/ls055386972/'

  def __init__(self):
      super(MoviesSpider, self).__init__()

  def start_requests(self):
      yield Request(self.start_url, callback=self.parse, headers=self.headers)

  def parse(self, response):
      #events = response.xpath('//*[@property="url"]/@href').extract()
      links = response.xpath('//h3[@class]/a/@href').extract()

      final_links = []

      for link in links:
          final_link = 'http://www.imdb.com' + link
          final_links.append(final_link)

      for final_link in final_links:
          absolute_url = response.urljoin(final_link)
          yield Request(absolute_url, callback = self.parse_movies)

          #process next page url
          #next_page_url = response.xpath('//a[text() = "Next"]/@href').extract_first()
          #absolute_next_page_url = response.urljoin(next_page_url)
          #yield Request(absolute_next_page_url)

  def parse_movies(self, response):

      title  = response.xpath('//div[@class = "title_wrapper"]/h1[@class]/text()').extract_first()

      yield{
                'title': title,
      }

Answer 3

你得到一个404，因为你的起始网址是不正确的。您需要删除start_urls中的尾部正斜杠：

start_urls = ['https://www.imdb.com/list/ls055386972/']

另外，您的allowed_domains不正确。它应该只包含域，而不是部分URL：

allowed_domains = ['imdb.com']

请参阅documentation。你知道吗