python的新特性,来自php。我想刮一些网站使用刮痧,并通过教程和简单的脚本很好。现在写真正的交易会出现这样一个错误:
Traceback (most recent call last):
File "C:\Users\Naltroc\Miniconda3\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks current.result = callback(current.result, *args, **kw)
File "C:\Users\Naltroc\Documents\Python Scripts\tutorial\tutorial\spiders\quotes_spider.py", line 52, in parse self.dispatchersite
TypeError: thesaurus() missing 1 required positional argument: 'response'
当调用shell命令scrapy crawl words
时,scray会自动实例化一个对象。在
据我所知,self
是任何类方法的第一个参数。调用类方法时,不将self
作为参数传递,而是将变量发送给它。在
首先称为:
# Scrapy automatically provides `response` to `parse()` when coming from `start_requests()`
def parse(self, response):
site = response.meta['site']
#same as "site = thesaurus"
self.dispatcher[site](response)
#same as "self.dispatcher['thesaurus'](response)
那么
^{pr2}${cd4>应该与这个调用相同。parse
显然是将{
完整代码:
import scrapy
class WordSpider(scrapy.Spider):
def __init__(self, keyword = 'apprehensive'):
self.k = keyword
name = "words"
# Utilities
def make_csv(self, words):
csv = ''
for word in words:
csv += word + ','
return csv
def save_words(self, words, fp):
with ofpen(fp, 'w') as f:
f.seek(0)
f.truncate()
csv = self.make_csv(words)
f.write(csv)
# site specific parsers
def thesaurus(self, response):
filename = 'thesaurus.txt'
words = ''
print("in func self is defined as ", self)
ul = response.css('.relevancy-block ul')
for idx, u in enumerate(ul):
if idx == 1:
break;
words = u.css('.text::text').extract()
print("words is ", words)
self.save_words(filename, words)
def oxford(self):
filename = 'oxford.txt'
words = ''
def collins(self):
filename = 'collins.txt'
words = ''
# site/function mapping
dispatcher = {
'thesaurus': thesaurus,
'oxford': oxford,
'collins': collins,
}
def parse(self, response):
site = response.meta['site']
self.dispatcher[site](response)
def start_requests(self):
urls = {
'thesaurus': 'http://www.thesaurus.com/browse/%s?s=t' % self.k,
#'collins': 'https://www.collinsdictionary.com/dictionary/english-thesaurus/%s' % self.k,
#'oxford': 'https://en.oxforddictionaries.com/thesaurus/%s' % self.k,
}
for site, url in urls.items():
print(site, url)
yield scrapy.Request(url, meta={'site': site}, callback=self.parse)
你的代码周围有很多微小的错误。我特意清理了一下,遵循了一些常见的python/scrapy习语:)
相关问题 更多 >
编程相关推荐