<p>如您所知,当我们尝试在Crawlera中使用Scrapy Splash时,我们使用以下lua脚本:</p>
<pre><code>function use_crawlera(splash)
-- Make sure you pass your Crawlera API key in the 'crawlera_user' arg.
-- Have a look at the file spiders/quotes-js.py to see how to do it.
-- Find your Crawlera credentials in https://app.scrapinghub.com/
local user = splash.args.crawlera_user
local host = 'proxy.crawlera.com'
local port = 8010
local session_header = 'X-Crawlera-Session'
local session_id = 'create'
splash:on_request(function (request)
request:set_header('X-Crawlera-Cookies', 'disable')
request:set_header(session_header, session_id)
request:set_proxy{host, port, username=user, password=''}
end)
splash:on_response_headers(function (response)
if type(response.headers[session_header]) ~= nil then
session_id = response.headers[session_header]
end
end)
end
function main(splash)
use_crawlera(splash)
splash:init_cookies(splash.args.cookies)
assert(splash:go{
splash.args.url,
headers=splash.args.headers,
http_method=splash.args.http_method,
})
assert(splash:wait(3))
return {
html = splash:html(),
cookies = splash:get_cookies(),
}
end
</code></pre>
<p>在lua脚本中有一个<code>session_id</code>变量,我非常需要它,但是如何从Scrapy的响应中访问它呢?在</p>
<p>我试过<code>response.session_id</code>或{<cd3>},但都不起作用。在</p>