如何在网页上获得推荐

Question

请看这一页：

https://www.michaelkors.com/anorak-rainbow-swimsuit-belt-bag-the-michael-tote-dylan-sneaker/_/L-MSTR101163

几天前我在Stackoverflow上问了一个问题，有人建议我应该研究一下scrapy splash，以便获得推荐。有了splash，我可以抓取大部分JS，但是，我只能抓取页面底部的推荐。这就是我迄今为止所尝试的：

recommendations = response.xpath("//div[@class ='you-may-also-like-section']/a/@href").getall()

这不会有任何回报

Answer 1

您是否尝试过此选择器：

response.css('div.you-may-also-like-section div.product-tile-container a::attr(href)').extract()

另外，您可以尝试在Splash浏览器中设置等待时间

但如果您选中浏览器->；网络->；XHR，您将发现这个请求https://api.rfksrv.com/search-rec/263221008/3 您应该做的一切就是发出这样的请求，但是对于您的数据，它是从源页面获取的。我建议这样

在卷曲中看起来是这样的：

curl 'https://api.rfksrv.com/search-rec/263221008/3' -H 'Accept: application/json, text/plain, */*' -H 'Referer: https://www.michaelkors.com/anorak-rainbow-swimsuit-belt-bag-the-michael-tote-dylan-sneaker/_/L-MSTR101163' -H 'Origin: https://www.michaelkors.com' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36' -H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyZWdpb24iOiJ1cy1lYXN0LTEiLCJzdWIiOiJhcGlLZXkvN255c1NhcnEiLCJzY29wZSI6eyIyNjMyMjEwMDgiOlsidzZndDQ0OHh1ZyJdfSwic3RhZ2UiOiJwcm9kIiwianRpIjoiNGI1M2MyOTItZDA4Ny00OGExLTkzYTctN2M5MTUzYjM2YWVmIiwiaWF0IjoxNTYzOTM2Nzk5LCJleHAiOjE1NjQwMjM3OTl9.UDqzF9cZHJ7KkCnrChvAV6vupP-gs6Bplv462rGII98' -H 'Content-Type: application/x-www-form-urlencoded'  data '{"data":{"batch":[{"widget":{"rfkid":"pdp1"}},{"widget":{"rfkid":"pdp2"}},{"widget":{"rfkid":"pdp_edt"}}],"context":{"page":{"uri":"/anorak-rainbow-swimsuit-belt-bag-the-michael-tote-dylan-sneaker/_/L-MSTR101163","sku":["126295789","314419197","287779605","287780826","321049671","512500966"],"locale_country":"us","locale_language":"en"},"user":{"uuid":"263221008-ox-ap-4u-1p-vws74v0y7idt0l5q27j4-1563955671571"}},"n_item":12,"content":{},"appearance":{}}}'  compressed