当我在chrome上测试tiktok时,我正试图用scrapingbee的API来刮tiktok,它可以运行,并提供json文件,如图所示:
Screenshot of the result of tiktok API Json
使用以下代码将其实现为scrapy后:
# -*- coding: utf-8 -*-
import scrapy
from scrapy.crawler import CrawlerProcess
from urllib.parse import urlencode
import json
from datetime import datetime
from ..items import TiktokscrapyItem
API = 'API key Hidden'
user_accounts = ['tahhan', 'amer_kurkar', '3.mr90', 'deeaali']
def get_url(url):
payload = {'api_key': API, 'username': url}
proxy_url = 'https://app.scrapingbee.com/api/v1/store/tiktok/user-feed?' +
urlencode(payload)
return proxy_url
class tiktokSpider(scrapy.Spider):
name = 'tiktok'
allowed_domains = ['app.scrapingbee.com']
custom_settings = {'CONCURRENT_REQUESTS_PER_DOMAIN': 1}
custom_settings = {'FEEDS':{'poststoday.csv':{'format':'csv'}}}
def start_requests(self):
for sss in user_accounts:
url = f'&{sss}'
yield scrapy.Request(get_url(url), callback=self.parse)
def parse(self, response):
items = TiktokscrapyItem()
jsonresponse = json.loads(response.text)
items['Bio'] = jsonresponse["Bio"]
yield items
我一直收到这个错误:500内部服务器错误
有什么想法吗?请帮忙,谢谢
目前没有回答
相关问题 更多 >
编程相关推荐