我在尝试刮取TiktokAPI时不断出错

2024-04-24 15:53:04 发布

您现在位置:Python中文网/ 问答频道 /正文

当我在chrome上测试tiktok时,我正试图用scrapingbee的API来刮tiktok,它可以运行,并提供json文件,如图所示:

Screenshot of the result of tiktok API Json

使用以下代码将其实现为scrapy后:

# -*- coding: utf-8 -*-
import scrapy
from scrapy.crawler import CrawlerProcess
from urllib.parse import urlencode
import json
from datetime import datetime
from ..items import TiktokscrapyItem


API = 'API key Hidden'
user_accounts = ['tahhan', 'amer_kurkar', '3.mr90', 'deeaali']


def get_url(url):
     payload = {'api_key': API, 'username': url}
     proxy_url = 'https://app.scrapingbee.com/api/v1/store/tiktok/user-feed?' + 
     urlencode(payload)
     return proxy_url


class tiktokSpider(scrapy.Spider):
     name = 'tiktok'
     allowed_domains = ['app.scrapingbee.com']
     custom_settings = {'CONCURRENT_REQUESTS_PER_DOMAIN': 1}
     custom_settings = {'FEEDS':{'poststoday.csv':{'format':'csv'}}}

def start_requests(self):
    for sss in user_accounts:
        url = f'&{sss}'
        yield scrapy.Request(get_url(url), callback=self.parse)
        
def parse(self, response):
    items = TiktokscrapyItem()
    jsonresponse = json.loads(response.text)
    items['Bio'] = jsonresponse["Bio"]  
    yield items

我一直收到这个错误:500内部服务器错误

有什么想法吗?请帮忙,谢谢


Tags: offromimportselfapijsonurlparse