从特定表元素中抓取特定文本时返回错误数据

2024-03-28 09:46:44 发布

您现在位置:Python中文网/ 问答频道 /正文

多亏了SO和@QHarr,下面的代码可以很好地处理url,比如

https://www.amazon.com/dp/B00FSCBQV2

但这样的网址不起作用-

https://www.amazon.com/dp/B01N1ZD912/

我的结果是-

'R1_NO' :'.zg_hrsr { margin: 0; padding: 0; list-style-type: 
none;}\n.zg_hrsr_item { margin: 0 0 0 10px; }\n.zg_hrsr_rank { 
display:inline-block; width: 80px; text-align: right; }'}'

它应该回来了

R1_NO = 42553 
R1_CAT = Baby Care Products
R2_NO = 6452
R2_CAT = Baby Bathing Products (Health & Household)

这是因为排名数据不在第一行。为了达到预期的效果,需要做些什么?这个脚本是否可以精简/更高效?你知道吗

我试过用bs4来处理选择1,得到文本条,我做什么都不管用。请帮帮我!你知道吗

fields = ['Amazon Best Sellers Rank']

            temp_dict = {}

            for field in fields:
                element = soup.select_one('li:contains("' + field + '")')
                if element is None:
                    temp_dict[field] = 'NA'
                else:
                    if field == 'Amazon Best Sellers Rank':
                        item='NA'
                        item = [re.sub('#|\(','', string).strip() for string in soup.select_one('li:contains("' + field + '")').stripped_strings][1].split(' in ')
                        temp_dict[field] = item
                    else:
                        item = [string for string in element.stripped_strings][1]
                        temp_dict[field] = item.replace('(', '').strip()

            ranks = soup.select('.zg_hrsr_rank')
            ladders = soup.select('.zg_hrsr_ladder')

            if ranks:
                cat_nos = [item.text.split('#')[1] for item in ranks]
            else:
                 cat_nos = ['NA']

            if ladders:
                cats = [item.text.split('\xa0')[1] for item in soup.select('.zg_hrsr_ladder')]
            else:
                cats = ['NA']

            rankings = dict(zip(cat_nos, cats))

            map_dict = {'Amazon Best Sellers Rank': ['R1_NO','R1_CAT']}

            final_dict = {}

            final_dict['R2_NO'] = 'NA'
            final_dict['R2_CAT'] = 'NA'
            final_dict['R3_NO'] = 'NA'
            final_dict['R3_CAT'] = 'NA'
            final_dict['R4_NO'] = 'NA'
            final_dict['R4_CAT'] = 'NA'

            for k,v in temp_dict.items():
                if k == 'Amazon Best Sellers Rank' and v!= 'NA':
                    item = dict(zip(map_dict[k],v))
                    final_dict = {**final_dict, **item}
                elif k == 'Amazon Best Sellers Rank' and v == 'NA':
                    item = dict(zip(map_dict[k], [v, v]))
                    final_dict = {**final_dict, **item}
                else:
                    final_dict[map_dict[k]] = v

            for k,v in enumerate(rankings):
                #print(k + 1, v, rankings[v])
                prefix = 'R' + str(k + 2) + '_'
                final_dict[prefix + 'NO'] = v
                final_dict[prefix + 'CAT'] = rankings[v]

我希望它能处理并返回问题中发布的两个url的值


Tags: noinfieldamazonforitemdictcat
1条回答
网友
1楼 · 发布于 2024-03-28 09:46:44

因此,由于html布局的不同,剥离的字符串导致返回内联css。你可以尝试缩短和使用正则表达式。一个可以收紧正则表达式,但我会等着看,如果你找到失败的案例第一。你知道吗

import requests
from bs4 import BeautifulSoup as bs
import re

links = ['https://www.amazon.com/dp/B00FSCBQV2?th=1','https://www.amazon.com/dp/B01N1ZD912/','https://www.amazon.com/Professional-Dental-Guard-Remoldable-Customizable/dp/B07L4YHBQ4', 'https://www.amazon.com/dp/B0040ODFK4/?tag=stackoverfl08-20']
map_dict = {'Product Dimensions': 'dimensions', 'Shipping Weight': 'weight', 'Item model number': 'Item_No', 'Amazon Best Sellers Rank': ['R1_NO','R1_CAT']}

# This handles when a ranking is from 1 to x,xxx,xxx
p = re.compile(r'#([0-9][0-9,]*)+[\n\s]+in[\n\s]+([A-Za-z&\s]+)')

with requests.Session() as s:
    for link in links:
        r = s.get(link, headers = {'User-Agent': 'Mozilla\5.0'})
        soup = bs(r.content, 'lxml')
        fields = ['Product Dimensions', 'Shipping Weight', 'Item model number', 'Amazon Best Sellers Rank']
        final_dict = {}

        for field in fields:
            element = soup.select_one('li:contains("' + field + '")')
            if element is None:
                if field == 'Amazon Best Sellers Rank':
                    item = dict(zip(map_dict[field], ['N/A','N/A']))
                    final_dict = {**final_dict, **item}
                else:
                    final_dict[map_dict[field]] = 'N/A'
            else:
                if field == 'Amazon Best Sellers Rank':      
                    text = element.text
                    i = 1
                    for x,y in p.findall(text):
                        prefix = 'R' + str(i) + '_'
                        final_dict[prefix + 'NO'] = x  
                        final_dict[prefix + 'CAT'] = y.strip()
                        i+=1
                else:
                    item = [string for string in element.stripped_strings][1]
                    final_dict[map_dict[field]] = item.replace('(', '').strip()
        print(final_dict)

相关问题 更多 >