使用请求抓取网站链接时出现问题

Question

我正试图从网页上获取一个连接到此L'atelier de willy餐厅的网站链接，但我做不到。你知道吗

Website address

这就是它在页面中的显示方式（在同一块中，餐厅名称以非常粗体的字母显示）：

我试过：

import requests
from bs4 import BeautifulSoup

link = "https://www.tripadvisor.fr/Restaurant_Review-g188644-d14788983-Reviews-Mozart_More_Than_Just_Ribs-Brussels.html"

res = requests.get(link,headers={"User-Agent":"Mozilla/5.0"})
soup = BeautifulSoup(res.text,"lxml")
website = soup.select_one("[class*='website']").get("data-ahref")
print(website)

我得到的输出：

q5aizCJEIWEVtIiYHVLaizCJEIWHEpttVcL4pIaQtipEnV1zS0pIaQaVMSpa1EVTVEEJc

我希望得到：

https://mozart-resto.be/

如何使用请求获取该网站链接？

Answer 1

为什么不签出https://html.python-requests.org/这是为了使用它来解析网页而编写的。你知道吗

Answer 2

该网站正在使用“asdf”-编码器（我不确定这是否是官方名称）。但是使用来自Converting JavaScript code to Python的回复，您应该能够解码这个字符串：

d = "q5aizCJEIWEVtIiYHVLaizCJEIWHEpttVcL4pIaQtipEnV1zS0pIaQaVMSpa1EVTVEEJc"

def asdf(d):

  b = ""
  h = {
    "": ["&", "=", "p", "6", "?", "H", "%", "B", ".com", "k", "9", ".html", "n", "M", "r", "www.", "h", "b", "t", "a", "0", "/", "d", "O", "j", "http://", "_", "L", "i", "f", "1", "e", "-", "2", ".", "N", "m", "A", "l", "4", "R", "C", "y", "S", "o", "+", "7", "I", "3", "c", "5", "u", 0, "T", "v", "s", "w", "8", "P", 0, "g", 0],
    "q": [0, "__3F__", 0, "Photos", 0, "https://", ".edu", "*", "Y", ">", 0, 0, 0, 0, 0, 0, "`", "__2D__", "X", "<", "slot", 0, "ShowUrl", "Owners", 0, "[", "q", 0, "MemberProfile", 0, "ShowUserReviews", '"', "Hotel", 0, 0, "Expedia", "Vacation", "Discount", 0, "UserReview", "Thumbnail", 0, "__2F__", "Inspiration", "V", "Map", ":", "@", 0, "F", "help", 0, 0, "Rental", 0, "Picture", 0, 0, 0, "hotels", 0, "ftp://"],
    "x": [0, 0, "J", 0, 0, "Z", 0, 0, 0, ";", 0, "Text", 0, "(", "x", "GenericAds", "U", 0, "careers", 0, 0, 0, "D", 0, "members", "Search", 0, 0, 0, "Post", 0, 0, 0, "Q", 0, "$", 0, "K", 0, "W", 0, "Reviews", 0, ",", "__2E__", 0, 0, 0, 0, 0, 0, 0, "{", "}", 0, "Cheap", ")", 0, 0, 0, "#", ".org"],
    "z": [0, "Hotels", 0, 0, "Icon", 0, 0, 0, 0, ".net", 0, 0, "z", 0, 0, "pages", 0, "geo", 0, 0, 0, "cnt", "~", 0, 0, "]", "|", 0, "tripadvisor", "Images", "BookingBuddy", 0, "Commerce", 0, 0, "partnerKey", 0, "area", 0, "Deals", "from", "//", 0, "urlKey", 0, "'", 0, "WeatherUnderground", 0, "MemberSign", "Maps", 0, "matchID", "Packages", "E", "Amenities", "Travel", ".htm", 0, "!", "^", "G"]
  }

  #for a in range(len(d)):     ## REMOVE and Change this to a WHILE loop (see below)
  a = 0                        #  Manually initialize your loop
  while a < len(d):
    j = d[a]
    f = j
    list = []

    for key in h:
      list.append(key)

    if (j in list) and (a < len(d)):
       a = a + 1                  ## CANNOT DO THIS, if you a use "for a in.." loop.  So, we'll use a WHILE loop instead
       f = f + d[a]
    else:
       j = ""

    g = getOffset(ord(d[a]))
    if  g < 0 :                   ## REMEMBER TO UPDATE THIS LINE to REMOVE # or type(h[j][g]) is str:
        b = b + f
    else:
        b = b + str(h[j][g])
        #print b                  # REMOVE this line

    a = a + 1                     # Manually increment your WHILE loop
  return b

def getOffset(a):
    if(a >= 97 and a <= 122):
        return(a-61)
    if(a >= 65 and a <= 90):
        return(a-55)
    if(a >= 48 and a <=71):
        return(a-48)
    print ("\n\nERROR\n\n")
    return(-1)

print(asdf(d))

印刷品：

https://mozart-resto.be/mozart-brussel/?utm_source=tripadvisor&utm_medium=referral

编辑（用于选择链接）：

from bs4 import BeautifulSoup
import requests

url = 'https://www.tripadvisor.fr/Restaurant_Review-g188644-d14788983-Reviews-Mozart_More_Than_Just_Ribs-Brussels.html'

soup = BeautifulSoup(requests.get(url).text, 'lxml')

s = soup.select_one('div[data-ahref]:has(span:contains("Site Web"))')
if s:
    site_web = asdf(s['data-ahref']) # this is the decoder function
    print(site_web)

印刷品：

https://mozart-resto.be/mozart-brussel/?utm_source=tripadvisor&utm_medium=referral

Answer 3

不幸的是，我还没有评论，这应该是一个评论：（

---

因此，您的代码正在寻找data_ahref，它应该得到您的输出所显示的内容。 网站来源显示

<div class="is-hidden-mobile blEntry website  ui_link" data-ahref="q5aizCJEIWEVtIiYHVLaizCJEIWHEpttVcL4pIaQtipEnV1zS0pIaQaVMSpa1EVTVEEJc" data-column="2" data-trackingkey="URL_EATERY" data-eventname="bl_contact_website" data-blcontact="URL_HOTEL" onclick="widgetEvCall('handlers.onWebLinkClicked', event, this)"><span class="primary_icon ui_icon laptop"></span><span class="detail ">Site Web</span></div>

它说data-ahref="q5aizCJEIWEVtIiYHVLaizCJEIWHEpttVcL4pIaQtipEnV1zS0pIaQaVMSpa1EVTVEEJc"。这意味着您的代码工作正常（和bs4）。你知道吗

另一件事是，当我点击Site Web链接时，我被带到了this link，而不是你想要的链接。在源代码中找不到您想要的链接。你知道吗

---

那么，你确定你在寻找正确的东西吗？你知道吗

编辑： 看着安德烈·凯斯利的答案，我意识到我答案的第一部分并不重要。我没有意识到data-ahref实际上是一个编码字符串，即url。你知道吗