使用BS4抓取和解析<script>标记（或者有更好的方法）

Question

我正试图从网站https://www.brewbound.com/breweries上搜集一份带有经纬度的啤酒厂名单。这是我感兴趣的网站源代码部分的样子：

    <script>
var locations = [['Wolf Pack Brewing Company', 44.6620529, -111.0994608, '/breweries/Wolf_Pack_Brewing_Co'],['Defiant Brewing Company', 41.0584046, -74.022847, '/breweries/Defiant_Brewing_Co'],

啤酒厂的名单。每个啤酒厂都列在[]之间，并带有名称、lat、long和网站。我想做的是刮取var locations并从中创建一个数据帧，每个brewery作为一行，列包含列出的信息

我已经能够从带有<script>标签（包括页面的多个部分）的网站上抓取所有内容。我不知道从那里去哪里

    url = "https://www.brewbound.com/breweries"
    r = requests.get(url)
    html_contents = r.text
    html_soup = BeautifulSoup(html_contents, 'html.parser')
    script = html_soup.find_all('script')

这是我用来获取所有<script>标记的代码

Answer 1

BeautifulSoup对<script>标记的内容没有帮助。但是您可以使用re和ast.literal_eval来提取信息：

import re
import requests
from ast import literal_eval
from pprint import pprint

url = "https://www.brewbound.com/breweries"
r = requests.get(url)

l = literal_eval(re.search(r'var locations = (\[.*?\]);', r.text, flags=re.DOTALL)[1])
pprint(l)

印刷品：

[['Wolf Pack Brewing Company',
  44.6620529,
  -111.0994608,
  '/breweries/Wolf_Pack_Brewing_Co'],
 ['Defiant Brewing Company',
  41.0584046,
  -74.022847,
  '/breweries/Defiant_Brewing_Co'],
 ['El Toro Brewing Company',
  37.1465525,
  -121.6219873,
  '/breweries/El_Toro_Brewing_Co'],
 ['Sebago Brewing Company',
  43.679212,
  -70.396424,
  '/breweries/Sebago_Brewing_Co'],

...etc.