使用bs4、python进行google搜索

Question

我想在python脚本中通过Google搜索找到“Spotlight 29 casino address”的地址。为什么我的代码不能正常工作？在

from bs4 import BeautifulSoup
# from googlesearch import search
import urllib.request
import datetime
article='spotlight 29 casino address'
url1 ='https://www.google.co.in/#q='+article
content1 = urllib.request.urlopen(url1)
soup1 = BeautifulSoup(content1,'lxml')
#print(soup1.prettify())
div1 = soup1.find('div', {'class':'Z0LcW'}) #get the div where it's located
# print (datetime.datetime.now(), 'street address:  ' , div1.text)
print (div1)

Pastebin Link

Answer 1

如果你想得到谷歌搜索结果。Selenium with Python是更简单的方法。在

下面是简单的代码。在

from selenium import webdriver
import urllib.parse
from bs4 import BeautifulSoup

chromedriver = '/xxx/chromedriver' #xxx is chromedriver in your installed path
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(" headless")
driver = webdriver.Chrome(chromedriver, chrome_options=chrome_options)

article='spotlight 29 casino address'
driver.get("https://www.google.co.in/#q="+urllib.parse.quote(article))
# driver.page_source  <  html source, you can parser it later.
soup = BeautifulSoup(driver.page_source, 'lxml')
div = soup.find('div',{'class':'Z0LcW'})
print(div.text)
driver.quit()

Answer 2

Google使用javascript呈现来达到这个目的，这就是为什么你不能用urllib.request.urlopen. 在

作为解决方案，您可以使用selenium-python库来模拟浏览器。使用“pip Install selenium”控制台命令进行安装，然后这样的代码可以工作：

from bs4 import BeautifulSoup
from selenium import webdriver


article = 'spotlight 29 casino address'
url = 'https://www.google.co.in/#q=' + article
driver = webdriver.Firefox()
driver.get(url)
html = BeautifulSoup(driver.page_source, "lxml")

div = html.find('div', {'class': 'Z0LcW'})
print(div.text)