Selenium python内容加载约束的类型

2024-03-29 12:47:23 发布

您现在位置:Python中文网/ 问答频道 /正文

我需要用javascript解析来自多个站点的新闻,并使用selenium+PhantomJS。但是在这些网站上有视频,对我来说毫无用处,我根本不需要它们。(有人建议我使用selenium+Chrome或selenium+Firefox,但在解析过程中我不需要打开任何窗口)。你知道吗

这些视频根据站点的逻辑自动开始播放,最后抛出异常http.client.RemoteDisconnected: Remote end closed connection without response。你知道吗

我认为它抛出,因为我的互联网是非常缓慢的,视频不能满载它。你知道吗

我怎样才能避免这个问题?你知道吗

selenium或PhantomJS中是否存在任何内容限制?你知道吗

完全回溯:

File "viralnova/viralnova.py", line 101, in parse_viralnova
    _parse_post_link(postlinktest, driver)
  File "viralnova/viralnova.py", line 9, in _parse_post_link
    driver.get(post_link)
  File "/Users/user/anaconda/envs/env/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 309, in get
    self.execute(Command.GET, {'url': url})
  File "/Users/user/anaconda/envs/env/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 295, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/Users/user/anaconda/envs/env/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 464, in execute
    return self._request(command_info[0], url, body=data)
  File "/Users/user/anaconda/envs/env/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 526, in _request
    resp = opener.open(request, timeout=self._timeout)
  File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 1346, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 1321, in do_open
    r = h.getresponse()
  File "/Users/user/anaconda/envs/env/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/Users/user/anaconda/envs/env/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/Users/user/anaconda/envs/env/lib/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

代码在这里

def _parse_post_link(post_link, driver):
    try:
        driver.get(post_link)
    except Exception:
        return None

    post_page_soup = Soup(driver.page_source, "lxml")
    title = post_page_soup.find('div', attrs={'class': 'post-box-detail article'}).h2.text
    print(title)

def parse_viralnova(to_csv=True):
    driver = webdriver.PhantomJS("/Users/user/.phantomjsdriver/phantomjs")
    postlinktest = 'http://www.viralnova.com/restroom-design-fails/'
    _parse_post_link(postlinktest, driver)

Tags: inpyenvhttprequestlibdriverselenium
1条回答
网友
1楼 · 发布于 2024-03-29 12:47:23

如果只是解析您要查找的文本内容,您可以考虑只使用Python and BeautifulSoup。你不应该以这种方式触发浏览器中的任何东西,因为你根本不会使用一个(你提到你不需要打开窗口),同时,解决方案会更快,没有浏览器开销。你知道吗

如果确实需要加载一些javascript,也可以尝试使用dryscape。你知道吗

相关问题 更多 >