我需要用javascript解析来自多个站点的新闻,并使用selenium+PhantomJS。但是在这些网站上有视频,对我来说毫无用处,我根本不需要它们。(有人建议我使用selenium+Chrome或selenium+Firefox,但在解析过程中我不需要打开任何窗口)。你知道吗
这些视频根据站点的逻辑自动开始播放,最后抛出异常http.client.RemoteDisconnected: Remote end closed connection without response
。你知道吗
我认为它抛出,因为我的互联网是非常缓慢的,视频不能满载它。你知道吗
我怎样才能避免这个问题?你知道吗
selenium或PhantomJS中是否存在任何内容限制?你知道吗
完全回溯:
File "viralnova/viralnova.py", line 101, in parse_viralnova
_parse_post_link(postlinktest, driver)
File "viralnova/viralnova.py", line 9, in _parse_post_link
driver.get(post_link)
File "/Users/user/anaconda/envs/env/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 309, in get
self.execute(Command.GET, {'url': url})
File "/Users/user/anaconda/envs/env/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 295, in execute
response = self.command_executor.execute(driver_command, params)
File "/Users/user/anaconda/envs/env/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 464, in execute
return self._request(command_info[0], url, body=data)
File "/Users/user/anaconda/envs/env/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 526, in _request
resp = opener.open(request, timeout=self._timeout)
File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 1346, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 1321, in do_open
r = h.getresponse()
File "/Users/user/anaconda/envs/env/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
File "/Users/user/anaconda/envs/env/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/Users/user/anaconda/envs/env/lib/python3.6/http/client.py", line 266, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
代码在这里
def _parse_post_link(post_link, driver):
try:
driver.get(post_link)
except Exception:
return None
post_page_soup = Soup(driver.page_source, "lxml")
title = post_page_soup.find('div', attrs={'class': 'post-box-detail article'}).h2.text
print(title)
def parse_viralnova(to_csv=True):
driver = webdriver.PhantomJS("/Users/user/.phantomjsdriver/phantomjs")
postlinktest = 'http://www.viralnova.com/restroom-design-fails/'
_parse_post_link(postlinktest, driver)
如果只是解析您要查找的文本内容,您可以考虑只使用Python and BeautifulSoup。你不应该以这种方式触发浏览器中的任何东西,因为你根本不会使用一个(你提到你不需要打开窗口),同时,解决方案会更快,没有浏览器开销。你知道吗
如果确实需要加载一些javascript,也可以尝试使用dryscape。你知道吗
相关问题 更多 >
编程相关推荐