无法从使用scrapy中提取数据

Question

我使用了各种方法从angel.co中提取数据

但仍然无法使用每次获取空列表时的数据进行刮取

results = self.driver.find_elements_by_css_selector(".results > div") for result in results: name = result.find_element_by_css_selector(".name") print(name.text)

另一个是

soup = BeautifulSoup(response.body) val = soup.findAll('div.name')

for post in response.xpath('.//div[@class="base startup"]'): item = {} item['title'] =post.xpath('.//div[@class="name"]//text()').extract()[0] print item 这些都是我试过的，如果有什么建议，那就帮我把这一页刮下来 链接到完整的蜘蛛是 here

Answer 1

您需要wait才能加载搜索结果，然后才能提取它们：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(self.driver, 10)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".startup")))

results = self.driver.find_elements_by_css_selector(".results > div")
for result in results:
    name = result.find_element_by_css_selector(".name")
    print(name.text)