刮网|靓汤|解析表

Question

在这方面有一些很好的线索（其中一些帮助我达到了这一点），但我似乎不明白为什么我的程序不工作。你知道吗

问题：程序正常工作，但似乎只返回第一行，而它应该在所有表行中循环。你知道吗

我使用的是python3.5

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = "http://www.the-numbers.com/movies/year/2006"

r = requests.get(url)
soup = BeautifulSoup(r.content)

data = []

for table_row in soup.select("table"):
    cells = table_row.find_all(['td'])
    release_date = cells[0].text.strip()
    movie_name = cells[2].text.strip()
    genre_name = cells[3].text.strip()
    production_budget = cells[4].text.strip()
    box_office = cells[5].text.strip()
    movie = {"Release_Date" : release_date, 
             "Movie_Name" : movie_name, 
             "Genre" : genre_name, 
             "Production_Budget" : production_budget, 
             "Box_Office" : box_office}
    data.append(movie)
    print (release_date, movie_name, genre_name, production_budget, box_office)

这将返回2006年1月BloodRayne Action$25000000$2405420，这是正确的，但是我需要表中的所有其他行。你知道吗

如果这个问题很容易解决，那么下一步就是将其放入数据帧中（但在响应中不是必需的）。你知道吗

任何帮助都将不胜感激。你知道吗

Answer 1

可以使用^{}进行一些数据清理：

df = pd.read_html('http://www.the-numbers.com/movies/year/2006', header=0)[0]
df = df.dropna(how='all')
df['Release Date'] = df['Release Date'].ffill()
print (df.head())
    Release Date          Movie   Genre ProductionBudget  \
0  January, 2006            NaN     NaN              NaN   
1      January 6     BloodRayne  Action      $25,000,000   
2      January 6       Fateless   Drama      $12,000,000   
3      January 6  Grandma's Boy  Comedy       $5,000,000   
4      January 6         Hostel  Horror       $4,800,000   

  DomesticBox Officeto Date  Trailer  
0                       NaN      NaN  
1                $2,405,420      NaN  
2                  $196,857      NaN  
3                $6,090,172      NaN  
4               $47,326,473      NaN  

您的原始解决方案：

data = []
#find first table
tab = soup.select("table")[0]
#find all tr elements
rows = tab.find_all(['tr'])
#loop anf find all td
for row in rows:
    cols = row.find_all('td')
    #parse text
    cols = [ele.text.strip() for ele in cols]
    #[:-1] remove last column
    data.append(cols[:-1])

cols = ['Release_Date','Movie_Name','Genre','Production_Budget','DomesticBox']
#[2:] remove first 2 rows
df = pd.DataFrame(data[2:], columns = cols)
print (df.head())
  Release_Date     Movie_Name   Genre Production_Budget  DomesticBox
0    January 6     BloodRayne  Action       $25,000,000   $2,405,420
1                    Fateless   Drama       $12,000,000     $196,857
2               Grandma's Boy  Comedy        $5,000,000   $6,090,172
3                      Hostel  Horror        $4,800,000  $47,326,473
4               Kill the Poor                                     $0