应用特定条件的有效方法

Question

我有出境航班的数据，包括日期、月份、机场等信息。 我想通过每一排，并为每一个，计算从同一个机场起飞的航班数在+-15米。 我的代码似乎可以工作，但速度非常慢（在10万行上，运行大约需要一个小时）。 有没有办法提高效率？ 下面是一个示例文件link 谢谢！你知道吗

time_allowance = 15
close_out = []
i=0
for index, row in df.iterrows():
    i+=1

    idf = df.loc[(df['Origin'] == row['Origin']) &
                 (df['Month'] == row['Month']) &
                 (df['DayofMonth'] == row['DayofMonth']) &
                 (df['DepTime'] < row['DepTime'] + time_allowance) &
                 (df['DepTime'] > row['DepTime'] - time_allowance), :]

    close_out.append(len(idf))   

col_name = 'close_out' + str(time_allowance)
df[col_name] = close_out

Answer 1

你的代码

import pandas as pd

cols = ['Month', 'DayofMonth', 'DayOfWeek', 'DepTime', 'UniqueCarrier', 'Origin', 'Dest', 'Distance']
data = [
  ['c-8', 'c-21', 'c-7', 1934, 'AA', 'ATL', 'DFW', 732],
  ['c-6', 'c-19', 'c-2', 1942, 'AA', 'ATL', 'CLE', 999],
  ['c-6', 'c-19', 'c-2', 1955, 'AA', 'ATL', 'CLE', 111],
  ['c-4', 'c-20', 'c-3', 1548, 'US', 'PIT', 'MCO', 834],
  ['c-9', 'c-2', 'c-5', 1422, 'XE', 'RDU', 'CLE', 416],
  ['c-11', 'c-25', 'c-6', 1015, 'OO', 'DEN', 'MEM', 872],
  ['c-10', 'c-7', 'c-6', 1828, 'WN', 'MDW', 'OMA', 423]
]

df = pd.DataFrame(data=data, columns=cols)
df['close_out15'] = 0

def algo_v1(df):


time_allowance = 15
  close_out = []
  i=0
  for index, row in df.iterrows():
      i+=1

      idf = df.loc[(df['Origin'] == row['Origin']) &
                  (df['Month'] == row['Month']) &
                  (df['DayofMonth'] == row['DayofMonth']) &
                  (df['DepTime'] < row['DepTime'] + time_allowance) &
                  (df['DepTime'] > row['DepTime'] - time_allowance), :]

      close_out.append(len(idf))   

  col_name = 'close_out' + str(time_allowance)
  df[col_name] = close_out

  return df

测试：

#print(algo_v1(df))
#%timeit algo_v1(df)

      Month DayofMonth DayOfWeek  DepTime  ... Origin Dest Distance  close_out15
0   c-8       c-21       c-7     1934  ...    ATL  DFW      732            1
1   c-6       c-19       c-2     1942  ...    ATL  CLE      999            2
2   c-6       c-19       c-2     1955  ...    ATL  CLE      111            2
3   c-4       c-20       c-3     1548  ...    PIT  MCO      834            1
4   c-9        c-2       c-5     1422  ...    RDU  CLE      416            1
5  c-11       c-25       c-6     1015  ...    DEN  MEM      872            1
6  c-10        c-7       c-6     1828  ...    MDW  OMA      423            1

[7 rows x 9 columns]
10 loops, best of 3: 28.4 ms per loop

---

仅仅使用groupby和apply方法的一些基本改进

def filter_and_count(df):
  time_threshold = 15

  for idx, row in df.iterrows():
    row['close_out15'] = df['UniqueCarrier'].loc[
      (df['DepTime'] <= row['DepTime'] + time_threshold)
      & (df['DepTime'] >= row['DepTime'] - time_threshold)
    ].count()


def algo_v2(df):
  df.groupby(['Origin', 'Month', 'DayofMonth']).apply(filter_and_count)

  return df

测试性能

#print(algo_v2(df))
#%timeit algo_v2(df)

  Month DayofMonth DayOfWeek  DepTime  ... Origin Dest Distance  close_out15
0   c-8       c-21       c-7     1934  ...    ATL  DFW      732            1
1   c-6       c-19       c-2     1942  ...    ATL  CLE      999            2
2   c-6       c-19       c-2     1955  ...    ATL  CLE      111            2
3   c-4       c-20       c-3     1548  ...    PIT  MCO      834            1
4   c-9        c-2       c-5     1422  ...    RDU  CLE      416            1
5  c-11       c-25       c-6     1015  ...    DEN  MEM      872            1
6  c-10        c-7       c-6     1828  ...    MDW  OMA      423            1

[7 rows x 9 columns]
100 loops, best of 3: 18.8 ms per loop

仅按某些字段分组，就可以看到约34%的改善。你知道吗

提高性能的下一步

主要有两种选择：

• 继续改进算法的性能
• 并行化

请注意，这两个选项也可以一起工作。你知道吗

提高性能

使用Numba或Ctyhon可能是good options。你知道吗

并行化

Multiprocessing module是另一种选择。另一个更高级的抽象选项是this one。你知道吗