在groupby（）对象上执行apply（）的次数要比组运行的次数多得多

Question

我继承了一些我正试图优化的pandas代码。一个DataFrame，results，已经用

results = pd.DataFrame(columns=['plan','volume','avg_denial_increase','std_dev_impact', 'avg_idr_increase', 'std_dev_idr_increase'])
for plan in my_df['plan_name'].unique():
    df1 = df[df['plan_name'] == plan]]
    df1['volume'].fillna(0, inplace=True)
    df1['change'] = df1['idr'] - df1['idr'].shift(1)
    df1['change'].fillna(0, inplace=True)
    df1['impact'] = df1['change'] * df1['volume']
    describe_impact = df1['impact'].describe()
    describe_change = df1['change'].describe()
    results = results.append({'plan': plan,
                              'volume': df1['volume'].mean(),
                              'avg_denial_increase': describe_impact['mean'],
                              'std_dev_impact': describe_impact['std'],
                              'avg_idr_increase': describe_change['mean'],
                              'std_dev_idr_increase': describe_change['std']}, 
                             ignore_index=True)

我的第一个想法是将for循环下的所有内容移动到一个单独的函数get_results_for_plan，并使用pandasgroupby()和apply()方法。但事实证明，他的速度更慢。跑步

%lprun -f get_results_for_plan my_df.groupby('plan_name', sort=False, as_index=False).apply(get_results_for_plan)

退货

Timer unit: 1e-06 s

Total time: 0.77167 s
File: <ipython-input-46-7c36b3902812>
Function: get_results_for_plan at line 1

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           def get_results_for_plan(plan_df):
     2        94      33221.0    353.4      4.3      plan = plan_df.iloc[0]['plan_name']
     3        94      25901.0    275.5      3.4      plan_df['volume'].fillna(0, inplace=True)
     4        94      75765.0    806.0      9.8      plan_df['change'] = plan_df['idr'] - plan_df['idr'].shift(1)
     5        93      38653.0    415.6      5.0      plan_df['change'].fillna(0, inplace=True)
     6        93      57088.0    613.8      7.4      plan_df['impact'] = plan_df['change'] * plan_df['volume']
     7        93     204828.0   2202.5     26.5      describe_impact = plan_df['impact'].describe()
     8        93     201127.0   2162.7     26.1      describe_change = plan_df['change'].describe()
     9        93        129.0      1.4      0.0      return pd.DataFrame({'plan': plan,
    10        93      21703.0    233.4      2.8                           'volume': plan_df['volume'].mean(),
    11        93       4291.0     46.1      0.6                           'avg_denial_increase': describe_impact['mean'],
    12        93       1957.0     21.0      0.3                           'std_dev_impact': describe_impact['std'],
    13        93       2912.0     31.3      0.4                           'avg_idr_increase': describe_change['mean'],
    14        93       1783.0     19.2      0.2                           'std_dev_idr_increase': describe_change['std']},
    15        93     102312.0   1100.1     13.3                         index=[0])

我看到的最突出的问题是每行的点击量。组的数目，按

len(my_df.groupby('plan_name', sort=False, as_index=False).groups)

是72。那么，为什么这些线被击中94或93次每？（这可能与this问题有关，但在这种情况下，我希望命中计数是num_groups + 1）

更新：在上面的%lprun调用groupby()中，删除sort=False将第2-6行的行命中率降低到80，其余行的行命中率降低到79。点击率还是比我想象的要高，但好一点。你知道吗

第二个问题：有没有更好的方法来优化这个特定的代码？你知道吗

Answer 1

以下是我在评论中的大意：

def append_to_list():
    l = []
    for _ in range(10000):
        l.append(np.random.random(4))
    return pd.DataFrame(l, columns=list('abcd'))

def append_to_df():
    cols = list('abcd')
    df = pd.DataFrame(columns=cols)
    for _ in range(10000):
        df = df.append({k: v for k, v in zip(cols, np.random.random(4))},
                       ignore_index=True)
    return df

%timeit append_to_list
# 31.5 ms ± 925 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit append_to_df
# 9.05 s ± 337 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

因此，代码最大的好处可能是：

results = []
for plan in my_df['plan_name'].unique():
    df1 = df[df['plan_name'] == plan]]
    df1['volume'].fillna(0, inplace=True)
    df1['change'] = df1['idr'] - df1['idr'].shift(1)
    df1['change'].fillna(0, inplace=True)
    df1['impact'] = df1['change'] * df1['volume']
    describe_impact = df1['impact'].describe()
    describe_change = df1['change'].describe()
    results.append((plan, 
                    df1['volume'].mean(), 
                    describe_impact['mean'],
                    describe_impact['std'], 
                    describe_change['mean'], 
                    describe_change['std']))
results = pd.DataFrame(results, columns=['plan','volume','avg_denial_increase','std_dev_impact', 'avg_idr_increase', 'std_dev_idr_increase'])