使用Pandas为大型数据集嵌套for循环

Question

我正在进行数据分析，我必须生成直方图。我的代码有7个以上嵌套的for循环。每个嵌套循环按类别中的唯一值过滤数据帧，以形成子类别的新数据帧，然后像前面一样进一步拆分。每天有大约40万条记录。我要处理过去30天的记录。结果是为最后一个不可拆分类别的值（只有一个数字列）生成直方图。如何降低复杂性？有其他方法吗？在

for customer in data_frame['MasterCustomerID'].unique():
    df_customer = data_frame.loc[data_frame['MasterCustomerID'] == customer]
    for service in df_customer['Service'].unique():
        df_service = df_customer.loc[df_customer['Service'] == service]
        for source in df_service['Source'].unique():
            df_source = df_service.loc[df_service['Source'] == source]
            for subcomponent in df_source['SubComponentType'].unique():
                df_subcomponenttypes = df_source.loc[df_source['SubComponentType'] == subcomponent]
                for kpi in df_subcomponenttypes['KPI'].unique():
                    df_kpi = df_subcomponenttypes.loc[df_subcomponenttypes['KPI'] == kpi]
                    for device in df_kpi['Device_Type'].unique():
                        df_device_type = df_kpi.loc[df_kpi['Device_Type'] == device]
                        for access in df_device_type['Access_type'].unique():
                            df_access_type = df_device_type.loc[df_device_type['Access_type'] == access]
                            df_access_type['Day'] = ifweekday(df_access_type['PerformanceTimeStamp'])

Answer 1

您可以使用pandas.groupby查找不同级别列的唯一组合（请参见here和here），然后在按每个组合分组的数据帧上循环。有大约4000个组合，所以在取消注释下面的直方图代码时要小心。在

import string
import numpy as np, pandas as pd
from matplotlib import pyplot as plt

np.random.seed(100)

# Generate 400,000 records (400 obs for 1000 individuals in 6 columns)
NIDS = 1000; NOBS = 400; NCOLS = 6

df = pd.DataFrame(np.random.randint(0, 4, size = (NIDS*NOBS, NCOLS)))
mapper = dict(zip(range(26), list(string.ascii_lowercase)))
df.replace(mapper, inplace = True)

cols = ['Service', 'Source', 'SubComponentType', \
    'KPI', 'Device_Type', 'Access_type']
df.columns = cols

# Generate IDs for individuals
df['MasterCustomerID'] = np.repeat(range(NIDS), NOBS)

# Generate values of interest (to be plotted)
df['value2plot'] = np.random.rand(NIDS*NOBS)

# View the counts for each unique combination of column levels
df.groupby(cols).size()

# Do something with the different subsets (such as make histograms)
for levels, group in df.groupby(cols):
    print(levels)

    # fig, ax = plt.subplots()
    # ax.hist(group['value2plot'])
    # ax.set_title(", ".join(levels))
    # plt.savefig("hist_" + "_".join(levels) + ".png")
    # plt.close()