Pandas数据帧按内存usag拆分

Question

有没有办法将一个pandas数据帧拆分成多个受内存使用限制的数据帧？在

Answer 1

最简单的方法是如果数据帧的列是一致的数据类型（即，不是对象）。你可以举个例子。在

import numpy as np
import pandas as pd
from __future__ import division

df = pd.DataFrame({'a': [1]*100, 'b': [1.1, 2] * 50, 'c': range(100)})

# calculate the number of bytes a row occupies
row_bytes = df.dtypes.apply(lambda x: x.itemsize).sum()

mem_limit = 1024

# get the maximum number of rows in a segment
max_rows = mem_limit / row_bytes

# get the number of dataframes after splitting
n_dfs = np.ceil(df.shape[0] / max_rows)

# get the indices of the dataframe segments
df_segments = np.array_split(df.index, n_dfs)

# create a list of dataframes that are below mem_limit
split_dfs = [df.loc[seg, :] for seg in df_segments]

split_dfs

另外，如果可以按列而不是按行拆分，pandas有一个方便的^{}方法。在

Answer 2

def split_dataframe(df, size):

    # size of each row
    row_size = df.memory_usage().sum() / len(df)

    # maximum number of rows of each segment
    row_limit = size // row_size

    # number of segments
    seg_num = (len(df) + row_limit - 1) // row_limit

    # split df
    segments = [df.iloc[i*row_limit : (i+1)*row_limit] for i in range(seg_num)]

    return segments