“record array”vs“structured array”vs“recarray”：对于其中一列是整数，另一列是可变长度整数的数据：

Question

这是这个问题的后续行动

NumPy "record array" or "structured array" or "recarray"

我不知道哪一个最适合我的情况

对于我的数据，一列是int，另一列是可变长度（2-150）的int批

下面是下载一小段（10 mbs）数据并在Pandas中打开的代码

import requests
import pickle
import numpy as np
import pandas as pd

def download_file_from_google_drive(id, destination):
    URL = "https://docs.google.com/uc?export=download"

    session = requests.Session()

    response = session.get(URL, params = { 'id' : id }, stream = True)
    token = get_confirm_token(response)

    if token:
        params = { 'id' : id, 'confirm' : token }
        response = session.get(URL, params = params, stream = True)

    save_response_content(response, destination)    

def get_confirm_token(response):
    for key, value in response.cookies.items():
        if key.startswith('download_warning'):
            return value

    return None

def save_response_content(response, destination):
    CHUNK_SIZE = 32768

    with open(destination, "wb") as f:
        for chunk in response.iter_content(CHUNK_SIZE):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)

download_file_from_google_drive('1-0R28Yhdrq2QWQ-4MXHIZUdZG2WZK2qR', 'sample.pkl')

sampleDF = pd.read_pickle('sample.pkl')

sampleDF['totalCites2'] = sampleDF['totalCites2'].apply(lambda x: np.array(x))

这是一个笔记本，用户不需要下载任何东西到他们的系统上

https://colab.research.google.com/drive/1kaaYk5_xbzQcXTr_DhjuWQT_3S4E-rML