将所有csv文件从s3转换为parqu

2024-03-29 01:12:44 发布

您现在位置:Python中文网/ 问答频道 /正文

我要做的是将所有文件从S3(AWS存储)转换为parquet格式,然后重新保存到S3中。在

我无法从s3转换所有文件。请帮帮我!!在

import boto3
import pandas as pd
import pyarrow as pa
from s3fs import S3FileSystem
import pyarrow.parquet as pq

s3 = boto3.client('s3',region_name='us-east-2')

obj = s3.get_object(Bucket='dstest-s3', Key='dstest/movies.csv')
df = pd.read_csv(obj['Body'])

table = pa.Table.from_pandas(df)

output_file = "s3://dstest-s3/dstest/parquetconversion1.parquet"
s3 = S3FileSystem()

pq.write_to_dataset(table=table, root_path=output_file, filesystem=s3)

print("File converted from CSV to parquet completed")

Tags: 文件fromimportpandass3astableboto3
1条回答
网友
1楼 · 发布于 2024-03-29 01:12:44

基本上,您需要使用list_objects_v2从bucket中获取所有密钥,然后迭代这些密钥,下载、转换和上传。在

s3 = bt3.get_client('s3')
response = s3.list_objects_v2(
        Bucket='dstest-s3',
        Prefix='dstest/'
)

for s3_obj in response['Contents']:
    obj = s3.get_object(Bucket='dstest-s3', Key=s3_obj['Key'])
    # Do your converting, and uploading here

相关问题 更多 >