我要做的是将所有文件从S3(AWS存储)转换为parquet格式,然后重新保存到S3中。在
我无法从s3转换所有文件。请帮帮我!!在
import boto3
import pandas as pd
import pyarrow as pa
from s3fs import S3FileSystem
import pyarrow.parquet as pq
s3 = boto3.client('s3',region_name='us-east-2')
obj = s3.get_object(Bucket='dstest-s3', Key='dstest/movies.csv')
df = pd.read_csv(obj['Body'])
table = pa.Table.from_pandas(df)
output_file = "s3://dstest-s3/dstest/parquetconversion1.parquet"
s3 = S3FileSystem()
pq.write_to_dataset(table=table, root_path=output_file, filesystem=s3)
print("File converted from CSV to parquet completed")
基本上,您需要使用list_objects_v2从bucket中获取所有密钥,然后迭代这些密钥,下载、转换和上传。在
相关问题 更多 >
编程相关推荐