s3iotools 0.0.3

使用量

将本地文件复制到s3并将文件对象从s3下载到本地很容易：

froms3iotoolsimportS3FileObjects3obj=S3FileObject(bucket="my-bucket",key="hello.txt",path="hello.txt")# get started, now we don't have file either on local or on s3ifs3obj.path_obj.exists():s3obj.path_obj.remove()asserts3obj.exists_on_local()isFalseasserts3obj.exists_on_s3()isFalses3obj.path_obj.write_text("hello world",encoding="utf-8)asserts3obj.exists_on_local()isTrues3obj.copy_to_s3()asserts3obj.exists_on_s3()isTrues3obj.path_obj.remove()asserts3obj.exists_on_local()isFalses3obj.copy_to_local()asserts3obj.exists_on_local()isTrue

您可以轻松地操作支持s3的pandas.DataFrame：

importboto3importpandasaspdfroms3iotoolsimportS3Dataframesession=boto3.Session(profile_name="xxx")s3=session.resource("s3")bucket_name="my-bucket"s3df=S3Dataframe(s3_resource=s3,bucket_name=bucket_name)s3df.df=pd.DataFrame(...)s3df.to_csv(key="data.csv")s3df.to_csv(key="data.csv.gz",gzip_compressed=True)s3df_new=S3Dataframe(s3_resource=s3,bucket_name=bucket_name,key="data.csv")s3df_new.read_csv()s3df_new.df# access datas3df_new=S3Dataframe(s3_resource=s3,bucket_name=bucket_name,key="data.csv.gz")s3df_new.read_csv(gzip_compressed=True)s3df_new.df# access data

json io类似。

s3df=S3Dataframe(s3_resource=s3,bucket_name=bucket_name)s3df.df=pd.DataFrame(...)s3df.to_json(key="data.json.gz",gzip_compressed=True)s3df_new=S3Dataframe(s3_resource=s3,bucket_name=bucket_name,key="data.json.gz")s3df_new.read_json(gzip_compressed=True)s3df_new.df# access data

parquet是一种列存储格式，对olap查询非常有效。您可以将数据放在s3上，然后使用aws athena查询拼花文件。s3iotools的拼花IO很简单：

s3df=S3Dataframe(s3_resource=s3,bucket_name=bucket_name)s3df.df=pd.DataFrame(...)s3df.to_parquet(key="data.parquet",compression="gzip")s3df_new=S3Dataframe(s3_resource=s3,bucket_name=bucket_name,key="data.parquet")s3df_new.read_parquet()s3df_new.df# access data

s3iotools不会自动安装pyarrow，您可以使用pip install pyarrow安装它。