允许同时下载数千个文件的python包
blackfeed的Python项目详细描述
黑饲料
BlackFeed is a micro python library that allows you download and upload files concurrently. You can download your files locally but you can also upload them to your cloud without writing them to disk.
所需包装
Installed automatically with pip
- 请求
- boto3型
安装
pip install blackfeed
使用
下载文件并上传到AWS S3 要使其工作，必须配置AWS CLI
^{pr2}$下载状态为
如果不想重新下载同一个文件两次，加载状态可能会很有用。在
fromblackfeed.downloaderimportDownloaderfromblackfeed.adapter.s3importS3Adapterqueue=[...]downloader=Downloader(S3Adapter(bucket='bucketname'),multi=True,stateless=False,state_id='filename')# You can add a callback function if needed# This function will be called after each bulk is processeddefcallback(responses):# response: {# 'destination': destination of the file can be local or can be S3 key,# 'url': URL from where the file was downloaded,# 'httpcode': HTTP code returned by the server,# 'status': True|False,# 'content-type': Mime type of the downloaded resource Example: image/jpeg# }# responses: response[]pass# Your logicdownloader.set_callback(callback)downloader.load_states('filename')# This will load states from "filename.txt"downloader.process(queue)stats=downloader.get_stats()# Statistics
弹性下载器
Let's you to download/retrieve files from FTP, SFTP and HTTP/S servers easily.
示例
从FTP下载文件
fromblackfeed.elasticdownloaderimportElasticDownloaderuri='ftp://user:password@ftp.server.com/path/to/file.csv'retriever=ElasticDownloader()res=retriever.download(uri,localpath='/tmp/myfile.csv')# localfile is optional# .download() function returns False if there was an error or return the local path of the downloaded file if it was a success.print(res)
/tmp/myfile.csv
从FTP
检索文件的二进制内容fromblackfeed.elasticdownloaderimportElasticDownloaderuri='ftp://user:password@ftp.server.com/path/to/file.csv'retriever=ElasticDownloader()res=retriever.retrieve(uri)# Return type: io.BytesIO | Falsewithopen('/tmp/myfile.csv','wb')asf:f.write(res.getvalue())
ElasticDownloader可以自动处理FTP、SFTP和HTTP URI。 使用方法download在本地下载文件，并使用retrieve方法获取文件的二进制内容。在
- 项目
标签：
欢迎加入QQ群-->： 979659372
